I noted you successfully avoided the phrase "object oriented programming" there. That’s an interesting question, because of course in Clojure you can use existing Java classes, you can implement Java interfaces, you can create new classes to be consumed either by Clojure code or Java code. So classes are fully there, but I think the thing that Clojure brings to that discussion is, that if you think about it, we talked about this a little bit in the first chapter of the book and how in many class-oriented languages the class is the abstraction. It’s the way that you do anything. If you have a bunch of somewhat related static methods that you what to group, you put them in a class. If you have data that you want to mutate with the functions that you want to use to mutate them, you put them in a class. If you have some code that is to be used in several different places, you might put it in a class and then inherit.
So it’s this one abstraction, a class and the one inheritance hierarchy that comes along with that you apply to all the problems, even when it doesn’t really fit. What Clojure does is it gives you each of those features - inheritance hierarchies, bundling method and data, near each other and separates them so you can choose the ones you want. In the second half of your question you asked "How do you go about learning that when that’s not what you are familiar with?" - That takes some mental work, I think. It wasn’t that long ago that people were trying to wrap their hands around the class-based approach; it’s not that old. It’s a different way of thinking about things and it’s almost more a process of unlearning than it is of learning.
The class is not really a great abstraction for some of those things I mentioned. So, once you start to take apart your expectation that the class is the answer to whatever design problem you are confronting, then it tends to flow fairly well. As you think about it, "I didn’t need the whole class, I just needed a group of functions and so I’ll put them in a namespace". It takes some practice, but I think a good book and a helpful online community and a couple of interesting practice problems like maybe http://projecteuler.net can get you a long way.
2. What types of problems is Clojure best suited for?
I hear that question quite a bit and I never know quite how to answer. It’s a general purpose language. I think the biggest drawback, the biggest disconnect between what Clojure is good at and what you might be trying to do; is where the JVM makes sense. The Java Virtual Machine isn’t everywhere or isn’t appropriate for every task and right now that’s the primary implementation. There is a CLR .NET runtime implementation as well, that people say good things about. For example I do a lot with Unix to work at the command line and there are a lot of tools that you want to start up and shut down within half second or less time periods and I haven’t found a way to start up and shut down a Java Virtual Machine that well.
There are utilities out there that sort of work around that problem, like Cake is one. If you look at Cake and Clojure you’ll see some things that they do to keep the virtual machine running all the time and contacted at the command line, but it’s a somewhat leaky abstraction you have to be aware that that’s actually what’s going on. There is that; running in a web browser can be an issue. So there are places where the Java Virtual Machine is either inappropriate or not ideal and where the CLR is also not available. Right now at least, Clojure has a hard time fitting into those spots.
But if you are doing anything that interacts with Java libraries, whether it’s contacting a SQL database or NoSQL database or doing user interface widget stuff or network stuff, any of that. Clojure is great at that and has useful tools to abstract away some of the boilerplate and complexity that often comes along with the Java library. When you get into your core problem space when you are not using libraries, that’s where all the design features, the concurrency features and all that really comes into play.
It’s interesting: the concurrency and parallelism before I started learning about Clojure, I sort of conflated the two. I think Guy Steel’s talk yesterday at the Keynote was good, where he made a clear distinction between them. Clojure has a lot more tooling right now around concurrency than it does parallelism. So concurrency is where you have a bunch of independent threads of execution or resources competing over the same process or memory and so on, as opposed to parallelism where you have several resources, maybe several cores all working on the very same problem in cooperation with each other.
There are a couple of things: Clojure has a function called "pmap", which stands for parallel map which is in the parallelism side and there is some work going forward using the fork/joint framework for doing parallelism. But the things the people talk about the most with Clojure have to do with concurrency. Clojure’s features and syntax: the syntax I’ll get out of the way first because the syntax is, for me at least, is mostly about malleability. You can adjust it to whatever your problem’s base is. So when it comes to concurrency, it removes a lot of the boilerplate work from what you might imagine you would need to do, just like it does for any other kind of specific domain.
The syntax is simple there, just like it is elsewhere. The features are interesting; it’s almost a philosophy, a way of thinking about your problem. Again, separation of concerns - figure out which parts of your problem can be expressed with pure functions and put those in one part of your code and then carefully identify the parts that actually have to mutate and then use Clojure’s reference types - "ref", "var", "atom" and "agents" - to address the specific needs you have where you have mutability issues. It takes a little bit of practice to think about your problem, but once you do, you’ll find I think that even in languages that don’t provide those specific features you can still organize your code that way and reap some of the benefits of it.
You absolutely need them. That’s a great question! I think people frequently dismiss Clojure when they think they are in a context where they don’t need multiple threads. Any time that you have a mutable collection in a local scope, starting local variable or whatever and you pass it to some function that maybe you don’t know very well, maybe somebody else wrote it, it’s in some other library, maybe that library is not documented very well or whatever. You have to be aware that there is a possibility that the function you are calling will change your collection. Hopefully, if they are going to do that, it’s considered questionable and they documented it and you can put a big comment there, "Watch out! This is going to rearrange your indexes", or whatever.
If you are passing in an immutable collection, you just don’t have to worry about it. It’s impossible for the method call you are doing to change your data structure, there is just no way to do it. And there is a lot of potential complexity that is simply wiped away by that simple fact. So it’s a huge win, even in the single-threaded program.
We have plenty of RAM now, but there is still a speed cost associated with copying large amounts of data. So Clojure’s data structure, the persistent data structures that come with Clojure, work very hard to prevent copying more data than necessary. So there is a feature involved there called "structural sharing" where, when you make a change to a collection by adding something to it or removing something, most of the old collection is left alone in memory and a small part of it is recreated with your new values and the parts that you didn’t change point to the old version of the old structure. Then the garbage collector can come along at some point and clean up the small part that you left behind that you didn’t need.
That’s structural sharing and all of Clojure’s persistent data structures do it. Vectors and maps and sets all have trees internally that allow for this and actually if you think about a simple linked list, that’s the easiest way to see how this can work. You mentioned the example of having 5 items and adding 2 more. If you keep a pointer to the head of a list of 5 items and it’s a regular list be it a single linked list, you can create a new head and point to that old list and as long as you promise never to change the old values, you can think of it as two lists. You have the old one, it hasn’t changed at all, and you have the new one and you didn’t copy anything. That same kind of principle then can be extended to trees and allow you to do the hash maps and vectors.
There is a cost, you can’t pretend there is no cost at all, but it’s not multiple times or orders of magnitude of memory or anything. It’s a small amount of overhead, especially on your write operations. Your reads can be at least as fast, sometimes faster because of the cache coherency of not changing things of the fly. They work out well. To help mediate some of the performance cost issues, several of Clojure’s persistence structures support a feature called "transients", which allow you within a controlled context to actually mutate your persistence structures. You might wonder whether that destroys the whole point, but what you do is you make a transient copy (it’s not a full copy, it’s just copying the head), a transient version of an existing persistent structure and now that you’ve got that, you created it locally, nobody else can see it, so it’s safe as long as you don’t let anybody else see it.
There are several operations that do work on regular persistence structures that don’t work on these transients so that you can quickly discover if you accidentally passed it to some function that then tries to do regular persistent things you’ll get an exception right away. The idea is that then within a tight loop, you have this transient that you can mutate with other special rules that apply to what you can and can’t do with it, but it prevents even that little bit of copying to a large extent, so that you can build up from one persistence structure to another. And once you’re done with the transient and it’s in the state you want, you flip a bit on it, because it is transient - you can change it - and once you flip that bit, you have a real persistent structure again.
The process of converting a persistent to a transient and a transient back to a persistent, each of these operations is costing time, but are very fast. There are actually several of Clojure’s built-in functions that do that internally and reap some of the performance benefits there.
When you switch from a persistent to a transient, you don’t switch the existing one, you get a new one. So that is actually a persistent operation - the conversion to transient. It’s interesting: the function names are "transient" and "persistent" that you call these operations. The transient one does not have an exclamation mark on the end because it’s safe. The persistent call, which actually mutates a transient and puts it back into a persistent state, does have the bang because you are actually changing that mutable collection. So the idea is that if somebody did pass in a persistent structure to you and then you want to do transient things on it, you’d call transient and now you have a new one.
That old persistent structure is still there, still persistent, still promising all the things that it did and so still providing the safety that you want for the people calling it. But now you’ve got this new thing that you can mutate more quickly and then, if you want to return that as a persistent, you’re good to go.
For the most part, the def types and protocols are really just about performance. They don’t really allow you to do things that you couldn’t do before, but they give you some tools for reaching Java level performance in even more contexts. It makes it very easy to create certain kinds of classes and adding methods to them. Part of the significant design requirements that I think Rich was applying to himself as he came up with those, was that the interface to def types and especially def records that came along with them as well as the interface to protocols. When you are using objects of those kinds, in as many cases as possible the API is identical if you are using Clojure’s other styles of collecting data and functions together. When you call a protocol method it looks exactly like a regular Clojure function call, for example.
The idea is that if you do want to convert some code that was using some of the older features, because you want some of the performance benefits of records and protocols, you can do it without having to completely rewrite your whole code base. A lot of the code will work exactly as you go into some key points. Actually I talked about that in my expression problem talk a little bit. I talked about how, for example, you can have a code base written using multi-methods and switch it to using protocols and the two or three things that you specifically have to look for and change in order to make that work.
I think that comes to personal development kind of thing. It might be worthwhile. It might be interesting to see whether there are many companies out there that are willing to put effort into training their teams to use languages that they don’t intend to actually use at the company. At least in my experience it seems that most of the people who are exploring those languages are not doing so with the intention of using them tomorrow at work, but because they like them and they are interesting. I mentioned in one of the earlier questions: it’s useful to experiment with learning other languages because you learn features about type inheritance or multi-methods or things that are interesting and can allow you to rephrase the problems you are thinking about in terms that you might not normally do.
C++ is a very flexible language; you can use it in a lot of different ways. So, if you know how to program functionally, you might see that there are some nice functional solutions that you can still use in C++. I think it’s worthwhile and I don’t know of any cotton-dry 1-2-3 step process to learn a language and then borrow its features and how you code in another language, but it all informs the creative process that is software development.
I think it is because software development is a creative process and you approach it with the tools that you are familiar with and Clojure allows you to put some more tools in your toolbox and allows you to think about your problems in new ways. It’s hard to say which languages are most useful for most quickly broadening you horizons (sometimes too quick is a problem as well). But I think that part of Clojure’s attraction for a lot of people is that, if they do come from a Java background, they can bring all of their knowledge of Java libraries and Java development environments and have all of that right there, wherever they need it and so they can start to combine Clojure’s way of thinking about design with all that knowledge they already have when they are writing Clojure code.
Then, when they go back to writing some Java code, again it’s those same libraries, maybe the same task they are trying to accomplish, but they have in their mind now some of the different ways that Clojure encourages you to solve them, so they can apply them pretty directly. It’s not as if there is a completely different sub-stratum of libraries that phrase the entire problems in completely different ways. Maybe it’s more applicable to move from Clojure back to Java then from some other completely different platform.