The next thing for Clojure will be sort of a maintenance release, version 1.4. The primary thing there is bug fixes and probably the major surface enhancement people will see is the extensible reader support. So an extensible reader haslong been asked for, what people wanted was something a lot like what Common Lisp used to do which is essentially give you reader macros which just allow you to commandeer some of the characters and say: "This means this" and I’ve resisted it throughout the history of Clojure because I wanted Clojure to be a strong library language and that capability doesn’t allow for interoperability.
If I say "This character means this" and you say "That character means that" then those two files and applications that are written that need to use both those things now can’t be composed. So I thought about the problem a lot and the answer I came up with was to do extensible reader where you use tags to identify novel things that can be put into data structures that you can read and because those tags are namespaced now they are free of clashes so there will be a few tags that don’t have namespaces which will be built-in. So for instance Clojure reader will gain the ability to read UUIDs with this enhancement and time stamps.
It’s an open system in that you can define your own tag with a namespace qualifier and then have that map to any reading function that you want. So by combining a tag and anything that can already be read, so you can tag a vector literal or tag a map literal or tag a symbol or string, you can enhance what can be read and I think this is the right way to do this. It’s an open system that is extensible, but that is also interoperable.
So I am very excited about it, for instance one of the things that is neat about it is prior to having this there was a way to, for instance, embed in a reader call to a constructor, construct the date when it was read and the problem you have is for instance when you have something like ClojureScript, now this isn’t really a transportable data format anymore, so we’d like it to be possible to write something, write a UUID from Java and read it in ClojureScript and each have their own independent representations of that but understand the semantics of what was transferred. So I think it’s a great feature, not everyone understands the power of it but it’s going to be a critical interoperability feature for Clojure and ClojureScript.
2. So you’ve given up the total power of being the only one who writes reader macros?
Well it’s not reader macros, it’s still less than reader macros. People who wanted reader macros will probably still be upset, but I think it’s a good solution that brings you a lot of the capabilities you’d be looking for it there, probably not as succinctly as you can do with reader macros, but you can write extensible data records and transfer stuff around without transferring Java implementation details around and I think that is an important power that we were missing so far.
3. So these tags are like prefixes to the data, is that it?
Yes it’s just a symbol prefix by Hash tags for extensions. Let’s say you have a handle and you say #mycompany.handle and then if you want to use maps to represent your handles or numbers then you could just put a map or a number. So the tag qualifies the next thing read and that causes your reader function to read it and interpret it as whatever you would like. And you can plug those in both the handlers and the tags.
4. And the reader function returns a Clojure data structure.
It returns whatever data structure you want, so you could read something in and get back at java.util.Date if you wanted, anything you want. And of course depending on the reader, if for instance, dates are a good example and maybe you have two applications and one is written in Clojure and one is written in Java and you have a reader on the Java side so that you can use it as a transfer medium. Maybe you are manipulating dates with Joda Time in one application and java.util.Date in another. You write date, now you are in a reader form that doesn’t dictate what type represents it and then when you read it you can have a different thing plugged in that reads it at something else and that is what is important,I think, to isolate what you are putting in the data files or in the data stream from the representations that end up coming up in memory.
Pods are an example of talking about something before you’ve made it, a feature you are going to be asked about forever, which I actually normally don’t do and because I did that and it’s actually shown up how long the cycle is around getting these ideas completed, I would say that the work I have done on Datomic has informed my thinking about Pods and probably will enable me to complete them, but I hate to continue to talk about them because I haven’t made them yet.
I personally haven’t had a lot of time to work on it after I got the core compiler going, but there is a whole sub community oriented around it and a bunch of people on the Clojure core team who are really enthusiastic about it and have been moving forward in the implementation, the libraries and the tooling. So there is a lot of activity around ClojureScript and I am really impressed with what they are doing.
8. It’s still a subset of Clojure?
That is right. It’s definitely designed to be Clojure and it’s a substantial subset. It doesn’t have concurrency because JavaScript engines don’t have concurrency, but it is Clojure and people have moved nontrivial amounts of Clojure code there without changes. So yes, it’s Clojure on the JavaScript engine, the parts that make sense.
Yes. We tried to fix up some of that, we can’t fix all of it. But it is important that ClojureScript implements Clojure semantics, so in many ways it’s a dramatic improvement over using JavaScript because the semantics there are difficult to ascertain. You don’t miss much. Actually it was a very exciting thing to build ClojureScript because it was the first time we were able to implement the abstractions of Clojure using Clojure’s own protocol and deftype and so ClojureScript is the best example so far of Clojure in Clojure being written entirely in Clojure and using deftype and protocols at the bottom. So the ClojureScript implementation is actually very exciting because as you want to enhance it, you have protocols for all those fundamental things where the Clojure implementation is still a hybrid of Java and Clojure.
10. So it’s basically a big proof of concept of protocols?
It definitely is a proof that they were viable for implementing the abstraction, that they were performant enough for that and that they were independent of the JVM as abstractions which they moved right over and of course the implementation is dramatically different on the JavaScript engine than otherwise. So it’s very exciting and when I get more time to get back to it I anticipate that the work we did on the ClojureScript will inform Clojure in Clojure substantially, if not be the starting point for it.
11. So you are still aiming for actual Clojure in Clojure to appear at some point.
So now you are asking me to talk about something I haven’t made yet. Yes, I think it’s inevitable that that happens. It’s a matter of someone having the time to work on it.
I think so, yes, I think ClojureScript is a good proof of that. The idea behind Avout was to, it’s David Liebke's work. He wanted initially to make a library for Zookeeper for Clojure and because once he had done that he would end up with locks and I said: "Distributed locks are as hard or harder to use than locks are in process, could you take it to the next level, could you take Clojure’s abstractions around state management in the reference system, so Atoms and Refs and put that again on top of that infrastructure?" and that is what he did. So Avout is fundamentally delivering the Clojure state abstractions in a distributed context and that is the idea. So it has Refs and Atoms running against not only Zookeeper now but other kinds of coordination engines and other kinds of storage engines, where you put the values that you are synchronizing, you can swap out. So it’s very interesting.
13. So it’s basically for distributing Clojure, is that sort of the solution?
No. I think when people move to something like Zookeeper usually it’s a very small part of their application, but that they desperately need to be distributed like that, like it would be process coordination or any kind of coordination or mastership of things or shared small things. You don’t want to pile a whole bunch of state in something like a Avout, it’s not a database. But it can be used for those particular kinds of things and it’s one of those things like when you need it, you need it and to have something that works reliably with those nice semantics I think it is a good value proposition. Hopefully you don’t need it that often. But that is true of references in Clojure too. People are surprised actually of how infrequently they need them. So that is what you want, you want to not need it very frequently and when you need it you want it to be sound.