Clojars and Leiningen Automate Library and Dependency Management for Clojure

Using libraries is easy but getting all the right ones onto your system can be a pain. Many languages solve the problem with repositories that host the libraries + metadata and tools that make it easy to pull the right versions of the libraries. In the Ruby space the solution is Ruby Gems, both a tool and packaging format; a default repository hosted at Rubyforge (Gems hosting will soon move to the GemCutter system, which offers more flexibility) makes it єasy to publish and consume Gems. In the Java space, Maven is a popular solution.

Clojars is a new repository that aims to make it easy to share and consume Clojure libraries. A Clojure specific build tool called Leiningen, created by Phil Hagelberg, makes it easy for producers to bundle and push Clojure libraries to Clojars, and for consumers to pull them and make sure they're in the right default place for Clojure programs.

InfoQ talked to Alex Osborne, creator of Clojars, about the motivation behind Clojars, the implementation and tooling.

InfoQ: How does Clojars use Maven? If I install something from Clojars, what do I get? A Maven install of the library? Something else?

Clojars itself is just a repository, so all it really uses Maven for is putting jars and metadata in the structure that most build tools expect. What "installing" something means depends entirely on your build tool. If you use Leiningen or Maven you'll get an install to your local Maven repository. Leingingen just treats the local Maven repository basically as a cache and copies all the dependencies to a "lib" directory under your project where other tools like swank-clojure can find them. It also means you can just set your classpath without any tools or magic, you can just do:

java -cp 'src:classes:lib/*' myproject.main

I quite like the way Ruby's gems can install executables that you can use from the command-line (common examples being "rails" and "rake"). I think we should adopt something like this as well, but I don't have any concrete plans yet. For now though Leiningen plugins cover the basic use-cases for this.

InfoQ: If I want to push a lib to Clojars, what metadata do I have to create and what's the best way to do that?

Clojars just expects a minimal Maven-style pom.xml file. At the bare minimum you will at least need to specify an artifactId, groupId, version and dependencies. It's also preferred to fill in some of the metadata fields such as description, url and license so that the Clojars site will be able to index and search on them, but I'm not going to enforce this. At the moment the search functionality is really basic but I hope to improve it and add a search command to Leiningen so you don't even have to leave your terminal.
The POM syntax is very tedious to type and usually full of XML namespace and schema goop, so unless you're using some IDE plugin that generates it for you I suggest using Leiningen's format, which looks like this:

(defproject myproject "0.1.0"
  :description "An example project."
  :dependencies [[org.clojure/clojure "1.1.0-alpha-SNAPSHOT"]
                 [org.clojure/clojure-contrib "1.0-SNAPSHOT"]
                 [compojure "0.3.1"]])

You can ask Leiningen to export a POM with:

lein pom

Then push it to Clojars with:

scp pom.xml myproject-0.1.0.jar clojars@clojars.org:

The lein-clojars plugin simplifies these two commands to just "lein push", but as you can see there's actually not all that much to simplify. ;-)
I've actually occasionally found myself using "lein pom" to create a template POM for Java projects. You're supposed to use Maven's archetypes for this but I can never remember the huge long command you have to type in order to use them.

InfoQ: What do you say to people who wake up screaming from nightmares about Maven?

*laughs* You're not the only ones! But building simple projects should be simple and I hope Clojars and Leiningen go some way to helping you sleep better.

InfoQ: You're using a Clojure web framework, Compojure, to build the web frontend for Clojars; what's is your experience with Compojure?

I'm used to using Sinatra and Haml with Ruby so Compojure was a natural fit. So far it's worked fine for me. I'm a big fan of minimalist tools that do *just enough* to make common things easy without becoming complex and difficult to troubleshoot.

InfoQ: Where's Clojars hosted and where do the repositories live?

At the moment it's all on a teeny-tiny Linode.com VPS and the repository just lives as flat files on disk (they're just statically served with lighttpd). The web part runs in Jetty and uses an SQLite database for the metadata, search and such. For pushing, I actually just use Nailgun to connect incoming SSH connections to the running app. I implemented an scp server in Clojure (it's a really trivial protocol) which checks that the uploaded POM and jar look okay and if they do has Maven deploy them to the repository. If the repository grows substantially in size I'll probably move the actual jar files to Amazon S3 but I don't expect to need anything beefier for the website and metadata.

InfoQ: You mention Clojars is supposed to look like Gemcutter; how do you approach things like reserving library names, forking, etc? Is it just first come, first serve?

That's a tricky question and I've gone back and forth on it a couple of times and I don't think there's a perfect answer. Fortunately unlike gems, POMs already have a namespacing mechanism (the groupId) so we have a few more options open here.
I find that with Maven, most of the time groupIds just get in the way. I just want to use the official version of "compojure" -- I don't want to have to think about who wrote it or where it's hosted. You often see Maven libraries where up until version 0.3 they might have been hosted on GitHub, so the older versions have a groupId like "com.github.weavejester.compojure" but then they got their own domain so suddenly it changes to "org.compojure". Then you pick the wrong one by mistake and 2 months later realise you've written your project against some ancient version of the library and that explains why the documentation doesn't seem to match. Surely I'm not the only one who does this all the time?

There's been a sort of de facto standard for the canonical version of a library where the groupId is set to the same value as the artifactId. Phil Hagelberg decided to adopt this for Leiningen, so when you just say "compojure" it's a shorthand "compojure/compojure" and I'm doing the same for Clojars. I know the Maven guys frown on this but I think they're being a bit idealistic. I think CPAN, PyPI and RubyGems have shown us that in practice it's not such a big deal, especially when you have a central repository of some sort. Sure there's the occasional problem, but namespacing everything with domains just makes things complex all the time for *library users*, instead of it just being a problem for the *library author* and only when there's actually a name collision.

But on the other hand maybe sometimes I need to tweak it a bit. Maybe the official version of Compojure depends on Jetty and I want to change it to work with Server X, but upstream don't agree and reject my patches. Or perhaps I want to use a library that someone else has written, but they haven't put it in Clojars or Maven Central and I don't want to squat on their name. For these sorts of situations it's really useful to have some sort of namespacing mechanism. So for this kind of usage I'm recommending pushing your jar with a groupId of the form "org.clojars.username". I toyed with the idea of making it just username/project instead of org.clojars.username/project, but I think the ugliness of including the full domain is actually a benefit here. It shows that this is clearly not the official version and encourages using the short form for official releases.

Currently the first time you push a jar with a particular groupId you become the sole owner of it and nobody else can push to it. You can add additional members to a group through the website. So yes first come, first serve. The Clojure community is pretty friendly and mature so I don't expect there to be many problems, but if a dispute does arise and nobody is clearly in the right I'll probably step in and just mark the canonical group name reserved until they sort it out. I think dealing with problems when they occur is better than having some sort of central approvals process which just wastes everyones time and causes submitters and approvers to get antsy with each other.

InfoQ: What tools do you need to interact with Clojars?

I really like what Phil's doing with Leiningen. It's got that same sort of feeling I mentioned before about Compojure and Sinatra, it does "just enough" and doesn't get in your way. But it also just uses Maven-style repositories so you don't have to go and repackage all the Java libraries you want to use in your Clojure programs. So I'll be recommending Leiningen, but really you'll be able to use Clojars with any dependency tool that can download from a Maven repository and most of the Java ones can.

For pushing you don't actually need Leiningen or Maven or anything other than scp, which is installed on virtually every unix system and many Windows developers likely have it installed alongside PuTTY or MSysGit or Cygwin. So if you really want to just manage your dependencies manually and build and push your library with a shell script by all means go ahead.