Today at EclipseCon, Shawn Pearce (maintainer of the Gerrit project) gave a talk on Improve your Java Builds with Buck (slides, additional slides). These gave the reasoning behind why the Gerrit project switched from Maven to Buck, ultimately moving the build to use Buck only from Gerrit 2.8 onwards.
Buck is a build system, loosely modelled on the internal Google build system 'blaze', developed by ex-Google employees who are now at Facebook. Whilst at Facebook, the build system was open-sourced under an Apache licence and is available at GitHub. Buck is based on Python, although it is used primarily for compiling Java in Gerrit.
The Buck language is a DSL which uses Python as its underlying build file. The following defines a Java library called 'printy_lib' which depends on Guava, which is located elsewhere on the file system:
java_library(
name = 'printy_lib',
srcs = glob(['src/main/java/**/*.java']),
deps = [':guava'],
)
prebuilt_jar(
name = 'guava',
binary_jar = 'guava.jar',
)
The Gerrit project has added support for resolving JARs from Maven Central, and then materialising them on the local system. The contents of the JARs are identified by a GAV and checksummed with SHA1, which is verified upon download. These extensions aren't available in Buck yet, but they may be contributed in future.
include_defs('//lib/maven.defs')
maven_jar(
name = 'guava',
id = 'com.google.guava:guava:14.0',
sha1 = '67b7be4ee7ba48e4828a42d6d5069761186d4a53',
license = 'Apache2.0',
)
The key advantage of Buck over Maven is that it is fast. This is due to several key advantages; builds are executed in parallel across modules, and the Buck build by default executes 1.25xCPU threads. (Maven and Make also have an ability to run with parallel builds, but still build the modules in parallel). The quoted speed improvements for Gerrit are:
- Clean build
mvn package -Dmaven.{javadoc,test}.skip=true ... 6m50s buck build :gerrit ... 2m 3s buck test --all ... 2m 5s - No-op incremental rebuild
mvn package -Dmaven.{javadoc,test}.skip=true ... 4m44s buck build :gerrit ... 2s - Buck is FAST; Even faster with buckd running in the background
~/buck/bin/buckd time buck :gerrit ... 0.5s
As well as the multi-threaded support by default, Buck speeds up compilation by doing incremental builds and only re-compiling what classes change. Instead of using filesystem timestamps (which may not be reliable on all systems), Buck uses an SHA content match based on the last time the code was compiled to know what the source files have been modified. The hash also includes the hash of the build files, so if the build system is modified then all files are recompiled.
Buck also has detailed knowledge of the Java relationships, so that it knows how to build the relationships between interdependent classes, ensuring that compilation for dependencies is only kicked off if the public API changes, and not if internal methods are changed.
As well as sharing the content for builds on the local system, it's possible to hook up Buck to an Apache Cassandra storage system to permit the libraries to be shared between multiple developers. This acts as a repository mechanism for downloading components which have already been built, or for older versions, and the build system can resolve against these. This permits easy bisecting to find a problem, and instead of having to do a build each time the assets from previous builds can be resolved. As the contents are hashed, the builds are available even if the versions aren't updated or the build hasn't been publicly released.
This increase in speed has enabled more flexible development; given that a build can execute in less than a second (if the Gerrit daemon is running) the development build of Gerrit will detect if it is running in development mode, and kick off a rebuild and reload for every HTTP request. This permits very fast turnaround of patches and development without needing to spend seconds or minutes doing a rebuild and deploy.
Buck is still in development, and perhaps currently lacks some of the community and documentation that Gradle provides. Finally, although Buck is written in Java, it uses Python as its DSL, and the testing is largely done on Mac and Linux machines, which means that Windows support is a second class citizen at the moment.
For more information on Buck, see the FaceBook page describing Buck
