This article contains details on the recently open sourced Buck and xctool, the build tools used internally by Facebook for their native Android and iOS applications.
Buck
Facebook has recently open sourced Buck, their build tool used for native Android applications. Inspired by Google Blaze, Buck was created to deal with the complexity associated with multiple Android libraries and to reduce the build time. After introducing Buck, the four native Android applications developed by Facebook ended up using a single code tree and build tool, making development simpler, more streamlined and with fewer errors. The initial 38 libraries became in time about 500 modules shared among four applications. The building time dropped from 3m 40s to 1m 30s during the first run against the code tree, Buck replacing the initial system based on Ant.
Buck is written in Java with build rules defined initially in Python, later in Jython for a quicker rule evaluation. The tool is currently used for building Android Java code, but the team is in the process of enabling it to build pure Java code, unrelated to Android, and it may be extended to build Python projects and possibly others.
Facebook’s Android code is organized in a tree with independent subtrees forming a directed acyclic graph which enables parallel execution of build rules associated with various subtrees, leading to a shorter build time. Build files, containing the rules, are spread throughout the code tree, one for each Android module existing. A build file is executed only if there is a changed file within the module, and these builds can be executed in parallel. Turning on the parallel capability using 8 threads reduced the unit tests time from 20 minutes to 4, according to Michael Bolin, the creator of Buck. He detailed the build system during a DevCon New York 2013 session.
Buck knows how to deal with multiple versions of a file, as explained by Bolin in a G+ thread:
Another one of the hard problems in building Android apps is that of generating the R.java file. Without getting too deep into the details, you may be building multiple Android apps whose Java files depend on the same class named R. For an ordinary Java class, you would compile it once and any number of other Java classes could depend on it. However, in Android, the version of R that you need to compile against depends on the Android app in which the Java code being compiled will ultimately be included. It would be inefficient to compile a different version of your Java code for every Android app you are building, so Buck employs some clever tricks (creating different R.java files for compile-time as opposed to run-time) to reduce this duplicate compilation.
In the near future, the team intends to add a daemon which monitors for file edits, starting a build immediately when it encounters one.
Buck runs on Mac OS and Linux, the team considering a port to Windows.
Shawn Pearce, creator of Gerrit, proposed using Buck instead of Maven, mentioning speed among other reasons:
buck: clean: 0.452s, full 1m21.083s [*], no-op: 7.145s,
maven: clean: 4.596s, full 2m53.776s, no-op: 59.108s,[*] full build includes downloading all dependencies, time can vary due to remote server performance.
Pearce also noted some of Buck’s drawbacks:
- No Windows support
- No native Maven Central support (added by macros)
- No native GWT, Prolog, or WAR support (added by macros)
- Bootstrap of Buck requires Ant
xctool
xctool is yet another build tool recently open sourced by Facebook, being used for building iOS applications. xctool replaces xcodebuild, featuring the following:
- Can run the same tests as Xcode.app
- The output of builds and test result is captured in JSON, making output parsing unnecessary
- xctool prints messages only when an error is found, and not for every source file as xcodebuild does
We wanted to know why Facebook has built another tool on top of xcodebuild, so we talked to Fred Potter, committer on xctool, asking him why their tool is better:
The biggest benefit of xctool is that it can build and run unit tests from the command-line the same way that Xcode.app does from the GUI. This is really important if you're setting up a continuous integration system for iOS. You want to be able to run the same tests under automation that your developers are running on their local machines, and xcodebuild doesn't build and run tests the same way that Xcode.app does. With Xcode 4, Apple really integrated unit testing into Xcode - there was a new 'Test' action along with 'Build' and 'Run'; with Xcode schemes you could select which of your unit tests were enabled or disabled; and, if you were writing tests that depended on the iOS Simulator (known as application tests), Xcode would automatically launch the simulator and run your tests. These were great improvements, but it seems Apple didn't retrofit these onto xcodebuild and that made automated building and testing a lot more difficult.
Another big problem is reporting of build and test failures. With xcodebuild, you get a giant stream of text output that contains compiler commands, compiler errors and warnings, and test output from OCUnit. If you want to automatically determine which component failed to build or which unit test failed, you have to write your own regex-heavy parser which is what we and others in the iOS community had been doing. It can work, but it's a bit cumbersome. With xctool, we trick xcodebuild and the OCUnit test runners into outputting their build and test results as a structured stream of JSON objects. This makes it easy to display build and test results in whatever format we need. For example, we built a reporter that shows results with attractive, colored output (https://fpotter_public.s3.amazonaws.com/xctool-uicatalog.gif). Others have used this to output test results as JUnit XML which displays nicely in the popular Jenkins build system.
So, we originally built xctool just for our continuous integration system, but many of our developers ended up using it on their local machines. It's handy if you want a command-line workflow for running tests.
