Benjamin Peterson, software engineer at Dropbox, recently shared how Dropbox leverages Bazel to improve their build and deploy experience. Using Bazel, Dropbox was able to scale their continuous integration and deployment pipelines to ensure quick feedback on commits. They achieved this by running only the affected tests within a grouping of commits and selectively pre-declaring which tests are gating to deployments.
Bazel is an open-source build and test tool derived from Google's internal build tool Blaze. Bazel offers a human readable, high-level build language for describing projects. By leveraging dependency analysis and caching, Bazel is able to produce incremental builds in which only the affected resources are rebuilt or tested.
Peterson notes that as more and more tests were added within Dropbox's monorepo, running the entire test suite became too time consuming. By leveraging Bazel's dependency graph, Dropbox was able to "greatly reduce the number of tests executed on most commits while still being correct". This was coupled with rolling up commits into groupings on which the identified tests would be run.
InfoQ chatted with Peterson about his experiences in integrating Bazel at Dropbox.
InfoQ: You indicate that the Dropbox server-side code resides in a monorepo. What led to this decision?
Benjamin Peterson: We previously had more repositories, and we decided to move away from that for many reasons.
Large-scale changes required manually submitting a code review to each affected repository affected. Sharing code was painful, so people tended to copy instead. Changes in one repository could break the tests of another. Each repository tended to develop its own "ecosystem" with a custom coding style, shared code, linters, and policies.
We did a large repository consolidation 5 or 6 years ago. I'm sure the pain points I've described would be much more severe at our size today.
InfoQ: Some argue that the solution to tests taking too long to run within a large repo would be to split out the repo into polyrepos based on service/library boundaries, each running an independent CI pipeline. Was this an option explored at Dropbox before adopting Bazel?
Peterson: No, we like the monorepo too much as discussed above. Releases tests can be thought of as an implementation of project-specific CI pipelines, though.
InfoQ: Have you found drawbacks to the roll-up method you use to avoid running all affected tests on each commit?
Peterson: The main difficulty was usability.
Due to aggregation, it can take longer for commits to become deployable than before. We addressed this latency problem by allowing engineers to trigger the release tests for a project on a specific commit. We also improved the UI in our deployment system to indicate why a commit is or isn't green. It took a few months to get right.
InfoQ: Are all automated tests run through Bazel now? Or have you reserved some tests (say Selenium/UI tests) to be run in alternate ways?
Peterson: Our Selenium tests run under Bazel. A small number of our tests don't work in the Bazel sandbox, but we still build them with Bazel. We'd like to fix them to be compatible with the sandbox in the long run.
InfoQ: In your blog post you mentioned that some of the benefits from implementing Bazel are faster feedback on commits and better resource consumption of build infrastructure. Were there any other benefits from this implementation?
Peterson: Bazel provides a uniform interface to building and testing across projects and programming languages, which is beneficial both to humans and our CI system.
Even if an engineer is unfamiliar with a particular part of the codebase, they'll know that they can build it with
bazel buildand test it withbazel test. Those are core parts of the software engineering workflow that Dropbox engineers only have to learn once.Our CI system only knows about how to invoke Bazel; it doesn't have to concern itself with language-specific package managers, build systems, or test frameworks.
InfoQ: What were the main challenges in integrating Bazel with your build pipeline?
Peterson: Bazel is an opinionated tool. To reap all of its benefits, you usually have to adapt your codebase to how Bazel operates. We invested substantial time in making our builds and tests hermetic. We taught Bazel how to natively build a lot of third-party projects. Almost all of these changes were good, independent of replacing the build system, but the move to using Bazel did prompt us to make them.
Bazel is open-source and freely available for download. Bazel 1.0 was recently released offering greater stability and backward-compatibility guarantees.
