Alexander Katz, a software engineering intern at Airbnb, recently wrote about improvements their team implemented into Airbnb's deployment system. The inhouse deployment service at Airbnb, Deployboard, handles thousands of deploys daily for the thousands of services that support Airbnb's product. However, the tool did not have any means of enforcing deployment ordering which allowed engineers to, in some cases, push code directly into production.
According to Katz, Deployboard is responsible for the deployments of the services that make up Airbnb's infrastructure and product. Each of these services has their own series of environments allowing for staged rollouts of changes to production. Katz also notes that "at Airbnb, we pride ourselves on giving our engineers a great deal of freedom and flexibility, which in this case means any engineer can deploy to any environment with minimal effort". This strategy is being increasingly used within large organisations, for example, the Netflix engineering team offer a "paved road" platform and deployment system that encouraged best practice, but does not prevent required adaptation.
Katz explains that their deployment tooling lacked guardrails preventing an engineer from deploying their change through the service's stages in the incorrect order. Additionally, the documentation for the service's deployment process tends to be stale. The challenge of keeping documentation up-to-date is a problem that has also been identified within several recent QCon talks, such as Michael Kehoe from LinkedIn who presented "Building production-ready applications". As engineers at Airbnb can freely contribute to services they don't regularly work on, Katz indicates that rectifying this gap became a high priority.
The proposed solution separated out the deployment stages for a service into mandatory and optional targets and presented them in sorted order. Comments were also added to help users understand the stage's purpose. Katz notes that focusing on an intuitive design was vital as "almost all Airbnb engineers interact with Deployboard regularly, these interactions are typically short and more of a means to an end".
Airbnb treats configuration as code, which meant that one of the first steps Katz and the team undertook was defining the specification for the configuration file that defines the deployment pipeline. Katz notes that deploys themselves are handled by pipelines, meaning a pipeline could be used to update its own configuration. Katz elaborates on this by stating that "if a pipeline somehow obtains a broken configuration, it cannot be fixed via direct methods". As such, they had to develop tooling to rescue the pipeline if this occurred.
With these changes, pipelines could be defined and owned by the service teams. As the pipeline definition is itself code, changes to those pipelines can be validated as part of the pipeline's own lifecycle. Katz notes that the team evaluated external solutions to this problem and will likely move to one in the near future. For interested readers, InfoQ has previously covered a number of potential solutions such as Jenkins X, Spring Cloud Pipelines and Spinnaker and Azure Pipelines.