Lindsay Holmwood recently recounted his learning experiences on building Continuous Delivery (CD) for infrastructure services. Holmwood offers advice to enable fast feedback loops, notably of Service Level Agreements (SLA) compliance, and to support small, discrete changes. Holmwood is the creator of Flapjack, a monitoring notification routing and event processing system.
Holmwood defines infrastructure services as "a configuration of software and data that is consumed by other software". This includes, for instance, DNS, databases or monitoring services.
Holmwood asserts that to get quality, not just fast, feedback there are five main issues to think about. Start with the CAP (Consistency, Availability, Partition Tolerance) theorem. Decide if the services you're building should go for consistency or availability. Examples of the former may include databases, of the latter, load-balanced servers. Once you've made that choice, it's time to agree with the infrastructure users the SLAs to commit to. For instance, "95 percentile response time for monitoring metric queries in a one hour window is < 1 second". Third, use your CD pipeline to check your SLA's, whenever possible. You must also have clear, well-defined, interfaces between infrastructure services. Those clear interfaces help to verify interactions between parts, both during testing and normal operation. Finally, Holmwood argues for pushing data to one place to enable immutable infrastructure.
On fast feedback, Holmwood and his team found that placing the continuous integration (CI) system near the IaaS provider (be it on-premises or in the cloud) is crucial to reduce latency times. Holmwood cites a project where the build time reduced from twenty to five minutes, just by moving the CI close to the IaaS API.
If your team mostly does its job through the command-line, context switching to and from the CI web interface can be an user experience blocker. To ensure that everyone complies with the rule of pushing every change through the CD pipeline, Holmwood suggests tools such as jenkins-cli. The build process can then be followed from the command-line:
git push && jenkins start $job && jenkins tail $job
You can even graph some data in the terminal with the help of gnuplot.
Fast feedback requires small, discrete changes. This means the tests must execute fast. Holmwood advises to make each test run in under ten seconds. Given we're in an infrastructure context, making tests run fast also means compromises. So Holmwood automates tests that look for unmistakable problems, such as checking for running services and executing simple queries. Monitoring checks complement the CD pipeline tests to verify end-to-end behaviour.
It should be noted that Continuous Delivery for infrastructure is getting more vendor support over time. Hashicorp's Atlas and the forthcoming Chef Delivery are examples of these developments.