Jez Humble, co-author of "Continuous Delivery" and "Lean Enterprise" and VP at Chef, presented the second Config Management Camp keynote, sharing the principles that enable high throughput and stability and the configuration management practices behind them, using models drawn from the Lean movement.
Humble shared some findings by the Puppet Labs "State of DevOps" report, where the highest correlation with IT performance came from:
- Code, application configuration and system configuration being in a version control system.
- Getting failure alerts from logging and monitoring systems.
- Developers merging code into trunk daily.
- Development and operations teams interacting.
- Developers breaking up large features into small, incremental changes.
With the top predictors of IT performance being:
- Peer-reviewed change approval process.
- Everything under version control.
- Proactive monitoring.
- High trust organization culture.
- Win-win relationship between dev and ops.
In order to have a performance oriented organization, a high trust culture is needed. A generative (performance oriented) organization is characterized by high cooperation, shared risks and implementation of novelty. Agile delivery is important even for firmware companies, for example when the Heartbleed bug was exposed, it highlighted the firmware companies limitations to deliver security updates. In today's world, lead time is important:
- How quickly can service be restored.
- How quickly can users get a critical fix.
- How quickly can a feature value be determined.
Experimenting with new features allows to measure the outcome value. For instance, at Microsoft only one third of well designed and executed experiments to improve a key metric were successful at improving the key metric. At Amazon, short lead times allows them to not deliver things that do not provide value.
Humble defined the DevOps movement as a cross-functional community of practice dedicated to the study of building, evolving and operating rapidly changing, secure, resilient systems at scale, and infrastructure as code as the ability completely reproduce production infrastructure just with code stored in version control.
Configuration management is only going away if you do not care about what is happening. If I look at what is in version control and I do not understand it, then it is no good.
Quoting W. Edwards Deming, quality is something that must be built into the product in the first place, not something to be inspected after the fact, and Continuous Integration is the way to do it.
Continuous Integration is the only way to test at scale. If testing is a different team you are screwed as quality is someone else's problem. Continuous Integration is not running Jenkins against your feature branches and not caring when it breaks.
Doing Continuous Integration right means developing in trunk, not in feature branches that get merged weeks later. That is Continuous Integration by definition.
Humble highlighted the need to build a resilient, scalable and testable architecture:
How do I orchestrate deployments so my architecture starts in the right order? You do not. Automating your broken delivery process is not good, you will end with a broken automated delivery process. You need to build resilient architecture, not trying to get away from it with configuration management.
Orchestration is a smell for a badly architected system.
Amazon is notoriously famous for its service oriented architecture, using cross functional teams to run each service. In Verner Wogels, Amazon CTO, words "you build it, you run it". Using cross functional teams is not needed, just one of the ways that Amazon chose. For Toyota, the structure of the organization is not important, the way people act and react to it, is what is important.
All the time happens. You buy the tool, change does not happen, you end with unused tools. When you buy the tools, but do not change the processes, you have zero impact on the outcome. Nothing changes.
Humble states that there are two architectures validated in the web:
- Microservices, used in Amazon or Netflix: build only the service and test the integration with other services, runtime binding.
- Enormous blocks and backends, used in Google or Facebook: needs Continuous Integration with a ton of tests, compile time binding. Goes through a lot of testing before it is released.
Jez Humble was interviewed by InfoQ about his recently published book "Lean Enterprise", and the slides for his talk are already available.