Continuous delivery means that a software product is production-ready from day one of the project, even if all features not implemented, and the product can be released to users on demand. Authors Jez Humble and David Farley discuss in the Continuous Delivery book, the concept that involves implementing the best practices in all three areas of an organization: process, people and tools to automate the various components of the software development process.
They talk about the problem of delivering software in organizations where the software is deployed manually, deploying production environment only after the development is complete (rather than more frequent deployments) and manual configuration of the environments. To address this problem, the authors explain the best practices in the areas of configuration management and continuous integration which include using a version control system to manage not only the application code but the other artifacts like the software configuration and environment details.
The deployment pipeline discussion in the book focuses on the practices of build and deployment phases of a project and the metrics to get feedback on the software delivery process. In the book, Jez and David also cover the topics of implementing testing strategies and how to test non-functional requirements.
This book was also just named as winner of the 2011 Jolt Awards for Books. InfoQ spoke with both authors of the book on the continuous delivery concept and how it can be used to automate the software product delivery process to be more efficient and effective.
InfoQ: Can you define "Continuous Delivery" concept and how it differs from other practices like Continuous Integration or traditional QA (Testing) tasks?
Jez Humble: Continuous delivery means that your software is production-ready from day one of your project (even when it's not "feature complete") and that you can release to users on demand at the push of a button. There are several practices and patterns that enable it, but in particular excellent configuration management, continuous integration, and comprehensive automated testing at all levels form the foundation. The key pattern is the deployment pipeline, which is effectively the extension of continuous integration out to production, whereby every check-in produces a release candidate which is assessed for its fitness to be released to production through a series of automated and then manual tests. In order to be able to perform these validations against every build, your regression tests - both at the unit and acceptance level - must be automated. Humans then perform tasks such as exploratory testing, usability testing, and showcases as later validations against builds that have passed the automated tests. Builds can be deployed automatically on demand to testing, staging and production environments by the people authorized to do so.
Through these practices, teams can get fast feedback on whether the software being delivered is useful (as required by methodologies such as the Lean Startup), reduce the risk of release, and achieve a much more predictable, reliable process for software delivery.
David Farley: Continuous delivery is fundamentally about establishing the shortest possible feedback loops for every aspect of development. We want early feedback of which requirements are effective when our software is delivered to its users, so we want to deliver software early and often. We want feedback if a change breaks an existing feature so we want to have great automated tests to highlight regression problems. We want feedback that our software can be successfully deployed to production so we need to exercise the deployment mechanisms before we get to that point.
Once this fundamental need is understood all of the technologies, techniques and practices are really about delivering this feedback as clearly and quickly as possible.
InfoQ: What type of organizational culture or level of process maturity is the best environment to use the Continuous Delivery practices discussed in the book?
Jez Humble: The backbone of continuous delivery is a culture in which everybody involved in delivery - developers, testers, infrastructure, DBAs, managers, customers - collaborates throughout the lifecycle of the product. It also relies on some level of maturity in automation of the delivery process: build, test, deployment, infrastructure and database management. But there's no real limit in terms of applicability. These techniques are valuable whether you're working on embedded systems, products, or web-based systems. However in return for rapid, reliable software releases you do incur a cost, so continuous delivery should only be used for strategic software projects where fast feedback and predictable, low-risk releases are valuable to the business.
Dave Farley: When I was a consultant, I always thought that establishing Continuous Integration in an organisation that wasn't used to it was like a beach-head for agile practices. It is the first and most obviously valuable practice and it drives good behaviours everywhere else. Continuous Delivery (CD) takes that a step further. For genuine CD to work you need high levels of discipline, and an across the board focus on quality, delivery and, to some extent, efficiency. This is a big challenge for many organisations, because it can often mean significant changes to the way in which organisations do things in a broader sense than only the development team: It affects how requirements are prioritised; how they feed into the development process; how testing is performed; how releases are done. CD has a significant, and in my opinion positive, affect on every role associated with software production from inception onwards. Any organisation can dip in and gain some of these benefits, but it does require significant commitment to get all of the benefits. I strongly believe that there is a significant pay-back from these techniques over a period of time, there is certainly an investment required to achieve that pay-back. My gut feel is that CD is appropriate for any software that will have changes made to it after a period of 3-6 months. If the software is done and dusted and will never change again within 3 months, then maybe not.
InfoQ: Can the Continuous Delivery practices discussed in the book also be used in software development environments that are using traditional waterfall development methodologies?
Jez Humble: Yes, absolutely. The techniques we discuss are engineering practices that are mostly independent of your project management process. Automation and collaboration are just as important if you're using waterfall, and they make a huge difference to the quality of your software and the predictability of your delivery process.
Dave Farley: I agree with Jez, a waterfall project that uses the techniques of CD will be better than a waterfall project that doesn't. However there is a sometimes hard to estimate cost to CD that can make scheduling tricky. Part of the process is to stay on top of failures. The process depends on a timely and effective view of the state of your software - it needs to stay working all of the time to achieve that. If your automated tests highlight a problem in your code then you need to react to it and fix it. Equally if your automated testing highlights a problem with the feedback cycle, tests running too slowly is the classic problem, you need to react to that too. So any process that employs CD needs the flexibility to react to problems and fix them as soon as they arise.
Inevitably I am going to say that although I believe that CD is an effective practice for any approach to software development, it is most naturally aligned with agile development practices and that these practices maximise its impact.
InfoQ: Software quality testing and validation process usually includes automated testing (via code scans), manual functional testing and manual code reviews. What is the best combination of these different quality validation efforts in order to get the best of each effort?
Jez Humble: Brian Marick created a test quadrant which classifies the various forms of tests that are essential in software development - and you need all of them. Developers should be using test-driven development to make sure the code they are delivering is of high quality. Developers and testers should work together to create suites of automated functional acceptance tests to ensure the software fulfills its requirements as part of the development process. Testers should perform exploratory testing and usability testing. Teams should also test cross-functional requirements such as security, availability, and performance from early on in the project, automating as much as possible.
One of my favourite software development quotes is from W Edwards Deming, who says "Cease dependence on mass inspection to achieve quality. Improve the process and build quality into the product in the first place." That means that testing is not a phase to be performed after development is complete. It needs to be done all the time throughout the delivery process by everybody, using techniques such as behaviour-driven development and acceptance test driven development. It also means that quality is not the responsibility of testers - it's the responsibility of the whole team. Testers are essential to creating high-quality software, but their job is to make the quality of the system transparent, not to be responsible for that quality.
Dave Farley: Jez has nailed it, quality is everyone's responsibility. One of the significant benefits of CD is that it gives better visibility to everyone associated with a software project of the state of that project and so anyone can react to something that doesn't look quite right. People are useless at complex, but repetitive tasks. Using any form of human based testing for regression is wrong. Computers are wonderful at that, so all regression testing, functional, performance, whatever, all repeated actions really, should be automated.
People are wonderful at pattern matching. People should be exploring the system, highlighting things that "just don't seem right" their role is to worry about usability and picking up on the subtler cues that suggest where there may be a problem. In my projects I use this distinction in all aspects of quality review, we automate everything that is repetitive, be that tests, build or deployment tasks or anything else and we encourage everyone to use their instincts, intuition and reasoning to highlight problems from whatever source, build problems, release problems, performance problems, code quality problems - anything. An important aspect of CD is a focus on continuous improvement.
InfoQ: What are some version control and release management best practices especially for geographically distributed teams?
Jez Humble: Firstly, everything that is required to build, test and deploy your system should be in version control. It should be possible to plug in a new workstation, and check out everything you need to build, test and deploy your system to any environment you have access to from version control. Deploying software to testing environments should also be a push-button process. This is even more important with distributed teams, since otherwise you waste cycles trying to work out why something works on their machine but not on the machine of somebody sat somewhere else in the world.
It's also essential that everybody checks in every day to trunk, again to make sure that everybody is working from a single source of truth as to the state of the system. Distributed version control systems such as Git and Mercurial can make this much easier when working in distributed environments since if your connectivity is bad, you're not reliant on waiting for a centralized version control system on a different continent to share your work. Instead you can check in locally multiple times per day, and push to the designated central repository out of band on a less regular basis (but still at least once per day).
Dave Farley: In CD every commit is considered to be a release candidate. This is means that it must work with everything else and be deployable and ready for use, we never commit changes that we intend to fix later. That means that in order to evaluate any commit it is essential that we evaluate it alongside every other change. In the ideal world every commit is made to HEAD, a single canonical representation of the state of the application, so that we can get the fastest feedback possible if our change causes problems. Technologies like distributed VCS are great, I am a big fan, but they have some common patterns of use that are antithetical to CD. My rule of thumb is that if you are not merging changes into HEAD at least once a day you aren't doing Continuous Integration and so will pay a significant cost when using a CD process.
InfoQ: How can the development teams integrate architecture practices into the software development process without impacting project deadlines too much, but at the same time creating software components and services that are useful for the specific project but also are reusable across the projects (in long term) in an organization?
Jez Humble: Wow, that's a big question! Books have been written on that topic. I don't think I can really do it justice in a short space. All I'd say is that you have to focus on creating a useful product first. Re-usability is important, but it is subordinate to creating working software. Frameworks and components are best created through harvesting them from similarities in existing systems. However how you structure your software is essential. The SOLID patterns as expressed by Bob Martin - using techniques such as encapsulation and low coupling to create modular software - is essential in order to create maintainable systems.
Dave Farley: This is a question close to my heart, but one that I can't do justice to here because it is such a big topic. First some simple pragmatic things, in my current project we have many tests that assert a level of architectural conformance (e.g. if you try to access a database directly from one of our web servers the build will fail). The tests are much more sophisticated than that, but it gives you a sense of the direction.
More generally I think that good design is orthogonal to development process. I very strongly believe that good design is an iterative process and that effective software evolves over time. This means that I think that agile, iterative development processes make it easier to create good designs, but you can write bad software as well as good using any development process. The only silver bullet is talented people.
Having said that I think that there are some common features of good design. A continual focus on quality and efficiency - good software does things simply if it looks complex it is probably wrong.
Modelling - good software has a shape and collections of abstractions that allow simplified descriptions of what is going on.
Automated tests - you can write bad software that is fully tested, you can write good software that has no automated tests but that is really hard. I think that it is hard enough to write good software already, so don't make it harder by ignoring the tests.
InfoQ: Another architecture related question. When is the right time to review and validate the software built to ensure that it's following the architectural and design standards?
Jez Humble: All the time. Pair programming - particularly having senior developers always pair with junior ones in order to teach them - and regular code reviews by senior developers are great ways to make sure that any problems are found early. However I don't recommend gating the delivery process so that each check-in must be reviewed before it can be merged. Instead, senior developers should monitor check-ins through feeds and then help people refactor through pairing when necessary.
Dave Farley: All the time. Every commit should assert some levels of architectural conformance. It is broader than that though. I like to maintain what I call a white-board model of the software that I work on. This is a diagram that shows the important abstractions. The whiteboard bit is significant in that it should be enough of a shared understanding of the system at a high-enough level of abstraction so that any reasonably senior member of the team can reproduce it on a whiteboard from memory - it should not be too detailed. New requirements should be evaluated against this whiteboard model to see how they fit and to see if they imply a change to the model. If the model needs changing we assemble the team and discuss the changes together.
InfoQ: Most of the organizations need to be in compliance with some type of regulatory or internal audit requirements. What are your thoughts on best ways to incorporate the security policies and compliance aspects into the software development process?
Jez Humble: There are two important areas where regulation applies. First to the delivery process itself. You need to make sure that errors or back-doors don't slip in to your systems, and that your software delivery process is auditable. The best solution to these problems is comprehensive configuration management, automation of the build, test, deploy and release process, and close collaboration between everyone involved in delivery. Pair programming and regular reviews are a great way to provide checks and balances on what gets introduced into the code base. Second, you need to make sure your software conforms to requirements such as security, and the ability to audit the data in the system. These are best enforced through automated validations - tests - that are run against each release candidate using the deployment pipeline. The deployment pipeline is what provides end-to-end traceability of every bit of information through your delivery process, and ensures that the necessary validations have been run against every release candidate.
I have an article coming out in the Cutter IT Journal later this month that discusses these issues in some detail.
Dave Farley: I am currently working in the finance industry and so I am no stranger to regulatory compliance issues. My recent experience has been that there is no better way than a process like Continuous Delivery to achieve this.
It can be complex to explain to people like auditors or compliance experts, but only because it is new to them. A full blown CD system offers soup to nuts traceability. In my organization we have a complete automated audit trail, as a free by-product of our techniques. Our requirements system records when a requirement was created, when development was started and when completed and who was involved at each step. This is tied in to our version control system that shows which changes were made related to that requirement. Our deployment systems are tied to the version control system via release numbers and audit who agreed to the release to a particular version of the software into a particular environment. We can answer almost any question about the life-cycle of any feature and show a complete audit trail of the change. As I said, this facility came as a side benefit, we did no extra work to get this level of traceability it comes free as a virtue of comprehensive version control.
The auditors that I have dealt with love this stuff.
InfoQ: There is more attention recently being given to the collaboration between developers (Dev) and operations (Ops) teams in integrating the development and testing tasks with the deployment and software delivery tasks. Can you talk about some best practices and guidelines in this area?
Jez Humble: Again, there's more detail in my forthcoming Cutter IT journal article, but essentially you need to make sure that developers, testers, and operations collaborate throughout the delivery process. That means all of them are present for project inceptions, showcases, and retrospectives. Unfortunately in most organizations developers are measured according to throughput, and operations are measured according to the stability of production systems. That leads to huge dysfunction. Developers should be accountable for the stability of the code they create - one way to encourage this is by having them wear pagers so that when things go wrong in production they have to help fix the problems. Operations have to be accountable for throughput - which means making sure that developers can self-service production-like environments for testing purposes to ensure that software can be released automatically without causing problems. Ultimately it's in everybody's interest.
Dave Farley: Delivery should be everyone's responsibility. The whole team should be focused on successfully delivering good quality software to its users. Providing good visibility into the lifecycle of releases is an important part of that. In addition making sure that the people that cause problems are responsible for fixing them is vital to the establishment of effective feedback loops.
InfoQ: Do the continuous delivery processes and practices differ in any way when the software product is deployed to a cloud instead of being hosted internally? What are the pros and cons of using a cloud solution when it comes to Continuous Delivery?
Jez Humble: Continuous delivery is really orthogonal to the issue of what platform you deploy your system on. The techniques are equally applicable whether you're releasing an embedded system, a cloud-based solution, or something that sits on a mainframe. That will depend on the cross-functional requirements of your software.
Dave Farley: No there is no real difference. The virtualization of the host system, inherent in cloud computing, can make it a little easier, from a technical perspective, to tear-down and re-commission a known-good environment into which the software will be deployed, but that is the only difference that I see, the process and techniques are the same.
InfoQ: To conclude the interview, how do you rate the significance of the elements of a successful continuous delivery environment in terms of Processes, People, and Tools? Is one element more important than the other?
Jez Humble: No, you need them all in order to achieve success. I think though that the most important thing to focus on is people. Most problems in IT boil down to people problems. Continuous delivery fundamentally requires discipline from the people developing software, and the ability of teams to adjust their environment in order to improve. Without that, you can't repeatably deliver valuable, high quality software. Tools and process are important, but they are enablers, not pre-requisites. One of my favourite papers on continuous integration is "CI on a dollar a day" by James Shore in which he shows how to do CI using an old developer workstation, a rubber chicken, and a bell. CI is a practice, not a tool. Running Jenkins on your feature branches does not mean you are doing CI.
Unfortunately most of the time people like to start by talking about the tools, because it's nice and concrete, and because it's easier to stick in a tool than it is to change people's behaviour. Tools are important enablers - I would recommend in practice that you use a CI tool - but they're not the deciding factors in whether you will be successful.
Dave Farley: Fundamentally I agree with Jez, all are important to get right. If pushed I would say that tools are the least important aspect, but only because if you have good people and effective processes then they will create the tools that they need (it's not actually all that hard). The things that are important about CD are not related to a particular product, tool or technology, but in general it is the effective use of technology to support the process that is an enabling factor.
Dave also said:
Fundamentally I think that software development, at its best, should be based on a process of exploration and measurement, a fairly scientific approach to the subject I guess. I think that CD is effective because it facilitates an explorative approach by providing real, valuable measurements of the output of the process and feeding those results back into the process.
InfoQ: Thank you both for your time with this interview.
About the Book Authors
Jez Humble is a principal consultant with ThoughtWorks and co-author of Continuous Delivery (Addison Wesley, 2010). He has been fascinated by computers and electronics since getting his first ZX Spectrum aged 11, and spent several years hacking on Acorn machines in 6502 and ARM assembler and BASIC until he was old enough to get a proper job. He got into IT in 2000, just in time for the dot com bust. Since then he has worked as a developer, system administrator, trainer, consultant, manager, and speaker. He has worked with a variety of platforms and technologies, consulting for non-profits, telecoms, financial services and on-line retail companies. Since 2004 he has worked for ThoughtWorks and ThoughtWorks Studios in Beijing, Bangalore, London and San Francisco. He holds a BA in Physics and Philosophy from Oxford University and an MMus in Ethnomusicology from the School of Oriental and African Studies, University of London. He is presently living in San Francisco with his wife and daughter.
Dave Farley has been having fun with computers for nearly 30 years. Over that period he has worked on most types of software, from firmware, through tinkering with operating systems and device drivers, to writing games, and commercial applications of all shapes and sizes. He started working in large scale distributed systems about 20 years ago, doing research into the development of loose-coupled, message-based systems – a forerunner of SOA. He has a wide range of experience leading the development of complex software in teams, both large and small, in the UK and USA. Dave was an early adopter of agile development techniques, employing iterative development, continuous integration and significant levels of automated testing on commercial projects from the early 1990s. He honed his approach to agile development in his four and a half year stint at ThoughtWorks where he was a technical principal working on some of their biggest and most challenging projects. Dave is currently working for the London Multi-Asset Exchange (LMAX), an organization that is building one of the highest performance financial exchanges in the world, where they rely upon all of the major techniques described in this book.