DevOps is a movement. DevOps is a mindset. DevOps is devs and ops working together. DevOps is a way of organizing. DevOps is continuous learning.
The intangibility of DevOps makes it hard for leaders to come up with a clearcut roadmap for adopting DevOps in their organization. The meaning of DevOps is highly contextual, there are no popular methodologies with prescribed practices to abide by.
However, healthy organizations exhibit similar patterns of behavior, organization and improvement efforts. In this series we explore some of those patterns through testimonies from their practitioners and through analysis by consultants in the field who have been exposed to multiple DevOps adoption initiatives.
This InfoQ article is part of the series "Patterns of DevOps Culture". You can subscribe to receive notifications via RSS.
"Build Quality In" is Steve Smith and Matthew Skelton's collection of experience reports (including their own) on Continuous Delivery and DevOps initiatives. The book features contributions ranging from in-house technical and process leaders to external consultants implementing technical and organizational improvements.
Since the book focuses on getting experiences into the light (and into words), readers shouldn't expect a consistently structured and thoroughly reviewed piece of work. Instead there's a welcoming mix of styles and prose akin to the diversity of organizations and working environments represented.
Reports range from pure Continuous Delivery (CD) to pure DevOps:
- Chris O'Dell (7digital) on end-to-end testing slowing down API delivery. How rollback, monitoring and blameless postmortems where key to increasing speed and moving from a monolith to service-oriented system;
- Niek Bartholomeus (large investment bank) on tackling gigantic releases (due to complicated inter dependence between applications). How an extremely centralized organization reflected in lack of application independence;
- Rob Lambert and Lyndsay Prewer (NewVoiceMedia) on how to reduce cycle time and act faster on customers' feedback. How testing as an activity (instead of a time-consuming phase) and monitoring throughout the entire lifecycle helped achieve smaller releases;
- Phil Wills and Simon Hildrew (Guardian) on quick feedback loops from customer input via multiple deploys per day. How changing traditional roles (dev, QA, ops and release managers) can bring about faster product-focused software delivery;
- Marc Cluet (consultant) on how company size influences DevOps adoption: from tech-savvy startups to risk and pain-averse corporations. Also the causality between steps in DevOps adoption;
- Alex Wilson and Benji Weber (Unruly) on applying good software engineering and continuous delivery practices in a full-stack team. How an incremental approach to monitoring, re-designing a monolith app and canary releasing resulted in business growth;
- Anna Shipman (UK's Government Digital Service) on building a DevOps culture in a government environment by growing a trust relationship with government agencies, showing work done and understanding risks. How keeping a security perspective can become an ally in promoting continuous delivery practices;
- Amy Phillips (Songkick) on trimming down the release process and moving forward as bottlenecks move along the delivery process;
- James Betteley (Fourth) on moving an ops team from a firefighting modus operandi to a "Scrumban" approach with high visibility and prioritization. How tracking daily interruptions along with backlog stories provided a clear picture of work in progress and increased predictability;
- John Clapham (MixRadio) on embracing risk pragmatically to improve cycle time and on the need to change culture and working habits to gain real benefits from DevOps and Continuous Delivery initiatives;
- Jennifer Smith (ThoughtWorks consultant) on shortening the feedback loop from running systems by making delivery teams responsible for production support;
- Sriram Narayanan (ThoughtWorks consultant at a large ticket-booking website) on build and release automation, monitoring and predictability. How effective changes in behavior require leading by example;
- Jan-Joost Bouwman (ING) on moving from a process-driven approach based on ITIL and CMMi frameworks to a content-driven approach kick-started by Lean and suplemented by DevOps. How internal DevOps Days and other activities can promote cross-team understanding;
- Martin Jackson (UK Government's Digital Transformation) on the technical path to establish trust in government software systems, namely via stable and predictable development and production environments;
- Rachel Laycock (ThoughtWorks consultant) on Conway's law as an obstacle to CD from an architecture point of view. How organizations swing from one extreme of “enterprise architecture” where developers don't understand why they are doing things to another extreme where fully autonomous teams care only about their layer in the stack thus causing integration nightmares;
- Matthew Skelton (a large ticket-booking website) on collaboration and improving the flow and visibility of work. How (some) technology can trigger cultural changes;
- Steve Smith (Sky Network Services) on growing a Continuous Delivery pipeline across an organisation, and how organisational structures can prevent cycle time reduction.
InfoQ asked the book authors their thoughts on the process of producing the book and on the state of Continuous Delivery and DevOps.
InfoQ: How did the idea and motivation for "Build Quality In" come to life? Why did you focus on practical examples instead of a more prescriptive book?
Steve Smith: Matthew and I had discussed writing a book together for some time, and just after the first PIPELINE conference in April 2014 we committed to doing it together. I had the idea of co-editing an anthology of articles, and Matthew had the idea of a Continuous Delivery/DevOps division. On the train home I asked the awesome Chris O'Dell to be our first contributor, and Matthew came up with the Build Quality In title.
Matthew Skelton: Steve and I realised that there were many people we knew doing really great work in organisations more typical of where most people work compared to the excellent but unusual examples of Netflix, Facebook, Google, Etsy, and so on; we wanted to demonstrate that Continuous Delivery and DevOps are accessible to all kinds of different organizations and all kinds of different people. We went with a set of experience reports because 'warts-and-all' is often more convincing than a theory-led approach, and also because there is no one-size-fits-all recipe for CD and DevOps - there are too many variables.
InfoQ: Agile practitioners often talk about "building quality in" in software but you are talking about building quality into an organization. What's the difference (if any)?
Steve Smith: The idea behind the Lean and CD principle of Build Quality In is to eliminate rework, which is an enormous drain upon lead times and productivity. Organisational problems such as poor inter-team communication or overly-centralised authority often lead to rework that can be far more expensive that reworking software, and it's interesting how many Build Quality In stories talk about organisational change as a prerequisite for success.
Matthew Skelton: Organizations building differentiating software systems that want those systems to be effective over any reasonable timeframe need to address organizational agility if the quality of their software is to be sustained. It's simply not possible to build and evolve high-quality software in the longer term if the organization fails to address things like team motivational factors, software operability, on-call responsibility, etc.
InfoQ: What criteria/approach did you follow to select the stories that made it into the book?
Steve Smith: We had a wishlist on a Trello board of CD practitioners we knew from LondonCD, PIPELINE, and the broader CD community. We explained to potential contributors our desire to highlight both the importance and inherent challenges of CD/DevOps, our donation of 70% royalties to Code Club, and asked for their involvement. The vast majority of people said yes, and we were really fortunate both Dave Farley and Patrick Debois agreed to write forewords for us. We will always be enormously grateful for the time people gave up to participate.
Matthew Skelton: We wanted a broad cross-section of angles, organizations, cultural challenges, and people represented in the book. By and large, I think we achieved that, although most of the examples are web-based, so desktop, on-premise, and embedded systems are not really represented, but I'm working to address that gap in other ways :)
InfoQ: What were your favorite stories? Or which ones helped you "see the light" on a particular problem, topic or context?
Steve Smith: I genuinely loved all of the contributions. Chris O'Dell talked about an API re-engineering that shrunk cycle times, Rob Lambert and Lyndsay Prewer talked about a company-wide vision that inspired a move from annual to weekly releases, Alex Wilson and Benji Weber talked about how they have enhanced eXtreme Programming practices, Anna Shipman talked about DevOps and the Badger Of Deploy at the heart of UK Government... and I found out after 6000 words I had been trying to solve the wrong problem for 3 years.
Matthew Skelton: It's really difficult to pick favourite stories, as they are all so inspiring! I think my top three are the story from Chris O'Dell on 7digital's metrics-driven approach to Continuous Delivery, Anna Shipman's honest appraisal of DevOps within the UK Government GDS team, and John Clapham's experience of introducing DevOps at Nokia Entertainment. All three really show how the transformation they helped make was a real mix of technology and cultural or organisational changes, not one or the other.
InfoQ: What are in your opinion the major obstacles today to successful DevOps and Continuous Delivery (CD) adoption?
Steve Smith: People. Organisational change is the single biggest barrier to CD adoption.
Matthew Skelton: I've spoken to people from dozens of organisations over the past 5 years trying to improve their delivery and operations capabilities. The biggest blocker to DevOps and CD I've seen is the lack of clarity on who should be doing what within an organisation, and how those responsibilities might need to change over the coming 2 or 3 years; I wrote recently about this, looking at how DevOps team topologies affect culture. There are many ways to make DevOps and CD effective, but closing our eyes and ears to the need to change team skills or responsibilities is not one of them!
InfoQ: Would you say that CD is "easier" than DevOps since "Continuous Delivery" (the book by Jez Humble and Dave Farley) is rather prescriptive on which practices to adopt while DevOps is more of a work in progress for each organization, without a definitive definition?
Steve Smith: I don't believe CD is easier than DevOps. Our main motivation behind Build Quality In was to show people how difficult CD and DevOps can be as well as the successes they can produce. I would say that CD and DevOps are both enormously contextual, and as a well-defined collection of practices and and principles CD is harder to misinterpet than DevOps.
Matthew Skelton: I would say that CD is easier to define than DevOps (due to the Humble & Farley book), but not that CD is 'easier'. The context - including teams, skills, technologies, end customer relationships, regulatory demands, and market demands - really dictate how effective a team or organisation can be with either DevOps or CD or both.
InfoQ: Despite the intangibility of DevOps, do you find there are common practices or patterns that organizations with a healthy DevOps culture exhibit?
Steve Smith: There are certainly common DevOps practices and patterns, and there is overlap with CD. I simply look for an environment in which people are free to talk to one another and collaborate on problems, regardless of role or team.
Matthew Skelton: Yes, organisations with a healthy DevOps culture place a strong emphasis on metrics-driven decisions, have a clear understanding of who does what (and why), and avoid a blame culture, thereby encouraging people to be specific almost to the point of pride in how they screwed up. The five pillars of DevOps - CALMS - are equally emphasised too, rather than a focus on just Automation or Metrics, for example.
InfoQ: If "culture eats strategy for breakfast" how can a DevOps culture ever take over a siloed culture?
Steve Smith: CD and/or DevOps need to be powered by a business vision. There needs to be a clearly-defined business problem people can relate to, and one that people can see is exacerbated by existing silos within the company. From that a CD and/or DevOps culture can spring, but it needs a lot of work to get it right.
Matthew Skelton: A DevOps culture can best emerge by simply demonstrating that it is more effective than a silo'ed culture. Use metrics to track deployments, uptime, performance, etc. and challenge the silo'ed culture to be more effective. However, some silos are actually useful: Google's SRE model for operating software is arguably an effective silo, it's just that the organisational conditions have been set up well in order for that silo to work well.
InfoQ: Steve, along with Dave Farley, you were part of one of the first projects where CD practices (and in particular the deployment pipeline) took shape. Did you realize back then that you were doing things radically differently from most organizations?
Steve Smith: The LMAX deployment pipeline I worked on with Dave between 2008-2010 was certainly a mature, high quality pipeline but CD practices have been around for some time. Dave and his co-author Jez Humble introduced CD practices to various ThoughtWorks clients in the mid-2000s, and in 2007 I worked on a team at Reed Elsevier that pushed new features to a website every few days. I would say CD is like eXtreme Programming - it feels like things are moving slowly at times and does not seem that radical, but actually the lack of feature branching, manual regression testing, rework etc. means you are delivering new functionality very quickly. When you are suddenly not doing CD it becomes quite obvious how opportunity costs pile up around you.
InfoQ: Was it difficult or demotivating for you to work on projects with little or no concepts of CD after that?
Steve Smith: I guess I've avoided that problem. Between 2007-2014 I worked for companies that adopted CD practices to varying extents, and since 2014 I've run my own consultancy www.alwaysagileconsulting.com to help companies understand how they can adopt CD for themselves.
InfoQ: In your opinion what have been the main changes in the CD world in the past 5 years? And how far are we from Continuous Delivery becoming a commodity like Continuous Integration?
Steve Smith: The biggest changes in CD in the past 5 years have been: PaaS e.g. Netflix on AWS, containerisation e.g. Docker, and build feature branching e.g. GitHub Flow. All of those have led to the evolution of CD practices, and I think that's perfectly natural. Neither Continuous Integration nor Continuous Delivery can become commodities as they are heterogenous practices rather than homogenous tools.
InfoQ: Your contribution to the book really brings home the interdependence between CD and DevOps (and other DevX relationships). Perfecting the pipeline still did not significantly reduce cycle time because of organizational barriers. What would you have done differently if you were in a similar situation again?
Steve Smith: If I was in the same situation again I would start with a shared consensus on a business problem and a CD-focussed solution, rather than one guy running around nagging people to adopt CD.... and I would measure the whole value stream from the outset to make sure I didn't miss the real constraint for 3 years again.
InfoQ: You've written a lot about CD patterns and anti-patterns. Have you collected any DevOps patterns and anti-patterns as well?
Steve Smith: I've not collected any DevOps patterns/principles, as my writing is based on experience and (shock, horror) I've not worked in a "DevOps organisation". I've worked in plenty of companies in which developers, testers, and operations staff collaborated closely together but I don't see anything reusable there other than reminding people to talk to each other. I have to admit I'm quite down on DevOps these days, and think it jumped the shark somewhere in the past year or two. I think Patrick Debois et al built an amazing global community over a 5 year period and a lot of hard problems were shared, but everywhere I look I see DevOps tools, certification programmes etc. I'm somewhere between Matthew's famous "it's raining DevOps" comment and Dave Farley's gentle debunking of Cargo Cult DevOps.
InfoQ: Matthew, collaboration, work flow and visibility are at the core of the story you contributed, despite all the technical improvements you implemented/promoted. When can technology be the messenger for cultural changes and when not?
Matthew Skelton: Technologies that we've seen work well as messengers for cultural change are things that help to shine a light on how well (or badly) the Production system works: metrics, monitoring, log aggregation, dashboard, and so on. These kind of tools generally bring a greater awareness of problems with software systems, highlighting what needs improving and resulting in more useful conversations with less guesswork.
InfoQ: You mention the episode where you printed out the table of contents from the CD book and stuck them on a wall, which then triggered many conversations. It sounds like this also doubled as a cheap visual roadmap for continuous improvement? In comparison, do you reckon that large organizations often get little value from formal process improvement projects that inherit the same siloing problems of their regular projects?
Matthew Skelton: The headings in the Humble & Farley CD book are great, and yes, we continue to use them as a simple CD roadmap for several organisations. The problem with the formality of many large-scale change programmes is that they do not address some of the key contributors to success with CD and DevOps, including team empowerment and engagement, true end-to-end responsibility, and a rejection of deadline-driven 'project'-based funding.
InfoQ: What's the impact on IT teams and individuals' motivation when organizations treat IT as a provider rather than a partner?
Matthew Skelton: The question of whether 'partner' or 'provider' is a better model for DevOps misses the point I think. The crucial thing is: do we have trust in the organisations with which we work? As clients/customers or as suppliers, do we meet the needs of the other organisation? Do we listen well enough to what they suggest? A little less JFDI and a bit more humility works well for both clients and suppliers in a DevOps context.
InfoQ: Can you expand on why you recommend programme-based funding rather than project-driven budgets in the book?
Matthew Skelton: Project-based funding tends to compromise the quality of software, prioritising instead an often arbitrary deadline or face-saving and bonus-collecting for project managers and others involved in 'delivery'. The integrity of the product or service suffers as a result. The wider horizon offered by programme-based (or product-based) funding means that the technical teams are given a view of the bigger roadmap in order to make more informed implementation decisions. This often means that the quality of the software remains higher than with project-based funding, because the team has more context for hosrt-term versus longer-term fixes or workarounds, and is empowered to clean up after an important delivery date.
InfoQ: Quoting you, Matthew: ‘NoOps’ != ‘no operations’ and ‘DevOps’ != 'developers running Production'. It sounds like in the IT world we're still too often looking for shortcuts to problems and skipping the need for genuine collaboration. Do you agree?
Matthew Skelton: Too many businesses that rely on IT solutions still have no idea about what software really is and where the value is added; they think that once the commercial teams have specified a new feature or product that all the hard work is done, and that IT just needs to implement it. To be fair, we technologists have a poor track record at explaining why and how the building of software systems needs a much more collaborative approach, and Build Quality In is a small attempt to help rectify that problem. We need to remember that collaboration literally means 'working together', and that working together does not simply mean a division of tasks from 'on high'.
InfoQ: You have generously donated 70% of the book proceedings to Code Club, a not-for-profit organisation that runs a UK-wide network of free volunteer-led after-school coding clubs for children aged 9-11. Why is this cause so close to your hearts?
Steve Smith: I think it's widely acknowledged now that the IT industry is in the midst of a diversity crisis. We both wanted to contribute to Code Club because we feel it's important children learn to program regardless of their gender or ethnicity.
Matthew Skelton: Code Club does amazing work in the UK to help get kids interested in and aware of software systems as things that they can create and run. I think it's really important that technology is something accessible to future generations, in every sense of the word. It's particularly good that over 40% of Code Club members are girls, because we desperately need more diversity in IT in the UK, and gender diversity is a good place to start.
InfoQ: Final question: will there be a follow up?
Steve Smith: Possibly, I'm always happy to work with Matthew and I love engaging with the CD community to find out what people are up to. But not in 2015, and probably not in 2016.
Matthew: We're thinking about a possible follow-up book, but probably not until late 2016 or 2017. In the interim we might expand on the experience of the contributors in the current book with additional blog posts or articles. However, if people are really keen, maybe we can be persuaded to do something sooner :)
About the Book Authors
Matthew Skelton has been building, deploying, and operating commercial software systems since 1998. Co-founder and Principal Consultant at Skelton Thatcher Consulting, he specialises in helping organisations to adopt and sustain good practices for building and operating software systems: Continuous Delivery, DevOps, aspects of ITIL, and software operability. He is co-editor of Build Quality In, a book of Continuous Delivery and DevOps experience reports.
Steve Smith is an Agile consultant and Continuous Delivery specialist at Always Agile Consulting Ltd. Steve is a co-author of the Continuous Delivery and DevOps book “Build Quality In“, a co-organizer of the monthly London Continuous Delivery meetup group, a co-organizer of the annual PIPELINE conference, and a regular conference speaker. Steve blogs at www.alwaysagileconsulting.com/blog.
DevOps is a movement. DevOps is a mindset. DevOps is devs and ops working together. DevOps is a way of organizing. DevOps is continuous learning.
The intangibility of DevOps makes it hard for leaders to come up with a clearcut roadmap for adopting DevOps in their organization. The meaning of DevOps is highly contextual, there are no popular methodologies with prescribed practices to abide by.
However, healthy organizations exhibit similar patterns of behavior, organization and improvement efforts. In this series we explore some of those patterns through testimonies from their practitioners and through analysis by consultants in the field who have been exposed to multiple DevOps adoption initiatives.
This InfoQ article is part of the series "Patterns of DevOps Culture". You can subscribe to receive notifications via RSS.