Call it DevOps or not, if you are concerned about releasing more code faster and at a higher quality, the resulting software delivery chain and process will look and smell like DevOps. But for existing development teams, no matter what the velocity objective is, getting from here to there is not something that can be done without a plan.
Moving your release cadence from months to weeks is not just about learning Agile practices and getting some automation tools. It involves people, tooling and a transition plan. I will discuss some of the benefits and approaches to getting there.
Waterfall to Agile, Agile to Continuous Integration, Continuous Integration to Continuous Deployment. Whatever your processes are, the theme is the same: find a way to get code to users faster without sacrificing quality. But speed and quality are sometimes at opposition to each other. Going faster means things can break faster, and when we only think about DevOps as releases, it’s easy to fall into this trap.
Established development shops cannot just jump from one flow to another. Unless you start out net new, the goal is to introduce new processes without delaying releases for three months or more to do transition in lump. This is often done using a pincer approach that addresses both bottom-up tactics and top-down oversight and culture at the same time.
However, because adopting DevOps tools is so easy, the trend is to focus on tactics only and adopt from the bottom up without consideration of the entire pipeline. The outcome is release automation tools that dictate your delivery chain for you, and not the other way around. Here are the key categories that get neglected when teams hit the accelerator without a plan in place.
Structured Automation: DevOps requires automation. But what is often not considered is automation that sustains and fits into the entire delivery chain. Considerations such as governance, artifacts organization and inventory, metrics and security need to be made. If an organization establishes a vetting process for all new automation and how it fits into the pipeline’s orchestration, then new automation will support what exists today and in the future.
For example, many organizations driving down the DevOps path have encountered challenges when trying to incorporate practices from security or governance teams. Historically these teams have resided outside of the dev and ops echo chamber and their processes were asynchronously aligned to the work being done. The challenge for many organizations is to determine the best ways to bring the people, processes, and technology supporting these initiatives into the fold without slowing things down. The best organizations are finding new ways to automate policies from security and governance teams by shifting away from artisanal, asynchronous approaches to synchronous processes earlier in the lifecycle.
Let’s take a look at an example of application security. A number of technology vendors in the application security arena are touting “automation” as key value point for their solutions in order to better fit them into a DevOps tool chain. In some instances, automation means that machines are now fully responsible for monitoring, analyzing, and fixing security vulnerabilities for those applications at wire-speed. In other instances, automation implies facilitation of human-centric workflows that might represent hours or days of asynchronous analysis not fit for continuous operations. In both cases, the technologies may accomplish similar ends, but their approaches could be dramatically different.
Also, one solution might be built to support asynchronous investigations by a security professional, while the other might provide synchronous support to a developer at the design and build stages of the SDLC. Establishing a vetting process can help determine if the automation levels required by a team or process can truly be delivered before investments are made. It is also worth noting that layers of obscurity frequently exist within words like “automation”, “integration”, “configuration”, and “continuous”.
Common Language: Part of the reason we have so many meetings is we are not all speaking the same language. Even if you can understand the other aspects of the software delivery chain, it does not mean that each team (QA, Development, IT Ops) speaks a unified language. And the time wasted to reconcile the differences is just vapor. But the solution is easy. Be deliberate. Have a guide and agree on terminology in the tools you use upfront for new aspects of the pipeline and application.
One approach I have used to establish a common language is to share a common story. In a recent meeting with a CIO, he told me his aim was to transform a diverse group of 50 people to operate under DevOps principles but that they lacked common understanding for starting the conversations. I recommended that he purchase a copy of The Phoenix Project for each of the people in the group and ask them to read it. The novel describes the efforts, challenges, setbacks, and accomplishments of a diverse group of people as they transform themselves into a DevOps practice.
For example, organizations could map their journey into DevOps similar to the way the main characters in the book did. They could discuss how their own organization might take on mastering “The Three Ways”.:
- The First Way: understanding the flow of work from left-to-right as it moves from Development to IT Operations to the Customer.
- The Second Way: ensuring a constant flow of feedback from right-to-left at all stages of the value stream.
- The Third Way: creating a culture that fosters continual experimentation and learning.
Readers can also gain a common understanding of the four types of work described in the book: business projects, internal IT projects, changes, and unplanned or recovery work.
Using such stories can provide a foundation for conversations and common vocabulary that fosters improved understanding, collaboration, and planning for the journey ahead of them.
Shared Goals: If the entire team — all the players that are a part of software releases — do not have a shared objective, then the competing goals will lead to competing strategies and delayed releases. This results in a Frankenstein pipeline. For example:
- If development is not accountable for bugs in QA, they will commit features faster, but releases are slowed because of new issues. Conversely, if there is a dedicated QA team that is rewarded for the quantity of issues found or closed, unnecessary work may be unduly entering the system; or
- If IT operations is not motivated by release frequency, they won’t consider things like full-stack deployments due to perceived risk; or
- If security teams rely on automated testing that takes four hours to complete, yet the development teams are pushing out new releases every hour, some application security checks may never happen.
Pipeline and Business Metrics: If you do not measure the release process (builds, speed, deploys, bugs, etc.), there is no way to know what to change in the future to increase the speed and quality of code. But pipeline metrics are not straightforward — and involve the entire team. The beauty of modern tooling is that it can collect the data and metrics for you. But it is up to the team to decide what is valuable. Metrics available from tools or measured processes can provide common ground for understanding what work is being done or what results are being achieved. Share metrics around work in progress, deployment frequency, lead time for changes, mean time to recover and things that matter most to the customer you serve. Tracking and sharing these metrics can help organizations better understand where constraints are within the systems and practices they are trying to optimize.
That said, you need to focus on the right metrics that drive business success. Metrics can help you see if you are progressing on the things your business and customers care about most – seeing the big picture. Deming called it, “appreciation of the system”. For example pipeline and business metrics to consider:
- Are you completing activities (e.g., releasing builds, shipping new features, enforcing quality, checking security, responding to inquiries) fast enough to matter?
- What percentage of time are you spending on innovation compared to maintenance/rework?
- What percentage of time is spent on manual / asynchronous vs. automated / continuous across your Dev and Ops processes?
- Is your user base expanding or contracting? Are your customers investing more in your solutions over time?
- Are your customers getting what they want in the time they expect?
- Are you spending more or less to acquire new customers / support existing customers?
Quality Strategy: Quality is not an afterthought. The best development operations make quality everyone’s responsibility and QA a strategy and automation practice, not an execution one. Then quality goes far beyond regression testing and standard unit tests. It also includes:
- Quality of the pipeline
- Component quality and security (e.g., binaries, images, tools)
- Test-driven development (testing new functionality before it’s implemented)
- Behavior-driven testing and development
Always improving the ways that bugs are caught means test coverage is always increasing and bugs are decreasing. It also means that if you have a dedicated QA team that they have an open communication with IT and developers, and that they have a seat at the table when it comes time to talk about quality.
Plans, Hurdles, and Wins
The best way to get from slow to fast consists of these practices:
- Have a plan. Cliché, huh? I am not talking about a 10-page document you put in a drawer. Build a lean model that covers the problem, path to solution and clear goals. It should not be static, but built in an Agile-type way. Think strategy as code – perhaps taking a walking skeleton approach. Until you talk/think through the entire flow, it’s nearly impossible to catch all the variables. And things are always more elaborate than they seem. If you have ever built a minimum viable product (MVP), you will know what I mean.
- Clear the hurdles. Don’t think of governance and change control after you set up a new process. Think about it from day one. And start your automation there. Pipelines are only as fast as the slowest gate. In many organizations, that is compliance and governance. So clear your governance hurdles by bringing on tooling that can do vulnerability checks on your components, validate licensing and maintain an audit trail of your entire build and release process.
- Socialize quick wins. To build enthusiasm and start to get some results. Address one aspect of your delivery chain, and make it better. They should be relatively low risk and high value. Once you have done so, socialize the benefits. These quick wins get everyone excited about what is possible. And give some direction on what to do next. Continuous integration — due to the great tools out there — is a place that you can start that is significant, team-wide and has a serious impact. (But don’t fool yourself by just implementing Jenkins, Bamboo or some other CI platform. Team members should be integrating their work frequently, and each integration needs to provide rapid feedback on the quality and stability of the build.)
Your environment is as unique as the combination of all of its individual parts. There is no one-size-fits-all. But ensuring attention to the categories I mentioned above can help existing operations move from where they are today to a regular cycle of sprints and, eventually, continuous delivery or even deployment.
A final tip that I will offer is to engage with your community. There are numerous DevOps Days conferences and over 1,750 DevOps meet-ups around the world. Right after you finish reading this article, find one in your area and plan on attending. The networking opportunities, open spaces agenda times, and presentations shared there can be invaluable.
But don’t just stop at attending the events, use the opportunity to connect with people in your local area that have already established DevOps practices and see if they might invite you on-site within their organizations for a half or full day of shadowing. This approach is more common than you might think across the community and can lead to valuable insight that you just can’t glean from online articles or conference presentations.
Image source: http://devops.meetup.com
In modern day development, it is easier to go fast than to make sure that you have the right agility to ensure future successes. It is important to recognize that taking the time to build and execute a plan that enables the right level of agility will be more productive than developing a plan defined only for speed.
About the Author
Derek Weeks currently serves as vice president and DevOps advocate at Sonatype. In 2015, he led the largest and most comprehensive analysis of software supply chain practices to date across 106,000 development organizations. As a 20+ year veteran of the software industry, he has advised many leading businesses on IT performance improvement practices. Derek shares insights regularly across the social sphere at @weekstweets, LinkedIn and online communities.
