Evolving DevOps to Enable Quality at Speed Software

DevOps started in 2009. This paradigm shift enables many organizations to accelerate the rate of software delivery and business performance as demonstrated in the Accelerate report and its well-known DORA metrics of speed, stability, and recovery.

Agile was the first popular combination with DevOps to equilibrate business value and technology delivery on the same flow of work.

More recent practices such as reliability or security embrace the same mindset coining terms of "SRE" and "DevSecOps".

Yet, the various State of DevOps from companies such as Google, Puppet, CircleCI, and Dynatrace, share similar findings in our ecosystem: a continued difficulty delivering timely business value and innovation, siloed teams lacking alignment, a fragmented toolchain, and quality sacrifices.

This article shares how to transform DevOps to bring more value and solve these identified pain-points by evolving the paradigm of DevOps through Quality Engineering, leveraging the experience and practices accumulated in the last few years.

DevOps for "Development & Operations" was only a start

DevOps has no fixed manifesto for one good reason: leaving space for evolution.

"We didn’t have a plan for what DevOps was. We kept on expanding in all directions, working out what improvements between silos could mean. People are really struggling with what DevOps is right now."
- Patrick Debois, Advisor, Snyk (Formerly DevOpsDays) - 2020 State of DevOps Report: Presented by Puppet and CircleCI

The original concept of DevOps fostering the flow between development and operations went in many directions. Security brought "DevSecOps", and Reliability was even added as part of the metrics by Google after thorough study of enterprise practices.

Figure 1: The evolution of DevOps continues to accelerate in many directions.

But the current level of innovation and competition in our ecosystem keeps raising the bar for all actors. Some new companies are able to reach one billion in valuation in less than 3 months, pressuring existing companies working in the same space. As a result, 85% of companies feel a real need to reinvent their business through software for survival.

And for many organizations, software delivery is still a business limiting factor.

Figure 2: Software limiting factors can directly limit the business.

The present state of DevOps is therefore not enough. Business value, security, and reliability are only part of the requirements to meet for staying competitive. Balancing quality and speed requires a paradigm shift across the entire software lifecycle, from business ideas to operations, of "Build Better, Build Faster".

That level of performance means that DevOps is evolving for continuous value delivery across all domains of the engineering system with Quality at Speed Software.

Streamline valuable activities with integrated methods

DevOps initially emerged from code to operations delivering numerous improvements on that perimeter. This led to many methodologies focusing on accelerating the flow between software engineers and operational teams.

Popular methodologies such as Agile and Lean were then combined with DevOps to focus better on value with more efficiency, aiming for "Software at Speed". Specific requirements were also added as continuous practices such as monitoring, security, or testing,

Figure 3: Even with the findings from Accelerate, quality requirements are only starting to be considered.

From these practices, the Accelerate report shared the positive correlation between software delivery performance and business outcomes, defining categories of performers where "Elite" performers release multiple times per day with stability.

But with delivering multiple times per day being the norm of today’s competition, competitive advantages can only come by delivering more value. That’s where DevOps must transform to integrate better with existing methodologies to create a Quality at Speed flow.

Figure 4: DevOps methodologies have to streamline dedicated quality flows.

The first change is to reconcile the two flows of digital transformation "from customer to code" and "from code to customer" as shared by Yves Caseau, Group CIO of Michelin in The Lean Approach to Digital Transformation.

DevOps methodologies have to integrate Agile, Total Quality, and Lean Startup with that global perspective from the start. Linking user stories to code deployments in their CI/CD platforms is a start. Shared ways of working and supporting tools have to integrate better to create faster, more valuable iterations from business ideas to operations.

The second change consists in creating dedicated value streams of experimentation, core business, and technology platforms. Aligned with the DevOps mindset of breaking silos end-to-end, each stream must be able to iterate decoupled from each other to ensure their requirements for quality and speed can be met.

Figure 5: Shift-left is one of the paradigms for Quality at Speed software.

SRE and DevOps are examples of dedicated flow for the non-functional requirements of reliability and security. The evolution is to be able to address other requirements like observability, usability, and sustainability along the software lifecycle to avoid missing them or trying to recover with costly reworks afterwards.

The last change is the systematic application of the theory of constraints on the end-to-end value stream. The weakest point across the lifecycle is determining the level of quality and speed for the entire system; value-stream mapping and limiting factors removal must be a habit for prioritizing DevOps improvements.

While these evolutions improve the flow of collaboration for more value, the desired acceleration requires more than methodologies. Technology has to enable organizations to evolve and adapt software with velocity.

Build platforms with minimum viable architectures

DevOps put a lot of emphasis on automation for integration and deployment stages to support faster iterations. The deployment of automation has contributed to improved quality through native pipelines available for developers and eased overall maintenance costs, with over 60% of companies stating to be mid-evolution on their DevOps stages in the 2020 Puppet report.

Figure 6: Modern DevOps accelerate with security, but lacking integrations. GitLab 2022 Global DevSecOps Survey.

The good part is the progress in providing self-service capabilities to developers leveraging technology and mindset changes. All this experience of automation progressively leveraging cloud-native capabilities enables us to better envision the target architecture that will enable us to "build better, build faster".

The improved automation and collaboration through self-service generates more data about processes execution, performance, and interactions. The evolving maturity of operational Data Science will enable the deployment of MLOps and AIOps components to provide continuous visibility and automation across the lifecycle like with Sealights.

Figure 7: The DevOps architecture must consolidate Quality at Speed platforms.

The DevOps transformation requires architecting the following platforms:

1. Infrastructure Platform
2. Developer Platform
3. Experimentation Platform

Each platform builds upon each other with an increasing maturity, meeting the quality requirements with minimal commitment and rework. This progressivity of implementation is the concept of Minimum Viable Architecture, pushing for delivering minimal architectural increments supporting product iterations for a better fit to purpose and faster adaptation.

The Infrastructure Platform allows operational teams to capture productivity gains through automation and cloud capabilities. These improvements can be directed to building a Developer Platform supporting faster iterations of core business changes while adhering to the multiple quality requirements, including functional usability, performance, and security.

The business can then leverage the accelerated cycles of software delivery meeting the quality requirements to iterate on business changes. This is the stage at which companies can develop their unique value proposition, and can do it even faster with self-service tooling and integrated solutions provided by an Experimentation Platform.

Figure 8: Under the Hood of Uber’s Experimentation Platform, Uber.

Engineers building the infrastructure, developer, and experimentation platforms all have users to consider as their customers. The satisfaction and performance of each platform is directly tied to their capability to experiment and improve the business, and in the end, deliver a successful user experience.

MACH (Microservices, APIs, Cloud, Headless) architectures significantly improve the modularity, composability, and scalability of the platform components. It is no longer about building isolated modules and frameworks; it’s about building the scalable platforms that will support accelerated cycles of value delivery for organizations.

This technology transformation requires the alignment of teams on a shared vision, with the inspiration and drive to make it a reality, overcoming organizational resistance, and effectively changing the habits of the many actors collaborating along the software lifecycle.

This is where management plays a crucial role.

Manage and drive end-to-end value creation

Cultural blockers raised in the Puppet DevOps Report are still keeping companies stuck in the middle of DevOps transformation where culture discourages risk-taking (21%), suffers from unclear responsibilities (20%), and lacks fast flow priority (18%) - leading to insufficient feedback loops limiting the speed at which companies can improve their value proposition.

Companies with successful DevOps transformations understand that effective leadership is necessary to break existing silos for faster iteration flow. But the significant efforts to transition from legacy cultural models to a DevOps model has left many managers overwhelmed by technology and engineering matters.

The DevOps team needs to focus on their work to improve the organizational maturity and flow. However, this has caused some people to think that DevOps is only about engineering. Managers must go beyond their comfort zones in order to have a bigger impact on the end-to-end flow of iteration to deliver more business value.

Figure 9: The management shift in DevOps must be end-to-end and increase value delivery.

The management role in DevOps have to change its positioning to:

1. Shift-up DevOps value proposition and delivery
2. Share the vision and empower for outcomes
3. Drive end-to-end transformational changes

Shift-up is a concept coming from software quality where the definition, communication, and measurement is done outside of operational teams to align the entire organizational system on the need for quality. One concrete exercise is to collaboratively define the quality requirements of a software product with C-level, Director, and business-line managers.

The benefits are improved change management by having key stakeholders aligned on what each persona values and expects from the software product. That common ground is then leveraged to get the necessary investment and support to implement quality requirements and organizational changes, especially when they involve technology.

The same shift is necessary to set DevOps as a necessary imperative to succeed in the digital transformation. The Phoenix Project is a good example of how DevOps contributes to improving the business. Managers pushing for DevOps must interact out of their comfort zone with business stakeholders to create the perception change that DevOps is not about engineering, but about accelerating the business reinvention.

The second management evolution is to systematically share the DevOps vision across all teams in the company, not restricting the ideas to within engineering's perimeters. This inspirational vision must share the business purposes, envisioned future, upcoming changes, risks, and plans. This will empower teams to make these changes a reality.

Figure 10: DevOps Management has to remove the most important limiting factors.

That requires managers to focus on end-to-end leadership transformations. They firstly need to implement control management aspects to empower their teams, solve roadblocks across the organization, and improve business value delivery. Effort must be allocated appropriately to drive the most change. For example, spending time to improve automation to shave off a few minutes when defining specifications takes teams days or weeks.

Management also has the responsibility to align the organization to create an aligned ecosystem with the target business architecture.

Align the organization on target architecture and flow

Many company reorganizations consist of visual organizational chart updates without any change to the day-to-day, and unfortunately, on the business. They usually try to involve everyone to be "participative" and "inclusive" but fail to bring lasting changes and improvements to companies suffering these restructuration cycles.

Superficial reorganizations have the common problem of not aligning the organization on the flow, lacking to focus on evolving interactions and strong decisions like which teams should be evolved first, which responsibilities must change, and how to implement a clear accountability model that works.

The DevOps organization is about enabling business reinvention.

Figure 11: DevOps organization must combine with business in a Product-centric organization.

Structuring changes must happen to magnify the impact of DevOps:

1. Define a true product-organization
2. Adopt real "platform as a product" teams
3. Align the responsibility & accountability model

Evolving the way companies adopt a product-organization model first requires making the company global by removing the silos of "product management" (i.e. business) and "product engineering" (i.e. "IT"). This artificial separation remains in many organizations, sending a message of the importance of these silos and thereby limits communication interaction, creating many inefficient hands-offs.

Teams must be acting as single cross-functional units dedicated to value streams where they have total ownership on the end-to-end flow. The challenge is to compose each team to cover the specific quality requirements in their scope of responsibilities. A unit working on the user experience has for example different challenges from a financial back-office.

The second change for true DevOps product organizations is to embrace the "platform as a product" approach as described in Team Topologies. The goal is to align the teams' structures and interactions on the desired target architecture by progressively creating a platform team that provides a self-service platform, expertise roles enabling cross-functional teams, or delivery lead roles.

Figure 12: Building an incremental organization for Quality at Speed with Team Topologies.

For all the above to work, leadership must make strong choices of investments and clarify them for the organization. Companies have to make trade-offs of investments due to limited resources, and too much compromise results in a performance at the lowest denominator, far from the target required.

Leaders must choose which teams get more resources, sometimes more than needed for service continuity while other teams remain limited. They need to clarify which teams have to work in best-effort for a complicated sub-system like monoliths, or which decisions remain central generating frustrations such as a central architecture for the benefit of a consistent foundation.

More recent evolution is also to dedicate roles to specific "Quality at Speed'' objectives. Where a "Delivery Owner" is responsible for the flow of value delivery, a "Practice Leader" is the one responsible to develop skills of expertises at a high-standard, and make sure that knowledge is shared.

Organizational alignment is one of the main pillars to foster and sustain the business and architecture transformation. On the day-to-day, the performance of defining structures and interactions depends on technology, but still mainly on the actors collaborating across the lifecycle. The last important transformation is Skills.

Foster an ecosystem of continuous skills development

The focus of DevOps skills reflects the maturity of the ecosystem with training for "Cloud Engineers" focusing on automation, infrastructure as code, or specific cloud services deployment. While these fundamental competencies are necessary, they are not sufficient to support the reinvention of companies at the required quality and speed.

Figure 13: The DevOps roadmap remains still on low technical layers, LinkedIn.

One main challenge is the specialization of skills that improve quality but reduce speed. On one hand, expertise introduces more verticals as dedicated areas such as "SRE" or "Security"; however, the collaboration between these more specialized roles is harder due to the skills distinction, potentially impacting the speed of value delivery.

The global scarcity of technology profiles is not helping in that challenge. Organizations fight to attract and retain the necessary talent for their transformation, with a worldwide pressure on resources that directly limit the transformation of companies. Some movements of reconversion and technology simplification are helping, but are not enough.

Yet, organizations must equilibrate skills for Quality at Speed with:

1. Composable and open career paths
2. Investment in scalable skills (i.e. platform)
3. Dedicated resources for skills development

The first transformation is to embrace more composable and open career paths. Many organizations replicated the Taylorism model even for DevOps, with strict positions of "Cloud Engineer" for doing specific tasks. But a lot of profiles can learn portions of that job while leveraging their complementary skills, further enabling companies to accelerate with more adaptation.

The concept of "T-shape" skills appeared for people able to combine a vertical skill like software engineer, and an horizontal one, like agile. Studies show that the model is evolving for "PI-Shape" and even "Comb-shape" for people able to combine multiple expertises, allowing for a faster collaboration with one person, and a richer career.

Figure 14: DevOps skills with more transversality through a continuous learning ecosystem.

This capability of adaptation and continuous learning means that organizations must evolve their paradigm and ecosystem to support that change. Dedicated investments must be allocated to mentoring programs, the introduction of the role of Practice Lead to inspire and develop people, and ensuring systematic time for training on a weekly basis, not just once per year.

Other practices like communities of practices, meetups, and talks remain useful to foster an attractive ecosystem where people can access self-development. Some organizations like Zalando, Uber, or Netflix successfully embrace open-source as a way to captivate talents and inner-source their products, enabling them to collaborate globally on their products.

While not all companies have the size and resources of these giants, all organizations can direct investments on reusable and scalable skills like technology integration and platform implementation. In an ecosystem where technological components become modular services, companies will have less to build and more to compose.

DevOps for Quality at Speed is continuous value delivery

This transformation of DevOps across the domains of methods, architecture, management, organization, and skills show the requirement for transversality to evolve the entire system of software engineering for continuous value delivery.

Culture, methods, and automation were all a good start for DevOps, but our competitive ecosystem will only sustain organizations able to rapidly reinvent and scale their value proposition with Quality at Speed software.

We can wonder if this evolution of DevOps should keep the same name as it pushes the boundaries beyond just "Dev" and "Ops", thereby becoming the integral pillar supporting an organization's digital transformation.

The future of DevOps is Quality Engineering.

References

• Dustin Smith, Accelerate State of DevOps Report. Google Cloud.
• 2020 State of DevOps Report: Presented by Puppet and CircleCI. CircleCI.
• The 2021 State of DevOps Report. Puppet.
• 2021 DevOps Report. Dynatrace.
• The GitLab 2022 Global DevSecOps Survey. GitLab.
• Gene Kim (2019), The Unicorn Project: A Novel about Digital Disruption, Redshirts, and Overthrowing the Ancient Powerful Order: A Novel about Developers, Digital Disruption, and Thriving in the Age of Data. IT Revolution Press
• Nicole Forsgren, Jez Humble, Gene Kim (2018). Accelerate: The Science of Lean Software and Devops: Building and Scaling High Performing Technology Organizations. Trade Select.
• Gene Kim, Kevin Behr, George Spafford (2013), The Phoenix Project: A Novel about IT, DevOps, and Helping Your Business Win. Revolution Press.
• Dave Farley (2021), Modern Software Engineering: Doing What Works to Build Better Software Faster. Addison-Wesley Professional.
• Yves Caseau (2021), The Lean Approach to Digital Transformation: From Customer to Code and from Code to Customer. Productivity Press.
• David N. Blank-Edelman (2018). Seeking SRE: Conversations About Running Production Systems at Scale. O'Reilly Media.
• Acacio Cruz, Ashish Bhambhani (2017), The Evolving SRE Engagement Model. O'Reilly Media.
• DevOps tech: Shifting left on security. Google Cloud.
• Matthew Skelton, Manuel Pais (2019), Team Topologies: Organizing Business and Technology Teams for Fast Flow. IT Revolution.
• Antoine Craske, Rémi Dewitte (2022), On Defining Quality Engineering. QE Unit.
• Scrum.org, DevOps Institute, The Convergence of Scrum and DevOps.
• Adrian Bridgwater (2016), Radical Agility as a Business Technology Principle. Forbes.
• Jérôme Louvel (2016), How Zalando Delivers APIs with Radical Agility. InfoQ.
• Lauri Apple (2016), Making open source fashionable. Open-source.com.
• Haseeb Budhani (2022), What You Should Know About The Rise Of The Platform Organization. Forbes.
• Uber Open Source in 2019: Community Engagement and Contributions. Uber Engineering.
• Accelerating Time to Market with the Business Agility Value Stream. Scaled Agile Framework.