Engineering Digital Transformation for Continuous Improvement

Software is becoming more important to the success of every company. Most organizations have this realization and believe they can become more efficient with software development and delivery but struggle to improve at the rate the business requires. This needs to change for these companies to remain competitive. They must invest in improvements and embrace new ways of working. But how do we make this happen?  

This is a problem I’ve been working on since I led my first major transformation at HP starting in 2007 that delivered 2-3X productivity improvements. Since realizing the benefits that come along with new ways of working in software organizations, my passion is doing everything I can (books, webinars, keynotes, workshops, and now a computer based training) to help as many people as possible with their digital transformation journey. As I worked with more organizations, I realized most develop and deliver software like HP did before the transformation journey began and they desperately needed help with productivity improvements. As I saw recurring patterns emerge, I continued to see organizations struggling to invest in improvement projects and embrace new ways of working.

My desire to help software organizations led me to doing research outside of the software industry for ideas that could be leveraged. I quickly realized that, although very different from software, manufacturing has a long and very successful track record of improving productivity and quality. Could there be some manufacturing principles to adopt if they were modified to take advantage of the unique characteristics and capabilities of software? Yes, but I also realized these principles required a new approach to help overcome the resistance to change by incorporating the best thinking in organizational change management.  

Getting people excited and willing to make changes is great, but is only the first step. We also need to do everything we can to ensure they are successful along the way. Improving how organizations work requires implementing new tools and techniques that might not be familiar to software engineers. As they start this work, we must enable their success as much as possible by helping them understand good design patterns and avoid common mistakes. To this end, I connected with David Farley, co-author of Continuous Delivery, to help. Dave has been working in organizations to implement hands-on process improvements since before he published his book with Jez Humble in 2011. Over the years, Dave’s breadth of experience has helped him understand which approaches work well and which don’t. There is plenty of information in the industry, but having someone with David’s experience curate it down to specific practices is perfectly designed to help engineers avoid frustrations that often result in giving up.

Lessons from Manufacturing Improvements

Manufacturing has a long history of successfully optimizing processes and quality. As Japanese automobile companies started to gain market share in the U.S. with better quality during the 1980s, U.S. companies realized they needed to adapt to survive. So, U.S. manufacturers started going to Japan to see what they were doing and copying their approaches. Toyota, the leader in the space, was more than willing to have any competitor tour their factories to see how they were winning in the market because they understood their own key to success was not how they were manufacturing in the present day. Instead, it was their systematic approach to continuous improvement. The U.S. companies did not realize this and wasted years trying to copy what worked well for someone else. After creating complex, detailed approaches to try and copy what they saw Toyota doing, they found limited success. This feels very similar to the current state of software organizations in companies around the globe. The industry observes what is working well for other organizations, distills it down into complex frameworks, and then implements that approach, which may or may not have similar problems. It’s the mindset that if you just follow these rituals, you also will be successful. It focuses on telling people what to do without ever taking the time to understand the real issues, and is frequently met with resistance.           

The U.S. automobile industry tried this approach for a long time before it realized it wasn’t working. That’s when it  started focusing on its own systematic approach for continuous improvement. Software organizations also need to understand that implementing complex frameworks that were designed to solve someone else’s problems aren’t going to deliver the magnitude of improvements their particular business requires. Creating a culture of continuous improvement through a systematic approach that is focused on addressing their unique challenges is more important. 

This systematic approach can’t just copy what was done in manufacturing because software is a very different beast. In manufacturing, you are creating the same (or very similar) products repeatedly. The quality is improved by controlling the process to reduce and eliminate defects. It is easy to measure the flow in terms of the number of widgets created in an hour -- so you can implement changes and measure the results to see if it helps. There also is one organization, R&D, focused on designing the product and another, manufacturing, fully focused on improving quality and flow.  

Software is very different. Every time software is developed, it’s for the first time. You are trying to understand a business requirement and implement software to meet that need. Every time you run the software manufacturing process of build, deploy, test and release, the product has changed. Each requirement is a different size and shape so you can’t really measure flow for software. The product is not created by a process that can be controlled to reduce defects. It is created by individuals working with the business and customers to design solutions to meet their needs. The same team is responsible for creating the product and improving the process. There also are steps, like creating an environment, building, deploying, testing, and releasing software, that are more repetitive and can be automated and optimized.  

Improving the creative part of the process is not about controlling it and eliminating defects like it is in manufacturing. Instead, it’s about providing the best quality feedback to developers as quickly as possible so they can learn about the unintended consequences of changes while still working on the code so it is fresh in their minds. To help them improve, we need to accelerate feedback on their code from tests, business owners, and ultimately customers. Ideally, we want to provide this feedback in small batch sizes with a limited number of changes, so it doesn’t take extra time and effort to determine which changes had the unintended consequence, and what changes provided the expected benefits. 

Improving the repetitive part of creating environments, building, deploying, testing, and releasing requires a different approach. These processes can be automated. This automation enables consistency that can be optimized and improved over time. It also changes the economics of running these processes so the iterations can be more frequent, and enable smaller batch sizes that are easier to triage and provide shorter feedback for developers. 

Manufacturing realized decades ago that just copying solutions that were designed for someone else’s problems wasn’t going to deliver the required business results. Instead, they created a systematic approach and a culture for driving continuous improvement. The software industry would be negligent to ignore this shift. Instead of focusing on rolling out complex frameworks that were designed to solve someone else’s problems, a more systematic approach to continuous improvement can enable teams to align on the improvements that will help their organizations the most.     

Lessons from Organizational Change Management

Getting organizations to invest in improvements and embrace new ways of working is a challenge. They don’t just need the right technical solutions, they also need to address the organizational change management challenges that are creating resistance to new ways of working. Organizations frequently have champions that have ideas for improvement and are trying to influence change without a lot of success. These champions find that the harder they push for change, the more people resist. We can have all the best approaches in the world, but if we can’t figure out how to overcome this resistance, organizations will never adopt them and realize the benefits.  

While pushing for change is the natural approach, research by organizational change management experts, like Professor Jonah Bergerin his book “The Catalyst,” suggests this is the wrong approach. His research shows that the harder you push for change, the more people resist. Whenever they feel like they are trying to be influenced, their anti-persuasion radar kicks in and instead they start shooting down ideas and resisting the change being offered. If we want to influence organizations, Professor Berger recommends focusing on understanding and reducing the barriers to change. He highlights five barriers to change and recommendations for reducing those barriers:

1. Reactance: People react and dig their heels in when they sense you are trying to influence them. Instead of trying to influence them to your way of thinking, work to understand their perspectives and give them options to pick a path to get where collectively we are hoping to go.
2. Endowment: Most people are used to what they’re doing and don’t want to switch. Instead of trying to influence them based on how much better your idea is, help them understand the risks of doing nothing.
3. Distance: If new information is within people’s zone of acceptance, they are willing to listen. But if it is too far away, in the region of rejection, everything flips and can increase opposition. Instead start by working with people that are more open to ideas first, which Berger calls the “moveable middle” or start by asking for something less that would fit within their zone of acceptance.
4. Uncertainty: Change requires uncertainty that can increase resistance. Reducing that uncertainty requires reducing the risk of the change. This can be done through things like reducing the effort to experience something new and/or making it reversible.   
5. Corroborating Evidence: Sometimes, one person’s opinion - no matter how knowledgeable - is not enough to convince people. They need to hear this idea from multiple sources. And, the more similar the source is to their current experiences, the more likely they are to be influenced by this evidence.  

Professor Berger’s book is highly recommended to any change agent trying to influence organizations. His ideas for understanding and reducing the resistance to change are highly impactful. Getting organizations to invest in improvements and embrace new ways of working is going to require leveraging ideas like the ones above from change management experts.   

Engineering the Digital Transformation

After the brief history lesson, how can we apply what we know to create a systematic approach for continuous improvement designed specifically for software? I took these learnings from my experience leading transformation journeys to author my fourth book, Engineering the Digital Transformation. Books are great, but they can be a non-starter for large organizations looking to get started on systematic continuous improvements today. They can also be construed as a forcing function from the top-down to get everyone to read the book and report back what was learned. So, to help meet people where they are at, during these ever-changing times, I developed a computer-based training program based on concepts in my book that are designed to be self-paced and guided along the way.

It leverages some of the principles that have proven to deliver dramatic improvements for manufacturing but modifies these approaches to take advantage of the unique characteristics and capabilities of software. It is slightly modeled after manufacturing’s Six Sigma (6σ) in that the basic level of training and certification of a “white belt” is based on ensuring the person taking the training first understands the concepts. The higher-level green belt and black belt certifications require the trainee to demonstrate they can apply these concepts to make their organization more effective and efficient.  

This next step is the goal. Knowledge for knowledge's sake is not the end goal. My goal is to motivate as many people as possible to start making improvements and help them as much as possible on their journeys.  It is not a complex methodology that should be implemented based on what worked for someone else. Instead, the white belt training is a straightforward, common sense approach designed to help someone analyze their current software development and delivery processes to make their biggest opportunities for improvement visible. The green belt certification is designed to motivate as many people as possible to implement an improvement project because that is where the real learning occurs. And, this is where the business value will be delivered. The green belt training leverages David Farely’s experience to provide as much help as possible for the improvement projects.  The black belt certification is designed to enable companies to be self-sufficient over time by allowing them to manage, coach, and certify their own green belt projects.      

Above all though, Engineering the Digital Transformation training program is designed to help reduce the barriers to change, instead of trying to tell people what to do based on what worked with others. It provides better corroborating evidence by helping them to understand and make inefficiencies in their current approach visible.  Not by telling them why a new approach is better, but by reducing the endowment barriers by allowing them to see the waste that will have to deal with every day if changes aren’t made. It helps reduce the reactance barriers by teaching people to analyze their processes and highlighting the types of things that can help but then allowing them to choose a green belt improvement project that is within their zone of acceptance. And, which project they believe will benefit their business the most. It is not a “big bang” approach with lots of barriers born out of the uncertainty of change, but instead creating the ability of a systematic approach to continuous improvement with lots of small changes that don’t require huge investments, and are easy to reverse.    

White Belt Certification

The white belt certification is self-paced, computer-based training. Through a series of video, learning modules, knowledge checks, and assessments, this course will prepare you to begin your journey to making your deployment pipeline visible, build in quality, optimize workflow, and identify inefficiencies in your process. Through scenario-based assessments, trainees apply the skills they’ve learned to identify issues commonly seen in the industry. This training teaches a systematic approach to analyzing a broad range of applications.

Because it is computer-based, it can easily integrate with learning management systems. It can be used across a large organization quickly and efficiently and gives everyone a common language and approach for thinking about the problem, and motivates people to further complete Green and Black Belt Certifications.     

Module 1: Manufacturing has had a long history of continuous improvement.  This module will review the major innovators and the improvements they championed to provide a background for concepts and approaches that can be applied to software.

Module 2: Provides a common approach for making your software development and delivery processes visible.  Because the software processes are so hard to see and measure, the first step to understanding the problem you are trying to solve is making it visible. Different applications, architectures, and organizations have different challenges. The first step to gaining alignment across the organization on the improvement journey is making the process visible so you have a common understanding of the challenges unique to your organization.

Module 3: Improving the creative side of software requires improving the quality and speed of feedback to developers. This requires automating and optimizing the repetitive tasks so you can shift to smaller batch sizes that are easier to triage and build in quality.  The problem is that it’s hard to automate all the tasks at once so you want to start with the improvements that will help your team the most. This module provides a systematic approach for analyzing and making your specific challenges visible. 

Module 4: Is focused on understanding and improving workflow for software. Since every new idea or requirement is a different size or shape, we can’t really measure flow like we can in manufacturing. Instead, we need to focus on identifying the sources of waste that are slowing down flow. This module provides a framework and metrics for making those issues visible.

Module 5: Looks across all aspects of your product, architecture, and processes to highlight different opportunities for improvement. Getting people to invest in improvements and embrace new ways of working isn’t about telling people what to do. Instead, we need to take Professor Berger’s advice and offer people a choice so they can pick something within their zone of acceptance and start the improvement journey. Module 5 looks across all of the different ideas so far letting people choose the improvements they are passionate about making successful.

Module 6: Is focused on the planning and requirements process. Software has some unique characteristics and capabilities that enable a different approach, but most organizations still leverage planning that was designed for physical assets.  This module reviews those differences, provides ideas for improvements, and makes the waste in the current approaches visible.

Module 7: Creating a systematic approach and culture for continuous improvements requires leadership. This module reviews their role in leading teams and prioritizing work across teams.

On completion of the training and white belt certification, trainees will be able to create a culture of continuous improvement, analyze a broad set of applications in their organizations, prioritize the best opportunities for improvement, and implement Green Belt improvement projects. 

Green Belt Training

In partnership with David Farley, his Green Belt training is designed to help with the improvement projects themselves. Once people are excited to make improvements, we want to ensure they maintain that momentum.  We don’t want them getting frustrated with new approaches and giving up on the improvement. We also want to help them avoid approaches that are fragile and hard to maintain. The training leverages David’s years of experience to provide his own content and curated content from others to help people understand what works and what doesn’t. It can’t cover every technology for different organizations, and it is not intended to show or tell people exactly what to do. Instead, it’s purpose is to demonstrate general design principles and approaches that can be applied to a wide variety of applications.     

David provides his training in a wiki format that enables people to focus on the content that will help with the issues they are addressing. There are many different ways to use the wiki, but David provides a few different starting points and recommended paths for topics. He starts with high level principles and definitions before diving deeper into implementation. His content includes very simple working examples so people can see the practices in action. The code can be reviewed to see examples of good design patterns and the consequences of bad design patterns.  

It includes definitions, tips, videos, code examples and everything he believes will help organizations with improving software development. And, like anything software related, it is designed to be updated and improved over time with feedback from the users.   

Black Belt Certification

The Black Belt Certification is further demonstration that the approaches and principles of White Belt training have been used successfully to transform how software is developed in your large organization. There are two types of Black Belt Certifications. First, there is certification to ensure that people proficient in White Belt principles can certify their own Green Belts, enabling companies to be more self-sufficient. This certification process is to ensure people can successfully coach Green Belt projects and ensure results are significant enough that the certifications are meaningful. Second is to recognize larger and longer continuous improvement projects that can last years instead of months. These are projects that may be of interest once the Green Belt Certification is completed.

Summary

The goal of Engineering the Digital Transformation is to lay the foundation for creating a systematic approach to continuous improvement that helps to reduce the barriers to change. This approach is focused on teaching organizations to understand and address their unique challenges instead of telling them what to do based on what worked for others. The approaches broadly used in manufacturing took generations and numerous thought leaders to fine-tune and improve. Software teams will need similar efforts to create approaches as valuable as has been accomplished in manufacturing. We can learn from manufacturing’s efforts and leverage as many as possible, but we need to understand that software is a very different beast that will require different thinking and ideas. My hope is to help as many people as possible lay the foundations to start that journey.  

About the Author

Gary Gruver is an experienced executive with a proven track record of transforming software development and delivery processes in large organizations. As the R&D director over the HP LaserJet FW team (~800), he led productivity improvements of 2-3X. Then, as VP of QA, Release, and Operations at Macy’s.com, he managed the journey toward continuous delivery. His current passion is helping as many people as possible achieve similar results through consulting, webinars, presentations, and books. Gary’s consulting approach is unique in that he stays engaged with organizations during their journeys to provide guidance when they hit specific challenges and to learn about what is and isn’t working during as many different transformations as possible. This experience enables him to help clients avoid common mistakes and share approaches that have been proven to deliver business results for a variety of companies.