At the recent DevOps Days in New York, Kevin Behr, co-author of “The Visible Ops Handbook” and ”The Phoenix Project”, and Jesse Palmer gave a talk on how they instilled a continuous improvement culture into an operations team that was in a permanent state of overwork.
According to Kevin and Jesse, through this process of continuous improvement the team’s work went through a radical change: Dev and Ops are working closely together and unnecessary work and rework are being steadily eliminated. The process also helped the team achieve an important breakthrough when they came to the conclusion that a coherent narrative to guide their work, a mission, was missing.
They started with a diagnosis of the current reality. This diagnosis was supported on several frameworks, like the theory of constraints and Cynefin, a framework to reason about the evolutionary nature of complex systems and their inherent uncertainty as the guiding frameworks. By using a current reality tree, they mapped the top problems the organization was having to the root causes that created them.
The team also started doing short, bounded experiments through a technique promoted by Mike Rother’s “Toyota Kata” book, the improvement kata. An improvement kata is a way to bring the scientific method to the daily activities of an organization, in order to solve problems and their root causes. The specific improvement kata used in this instance was based on a daily stand-up meeting where five questions were answered:
- What is your target condition?
- What is the current condition?
- What are the main obstacles to achieving the target condition?
- Which obstacle will you experiment on today?
- What is your experiment and how can we see the results of the experiment?
For instance, one common complaint was “We don’t have enough time”. The team dug deeper and found that everyone was constantly trying to find the most useful thing to do on a given point in time, but without visibility to the whole picture, so they adopted Kanban. Kanban allowed the team to visualize their entire value stream, manage their flow and set the stage to measure their experiments and progress.
In the process, the team identified several target conditions, such as “Pay down technical debt” or “Manage unplanned work”. To achieve these target conditions several experiments were done, such as defining critical technical debt projects or creating simple project plans so the team didn’t loose track of bigger activities. These experiments led to concrete results: doing root cause analysis to avoid the Groundhog Day effect was one of them. The operations team also started to solve common issues with the development team, such as improving the deployment procedures. This was possible because a feedback loop was built between the two teams.
InfoQ asked Kevin about some of the techniques and practices mentioned in the talk:
During your talk, you explained the improvement kata used on that organization and on that context. Do you always use the same improvement kata or are there multiple improvement katas to choose from, depending on context?
There are many forms. We often start with the improvement kata similar to what Mike Rother describes in his book “The Toyota Kata”. Our version is part of an approach or broader form we call Opsflow. It uses various methods and assemblies to hit the seven dimensions of organizational learning in order to build co-adaptive innovation capabilities and to evolve/cultivate latent talent while reducing cycle and wait time for value delivery.
You mentioned that a target condition is not just a goal or a KPI. Can you elaborated on that? What is a target condition?
Goals are too far off to impact with minute by minute work decisions. Challenges may be smaller units of accomplishment than goals (several challenges may lead you closer to a goal) but even smaller and closer to the worker is the target condition: a measurement or measurements that are at the individual worker's metaphorical workstation or work center. Could be a process level metric (widget output per day) or a knowledge work metric. Think how I must work at my desk so that we can achieve a challenge. Figure out how to measure that as a condition you could obtain. It helps to be coached in this extensively at the onset. It is easier caught then taught.
What mechanisms do you use to find target conditions?
We use many techniques but the best leverage are organization’s goal/missions/tactics/plans/sorties, what ever the management approach is to mission definition. All target conditions need to be inside of a corridor that management establishes - then they coach only to make sure the target conditions will get us closer to the challenges which move us all closer to the goal.
How do you decide what target condition to focus on?
We use thinking tools adapted from Eli Goldratt’s theory of constraints such as current reality tree, future reality tree, transition tree and goal mapping. We also use methods and assembles designed by David Snowden such as future backwards, ritual dissent and others to help get around cognitive bias and identify root conditions that need to be changed.
One of the most difficult things on continuous learning, or continuous adaptation is the "continuous" part. It is easy to start with lots of momentum, but that can fade over time. Do you have any tips and tricks on sustaining the continuous adaptation on the long run?
Yes! First make sure the forms are learned correctly and deeply. Establish and cultivate a culture of scientists by making learning about the scientific method. Create a coaching and second coaching program that establishes culture keepers. Also introduce weekly retrospectives that talk about failure and celebrate success. We see that the hardest part to sustaining efforts is that each challenge when met is immediately replaced with another so it can feel like you are on a treadmill (in same place working - but no big bang success). Since the improvement to fail ratio is low it can feel discouraging. When teams regularly look back on their success and lessons they can use that sense of accomplishment to motivate even more improvement. That is key.