Codeweavers combines the Toyota way with extreme programming and continuous delivery in development and support to do small, frequent releases. The advice from Paul Boocock to apply the Toyota way is to start with the books, understand the philosophy, and begin teaching it to others.
Paul Boocock, academy leader at Codeweavers, spoke about how Codeweavers applies principles from the Toyota way at Agile Cambridge 2017. InfoQ is covering this conference with Q&As, summaries, and articles.
InfoQ interviewed Boocock about why they decided to use the Toyota way, how they do continuous integration, how the Toyota way is applied in support, what they learned from applying the Toyota way, and which benefits it has brought them.
InfoQ: What made you decide to start using the Toyota way at Codeweavers?
Paul Boocock: We’ve been practising agile techniques since 2007 and after a very short stint with Scrum we soon moved towards Extreme Programming (XP) and Continuous Delivery (CD). As we continued on our agile journey we started adopting a variety of ideas including Lean and Kanban which led us towards The Toyota Production System. Whilst on the surface it is all about manufacturing cars if you dig a little deeper, particularly into Taiichi Ohno’s work, you soon start to see how these rules can be applied to all kinds of manufacturing and we saw many parallels in what we were reading about the Toyota Production System and the way we delivered our software. The Toyota Way then takes the guidelines set out in The Toyota Production System and gave us 14 principles that we kind of use as a guiding light when making decisions on how we work. We’re still within an Extreme Programming methodology but the Toyota Way helps us when we come to a tricky junction and have a decision to make.
Not too long ago, we were faced with an issue of increased numbers of exceptions occurring on production servers. There were no glaring issues, just occasionally exceptions in code paths we hadn’t expected to be used in a certain way. On the surface, it might have been tempting to form a development squad to tackle this issue or perhaps assign it to someone to take away from the meeting as an action point to look into further. That mentality doesn’t work for us; The Toyota Way teaches us two things that prevent this as a viable option. First, "Become a learning organization through relentless reflection (hansei) and continuous improvement (kaizen)" tells us to dig deeper; we must find the root cause of the problem so we can improve. Noticing we have increased exceptions and patching the Null References and Timeouts would be one thing, but why are we getting those in the first place? Why did the code with these defects make it to production? Why were we focused on delivering new features with defects in production? Why didn’t we stop to fix the issues first? Performing this sort of "five whys" helps us find the root cause rather than just considering the initial problem. Secondly, "Teamwork is everything"; so we cannot delegate this responsibility to a single person or small group of developers to fix. We must get the entire team working in harmony so we don’t end up back here in the future, one person going faster than the rest usually makes the system actually go slower. In this instance our root cause helped us find that our teams had become "immune" to the noise on the monitoring screens when exceptions and timeouts occur, they had slowly started creeping up and it wasn’t obvious through our monitoring that were becoming more severe and what impact they were having. We adapted by looking at new ways of designing our live monitoring screens (more colours and sounds) to make it clearer not only where the errors were occurring in our platform but the knock on consequences they can cause when they go unresolved (monitoring and displaying deploy times from local to live, hourly deployment counts and HTTP error code responses).
InfoQ: What is it that makes the principles of the Toyota way so valuable?
Boocock: Every day we are required to make a variety of choices and when we don’t have any help, the ambiguity can paralyse us. This is where ideas like The Toyota Way help us as we use these ideas as a guide and philosophy. One idea that we rely on heavily is that of "Repeating Why Five Times" whenever we have a problem and we want to find the root cause; before we used this technique we found ourselves in that typical situation where we are looking for solutions to the problem rather than finding the root problem and looking to solve the root cause. We use this technique in all areas of our business, whether we are looking at issues with our current processes, looking into a production service failure or considering why something is performing the way we expected. Another core idea is to "Eliminate Waste" and we see this idea applied mainly in the way we continuously deliver our software - from reducing build times and reducing inventory in development projects to not planning ahead months at a time. There are other ideas too such as "Teamwork is everything" and "The Skill of Passing the Baton" that unpin our culture of working together and not having certain team members working twice as fast as everyone else and upsetting the boat (there’s a great analogy about a team in a rowing boat, when one goes faster than everyone else the boat gets to its destination slower as it doesn’t go in a straight line and others have to try and keep up or compensate for this).
InfoQ: How are you doing continuous integration?
Boocock: We run a single trunk workflow (no branching) across our entire codebase; this leads to more frequent integrations and less inventory. Each commit to our trunk, builds and tests the affected application and then queues that up for deployment. We don’t branch and we advocate small frequent releases, often performing a release of a single line of code. We can do this as we follow the idea of Autonomation from Toyota, which is Automation with a Human Touch. We have automated almost our entire delivery pipeline, being able to go from a commit at a local development machine to our production environment in 10-15 minutes. However, if something goes wrong our delivery platform instantly highlights this issue and stops all work which helps us eliminate any defects before they reach our production environment. This is often referred to as Jidoka in The Toyota Production System.
InfoQ: How have you applied the Toyota way in support at Codeweavers?
Boocock: Our support team is very much driven by our customers and production levelling is a constant challenge. We have recently been considering ways to improve our levels of customer support by applying ideas from The Toyota Production System. An interesting development in our support team over the past 12 months has been its growth from 3 people to a team of 8; this sort of growth presents a challenge as we believe in developing our people to add value to our organisation, and this puts a strain on those at Codeweavers who understand our philosophy and can teach it to others. Growth brings other challenges in a team and having the courage to stand up to traditional thought, and often common sense, when it comes to customer support is something we have been known to do. It’s all too easy to slip into specialisms, with certain people doing one area of support and others doing another thing, but it’s much easier for the team as a whole if everybody is skilled in all work areas; in a car manufacturing plant a welder who has been taught how to paint is more useful to the production of cars than someone who can just weld. We try to get this philosophy out across our support team, ensuring a wide level of knowledge of all areas of our platform so the team can help each other out. Specialisms are still important but working as a team comes first, much like in many sports. Trying things out to help level the workload across our support team has been really important; this has helped get the entire team up to speed and working with more cohesion.
InfoQ: What have you learned from applying the Toyota way?
Boocock: I’ve learnt that traditional western ways of working are not necessarily the best, batching up work (and increasing inventory) and mass production are wasteful and it’s become clear why small, frequent releases (think XP and CD) are a step in the right direction. I’ve adopted working techniques that help me work better with the team and ensure my workload is sustainable. A game changer for me was to focus on single piece flow, that is to do a single piece of work at a time until it can be marked as "Done"; I feel more productive and it’s a great motivator when I get to move the card into the done column.
InfoQ: Which benefits did it bring Codeweavers?
Boocock: The speed at which Codeweavers can deliver quality working software still consistently amazes me. The principles that have been taken from Toyota unpin many of the ideas that have made this possible. It’s created a great way of thinking about delivering work and ultimately that has made it a great place to work. The Toyota Way Book description states, "Toyota consistently makes the highest-quality cars with the fewest defects of any competing manufacturer, while using fewer man-hours, less on-hand inventory, and half the floor space of its competitors," and it feels like Codeweavers are also feeling the benefits of this having adopted practices from The Toyota Way.
InfoQ: If an organization would like to use the Toyota way, what advice do you have?
Boocock: Start with the books. Once you start to understand the philosophy, begin teaching it to others. There are three important books for me: The Goal by Eliyahu M. Goldratt, Toyota Production System: Beyond Large-Scale Production by Taiichi Ohno and The Toyota Way by Jeffrey K. Liker. Reading these books is not a silver bullet and at times the ideas may seem woolly and hard to follow, but Codeweavers are proof that these ideas and principles can help transform a business from one that releases software to customers a couple of times a year to one that releases over a 100 times a day.