Q&A on the Book It's All Upside Down

In the book It's all Upside Down, Paul McMahon provides stories from software development teams supported by upside down principles and coaching tips for applying them. He explains how you can use Essence to improve processes leading to better organizational performance.

InfoQ readers can download a sample of It's All Upside Down.

InfoQ interviewed Paul McMahon about why there is still a need for processes when organizations adopt agile, things that work and don't work for improving processes, dealing with incomplete or vague requirements, the benefits that teams can get from coaching and how tech leaders can improve their coaching skills, why he choose Essence to explain the software development stories, and advice to organizations that want to use Essence to improve their performance.

InfoQ: Why did you write this book?

Paul McMahon: The subtitle of the book, What I've Learned About Software Development and Why it Seems Opposite to Everything I was Taught, has a lot to do with my motivation for writing the book. In just the past few years I have made such drastic changes in how I help software development teams as a coach that it often appears what I am recommending to my clients is the complete opposite of many well established long held software engineering principles. This was my major motivator for writing this book. I wanted to share true stories of what we have recently discovered really works for successful software development teams in practice. There are eight stories in Part I of the book that all occurred in 2015 and 2016.

InfoQ: For whom is this book intended?

McMahon: In this book I highlight 26 upside down principles along with upside down principle clarifying thoughts. I also highlight 18 coaching tips. If you are wondering if a book that highlights coaching tips is for you, it is. I believe everyone in an organization should view themselves as a coach. This book is particularly for anyone in an organization who can affect or is affected by software development decisions.

InfoQ: The manifesto for agile software development suggest to focus on people over processes. Is there still a need for processes when organization adopt agile?

McMahon: Yes. There will always be a need to provide team's appropriate guidance to ensure the team understands its mission and how to handle the common situations the team faces. Unfortunately, too many people today equate processes (or practices) with over-weight written guidance. There will always be a need for processes as long as you accept my definition of process as appropriate guidance to teams. The challenge we face is how to determine the right level and right vehicle to provide this guidance. One of the insights I have recently discovered is how to capture processes that fit agile organizations by focusing on principles. Principles focus more on the goal, whereas processes (or practices) focus more on the steps to achieve a goal. I now view principles as "mini-practices" (or mini-processes). This approach helps teams focus on the right behaviors for higher performance.

InfoQ: Can you give examples of things that you learned don't work for improving processes?

McMahon: Great question. "Things that don't work for improving processes" actually could have been a good subtitle of the book because this is really what the entire book is about-as well as being about things that do work. So I will just give a few examples here. One of the things that doesn't work is trying to "improve" processes by defining the process in detail in the written word before your team has proven out the "process" on a real project. This doesn't work because we have learned that no matter how much effort you put into defining processes up front the team is going to learn important new things as soon as they start using the process on a real project. Another thing that doesn't work is buying into the false belief that you have to define measures first in order to be sure you have improved. It is true that you do need to measure to ensure you aren't fooling yourself, but some of the best measures are discovered right when the improvement is being implemented on a real project. An example can be found in Story One of the book. A third thing that doesn't work is believing that a repeatable process is necessary to gain repeatable results. Sometimes the best path to repeatable results is to allow the team to vary their process to address a specific challenge they are facing. This is the point of Story Two.

InfoQ: What are some things that you learned do work?

McMahon: The most important thing I have learned that does work is to recognize that-- while the agile movement has highlighted many sound practices-- most projects have specific constraints and/or regulations that required a degree of tailoring of "pure agile" practices. So what works best is to recognize this and to coach teams and team leaders in how to go about finding that right level of "hybrid proven best practices" that fit their culture and environment. Too many organizations think "tailoring" is only relevant to the start of a project (if at all), whereas in reality tailoring needs to be considered continually throughout the project. At the same time, unbridled tailoring can breed chaos. Therefore another thing I have found that does work is to provide a minimal set of "must do" guidelines that must NEVER be "tailored out".

InfoQ: What do you suggest teams should do when they are dealing with incomplete or vague requirements?

McMahon: Another great question! One of the things I like to do to help teams-- instead of talking about theories-- is to share stories of what works. One of my favorite stories I like to share relates to incomplete or vague requirements. I was giving a workshop many years ago to a client that develops simulation training software for the US Navy. We were discussing incomplete requirements and one participant interrupted saying, "Let me tell you about the problem I have with my customer". He continued saying that he was trying to get his customer to clarify a vague requirement, and the customer responded with, "We need simulated missiles that fly high and go fast!". He then asked his customer, "How high and how fast?", and the customer responded with, "I don't know, just build something and we'll let you know if it's good enough".

What was most interesting was that the discussion among the other workshop participants did not result in any agreement as to whether this was actually a problem or not. One participant replied, "I work with my customer that way all the time, and they love us!" While another said, "This is a huge problem because my customer is always complaining that we can't hit cost and schedule".

The point of this story is that to determine if this scenario is a problem or not, you have to understand your customer and the project situation. You have to ask more questions first, such as, "How much money does your customer have?" and "Does your customer understand the cost of iterating this way to discover the requirements?"

If you customer is not collaborative and if you have a fixed cost and schedule, this could be a huge problem and you probably need to take action immediately. But, on the other hand, if your customer understands the cost of iterating to determine requirements, and your customer wants you to help him/her discover the requirements and has enough money to pay for it, then you probably should just keep doing what your customer is asking you to do and paying you to do.

InfoQ: What benefits can teams get from coaching?

McMahon: It is only in the past few years that I have come to realize how valuable coaching can be. I used to put a great deal of energy into developing my formal workshops for my clients that included literally hundreds of PowerPoint slides that I would use in workshops that would often last three or more full days. I did tailor these workshops to specific client needs and the workshops were well received. Therefore, I continued this approach until just the last few years.

But in 2015 I started to realize some important things that were missing from my formal classroom training that I could provide much more effectively through on-the-job personal coaching. When workshop participants are in the classroom, it is often difficult for them to translate what they are learning to how this new knowledge affects their actual project tasks. Therefore, too often when they leave the classroom and face a situation on the job where something I tried to teach them in the classroom was applicable, they miss the opportunity to apply it. This is because they just fall back into old habits, which is natural for many of us. The primary benefit of a coach is that as a coach you can be closer to the real challenges the student is facing, and can thus recognize the opportunity to apply a new behavior right at the point it is called for. We miss this opportunity when we limit our interactions with our clients to formal classroom training alone.

Some have argued that coaching, especially one-on-one is significantly more expensive than classroom training, but it does not need to be.

Many organizations still train their people by sending them to multi-day comprehensive formal workshops. But replacing these lengthy workshops with short streamlined classroom training supplemented with on-demand coaching can be more efficient and more effective. This saves cost in the formal training, and it allows coaches to work with more clients in parallel by providing the coaching just as needed through email, short phone calls and short focused webinars as the situation demands.

This approach is proving to be both more effective and efficient for coaches as well as clients. Another great positive side-effect of emphasizing coaching over formal classroom training is that it doesn't need to be viewed as purely an "overhead" cost for clients, because the coach and the student are working on a real project challenge.

InfoQ: How can tech leaders improve their coaching skills?

McMahon: First, I would like to point out that when I was a young software developer the first two times my manager tried to convert me into a technical leader I hated the new job, and as soon as possible I got out of that job and went back to being a developer. In hindsight I realize I did this because I was given no training or guidance in what a technical leader's responsibilities are and how to carry out those responsibilities. The first point I want to make on this topic is that organizations should make it clear to their practitioners who is responsible for technical leadership and how technical leadership skills are acquired.

Today, I believe becoming a technical leader should not be a choice that developers have to make that includes giving up developing software. The best technical leaders should also be developers so they keep aware of the latest practices, tools and issues that developers are facing. Therefore, in today's world, technical leadership should come from wherever the knowledge exists on your team that is needed, and when it is needed.

Given this line of thinking, improving coaching skills of technical leaders has more to do with team communication and understanding team member responsibilities, than anything technical. It is about listening first, understanding the issues your team is facing, and then having access to the people with the experience that can help. It also means communicating to every team member the fact that part of their responsibilities include helping teammates when they have the knowledge needed, and one of their teammates needs a little help.

InfoQ: What is Essence?

McMahon: Essence is based on the Software Engineering Method and Theory (SEMAT) initiative which was initiated at the end of 2009 envisioned by Ivar Jacobson, and two advisors Bertrand Meyer and Richard Soley. There are many people from around the world, including industry, academia and research, who worked as volunteers leading to the Essence Object Management Group (OMG) standard. While Essence has only been an OMG standard since 2014, its roots can be traced back to 2005 when Ivar Jacobson International (IJI) redesigned the Rational Unified Process (RUP) and presented this as ESSUP (the Essential Unified Process). One of the prime motivators behind SEMAT is stated on the SEMAT web site: "There are millions of software engineers on the planet in countless programs, projects and teams; the millions of lines of software that run the world are testament to their talents. However, as a community we still find it difficult to share our best practices, truly empower our teams, seamlessly integrate software engineering with other engineering disciplines and into our businesses, and maintain the health of our endeavors, avoiding embarrassing and unnecessary catastrophic failures. The industry's habit of constantly switching between no method and the latest "one true way" (an affliction that sadly is even affecting the agile community) is not the way forward".

The SEMAT community is not alone in observing this trend. Capers Jones, who has been collecting data on software quality, risks and best practices for many years, told me that software engineering has created more variations of software methods than any other engineering discipline in human history.

InfoQ: Why did you choose Essence to explain the software development stories in your book?

McMahon: In the stories in my book you will learn many questions that I asked my clients and coaching tips I gave them to help them overcome common obstacles and achieve higher performance. Many of the questions I ask and the tips I provide are based on experience. But Essence, I have found, provides a simple way to share these questions and tips with practitioners in a form they can easily access when a coach isn't immediately available.

I sometimes refer to Essence as a "thinking framework", but, in fact, Essence is more than just a "thinking framework". Essence provides a "common ground" set of essentials relevant to all software development efforts. This point is also relevant to a previous answer I gave related to the need for a minimal set of "must do" guidelines that must NEVER be "tailored out".

Essence includes a set of checklists that can stimulate the kinds of questions that I ask my clients. You can learn more about these questions in my stories in Part I of the book. Essence also includes a way for anyone, including consultants, coaches, practice/method authors and practitioners, to express their practices and tips in a common language highlighting the unique discriminators of their approach. This way of expressing practices in the Essence language is referred to as "essentialization". This is fundamentally why I chose Essence to explain the software development stories in my book.

InfoQ: What's your advice to organizations that want to use Essence to improve their performance?

McMahon: The best place I have found to start using Essence is to use the framework as an aid to stimulate discussion leading to root causes and resolutions of key problems that are getting in the way of an organization's higher performance. A good example of this is the way we used Essence in the NORO case study described in my book.
NORO is a small company that broke off from a larger CMMI Level 3 organization that developed complex software systems for the US Department of Defense. NORO wanted to be more agile so they dropped the heavyweight processes of their previous parent organization, but soon realized they had gone too far and needed to add back a degree of disciplined processes.

The goal with NORO was to add back just enough discipline to solve the key pain points that led to dissatisfied customers, but not add back the bureaucracy that hindered efficient and effective operations. By using Essence to help us conduct a root cause analysis we were able to rapidly identify two key problem areas including:

• Not adequately involving key stakeholders
• Poor testing leading to escaped defects

We then gave the NORO software team some streamlined training focusing on key goals together with a few simple practices that helped the team resolve both problems.

Within just eight weeks the NORO team was able to achieve 100% measurable performance improvement that was validated by surveys with previously dissatisfied customers. Using Essence to aid root cause analysis is not the only area where Essence can help organizations, but, as we demonstrated with the NORO case study, it is a great place to start to demonstrate the power of Essence to help teams rapidly improve performance.

About the book author

Paul E. McMahon is Principal Consultant at PEM Systems where he has been coaching large and small teams in practical approaches to increase agility and process maturity for the last twenty years. McMahon just released his fifth book, "It's All Upside Down: What I've Learned About Software Development and Why Its Seems Opposite to Everything I was Taught" where you can find more true upside down stories of successful software development teams. When he isn't coaching you can find McMahon running marathons. He has completed 12 Boston Marathons. You can reach MacMahon here.