So, How Do You Make Agile Successful?

Last year, I wrote an article (“Why Agile didn’t work”) that provoked a lot of discussion. Some people have been badly hurt by Agile, while others have had good experiences with Agile. These discussions usually boil down to the same conclusion: “it is not about process, it is about people.” That is hardly surprising. A team is made up of people, so of course Agile is about people. But software is no longer an elitist industry. Normally in a team, there are stars and there are average engineers and average managers. When teams come to Agile for help, Agile’s answer should not be “First, get yourself together, then I will show you the Agile magic.” In fact, that is what I do not like about Agile. When Agile works, it is because Agile’s “principles” are brilliant; and when it doesn’t work, it is because “I only prescribe principles, you are interpreting and implementing them wrongly.” Answers like this are dodging responsibility. We need a more constructive response than that.

To formulate one, let us first go to the beginning and figure out what the essence of Agile is.

The essence of Agile

Scrum, Lean, and Lean Startup have one thing in common: deliver faster to receive quicker feedback. But why do we need quicker feedback? Because we need to learn from feedback. When we start out, we (everyone involved in producing a software product) don’t know much, so we need fast feedback to clarify requirements and discover risks; and at every stage of development we need rapid feedback to catch bugs and improve product quality.

Some managers may say, “I pay you to work, not to learn!” Well, it’s a tough awakening for these managers. Software engineering is knowledge work: there are no assembly lines in which you can just take any group of software engineers and demand them to produce high-quality software.

Agile provides a learning opportunity (or challenge) for users and customers too. Statistics have shown that only 20% of features are frequently used, while 64% are rarely or never used. With Agile, we hope to identify those 20% of features and not to waste time on the 64%. Agile doesn’t mean that users can change their minds on a whim, rather it means they need to make an effort to think carefully what they need and to participate in the development process in order to “maximizeing the amount of work not done”.

But be careful. Learning is easier said than done. To be able to learn, we first need to be humble and open-minded enough to admit that there are things we do not understand. Then we need to have the skills to spot mistakes or opportunities for improvement. We also need to have the skills to think systematically, drill down to root causes, and come up with solutions. And finally, we need to have the courage to try out the solutions and the perseverance to see changes through.

Managers have a lot to bear in this learning process. Because Agile facilitates fast feedback, it will expose more issues more quickly, which places more demand on managerial skills. Don’t ever think that management can be delegated to Scrum masters. In real corporations, “self-organizing” teams are hard to achieve, and it is impractical to expect a “servant” Scrum master to navigate through corporate bureaucracy to push through changes. To predict if an Agile transformation will be successful one just needs to watch how a manager reacts to issues. If his reaction is to have more meetings and more controls, the possibility of successful Agile transformation will be low.

In terms of leadership, I like Toyota’s definition (Liker):

1. Active and open-minded observing of the work of the organization

2. Active listening to hear what people are really saying

3. Systems thinking

4. Clearly defining problems and identifying the root cause

5. Creatively identifying countermeasures to the true root causes

6. Planning

7. Translating plans into action with clear accountability

8. Taking the time and energy for deep reflection to identify further opportunities for improvement

9. Understanding the actual strengths and weaknesses of each person

10. Motivating and influencing people across the organization (with no direct authority) toward common objectives

11. Being able to teach others all of the above

These leadership skills are oriented towards solving issues. Items 1-7 are all about necessary skills and the right approach to solving problems; item 8 is about preventing problems from happening again and making further improvements; items 9, 10, and 11 are about people management skills, with the ultimate goal of teaching people the right approach to solving problems.

Scrum traps

On the surface, Scrum seems to match with the Agile principles perfectly. Time-boxed delivery means we get users’ feedback quickly; user stories created from users’ point of view mean we set priority based on business value; user story reprioritization in each iteration means we adapt to user feedback; retrospective meetings mean we learn and improve continuously.

Scrum is usually the first and sometimes the only Agile process some teams adopt, because unlike Agile principles, it is a tangible process that can be implemented in a short time. In its crudest form, it just adds a few more meetings and splits work into multiple deliveries. Teams who adopt the crudest form will find themselves complaining about long and tedious meetings. Instead of a “sustainable pace”, teams find themselves having to “sprint” in each iteration to get something ready for review.

It is not Scrum’s fault per se. However, since many Agile consultants sell Scrum as the de facto Agile process, resulting in teams adopting the crudest form, we have to face the fact that Scrum may have done more harm than good, and understand why it is so. If you argue that it is not the true Scrum (just like when Agile fails, it is not the true Agile), just replace Scrum with Scrum preached by bad Agile consultants.

Scrum’s success is based on self-organizing teams, I would argue “self-organizing teams” is an outcome, not a premise. Some Scrum practices (such as user stories, velocity measurement, and meetings), are tricky in that they can easily be abused and lead to short-sighted technical decisions and micromanagement.

Self-organizing team

Scrum groups people into cross-functional teams. If a team has all the expertise it needs, according to motivation theory (see Drive: The Surprising Truth About What Motivates Us), it should be happy to be self-organizing, right? Well, in reality, it doesn’t often play out according to the text book.

If a team is used to receiving commands from management and delivering software according to specifications, it could be overwhelmed by the unexpected freedom. As Spiderman says, “With great power comes great responsibility”, people in a dysfunctional team do not want to make decisions on their own, because they do not want to take responsibility. If such a team practices Scrum, a power structure emerges, with the PO serving as the functional leader to issue commands, and the Scrum Master assigning tasks and enforcing commitments.

Management adopting Scrum shouldn’t take self-organization for granted. It takes a lot of trust-building to motivate people to be engaged and take the initiative. It is the outcome of good Agile implementation, not a premise.

This is assuming that management really wants to give power to the team. In reality, this is sometimes just lip service. Management states teams have the power to work at a sustainable pace, while in reality the unscrupulous manager maneuvers Agile’s claim of embracing changes to cram more work into the backlog. As a result teams have to “sprint” in every iteration to get things done. No wonder that many teams want to go back to “waterfall” days, when they only had to “sprint” in the last stage.

User story

The purpose of user stories is to facilitate the conversation between users and delivery teams. It is difficult to elicit requirements by asking users “what features do you want?” Using the narrative statement “tell me how you perform this part of your job from the beginning to the end” is an easier way to discover features and opportunities for improvement. The purpose of user stories is not to fill up the backlog. Otherwise, we may fall into these traps:

1. Everything has to be justified from the users’ point of view. Admit it: software users usually do not have the technical skills to understand how software is made. They can tell you whether a feature looks good or not, but they can’t judge the value of technical details underlying the feature. If you tell them this story takes 2 points, but that story takes 13 points, because you need to refactor the code to make it more modular, more testable; they won’t understand.

Some teams resort to “technical stories” to describe the work that aims at improving code quality and reducing technical debt. The fact that we have to create “technical stories” indicates that there is (still) a big gap between business and IT. Having “technical stories” and “user stories” side by side creates an illusion that “technical stories” can be sacrificed or be worked on later as “user stories”.

As long as business favors short-term values over long-term values, the gap won’t be easily bridged.

2. In Scrum, user stories are groomed at the beginning of an iteration. This does not mean that we should just focus on the next batch of user stories and be ignorant of all others. Failing to detect important needs beyond the current iteration will cause big disruptions down the road.

Meetings

Ah, yes, meetings. It is probably the most complained about aspect of Scrum. Daily standup meetings, backlog grooming meetings, planning meetings, review meetings, retrospective meetings – will they ever end?

In my experience, in a small team where members are on good terms, people usually know who to turn to for help and what others are doing. They have plenty of opportunities to know each other. There is no need to have a formal daily standup. Even though a daily standup meeting only takes 15 minutes, the interruption it causes is more than 15 minutes. Engineers are most efficient when they immerse themselves in a “flow” so that they can devote all their minds and hearts to solving issues. A scheduled morning standup is like a scheduled interruption, making it hard for engineers to get into the flow.

It is easy to tell if a meeting is held successfully. Just observe whether people are engaged and whether people are not afraid of voicing their opinions. If people are playing with their phones or have glazed eyes, think of other ways to engage them, instead of continuing to torture them by meetings.

Velocity

A consistent velocity is the Holy Grail for project managers, but unfortunately, it is not always achievable. If a team has done a similar project before and has advanced technical skills, the velocity diagram might be consistent. But in reality, a velocity chart might look like the following:

The team starts to discover user stories as it goes along, so the burn down line (the blue one) at first looks like a burn up line! I guess there should be a point where the burn down line should start burning down, but the location of the point would be based on the nature of the project.

The trouble with velocity charts is that it can be so easily abused. Project managers will push teams to generate such charts so they can make “reliable” predictions. But you know software engineers are smart enough to game any metrics when push comes to shove (see The Original Sin of Software Metrics).

Micromanagement

“What have you done yesterday?”

“What do you plan to do today?”

“Why haven’t you done what you said you were going to do yesterday?”

“Why is this story 13 points? Someone has done something like this with 5 points?”

You see, with daily standup, story point size, and a velocity chart, it is so easy to turn the whole Scrum process into micromanagement. It is worse than it was in Waterfall days, because now you have to justify your work every day! Morale and innovation are killed in this day-after-day grinding process.

This kind of micromanagement hurts senior engineers most. Senior engineers know from experience how to approach a technical issue, how to break things down into chunks of work, how to plan their work based on those chunks, and at what moment they should stop writing feature code and start refactoring. I would like them to share their thinking process so that junior engineers can learn from them, but I would not, for the purpose of a neat and pretty burn down chart, ask them to break small things down into even smaller tasks or to make up “technical stories” to be approved before they start working on them (unless of course the “technical stories” are of high impact; but if so, senior as they are, they must know when to bring them up and involve others in the discussion).

Invest on skills, not on processes

Strapping a process, a strict one like Scrum, on the same people having the same mindset and skillset, and expecting a miraculous result (capable of embracing changes) is not realistic. There are two layers of Agile, each calling for different mindset and skills.

1. Product level. One of the common misconceptions I heard is “we have a fixed scope, so we can’t be agile”. To this, my question is “Is this fixed scope guaranteed to deliver the value that it promises, or does it contain assumptions?” If a product manager clings to his/her list of features, it is more likely that he/she doesn’t know which feature will strike gold, and so he/she wants to deliver them all. Even if you have a feature list directly from end users, chances are that they will change their minds upon seeing or using these features. In Agile, product development calls for an experimental mindset, treating features as assumptions to be validated. It will be very discomforting for someone who is accustomed to a feeling of certainty (even if it is just an illusion) to transition to this sort of trial-and-error style. Besides the mindset change, the product team and the delivery team also need to practice and master the skills required to validate assumptions quickly. Just like a scientific experiment, they need to first have a hopeful hypothesis, design an experiment, and gather data to prove or disprove the hypothesis. This structured approach of “trial-and-error” will ease the fear of chaos.
2. Delivery level. Since Scrum is so easily misused despite its good intensions, if one is brave enough to try Scrum, I suggest trying CICD (continuous integration and continuous delivery) at the same time. If your business model doesn’t allow you to continuously deliver your product to end users, continuously delivering the product to a demo environment or staging environment is still useful. CICD counteracts Scrum traps in the following ways:
   1. It balances the “user centric” bias. User stories encourage users’ participation and make sure software is focused on maximizing that elusive 20% of features, but it doesn’t mean users know how to or even have the right to direct development work. To make CICD successful, code has to be maintained healthy. Accumulated technical debt will throw a wrench into the CICD chain. If technical debt is ignored and piled on, the CICD chain will quickly break.
   2. CICD really challenges teams’ capabilities. It requires advanced technical skills to oil the chain and make it run smoothly. It requires the product to be well-designed and written for testability. What emerges from the CICD chain is working software, which is what really should be measured, not story points or velocity charts.

One good thing about CICD is that it contains a lot of information which is naturally captured from teams’ actions, for example:

1. From the source control system, you can perform code static analysis and understand if technical debt is accumulating over time; you can also get information as to how actively team members are participating in code reviews. For example, if you are using Git, you can analyze how quickly pull requests are merged and how many comments are posted. This can give you a clue as to whether teams are following good practices and working together closely.

2. From the build system, you can see how many builds fail each day and how quickly a failed build is fixed. This will give you a clue as to code quality and teams’ discipline.

3. From the defect system, you can analyze how many defects are captured by automation and how many by manual tests. You can further categorize defects into groups such as usability defects, security defects, performance defects, and functional defects. Each group might point to areas that teams need to strengthen.

One good way to hone your systems thinking skills is to gather information from multiple areas and analyze their relationship, instead of just focusing on one set of information (such as velocity) and optimizing one single part.

A common problem that affects CICD is lack of automation or lack of stable automation. A manager without systems-thinking skills will simply push for teams to write more automation tests. What is wrong with this approach? It violates the 11 laws of systems-thinking (Senge):

1. Today's problems come from yesterday's solutions.

2. The harder you push, the harder the system pushes back.

3. Behavior grows better before it grows worse.

4. The easy way out leads back in.

5. The cure can be worse than the disease.

6. Faster is slower.

7. Cause and effect are not always closely related in time and space.

8. Small changes can produce big results – but the areas of highest leverage are often the least obvious.

9. You can have your cake and eat it too – but not all at once.

10. Dividing an elephant in half does not produce two small elephants.

11. There is no blame.

Every software manager will probably have witnessed several corpses on the journey to accumulated technical debt; they should in theory understand the damage resulting from technical debt. So why is it so hard for them to resist the temptation of a symptomatic fix – in this example, pushing teams to write more automation? Part of the reason lies in the above laws, in particular, law 3 (behavior grows better before it grows worse) and law 7 (cause and effect are not always closely related in time and space). Having more automation tests, even if they are badly written, can be useful in the short term, until the burden of maintenance outweighs the benefits. Hopefully with CICD, the feedback delay will be shortened and the accumulating technical debt will be revealed quickly.

Part of the reason is in law 4 (the easy way out leads back in). The more experienced a manager is, the more entrenched his mental mode might be. "Mental models" are deeply ingrained assumptions, generalizations, or even pictures or images that influence how we understand the world and how we take action. Very often, we are not consciously aware of our mental models or the effects they have on our behavior (Senge). In this example, what is the underlying mental model for the manager? 

As pointed out by Peter Senge, the manager probably won’t even be aware of his mental model, he certainly won’t admit that he thinks testers are lazy or stupid or that it is the testers’ job to perform “quality assurance”. Scrum’s practice of having a cross-functional team is useful, because it brings together people with widely different perspectives. It is the manager’s job to build a trusting environment, so that different mental models can surface and get challenged, including, probably most importantly, his own mental model.

While discovering and probing into deeply-held mental models, the team might ask:

1. Is the system written for testability? A system written without testability in mind is hard to perform unit and API tests on, and testing is therefore done more on the UI level, which is fragile.

2. Should developers have a testing mindset? Most developers know how to use XUnit, but knowing how to write testMyMethod is not enough. They need to think like a tester: where might this method fail, what if this parameter is null, and what will happen if this method fails.

3. Should testers have good programing skills? Technical debt is a well-known concept these days, but test technical debt doesn’t get enough attention. Automation test is code, and requires good programing skills as well. The automation framework needs to be written in a way that testers with less programming skills can easily write automation tests. Like production code, automation tests need to be stable and perform well. And since automation test cases serve as documentation for the system, they must be readable. Badly-written automation test cases are not assets but debts.

4. Do developers and testers know how to work with each other? While discussing user stories, are testers pointing out vulnerable areas for developers? Are developers considering testability? Or are developers handling user stories over to testers, and testers subsequently checking whether developers have done what they said they did?

5. Is testing just an activity for verifying software implementation against specifications? Or can testing be used to flesh out specifications and drive system design? This is probably a mental model leap for both developers and testers, a new skill for both to master.

Agile is hard, it will expose more issues more quickly, it is a challenge for both the team and management. I’ve witnessed teams that have been paralyzed by issues cropping up constantly, they tried to use the same solutions or mental models to solve them, failed, and felt helpless. As Albert Einstein said, “We cannot solve our problems with the same thinking we used when we created them.” It is a learning journey; teams that look at issues holistically, understand the short-term and long-term consequences, pick up new skills, and update their mental models will be successful.

References

• Liker, J. (n.d.). The Toyota Way: 14 Management Principles from the World's Greatest Manufacturer.
• Senge, P. (n.d.). The Fifth Discipline: The Art & Practice of The Learning Organization.

About the Author

Chen Ping lives in Shanghai, China and graduated with a Masters Degree in Computer Science in 2005. Since then she has worked for Lucent and Morgan Stanley. Currently she is working for HP as a Development manager. Outside of work, she likes to study Chinese medicine. She blogs here.