Bridging the Management Gap

As Agile grows up and its principles become widely accepted within IT organizations, one roadblock to more significant organizational change is becoming clear - resistance from management. This came as no surprise for anyone who had been paying attention though. Long before the Agile Manifesto, other fields such as Complex Adaptive Systems, Lean and Systems Thinking had already made it clear that our traditional command & control, manufacturing mentality would no longer suffice in a globalized, knowledge-based economy. Yet the interesting question has remained the same - when will we reach the tipping point where most organizations unshackle themselves from the many limitations of command & control?

I’ve experienced this resistance first hand while working with large organizations in implementing Agile transitions. Short-term gains such as improved communication, increased transparency and better teamwork came by fairly easily when companies began adopting Agile practices, but true organizational change was not achieved. The same companies refused to review their management practices and the bigger picture of how they deliver value to their customers. Their focus was on adopting Agile almost exclusively within the IT organization, and even there, without any serious re-thinking of how they manage their people or projects. I call this pattern the "management gap" and it is one frequently experienced by Agile change agents.

(note: I use the phrase "Agile change agents" to mean anybody responsible for driving changes and improvements in organizations and making them more agile. This includes Agile coaches and consultants, internal change champions and managers responsible for improving their organization.)

Troubled by this predicament, I wanted to understand what could be done about it. So I began to focus on two questions:

1. Why is there a "management gap"?
2. How can we bridge this gap?

If we want to understand the root cause of the management gap (why), we have to go back before the Industrial Revolution, when the issue of central control was becoming a hot topic in socio-political circles.

The Age of Reason and Rise of Central Control

As Europe transitioned from the Middle Ages (and Feudalism) to the Age of Reason (and Central Government), the topic of how to effectively manage nations and societies was hotly debated among leading philosophers of the time.

Famously, David Hume and Thomas Hobbes disagreed on central control. Hobbes believed it was a requirement for any civilized society and that its absence would undoubtedly lead to "solitary, poor, nasty chaos" ("Leviathan", 1660). Hume, on the other hand, introduced the concept of "spontaneous order", claiming that the lack of central control would actually facilitate the development of an organized society. It would create a silent, emergent agreement on the necessary social conventions, "where every act is performed in expectation that others will perform the like" ("A Treatise on Human Nature", 1739). It is mildly amusing that today, almost 300 years later, you can hear the exact same discussion happening in many organizations between teams and their managers.

Following on Hume’s footsteps, Adam Smith (who was a close friend of Hume) took the concept of the "spontaneous order" and applied it to economics. The result was his famous book "The Wealth of Nations" (1776), which became the philosophical foundation for the free market economy. Critically, Smith claimed the collective actions of individuals combine to create an invisible hand, which will always be more capable than any form of central control to carry a society to prosperity.

The Manufacturing Dichotomy - Free Markets vs. Dictatorial Organizations

The industrial revolution drove economic growth in the 19^th century as Europe and the US embraced Enlightenment concepts such as democracy and free markets. Yet while the emerging corporations competed against one another in the market as free, independent agents, they were internally organized like feudal societies. Indeed it was the very sentiment of people as productive resources (instead of knowledge workers) that made the rise of the next protagonist in this story inevitable. Enter Taylor.

In the late 1800's, Frederick Winslow Taylor began developing what he would later call "scientific management", which focused on the separation of planning and doing. The steps involved in the production process were diligently analyzed, then synthesized and standardized so that workers could be easily trained to perform their planned tasks. One of the main benefits organizational leaders saw in this approach was that it facilitated their growth due to the plentiful supply of manual workers. This ushered in the era of command & control management, where thinking was the job of a select few at the top of the hierarchy.

At this point it's probably pertinent to ask why did organizations almost unanimously favor Taylor's "scientific" approach to management? If today we're saying that they need to re-think their organizational structure to leverage the full potential of their people, which is similar to what Hume, Smith and many others were saying 250 years ago, why did corporations get away with the simplistic command & control structure that is still prevalent today?

The answer is simple - at that time, it generated results. Impressive results. By standardizing labor tasks companies were able to quickly expand their operations and increase their profits. The management techniques introduced by Taylor were proven time and again during this period.

From an organizational structure perspective, the organizational paradigm resulting from Taylor’s work rested on two success factors:

1. Size (to benefit from economies of scale)
2. Standardization (to enable growth without losing control)

(Click on the image to enlarge it)

Figure 1

Enter the Knowledge Economy, Exit Command & Control 

While scientific management and the manufacturing paradigm became accepted truths, the second-half of the 20th century saw important advances in technology and social thinking that would fundamentally change the nature of companies. Competitive advantage was the name of the new game as companies began to look externally towards the market. Process-focused scientific management was no longer enough to make a company successful, you needed to be innovative to stay ahead of your competitors.

While organizational leaders recognized the shift from a manufacturing economy to a knowledge economy, they failed to see that they also needed to re-think their internal structure to be successful. As a result, they simply accepted the same command & control management model without second thought. And this, in essence, is the root cause of the current friction between Agile change initiatives and organizational leaders - we're operating in a knowledge economy while using command & control management principles from the industrial revolution.

As we have seen, this friction is not recent, but rather a debate that started during the Enlightenment. Indeed, thought leaders of the knowledge economy have been proposing alternatives to the command & control paradigm since the 1960’s. In the software industry, these alternatives were being put forth by people such as DeMarco and Lister, Fred Brooks and Jerry Weinberg. Later these thoughts were gathered under the common umbrella of Agile. In the manufacturing industry, Edwards Deming and Taiichi Ohno were doing the same in what is now known as Lean Manufacturing. In the academic field, people such as Henry Minzberg where bringing into question the actual role of managers and strategies. These emerging alternatives were based on empiricism and encompassed thoughts from a variety of fields including:

• Complex Adaptive Systems
• Sociology
• Psychology
• Cognitive science
• Anthropology

The interesting question emerging from all this is - what would an Agile organization look like?

I propose that in the knowledge economy, the new paradigm for effective organization design is based on two criteria:

1. Non-hierachical (emergent leadership)
2. Adhocratic (emergent order)

(Click on the image to enlarge it)

Figure 2

The vertical axis (hierarchy) represents the "command" part of command & control. Towards the top-half of the graph, leadership is flexible, emerging based on the challenge at hand. In the bottom-half of the graph, leadership tends to be rigid and dictated by title and hierarchy.

The horizontal axis (bureaucracy) represents the "control" part of command & control. Towards the right of the graph, companies are more adhocratic. Their structure is more fluid, with rules and processes changing in emergent fashion based on the challenges at hand. The left of the graph is more rigid and bureaucratic, with inflexible rules and very detailed processes.

A quick study of this new organizational paradigm highlights three important observations:

1. The very structure that made a company successful in the manufacturing paradigm is what makes a company slow and unresponsive in the knowledge economy - if your company is in the bottom-left of the graph (red), this change is going to hurt.
2. Moving into the more desirable areas of the graph (green) must be a gradual journey. If leadership and order are to be truly emergent and adapting to the needs of the company, they cannot, by definition, be fully planned upfront - there must always be room for discovery during the change initiative.
3. Being on the green areas of the graph though necessary for success in today's economy, is not easy. Self-organization is powerful, yet combustible in its infancy. And even then, success is never guaranteed. You still need to sell something people want to buy. This is just the organizational structure that will give a company the best chance to succeed in today's market.

Bringing the point home about the potential of the knowledge paradigm, we are confronted with an increasing number of companies in the forefront of their industries who are finding creative ways to breakdown hierarchies and bureaucracies. Companies such as Valve, Google, Dropbox, Spotify, Morning Star, Semco and Gore & Associates have shown some of the vast untapped potential of being lean, agile and anti-fragile as they are out-performing their command & control competitors. Indeed, management publications such as the Harvard Business Review and Forbes have been dedicating a lot of their pages recently to talk about how large organizations from any industry can benefit from the "start-up mentality" with concepts such as teaming, agility and collaboration.

And while many argue we are reaching a tipping point for management, we are not there yet. When we understand the history of how we got here, it also helps us to understand the size of the challenge at hand - we need to move beyond a paradigm that, though successful for over 150 years, has now reached its natural limit and must be replaced.

So, how can we go about doing that?

Bridging the Management Gap

Within the context in which I have formulated this issue, successfully bridging the gap means agreeing with organizational leaders that management has fundamentally changed in the knowledge economy. This change has created a new paradigm in organizational structure, rendering command & control a thing of the past, much like feudalism. Bridging the gap, thus, means management and change agents agree under which paradigm we are operating.

(Click on the image to enlarge it)

Figure 3

Now, there are many things that can be done to help bridge the gap, such as:

• realizing that cookbook-style change recipes (yes, even Scrum) cannot, by themselves, make a company Agile
• thinking beyond software and looking towards other disciplines for learnings and ideas
• introducing agile management into education, so that future leaders will no longer be taught that command & control is an effective management alternative

While actions such as these are critical, they require a separate article for adequate discussion. However, we can already touch on two basic actions change agents can take to help bridge the management gap:

1. agree with leaders on the context of the change initiative
2. share their results

Understanding the Change Initiative’s Context

The knowledge paradigm in Figure 2 can be used to define the scope of the change initiative, but to do that we must first listen to the organization’s leadership. What are their biggest challenges? What kind of company do they want to be? What kind of company are they right now? While the answer to these questions might not be fully clear at the start, discussing them upfront is a valuable exercise for three reasons:

1. secures support from organizational leaders
2. defines objectives and approach for the change initiative
3. helps identify internal change champions

Engaging leaders and asking them to identify their main challenges is a key first step, keeping them from hiding behind the generic objectives of a specific cookbook. Rather, it pushes them to understand the history of the manufacturing paradigm and the principles behind the knowledge paradigm, so that they can then translate those principles into more specific goals based on their own reality and competitive environment.

A further benefit of defining a change approach within the framework proposed above is that it becomes possible to identify priorities and attention points for the change initiative. Organizational change is an incremental, gradual process and companies can choose very distinctive change approaches.

(Click on the image to enlarge it)

Figure 4

Best Practices: this section could also be renamed “pixie dust”, since it involves adopting some of the best practices found in the cookbook without aiming at a more significant change. Though not explicitly stated upfront (who wants to promote a change initiative that doesn’t actually promote any significant change?), implicitly it is actually the approach most adopted nowadays, since companies are still not convinced on the benefits of the knowledge paradigm. Agile change agents should highlight to organizational leaders that they are not significantly altering their internal structure and processes, so they should not expect the result to be an agile company ready for the challenges of the knowledge economy. In the lower-left area (red), you’re dealing mainly with local optimums.

Benevolent Dictatorship: this change approach makes sense when organizational leaders believe their main challenges are related to their ability to capitalize on existing internal knowledge. Organizational silos can create significant barriers to internal collaboration, with departments more concerned about their own limited objectives. This approach would focus on loosening up procedural rules and establishing common objectives to promote cross-organizational collaboration. It also is clear that special attention must be paid to training managers so they understand the role of an Agile manager in this environment.

Structured Start-up: this approach makes sense if, for example, organizational leaders identify responsiveness to market opportunities as one of their main challenges. By making their organization less hierarchical and empowering their employees to make appropriate decisions, they can expect to improve their responsiveness and benefit from the knowledge of the people "in the trenches". It usually will also mandate a re-thinking of the project management and budgeting offices since you want to have smaller projects (budgets) being released to further enable faster reactions and exploration of new opportunities while limiting your risk (Lean Startup). Clearly, special attention must be paid to creating cross-functional teams with clear objectives.

(note: there is a common theme between the “Benevolent Dictatorship” and “Structure Start-up” approaches - cross-functional teams with common objectives. This is natural as it’s one of the key characteristics of innovative, agile companies.)

Whichever approach is chosen, change is a gradual process so it's important to make sure organizational leaders are able to contextualize it within a bigger picture and understand the scope of the challenge. Their responsibility is to re-shape their organization to thrive in the 21st century and the knowledge paradigm offers them a foundation to start that journey.

Sharing Results, Experiences and Learnings

It is still unclear what (and when) will be the tipping point for the transition from command & control to the new knowledge paradigm, but what certainly will help is a growing movement driven by a sharing of results and experiences. This sharing needs to happen on two levels:

• Internally, as companies proceed in their change initiatives
• Externally, between companies and change agents across different industries

Sharing internally allows people to hear about how other teams in the organization are handling the challenges and opportunities brought about by the change initiative. This enables key learnings to be shared and also new ideas and collaboration opportunities to emerge.

By sharing externally, companies and change agents can better understand the real principles behind the knowledge paradigm, since they can see significantly different change initiatives and organizational cultures. This bazaar of ideas is beneficial to all, and much like the open-source movement, will ensure the results are greater than the sum of its parts.

Another benefit of sharing ideas and experiences of such change initiatives is that inherent to sharing is the humility to know there is no single right answer. Once we stop trying to convince ourselves that pixie dust is all that is needed, we're forced to accept that the answers are much deeper and varied. It is the challenge of every organizational leader to not just follow a prescribed recipe, but to understand the principles behind what it takes to be at the forefront of the knowledge economy. This will require re-thinking their existing organizational structure and coming up with their own, unique implementation of those principles. Making the change their own is the only way for it to stick.

In this context, the role of the change agent is to help these organizational leaders start to define the way forward. The sharing of experiences (successes and failures) will ease this process by allowing people to learn from the experiments of others and find their own way. The Beyond Budgeting Round Table, BetaCodex, and Stoos are examples of groups or sharing platforms that are serving precisely this purpose.

This topic has been gaining a lot of traction recently and hopefully these initiatives will continue to grow and branch out so we can start adding more momentum towards effectively bridging the management gap. In the meantime, understanding the history of the manufacturing paradigm, as well as the principles and benefits behind the knowledge paradigm will better equip change agents to handle their current improvement initiatives, while leading the way to that tipping point.

About the Author

Tiago Garcez is an Agile coach and partner at Agilar.  He has worked in multiple large-scale Agile transitions, supporting change leaders and teams in their continuous improvement efforts.  Previously, Tiago worked as a Product Owner, Agile Project Manager and Business Analyst.  An active member of the Belgian Agile community, you're likely to bump into him in the Agile events in the area.