Soulful Socio-Technical Architecture

When I read the data about employee engagement worldwide in the State of the Global Workplace 2021 report by Gallup, I was shocked to find out that, on average, the engagement level of people is about 20%. That means only 2 people out of 10 are actively satisfied and thriving in their job. Only the USA and Canada surpass 30%. Most other countries are way behind, with Western Europe scoring an ashaming 11%. This is a general statistic, regarding any type of job, but it properly reflects the unhappiness I found in developers inside many organizations, especially the largest ones. I also observed how the most expert of them seem to be those suffering the most.

I wanted to understand more, so at Alcor Academy, we started our research as well.

We have been interviewing several IT industry leaders and managers asking what are, in their opinion, the most relevant issues they face in their environment.

We found out what they perceive to be their top problems:

• The development teams are not aligned with business value
• Teams are entangled with dependencies driving low productivity
• An unclear, top-down decision making jeopardize trust in the management team
• Attracting and retaining expert talents given the high turnover
• Most of the people remaining are disengaged and disconnected

I agree with Patrick Lencioni when he says that "a healthy organization has fewer politics and confusion, higher morale and productivity, lower unwanted turnover, and lower recruiting costs than an unhealthy one".

From this point of view, both the Gallup stats and our findings are showing all the signs of an unhealthy industry. So, what can we do to change this disastrous trend? How can the industry heal?

I believe a collective mindset shift has to occur.

It’s the collective mindset of an organisation that determines an organisation’s overall effectiveness, productivity and degree of success. - Bob Marshall

That's the reason why at Alcor Academy we focus on soulful socio-technical architectures that help promote healthy organizational cultures, where everybody wins: stakeholders, with higher productivity and reduced cost; customers, delighted by great products; and developers engaged in their work.

Socio-Technical

The term "socio-technical" was introduced after a study by Eric Trist & Ken Bamforth published in 1951. Their article analyzed the detrimental consequences and broken promises of the British nationalization of the coal mining industry that happened in the previous decades. The research was titled "Some Social and Psychological Consequences of the Long-Wall  Method of Coal Getting: An Examination of the Psychological Situation and Defences of a Work Group in Relation to the Social Structure and Technological Content of the Work System".

The "Long-wall method of coal getting" was a new way to extract coal, based on either the introduction of new technology (like underground railroads and mechanical underground diggers) and a social review of tasks and hierarchies involved in the process.

In the previous "Hand-got" method people worked in little, self-organized groups to mine coal (2-20 people). Their tasks were multiple, and the choice of teammates was critical for high productivity. Salary and bonuses were negotiated directly by each group, which had internal coordination and supervision on the final execution of the job. They were capable of responsible autonomy and able to vary the work pace in correspondence with changing conditions. This social structure proved to be ideal for adapting to the risky and uncertain underground situation.

The introduction of the new mechanized "Long-wall" also brought an entirely different social structure. The new method split the activities over three shifts instead of only one, and each shift did different things. People were spread out more spatially and temporally. Furthermore, they were segregated and paid by skill. So the social groups were defined by occupational roles, rather than cohesive personal relationships.

Even the three shifts were segregated, with no handover. Mistakes in one shift carried over to the next. Finally, also the role of management changed and became directly involved in the coordination and control of workers. As a result, workers felt they were driven by management, establishing a norm of low productivity.

The reform was meant to get the most out of the new technology, but the outcome was far from expectations. There were continuous problems with management. Absenteeism and turnover were at an all-time high, while strikes became the norm.

Faced with low productivity despite improved equipment, and with drifts from the pits despite both higher wages and better amenities, those in authority have increased their interest in organizational innovations. [...] A qualitative change will have to be effected in the general character of the method so that a social, as well as a technological whole, can come into existence.

If attention is paid to the social as well as the technical system, then a choice can be made that leads to optimal performance of the system as a whole, even if that requires a less than an optimum state for the separate dimensions.

What Trist claims, is a direct consequence of the Theory of Constraints. In a complex environment, which is a system of systems, increasing the performances of only one internal system in isolation has not the effect of improving the whole. It often has the opposite effect, as shown by the dreadful mining reform.

In the Cambridge Dictionary, we find two interesting definitions for Architecture, if we drop those referring to the most common one related to the design of buildings.

1. "The design and structure of a computer system, which controls what equipment can be connected to it and what software can operate on it."
2. "The structure of an organization, which influences the relationship between its parts and employees.”

If we think of Conway's law, however, we can see how the two definitions are related to each other: the structure of the organization constraints the design of its digital systems. This clearly proves why the most common anti-pattern I've found in organizations is the "Monolithic, Big Ball of Mud". Most of them are fixed, highly centralized hierarchies with poor communication structures entangled with entropy, after all. What would you expect?

 Russell Ackoff wrote that "your understanding increases the larger the system you comprehend. Knowledge increases the smaller element you comprehend." So, let's journey from knowledge to understanding. Let’s start from the smallest element of our architecture and go inside-out this time (for a change).

The smallest element in our architecture is our code. At its best, it looks similar to the picture.

There are several variations and names over this idea, developed and expanded by several people. So the literature is vast and it doesn't need more. (onion, hexagonal, port and adapters, clean...). The consensus here is that good architecture is modular and loosely coupled.

But our code neither runs by itself, nor it is developed for its own sake. Our code is always meant to support another activity and has to interact with other people and/or other activities. These interactions between activities and people, driven by the very basic flow of value, are not at all valued in hierarchical environments, because they cross departments orthogonally to the organizational chart.

Yet, "the key in making great and growable systems is much more to design how its modules communicate rather than what their internal properties and behaviors should be." (Alan Kay)

That's where, in my opinion, Eric Evan's idea of Bounded Contexts and Context Mapping comes into play. The modular architecture has to live inside its bounded context of Communication. And this context represents a modular sub-system of the organization. Autonomous in how to perform its activity and how to implement things internally, but not in the way it interacts with other subsystems. That's the critical aspect that needs to be well designed and defined, but in my experience is often overlooked.

To manage a system effectively, you might focus on the interactions of the parts, rather than their behavior taken separately.

So how do Bounded Contexts interact? Technically, the most relevant decision is between synchronous or asynchronous communication (this topic is widely covered in the literature as well). But before thinking about that, there must be an architecture of bounded contexts in place. How do we split them? In my experience that seems to be the point where organizations struggle the most.

The key here is to consider the organization's value stream and analyze how the value is added to the end product or service. We can simply think of it as a temporal sequence of activities necessary to design, produce and deliver the value proposition. And, ideally, most of these activities would benefit from the support of an information system.

If this holistic view of the organization’s activities is missing, what chance is there of creating sub-systems that communicate in harmony with each other, smoothly mapping the flow of value with the flow of information?

If we look at software development from a value stream perspective, rather than simply looking at these activities as a process, we can emphasize the creation of value. As we have seen before, it’s all about improving the whole process, not just the parts.

The activities of the flow are independent but interconnected; this is one of the properties of a Complex System. Each activity performs different tasks in autonomy but feeds information back to other parts of the System. They are needed for the chain of value to progress to the next stage.

For example, in an e-commerce company selling physical items online, the value stream activities include:

• Acquiring (physically or remotely) the items to be sold
• Describing their physical details (color, qualities, pictures, and other information useful for their placement on the market)
• Assessing and interacting to handle the quantity in stock
• Creating and managing the mechanism for customers to browse the catalog, place an order and pay, dispatch the items

In parallel, there’s a marketing system to place advertisements, and after the sale is done, there will be some customer care functionality. This and all of the above are activities of the value stream.

They can happen in parallel, and the stream might be somewhat complicated, but they can be carried on independently of one another. However, they have dependencies; for example, the system in charge of letting customers place an order obviously needs to know the stock of the available items – they are interconnected.

To keep the overall entropy of the information system under control, the business must correctly separate and group these activities into their contexts. The crucial point is finding the perfect balance between independence and interconnections. (Incredibly similar to balancing coupling and cohesion in Object-Oriented programming, after all.)

But our contexts aren't just for digital systems. In my view, the activity itself is part of the context, as well as the business people performing it. This is the point where the communication between technical people and business folks should be frequent, clear, and open. And that's where the siloed departments divided by skill fail miserably. Exactly like in the coal mining reform!

To understand more about principles and strategies for evolving the teams in that direction, I suggest reading “Team Topologies” by Matthew Skelton and Manuel Pais. In their great work, they explain that there are many different ways teams can interact (collaboration, x-as-a-service, facilitation), with several different purposes (stream-aligned, enabling, complicated-subdomain, platform). Their size matters as well, because there is a limit in the cognitive load of individuals. Furthermore, the bigger the team, the weaker the relationships among its members.

These considerations are the most important when we do architecture design, in my opinion. Yet in most environments, there is not a defined strategy for how teams collaborate and what is their purpose in relation to the flow of value. This means that the design of these key aspects is often accidental.

Instead, imagine a network of small, autonomous, self-organizing, multi-skilled teams, united by purpose and values: how far could you go?

Unfortunately, the reality is often very different in too many environments. A rigid hierarchy of control is in charge to perpetuate itself, united only by command-and-control. A Ponzi scheme of power gets in the way of any innovative idea and systemic change.

When everyone feels just like a cog in the mechanism, it is not a surprise to see low levels of engagement. That's what happens when people feel unheard, unsupported, and have no influence in their environment for creating a better future. And the higher the knowledge of the professionals, the deeper the feeling that their knowledge is being scorned. I sadly observed how the morale of most talented developers in these environments is often gloomy.

A healthy organizational culture is deeply important: when there is no joy, there is also very little creativity. If you leave the house and realize that your car has been stolen, will you be in the mood of creating something amazing?

Joy, feeling one's own value, being appreciated and loved by others, feeling useful and capable of production are all factors of enormous value for the human soul. - Maria Montessori

That's why I believe good architecture also has to consider the soul of the individuals: we need Soulful Socio-technical Architectures!

Soulful Socio-Technical Architecture