Thriving in the Complexity of Software Development Using Open Sociotechnical Systems Design

Although the ICT industry is trying to adjust and adapt as best as it can to the increasingly turbulent social and technical environment it operates in, where workers are demanding more and the fourth industrial revolution is in the making, the success rates are dismally low. Many try to faithfully copy winning formulas shared by the few companies that seems to have cracked the code, while others double down protecting their existing assets by doing more of what used to work in the past, hoping that they can ride the storm.

The main challenge may very well be that many are the victims of their own success, that the amazing progress made in technology has led to blindly following the technical imperative at the cost of the social and human dimensions. Following the proud traditions of physical engineering may seem like a sensible choice, with its rigid and battle tested processes and expert lead organisations, but the technology domain is not as repetitive and predictable, as there are design decisions being made at any time and every level, from strategies laid in the board rooms to the coder pushing commits to Git. The complexity is too high and the speed of change is so fast that other ways of working are needed, preferable where agility, resilience, and active adaptations to the uncertain environment naturally emerge. Ironically, the solution may be found in the midst of exactly that industrial tradition that we are an extension of, based on scientific research in the post-war mechanical industries.

The ICT industry is the driving force of the digital revolution, with its game changing technical innovations that are starting to dominate the world economy and enhance the lives of people across the globe. We got tech down! No doubt. What we are not so good at is the social dimension of what all software companies are, namely sociotechnical systems. Companies frequently proudly claim to have the best product managers, designers, coders, and SREs in the world, that the talent and the human expertise is the new gold and critical to their success. Still, many seem to forget that “software development is a fundamentally socio-technical enterprise” as GeePaw Hill called it. Software is written by people for people. There is no ‘disembodied’ technical system; it is inevitably joined to a social system. Maybe we ought to get a better grasp of social sciences, inviting them in to help us create a work environment where people feel more at home and proud of what they produce – just like the mechanical processing industries did 70 years ago.

The discovery of the semi-autonomous teams

Social scientists have been working with the mechanical processing industries since the Tavistock Institute entered the coal mines of northern England in the late 1940s to investigate a new way of organising work that had emerged there. Some seams went against the new practice of division of labour and expert supervision and control that had established itself with the introduction of new technology, going back to the earlier way of organising the work by having relatively autonomous groups which not only managed to improve productivity, but also worker commitment and their mental and physical health. What those studies and the decades of action research that followed have shown is that in order to make a sociotechnical system work at its best, both the social and the technical aspects must be taken into account in its design. Focusing on the technical side alone often leads to a dehumanising environment, while a focus on the social in isolation will not make the technology work at its potential. The technical and the social aspects need to be jointly optimised. 

What the workers in those coal mines did was return to their age old social system, where they themselves where in control and had the respect as dignified human beings. It was an act of survival and a reaction to the dehumanisation of the industrialisation of their work situation brought about by the new technology with little to no regards to the social impact. Many in the ICT industry will probably recognise this pattern, for example the XP teams that went against the prevailing best practice of the huge and multi-year projects in the 90s. Their frustration and disgust at the inefficacies and stern belief that sociotechnical systems can be predicted with any level of certainty led them to develop new way of working that would create better result faster and would make people more engaged and proud of their work.

This is one of many interesting parallels between the research done on what became Sociotechnical Systems Design (STSD) and what we today know as agile way of work. The main difference, apart from the obvious time and place, is that STSD was driven by scientific research and grounded in a solid theoretical framework. It has an epistemology and an ontology.

Forming a theoretical foundation

A seminal paper in the history of Open Systems Theory (OST), of which the sociotechnical perspective is a part, was published in 1965 by Fred Emery and Eric Trist. It was called “The Causal Texture of Organizational Environments” and went into an area called socio-ecology, people in environments, and introduces two important concepts that became the foundation for the theoretical framework. One was the extension of von Bertalanffy’s open systems idea, where a system is fully exposed to its environment and needs to have a transactional relationship with it in order to survive; not only by learning and adapting to it, but also actively engaging with to secure its future. The other was the identification of what they called ideal types of environments that a system may find itself in, referred to as causal textures, and can be seen as stages in an evolution. They can be described as follows:

• The first, placid random, is largely a theoretical one where all things are random and the environment is neutral; 
• the second, placid clustered, is more ordered as the elements are clustered, but the environment is still neutral; 
• the third, distributed reactive, is where competition occurs as there are not only clusters, but more than one element of the same kind; 
• and the fourth, turbulent fields, is the most complex environment where the dynamic processes not only occur due to the system’s interaction with it, but also from the field itself – “the ‘ground’ is in motion.” 

Three of the relevant causal textures illustrated as organisational structures, illustrated by the author for the purpose of this article.

The second field is how the world looked before the industrial revolution, where people lived and worked in self-managing groups in small self-sufficient communities, while the third came about as a direct result of the industrialisation of work and the competition that started at the end of the 18th century and went on to the 1950s. This was the time of scientific management, mass production, assembly lines, bureaucracies, command & control, division of labour and all that; a highly competitive and dehumanizing environment, where companies fought over the same resources, be it materials or people, but there was enough for all to go around, which created stability for a couple of centuries, up till the major disruption of the two world wars. 

People generally hate living and working in bureaucracies with all the competition, on any level, including nations committing industrial scale murder during the world wars, and one of the reliable products of competition is segmentation and division. People reacted to the dehumanisation of Type III, a rejection of its assumptions and structures and increasingly took things into their own hands. They no longer accepted just being a cog in the machine of the bureaucratic hierarchies of personal dominance and wanted more autonomy, involvement, and social support at work. The old values based on superior-subordinate relations gradually fell into decline as people set about deciding on new value systems based on more egalitarian and democratic relations. The world was entering into the turbulent Type IV environment.

Back in the ICT sweat shop of today, coders are huddled together in large open spaces, not unlike the textile workers in the industrial era, and given simple tasks to complete as quickly as possible, trying to catch the release train that runs on tight schedules. This is also where fear of failure is common, occasionally saved by heroes who go beyond the call of duty and are rewarded handsomely. This is exactly what the STSD researcher found to be anti-systemic, not jointly optimising for the technical and the social, but rather optimising each one in isolation. The fight for agile and empowered product/BizDevOps teams is similar to the return to semi-autonomous work groups in the industrial era and it takes us from a short-lived but stable Type III environment to a turbulent Type IV. The limitations of command & control are becoming more and more obvious as the extended technical field is getting increasingly unpredictable, driven by the enormous scale of the internet and what many refers to as the fourth industrial revolution, with its internet of things, large-scale M2M, machine learning, AI, and all the new business models that follow. This turbulence and exponential increase in variety can no longer be left in the hands of the few at the top – neither for the business nor the people working there.

Another similarity with the post-war era is that the workforce is becoming more demanding, especially where there is limited access to it as is the case in the ICT industry, and that every technical advancement requires more and more specialised skills. Also, the new generations are digital natives, be it the millennials, the Zs and the Alphas, and their values are even less congruent with the industrial bureaucracies than the previous generation X and the baby boomers before them. They care deeply about autonomy, meaning, personal development, self-expression and fun. In short, people require better quality of work as their basic needs are catered for. A good example of this is the so-called Great Resignation that has followed in the wake of the COVID-19 pandemic.

The two genotypical organisational design principles

How can companies then make headway in such a turbulent environment? And how can agile and the pipe-dream of a true product team come true? Let us once more take a look at the theoretical underpinnings of STSD, especially the OST extension developed by Fred Emery during the Norwegian Industrial Democracy Program in 1967. He identified what he referred to as the two genotypical organisational design principles, simply called DP1 and DP2 (DP – design principle) and they were defined by the way organisations get redundancy in order to operate efficiently.  DP1 is where there is redundancy of parts, i.e. each part is so simple that it can easily and cheaply be replaced. The simpler, the better, but that also means that they need to be coordinated or supervised in order to complete a whole task. This is what we all know as the classical bureaucratic hierarchy with maximum division of labour. The critical feature of DP1 is that responsibility for coordination and control is located at least one level above where the action is being performed.

DP2 on the other hand is where the redundancy is handled by functions, where each part, a person, can do many things and can fill in for any missing parts. Responsibility for the coordination and control is then moved down to the group and this is what we know as the semi-autonomous team with minimum division of labour. This is now called a self-managing group as it can encompass varying levels of autonomy. It also emphasizes the fact that these groups do not require external management. Management functions inhere in all levels from the strategic to the operational. One of the important differences between DP1 and DP2 structures is that the organization is now acknowledged to be an open system with a permeable boundary to its environment, be it competing companies, customers and their users, and even the extended social and technical fields. With highly motivated employees working in self managing groups from top to bottom of the organization, all the opportunities inherent in the environment can be maximized and the obstacles minimized.

From Merrelyn Emery's paper "The current version of Emery's open systems theory" (2000)

In the context of the causal textures, the first design principle, DP1, is what ruled in the Type III world, while the second, DP2, was the one we humans had employed since the dawn of mankind up until the industrial revolution. How about Type IV, the turbulent environment? This is where we see people trying to get back to DP2 with the modern cross-functional teams and people dropping out to start their own businesses. These are reactions to the problematic and increasingly maladaptive industrial bureaucracies.

The problem then, and now, is that it is mainly done at the “shop floor” level, where the primary work is done, while the rest of the organisation frequently is still in DP1 mode. This is neither DP1 nor DP2 at the organizational level, really, neither autocratic nor democratic, but a mixed mode. Many attempts at teams are actually laissez-faire where democracy is misunderstood as unconstrained individual freedom, or control is assumed where there is none. There are either no goals or the goals are not aligned and this can be seen as a maladaptation to the turbulent Type IV field, where things get progressively worse by maladapting to an existing maladaptation. It can be argued that industrialised Agile in many places fits this description, especially where it is scaled to account for the existing bureaucracy and thereby making it worse for both the DP1 style organisation and the DP2 style agile teams. 

Designing participative democratic organisations

What OST suggests is that an assisted and controlled transition from DP1 to DP2 is needed, without the risk of making the turbulence worse. Arguably, the only way to do this is a radical structural change that removes any remnants of DP1, moving from autocracy and command & control to a participative democracy where every human is given the opportunity to grow and fulfil their potential. Most people can agree on this and many companies try to make a change by a sort of human relation game, introducing new company values, but this has time and time again proven to make little to no real impact. The way that change actually happens is action or behaviour first, values and attitudes second, not the only way around. Merrelyn Emery puts it succinctly: “And no matter how many times you sing the song, you need more than love.”

Equally important, this type of change is not something that should be done only in parts of the organisation; this must be complete structural change, utilising the second most effective leverage point in Donella Meadow’s list of places to intervene in a system. “The mindset or paradigm out of which the system — its goals, structure, rules, delays, parameters — arises.”

Transition from the hierarchy of personal dominance to a non-dominant hierarchy of functions in the ICT industry, illustrated by the author for the purpose of this article.

The goal is a radical change in structure, from a hierarchy of personal dominance to a non-dominant hierarchy of functions, composed throughout of self-managing groups who cooperate across functions and up and down functional levels. This seems very close to what many agile and product proponents talk about, like Steve Denning saying “achieving business agility requires an organization to evolve from a hierarchy of authority to a network of competence” or Michael Dubakov’s “hierarchical organizations can't react to new market opportunities and changes fast enough, this impedes the company’s survival in the long run,” as well as similarities to the Spotify Model, Holacracy, Dialogic OD, and even the so-called teal organisations. They are all searching for DP2 without anchoring it in OST.

What could this radical change look like for a developer working on a system deep down in both the organisational hierarchy and the tech stack today? It would mean that instead of being someone who is managed and given concrete tasks to do, they would be part of a team that takes on the task of designing, developing, and maintaining a business capability, from start to end. The team would have all the roles and expertise needed to solve the whole task and nobody else would have a say in how it is done simply because those who can are in the team already, including the product managers. It has all the authority and agency to own the whole flow of value with a clear purpose not only aligned with the organisation as whole, but also their own intrinsic motivations as they are participating in the design of their own work and have psychological ownership of it.

Somewhat ironically, this may sound like a well-organised project team, staffed with all the necessary people and authority to drive changes efficiently with optimised flow, but this is a temporary structure frequently very much part of a bureaucratic DP1 structure with PMOs and the ordering-supplier divide between IT and the business. Unfortunately, most matrix organisations are really bureaucratic structures as Spotify experienced.

Not only were STSD and OST in a sense the original agile and product-based organisational design approaches, they were also where the concepts of team-first in a work environment originated. There was a huge amount of work done on sociotechnical studies before Emery discovered the genotypical design principles but most of that has been surpassed by building the design principles into the method called the Participative Design Workshop where people redesign their own section of the organization from DP1 to DP2. Also during the Norwegian Industrial Democracy Program, Emery and Thorsrud confirmed the power of the psychological requirements of productive work, called the six criteria: 

1. Jobs to be reasonably demanding;
2. opportunity to learn;
3. an area of decision-making;
4. social support; 
5. the opportunity to relate work to social life,; and
6. a job that leads to a desirable future.

These generally define satisfaction and are correlated with DP2 working. They have been codified and are routinely measured in every Participative Design Workshop in the early analysis stage where employees assess their organization and its effects on them. These requirements are now actually part of Norwegian law, explicitly included in Section 4.2 of the Work Environment Act.

Summary

To summarise, STSD as part of  OST, and the current demand for agile teams, have very similar goals to a democratic team-first based organisational structure which is designed participatively by the people themselves because, as Ackoff so wisely said, "it is absolutely essential that the people that have to implement a plan do the planning.” Not only will the companies get happier and more committed employees that produce better products and services they are proud of, they will be much more equipped to handle the turbulence of the Type IV environment as DP2 is variety increasing, not variety decreasing as the DP1. Move from the fragile industrial age mechanistic bureaucracies that reduce people to parts, to purposeful teams built by participative design where resilience, active adaptation (agility), and empowerment naturally emerges.

For software developers this can mean:

• Be more aware that the work has both social and technical impact, not only for yourselves but also for the digital products produced. Start early with a separate social analysis and cater for that dimension going forward.
• Embrace the collaboration with fellow developers and every other person involved in the design, development, and maintenance of the product produced. It is a team effort and every contribution is valuable. Be a team player as that is best for product, and yourself.
• As a team, take charge of as much of the process, information, variance control, and your own well-being as possible. This will reduce the need for outside control, and hopefully fully removes it so that you get full agency.
• Taking charge also means finding ways to reduce dependencies to other teams, enabling autonomy for all and being able to take ownership of the whole flow of work. Together you can make the world better for everyone in the company and be aligned with your common goals and values.
• If a network of semi-autonomous teams seems utopian, a major shift is needed to make it happen. Work for that, from wherever in the organisation you are. Not only will this massively increase your own quality of work life, but also the organisation’s chance of survival in the increasingly turbulent environment. Employ OST and participative design to help in that process, designing your own work, and thereby get psychological ownership of it.

Acknowledgment

I would like to acknowledge the lifetime efforts made by Eric Trist, Fred Emery, and Merrelyn Emery for helping the world move to both a more humane and environmentally sound way of living our lives, especially where we spent a lot of our time and effort: the companies we work for.

I would also like to thank Merrelyn Emery for all her constructive and honest feedback and contributions to this article, not only making it better, but also more aligned with recent research in the field of social sciences (which I am not part of).

About the Authors

Trond Hjorteland is an IT architect and aspiring sociotechnical systems designer from the consulting firm Scienta.no and has many years’ experience working with large, complex, and business critical systems, primarily as a developer and architect on middleware and backend applications. His main interests are service-orientation, domain-driven design, event driven architectures, and sociotechnical systems, working in industries like telecom, media, TV, and public sector. His mantra: great products emerge from collaborative sense-making and design. Trond tweets at @trondhjort and blogs here.