Key Takeaways
- Software development is driven by learning experiences that generate knowledge
- Knowledge is contextual to a perspective’s role and responsibilities
- Software systems also need to cope with relativity of information
- Relativity means that the content of information is also inconsistent across locations
- To build great software, we need to account for perspective and relativity
An important part of software development is that it is driven by experiences—we learn about our systems through hands-on work—spiking and prototyping, experimenting and testing.
We try to have rich experiences using a variety of tools, practices, events, and concepts. We use those experiences to generate ideas and data, confirm or deny assumptions, and to solve problems.
This is our process of creation: write something, build something, see it work, evaluate. The code we write, the way we write it, the tests we use, the packages we import, the systems that we deploy it onto—we are going to bring all these all together through our processes of creativity and experimentation that coalesce into an overall structure and architecture.
So, we cut some cards on the Kanban, get a few people together, pull some work. Soon enough we have something working, we demo it to ourselves, we demo it to a customer, and then hey-whaddya-know, someone is actually getting value from our creation.
Now I ask you—at what point did the software exist?
When did we move from nothing to something?
We all come to the table with a different view of the world. We all understand what it means for something to “exist” differently. For one person it’s when the wireframes are built. For another, it’s not until the first tests are passing. And for the finance team? Way back when the budget got approved. You get the idea.
We can call this "perspective"—things have different meaning depending from where you observe them.
It is interesting to think that something as fundamental as existence can differ depending on what "hat" you are wearing. Compared to objects in the real world, this is an awkward thing to wrap your head around. The furniture in my house exists whether I’m an engineering manager working from my living room or a travelling salesman trying to find the shortest path between cities.
But for "processes," this is actually quite normal (which is what we have been describing so far —a process). They are abstract n-dimensional graphs, and materially different depending on who is examining them, in ways that one might not expect if one is not used to working with abstractions.
This is cool, but it is not novel, because it is true about other processes in business. If I am building a skyscraper, I work with many different disciplines to do the construction, and all of them will look at different criteria to assess their overall state of existence. (Though, again, this is much less obvious because the focus of the work is on real objects and not abstractions).
Where perspective gets really interesting is when you bring in the concept of relativity.
Relativity
Relativity was introduced at the beginning of the last century when Einstein proved that reality is fundamentally different depending on your frame of reference, a distortion of the spacetime continuum. The concept has led to the discovery of black holes, gravitational lenses, time dilation, and all kinds of other fantastic things.
Relativity is not at all what one would expect based on our regular day-to-day lives that operate according to classic laws of physics. It changes what it means to observe and to be an observer—it means that how we experience the world differs not just in how we interpret it. There are circumstances where the world I experience is inconsistent with yours.
It turns out that communication has these same circumstances that also work in this same peculiar way. Information is distorted depending on the location of the observer.
Mark Burgess calls this "information relativity": messages can take multiple paths and interfere with one another, information can be reversed in its order as it travels along one path, the speed of communication can be different from the speed of communication on another path. All this extends the problem of communication beyond the classic problem of how meaning differs, and into issues of consistency.
We are well aware of consistency issues in building distributed systems. We have the CAP theorem to describe some of the trade-offs we need to make against availability and partitioning. We have the idea of eventual consistency, which is now a household word (in my house). This is a tremendous problem to solve on a day-to-day basis in the dynamics of a distributed system, but it also plays an important role in our organizational dynamics, too.
Unlike building a skyscraper, in a software system, we are only ever working with representations, never “real things.” Our ability to obtain representations depends on the fidelity of our information networks—systems of people, tools, processes, and artifacts. The impact of information relativity on the dynamics of those networks is a key impediment to value that we all need to think about when we are building systems.
Take the problem of calculating capacity for a service across a multitude of microservice owners, perhaps working together to build a capacity plan for customer forecasts. Depending on which information sources they access, the way they access them, the queries they run, and even the time of day they run them, each group is liable to arrive at different results. This is true whether we are talking about infrastructure or sales data.
Or take the problem of managing an incident, where multiple teams need to work together to determine what is failing and how to fix it, then return later to understand why it failed and what can be done to improve the system. Depending on what team you sit on, you will use different tools to look at a phenomenon on different scales. We observe traces of the same phenomena in different ways.
These are difficult problems, and we have not even started talking about the classic issues of whether we understand our measurements to mean the same thing or whether we are measuring the same things in the same way.
So to build great software, we start with those classic problems of perspective, but we also need to grapple with questions of relativity, which does not operate according to the classic deterministic principles that we all know and love, the principles that operate when you eat your breakfast or bike to work.
We depend on unreliable materials—stories and data that build mental models about what is happening, and which are only ever in a partially-correct, never-stable state.
Though our systems are based on 1s and 0s, the answers to our questions do not conform to nice neat logical states of true or false. Quite the opposite, they exist somewhere on a spectrum of strong and weak inferences.
And this is the heart of our synthetic systems. To answer questions, we need to gather data and use a statistical, inductive approach, not a deductive one.
To be successful we need to think in terms of probability, not determinism. The unsuccessful will assume that what they hear (and think, and see) has been determined by the past or determines something in the future. But in our world of information dynamics, determinism is a harmful paradigm to try to work inside.
Leaders need to apply Synthetic Management practices, which means using empirical, experiential, and emergent approaches to managing work. Key principles include:
- Assume variability: the impact of relativity is difficult to predict, so we are better off focussing on strategies that emphasize converging eventually rather than trying to get it upfront.
- Use systems thinking: we won’t factor in relativity when we are only thinking about a subset of the system, or ignoring human factors.
- Get fast feedback: we cannot detect and respond to problems of inconsistency if we are not getting feedback from our environment.
These principles are not self-evident. The same way Einstein discovered that the world works differently underneath the atom, we are discovering the need to work differently in the world of information. What are the black holes and gravitational lenses in the physics of synthetic systems? What is still waiting to be discovered in the world of software development? History says, a lot more.
About the Author
John Rauser is a software engineering manager at Cisco Cloud Security, based out of Vancouver, Canada. His team is building the next generation of network and security products as cloud services. John has spent the last 10 years working in a variety of different roles across the spectrum of IT, from sysadmin to technology manager, network engineer to infosec lead, developer to engineering manager. John is passionate about synthetic management, new ways of working, and putting theory into practice. He speaks regularly at local and international conferences and writes for online publications.