Moldable Development: Guiding Technical Decisions without Reading Code

Why do developers read code?

They read code because that is where the ground truth lies. And they read in multiple circumstances, too. They read when they want to learn how to use an API. They read when they seek the area to add a new feature. They read when they search for the root cause of a bug. They read when they hunt the reason why the build configuration does not work. They read when they investigate why a test fails.

Reading is relied on pervasively during development. In fact, developers spend the largest amount of the development time doing just that. Multiple studies reveal that typically more than half of the development time is being devoted to reading.

Reading is so ingrained in everyday development work that people take it for granted. And why should they not, after all? A common view on programming is that code is like a book that we write and read just like we do with any other book. Yet, code is not quite like a book. A key differentiator is that code is typically much larger than a typical book. Say a system has 250'000 lines of code. This would correspond to the size of some 64 copies of Hamlet and reading it fast, say 1 line in 2s for 8h per day, would take about a month worth of work. And that's not quite all. While Hamlet has few characters and a plot that can be summarized succinctly, a system has little code that one can do away with. Reading the code once is rarely sufficient to get an idea of the system. Because of the intertwined nature of a software system, for one to understand it, it is necessary to understand it as a whole by taking into account all the inter links. Add to this the fact that a living system changes all the time at a much faster pace than we have a chance to consume through reading, and we get a problem that is qualitatively and quantitatively distinct from that of reading a book.

If we take a step back, we can observe that while the circumstances can be diverse, the motivation isn't: developers typically want to understand something about the system enough to figure out what to do next. They read because they want to make decisions. This is quite different from reading Hamlet. And it's different from reading a text book on astronomy, too.

From this perspective, reading is merely a tactic to retrieve information from the system. When there is a way, there can be another way. Indeed, everything about a system- be it code, configuration, logs, or data in the database- is just that: data. And data is best dealt with through tools.

Of course, there hasn't been a shortage of analysis tools of all kinds over the past decades. Still, even today reading is still the primary source of gathering information from a system. "So what?" you may ask.

Well, given that it also accounts for the single largest cost in software development, we should want to optimize it. The good news is that, as reading is the most manual way to extract information from data, we should be able to optimize it quite dramatically. After all, this is the very activity that makes our industry so valuable: most systems help humans make decisions about data they will never see, and we, developers, have the ability to create this magic. We just need to apply our abilities to our own problems!

“Ok, but how?”

There is one more peculiarity of software systems that we should take into account: they are highly contextual. Two systems might work in the same domain, be built from the very same technologies, and still be radically distinct. A few years ago, we did an experiment that tried to replicate such a scenario. Multiple student teams at the University of Berne were given the same small application to build, and had to use the same libraries. They were facing the same requirements and were tested for acceptance in the same way. After seven weeks of development, we looked at the inside of their systems.

The picture shows seven systems, and for each we depict classes in black, methods in red, attributes in blue, and all their inter-dependencies. The graph is arranged following a force layout. Even though the teams were highly constrained, they still produced distinct structures. The inner workings of a system emerge and are influenced by many factors that go well beyond technical specifications.

It is because of this contextual nature that generic tools never really work in practice. They can produce beautiful pictures, but they do not affect the way humans make decisions. For decisions to be relevant, they need to take the context into account, and generic tools are simply not aware of it. For example, the visualization from above is generic in that it is applicable to any system. It is useful to depict the general nature of software, but it is not useful for addressing any specific problem related to our systems.

The alternative is to create custom tools that take that start from the context and offer an experience that matches it. The primary focus of the tool should be on showing the problem in a way that makes it comfortable to understand. The solution typically follows quickly afterwards. We call this Moldable Development.

Let's consider a concrete example.

Example: Reasoning about Feature Toggles

Dependencies occupy a prominent space in the consciousness of developers. They add dependencies, they remove dependencies, they manage dependencies, and they break dependencies cycles all the time. This is a generic problem. Still, not all dependencies are created equal. Some are more interesting than others. And some are more obvious than others.

While describing Moldable Development by Example at QCon Plus 2021, I showed a story related to dependencies induced by system settings and feature toggles. You see, when a feature toggle is defined in a component and is used in another component, a dependency appears between the two.

My colleagues and I recently had the opportunity to explore the feature toggles from Open edX, an open-source system written in Python. The visualization below highlights in red all places that use a feature toggle on a treemap, a compact visualization of hierarchical data shown as a set of nested rectangles. In this case, the treemap shows the folders and files from the system. The files that use at least one feature toggle are highlighted in red. Indeed, this is a system that relies heavily on the use of feature toggles.

In its most basic form, a feature toggle is a boolean setting. Of course, in a system there are many booleans, but only a few of them are toggles. Typically, defining and using toggles relies on either patterns, or on libraries. It so happens that at the time of our investigation, there were three ways of expressing toggles which were based on different libraries. Thus, in order to draw the visualization above, we first had to create a dedicated tool that could understand those contextual libraries.

We use Glamorous Toolkit, the moldable development environment. Glamorous Toolkit is specifically focused on empowering developers to create custom tools inexpensively. Here is an example of using the environment for an analysis. In the left pane we see a live notebook page containing a query to find toggles that are using a legacy mechanism. In the middle pane we see an inspector over a list of toggles. And on the right we see an inspector over one toggle showing the Python source code.

In our example, the detection reveals a toggle defined as:

HIDE_ACCESS_DENIALS_FLAG = LegacyWaffleFlag(
    waffle_namespace=COURSE_BLOCKS_API_NAMESPACE,
    flag_name='hide_access_denials',
    module_name=__name__,
)

As the name suggests, this is a legacy mechanism for defining a variable in this system. The intent is to migrate these. Of course, before we can migrate, we first need to find the offenders. And to find them, we need a custom tool that understands the framework. That is what the query from the notebook captures:

Gt4EdxGlobals toggleModel allEdxToggles select: #isLegacy

This is a query that relies on an existing model of toggles and selects them based on a flag that the model creation logic set previously. We created this model and the logic to populate it specifically for the Open edX system.

This scenario captures a couple of further things. First, notebooks are as useful for data science as they are for software engineering. Second, the analysis is supported by inspectors relying on custom views over a custom model of the system. For example, on the right we see an inspector over an instance of a class called Gt4EdxToggleDefinition. That is an entity in a model created specifically for the analysis.

Once we have a model capturing facts about our system we can formulate questions that are less obvious. At the beginning of this section we talked about dependencies induced by feature toggles. Here is how they looked in our system. On the left we see a visualization of top level modules and their dependencies induced by feature toggles. Clicking on one of the arrows reveals the feature toggles that induce that dependency. On the very right we see the details of such a use of a feature toggle.

The specific example shows the retrieval of a feature toggle and storing it in a context variable:

context = { 
  ... 
  'enable_account_deletion': configuration_helpers.get_value(
            'ENABLE_ACCOUNT_DELETION', ...)
  ... }

The use of the toggle happens in the openedx component of the system. However, the definition of the toggle is in the lms component, therefore it generates a hidden dependency between the two. This is but one example.

This section showed a couple of typical questions that appear during development. For example, should we want to split the system, we would want to know what the hidden dependencies are. We would want to know that we are always in the position to answer any questions about the system. In our case, we answered the questions through static analysis. But the idea of Moldable Development applies to any aspect in a software system, static or otherwise, in a similar way: we start from a question, build a custom tool, and interpret the results to find the path to action. And at the end, we find we can reason about much of our system without actually reading code.

Indeed, the idea of guiding our decision-making through custom tools represents quite a departure from the current state of practice. At first sight, the proposition can perhaps appear as a waste. After all, does this not mean that developers will just spend a ridiculous amount of time building and polishing tools thereby decreasing productivity? In short, no. Quite the contrary. But let's get back to it a little later.

Glamorous Toolkit: the Moldable Development Environment 

In the example above, we used Glamorous Toolkit to reason about a system written in Python. Glamorous Toolkit is a moldable development environment specifically designed to decrease the cost of custom tools significantly. For instance, a custom view is often a matter of minutes to create. Glamorous Toolkit is a broad environment that offers an experience that unifies programming, data science and knowledge management. It is a Smalltalk system based on Pharo, but the key aspect of it is that it can be programmed and customized in many ways.

Glamorous Toolkit is also a case study in that we have created it following Moldable Development. To this end, we have thousands of small tools right in the base distribution. How do we know that? That's a good question, and like any question about a system, we can answer it through a custom tool. In this case, we can write a simple query about Glamorous Toolkit using Glamorous Toolkit.

In the screenshot we see on the left a notebook that contains a piece of code:

#gtView gtPragmas

Executing that script shows the result in an inspector to the right. In this case, the result is something called a filter that knows how to show the actual places in the code, but also an overview of metrics. In this case, we see there are 1984 methods. These methods define the custom views that the inspector shows. For example, the Metrics view itself is a custom view that is defined as:

gtMetricsFor: aView
	<gtView>
	^ aView columnedList
		title: 'Metrics';
		items: [ {'Number of methods' -> self amountOfMethods.
				'Lines of code' -> self linesOfCode.
				'Lines per method' -> self linesOfCodePerMethod} ];
		column: 'Metric' text: #key;
		column: 'Value' text: [ :assoc | assoc value asString ]

Why do we have this view? Because we once wanted to learn how many custom views we have. The Moldable Development flow typically starts from a question or hypothesis; we then ask what kind of tool would offer the interesting summary; and if we don't have the appropriate tool, we create one. And we do this for every question we have about our system. And often, these tools are only used once.

"What folly is that?! Build a tool and only use it only once?" Yes. It's merely a question of cost. If the cost is small enough, it's actually more profitable than the manual alternative. This is not as far-fetched as it might appear. The flow is similar to what can be observed in data science, where people create custom narratives for each question.

“So, what can a tool be?” A custom view over an object is a tool. A custom debugger is a tool. A custom search is a tool. A custom API browser is a tool. A custom log analyzer is a tool, too. All these are tools and it's the job of the environment to make them inexpensive to mold. Once this happens, the flow of programming changes fundamentally.

"Still, would it not at least be more economical to aim for reusing tools?" If you can reuse your custom tools, do it. For example, those 1984 custom views from the default distribution of Glamorous Toolkit are reusable. However, do not start with the reuse in mind. Instead, aim to amortize the cost on the first use. This will free you from having to start with the economic case before starting creating your tools.

"Ah, but should the tools be necessarily custom?" If we look at our example, we cannot answer questions about settings and toggles without understanding the details of the framework used to define them. In fact, in this system there are more than one such frameworks. Most decisions in software development are about specific situations that are not predictable from the outside. In fact, if your question is not specific, it's likely not useful. And if you find yourself using a tool that is usable as is in other systems, you likely are underserved.

Let's consider a common case of looking at a build log. Here we see three panes. In the first pane, we see the build status embedded in a page from the knowledge management system. In the second pane, we see a graph of the stages. The selected build actually failed, and the map shows that the problem appeared in the very last stage. Selecting that last stage shows the detailed steps that the build executed. One of these steps failed. And when we look inside the step, we immediately spot the specific error highlighted in red.

In this case, the error was:

Error: Expected version v0.8.1839 does not match the actual v0.8.1838

This error is not generic. It mostly makes sense in the context of our build system. Therefore, to highlight it, the view must necessarily take our context into account. In fact, I saw this error recently and it was not highlighted because we did not encounter it before. A few minutes later, I extended the highlighter with a few lines of code:

('Error: Expected version' asPParser token
	, #endOfLine asPParser negate star token , #endOfLine asPParser)
	==> [ :tokens | 
		(aText from: tokens first start to: tokens second stop)
			highlight: Color red ]

Anything in the environment can and should be adapted to the context. Of course, it helps a great deal if the environment already makes it possible to quickly create custom experiences. In our example, you might notice that we did not go to a separate tool to gather the information about the build. The I in IDE stands for integrated. If the environment is to be integrated, we should never want to leave to find information. Instead, we want the information to come to us so that we can manipulate it in the way that fits our context. That's the essence of Moldable Development.

To make this practical, the environment must make the creation of tools inexpensive. That is why we see the environment not so much as a ready-made tool, but as a language made out of interactive and visual operators that can be combined in many ways. Glamorous Toolkit offers many such small parts that can be pieced together to form a contextual experience.

Implications of Moldable Development

Today, developers spend most of their time reading code. However, they rarely talk about it. And when they do, as in the excellent recent work of Felienne Hermans on code reading, the reading part is never actually challenged. Reading is a great strategy when you learn a new language or when the problem is captured on a single screen, but it’s not effective when we have to deal with problems that are larger.

Moldable Development challenges reading as a means to gather information from the system. Instead, it offers a systematic approach that relies on creating dedicated experiences that summarize the system. In this article we detailed a scenario in which we could indeed summarize a situation around a system without relying on reading. This was, of course, just an example that could fit in a short article like this one. There are many more because developers make many decisions per day, and each of these can be sped up through automated, yet contextual summaries.

We talk here about the largest chunk of the development budget which today is being spent in an implicit, unoptimized way. The good news is that this represents a huge opportunity: if we affect this effort significantly we directly affect the entire development. We, as an industry, automate the tedious parts of everyone else's decision-making. We do that by creating dedicated experiences that summarize data and through which people reason without ever seeing the raw data. It's a magic skill that touches everything in our society. We only have to apply it to our own work, too. The flow looks as depicted below.

It all starts with a hypothesis or at least an explicit question about our system. Because of the size of the system, we want to gather the information through tools. Once we have results, we interpret them. If we are confident in our conclusions, we act. If not, we refine and retry. This is nothing new. It's the scientific method. What makes Moldable Development distinct is that given that software is highly contextual we cannot predict the exact problems people will encounter, and therefore we cannot produce ready made tools. For a given hypothesis, it is likely that we do not have an appropriate tool ready. In that situation we should mold a suitable one. That's it.

Moldable Development requires new technology that makes the creation of custom tools as inexpensive as possible. We develop Glamorous Toolkit and we are offering it free and open-source to show how the approach works in practice and to reveal the extent to which systems can be made explainable. However, Glamorous Toolkit is just a technology. Moldable Development is a general approach that we believe is fundamental to software engineering.

Moldable Development requires dedicated skills, too. The most obvious is the skill of creating custom tools. This is not that different from that of a data scientist that creates custom visualizations. But that is not sufficient. The most important skills are hypothesis forming and decision-making. While the technology side can be adopted and learnt pretty much like any other technology, the formulation of specific questions is a skill that can be built only over time.

Once the technology and skills are readily available, there are new opportunities that open up. Let's untangle them through a Wardley Map.

Both technical and non-technical people face challenges and need to make decisions regarding their systems. Those challenges always relate to specific problems. Typically the decisions are informed by views put together through manual inspection of the system. Given that the system is much larger than can be gathered through manual inspection, it follows that manual views are always incomplete, and often wrong. The alternative is to generate the views automatically. Of course, for the views to have value, they need to be specifically defined. This requires specific coding. All this is possible only if tools become moldable, too.

The effort associated with creating custom tools does not increase the development cost. It actually decreases it. You see, the budget is already being spent on manual exploration. Transforming some of that energy into coding optimizes the manual work and the total cost ends up being smaller.

We can view Moldable Development as a way to reduce costs. That alone would be worth it. However, today, the speed and quality of decisions are paramount. Not only does the generation of custom view replace the largest cost today, it also enables companies to make decisions faster, and more accurately. That is where the greatest potential lies.

Consider a case of a recent company we worked with. They were trying unsuccessfully for a couple of years to optimize the performance of a data pipeline used for critical marketing campaigns.When they first described their system, they drew an architectural diagram containing four large components. A closer inspection revealed that, in fact, there were five large components. We do not talk here about fine grained details of data lineage. We talk about the coarse grained understanding of the system. Essentially, they did not really know what the system was made of. This sounds preposterous, right? How can people that work on the system not know it? Yet, this is not far from the typical experience we have with most companies that have legacy systems. Over and over, we meet teams that believe they know the system only to learn that the reality is significantly different. In general, if the basis for decisions is a hand-produced diagram, we can assume that the knowledge is lacking. Or, worse, that it's wrong.

Systems evolve over time in ways that make initial knowledge obsolete. We cannot rely on memory in these situations. Instead, we want to be able to draw a picture by asking the current system through automatic means. Obtaining accurate views for questions at low costs opens the possibility to ask more questions, thereby leading to a new feedback loop.

These questions are technical in nature, but have non-technical implications. Software systems increasingly document and automate the critical knowledge of an organization. The ability to access and change the internals of systems will be the key source for competitive advantage over the next decade.