Key Takeaways
- "Data plus architecture" is the idea that, more frequently, software architecture is adapting to consider data. This holistically includes data quality, data pipelines, and traceability to understand how data influenced decisions and AI models.
- Innovative software architecture is facilitating data quality the way we’ve improved code quality. Catching bad data early is as important as catching bugs early.
- The practice of software architecture does not belong solely to people with the job title of architect. Every engineer can actively participate in the architecture, and architects should help facilitate that process.
- One positive benefit of the pandemic and the shift to remote and hybrid work is increased asynchronous communication, which can manifest as Architecture Decision Records (ADRs).
- Software architects are adapting their feedback loops, which can be challenging when dealing with colleagues across many time zones or other remote work constraints. Good architects are learning from distributed working how to design better distributed systems.
Each year, InfoQ editors discuss what we’ve been observing across the entire software development landscape, and create several trends reports, each with its own graph of the adoption curve. This helps the editorial team focus its reporting on innovative technologies and ideas, and also provides our readers with a high-level overview of topics to keep an eye on.
There are two major components of these reports. The first is the written report you are currently reading, and includes the trends graph and details on individual items that have been added or changed in the past year, as well a general analysis from the InfoQ editors.
The second part of the report is an episode of the InfoQ Podcast, which is a chance to listen in on part of the editors’ conversation and hear some anecdotal examples from our panel of expert practitioners.
Updates to the Trends Graph
In 2021, we looked at many ways architects shape their designs, considering major "-ilities." This year, we’ve added "design for security" to that list, and consider it at the early adopter stage. While security has always been a factor, it is increasingly important to consider security early and repeatedly when designing a system and its components.
Architecture may not be part of the portmanteau triplet DevSecOps, but it is an underlying component of that idea. There are also many similarities between cybersecurity and software architecture, as discussed with Maxime Lamothe-Brassard on the InfoQ Podcast.
We considered adding “design for scalability” but decided it was captured under the idea of “design for resilience” and other trends that exist to accommodate scalability. In the second year of the pandemic, consumption of online services continued to grow, and companies had to be able to scale to meet the demand.
eBPF, extended Berkley packet filters, are a way of programming the Linux kernel, and allowing developers a way of adding capabilities to the operating system at runtime. We’ve added eBPF as an innovator trend and will watch its adoption over the coming years. Liz Rice provided a great overview of the technology in a recent episode of the InfoQ Podcast.
The term "Next-gen GraphQL" was changed to GraphQL Federation, as it is the foremost aspect being adopted by companies building on top of their existing GraphQL implementations.
“Data gateways” was removed, and replaced with the “Data + Architecture.” Also, Architect Elevator was replaced with Architecture Decision Records. These changes are discussed in detail later in this report.
Both Dapr and WebAssembly have moved from Innovators to Early Adopters.
A couple of items were removed due to a lack of adoption and insufficient innovation being mentioned in the past few years. These were Open Application Model (OAM) and RSocket & Reactive Streams.
Data + Architecture
We’re seeing a transition from data being considered only at the storage or transport layers in a system to data being a defining element of the system. This is apparent in AI/ML systems, which are built from data, as well as in design patterns such as event sourcing. Sometimes this is manifested in a well-defined concept, such as data mesh, which we are continuing to watch as a separate innovator trend. More often, architects are just considering data differently, which led to the addition of “data + architecture” on this year’s trends graph.
In an interview on the InfoQ Podcast, Neal Ford and Mark Richards talked about the need to include chapters about data in their new book, Software Architecture: the Hard Parts. Neal said:
We wanted to make sure that we have a more complete picture because we firmly believe that data plus architecture is going to be one of the themes over the next few years. And it is very much the theme of a lot of our professional work that we’re doing right now—untangling that relationship and getting that to be a smooth working relationship is a collaboration again, between DBAs, architects, and getting those practices into the same kind of practices we’re getting accustomed to in architecture and design.
Data + architecture encompasses some of the SEO-friendly terms, such as data gateways which it replaced on the graph. It also includes ideas like data quality, and the introduction of tooling that tries to solve the problem of “How do we make sure that the data we use is correct? How do we know that it is what we expected?” The consequences of not dealing with this question can be disastrous.
InfoQ editor Eran Stiller considers data quality in his work at Badook and has the following to say on the subject:
Companies today are increasingly relying on Artificial Intelligence and Machine Learning algorithms. They use them to power their customer experience, forecast their sales and stock level, and gain valuable insight into their business. In some cases, they even make fully automated decisions without human intervention. These algorithms rely on data. If the data is good, they can build a usable model. But unfortunately, our data changes over time, and its quality decreases. Errors in data collection occur, data pipelines can inadvertently wreak havoc on our data, and assumptions that we previously had on our data change over time and might no longer be valid, causing potentially catastrophic business results.
For example, real-estate company Zillow recently wrote down more than half a billion dollars after its data model for predicting house prices and driving its business decisions failed to adapt to a changing reality.
In addition, regulators around the world, such as the European Union, are in the process of regulating the way Artificial Intelligence algorithms are created and used. A similar Algorithmic Accountability Act was recently introduced in the US Congress. A crucial part of these regulations will be ensuring that the data used to derive the algorithm is valid.
Consequently, I predict that data quality and reliability will be a topic of growing concern with most, if not all, tech companies. Like we test our code, we should test and validate our data and ensure that it behaves as we expect it to. If we don’t do that, we might wake up one morning and base our business strategy on corrupt forecasts, or even worse—make catastrophic fully-automated financial decisions that could cost us billions. I’m sure that none of us wants to be in that position.
Architects and Architecture
Across the industry, there is a wide spectrum for defining the role of an architect and the practices of software architecture and design. Many companies do not have a job title for a software architect, and those that do can have very different responsibilities. Despite this, software is always designed and always has an architecture. Our challenge is to identify the innovative ways architecture is being done, and how this can manifest in the role of software architects.
This is a hard idea to condense into a short concept on the trends graph. Last year, we had “architect elevator” Gregor Hohpe’s idea of the architect needing to communicate at multiple levels of the organization. Before that, we had “architect as technical leader.” Both of those were in the early adopter category, mostly from a gut feeling that they weren’t necessarily innovative, but also not widely adopted.
This year, we have included “architecture decision records (ADRs)” as an early adopter idea. A great reference for ADRs is from a 2011 article by Michael Nygard. Most software teams are familiar with “boxes and arrows” architecture diagrams, which are useful for documenting either a desired or current state of a system.
However, it is often useful to understand why those design decisions were made, which is the purpose of ADRs. By capturing the decision, along with other options considered and the trade-off analysis done at the time, an ADR can be very useful when a design has to be updated in the future, or simply as a reminder when the design is questioned.
One side-effect of building distributed systems, and microservices built by semi-autonomous teams, is that architecture decisions also become distributed. Therefore, architects need to help empower teams to perform architecture tasks, such as trade-off analysis, making good decisions, and documenting and communicating those decisions. ADRs can be an important part of scaling software architecture, according to Andrew Harmel-Law.
To hear more discussion on the role of architects and the practice of software architecture, I highly encourage you to listen to the accompanying podcast.
