The Major Software Industry Trends from 2021 and What to Watch in 2022

As 2021 was coming to a close, the co-hosts of the InfoQ podcasts met to discuss the major software industry trends we've observed over the past year, and what they will continue to watch in 2022. In addition to this summary, the podcast episode is available for listening, as well as a full transcript of the discussion.

Daniel Bryant led the discussion and was joined by Wesley Reisz, Shane Hastie, Charles Humble, Srini Penchikala, and Thomas Betts.

Hybrid and remote working

The most significant change to how teams are working was the forced shift to remote working in 2020 has evolved to "hybrid" working, and it is likely to be the standard going forward. Shane Hastie, InfoQ's lead editor of Culture and Methods and host of the InfoQ Culture Podcast, said, "We're never going to be back to five days a week, three-hour daily commutes and all of that. Most of the people in the software space don't need to spend every day in an office in order to be productive." 

However, this does lead to a new set of challenges. A hybrid working model has teams mostly at home, and occasionally in the office, which requires finding the right balance for teams to be together in person. Some organizations are adopting more asynchronous patterns, and minimizing or almost eliminating synchronous meetings, such as Zoom calls. Hastie says these new ways of working require shifts in culture, philosophy, and mindset to be successful. He thinks the biggest long-term impact will be seeing which experiments are successful, and learning and adapting from the failures.

As we're in the middle of the "great resignation," it's becoming clear that individuals are placing more emphasis on their personal and family well-being. This relates to developer and employee experience, and organizations have to work harder to demonstrate why they are a great place to work.

In the area of diversity, inclusion, and equity, the changes over the past year have been mixed. In some respects, there have been steps backward, most notably women shouldering more of the pandemic-related burdens than men. However, diversity and inclusion have improved among companies that now hire remote employees, rather than a limited demographic who live near an office or can relocate.

The hybrid environment can potentially create a new division of haves and have-nots, and companies need to intentionally make an effort to treat all employees the same, regardless of whether they are in the office or remote. Wesley Reisz referring to a QCon Plus track focused on the return to offices including examples of Netflix taking deliberate steps so leaders can relate better to their remote employees. 

Thomas Betts brought up The Spectrum of Synchronous, which was the subject of James Stanier's talk in that track. The extreme ends are everyone in the office, with fully synchronous communication, to everyone is remote and all communication is asynchronous. The hybrid environment is somewhere in the middle, with a good balance. Async is fine for documentation, but you lose the humanity, so some face-to-face time is still required for a healthy environment.

Another aspect of remote and hybrid work is making connections and establishing a network. This is particularly challenging for new, younger employees, who may start their first job remotely. Companies and senior employees need to be intentional in helping the junior employees make those connections.

Architecture trends

Microservices continue to be a major architectural pattern, and in 2021 Thomas Betts raised the idea of a COVID Corollary to Conway's Law, where a company's ability to develop a distributed system is directly related to their ability to sustain a fully remote workforce. Conversely, if your process depends on a lot of people being in the same room, with synchronous communication, then you probably struggled with the transition to remote working, and are likely to be unsuccessful with microservices. Furthermore, effective asynchronous communication among remote teams can be a signal for the ability to develop asynchronous software. 

As for tools and standards, 2021 saw a lot of maturation of AsyncAPI, with many new tools and languages supporting the specification. It's a few years behind the OpenAPI spec it's based on, and the growth and adoption is increasing in a similar fashion. This will lead to more use of contracts to define async communication boundaries, and code generation to assist developers that integrate with event-driven architectures.

Data mesh is also starting to emerge as a new trend. The idea applies concepts from domain-driven design to data, with data products being just one more responsibility for a development team. In 2021, we saw many people talking about data mesh, and Betts looks forward to seeing if 2022 is when companies will start implementing it and sharing their success stories, and also the real world challenges they had to overcome.

AI and ML

Srini Penchikala identified four categories of AI and ML trends he was watching in 2021. First was data management. "Data management includes the data ingestion, data storage, data processing, data integration using GraphQL or other API-based data access solutions, and then data processing and analytics." Streaming is the main trend, with data being generated very frequently, such as from IoT devices or autonomous vehicles.  

In the next two categories, infrastructure and operations, Penchikala said the data space, which typically lags behind in innovation, is now catching up to trends we've seen in development and architecture. For infrastructure, companies are leveraging containers to run Spark jobs, using Kubernetes to scale up and down as demand requires for machine learning solutions, and are implementing GPU-based solutions. 

Data Ops is a DevOps-based approach to operating data systems, which then leads to ML Ops. Typical machine learning life cycles take models, training data and test data, and then require iterating to find which model will work well and be deployed into production. Doing this manually does not meet the needs of business users, and ML Ops tries to automate the process, with versioning and dynamically applying the models.

Ethics, energy, and the environment

Penchikala's final category for data is ethics, where the industry is trying to use AI, ML, and data engineering solutions with responsibility and fairness in mind. But data is just one aspect of ethics in software, and the subject is only recently being taught as part of a computer science curriculum. Humble said, "The systems that we build can have profound impacts on people's lives, and we don't talk about it enough, and we don't reason about it enough, and it's really scary."

While we're seeing ethics being taught for computing degrees, it's not to the same extent as what is done for engineering, where a certified professional engineer is signing off on a design as safe. One challenge with teaching ethics for software, as Reisz points out, is "we haven't had ethics with the type of software that we're writing today." When software uses ML models to determine which people receive resources, that's a more complicated issue than the trolley car scenario.

The environmental impact of software is also a growing concern, and only recently being discussed. Software uses 10% of global power, and has a bigger impact on the environment than the aviation industry. There are specific examples, such as cryptocurrency mining, but all AI and ML have significant impacts on people and the environment. 

Humble thinks we tend to focus on what individual developers can do, but this is actually a problem for the big cloud providers. If you shift your code to the cloud, then demand that all of the energy that your cloud provider is using is green, that would be a really good start, and will have more bang for the buck than anything else. The idea of demand shaping, where your code runs in the greenest possible way was the subject of an InfoQ podcast with Asim Hussain. 

Reisz went as far as to say that "sustainability and ethics is probably the largest problem we face as software developers." He referenced Astrid Atkinson who left Google to start Camus Energy and gave a presentation at QCon Plus . Reisz saw this as just one example of people who are applying their software skills to focus on the problem of energy and the environment.

Storing data on people

One subset of ethical concerns is data about people, how it is stored, used, and made available. The past few years have seen GDPR, CCPA and other similar initiatives, which has forced people to think about these issues. In the past, regulations have typically focused on forcing companies to implement a solution, which may already be out of date. In contrast, GDPR raised issues which didn't yet have clear solutions, and this led to more open discussion about the solution. This is still an ongoing issue, with many new questions to be answered. How do we secure individuals' privacy? Who should own the data? Where should the data live?

GitHub Copilot and coding assistance tools

The introduction of GitHub Copilot and other AI-based coding assistance tools has raised the possibility that, in an extreme case, developers will no longer be needed. Reisz does not think we will see true, artificial general intelligence anytime soon, but we will see AI being able to augment what humans do. He used cruise control in a car as an analogy. What started with just maintaining a set speed, has progressed to lane keeping assistance, and eventually full self-driving. Copilot helps with the boiler plate code, which developers have been complaining about writing for decades. This allows developers to focus more on their specific business needs.

According to Charles Humble, these tools take care of the easy bits, but, "The hard bit about software is figuring out what it is you're trying to build, and none of these tools really help with that." The same is true of low code solutions, which allow business users to quickly solve small problems. But those solutions fall down when you need software built in a repeatable fashion that can scale and grow. In a keynote at QCon San Francisco in 2018, Grady Booch described software is the art of working at higher and higher levels of abstraction. Reisz sees this as a higher level of abstraction.

Cloud computing

Cloud computing is a $90 billion (USD) industry, and is not getting any smaller. Multi-cloud and hybrid cloud are a reality, most commonly because of mergers and acquisitions. Because of this reality, we're now seeing technical solutions for hybrid cloud needs, such as GCP Anthos and AWS Outposts. Bryant thinks control planes will be interesting to watch, as companies deal with governance, security, and other interactions that span cloud providers. 

Adopting cloud native patterns will continue to be vital, as it allows for a more cloud-agnostic design, which could be deployed in different ways. A subset of this is cloud-native data practices, which have an integration layer in the architecture to handle common policies such as user authorization. This can help alleviate the problem of "data gravity" where everything in a system is built based on where the data is stored.

There are tools being created to help with the "paved road" for the cloud platform. Two that were called out were Open Policy Agent, which Reisz discussed with the projects co-creators, and Crossplane which allows you to build your own PaaS following the Open Application Model.

Software supply chains, blockchain and NFTs

Reflecting on some of the high-profile cyberattacks in 2021 led to a discussion about the software supply chain, and how we handle dependencies. Almost all software has dependencies, be it libraries or cloud services, which leads to concerns about how well you understand the full stack of code you are running. There are some tools being created, but currently, we don't have a way of saying, "I can verify that this piece of software is what I think it is." Sadly, this complexity will probably lead to more supply chain attacks, whether that's through injections when software is compiled or through distribution updates. 

Reisz is looking forward to seeing some viable uses of blockchain and non-fungible tokens (NFTs) in 2022. Initially, we'll see them in gaming and the metaverse, where you are able to buy something that is truly unique or in limited quantities, and it stays with you. This will lead to other types of software that can travel with you across different games. Blockchain could also be used for privacy use cases, such as a personal, private healthcare record.

While not currently in use, blockchain could also be used for certification of software, to verify the authenticity of a piece in your supply chain. Docker, VMware, and other companies are providing catalogs of approved components and containers, giving companies an extra layer of trust and verification of what they are using.

Looking forward to 2022

Bryant asked each of the participants what they are looking forward to seeing in the new year.

Penchikala noted that all the trends, technologies, and practices are interconnected. He's looking forward to seeing how data engineering, AI, and ML solutions can become more ethical and more fair. Machine learning and AI may not be replacing us, but for them to really make our jobs better we need to take time to make sure they are ethical and fair. This could be accomplished by standardizing the way we validate the ethical nature of these solutions, including defining fairness across all parties involved. 

Humble is really interested in the idea of improving how we read and understand code. For the past 30 years, our industry has been focused on making it easier to write code. But, as a programmer, you probably spend a majority of your time reading code, whether it's code you or others wrote, and trying to understand what it does. He's been looking at the Ballerina language, which has built-in sequence diagrams to represent the concurrency model. Changing the diagram will change the code, which leads to better understanding of how the concurrency model works. He recently spoke with James Clark about it.

Hastie is looking for, and seeing, more humanistic workplaces that treat people well. "I wish it was purely because organizations just cared about people, but the cynic in me says they've got to do it because they're losing good people. And if you want to attract good people, you need to give them a great environment to work in."

Reisz observed the common theme of working at higher and higher levels of abstraction. This gives us more ability and more reach in what our software can do.

Betts is really hoping to see a good example of a company implementing a data mesh, as opposed to just the high-level architecture designs. He thinks there will be some really interesting developments, and great stories from people saying how they handle data, how it all works, and why it's an improvement on the past. He also wants to see if it's possible to prove his COVID Corollary to Conway's Law, that hybrid and remote-first companies are better at writing distributed software.

Everyone had something to say about ethics and sustainability and sees those as the primary drivers for everything else in coming years. Humble plans to keep raising as an issue for our industry in whatever little way he can. Everyone is hopeful that we'll see progress there in the new year.