Organizations are moving more and more of their processes into software, Jay Kreps notes in a blog post, and adds that he thinks an accompanying change is that businesses are increasingly defined in software – the core processes are specified and executed in software. To support this transition, he believes we have to move away from traditional databases into working with the concepts of events and events streams.
Kreps, CEO of Confluent and one of the co-creators of Apache Kafka, refers to an article by Marc Andreessen from 2011 where he described the world as being in the middle of a technological and economic shift with software companies taking over large parts of the economy. Kreps summarizes the article into the idea that any process that can be moved into software, will be.
Databases have been an extremely important part of application development, irrespective of how they store data, but Kreps notes that they follow a paradigm where data is passively stored, waiting for commands from an external part to read or modify the data. Basically, these applications are CRUD-based with business logic added on top of a process run by humans through a user interface (UI).
A problem with these CRUD style applications is that they commonly lead to an infrastructure with lots of ad-hoc solutions using messaging systems, ETL products and other techniques for integrating applications in order to pass data between them. Often code is written for specific integrations, and all this causes a mess of interconnections between applications in an organization.
To move away from relying on humans working through a UI to a platform that is able to trigger actions and react on things happening in software, Kreps believes that the solution is events and event streams. They represent a new paradigm where a system is built to support a flow of data through the business and reacting in real-time to the events occurring.
The core idea is that an event stream is a true record of what has happened. Any system or application can read the stream in real time and react on each event. Kreps compares this with a central nervous system but for a software defined company, and he points out that a digital organization needs the software equivalent to a nervous system for connecting all its systems, applications, and processes.
For this to work, Kreps believes that we have to treat the streams of everything that is happening within an organization as data and enable continuous queries that process this data. In a traditional database, the data sits passively and an application or a user issues queries to retrieve data. In stream processing, this is inverted: data is an active and continuous stream of events and queries are passive, reacting to and processing the events in the stream as they arrive. Basically, this is a combination of data storage and processing in real time, and for Kreps a fundamental change in how applications are built.
Other experts however believe there are benefits to using more types of messages besides events. In a presentation at this year’s Event-driven Microservices Conference in Amsterdam, Allard Buijze, CTO at AxonIQ, pointed out that there are different reasons why services or applications want to communicate. With only events available, they are also used to indirectly request something to happen in another service. This often increases the coupling between services and can create a very entangled choreography. Therefore, besides events, Buijze thinks there is a need for two other types of messages: Commands, which represent an intent to change something, and Queries to fulfil a need for information.
In a joint meeting with people from Confluent and Camunda earlier this year, Bernd Rücker, co-founder of Camunda, argued that we should talk about record or message streams instead of event streams, and emphasized that the term in Kafka API is records, not events. For him, Kafka can be used for different types of messages, which include both events and commands. He also noted that when only events are used, it can be hard to get a picture of the overall flow from a business perspective, and referred to an article by Martin Fowler where he points out that although the event notification pattern can be useful, it also adds a risk of losing sight of the larger-scale flow.