From Monolith to Event-Driven: Finding Seams in Your Future Architecture

Why Migrate?

Beyond the single-tiered monolith, a common architectural pattern is the 3-Tier Architecture which traditionally consists of presentation, business and data tiers. However, complexity is hardly ever proportional among those tiers. In some cases, we find business logic in the presentation tier or in stored procedures within the data tier. Yet, in other cases, we find the application tier divided into as many services as there are functions in the system. Some may even call this Microservices, but without the appropriate isolation, it is merely a distributed monolith.

Figure 1: Comparison of some architectural styles.

These of course are not all bad architectural styles but each have risks. The common threshold for migrating to a new architecture is when the risk associated with changes is great enough or cannot be mitigated. It is just as important to recognize the reasons for making an architectural switch as it is to validate them prior to jumping in.

EDAs are All About Events

Event-driven architectures (EDAs) are not new but we may find that the practices we use in implementing them often distract from the core principles consequently causing us to overlook that EDAs are really about the events’ primary role of being units of change within the system. Practices often used in EDAs such as messaging and asynchrony may introduce such noise. You can build an event-driven system inside a monolith just as much as you can with a distributed system; simply look no further than most operating system kernels such as Windows or Linux.

But events do not simply be. We don’t just end up with a PantsStainedEvent; a SpillCoffee action must have been invoked, albeit, inadvertently. An event must be caused by something because it is naturally a historical element and, similarly to stained pants, immutable. In a system, the creation of an event will be caused by actors such as users or time and usually communicated through a message such as an API call or a queue message.

EDA + CQRS

The CQRS pattern strongly suggests that it is about the segregation of commands and queries, but realizing that there is a difference between 1) asking for the state of a system and 2) asking the system to change its state is more fundamental than the separation itself. In fact, you’ll find many variants of CQRS implementations ranging from logical to physical. 

Combining EDA with the CQRS pattern is a natural increment of the system’s design because commands are the generators of events. With CQRS and commands, the migration of data during the transitional state of an architecture provides a seam by which, once the migration is over, can be removed. This seam will be covered in more details in the Data Migration Seams section.

EDA + Event Sourcing

Event sourcing a system means the treatment of events as the source of truth. In principle, until an event is made durable within the system, it cannot be processed any further. Just like an author’s story is not a story at all until it’s written, an event should not be projected, replayed, published or otherwise processed until it’s durable enough such as being persisted to a data store. Other designs where the event is secondary cannot rightfully claim to be event sourced but instead merely an event-logging system.

Combining EDA with the event-sourcing pattern is another increment of the system’s design because of the alignment of the EDA principle that events are the units of change and the event-sourcing principle that events should be stored first and foremost.

Event Types

Reflecting on our prior knowledge of the CRUD acronym, we can relate to it because all of the applications that we have built revolve around these core actions of creating, reading, updating or deleting something. We even find similarities in RESTful interfaces where HTTP methods map to these actions. As historic as CRUD may be, we also find it in modern architectural styles and engineering patterns such as EDA, CQRS and Event Sourcing. Let’s take a look.

When we look at the relationship between CRUD and CQRS, we find that when commands are separated from queries that CRUD is also distributed. The C, U and D letters are grouped with commands, while R solely groups with queries. This should be intuitive and only mildly interesting.

But, when we look at CRUD’s relationship to events along with CQRS, we see a similar association. If commands generate events, then an association between CUD and events also exists. We imply that all events must also be creational, updational, or deletional types. If you classify your events in this way you may have a property on the event, such as action or type, that contain values of either “create”, “update” and “delete.”

We will combine this notion of these three event types with another concept called Inception and Iteration.

Inception vs Iteration

Events in any system are historical and immutable. You can visualize this with a line graph with the line representing time and any event has one place on the line. The events on the line graph can be divided into 2 sets; the first set is called Inception and contains only one event, the first one, and the second set is called Iteration contains all the other events after the first event. They are called Inception and Iteration because they represent what is happening to an entity.

You thought we were done with CRUD? There’s one more grouping to look at. Since all events are CUD (recall the event types), then when we relate Inception and Iteration to CUD, we find that C associates with Inception, and UD associates with UD.

Sometimes, to find seams in your architecture, you will have to look pretty deep. In this case, we took the simple CRUD concept and mapped it pretty deeply to more concepts such as Inception and Iteration.

Modern vs Transitional

In a modern system, the event timeline is simple. The only way to incept or create a new entity is with an event type that is creational (event.type = “create”). If the system is in a transitional state, then introducing a new entity into the system is not confined to just “event.type = create”. You may also have migrational event types such as “event.type = migrate” as another way to create a new entity in the modern system.

Why is this important? For starters, if your system replays events, you should have strict or deterministic rules for how you do that. For example, you may have a rule that says “The very first event must be event.type = create” That should make sense. This rule won’t account for snapshots but the point is, even snapshots are derived from the same rule. If your system is in a transitional state then you can update this rule to also include “event.type = migrate”

Another reason for having the “event.type = migrate” be recognized by the system is because this is one of those seams that you’ve read about. This is the part where while you're transitioning to the new architecture, you have two ways to create an entity, and when you’re done, you can just remove it and reduce it to just one. You could essentially just delete all the code related to “event.type = migrate”.

The final two sections will unpack this and look into it much deeper. With data migration, we’re concerned with when the new, modern system first comes online and you want to feed it the initial state, whereas, data synchronization is where both legacy and modern systems are both online and state-changing events are occurring on both sides.

Data Migration Seams

Migrating data is when you want to load up your new system with the previous state from your legacy system. Sometimes, this is a one time event but you can also get away with doing this with some cadence, such as daily, given your migration is idempotent and does not duplicate state. So what does this really look like under the hood? Let’s look at it logically.

As we can see, one solution is to introduce a command loosely called MigrateDataCommand. When this command is invoked, it will reach into the legacy system, retrieve state, and create “migrate” events from them. The important thing to remember is that migrate events *are* actually very different from “create” events when you look at them from a business-rule perspective. While it is very possible to associate a set of business rules to a “create” event, such as validations (e.g. a valid Country Code) and generations (e.g. create a new ID), with “migrate” events, it should be rare that you have to attach business rules to them. The events resulting from a migration is the movement of preexisting states. That state became what it is because it in fact did go through business rules, albeit on the legacy system. The purpose of the “migrate” event is to simply carry that state to the modern system similar to what a snapshot does.

This is a seam because when you’re done being in a transitional state, you can just delete your MigrateDataCommand. There won’t be a need to invoke it anymore and you will stop generating “migrate” events. You won’t really have had an event handler for it, but if you did, you would delete that too.

Data Synchronization Seams

If you’re lucky, you can just turn on your new system and turn off your old one. But, most of us aren’t so lucky with such simplicity. Oftentimes, there is still business value left over in the legacy system before you’ve fully relocated it to the new system. And this means that when both systems are operational, you have events happening on both sides, and more importantly, probably against the same set of entities.

So how do you synchronize both systems? By using events of course. Now, this section does make some assumptions. It assumes you have some control of the legacy system. And by control, I mean, you can change its code and redeploy, have access to the data or otherwise, some way to hook into the actions performed by the users of that system.

Once again, let’s look at this logically.

As we can see, we had to make a modification to the legacy system and had to make it, not event-driven, but event-aware. We changed it so that right after it persists state to its data store it also generates an event. This event has the connotation of being “legacy” and naming of it should reflect that. On the modern system, we have handlers for those events. The handlers themselves don’t really know or care which system generated it, and to some extent, it shouldn’t care too much that it’s legacy at all. These handlers will replay these events with the same rules as others.

Now, of course, synchronization goes the other way to. Let’s take a look at what that looks like.

With back synchronization, when a command generates a new event, the new event is read by a legacy event handler. It’s sole responsibility is to write a new state into the legacy system.

This is a seam because when you’re done being in a transitional state, you can just delete your legacy handlers once the legacy system is offline and no longer produces legacy events.

Putting It All Together

So, what does a monolithic architecture transitioning to an event-driven look like in practice? And, just as importantly, where are the seams in the code that keep legacy and modern elements isolated? Let’s imagine a note-taking system called Notez that, as the name suggests, allows you to create and manage notes. We can’t really imagine where these seams are unless we take a much deeper look into what the modern system looks like.

The Notez Systems

We’ll assume that there are disadvantages with the legacy system to allow us to focus on the modern system. The modern system realizes the CQRS pattern in a maximum way by physically separating both command and query services into different services. Your implementation may vary. The command service’s job is to accept commands, and if valid, create events. These events end up in two places; first, the event is stored in the event store, and when that is successful, it is published to an event stream. The query service listens for events and delegates these events to an event handler. The event handlers’ primary job is to create any projections based on that event. Finally, as the name implies, the query service serves up queries by leveraging the projections. This is an eventually-consistent system.

Note: This is a very expensive system to build but it provides an evolutionary system quality. As with any evolutionary assessment of a system, you decide whether you want to pay for change up front, or later. This system pays for change up front which subsequently minimizes the cost of future changes.

Anatomy of the Command Service

The Command service uses a layered design to separate responsibilities. The separation is necessary to allow for each of them to evolve at different rates; for example, you may decide to add another API in the future. Layers are expensive because of the integration and redundancy cost but their benefits are what allows us to find seams in the architecture. Although each layer is important in their own rite, in a transitional phase, the Application layer is arguably the most important one. In the Application layer is where we’ll primarily find migration and synchronization commands. Let’s reimagine that the legacy system contains a bunch of notes that need to be migrated so that the modern system can recognize them, and that both systems are active together.

Let’s recall that there are two major aspects to pay attention to in a transitional architecture: migration and synchronization, where migration is the ability to present existing data to the modern system and synchronization is the ability for new data to be realized on both legacy and modern systems. You may not need both.

Migration Scenario

In the migration scenario, a new command MigrateNotesCommand is created. This command has knowledge of the legacy system and is the seam between the two worlds and provides a bridge for information to cross over. The most important aspect is perhaps the generation of NoteMigratedEvent events. As far as the modern system is concerned, it’s just another inception event that needs to be processed (e.g. a projection to be created for).

What’s not shown in this diagram is what happens to the published event. The processing of the migrate-event will be similar to any other event and is implied knowledge of having built the system to begin with. It’s actually one of the hidden points that the seam is in the Command-Business-Application layer and thus, focus is emphasized there.

Synchronization Scenario

From the Modern System

When the modern system’s state is changed by a command such as a CreateNoteCommand, the resulting event NoteCreatedEvent is published. This new state needs to be realized by the legacy system. With the event published, the command service can listen for it using an event handler. As the event handler receives NoteCreatedEvent events, it will invoke the synchronization SyncNoteCommand. Finally, the SyncNoteCommand will update the legacy system. In this diagram, the update is shown as a direct interaction with the data store, however, if applicable, web services or other interfaces may also be called.

From the Legacy System

When the legacy system’s state is changed, then the modern system is notified by an event. The event is created by a new part in the legacy system. This may be a command or other similar patterns that allows you to invoke it right after successfully persisting the entity. The message that it sends to the message broker is an event; in this example a LegacyNoteCreatedEvent. This event is a signal to the Command service to store a similar event. There’s a choice here: either store the legacy event as-is or remove the “legacy” moniker. It’s generally better to remove the moniker because it avoids confusion on the event handlers. In other words, there should be no event handler that listens to legacy events because they should rely on the command service to generate a similar one. You want to avoid responding to an event that may not have been persisted to the store, yet.

Where are the Seams?

Interestingly, the seams are right there in front of us. In the migration and synchronization scenarios, the seams are depicted by the blue elements. These are the elements in the architecture that you can add to enable those scenarios, but just as quickly remove them when they are no longer needed.

About the Author

Jayson Go (he/him/his), or just JGo, is a Principal in Software Development and Engineering for a fintech company focused on investments technology. He has over 20 years of experience in software systems, and currently assumes a platform architect role. His passion for software engineering is distributed among writing code, describing solutions, mentoring others and designing for stakeholders’ needs. If you meet him, ask him about domain-driven design, event-sourcing, defensive-design, functional-styles in object-oriented paradigms, distributed systems and software architecture governance.