Synchronous request-response communication in microservices systems can be very complicated. Fortunately, asynchronous event-based architectures can be used to avoid this, Yaroslav Tkachenko claimed in a presentation at QCon London 2018, where he described his experiences with event architectures and how Actors can be used in systems built on this architecture.
Tkachenko, senior software engineer at Demonware, started by emphasizing that you cannot make synchronous requests over the network behave like local ones. There are a lot of challenges related to network communication that can be hard to overcome, especially if you lack experience working with microservices and distributed systems. For him, an event-driven architecture is a more straightforward approach to solve the problems that may occur.
At Demonware they still have a big application monolith but are slowly migrating towards independent services. Historically they have used a lot of synchronous request-response communication, but are currently working towards a more asynchronous communication and have decided to use domain events between services.
A typical service at Demonware is based on domain-driven design (DDD) and the hexagonal architecture, and with Kafka as a message transport layer. Events received are transformed to commands in an event adapter before sent to the business logic core. Correspondingly, events created by the service are added to Kafka, using fire and forget or a guaranteed at least once delivery, backed by a remote or local event store.
In short, Demonware are using an event framework with:
- Decorator-driven event consumers using callbacks
- Reliable producers
- Non-blocking IO
- Apache Kafka for transport
A way of simplifying this design is to use actors as they natively support message passing, both for consumers and producers. Briefly, actors:
- Communicate with each other using messages in an asynchronous and non-blocking way
- Manage their own state
- When responding to a message they can create child actors, send messages to other actors and stop child actors or themselves
When Tkachenko started to work with the actor model, he realized that the whole system started to look like a messaging or event-driven system. For patterns in this area he recommends the Enterprise Integration Patterns book and Vaughn Vernon’s book Reactive Messaging Patterns with the Actor Model.
For an implementation based on the actor model, Tkachenko referred to Bench Accounting where they transformed a large Java enterprise monolith into a series of Scala microservices and actors using Akka. Key components in this implementation were ActiveMQ to handle message queues, Apache Camel which is an integration framework and akka-camel, an official Akka library, although it’s now deprecated and replaced by Alpakka.
In a typical event listener in this implementation, akka-camel gets messages from the queue, deserializes, translates and does other work needed before they are sent to the actor. A message sender is similar; the actor uses akka-camel for serialization and other work before the message is sent to the queue.
In short, this implementation used an event framework with:
- Actor-based consumers and producers using Apache Camel
- Producers with ACKs
- Non-blocking IO
- ActiveMQ as transport
For Tkachenko, the lessons learned working with actors include:
- Semantics is important. Natural message-passing is a huge advantage
- Asynchronous communication and location transparency by default makes it easy to move actors between service boundaries
- More advanced advantages include supervision hierarchies, the cultivation of a "let it crash" philosophy, and excellent concurrency
His recommendations include:
- Domain-driven design (DDD) and the book Enterprise Integration Patterns are very useful for bootstrapping your understanding of this space
- Understand your domain space and choose the concepts you need among events, commands and documents
- Explicitly handle all possible failures, as they will happen eventually
- Avoid exactly-once semantics unless absolutely necessary, because this is very expensive
- Message formats and schemas are extremely important. Tkachenko recommends using binary formats like Protobuf or Avro. If using JSON he thinks it's important to design a schema evolution strategy early on
Some of the challenges he has experienced:
- This is difficult work, and it takes time to move from a synchronous paradigm to an asynchronous world
- High coupling will kill you. Avoid coupling by using events and minimize the use of commands
- It’s straightforward to implement event-based communication for writes, but really challenging for reads. DB denormalization, in-memory data structure or stream processing are common solutions to explore
- Managing a message broker cluster is still a non-trivial problem
The question of whether you should use actors depends on your requirements. Tkachenko notes that it’s possible to build asynchronous, non-blocking event frameworks in many languages, but actors-based frameworks are asynchronous and message-based by default, which is an advantage when you need it.
Most presentations at the conference were recorded and will be available on InfoQ over the coming months. The next QCon conference, QCon.ai, will focus on AI and machine learning and is scheduled for April 9 – 11, 2018, in San Francisco. QCon London 2019 is scheduled for March 4 - 8, 2019.
