Amazon MQ is a managed message broker service for Apache ActiveMQ Classic and RabbitMQ, simplifying the setup, operation, and management of message brokers on AWS. Recently, AWS announced support for quorum queues, a replicated type designed for higher availability and data safety, for Amazon MQ for RabbitMQ.
The RabbitMQ quorum queue is a modern queue type that implements a durable, replicated FIFO queue based on the Raft consensus algorithm. The Raft Consensus Algorithm is a protocol for managing a replicated log of states across a distributed system to ensure consistency and reliability by electing a leader who coordinates the order and commitment of log entries.
In a LinkedIn post, Saineshwar Bageri, a solutions architect, explains:
Quorum queues provide the replication capability to ensure high data availability and security. They can replicate queue data between RabbitMQ nodes, ensuring that queues can still run when a node breaks down. In Quorum queues, messages are written on the disk for Persistence. You need at least a Three-Node Cluster to Use Quorum queues.
(Source: LinkedIn Post)
In the RabbitMQ documentation, the queues are described as:
Quorum queues are designed to be safer and provide simpler, well-defined failure-handling semantics that users should find easier to reason about when designing and operating their systems.
Furthermore, in an AWS Compute blog post, Vignesh Selvam, a senior product manager at Amazon MQ, and Simon Unge, a senior software development engineer at Amazon MQ, write:
Quorum queues can detect network failures faster and recover quicker than classic mirrored queues, thus improving the resiliency of the message broker as a whole.
Developers can use quorum queues by explicitly specifying the "x-queue-type" parameter as "quorum" on a RabbitMQ broker running version 3.13 and above. The company recommends changing the default vhost queue type to ’quorum’ to ensure all queues are created as quorum queues by default inside a host.
RabbitMQ queues console (Source: AWS Compute blog post)
Quorum queues suit scenarios where data durability and fault tolerance are critical. Yet, they are not meant for temporary use, do not support transient or exclusive queues, and are not recommended for unreplicated queues.
In addition, quorum queues perform better when they are short. Developers can set the maximum queue length using a policy or queue arguments to limit the total memory usage of queues (max-length, max-length-bytes).
Other message brokers use quorum mechanisms to enhance reliability, especially in distributed systems where consistency and fault tolerance are critical. For instance, Apache Kafka, a broker primarily using a log-based approach for message durability, can support quorum-based replication for increased reliability and consistency. Another example is Apache Pulsar, which uses a distributed ledger for message storage managed by BookKeeper, supporting quorum-based replication for data consistency and durability. Lastly, NATS Streaming (now part of NATS JetStream) supports quorum-based replication to ensure message durability and fault tolerance.
More details on the quorum queues in Amazon MQ for Rabbit MQ are available on the documentation pages.
