The Apache Software Foundation has released Apache Kafka 3.3.1 with many new features and improvements. In particular, this is the first release that marks KRaft (Kafka Raft) consensus protocol as production ready. In development for several years, it was released in early access in Kafka 2.8, then in preview in Kafka 3.0.
KRaft is the consensus protocol developed to allow metadata management directly in Apache Kafka. This greatly simplifies Kafka’s architecture by consolidating responsibility for metadata into Kafka itself without the requirement of a third-party tool like Apache ZooKeeper. This new KRaft mode improves partition scalability and resiliency while simplifying deployments of Apache Kafka that now can be deployed standalone.
KRaft makes use of an event-based variant of the Raft consensus algorithm, hence its name.
The new quorum controller introduced with KRaft ensures that metadata is accurately replicated across the quorum. The active controller stores the metadata in an event-sourced log topic while the other controllers within the quorum follow the active controller by responding to the events that it creates. The event log is periodically snapshotted to guarantee that the log cannot grow indefinitely. In case of issues, unlike the ZooKeeper-based controller, the quorum controller does not need to load state from ZooKeeper since the internal state of the cluster is already distributed in the metadata topic. This significantly decreases the unavailability window, improving the worst-case recovery time of the system.
The image below shows a much faster shutdown of a Kafka cluster with two million partitions using the new quorum controller versus ZooKeeper.
The new KRaft consensus and quorum controller enables Kafka clusters to scale to millions of partitions through improved control plane performance with the new metadata management; improves stability, and makes it easier to monitor, administer, and support Kafka; allows Kafka to have a single security model for the whole system and makes controller failover near-instantaneous.
The Kafka community plans to deprecate ZooKeeper in the next release (3.4) and then remove it entirely in version 4.0.
In addition, Apache Kafka 3.3 comes with other new features like adding metadata log processing error-related metrics, allowing users to create delegation tokens for other users and strictly uniform sticky partitioner to improve the partition time.
For Kafka Streams this release adds source/sink node metrics for consumed/produced throughput, pause/resume topologies, and consolidates the KStream transform() and process() methods. Kafka Connect adds Exactly-Once support for source connectors.