Recently announced by HiveMQ, HiveMQ Enterprise Extension for Kafka aims to integrate Kafka and MQTT to enable real-time streaming for IoT applications.
The HiveMQ Enterprise Extension for Kafka is an implementation of the Kafka protocol for the HiveMQ broker, which enables the broker to act as a Kafka client and to stream messages coming from IoT devices to one or more Kafka clusters.
According to HiveMQ CTO Dominik Obermaier, Kafka can bring lot of value to IoT solutions thanks to its capacity to process large amounts of real-time data providing high throughput and low latency.
In conversations with HiveMQ customers, some operating clusters with millions of devices and very high message throughput, we saw the need for creating an MQTT broker extension for Kafka. [...] They want to use MQTT and HiveMQ to move data from devices to different back-end systems. The back-end systems include Kafka and also non-Kafka systems.
Kafka, though, is not by itself especially geared towards IoT applications due to a fundamental mismatch between the requirements of a typical stream-processing solution and those of an IoT solution. In particular, Obermaier mentions the restricted number of topics Kafka consumers and producers may handle in comparison with potentially millions of topics used in an IoT application; the complexity of typical Kafka API implementations which does not lend itself easily to use on constrained devices; the unavailability of key IoT features such as keep alive and last will and testament; and others.
This is where the HiveMQ Enterprise Extension for Kafka comes into the picture. One of its key features, indeed, is its ability to map MQTT topics to Kafka topics, thus allowing to overcome Kafka unsuitability to handle large numbers of topics. Similarly, all IoT devices will only see the MQTT broker, so they can keep using MQTT with its reduced requirements and full support for keep alive and last will and testament. Additionally, in cases where the unreliability of the network may break the connection between the broker and the Kafka clusters, the former will buffer all received messages, ensuring they are not lost.
HiveMQ Enterprise Extension for Kafka is not the only solution out there enabling MQTT-Kafka interoperability. Competitor solutions are Confluent Kafka Connect MQTT, which provides a Kafka Connect plugin, and several open-source solutions.
Based on a publish/subscribe model, Kafka is one of the most widely used platforms to process and distribute real-time data streams. MQTT is a publish/subscribe protocol particularly suited to IoT applications thanks to its small footprint, real-time guarantees, and suitability for use in high-latency, low-throughput, and unreliable networks.
On a related note, HiveMQ recently announced the open sourcing of HiveMQ Community Edition, an MQTT broker, and HiveMQ MQTT Client. The MQTT client was created in collaboration with BMW Car IT and is especially tuned for high-throughput and performance on memory and CPU-constrained devices.