During Build 2018, Microsoft announced it would support Kafka clients to integrate with Azure Event Hubs. The Microsoft engineering team responsible for Azure Event Hubs made a Kafka endpoint available for users of their service to stream event data into it.
Event Hubs are an event ingestion service in Microsoft Azure and provides a highly scalable data streaming platform. Developers can leverage Event Hubs instead of a running Kafka cluster, and benefit from features like Capture (automatic delivery to Blob or Data Lake storage), Auto-Inflate (auto-scalling of throughput), and Geo Disaster-Recovery. Tom Kerkhove, an Azure consultant at Codit and a Microsoft Azure MVP, told InfoQ:
By using Event Hubs for Kafka Ecosystems, I can use the Kafka protocol without worrying about infrastructure and focus on my events.
Both Kafka and Azure Event Hubs are scalable distributed event-driven messaging platforms with multiple producers and consumers. Furthermore, both have been designed to handle large-scale stream ingestion driven by real-time events. Clemens Vasters, messaging architect at Microsoft, stated in the blog post on the announcement:
Conceptually, both are a distributed, partitioned, and replicated commit log service. Both use partitioned consumer model offering massive scalability for concurrent consumers. Both use a client-side cursor concept and scale very high workloads.
A typical integration of a Kafka client application with Event Hubs is through enabling a Kafka endpoint while creating an Event Hub namespace.
Subsequently, on the Kafka client, a developer will need to change the configuration by switching the parameter to the Kafka endpoint - the fully qualified name of the event hub namespace. Furthermore, the developer needs to set the security mechanism to PLAIN and the event hub connection string as a password.
bootstrap.servers={YOUR.EVENTHUBS.FQDN}:9093
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="$ConnectionString" password="{YOUR.EVENTHUBS.CONNECTION.STRING}";
Note that the public preview of Azure Event Hubs for Kafka Ecosystem supports Apache Kafka 1.0 and later. However, the preview has limitations with the lacks of support for Kafka features like an Idempotent producer, HTTP Kafka API support, and Kafka Streams.
By supporting a Kafka endpoint on Azure Event Hubs, Microsoft has put a facade to mimic the Kafka capability. Richard Seroter, senior director of product for Pivotal, wrote in a recent blog post about the Kafka interface on top of Azure Event Hubs:
Azure Cosmos DB has its own native interface, but also ones that mimic MongoDB and Apache Cassandra. Azure Event Hubs got into the action by recently adding an Apache Kafka interface. If you have apps or tools that use those interfaces, it's much easier to adopt the Azure equivalents now. Azure is not offering MongoDB, Cassandra, or Kafka, but their first party services resemble them enough that you do not have to change the code to get the benefits of Azure's global scale. Good strategy.
The Kafka support for Event Hubs is currently only available in the East- and West US region and pricing details for Event Hubs are on the pricing page. Furthermore, more information around the Kafka support is available on the Microsoft docs – Event Hubs for Kafka Ecosystems.
