Recently, Microsoft announced the general availability of Azure Event Hubs for Apache Kafka. Microsoft claim that customers will "get the best of both worlds"—the ecosystem and tools of Kafka, along with Azure’s security and global scale. Furthermore, by bringing together two powerful distributed streaming platforms, customers can have access to the breadth of Kafka ecosystem applications without having to manage servers or networks.
At the beginning of June, InfoQ reported on the preview of the integration of Kafka with Azure Event Hubs as the engineering team responsible for Azure Event Hubs made a Kafka endpoint available for users of their service to stream event data into it. Now the integration is ready for general availability, and the team have added a new capability that enables customers to start streaming events from applications using the Kafka protocol directly into Event Hubs by merely changing a connection string. Moreover, customers can keep using their existing Kafka applications, frameworks, and tools to talk to Event Hubs.
With EventHubs for Kafka, customers can start streaming events from applications using the Kafka protocol directly into Event Hubs. Microsoft manages the service, and the customer does not need to run Zookeeper, or control or configure any clusters. Shubha Vijayasarathy, program manager, Azure Service Bus, said in a recent Azure Friday episode:
The biggest benefit of using Azure Event Hubs for Apache Kafka is that you're not changing any of your clients, you're not changing your code, your producers, your applications, your tools, or your frameworks.
Customers can use Kafka Connect or MirrorMaker to talk to Event Hubs without changing a line of code.
Other benefits of the integration of Kafka with Azure Event Hubs are according to the announcement blog:
- Leverage Event Hubs to send data to Blob storage or Data Lake Storage for long-term retention or micro-batch processing with Event Hubs Capture.
- Scale from streaming megabytes of data to terabytes while keeping control over when and how much to scale with Auto-Inflate.
- Support of Event Hubs for Geo Disaster-Recovery.
- Event Hubs are deeply-integrated with other Azure services like Azure Databricks, Azure Stream Analytics, and Azure Functions. Therefore, customers can unlock further analytics and processing.
EventHubs for Kafka provides support for Apache Kafka 1.0 and later – the engineering team has mapped the Apache Kafka Protocol to the native AMQP 1.0 protocol. This protocol translation allows other AMQP 1.0 based applications to communicate with Kafka applications. Also, JMS based applications can use Apache Qpid to send data to Kafka based consumers.
Both Azure EventHubs and Kafka are designed to handle large-scale, real-time stream ingestion and are primarily distributed, partitioned, and replicated commit log services. Furthermore, both are using a partitioned consumer model with a client-side cursor concept that provides horizontal scalability for demanding workloads. By adding an endpoint for Kafka, Event Hub is effectively mimicking Kafka. In the Azure Friday episode, Shubha Vijayasarathy explained the conceptual architecture of both:
In the conceptual architecture of event hubs, you have an Event Hub or Kafka Topic; both are very similar. The way Event Hubs or Kafka get data is that they both have both partitions in their architecture, where the data gets distributed. You have a bunch of event producers, and a bunch of event receivers who are interested in this data and the data gets distributed within Event Hubs or Kafka topics in a uniform pattern.
Source (screenshot): https://channel9.msdn.com/Shows/Azure-Friday/Azure-Event-Hubs-for-Apache-Kafka
More details on Azure Event Hubs for Apache Kafka is available on the documentation website and pricing details on the pricing page.