Azure Event Hubs is Microsoft’s managed real-time event ingestion service designed to serve demanding big data streaming and event ingestion needs in the Cloud. Microsoft announced the public preview of Event Hubs Premium during the annual Build conference as a new product SKU tailor-made for high-end event streaming scenarios requiring elastic, superior performance with predictable latency.
Source: https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-about
The new premium offering of Event Hubs comes with a multi-tenant offering with resource isolation, enabling more predictable high-throughput and low latency workloads while providing higher limits and extending support to more Apache Kafka workloads. According to a Microsoft Tech Community blog post by Kasun Indrasiri, senior program manager at Microsoft Azure Messaging Services, the premium Event Hub comes with reserved compute, memory, and storage resources - increasing performance and minimizing cross-tenant interference in a managed multi-tenant PaaS environment. Moreover, Clemens Vasters, principal architect, Messaging Services and Standards at Microsoft, provides InfoQ with some more details:
Event Hubs Premium is a compromise that gives you single-tenant-like, isolated compute and memory resources and reserved I/O capacity but runs inside a multi-tenant cluster. The multi-tenant cluster model has the advantage that we can scale up and down the capacity of any tenant flexibly within seconds and that the substantial reserve capacity required for catastrophic zone failures is easily shared tenants - and thus dramatically reducing cost compared to the Event Hubs Dedicated model.
We also asked Vasters how the new Premium SKU originated:
Event Hubs Premium results from more than two years of internal rearchitecting of the Azure Event Hubs backend, adding a brand-new storage service layer between Azure Storage, still used to keep long-term data, and the Event Hubs broker and protocol gateway logic. This new layer, called "local store" , is a brand-new, native-code log engine, tailored to the Azure environment and deeply integrated with Azure Service Fabric. We can replicate and disk-flush event data across Azure availability zones in less than a millisecond with this new engine. With all the extra broker and protocol added on top, we can accept and acknowledge events from a sender in under five milliseconds and flow them to a receiver in a total end-to-end time of under ten milliseconds.
In addition, he said:
With availability-zone distributed, flushed-to-disk persistence, that’s industry-leading performance, and we are optimistic that we know where to shave off some more time.
Furthermore, Vasters also mentioned another significant aspect:
The second significant aspect of the rearchitecture of Event Hubs is that we’ve taken the proven process/memory isolation model from Service Bus Premium and adapted it for Event Hubs. In Event Hubs Standard, a "throughput unit" is a throttling allowance you buy to use a slice of a large, multi-tenant cluster. In Event Hubs Dedicated, you rent fully isolated, single-tenant cluster capacity similar to how we run Event Hubs Standard ourselves, whereby 8 "capacity units" are the smallest size at which we operate across availability zones.
With Azure Event Hubs, Microsoft competes with other event ingestion services in the Cloud like Confluent with their managed Apache Kafka platform Confluent Cloud and AWS with managed Kafka service MSK. Concerning that, Vasters told InfoQ:
With Event Hubs Premium supporting AMQP as well as the Apache Kafka producer - and consumer APIs and with the latency advantage we offer, Event Hubs is a serious alternative to running a log broker (including Apache Kafka) yourself, even if your workload is not inside of Azure. We don’t say that aloud much, but outbound data transfer charges are not a factor with Event Hubs. If you compare Event Hubs against other PaaS services in the same category, we suggest that you take a very serious look at the cost savings potential.
Customers can currently try out Event Hubs Premium in the Azure Portal. The Premium SKU is now available in the Australia East, Canada Central, East US, North Europe, Southeast Asia, West Europe, and West US 2 regions – and the pricing details of Event Hubs are available on the pricing page. Lastly, Vasters said:
Event Hubs Premium, now in public preview and expected to GA before the end of the year. Event Hubs Premium is the launch vehicle for a new generation of storage backends, and you will see most of our other messaging services accelerate as well over the course of the next year or so.
