Amazon Kinesis Data Streams is a fully-managed, serverless service on AWS for real-time processing of streamed data at a massive scale. Recently, the company released a new capacity mode On-demand for the service, which eliminates capacity provisioning and management for streaming workloads.
With Amazon Kinesis Data Streams, customers can take any amount of data from any number of sources and scale up and down as needed. Earlier, they needed to adjust and define the capacity of the service. Now scaling with the "on-demand capacity" mode is done automatically in response to varying data traffic.
In a recent AWS news blog post on Amazon Kinesis Data Streams, Marcia Villalba, a senior developer advocate for Amazon Web Services, explains when customers can benefit from the On-demand mode:
On-demand mode is great for customers that have an unknown or variable workload, or who simply don’t want to deal with capacity management. On-demand mode works best for workloads that have even partition key distribution. For example, you run a mobile game that has variable traffic through the week or day, as customers play mostly on nights or weekends. Or, you run a streaming platform that hosts live shows, and you see a sudden increase in demand depending on the guests you have.
Users of Amazon Kinesis Data Streams can create a new data stream with Kinesis Data Streams On-demand from the AWS console, and through AWS SDKs, AWS Command Line Interface (CLI), and AWS CloudFormation. When using the console, the capacity mode can be selected in the Create data stream section.
Adding the On-demand mode capacity mode for Amazon Kinesis Data Streams is a response by AWS to its competitors, which offer similar services that also auto-scale. For instance, Azure Event Hubs Microsoft’s streaming service had the auto-inflate feature for some time now responsible for automatic scaling of capacity. And Google’s Pub/Sub streaming service also has auto-scaling and provisioning features.
James Urquhart, global field CTO at VMware, stated in a tweet:
This is the purchasing experience I expect all flow and event processing/queueing services to adopt over time. Pay for throughput.
And Clemens Vasters, principal architect messaging at Microsoft, responded to that a tweet:
Azure Event Hubs Standard and Premium already had auto-scale of Throughout/Processing Units independent of how many partitions you set up. Eventually, everyone gets there.
Currently, Amazon Kinesis Data Streams On-demand is available globally in all commercial regions. Customers are charged per gigabyte of data written, read, and stored in the stream, in a pay-per-throughput fashion – more details are on pricing can be found on the pricing page.
