Amazon Web Services (AWS) recently announced that Amazon SNS supports Amazon Kinesis Data Firehose subscriptions to send messages to "data lakes, data stores, and analytics services [...] without writing custom code". The new event destination also simplifies the integration of third-party service providers.
Amazon Simple Notification Service (Amazon SNS) is a managed service that provides asynchronous message delivery from publishers to subscribers "for both application-to-application (A2A) and application-to-person (A2P) communication". It supports a wide variety of native AWS event sources to implement high throughput and low latency event-driven architectures (previous coverage).
Amazon Kinesis Data Firehose is a managed service to "prepare and load real-time data streams into data stores and analytics services" without the need to implement anything but an optional serverless data transformation via an AWS Lambda function (previous coverage).
Amazon SNS has previously supported general-purpose A2A event destinations like Amazon SQS, AWS Lambda, and SNS aware HTTP/S endpoints, which required custom solutions for popular destinations with more specific requirements. AWS has now added an Amazon Kinesis Data Firehose event destination that can in turn send records to delivery stream destinations including Amazon S3, Amazon Redshift, Amazon Elasticsearch Service, and "any HTTP endpoint owned by you or any of your third-party service providers". Coupled with the built-in Kinesis Data Firehose data transformation capability, this obsoletes most custom code to "capture, transform, buffer, compress and upload data" en route to these destinations.
/filters:no_upscale()/news/2021/05/amazon-sns-kinesis-data-firehose/en/resources/amazon-sns-delivery-protocols-1622507573564.png)
Image: Amazon SNS event destinations (via AWS documentation)
The introductory blog post outlines several example use cases that are now easier to achieve:
- Store and query SNS messages via Amazon S3 destinations – backup and archive data in S3 buckets, and analyze it with Amazon Athena using SQL
- Index and search SNS messages via Amazon ES destinations – publish data to the Amazon Elasticsearch Service, and explore and visualize it with Kibana
- Analyze SNS messages via Amazon Redshift destinations – publish data to a Redshift data warehouse, and analyze it with Amazon Redshift using SQL
- Forward SNS messages via HTTP destinations – route data to custom endpoints and third-party service providers
- Transform SNS messages via Kinesis Data Firehose transforms – process data with a Lambda function, e.g. to redact personally identifiable information (PII)
The new Kinesis Data Firehose integration supports regular SNS features such as dead-letter queues (DLQs) and message filtering, which allows to "offload the message filtering logic from subscribers and the message routing logic from publishers".
It is worth noting that the event bus based Amazon EventBridge service also provides DLQs, content filtering, input transformation, and has native event archive and replay capabilities (previous coverage). James Beswick, principal developer advocate for AWS Serverless, acknowledges in a blog post on choosing between messaging services that "SNS and EventBridge have many similarities":
Both can be used to decouple publishers and subscribers, filter messages or events, and provide fan-in or fan-out capabilities. However, there are differences in the list of targets and features for each service, and your choice of service depends on the needs of your use-case.
The post then describes the EventBridge exclusive SaaS partner event source and schema registry capabilities and provides a detailed feature comparison table. Beswick also followed up with example use cases combining features of both services in a subsequent blog post on building resilient serverless patterns.
In related news, Amazon SNS recently introduced additional operators for its message filtering capability, meanwhile supports ten-digit long codes and toll-free numbers for sending SMS to US destinations, and also improved Amazon CloudWatch monitoring by publishing metrics at 1-minute resolution.
The Amazon SNS documentation features a developer guide, including a section on message archiving and analytics, the AWS CLI reference, and the API reference. Amazon SNS is subject to usage-based pricing that is primarily based on notification volume and varies by destination. Charges associated with destination services like Amazon SQS, AWS Lambda, and Amazon Kinesis Data Firehose are billed separately. Support is provided via the Amazon SNS forum.
