At re:Invent, AWS introduced Amazon EventBridge Pipes, a new feature in Amazon EventBridge providing developers a more straightforward way to connect events from multiple services.
With Amazon EventBridge Pipes, the company expands the EventBridge offering beyond event buses and scheduling. The new feature allows developers to create point-to-point integrations between event producers and consumers. Werner Vogels, chief technology officer at Amazon, explained the feature during the re:Invent keynote:
So the idea is that you should no longer have to write the glue code because you can easily stitch these services together. And if you would want to actually manipulate the events before they reach the consumer, you can actually provide a lambda function or a point-to-step function or API gateway to actually run some code to manipulate the events that are flowing through your pipe. And it has built-in filtering meaning that if you only want to get a real subset of the events that need to flow to the consumer you can add that too.
Source: https://aws.amazon.com/eventbridge/pipes/
Developers can create pipes by clicking "Pipes" in the Amazon EventBridge console or using the AWS Command Line Interface (CLI), AWS CloudFormation, and AWS Cloud Development Kit (AWS CDK). Next, they choose a source producing events like Amazon DynamoDB, Amazon Kinesis, and Amazon MQ. Then they can optionally specify an event filter to only process events that match the filter. Subsequently, they can also optionally transform and enrich events using built-in free transformations, or AWS Lambda, AWS Step Functions, Amazon API Gateway, or EventBridge API Destinations to perform more advanced transformations and enrichments. And finally, they choose a target destination such as AWS Lambda, Amazon API Gateway, and Amazon SNS.
Ian McKay, a DevOps lead at Kablamo, tweeted:
EventBridge Pipes allows for data pipelines between AWS services such as DDB, SQS, Kinesis, MSK etc., to destinations such as Lambda, Kinesis, and other 3rd parties - including field-level mapping/filter/enrichment support, suspiciously similar to the way AppFlow works.
In addition, Nik Pinski, a principal engineer at AWS, tweeted:
This is something our friends at Lambda solved quite elegantly already with the EventSourceMapping - giving an easy way to run a Function in response to an event source. But some Lambda Function code is also undifferentiated glue to downstream systems.
Amazon EventBridge Pipes are currently generally available in all AWS commercial regions except Asia Pacific (Hyderabad) and Europe (Zurich). Pricing details of the service are available on the pricing page.