Amazon Web Services (AWS) recently announced that AWS Step Functions supports synchronous Express workflows to "easily build web-based applications and orchestrate high-volume, short-duration microservices". Express workflows, a cost-effective, high-performance workflow type with a runtime of up to five minutes, can now be synchronously invoked from Amazon API Gateway or directly via the API.
AWS Step Functions is a serverless workflow orchestration engine that allows to "sequence AWS Lambda functions and multiple AWS services into business-critical applications". Step Functions provides two distinct workflow types: Standard workflows can run for up to one year with exactly-once execution guarantees, which targets potentially long-running and auditable workflows. In contrast, Express workflows can run for up to five minutes with at-least-once execution semantics at a significantly higher execution rate and respectively lower pricing, which makes them "ideal for high-event-rate workloads".
The Step Functions API had previously only offered asynchronous workflow execution, which necessitated custom code and polling logic when microservices required an immediate response. To simplify respective architectures, Express workflows can now also be executed synchronously so that clients "receive a response without needing to poll additional services". This aims at "high-volume microservice orchestration and fast compute tasks that communicate via HTTPS", which Amazon API Gateway supports via the new HTTP API StepFunctions-StartSyncExecution service integration.
In the introductory blog post, AWS Developer Advocate Benjamin Smith walks through a comprehensive synchronous Express workflow example that analyses the sentiment of a web form submission by orchestrating several AWS services to respond with a customer-centric message:
/filters:no_upscale()/news/2021/01/synchronous-aws-step-functions/en/resources/AWS-Step-Functions-synchronous-workflow-example-1612123887165.png)
Image: Web form submission with synchronous sentiment scoring (via AWS blog post)
The example illustrates how a regular HTTP POST request (1) synchronously triggers a potentially complex Step Functions workflow with arbitrary steps and service integrations (2-5) via an API Gateway HTTP API (6), which then returns the workflow result to the web client immediately through the HTTP POST response (7).
The AWS team has also launched additional service integrations with AWS Step Functions. Not to be confused with abovementioned Amazon API Gateway service integration to call Step Functions workflows, Step Functions can in turn now orchestrate HTTP and REST APIs that are acting "as the ‘front door’ for business logic" running on other services like AWS Lambda or the Amazon Elastic Container Service. As outlined in more detail in Benjamin Smith's sibling introductory blog post, supported integration patterns are 'Request Response' and 'Wait for Callback' (previous coverage).
The ability to optionally authenticate and authorize Step Functions workflow calls to API Gateway resources protected via resource or IAM policies has been well received by the AWS community, with Ben Kehoe summarizing that "fully-managed authorization is the best authorization". However, Yan Cui tweeted his irritation about the inability to call endpoints outside of AWS:
It's great to see Step Functions adding direction integration with API Gateway, but I don't understand why it's limited to API Gateway endpoints and not "any HTTP endpoint"? [...] Am I missing something here?
In addition, Step Functions now integrates with the Amazon Elastic Kubernetes Service (EKS) to enable the orchestration of both Kubernetes jobs and EKS cluster operations, which implies the ability to create "serverless workflows that instantiate short-lived Amazon EKS clusters to execute jobs". Similarly, the new support for AWS Glue DataBrew jobs in conjunction with the previously released AWS Athena integration enables the orchestration of "end-to-end ETL pipelines" based on DataBrew jobs and Athena queries.
In related news, AWS Step Functions meanwhile supports larger state payloads up to 256 kilobytes, allows to view maps and timelines of a workflow's underlying components via an integration with AWS X-Ray, and also visualizes associated workflows within the AWS Lambda console. Earlier last year, the Amazon State Language (ASL) has received significant enhancements with updates to the ‘choice’ state, global access to the context object, dynamic timeouts, result selection, and intrinsic functions, all of which can further simplify workflows and reduce cost by requiring fewer Lambda calls and state transitions.
Microsoft's Azure Logic Apps service also provides the ability to respond synchronously to inbound HTTPS requests, with a timeout of up to four minutes depending on the service version and environment.
The AWS Step Functions documentation features a developer guide, including a section on service integrations, the Amazon States Language (ASL), and the API reference. The AWS CLI Step Functions commands are documented separately. Support is provided via the AWS Step Functions forum. Synchronous Express workflow execution is provided at no additional charge beyond the regular usage-based Step Functions pricing.
