Removing a Hidden Round Trip from a Multi-Region AWS API

Until recently, every new client session to our AWS globally routed service started with a request whose only purpose was to figure out which AWS region to authenticate against. We carried that workaround for years before removing it — and the push to do so didn't come from latency concerns. It came from a series of regional outages.

The constraint we built around

This article is related to a user-settings service that my team and I run inside AWS and is used internally by other AWS teams that need fast, low-latency access to per-user settings. Our service stores per-user state that gets accessed at application startup - think of these as settings that need to load fast, from wherever the user is. The service architecture was: deploy API Gateway (APIGW) REST APIs in two regions, wire up Route 53 latency-based routing, and let the infrastructure decide where traffic lands.

We faced a constraint when we integrated AWS Identity and Access Management (IAM) authentication. We needed to use IAM authentication in APIGW since we were storing settings on a per-identity basis. At that time, the only option was to use AWS Signature Version 4 (SigV4).

SigV4, the authentication scheme used for IAM-secured API Gateway requests, binds each signed request to a specific service and region. The region is embedded in the credential scope during the signing process, which means a request signed for us-west-2 (Oregon) is cryptographically invalid if it arrives at eu-west-1 (Ireland). The signature check fails before the request gets processed.

This region-pinning constraint resulted in a tradeoff: we wanted infrastructure (the DNS service Route 53) to decide which region to serve, but the client had to commit to a region before it could even construct a valid request. Therefore, we built our service around this constraint.

We implemented a pre-flight region discovery step as a workaround. The client would first call a lightweight DiscoverRegion endpoint — unauthenticated, since its only job was to return a region name — learn which region was "closest", cache that result, and then sign all subsequent calls for that region using SigV4. In total, there were two requests to do one operation for the initial request.

This became our production architecture, and it worked well for years. However, a series of events in us-east-1 (N. Virginia) including networking (12/2021), Lambda (06/2023) and DynamoDB (10/2025) directly affected our service. Because we relied on API Gateway, Lambda and DynamoDB, we couldn’t shift traffic away from the impacted region even though the service was architecturally global. Clients that had completed region discovery and signed their requests for us-east-1 were cryptographically bound to it — a SigV4 signature for us-east-1 is invalid if rerouted to us-west-2. The only recovery path was re-running the discovery flow.

These events pushed us to re-architect our settings service for regional resiliency in Q4 2025, and the discovery step - along with the client-side region pinning it required - became an obstacle. Asymmetric Signature Version 4 (SigV4a) had been available for a while (launched in Q3 2021) by then; we simply did not have any reason to reach for it until now.

What the traces showed

Once we started looking at the request flow as part of the global redesign, we noticed the latency impact. At the 90th percentile (p90), the DiscoverEndpoint latency applicable to new client sessions was 75–100ms when the user is in the same region as the discovered endpoint. In the average case (say user is in us-west-2 while the discovered endpoint was us-east-1 – not too far away but also not too close, the DiscoverEndpoint p90 latency increased to ~315ms). In the worst case, say us-west-2 user hits ap-south-1 (Mumbai), the p90 latency was 1s.

While this can be attributed to multiple factors such as client-side roundtrip, inefficient regional routing, slow network connection - this was still a request roundtrip we would rather not have and therefore reduce latency.

This latency overhead was an acceptable compromise when the architecture was stable and there was no obvious alternative other than building a new global service. However, after the outages, once we knew SigV4a could eliminate the discovery step entirely, 100ms on every new session was no longer something we were willing to carry forward into a redesigned global service.

While the latency numbers were the visible part, there were other costs that became clear only when we started designing around them:

Client state. Once a client pinned to a region, it held that selection for the lifetime of the session. If that region became degraded after the discovery call, the client's signed requests would start failing - and retry logic now had to detect this, invalidate the cached region, and re-run discovery before retrying the original call. That failure path was more complex than it looked.

Operational coupling. Designing for global failover meant we needed to shift traffic between regions cleanly. That required updating the discovery endpoint's response and ensuring clients would pick up the new selection in a reasonable time window. The discovery layer, which had been a minor detail during steady-state operation, was now directly in the critical path of the failover design.

Client library complexity. Because we exposed this service to multiple internal callers, we shipped a client library that encapsulated discovery and signing. That library had to manage region cache lifetimes, retry logic, and credential scoping. It was not a small abstraction.

The flow itself looked like this: client calls GET /discover (unauthenticated), gets back a region name like us-west-2, then signs and sends the actual POST /settings request to that region. Two round trips to authenticate. The discovery step doesn't appear in most architecture diagrams – it’s inside the client library but it always adds latency. 

Figure 1: The SigV4 flow — the client first calls the DiscoverRegion endpoint, caches the returned region, then signs and sends the request pinned to that single region. (Image created by author)

What SigV4a changes

AWS Signature Version 4A (SigV4a) changes the scope of a signature. A SigV4a signature is valid for a set of regions rather than a single one. This is possible because SigV4a uses ECDSA-P256 (an asymmetric elliptic-curve signing algorithm) rather than HMAC-SHA256: the asymmetric model allows services to verify a signature without needing the exact region that produced it, as long as the signature is valid for an allowed region set.

As a consequence, the client no longer needs to know the destination region before signing the request. It can sign once, for a declared region set, and send to a global entry point. The global infrastructure — for example Route 53 or Global Accelerator— resolves the actual endpoint. The request remains cryptographically valid regardless of which regional deployment ultimately serves it.

Figure 2: The SigV4a flow — the client signs once for a region set and sends to a global entry point, which routes to the nearest healthy region. (Image created by author)

After the switch, the client signs a single POST /settings request for the region set {us-west-2, eu-west-1} and sends it to the global entry point (https://global.settings.service – a latency based routing endpoint instead of https://region.settings.service). Route 53 resolves the destination. The client never learns which region served the request, and doesn't need to. Alternatively, the global entry point can be a CloudFront destination which brings the requests inside Amazon’s backbone and routes efficiently within their network. Retry logic becomes simpler too — if a request fails, just retry it. The signature is valid for either region, so infrastructure can route the retry elsewhere without the client knowing.

What the migration actually involved

Our previous architecture’s region discovery step was not an oversight. SigV4 was the only available option when we built the service, and the workaround was the right call at the time. The migration was not fixing a mistake. Instead, it was taking advantage of a capability that hadn't existed when the original design was made.

The signing change itself was small. The key difference at the library level is that instead of committing to a single region string at signing time, you declare a region set:

// SigV4:  sign for one region -- must match the destination
sign (request, region= "us-west-2")

// SigV4a: sign for a set -- infrastructure resolves the destination
sign (request, regionSet= ["us-west-2", "eu-west-1"])

The region set is just a list - it can be as narrow as two adjacent regions for active-passive failover, or scoped to a single geography (ex: all EU regions) to satisfy data residency requirements. It also supports wildcards, for example, X-Amz-Region-Set=us-west-*, where a request can be made in us-west-1 (San Francisco) or us-west-2 (Oregon). The SDK call is otherwise identical in structure. If your signing logic is encapsulated in a client library (ours was), the diff is localized to one place.

The harder part was the rollout, not the code. Because multiple internal teams depended on our client library, we could not flip a switch: we stood up a new SigV4a-compatible endpoint alongside the existing SigV4 one. Creating a new endpoint wasn't strictly required by SigV4a itself, but it made the migration easier to reason about: the newer design collapsed region discovery and manual routing down to a single global entry point, gave us a cleaner DNS layout for future work like IPv6, and aligned with a broader security-driven infrastructure change where new API Gateways were deployed in different AWS accounts.

Both endpoints ran in parallel. We released a new major version of the client library that used SigV4a signing and pointed at the new endpoint, and ran a migration campaign: announcements in internal channels, direct outreach to teams with high call volume, and a deprecation timeline communicated well in advance. The parallel period ran for about three months, and we tracked adoption by library version through our service metrics — watching the ratio of SigV4 to SigV4a requests shift as teams updated.

The rollout issues were process-oriented, not algorithmic.

First, some customer environments had corporate firewalls or network controls that allowlisted the old domain but not the new one. Requests to the new endpoint were silently dropped until those allowlists were updated.

Second, not every team was using our vendored SDK client. Some had implemented custom signing logic and needed direct outreach — updating a library dependency wasn't enough.

Third, SDK versioning became a practical constraint. Some clients were on older SDK versions that couldn't support SigV4a without first doing unrelated refactoring to get onto a newer version.

When traffic to the old endpoint dropped to a negligible level, and remaining callers had confirmed migration plans, we retired it — along with the DiscoverRegion endpoint. End-to-end, the work spanned about six months: roughly three months to build and deploy the SigV4a endpoint and three more to run the parallel period and complete the caller migration. The technical change was small. The coordination cost was not.

After the migration, we confirmed the discovery round trip was gone. We didn't measure the CPU cost of ECDSA signing versus HMAC — on modern hardware, it's sub-millisecond and not relevant to compare against a network round trip.

When SigV4 is still the right choice

SigV4a is not a universal improvement. It solves a specific problem — dynamic routing where the destination region is not known at signing time. If that problem doesn't exist in your system, SigV4 is a simpler solution.

A hard constraint to check first is that not all AWS services and API types support SigV4a. S3, for example, supports SigV4a but only on Multi-Region Access Points,  not on standard regional endpoints. API Gateway REST APIs with IAM auth support it; HTTP APIs do not. Before evaluating whether SigV4a solves your problem architecturally, verify that the specific service and endpoint type you're using accept SigV4a signatures. 

SigV4a support surface is narrower than you'd expect, and the failure mode when it’s missing is a 403 that gives you nothing useful to diagnose.

Use SigV4 when:

• The AWS service or API type you're using doesn't support SigV4a.
• Your service is deployed in a single region, or clients always target a known, stable region.
• Regulatory or data residency requirements mean requests must be constrained to a single specific region — SigV4's region-scoping is a feature in these cases, not a limitation.
• Your client tooling doesn't have reliable SigV4a support.

Consider SigV4a when:

• The service you're using explicitly supports it. 
• Clients are doing a pre-flight discovery step purely to satisfy region-scoped authentication.
• You're building for latency-based routing or active-passive failover across regions.
• Client simplicity (removing region-cache state) is a meaningful goal.

What made it worth doing

The discovery step was the right call when we built the service — SigV4 was the only option, and it worked for years. The migration became worth doing when regional outages pushed us to take global failover seriously and the discovery step turned out to be the thing standing in the way.

The core issue was coupling: SigV4's region-scoped model forced clients to make a routing decision that should have belonged to infrastructure. SigV4a removes that constraint. If your clients are running a pre-flight step whose only purpose is to determine which region to sign for, that's the signal. The step exists because of the authentication model, not because it adds value.