Microsoft has released a reference architecture for routing agent traffic on Azure Kubernetes Service. It breaks down the issue into three key choices: which model answers a call, how the call is managed, and which GPU replica handles it. The design combines the Kubernetes Gateway API Inference Extension for load balancing, agentgateway as an AI proxy, and RouteLLM for semantic routing. All three connect into one OpenAI-compatible endpoint.
The motivation is agentic workloads specifically, not chat. A single agent task can fire hundreds of LLM calls in a plan-act-observe loop, and most of those calls (a tool-argument fill, a yes/no gate, a summary) don't need a frontier model. Sending every call to a top-tier model increases costs and delays based on loop length. A simple round-robin load balancer makes this even worse. It might queue a 200-token completion behind a 100k-token prefill on a busy GPU pod, while another idle pod sits unused nearby.
The suggested architecture divides issues exactly by signal. RouteLLM checks the prompt and predicts if a cheaper model can match a stronger model's answer quality. It uses a matrix-factorization router trained on human-preference data. Agentgateway is an open-source proxy that works with OpenAI. It manages policies like authentication, rate limits for each agent, cost tracking, and guardrails. It does all this without checking the meaning of prompts. The Gateway API Inference Extension's Endpoint Picker checks the live GPU state. It looks at vLLM's KV-cache occupancy and queue depth. This helps it decide which replica of the chosen model will handle the request. Agentgateway calls the Endpoint Picker directly over ext-proc for the self-hosted path, allowing the design to bypass a separate Gateway API gateway entirely.
KAITO provides GPU node pools as needed and runs vLLM. It shows metrics like vllm:num_requests_waiting and vllm:kv_cache_usage_perc, which the Endpoint Picker uses. The strong path goes to Azure OpenAI via an AI backend in agentgateway. The weak path routes to KAITO-served pods through a service backend. This backend uses an inferenceRouting policy. It directs placement to the Endpoint Picker, and the destinationMode is set to passthrough. Azure Managed Prometheus and Grafana scrape both agentgateway's routing and cost metrics and vLLM's GPU metrics for a combined view.
The load-bearing number in the whole setup is RouteLLM's escalation threshold. On the pairing RouteLLM tested, the mf router achieved about 95% of GPT-4's MT-Bench quality. It only sent around 26% of calls to GPT-4, saving up to 85% in costs compared to routing all calls to the strong model. Microsoft states that this figure doesn’t apply automatically. It’s tied to the model pair used for training RouteLLM, not to a phi-4-mini/GPT-5.1 pairing. So, the users need to calibrate the threshold against actual traffic and adjust it based on agentgateway's strong/weak split, instead of RouteLLM’s estimate. The post highlights that prompt caching makes token costs tricky. A cached input token gets a discount, and switching models cools both caches. This means the true cost of a "strong" call is lower than it appears.
The author warns that the pieces used in this post are young:
Several pieces here are young, and field names drift between releases — the Inference Extension renamed and restructured CRDs on its way to v1. Everything below was validated end-to-end on AKS in mid-2026 against Inference Extension v1.0.0 and agentgateway v1.3.1. Treat the manifests as the shape of the solution, pin your versions, and confirm fields against the docs at the end. The decomposition is stable; specific flags and CRD fields move quickly.
For example, InferencePool and InferenceObjective are in different API groups. All three open-source layers operate within the AKS cluster. Meanwhile, KAITO serving, Azure OpenAI, and the Prometheus/Grafana observability stack are managed by Azure. Microsoft notes that its Foundry model router is a managed version of RouteLLM's semantic layer. This helps teams that prefer not to manage a router themselves. However, there is currently no managed option for the Endpoint Picker's GPU-aware placement. It must run in-cluster, no matter which gateway is used.
Teams adopting this can take it in slices rather than all at once. A single hosted model behind a handful of agents mainly needs agentgateway for governance, with nothing to route to and no GPUs to place against. Self-hosting a single model class benefits from KAITO and the Inference Extension without any semantic routing layer, since there's only one model to choose. RouteLLM is worth adding when a strong model has a clear price gap over a weak one, and there's a good amount of easy traffic. The post suggests this applies to nearly any agent running in a loop.