Routing requests to models

Agents send their model requests to a proxy, such as Agentgateway. The proxy calls the gateway router over gRPC. The router runs the exchange through a pipeline of two stages and returns a destination, and the proxy forwards the request to the model provider named in that decision.

An agent does not address the router and does not name a model. Putting the proxy in the request path is what applies routing to every request, so the set of providers behind it can change without a change to agent code.

The router decides about agentic traffic, such as LLM and MCP calls. It is not a general purpose proxy.

Routing on request attributes runs first. It matches declared rules against named attributes of the request, such as the protocol, the model asked for, or a header.

Routing on request meaning runs second, and only for requests the first stage did not claim. It matches the request against use cases discovered from live traffic, using what the request means rather than a declared rule.

Choose the first stage when the condition can be written down: a tenant header, a protocol, a model name. Choose the second when it cannot, because the thing that decides is what the user is asking for.

A destination is a backend registered with the gateway proxy. Providers behind one gateway carry different characteristics: a general-purpose model offers broad capability at a higher cost per token, and a smaller model returns faster answers to simpler requests. A model you run yourself, described in Serving models, is one destination a route can name.