Inference

An inference call differs from a function call: the result varies, arrives as text that the calling code must validate, can take minutes, and is billed per token. An agentic system makes several inference calls for each request, so latency, cost and the chance of a failed call grow with the depth and width of the call chain. See Models and inference for these properties and for how Akka observes, routes, trains and serves model traffic as one loop.

Serving a model means running it on hardware in your project and exposing it on a hostname that OpenAI-compatible clients can call. Akka runs the model, allocates the hardware it needs, and provisions a TLS certificate for the hostname.

You describe what you want to run in a model descriptor, a YAML file that declares three kinds of resource. Applying that file is what creates them.

Applying a descriptor writes these in dependency order, so nothing refers to a resource that does not exist yet: allocations, then models, then endpoints. See Model descriptor for every field of every resource.

A model descriptor does not hold secrets or routes. Create the secrets that the models need, such as the token for gated weights and the endpoint’s API keys, with akka secrets and akka models endpoints keys. A project route puts an endpoint on a hostname.

Serving your first model is a tutorial that does all of this in one pass. These pages describe each step in detail:

Serving decides what runs. Routing requests to models decides which of several providers answers a given request, from rules you declare or from the meaning of the request. A model you serve yourself is one destination a route can name. The two are independent: a served model is reachable on its own hostname whether or not any route names it.

Tracking inference spend reports what model traffic in a period consumed and cost, priced from a pricing catalog that you publish. Spend is attributed to systems by the system id that each request names.