Exposing models on a hostname

Feature set: Inference Contact our support for access.
This functionality evolves quickly, the behavior and APIs might change between releases without further notice.

An inference route gives a set of model deployments one hostname. The unit is the hostname, not the deployment: one host serves many models, and clients choose between them with the model field of the request body. A single base URL therefore covers everything you run.

Declaring a route

resource: InferenceRoute
resourceVersion: v1
metadata:
  name: models
spec:
  host: models.acme.example
  models:
    - name: docs
    - name: agent

Each entry names a ModelDeployment. The names clients may send are the deployments' own served names, adapters included, so one entry can expose several names.

Served names do not have to be globally unique. Two projects may both serve a model called agent, because their hostnames differ.

See InferenceRoute for every field.

Reading a route

akka models routes get models
Name:           models
Host:           models.acme.example
Ready:          True
Message:        2 model(s) on models.acme.example
Endpoint:       https://models.acme.example/v1
Own listener:   models
Auth:           api-key
Accepts:        docs, agent

DEPLOYMENT      SERVES   STATE
docs            docs     Serving
agent           agent    Serving

Accepts is the list of names clients may send as model. Endpoint is the base URL to give an OpenAI client. The endpoint is HTTPS with a real certificate, so no client needs special configuration.

To see every hostname, the models on each, and whether each requires a key:

akka models routes list

Changing which models a hostname serves

Add a model to the route in the descriptor and apply the file again to expose it. Remove it and apply again to withdraw it. Exposure resolves continuously rather than only at apply time, so a route applied before its models still works once those models arrive.

When a route is not ready

akka models routes get NAME prints the reason. Two are common.

A route naming a hostname your project does not own applies successfully and then sits at Ready: False with host "…" is not configured on project. The file is not at fault. Registering a hostname is a separate step, with DNS verified and a certificate issued, and the route starts serving as soon as it names a hostname you own.

no exposed model resolves means the route names a deployment that is not deployed. Check that the names in spec.models match your deployments.

A hostname the platform generated needs nothing further. A hostname you supplied carries its own certificate and gets its own listener, and certificate provisioning takes a minute or two after the first apply.

Next steps

The hostname refuses every request until a key authorizes it. See Issuing API keys.

 

The features described in this section are an add-on to Akka Automated Operations. They are not included in the base product.

See also