Serving models
Serving a model means running it on hardware in your project and exposing it on a hostname that OpenAI-compatible clients can call. Akka runs the model, allocates the hardware it needs, and terminates TLS on a hostname you own.
You describe what you want to run in a model descriptor, a YAML file that declares four kinds of resource. Applying that file is what creates them.
| Resource | What it declares |
|---|---|
|
Credentials the deployment needs, such as the token that downloads gated weights. |
|
A claim on a slice of the GPU inventory available in your region. |
|
One model, the accelerator it runs on, and how the engine serves it. |
|
A hostname, and the deployments reachable on it. |
Applying a descriptor writes these in dependency order, so nothing refers to a resource that does not exist yet: secrets, then accelerators, then model deployments, then routes. See Model descriptor for every field of every resource.
Each resource also has commands that create and inspect it directly, without a descriptor. Use the descriptor for anything you intend to keep, because a descriptor can be version controlled and re-applied. Use the commands to look at what is running and to make a one-off change.
Serving a model for the first time
Work through these pages in order. The result is a model answering requests on a hostname, with a key that authorizes them.
-
Reviewing available hardware shows the GPUs your region has and how much of each is free.
-
Allocating hardware to models claims a slice of that inventory.
-
Deploying models declares the model, applies the descriptor, and waits for the model to become ready.
-
Exposing models on a hostname puts the model behind a URL.
-
Issuing API keys authorizes clients to use that URL.
-
Calling a deployed model sends the first request.
Removing deployments covers exporting what is running and tearing it down.
Serving and routing
Serving decides what runs. Routing requests to models decides which of several providers answers a given request, and a model you serve yourself is one destination a route can name. The two are independent: a served model is reachable on its own hostname whether or not any route names it.
|
The features described in this section are an add-on to Akka Automated Operations. They are not included in the base product. |