Model commands

Feature set: Inference Contact our support for access.
This functionality evolves quickly, the behavior and APIs might change between releases without further notice.

The akka models commands deploy models to an inference cluster, expose them on a hostname, and record what an organization sanctions. See Serving models for the tasks these commands support, and Model descriptor for the file they apply.

Every command supports --help. Most support -o json for output consumed by another program. validate and export accept the flag and ignore it, and always print their usual format.

Model deployments

A deployment is one serving engine running one model on an accelerator.

Command Description

akka models list

Every deployment, with the model it serves, its KV token budget, its concurrency estimate, and whether it is ready.

akka models get NAME

One deployment. When a model is not ready, this prints the reason in plain text.

akka models get NAME --wait

Wait for a deployment to become ready. The first deployment downloads the weights, so this takes several minutes.

akka models validate -f FILE

Check a descriptor offline. Reports every problem in the file at once and deploys nothing.

akka models apply -f FILE

Create or update everything the descriptor names. Safe to repeat: it updates what changed and leaves the rest alone.

akka models apply -f FILE --dry-run

Run the checks that need the cluster, without persisting anything.

akka models delete -f FILE

Remove what the descriptor names, routes first. Cached weights survive.

akka models delete -f FILE --include-cache

Remove what the descriptor names and the cached weights with it.

akka models delete NAME

Delete a single deployment by name, without a descriptor.

akka models export

Write everything running back out as a descriptor, routes included. Secret values are masked with NOT EXPORTED.

list reports MAX CONC, the number of concurrent requests the KV cache comfortably holds at the configured context length. It is a sizing estimate rather than a limit, and nothing enforces it.

Proxying to a deployment

Command Description

akka models proxy NAME

Open a local tunnel to one deployment on localhost:8080, bypassing the route and its API keys.

The tunnel reaches one model, so the model field of a request body is echoed back rather than used to choose between models.

Devices

A device is a kind of GPU the platform has in a region. Devices are inventory and cannot be created from a descriptor or from the CLI.

Command Description

akka models devices list

Every kind of GPU available in the region, with its memory, architecture, and how many cards are free.

akka models devices get NAME

The individual cards behind one device name, and what is running on each.

Accelerators

An accelerator claims a slice of the device inventory. A deployment is placed on an accelerator, never on a device.

Command Description

akka models accelerators list

Every accelerator, its device, its tenancy, and its free slots.

akka models accelerators get NAME

One accelerator, stating the hardware, the tenancy, and the free slots in one line.

akka models accelerators create NAME --device DEVICE

Create an accelerator with dedicated tenancy.

akka models accelerators create NAME --device DEVICE --tenancy Shared --max-models N

Create an accelerator that splits one device between N models.

akka models accelerators delete NAME

Remove an accelerator, releasing the capacity it held.

accelerators create does not check for free hardware. Run it against a fully allocated card and it succeeds, reporting READY True with CAN PLACE 0: an accelerator that looks healthy and can never place a model. Check CAN PLACE, not READY, before deploying against a new accelerator.

An accelerator is immutable once it exists. Applying a descriptor leaves an existing accelerator alone rather than reshaping it.

Routes

A route gives a set of deployments one hostname. Clients choose between the models on it with the model field of the request body.

Command Description

akka models routes list

Every hostname, the models on it, and whether it requires a key.

akka models routes get NAME

One route: its endpoint, the model names it accepts, the state of each deployment, and why a route is not ready.

routes get reports Accepts, the list of names clients may send as model. Adapter names appear there alongside the base name of the deployment that serves them.

API keys

A route refuses every request until a key authorizes it. Keys are issued per client.

Command Description

akka models routes keys add ROUTE CLIENT

Issue a key for one client. The value is shown once and cannot be read back.

akka models routes keys list ROUTE

Which clients hold a key. Never the keys themselves.

akka models routes keys remove ROUTE CLIENT

Revoke one client’s key.

Deleting a route does not revoke the keys issued against it. Rebuild the route under the same name and every key ever issued to it works again. A key is only gone once routes keys remove has removed it.

Model governance

The same akka models noun records which models an organization sanctions and the conditions attached to them. What a model is and what is running are the same resource seen twice, so there is no separate registry listing: list and get, described in Model deployments, read that resource, and the verbs below act on the same names.

Command Description

akka models apply NAME -f FILE

Register a model or update its metadata.

akka models promote NAME

Promote a candidate to active.

akka models deprecate NAME

Discourage new use of a model.

akka models retire NAME

End the sanction to use a model.

akka models resolve NAME --action ACTION

Resolve a decision on the model that owns it.

akka models signoff NAME --condition C

Record human sign-off closing a risk condition.

akka models waive NAME --condition C

Record a time-bound waiver on an open condition.

akka models strike NAME --condition C

Turn off a condition that does not apply.

akka models conditions list NAME

Every condition on a model and its state.

akka models attestations list NAME

Every attestation recorded against a model.

akka models attestations create NAME -f FILE

Record an externally-run evaluation or report.

akka models actions list NAME

What may be decided about this model, and how guarded each choice is.

apply takes two forms, told apart by whether a name precedes the file. apply -f FILE applies a descriptor and creates or updates everything it declares. apply NAME -f FILE registers metadata against one named model.

The singular alias model is accepted.

 

The features described in this section are an add-on to Akka Automated Operations. They are not included in the base product.

See also