Deploying models

Feature set: Inference Contact our support for access.

A model is one checkpoint served by an engine on a device of an allocation. You declare the model in a model descriptor and apply the file, which downloads the weights and starts the engine.

Declaring a model

A model names the checkpoint to serve, the engine, and the allocation entry and device that it runs on:

resource: Model
resourceVersion: v1
metadata:
  name: assistant (1)
spec:
  model: mistralai/Ministral-3-3B-Instruct-2512 (2)
  source:
    type: huggingface
    huggingface:
      secretRef: hf-token (3)
      revision: COMMIT_SHA (4)
  engine:
    type: vllm
    vllm:
      checkpointFormat: mistral
  placement:
    allocation: gpu (5)
    device: 0
  serving:
    contextLength: 8192 (6)

1	The model name. Clients send this name in the `model` field of a request.
2	The checkpoint to serve, passed to the engine as it is written.
3	The project secret that holds the token for gated weights. See Supplying a token for gated weights.
4	The full 40-character commit SHA of the checkpoint. A branch or tag name is refused, because it can change under a running model.
5	The name of an allocation entry. See Allocating hardware to models.
6	The maximum number of tokens per sequence, prompt and completion together. Without this field, the engine uses the checkpoint’s own maximum, which can exceed the device memory.

spec.model, spec.source, spec.engine, and spec.placement are required. Only huggingface is a supported source and only vllm is a supported engine. See Model for every field.

By default a model generates text. Set spec.task to embed to serve an embedding model. A model runs one replica, and the descriptor does not accept a higher count.

Tuning the engine

The spec.engine.vllm block decides how the model is served. Every field is optional. A field that you leave out, or set to auto, keeps the engine default.

Field What it does

Field	What it does
`dtype`	The data type of the weights and activations: `auto`, `bfloat16`, `float16`, or `float32`.
`quantization`	The weight quantization method that the engine applies or expects. Leave it out to let the engine detect a pre-quantized checkpoint from its configuration. Values include `fp8`, `awq`, `gptq`, `bitsandbytes`, and `gguf`.
`kvCacheDtype`	The data type of the KV cache: `auto`, `fp8`, `fp8_e4m3`, or `fp8_e5m2`.
`checkpointFormat`	The checkpoint layout: `hf` for transformers, or `mistral` for consolidated Mistral-format weights, tokenizer, and configuration. Use `mistral` for a checkpoint in that layout.
`chatTemplateRef`	The project secret whose `template` key holds the chat template, for fine-tuned checkpoints.
`extraArgs`	Arguments appended to the engine command line, last and verbatim. The platform refuses flags that it owns, such as the model, port, served name, memory fraction, adapters, and parallelism.

dtype

The data type of the weights and activations: auto, bfloat16, float16, or float32.

quantization

The weight quantization method that the engine applies or expects. Leave it out to let the engine detect a pre-quantized checkpoint from its configuration. Values include fp8, awq, gptq, bitsandbytes, and gguf.

kvCacheDtype

The data type of the KV cache: auto, fp8, fp8_e4m3, or fp8_e5m2.

checkpointFormat

The checkpoint layout: hf for transformers, or mistral for consolidated Mistral-format weights, tokenizer, and configuration. Use mistral for a checkpoint in that layout.

chatTemplateRef

The project secret whose template key holds the chat template, for fine-tuned checkpoints.

extraArgs

Arguments appended to the engine command line, last and verbatim. The platform refuses flags that it owns, such as the model, port, served name, memory fraction, adapters, and parallelism.

spec.serving bounds the serving envelope in terms that do not depend on the engine: contextLength, maxConcurrentRequests, and the CPU and memory resources of the engine container. Always set contextLength, because some checkpoints declare a maximum that no single device can hold.

Enabling tool calling

Set toolCalling.parser to the parser that matches the model family:

engine:
  type: vllm
  vllm:
    checkpointFormat: mistral
    toolCalling:
      parser: mistral

The block enables OpenAI tool calls with the named parser. The region refuses a parser name that the engine does not support. It cannot check that the parser matches the model, so choose it deliberately. A wrong parser starts healthy and returns tool calls as plain text.

Supplying a token for gated weights

A checkpoint behind an account needs a token. Store the token in a project secret under the key token, and name the secret in source.huggingface.secretRef:

akka secrets create generic hf-token --literal token=HF_TOKEN

Replace HF_TOKEN with your token. A model descriptor cannot contain a Secret resource, so the descriptor holds only the name of the secret and no credential.

Persisting the weights

Set source.cache.sizeGiB to keep the weights across restarts. Without a cache, every restart pays the download again.

source:
  type: huggingface
  huggingface:
    revision: COMMIT_SHA
  cache:
    sizeGiB: 100

Sharing a device between models

placement.device is the index of a device within the allocation entry. Models that give the same entry and index share the physical device. placement.memoryGiB is the engine’s device-memory budget. Without it, a model takes the whole device less the type’s headroom, so set it on every model that shares a device.

placement:
  allocation: gpu
  device: 0
  memoryGiB: 12

Serving LoRA adapters

A LoRA adapter is served next to the base model and addressed by its own name. Each adapter states its name, its checkpoint, where the weights come from, and its LoRA rank:

spec:
  loraAdapters:
    - name: docs-support (1)
      model: acme/support-adapter
      rank: 16 (2)
      source:
        type: huggingface
        huggingface:
          revision: COMMIT_SHA

1	The name that clients send in the `model` field to reach the adapter. It must be unique across the project’s routing table, and the region refuses a duplicate.
2	The rank is required. The engine computes its maximum rank from the declared ranks.

An adapter can also come from object storage. Set source.type to objectStorage and give the logical name of a bucket of the project, the object path of the adapter directory, and the sha256 of adapter_model.safetensors. The fetch fails when the weights do not match the digest.

The set of adapters is static. Changing it restarts the engine.

Applying the descriptor

Run the region’s admission checks without persisting anything:

akka models apply -f models.yaml --dry-run

The CLI checks the shape of every document first and reports every problem at once. The region then checks, among other things, whether the requested devices fit the pool.

Apply the descriptor:

akka models apply -f models.yaml

apply creates or updates each resource by name, so it is safe to run again after you edit the file. It applies the allocations first, then the models, then the endpoints.

Waiting for a model to become ready

akka models get assistant --wait

The first deployment pulls the engine image and the checkpoint, so several minutes are normal. --wait waits for 10 minutes by default, and --wait-timeout sets a different time. When the model is not ready, akka models get prints the reason from the Ready condition. A model that is still pulling weights, a device that does not bind, and a revoked entitlement each name themselves.

To list the models in the region:

akka models list

Column Value

Column	Value
`NAME`	The model name.
`MODEL`	The checkpoint.
`SERVES`	The names that clients can send in the `model` field, adapters included, or `-` while there are none.
`ALLOCATION`	The allocation entry that the model is placed on.
`READY`	`True` when the model is serving.
`AGE`	How long the model has existed.

NAME

The model name.

MODEL

The checkpoint.

SERVES

The names that clients can send in the model field, adapters included, or - while there are none.

ALLOCATION

The allocation entry that the model is placed on.

READY

True when the model is serving.

AGE

How long the model has existed.

Removing models

Tearing down what a descriptor created releases the hardware it held. Deletion runs in the reverse of apply order, so no endpoint is left pointing at a model that has already gone.

Removing the route

If a project route sends a hostname to the endpoint, delete the route first:

akka routes delete assistant-route

Removing what a descriptor names

akka models delete -f models.yaml

The command lists the resources that it will delete and asks for confirmation. Pass --force to delete without confirmation.

Endpoints are removed first, then models, then allocations. A resource that is already absent is skipped, so the command is safe to repeat.

Removing a single resource

akka models delete deletes the resources that a descriptor names, so to delete one resource, pass a file that names only that resource by resource, resourceVersion, and metadata.name:

resource: Model
resourceVersion: v1
metadata:
  name: assistant

Deleting a model releases the device capacity that it used. The allocation stays, and its devices become available to another model. To release the devices too, delete the allocation. See Releasing capacity.

Secrets survive teardown

akka models delete never deletes secrets:

The key secret of an endpoint is left alone, because keys are handed to clients and destroying them is a separate decision. Rebuilding an endpoint that names the same secret makes every key in it work again. Remove keys with akka models endpoints keys remove, described in Issuing API keys.
The token secret for gated weights stays until you delete it.

Delete a secret that you no longer need:

akka secrets delete assistant-keys
akka secrets delete hf-token

Next steps

A deployed model is not yet reachable from outside the project. See Exposing and calling models.

The features described in this section are an add-on to Akka Automated Operations. They are not included in the base product.