Model Descriptor reference

Feature set: Inference Contact our support for access.
This functionality evolves quickly, the behavior and APIs might change between releases without further notice.

The model descriptor is the YAML applied by akka models apply to an inference cluster. It declares model deployments, the accelerators they run on, the secrets they need, and the hostnames that expose them. See Serving models for the tasks these resources support.

Two commands check a descriptor without deploying it:

akka models validate -f models.yaml
akka models apply -f models.yaml --dry-run

validate checks the file alone. --dry-run also checks it against the cluster and persists nothing.

Document envelope

A descriptor is a multi-document YAML file. Documents are separated by a line starting with ---, and every document carries the same four top-level fields.

Field Type Description

resource

string required

One of Secret, Accelerator, ModelDeployment, InferenceRoute.

resourceVersion

string required

Must be v1.

metadata.name

string required

Resource name. See Naming rules.

spec

object required

Shape depends on resource.

Specs are parsed strictly. An unrecognized field is an error rather than a value that is silently ignored, so a mistyped field name fails the file instead of quietly changing behavior.

Validation reports every problem it finds, not only the first.

Naming rules

metadata.name, served model names, adapter names, and accelerator references all follow the same rules:

  • lowercase alphanumeric characters and dashes, matching [a-z0-9]([-a-z0-9]*[a-z0-9])?

  • 42 characters maximum

The cap is lower than the DNS label limit of 63 because the operator derives object names from yours by adding suffixes, and those derived names have to fit.

Apply order

Applying a descriptor writes resources in dependency order, so nothing refers to an object that does not exist yet:

  1. Secrets

  2. Accelerators

  3. Cache volume claims, then model deployments

  4. Inference routes

Routes come last because a route names deployments. A route applied before its models still works, because exposure resolves continuously rather than only at apply time.

Apply is authoritative for what it names. An existing resource has its spec set from the descriptor. Labels and annotations added by other tools are preserved, and the values in the descriptor win where both set the same key.

Two resources are never mutated once they exist:

  • Accelerators, whose shape is immutable on the server.

  • Cache volume claims, which may hold a pre-warmed cache. Most of a claim’s spec is immutable in any case.

Secret

Creates a secret on the inference cluster. Values are written as plain strings, which keeps the YAML readable.

resource: Secret
resourceVersion: v1
metadata:
  name: hf-token
spec:
  type: generic
  data:
    token: hf_xxxxxxxxxxxxxxxxxxxx
Field Type Description

spec.type

string required

Only generic is supported on an inference cluster.

spec.data

map required

At least one key. Keys match [-._a-zA-Z0-9]+.

akka models export masks secret values with NOT EXPORTED. Applying an exported file unedited fails validation rather than overwriting a live secret with the mask. Replace the value, or delete the document and create the secret ahead of time.

Accelerator

An accelerator carves the platform’s device inventory into capacity a model deployment can be placed on. Devices are platform inventory, and an accelerator is how a project claims a slice of it. Run akka models devices list to see the devices available in a region.

resource: Accelerator
resourceVersion: v1
metadata:
  name: l4-dedicated
spec:
  device: l4
  tenancy: dedicated
Field Type Description

spec.device

string required

Device kind from platform inventory, for example l4.

spec.tenancy

string

dedicated or shared. Defaults to dedicated.

spec.devicesPerReplica

integer

Devices each replica claims. Defaults to 1.

spec.shared.maxModels

integer

How many models split one device. Required when tenancy is shared, minimum 2.

Dedicated tenancy

Each model gets whole devices. This is the default.

Setting devicesPerReplica above 1 requires a node holding that many devices, and the deployment’s parallelism factors have to multiply to exactly that number.

Shared tenancy

Several models split one device’s memory. Every model placed on a shared accelerator has to state its engine.memoryPercent. A member that does not declare its share is sized against the whole card and runs out of memory after a full weight download.

spec.shared is only valid when tenancy is shared, and maxModels has to be at least 2. One model per device is dedicated tenancy, not shared.

A shared accelerator cannot set devicesPerReplica above 1. It splits one device between models rather than claiming several.

resource: Accelerator
resourceVersion: v1
metadata:
  name: l4-shared
spec:
  device: l4
  tenancy: shared
  shared:
    maxModels: 3

ModelDeployment

One serving engine running one model.

resource: ModelDeployment
resourceVersion: v1
metadata:
  name: agent
spec:
  model: google/gemma-3-4b-it
  servedModelName: agent
  replicas: 1
  placement:
    accelerator: l4-dedicated
  engine:
    maxModelLen: 16384
    memoryPercent: 90
Field Type Description

spec.model

string required

Checkpoint to serve, for example google/gemma-3-4b-it.

spec.servedModelName

string

Name clients send in the OpenAI model field. Defaults to metadata.name.

spec.replicas

integer

Engine count, minimum 1. Each replica needs its own device.

spec.placement.accelerator

string required

Accelerator to run on.

There is no path that lets the scheduler pick the hardware. placement.accelerator is required because the hardware a model runs on is part of how it behaves, and a model that lands on whatever device is free has its performance decided by the scheduler. The first sign of that is a latency change nobody asked for.

spec.engine

Field Type Description

maxModelLen

integer

Maximum context length, prompt and output combined.

memoryPercent

integer

Share of device memory the engine may use, 1 to 100.

quantization

string

none, fp8, awq, gptq, or bitsandbytes.

textOnly

boolean

Drop the vision tower on a multimodal checkpoint.

checkpointFormat

string

auto or mistral.

toolCalling.enabled

boolean

Enable function calling.

toolCalling.parser

string

Parser for the model’s tool-call syntax.

Always set maxModelLen explicitly. Some checkpoints declare a default of 262144, and the engine sizes a KV cache for it.

memoryPercent is required when the deployment is placed on a shared accelerator, where it has to fit inside the model’s share of the device.

textOnly returns the vision tower’s memory to the KV cache. The engine then cannot accept images at all, so it is only correct for a multimodal checkpoint used for text.

checkpointFormat: mistral selects the Mistral-native tokenizer, config, and weight formats. Without it, a Mistral-native checkpoint is read along the standard path and its files are parsed incorrectly. It is never inferred from the model name.

toolCalling.parser is required when enabled is true, and cannot be set when it is false. Valid names come from the serving engine the platform pins rather than from the CLI, so the value is only checked for shape: lowercase alphanumeric with dashes or underscores. Common values are pythonic, mistral, and hermes.

Choose the parser deliberately. The engine does not verify that a parser matches the model, and the wrong one starts healthy and returns tool calls as prose.

spec.parallelism

Field Type Description

tensor

integer

Splits the model across devices on one node.

pipeline

integer

Splits layers, and may cross nodes.

data

integer

Runs independent copies of the tensor-parallel group.

The factors multiply to the number of devices one replica uses. Unset factors count as 1. When the accelerator sets devicesPerReplica above 1, the product has to equal it exactly.

spec.adapters

LoRA adapters served from the base model. Each adapter is addressable as its own model name, so one deployment can answer to several names.

adapters:
  - name: agent-support
    path: s3://acme-adapters/support/
Field Type Description

name

string required

Model name clients use. Must be unique and must not collide with servedModelName.

path

string required

URI with a scheme, for example s3://bucket/path.

Which schemes can be fetched is the operator’s decision, so the CLI only checks that a scheme is present.

spec.source

Where weights come from and how they are cached.

source:
  huggingFaceTokenSecret: hf-token/token
  cache:
    enabled: true
    size: 100Gi
Field Type Description

huggingFaceTokenSecret

string

Reference in secret-name/key form, for gated checkpoints.

cache.enabled

boolean

Persist weights and compiled artifacts across restarts.

cache.size

string

Volume size, required when the cache is enabled.

cache.size is a Kubernetes quantity, for example 100Gi. An ephemeral cache pays the compilation cost on every restart.

The referenced secret has to be declared in the same descriptor or already exist in the namespace. Apply checks this before writing anything.

InferenceRoute

Exposes model deployments on a hostname.

The unit is the hostname, not the deployment. One host serves many models, and clients choose between them with the model field of the request body, so a single base URL works with any OpenAI SDK.

resource: InferenceRoute
resourceVersion: v1
metadata:
  name: acme
spec:
  host: models.acme.example.com
  models:
    - name: agent
    - name: docs
client = OpenAI(base_url="https://models.acme.example.com/v1")
client.chat.completions.create(model="agent", ...)
client.chat.completions.create(model="docs", ...)
Field Type Description

spec.host

string required

Hostname already registered on the project.

spec.models[].name

string required

Name of a ModelDeployment, unique within the route.

Two projects may both serve a model called agent. Their hostnames differ, so served names do not need to be globally unique.

The names clients send are the deployments' own served names, adapters included, so one models entry can answer to several names.

Registering a hostname is a separate step, with DNS verified and a certificate issued. The CLI checks only that spec.host looks like a hostname. Whether the project owns it is the platform’s answer, so a descriptor that validates locally can still be refused on the cluster with host is not configured on project.

A hostname the platform generated needs nothing further. A hostname you supplied carries its own certificate and gets its own listener.

Cross-document rules

Checked across the whole file:

  • Names must not repeat within a resource kind.

  • Served model names must not collide across deployments, counting adapter names. Requests for a duplicated name would be ambiguous.

  • A deployment placed on a shared accelerator declared in the same file has to set engine.memoryPercent.

  • A deployment’s parallelism factors have to multiply to the accelerator’s devicesPerReplica when that is above 1.

Placement rules are only checked in the file when the accelerator is declared there too. An accelerator that exists only on the cluster is checked at apply time.

What apply checks against the cluster

Before writing anything, apply verifies:

  • Served model names do not collide with deployments already in the namespace that are not part of this descriptor.

  • Every referenced secret is declared in the descriptor or already exists.

  • Every referenced accelerator is declared in the descriptor or already exists.

These need the cluster, so akka models validate cannot do them. Use --dry-run to run them without persisting.

Complete example

resource: Secret
resourceVersion: v1
metadata:
  name: hf-token
spec:
  type: generic
  data:
    token: hf_xxxxxxxxxxxxxxxxxxxx
---
resource: Accelerator
resourceVersion: v1
metadata:
  name: l4-dedicated
spec:
  device: l4
  tenancy: dedicated
---
resource: ModelDeployment
resourceVersion: v1
metadata:
  name: agent
spec:
  model: google/gemma-3-4b-it
  servedModelName: agent
  replicas: 1
  placement:
    accelerator: l4-dedicated
  engine:
    maxModelLen: 16384
    memoryPercent: 90
    toolCalling:
      enabled: true
      parser: pythonic
  source:
    huggingFaceTokenSecret: hf-token/token
    cache:
      enabled: true
      size: 100Gi
---
resource: InferenceRoute
resourceVersion: v1
metadata:
  name: acme
spec:
  host: models.acme.example.com
  models:
    - name: agent

Resources that belong elsewhere

Devices cannot be created from a descriptor. They are platform inventory. See Reviewing available hardware.

 

The features described in this section are an add-on to Akka Automated Operations. They are not included in the base product.

See also