Training runs

A run adapts a base model with one dataset and configuration. Use runs to train a model, inspect its progress, and manage its lifecycle. A run keeps the same identity across attempts.

The descriptor

training runs start accepts a JSON or YAML descriptor. The following descriptor from the support triage example defines a complete supervised run:

{
  "kind": "run",
  "project": "support-triage",
  "baseModel": "unsloth/Llama-3.2-1B-Instruct",
  "config": {
    "dataset": "support-triage/train",
    "method": "sft",
    "hyperparams": {"lora": {"rank": 8}, "optimizer": {"learningRate": 0.0001}, "training": {"epochs": 2}},
    "maxSeqLength": 2048,
    "model": {"precision": "BF16"}
  },
  "execution": {"gpus": 1, "saveEvery": 20, "keepAdapters": 4},
  "description": "two epochs at rank 8",
  "configName": "r8-e2"
}

The descriptor has the following top-level fields:

Field Description

kind

run. Required. The CLI reads it to tell a run descriptor from a pipeline descriptor.

project

The training project for the run, by name or ID. Required.

baseModel

The base model repository. Required.

modelName

The name under which the run registers its model in the training project’s namespace. Defaults to the base model’s name, so a run over Qwen/Qwen3-0.6B in the triage project registers triage/qwen3-0.6b:v1. Set it when the project’s name is the base model’s owner, such as a qwen project over Qwen/Qwen3-0.6B, because the default is then the base model’s own named model and the run is refused. It follows the registry’s naming rules. It is not part of the run’s configuration, so two runs that differ only in modelName share a column in training projects scores.

config

The configuration: what model is trained. See Configuration.

execution

The execution settings: how the run is executed. See Execution settings.

grader

A grader for the run’s evaluations. It overrides the training project’s grader. See Graders and rewards.

graderInstructions

The instructions given to a model: grader. They replace the instructions the judge would otherwise receive. Optimize refuses them with any other grader. See The judge’s instructions.

warmStartModelId

The registry version of an adapter that a reinforcement run starts from, instead of the base model, such as triage/qwen3-0.6b:v2.

submissionId

An ID for this submission. Optimize derives the run ID from it. Repeating a request with the same ID after a dropped connection returns the run started by the first request. If the descriptor omits this field, the CLI generates a new ID for each command. Specify an ID to make every submission of the descriptor refer to the same run.

smoke

Run a smoke check instead of the run. See Smoke check.

description, configName

Annotations. configName heads the configuration’s column in training projects scores. Neither field is part of the run’s identity. You can change them later with training runs annotate.

Configuration

Every field of config affects the trained model. The trainer hashes the configuration and uses the hash as its identifier. Two runs with the same configuration and base model use the same training recipe. The start response lists previous runs of that recipe with their scores.

config has the following fields:

Field Description

dataset

The training dataset, specified by registry reference, such as triage/train or triage/train:v3. Required.

method

sft, grpo, or gspo. Default sft.

hyperparams

Supervised hyperparameters, for sft.

rlHyperparams

Reinforcement hyperparameters, for grpo and gspo.

reward

The scorer for each sampled completion in a reinforcement run. Defaults to the run’s grader. See Graders and rewards.

warmStartConfigHash

The configuration hash of the model that a reinforcement run starts from. The trainer sets it from warmStartModelId. Leave it out.

model

How the base model is loaded and what the forward pass keeps. See Model settings.

maxSeqLength

The longest example that the run trains on, in tokens. A longer example is refused, unless data.truncation is right. See Data. Default 2048.

data

How the run reads its data, including the dataset it validates on. Optional. See Data.

loss

Which tokens in a conversation contribute to the run’s loss. Optional. A reinforcement run that states values other than the defaults is refused. See Loss.

engineSettings

Settings the run gives the training engine directly. Optional. See Engine settings.

An omitted field uses its default. A configuration that specifies a default and one that omits it produce the same hash, except for engineSettings, hyperparams.training.tokensPerGpu and hyperparams.training.validationBatchSize. These are part of the hash whenever they are set, even at their default value.

Supervised hyperparameters

hyperparams has four groups: lora, optimizer, scheduler, and training. A refusal names a field by its group, for example optimizer.weightDecay. Every field is part of the configuration hash.

Specify any fields that you need to change. Each omitted field uses the default for the selected training method, so specifying one field in a group does not change the other fields.

The lora group describes the shape of the adapter:

Field Default Description

rank

8

The adapter’s rank. The endpoint that serves the model must support this rank.

alpha

Twice the rank

The scale at which the update is applied.

dropout

0

The probability that an adapted activation is dropped during training. Must be at least 0 and less than 1.

targetModules

default

The modules that the adapter covers, as a regular expression that matches whole module paths. default covers every linear module of the language backbone. A pattern that matches no module fails the run.

The optimizer group describes what performs the update:

Field Default Description

kind

adamw

The algorithm. Only adamw is accepted.

learningRate

0.0001

The peak learning rate.

weightDecay

0

The decoupled decay applied to every adapted weight. Must not be negative.

maxGradNorm

1

The norm to which the gradient is clipped before the update. Each step reports the factor used to scale the gradient during clipping. See Metrics.

The scheduler group describes how the learning rate changes over the run:

Field Default Description

kind

linear

linear, cosine, or constant.

warmupRatio

0

The fraction of the run’s updates spent increasing to the configured learning rate. Must be at least 0 and less than 1.

minLearningRateRatio

0

The fraction of the learning rate that the schedule decays to, instead of zero. Only cosine decays to a floor. A floor with another kind is refused.

The schedule spans the run’s total updates: training.maxSteps if set, or the updates in all of the run’s epochs otherwise. A supervised branch’s schedule spans its own update budget. In a supervised run, the warmup is warmupRatio times the total updates, rounded up to a whole update. For example, a ratio of 0.03 over 939 updates warms up over 29 updates.

The schedule counts from zero. The first update takes the schedule’s value at 0, and update n takes its value at n - 1. With a warmup, the first update therefore runs at a learning rate of 0, and the rate reaches the full value at the first update after the warmup. Without a warmup, the first update runs at the full rate. A linear or cosine schedule reaches its end value after the last update, so the last update runs at a small rate above it. A resumed run restores the schedule’s position from its checkpoint, so each update takes the rate it would have taken without the interruption.

The training group describes the loop:

Field Default Description

epochs

3

The number of passes over the dataset.

maxSteps

None

The number of optimizer updates. A positive value overrides epochs.

microBatchPerGpu

Sized from maxSeqLength

The number of sequences in one forward and backward pass on one device. If tokensPerGpu is not set, the run is refused if this value times maxSeqLength exceeds the qualified limit in the base model’s catalog entry, where the entry states one.

gradientAccumulation

Sized from microBatchPerGpu

The number of passes summed into one update. Multiplied by microBatchPerGpu, this is the number of sequences in one update on each device. On more than one device, each device packs its sequences into passes. The token count and tokensPerGpu determine the number of passes.

validationFrequencySteps

100

The number of optimizer updates between evaluations on the validation dataset. A run without a validation dataset performs no evaluations, regardless of this value.

seed

42

The seed that initializes the adapter, orders the data, and drives dropout. Zero uses the default. Specify another number to set a seed.

tokensPerGpu

microBatchPerGpu times maxSeqLength

The maximum number of tokens in one forward and backward pass on one device. A pass holds whole conversations up to this budget, with no padding. Set it to fill a device with conversations of different lengths without raising maxSeqLength. It must be at least maxSeqLength. The run is refused if it exceeds the qualified limit in the base model’s catalog entry, where the entry states one. For a flex_attention run, the budget is compared after padding to engine.pad_to_length_bucket. Optimize refuses it on a run on one device. The configuration hash includes it when set.

validateEachEpoch

false

Also evaluate on the whole validation dataset before the first update and at the end of each epoch, in addition to the evaluations every validationFrequencySteps updates. An evaluation due at both intervals runs once. A run that sets it without a validation dataset is refused. A full-state branch takes neither the evaluation before its first update nor the epoch evaluations. See [validation].

validationBatchSize

The conversations in one update

The number of validation conversations evaluated in one call. A larger value makes fewer calls, so an evaluation can take less time. It does not change the measurement. Each device splits its share of a call into passes as it does in training, so it does not change the memory a pass needs. Optimize refuses it on a run on one device. A run that sets it without a validation dataset is refused. The configuration hash includes it when set.

If omitted, microBatchPerGpu is the largest of 8, 4, 2, and 1 sequences whose tokens at maxSeqLength fit within 4096 tokens. It is 1 if none fits or if engineSettings sets data.use_dynamic_bsz to false. An omitted gradientAccumulation accumulates passes to 8 sequences in one update: 8 divided by microBatchPerGpu, and at least 1. The service resolves both values before hashing the run, so the configuration hash includes the values used for training.

On more than one device, one update takes microBatchPerGpu times gradientAccumulation conversations on each device. Each device receives the same number of conversations. Token counts are balanced across devices as evenly as conversation lengths allow. Each device then packs its conversations into passes of up to tokensPerGpu tokens, or trains one conversation per pass where engineSettings sets data.use_dynamic_bsz to false. The loss is the mean over all trained tokens in the update, across all devices. Each trained token has the same weight, regardless of the budget or how conversations are divided. Each epoch trains every conversation once. The last update in an epoch takes all remaining conversations. If their count is not a multiple of the device count, the update adds copies that train nothing. training runs get --settings and a dry run show the update size, and the budget or one conversation per pass.

Most numbers in these groups use their default when omitted or set to zero. optimizer.weightDecay set to 0 trains without weight decay, and only an omitted value takes the default. Optimize refuses a negative count, step budget or token budget and names the field.

Data

data specifies how the run reads its training data. It has the following fields:

Field Default Description

validationDataset

none

A second registered dataset that the run evaluates on while it trains, by registry reference. Optimize refuses an unregistered dataset or the training dataset itself.

truncation

error

What happens to a row longer than maxSeqLength. error refuses the row and names it. right drops tokens from the end, and refuses the row when a dropped token was one the run trains on.

packing

false

Whether short rows are packed into one sequence. Packing makes a step cover more rows and makes two runs over the same data see different batches. Only a run on one device packs. A run on more than one device already packs whole conversations into each pass, up to hyperparams.training.tokensPerGpu, or trains one conversation per pass where data.use_dynamic_bsz is false, so Optimize refuses packing set to true for it.

shuffle

true

Whether the rows are shuffled before the first epoch. The order is drawn from hyperparams.training.seed, which is the run’s one seed.

A supervised dataset holds chat examples or prepared token rows, and the registered dataset records which, so there is no format field. The base model’s tokenizer provides the chat template for chat examples, so there is no chatTemplate field.

With a validation dataset, the run evaluates every validationFrequencySteps updates. With hyperparams.training.validateEachEpoch set to true, it also evaluates before the first update, which provides the baseline for later measurements, and at the end of each epoch. A resumed run does not repeat the measurement before the first update. A supervised run on more than one device also evaluates at its last update. After a resume, it evaluates again at a due step whose measurement was lost.

Each evaluation covers the whole validation dataset. Its loss is the mean over every supervised token in the set, so a long answer counts for more than a short one, and its perplexity is the exponential of that loss. The run evaluates validationBatchSize conversations in each call and weights each call’s loss by its supervised tokens, so the measurement is the same at any validationBatchSize, up to rounding. The run metrics contain the validation loss, its perplexity, and the training loss, and the run’s curves show the validation measurements under eval/loss and eval/perplexity. runs get shows the latest values. Read the losses together. A training loss that falls while the validation loss does not indicates that the run is memorizing its data.

Loss

loss specifies which tokens in a conversation contribute to the run’s loss. Without an explicit mask, system and user tokens are excluded from training regardless of these settings. A mask from explicitLossMaskColumn is used as given. It has the following fields:

Field Default Description

assistantTurns

last

Which assistant messages are trained on: all, or last alone. On a single-turn row, last is the whole completion.

trainOnToolCalls

true

Whether the tool-call tokens in a selected assistant message count. This setting works with assistantTurns. An assistant message that contains calls is never the last message of a valid row, so without an explicit mask a tool call is never trained under last, regardless of this setting.

trainOnToolResults

false

Whether a tool result’s own tokens contribute to the loss.

explicitLossMaskColumn

none

A dataset column containing an explicit loss mask. For rows with this column, the mask is used as given and the other settings in this group do not apply.

A supervised run on more than one device computes the model’s output layer only for tokens included in the loss. This saves memory and time on conversations where most tokens are excluded from training. The loss and gradients are the same as in the full computation. This applies when the deployment computes log-probabilities in chunks of tokens, as it does by default. Optimize refuses this computation for a base model it has not verified. A model that soft-caps its final logits, such as Gemma, retains that cap.

Optimize refuses a loss or data group on a reinforcement run. A reinforcement run takes its loss from the reward over a rollout and reads its dataset as rollout prompts. A group that contains only defaults is accepted. A branch to a reinforcement method inherits neither group, so a group on the supervised parent does not cause the branch to be rejected.

Reinforcement hyperparameters

rlHyperparams has the following fields and the same lora, optimizer, and scheduler groups as hyperparams. Three defaults differ for reinforcement training: optimizer.learningRate is 0.00001, optimizer.weightDecay is 0.01, and scheduler.kind is constant. The reinforcement engines train the adapter without dropout and follow a constant or cosine schedule. Optimize refuses a lora.dropout above zero or a linear schedule.

Field Default Description

epochs

1

The number of passes over the prompts.

maxSteps

None

A cap on optimizer updates. A positive value overrides epochs.

groupSize

4

The number of completions sampled per prompt. The trainer computes group statistics from these completions.

promptsPerStep

8

The number of prompts rolled out per step. A step generates promptsPerStep times groupSize completions. Set it to a multiple of execution.gpus so each device receives the same number of prompts.

klBeta

0.04

The KL penalty toward the reference policy. Zero disables the penalty.

temperature

0.7

The sampling temperature. Zero uses greedy sampling.

maxCompletionLength

256

The longest completion sampled, in tokens.

stalenessThreshold

0

Not supported. Optimize refuses a value above zero: every step trains on-policy, on rollouts generated from the current weights. Leave it out or set it to 0.

The trainer preserves the configured values of klBeta, temperature, maxSteps, and stalenessThreshold because zero is valid for each field. optimizer.weightDecay set to 0 trains without weight decay, and only an omitted value takes the default of 0.01. The other fields use their defaults when omitted or set to zero.

Model settings

The model object has the following fields:

Field Default Description

precision

BF16

How the trainer loads the base model’s weights: BF16 or INT4. With INT4, the adapter trains against the quantized base, which is a different model from the one used with BF16. Only a run on one device trains at INT4. Optimize refuses INT4 for a run on more than one device when you submit the run.

gradientCheckpointing

true if maxSeqLength is greater than 4096

Whether the backward pass recomputes activations instead of keeping them. This reduces memory use at the cost of more computation. If omitted, the value is derived from maxSeqLength, and the derived value is part of the hash.

attentionImplementation

sdpa

The kernel used by the attention layers: sdpa, eager or flex_attention. sdpa is PyTorch’s fused implementation. It is faster and uses less memory where the architecture supports it. eager is the architecture’s own implementation and is always available. Use it when a model card requires it or when the fused kernel does not work correctly on your hardware. flex_attention is accepted only for a model it is verified on. See Attention in a packed pass.

weightPrecision

The deployment’s, usually fp32

The precision of the trained model’s weights: fp32 or bf16. At bf16, the weights take half the memory, but an update the size of a typical learning rate can round away and the run learns less. Use bf16 only when the fp32 weights do not fit the device. Part of the configuration hash. See Memory settings.

Attention in a packed pass

A supervised run on more than one device packs several conversations into one pass. Under sdpa and eager, each conversation attends only to its own tokens, so its log-probabilities do not depend on the other conversations in the pass.

flex_attention is offered only for a model type it is verified on: the types of Gemma 4 E2B, E4B and 12B. Before claiming devices, Optimize refuses a run that names it for a model of another type, and the error names the model. A dry run reports the same refusal.

A flex_attention run pads every pass to a multiple of engine.pad_to_length_bucket, 1024 tokens by default, because flex attention has no kernel for a pass shorter than 128 tokens. Optimize refuses a run that turns engine.pad_to_length off or sets the bucket below 128. Flex attention also needs a device with enough shared memory per block. On a smaller GPU, the run fails at startup and the error names the device.

Engine settings

Use engineSettings to set the training engine’s settings directly. Most settings affect only memory use and speed. Use them to fit your model to its devices. Precision settings change the arithmetic used during training. engineSettings has the following fields:

Field Description

engine

verl. Required. No other engine takes engine settings.

settings

Each setting’s value, by its name. Required.

A run can set the following settings, where its deployment allows each one:

Setting Default Description

engine.forward_prefetch

false

Fetches the next layer’s weights while the current layer computes. A step can be faster, and it needs more memory.

engine.reshard_after_forward

true

Frees each layer’s gathered weights after the forward pass. false keeps them for the backward pass, which can be faster and needs more memory.

engine.pad_to_length

false, true for a flex_attention run

Pads each pass up to a multiple of engine.pad_to_length_bucket tokens, so the kernels see fewer shapes. It can speed up a run whose sequence lengths vary. Padding requires additional computation.

engine.pad_to_length_bucket

1024

The multiple engine.pad_to_length pads to.

engine.use_no_sync_for_gradient_accumulation

true for a run that trains an adapter

Accumulates the gradients from each pass on each device and combines them across devices once, after the last pass. Each device holds the whole gradient until then, which is small for an adapter. false combines them after every pass, so every device waits at each pass for the device whose pass takes longest. That is slower when passes differ in length, such as one conversation per pass. With bf16 computation, a device adds up its passes in bf16 before the gradients are combined in fp32, so the two differ by rounding. A reinforcement run that trains the full model keeps false.

engine.mixed_precision.param_dtype

bf16

The data type used for computation in the forward and backward passes: bf16 or fp32. fp32 is slower and needs more memory. Supervised runs only.

engine.mixed_precision.reduce_dtype

fp32

The data type used to sum gradients across devices: bf16 or fp32. bf16 transfers half as many bytes but reduces the precision of the sum. Supervised runs only.

engine.mixed_precision.buffer_dtype

fp32

The data type used to store the model’s buffers, such as its rotary tables: bf16 or fp32. Supervised runs only.

compile.recompile_limit

8

The maximum number of compiled versions of a function that Torch compile keeps for one layer. For example, each sequence shape can have its own version. After this limit, the function runs without compilation. Supervised runs only.

compile.accumulated_recompile_limit

256

The maximum number of compiled versions of a function that Torch compile keeps across all layers that run it. Supervised runs only.

compile.fail_on_recompile_limit_hit

false

true fails the run when either recompile limit is reached. false runs the function without compilation from then on. This is slower and does not report that the limit was reached. Supervised runs only.

liger.rms_norm

false

Uses Liger’s fused kernel for the model’s RMS normalization layers. This can improve speed and reduce memory use. On Gemma 4 E2B and E4B, the kernel applies to every RMS normalization layer in the language model except the attention value norm v_norm, post_per_layer_input_norm and per_layer_projection_norm. These keep their own forward methods. Not available for Gemma 4 12B. Supervised runs only, for model types verified for this kernel.

liger.geglu

false

Uses Liger’s fused kernel for a GeGLU feed-forward layer, such as Gemma’s. This can improve speed and reduce memory use. Not available for Gemma 4 12B. Supervised runs only, for model types verified for this kernel.

checks.start

true

Checks the run’s numerics before its first update. false trains without the checks. Supervised runs only. See Checks before the first update.

checks.head.max_logprob_error

0.01

The head check’s bound on the largest difference in a log-probability. Supervised runs only. See Checks before the first update.

checks.head.max_gradient_error

0.005

The head check’s bound on the gradient’s difference relative to its size. Supervised runs only.

checks.isolation.max_logprob_error

0.0001

The isolation check’s bound on how far a conversation’s log-probabilities move. Supervised runs only, with data.use_dynamic_bsz left on.

checks.packing.max_loss_error

0.005

The packing check’s bound on the difference in the step’s mean loss. Supervised runs only, with data.use_dynamic_bsz left on.

checks.reference.max_logprob_error

1.0

The reference check’s bound on the largest difference in a log-probability at the run’s precision. Supervised runs only.

checks.reference.mean_logprob_error

0.03

The reference check’s bound on the mean difference in a log-probability at the run’s precision. Supervised runs only.

checks.reference.max_nll_error

Calculated from the measured differences, at least 0.005

The reference check’s bound on the difference in the mean loss at the run’s precision. Supervised runs only. See Checks before the first update.

checks.reference.fp32_max_logprob_error

0.002

The reference check’s bound on the largest difference in a log-probability at fp32. Supervised runs only.

checks.reference.fp32_mean_logprob_error

0.0001

The reference check’s bound on the mean difference in a log-probability at fp32. Supervised runs only.

head.chunk_tokens

512

The number of positions the output head computes log-probabilities for at once. Each position in a chunk holds its logits over the whole vocabulary, so a smaller value needs less memory and launches more kernels. Any whole number above 0 is accepted, and the log-probabilities and gradients are the same at every value up to rounding. Applies when the deployment computes log-probabilities in chunks, which it does by default. Supervised runs only. A reinforcement run computes 512 at a time.

checkpoint.compare_replicas

true

With memory.sharding set to none, compares every device’s optimizer state with the first device’s before the first save of each attempt, and fails the run if any differs. Only the first device’s copy is saved. false skips the comparison. Supervised runs only. See [recovery].

rollout.enable_chunked_prefill

true

Splits a long prompt’s prefill into chunks. Reinforcement runs only.

rollout.enable_prefix_caching

true

Reuses the cache for prompts with a common prefix, such as the completions of one group. Reinforcement runs only.

rollout.enforce_eager

false

Runs the rollout engine without CUDA graphs. It starts faster and uses less memory, but generates more slowly. Reinforcement runs only.

rollout.max_num_seqs

The step’s rollouts

The most sequences the rollout engine generates at once. Reinforcement runs only.

rollout.max_num_batched_tokens

The larger of maxSeqLength and the step’s rollouts

The most tokens the rollout engine processes in one batch. Reinforcement runs only.

data.use_dynamic_bsz

true

Packs whole conversations into each pass, up to hyperparams.training.tokensPerGpu. false trains one conversation per pass, so a step takes one pass for each of its conversations. The loss is the same mean over the update’s trained tokens either way. With false, an unset hyperparams.training.microBatchPerGpu is 1. A run that sets it above 1, or sets tokensPerGpu, is refused. Supervised runs only.

The following example sets two of them:

"engineSettings": {
  "engine": "verl",
  "settings": {"engine.forward_prefetch": true, "rollout.enforce_eager": true}
}

The settings are part of the configuration hash. training runs get --settings and training runs start --dry-run --show-resolved show them after the engine, one line for each group of settings, such as compile or liger. Settings reported on other lines are omitted from this list. Explicit data types appear on the precision line, and data.use_dynamic_bsz set to false appears on the batch line. For a supervised run on more than one device, both commands also show the data types used for training. These include the parameter type from config.model.weightPrecision and the types used for computation, gradient reduction and buffers. A run that specifies engine settings uses the engine they name. Before claiming devices, Optimize refuses the run if the base model’s catalog entry does not list that engine for the run’s devices. It also rejects a supervised run on one device that specifies them.

Optimize also refuses a run, naming the setting, when the setting:

  • is not one the deployment allows,

  • is for the rollout engine and the run is supervised,

  • applies only to supervised runs and the run is a reinforcement run, or

  • has a value of the wrong kind, such as a number for engine.forward_prefetch.

Before claiming devices, Optimize also refuses your run if you enable a Liger kernel for a model that is not verified for that kernel. The error names the kernel and the model. Liger’s kernels are verified for Gemma 4 E2B and E4B, and not for Gemma 4 12B, which takes flex attention alone.

A run cannot specify a setting already determined by its descriptor, a setting that names a path or code, or the seed.

A branch keeps its parent’s applicable engine settings. For example, a supervised branch drops rollout settings.

Execution settings

execution has the following fields:

Field Default Description

gpus

1

The number of devices that the run claims. A supervised run on one device and one on more than one device train differently, and several fields on this page apply to only one of them. A reinforcement run on more than one device needs all of them on one node. Use a value that the deployment’s catalog admits.

rolloutGpus

0

Not supported. Optimize refuses a value above zero: the rollout engine runs on the training devices. Leave it out or set it to 0, and count every device in gpus. See Where a reinforcement run generates rollouts.

microBatchSequences

chosen by the job

The maximum number of sequences per device in one forward and backward pass. It changes only peak memory. Accumulation combines the same rollouts in the same optimizer step.

saveEvery

off

The periodic checkpoint interval, in optimizer updates. When it is off and saveEachEpoch is not set, the run writes only one checkpoint at the end. An interrupted run therefore has no checkpoint from which to restart.

keepAdapters

5

The number of unreferenced adapter checkpoints retained for the run. Adapter checkpoints saved at the end of an epoch do not count toward this limit.

keepRecoverable

1

The number of unreferenced saves that keep their training state, so that a resume can start from any of them. Must be at least 1. Each retained save uses storage for its training state. See [recovery].

saveEachEpoch

false

Set this to true to also save at the end of each epoch in a supervised run. These adapters are retained without counting toward the keepAdapters limit. Each adapter is registered as a version of the run’s model. The save at the run’s last update is the run’s own model and is not registered twice. A deployment that stores checkpoints on a shared filesystem refuses this setting. See Epoch adapters.

deviceProfile

chosen by the catalog

The accelerator to train on, as the catalog names it. Omit it when a deployment has only one kind of device. If the named device has no qualified capability for the model, Optimize refuses the run and names the devices that do.

engine

chosen by the catalog

The training engine, trl or verl, among the engines that the catalog lists for the device. Use it to test an engine. A deployment can refuse it.

rollout.memoryShare

The deployment’s, usually 0.5

The share of each device that a reinforcement run’s rollout engine takes for its weights and its cache. See Memory settings.

memory.offload.weights

The deployment’s, usually false

Whether the trained model’s weights move to host memory between passes. See Memory settings.

memory.offload.optimizer

false

Whether the optimizer’s state moves to host memory between updates. See Memory settings.

memory.offload.activations

false

Whether activations saved during the forward pass stay in host memory until the backward pass needs them. See Memory settings.

memory.sharding

all

How your trained model is distributed across your run’s devices. all splits it across the devices. none keeps a complete copy on each device. See Memory settings.

hostMemoryGiB

none

The host memory required to load the run’s model. Optimize refuses the run at submission if this value exceeds the host memory listed for the named device. This field does not set the run’s host memory allocation. Omit it if the device lists no host memory, because the device then accepts any value.

Execution settings are recorded with the run and are not part of the configuration hash.

For example, to save every 50 updates, keep the three latest saves resumable, and retain each epoch’s adapter separately, set:

"execution": {"saveEvery": 50, "keepAdapters": 3, "keepRecoverable": 3, "saveEachEpoch": true}

Where the deployment stores checkpoints in object storage, each save is a snapshot. A save holds the adapter weights and the full training state: the optimizer, the scheduler, the random-number generators and the step reached. The run also saves at its last update. Each time a save is recorded, Optimize retains the following and deletes the rest:

  • The adapter weights of the keepAdapters latest saves that nothing references, the save at the last update included.

  • The training state of the keepRecoverable latest saves that nothing references. Earlier training state is deleted, so of the unreferenced saves only those can be resumed.

  • The training state of the save that a retry or a resume of the run would restore, whatever its rank.

  • The adapter weights of every epoch save, until you release its epoch hold with training snapshots release. Their training state follows the rule above.

  • Any part of a save that a branch or a pin references, for as long as the reference exists. A registered model version keeps its own copy of the weights and does not keep the save.

A deployment that stores checkpoints on a shared filesystem lists no snapshots, so these retention rules do not apply, and it refuses a keepRecoverable above 1.

A retry, and training runs resume without --step, restore the newest save. To resume a paused run from an older save that still holds its training state, name its step with training runs resume RUN_ID --step STEP. training runs snapshots lists the saves and their steps. A branch with --restore trainingState can start from any of them by its snapshot ID. See Resume a run.

Training state is the optimizer’s two moments for every trained parameter, plus the adapter, the scheduler and the random-number generators, for every device. For Gemma 4 E4B with a rank 64 adapter on the attention and feed-forward projections, the adapter has about 147 million parameters. At fp32 weights, one copy of the adapter and its moments is about 1.8 GB, and the training state also holds an exported copy of the adapter, about 0.6 GB. With either value of memory.sharding, a save’s training state is about 2.4 GB regardless of the number of devices. Setting keepRecoverable to 3 retains about 7 GB.

With none, before the first save of each attempt, the run compares a hash of every device’s optimizer state with the first device’s, and fails if any differs. The comparison takes about a second for each gigabyte of optimizer state. Set the engine setting checkpoint.compare_replicas to false to skip it.

After a resume from an older save, the later saves from the earlier attempt remain until retention deletes them. Retention ranks the resumed attempt’s saves first. It deletes unreferenced saves from the earlier attempt before those from the resumed attempt.

Memory settings

Use these settings to adjust memory use, training speed, or learning. You cannot set them on a supervised run on one device. If you omit a setting with a deployment value, your run uses that value. execution.memory.offload.optimizer and execution.memory.offload.activations default to false. execution.memory.sharding defaults to all.

Setting Trade

execution.rollout.memoryShare

A larger share gives the rollout engine more room for its cache, so it generates more completions at once. It leaves less memory for training. A model that fills most of the device needs a larger share to load at all. A reinforcement run only, since a supervised run has no rollout engine.

config.model.weightPrecision

bf16 halves the memory that the weights take. A small update can round away, so the run can learn less than the same run at fp32.

execution.memory.offload.weights

true moves the weights to host memory between passes, which frees the device for the rollout engine. Each step takes longer because the weights move back and forth.

execution.memory.offload.optimizer

true keeps the optimizer state in host memory. It moves the state to the device only during each update. If your run trains an adapter, it keeps optimizer state only for the adapter. In that case, offloading optimizer state frees less memory than offloading weights. Each update takes longer. The state uses host memory.

execution.memory.offload.activations

true moves activations saved during the forward pass to host memory, one layer at a time. It returns each layer’s activations to the device when the backward pass needs them. This frees the most memory on long sequences, where activations fill device memory. Each step takes longer. The activations use host memory, which hostMemoryGiB does not count. You cannot use this setting with an image-text model, such as Gemma 4, whose vision layers do not run on text rows.

execution.memory.sharding

all splits the weights, gradients, and optimizer state across your run’s devices. Each pass gathers the parts it needs. none keeps all of them on each device. The devices combine their gradients once per update, or after each pass if engine.use_no_sync_for_gradient_accumulation is false. Use none when your model’s training state fits on one device. A step does not gather model state, so it can be faster. On one device, both settings have the same effect.

The deployment determines which settings a run can specify and the allowed range of each. Optimize refuses a run before it claims devices, and names the setting, when:

  • the deployment does not allow a run to specify the setting,

  • memoryShare is outside the deployment’s range, or the run is supervised,

  • weightPrecision is not one the deployment allows,

  • offload.activations is true for an image-text model, or

  • no engine that the catalog lists for the run can honor the setting.

If you set any of these settings, your run uses an engine that supports them. Optimize refuses your run if its catalog entry lists no such engine for the run’s devices.

A run records the value of each setting when it is created. A later change to the deployment’s values does not change the run, a branch of it, or a retry. training runs get --settings shows the recorded values for a run that takes these settings and marks the ones the run stated. training runs engine-config shows the values the job passed to the training engine.

Where a reinforcement run generates rollouts

A reinforcement run generates rollouts on the same devices used for training. During each step, every device generates its share of the rollouts and then takes part in the update. To run on four GPUs, set gpus to 4 and leave rolloutGpus out. Separate devices for generation, such as three for training and one for rollouts, are not supported.

The deployment states the most devices a run can use, and counts gpus and rolloutGpus together against it. The catalog states the counts each model is qualified at on each device. Optimize refuses a run at any other count and names the qualified counts.

Start a run

Validate a descriptor, start the run, and wait for it to finish:

akka training runs start -f run.json --dry-run
akka training runs start -f run.json -o json --jq .runId
akka training runs wait RUN_ID --exit-status

--dry-run validates the descriptor against the service without training. Without it, the command validates the descriptor and asks for confirmation. --force skips the prompt. Without a submissionId in the descriptor, every submission starts a new run, so submitting the same descriptor twice trains it twice. With one, a later submission reaches the run that the first started.

Add --show-resolved to --dry-run to see the configuration and execution settings that the run would use. The output includes every default and every resolved dataset and reward reference:

akka training runs start -f run.json --dry-run --show-resolved

This command reports the same values that training runs get --settings reports after the run exists. Use it to resolve a descriptor before starting the run. -o json includes the resolved configuration as structured data.

Smoke check

A smoke check trains on the first rows of the dataset until the model has memorized them. It then checks that the model reproduces them. If a model cannot reproduce rows that it has seen repeatedly, the data, masking, or tokenizer has a problem.

A smoke check takes everything else from the descriptor. Add smoke to the descriptor to run one:

Field Default Description

sliceSize

200

The rows taken from the start of the dataset. These are also the rows scored afterwards.

maxSteps

2000

The cap on optimizer updates. Raise it with sliceSize, so the slice can still be memorized.

threshold

0.98

The reproduction score a working run reaches. It must not be above 1.

training runs get shows the verdict. training runs wait --exit-status exits with a non-zero status when the check fails.

Base models

A run can specify only a base model that the deployment allows and that someone has selected. To list the allowed models and select one, see Base models.

If a descriptor specifies a model that the deployment does not allow, Optimize refuses the run. Specify a model from the list, or ask your operator to allow the model that you need.

Before it admits a run, the trainer also checks the adapter rank and sequence length against what the deployment supports.

Methods

Supervised fine-tuning

sft learns from the assistant messages in the dataset. Use it when every example has a known correct output, such as classification, extraction, or another task with a manually written or curated dataset.

Checks before the first update

A supervised run on more than one device checks its numerical results before its first update, using its model, devices, and first step’s rows. While the checks run, the run is PREPARING at the stage checking the run before its first update. Each attempt checks again.

Check What it compares Bounds, with their defaults

head

The output head that computes the log-probabilities, on fp32 copies of a few positions of the first pass. Its log-probabilities against the model’s own head, and its gradient against the exact gradient of the model’s own head

checks.head.max_logprob_error 0.01, the largest difference in a log-probability. checks.head.max_gradient_error 0.005, the gradient’s difference relative to its size

packing

The step’s mean loss with its conversations packed as the run packs them, against one conversation per pass

checks.packing.max_loss_error 0.005

isolation

Two conversations packed into one pass, with every token of each replaced in turn. The other conversation’s log-probabilities must not move

checks.isolation.max_logprob_error 0.0001

reference

Two conversations through the run’s training path, against plain Transformers on each conversation alone, at the run’s precision and at fp32

At the run’s precision: checks.reference.max_logprob_error 1.0, checks.reference.mean_logprob_error 0.03 and checks.reference.max_nll_error, the difference in the mean loss, calculated as described below. At fp32: checks.reference.fp32_max_logprob_error 0.002 and checks.reference.fp32_mean_logprob_error 0.0001

The difference in the mean loss is a mean over the positions the reference check compares, which are the supervised tokens of its two conversations. Conversations with few supervised tokens provide few positions to compare. A large rounding difference at one position can then have a large effect on the mean. Rounding can shift positions in either direction. A fault in the run’s computation shifts every position in the same direction. When you leave checks.reference.max_nll_error out, its bound is calculated from the differences the check measures: the largest difference divided by the number of positions, plus three standard errors of the mean of the other differences, and at least 0.005. For example, over 80 positions with one differing by 0.61 and the rest rounded by about 0.01, the bound is about 0.011, and the measured difference of 0.0078 passes. If every position is shifted by 0.015, the bound is 0.005, and the check fails. training runs get -o json reports positions and the bound used, under startChecks.

Each bound is an engine setting that your deployment must allow. A bound you leave out takes its default. For example, to set the reference check’s maximum log-probability error at the run’s precision to 0.002:

"engineSettings": {
  "engine": "verl",
  "settings": {"checks.reference.max_logprob_error": 0.002}
}

A bound must be a number of at least 0. Optimize refuses a bound when checks.start is false, and a packing or isolation bound when data.use_dynamic_bsz is false, since those checks then do not run.

A check that does not apply is skipped, and the record says why. head runs only when the run computes log-probabilities with fused kernels. packing and isolation run only when a pass can hold more than one conversation. reference runs only on an attempt that starts from a fresh adapter, so a resumed attempt, a branch and a warm start skip it. It is also skipped when the first step has fewer than two conversations, or when the base model at fp32 does not fit beside the run on the first GPU.

If a check exceeds a bound, the run fails before it trains. training runs failure shows the check, the measured value, the bound and the setting that sets it. Whether the checks pass or fail, Optimize keeps their values and bounds, and training runs get shows them. With -o json, startChecks holds one entry for each attempt, and each check’s boundSettings names the setting of each bound.

akka training runs get RUN shows one line for the checks of the last attempt: how many passed and how long they took, or the first check that failed with each value over its bound. --checks adds one row for each check, with its verdict, each value against its bound, and the setting that sets the bound. -o json includes the whole record under startChecks.

The checks add roughly the time needed to load the base model once, run two forward passes over the first step, and run three short passes over two of its conversations. To train without them, set the engine setting checks.start to false. Your deployment must allow the setting.

Reinforcement learning

grpo and gspo sample groupSize completions per prompt and score each completion with the reward. They update the model toward completions that score above the group mean. Use them when you can verify but cannot enumerate correct output, such as executable code, SQL that must return the correct rows, or an answer with a verifiable final value.

A reward with only a strict correctness term gives an untuned model no signal. Every completion in a group scores zero, so the advantage is zero. Add a format term so that the model has a gradient before it produces a correct answer. Alternatively, warm-start from a supervised adapter with warmStartModelId.

You can train an image-text model such as Gemma 4 E4B with grpo or gspo over a text dataset on one device. As in a supervised run, the adapter updates the language backbone and leaves the vision and audio towers unchanged. On more than one device, your deployment must enable grpo and gspo for image-text models. Otherwise, the model supports only sft there. Run base-models get with the model’s repository to see the methods that your deployment supports. Reinforcement runs cannot train on images. Optimize refuses a grpo or gspo run whose dataset contains images, regardless of the base model.

The following descriptor, from the GSM8K example, is a descriptor for a reinforcement run:

{
  "kind": "run",
  "project": "gsm8k",
  "configName": "grpo-100-steps",
  "baseModel": "Qwen/Qwen3-0.6B",
  "config": {
    "dataset": "gsm8k/train",
    "method": "grpo",
    "rlHyperparams": {
      "epochs": 1,
      "maxSteps": 100,
      "groupSize": 8,
      "klBeta": 0.001,
      "temperature": 1.0,
      "maxCompletionLength": 512,
      "lora": {
        "rank": 32
      },
      "optimizer": {
        "learningRate": 1e-5
      }
    },
    "reward": {
      "terms": [
        {"kind": "rule", "weight": 1.0, "bundle": "gsm8k/scoring", "module": "reward"}
      ]
    },
    "maxSeqLength": 1536
  },
  "execution": {"gpus": 1, "saveEvery": 10, "keepAdapters": 4}
}

Lifecycle

Run the following commands to read a run, list open runs, and cancel a run:

akka training runs get RUN_ID
akka training runs list --phase open
akka training runs cancel RUN_ID

The run detail reports the phase, progress, cost, timing, and any failure reason. observedStep counts the updates reported by the worker. savedStep is the selected restorable checkpoint. The difference between these values is the work that a restore repeats.

A run reports one of the following phases. A first attempt passes through them in order and skips phases that it does not need. A supervised run on more than one device also reports PREPARING before AWAITING_CAPACITY, while it prepares its rows without GPUs. A resumed run re-enters at AWAITING_BASE_MODEL, DEPLOYING_GRADER or AWAITING_CAPACITY. If the cluster stops a job, its run moves from TRAINING to PAUSING.

Phase What the run is doing

RESOLVING_DATASET

Reading the dataset’s manifest.

AWAITING_BASE_MODEL

Waiting for the run’s base model to finish preparing.

DEPLOYING_GRADER, AWAITING_GRADER

Starting the judge specified by a reinforcement run’s reward and waiting for it to respond.

AWAITING_CAPACITY

Waiting for the cluster to allocate the devices requested by the run.

STARTING_JOB

Submitting the training job.

QUEUED

Waiting for the cluster to schedule the submitted job. The wait depends on the cluster’s other work.

PREPARING

A supervised run is preparing before its first reported step. Its stage names the current action and reports completed and total rows during tokenization. A supervised run on more than one device enters this phase before AWAITING_CAPACITY, while row preparation waits to be scheduled or runs. It requests GPUs after preparation succeeds. Cached rows need no rendering. For a prepared dataset, Optimize checks the rows without rendering them. You cannot pause the run during this preparation. After claiming devices, the run also uses this phase while reading data or starting training. A supervised run on one device tokenizes after claiming its device. A reinforcement run goes from QUEUED to TRAINING.

TRAINING

The job is running. runs metrics reads its steps.

PAUSING, PAUSED

Stopping the job and releasing its resources until the run resumes.

RECORDING

Registering the trained adapter as a model. While the registry does not answer, the run waits here and tries again. If the registry refuses the model, the run fails.

CLEANING_UP

Releasing the run’s resources before a terminal phase. A run also waits here, with its devices released, until its last saves are recorded and committed or their commit fails, for at most 15 minutes, so a finished run reports its last saved step.

COMPLETED, FAILED, CANCELLED

Terminal.

A completed run registers its final adapter as a model. training runs get shows the model. Cancelling a run releases its devices but retains its committed snapshots, which you can still register or use as branch sources.

For training runs pause and resume, see Checkpoints. For training runs branch, see Branches.