Training runs

A run adapts a base model with one dataset and configuration. Use runs to train a model, inspect its progress, and manage its lifecycle. A run keeps the same identity across attempts.

The descriptor

training runs start accepts a JSON or YAML descriptor. The following descriptor from the support triage example defines a complete supervised run:

{
  "kind": "run",
  "project": "support-triage",
  "baseModel": "unsloth/Llama-3.2-1B-Instruct",
  "config": {
    "dataset": "support-triage/train",
    "method": "sft",
    "hyperparams": {"lora": {"rank": 8}, "optimizer": {"learningRate": 0.0001}, "training": {"epochs": 2}},
    "maxSeqLength": 2048,
    "model": {"precision": "BF16"}
  },
  "execution": {"gpus": 1, "saveEvery": 20, "keepAdapters": 4},
  "description": "two epochs at rank 8",
  "configName": "r8-e2"
}

The descriptor has the following top-level fields:

Field Description

kind

run. Required. The CLI reads it to tell a run descriptor from a pipeline descriptor.

project

The training project for the run, by name or ID. Required.

baseModel

The base model repository. Required.

modelName

The name under which the run registers its model in the training project’s namespace. Defaults to the base model’s name, so a run over Qwen/Qwen3-0.6B in the triage project registers triage/qwen3-0.6b:v1. Set it when the project’s name is the base model’s owner, such as a qwen project over Qwen/Qwen3-0.6B, because the default is then the base model’s own named model and the run is refused. It follows the registry’s naming rules. It is not part of the run’s configuration, so two runs that differ only in modelName share a column in training projects scores.

config

The configuration: what model is trained. See Configuration.

execution

The execution settings: how the run is executed. See Execution settings.

grader

A grader for the run’s evaluations. It overrides the training project’s grader. See Graders and rewards.

graderInstructions

The instructions given to a model: grader. They replace the instructions the judge would otherwise receive. Optimize refuses them with any other grader. See The judge’s instructions.

warmStartModelId

The registry version of an adapter that a reinforcement run starts from, instead of the base model, such as triage/qwen3-0.6b:v2.

submissionId

An ID for this submission. Optimize derives the run ID from it. Repeating a request with the same ID after a dropped connection returns the run started by the first request. If the descriptor omits this field, the CLI generates a new ID for each command. Specify an ID to make every submission of the descriptor refer to the same run.

smoke

Run a smoke check instead of the run. See Smoke check.

description, configName

Annotations. configName heads the configuration’s column in training projects scores. Neither field is part of the run’s identity. You can change them later with training runs annotate.

Configuration

Every field of config affects the trained model. Optimize hashes the configuration and uses the hash as its identifier. Two runs with the same configuration and base model use the same training recipe. The start response lists previous runs of that recipe with their scores.

config has the following fields:

Field Description

dataset

The training dataset, specified by registry reference, such as triage/train or triage/train:v3. Required.

method

sft, grpo, gspo, or dpo. Default sft. See Methods.

hyperparams

Supervised hyperparameters, for sft and dpo.

preference

Preference settings, for dpo. Optimize refuses the group on a run of another method. See Preference settings.

rlHyperparams

Reinforcement hyperparameters, for grpo and gspo.

reward

The scorer for each sampled completion in a reinforcement run. Defaults to the run’s grader. See Graders and rewards.

warmStartConfigHash

The configuration hash of the model that a reinforcement run starts from. Optimize sets it from warmStartModelId. Leave it out.

model

How the base model is loaded and what the forward pass keeps. See Model settings.

maxSeqLength

The longest example that the run trains on, in tokens. By default, Optimize refuses a longer example. data.truncation controls whether Optimize shortens it instead. See Data. Default 2048.

data

How the run reads its data, including the dataset it validates on. Optional. See Data.

loss

Which tokens in a conversation contribute to the run’s loss. Optional. A reinforcement run that states values other than the defaults is refused. See Loss.

engineSettings

Settings the run gives the training engine directly. Optional. See Engine settings.

An omitted field uses its default. A configuration that specifies a default and one that omits it produce the same hash, except for engineSettings, hyperparams.training.tokensPerGpu and hyperparams.training.validationBatchSize. These are part of the hash whenever they are set, even at their default value.

Supervised hyperparameters

hyperparams has four groups: lora, optimizer, scheduler, and training. A refusal names a field by its group, for example optimizer.weightDecay. Every field is part of the configuration hash.

Specify any fields that you need to change. Each omitted field uses the default for the selected training method, so specifying one field in a group does not change the other fields.

The lora group describes the shape of the adapter:

Field Default Description

rank

8

The adapter’s rank. The endpoint that serves the model must support this rank.

alpha

Twice the rank

The scale at which the update is applied.

dropout

0

The probability that an adapted activation is dropped during training. Must be at least 0 and less than 1.

targetModules

default

The modules that the adapter covers, as a regular expression that matches whole module paths. default covers every linear module of the language backbone. A pattern that matches no module fails the run.

adaptExperts

false

Whether a mixture-of-experts model’s experts are adapted too. Off, the adapter covers attention and the shared MLP, as over a dense model, and the router and the experts stay frozen. Optimize refuses true, because the serving engine does not apply an adapter’s expert weights for Gemma 4, so an adapter over the experts could not be served.

The optimizer group describes what performs the update:

Field Default Description

kind

adamw

The algorithm. Only adamw is accepted.

learningRate

0.0001, or 0.000005 for dpo

The peak learning rate.

weightDecay

0

The decoupled decay applied to every adapted weight. Must not be negative.

maxGradNorm

1

The norm to which the gradient is clipped before the update. Each step reports the factor used to scale the gradient during clipping. See Metrics.

The scheduler group describes how the learning rate changes over the run:

Field Default Description

kind

linear

linear, cosine, or constant.

warmupRatio

0

The fraction of the run’s updates spent increasing to the configured learning rate. Must be at least 0 and less than 1.

minLearningRateRatio

0

The fraction of the learning rate that the schedule decays to, instead of zero. Only cosine decays to a floor. A floor with another kind is refused.

The schedule spans the run’s total updates: training.maxSteps if set, or the updates in all of the run’s epochs otherwise. A supervised branch’s schedule spans its own update budget. In a supervised run, the warmup is warmupRatio times the total updates, rounded up to a whole update. For example, a ratio of 0.03 over 939 updates warms up over 29 updates.

The schedule counts from zero. The first update takes the schedule’s value at 0, and update n takes its value at n - 1. With a warmup, the first update therefore runs at a learning rate of 0, and the rate reaches the full value at the first update after the warmup. Without a warmup, the first update runs at the full rate. A linear or cosine schedule reaches its end value after the last update, so the last update runs at a small rate above it. A resumed run restores the schedule’s position from its checkpoint, so each update takes the rate it would have taken without the interruption.

The training group describes the loop:

Field Default Description

epochs

3

The number of passes over the dataset.

maxSteps

None

The number of optimizer updates. A positive value overrides epochs.

microBatchPerGpu

Sized from maxSeqLength

The number of sequences in one forward and backward pass on one device. If tokensPerGpu is not set, this value times maxSeqLength is compared with the measured token limit per pass. Exceeding the measurement gives a warning in an advisory catalog and a refusal in a strict catalog. For a dpo run, this value counts pairs, and each pair counts as two sequences. See Measured training envelopes.

gradientAccumulation

Sized from microBatchPerGpu

The number of passes summed into one update. Multiplied by microBatchPerGpu, this is the number of sequences in one update on each device, or of pairs for a dpo run. On more than one device, each device packs its sequences into passes. The token count and tokensPerGpu determine the number of passes.

validationFrequencySteps

100

The number of optimizer updates between evaluations on the validation dataset. A run without a validation dataset performs no evaluations, regardless of this value.

seed

42

The seed that initializes the adapter, orders the data, and drives dropout. Zero uses the default. Specify another number to set a seed.

tokensPerGpu

microBatchPerGpu times maxSeqLength

The maximum number of tokens in one forward and backward pass on one device. A pass holds whole conversations up to this budget, with no padding. Set it to fill a device with conversations of different lengths without raising maxSeqLength. It must be at least maxSeqLength. Exceeding the measured token limit per pass gives a warning in an advisory catalog and a refusal in a strict catalog. For a flex_attention run, the budget is compared after padding to engine.pad_to_length_bucket. Optimize refuses it on a run on one device. The configuration hash includes it when set.

validateEachEpoch

false

Also evaluate on the whole validation dataset before the first update and at the end of each epoch, in addition to the evaluations every validationFrequencySteps updates. An evaluation due at both intervals runs once. A run that sets it without a validation dataset is refused. A full-state branch takes neither the evaluation before its first update nor the epoch evaluations. See [validation].

validationBatchSize

The conversations in one update

The number of validation conversations evaluated in one call. A larger value makes fewer calls, so an evaluation can take less time. It does not change the measurement. Each device splits its share of a call into passes as it does in training, so it does not change the memory a pass needs. Optimize refuses it on a run on one device. A run that sets it without a validation dataset is refused. The configuration hash includes it when set.

If omitted, microBatchPerGpu is the largest of 8, 4, 2, and 1 sequences whose tokens at maxSeqLength fit within 4096 tokens. For a dpo run it is the largest of 8, 4, 2, and 1 pairs whose two sequences each at maxSeqLength fit within 4096 tokens, so a pass holds 2 pairs at 1024 and 1 pair from 2048. It is 1 if none fits or if engineSettings sets data.use_dynamic_bsz to false. An omitted gradientAccumulation accumulates passes to 8 sequences in one update, or 8 pairs for a dpo run: 8 divided by microBatchPerGpu, and at least 1. The service resolves both values before hashing the run, so the configuration hash includes the values used for training.

On more than one device, one update takes microBatchPerGpu times gradientAccumulation conversations on each device. Each device receives the same number of conversations. Token counts are balanced across devices as evenly as conversation lengths allow. Each device then packs its conversations into passes of up to tokensPerGpu tokens, or trains one conversation per pass where engineSettings sets data.use_dynamic_bsz to false. The loss is the mean over all trained tokens in the update, across all devices. Each trained token has the same weight, regardless of the budget or how conversations are divided. Each epoch trains every conversation once. The last update in an epoch takes all remaining conversations. If their count is not a multiple of the device count, the update adds copies that train nothing. training runs get --settings and a dry run show the update size, and the budget or one conversation per pass.

Most numbers in these groups use their default when omitted or set to zero. optimizer.weightDecay set to 0 trains without weight decay, and only an omitted value takes the default. Optimize refuses a negative count, step budget or token budget and names the field.

Preference settings

preference sets what a dpo run optimizes beyond its hyperparams. Every field is part of the configuration hash. A fusedLoss of true hashes as if it were left out. See Direct preference optimization.

Field Default Description

beta

0.1

How far the trained adapter may move from the reference model. A smaller value lets it move further. It also scales each reward, so the margin of a run at a larger beta reads larger for the same preference. Must be greater than 0.

lossType

sigmoid

The loss over each pair’s margin. sigmoid is the loss of the original method. The margin keeps increasing, more slowly as it grows. ipo is a squared loss that regularizes each pair toward a target margin set by beta, where a smaller beta sets a larger target. Its target is per answer token, so the margin a run reports is not expected to settle at a given value.

fusedLoss

true

Whether the run may compute its loss with Liger’s fused kernel. false computes it with TRL’s own loss wherever the run would use the fused one, for example to compare the two. See Direct preference optimization.

Data

data specifies how the run reads its training data. It has the following fields:

Field Default Description

validationDataset

none

A second registered dataset that the run evaluates on while it trains, by registry reference. Optimize refuses an unregistered dataset or the training dataset itself.

truncation

error

What happens to a row longer than maxSeqLength. With error, Optimize refuses the run and names the row. Before the run claims a GPU, Optimize checks each chat dataset against the lengths that a dataset analysis measured under the base model. See [row-lengths]. With right, Optimize removes the tokens past maxSeqLength, and refuses the run if this removes a token selected for training. Without an explicit mask, the final assistant answer is trained under either assistantTurns setting, so right can shorten a row only when its explicit mask excludes every token past maxSeqLength. Optimize refuses right at submission unless the run sets loss.explicitLossMaskColumn. With drop, Optimize leaves a row longer than maxSeqLength out, training and validation alike, and the run reports how many training rows it left out. Optimize refuses a run where drop leaves no row in its training or validation dataset, and a run under drop that also asks for a smoke check. With oldest, Optimize removes whole turns from the start of a row longer than maxSeqLength, oldest first, until the row fits, and removes as few as it can. A turn is a user message and the assistant messages, tool calls and tool results after it, so a tool call and its result are removed together. Anything before the first user message, such as an assistant message that opens the conversation, is a turn of its own. The system prompt and the final turn always stay, so the final answer is still trained. Under assistantTurns: all, the answers of the removed turns are not trained. Optimize refuses a row that does not fit however many earlier turns it removes, and names the row. A long single-turn row has no earlier turn, so Optimize refuses it. With oldest_or_drop, Optimize trims a row the same way, and leaves out a row that does not fit however many earlier turns it removes, as drop does. Both apply to training and validation alike. Optimize reports how many training rows it trimmed and how many turns it removed. Optimize refuses oldest and oldest_or_drop for a dataset with images and for a run that also asks for a smoke check. Under both values, Optimize refuses a long row carrying the column named by loss.explicitLossMaskColumn. The mask covers the whole conversation. oldest_or_drop does not leave such a row out. Under oldest, Optimize checks each chat dataset before the run claims a GPU, by the length each row reaches with its oldest turns removed. See [row-lengths]. In a dataset with images, a row’s length is what the model’s processor counts, image tokens included, for every row of that dataset, with or without images of its own. A dataset of prepared token rows takes only error.

packing

false

Whether short rows are packed into one sequence. Packing makes a step cover more rows and makes two runs over the same data see different batches. Only a run on one device packs. A run on more than one device already packs whole conversations into each pass, up to hyperparams.training.tokensPerGpu, or trains one conversation per pass where data.use_dynamic_bsz is false, so Optimize refuses packing set to true for it.

shuffle

true

Whether the rows are shuffled before the first epoch. The order is drawn from hyperparams.training.seed, which is the run’s one seed.

A supervised dataset holds chat examples or prepared token rows, and the registered dataset records which, so there is no format field. The base model’s tokenizer provides the chat template for chat examples, so there is no chatTemplate field.

With a validation dataset, the run evaluates every validationFrequencySteps updates. With hyperparams.training.validateEachEpoch set to true, it also evaluates before the first update, which provides the baseline for later measurements, and at the end of each epoch. A resumed run does not repeat the measurement before the first update. A supervised run on more than one device also evaluates at its last update. After a resume, it evaluates again at a due step whose measurement was lost.

Each evaluation covers the whole validation dataset, less any rows that data.truncation set to drop or oldest_or_drop left out. Its loss is the mean over every supervised token in the set, so a long answer counts for more than a short one, and its perplexity is the exponential of that loss. The run evaluates validationBatchSize conversations in each call and weights each call’s loss by its supervised tokens, so the measurement is the same at any validationBatchSize, up to rounding. The run metrics contain the validation loss, its perplexity, and the training loss, and the run’s curves show the validation measurements under eval/loss and eval/perplexity. runs get shows the latest values. Read the losses together. A training loss that falls while the validation loss does not indicates that the run is memorizing its data.

Optimize checks the rows of a supervised run before the run claims a GPU, under every data.truncation value. Each chat dataset of the run, training and validation, is checked against a dataset analysis of the same examples under the run’s base model commit. Under error, Optimize checks each row’s length. Under oldest, it checks each row’s trimmed length, the shortest the row reaches with its oldest turns removed, so a row that fits once trimmed passes. Under drop, right and oldest_or_drop, Optimize refuses no row for its length, and the check refuses only the rows whose answer the chat template cannot train. A smoke check covers the training rows in its slice and every validation row. Datasets of prepared token rows are not checked, and a run whose datasets all hold prepared token rows skips the check. A dpo run under error is checked the same way, by each pair’s length: the longer of the prompt with the chosen answer and the prompt with the rejected answer. Its refusal names pairs. Under error and drop alike, Optimize also refuses a dpo run whose analysis found a pair the run cannot train, such as one whose answer the chat template does not render after its prompt, and names the first such pair. The run would refuse that pair under every data.truncation value.

  • At submission, Optimize refuses the run if an analysis that is already done finds a row that the check refuses. A dry run reports the same refusal. Submission does not start an analysis.

  • Otherwise, the run checks in PREPARING, at the stage checking row lengths against maxSeqLength. If the check refuses a row, the run fails before it claims a GPU.

    • A run that prepares its rows before claiming a GPU uses only a completed analysis. Without one, Optimize records not run and checks the lengths while it prepares the rows.

    • Otherwise, Optimize starts or reuses an analysis and waits up to 30 minutes by default.

  • Pipeline stages and branch submissions are not refused at submission. Their runs are checked in the run, before a GPU is claimed.

The refusal states how many rows are too long and which dataset holds them. It names up to 20 rows, longest first, with their lengths. It also names the analysis and the base model commit that measured the lengths. To fix it, raise maxSeqLength to at least the longest row, shorten or remove the rows, or set data.truncation to drop. Under oldest, the refusal names the rows' trimmed lengths, and the last fix is to set data.truncation to oldest_or_drop, which leaves such rows out.

Optimize refuses a row with an assistant message that calls tools, where the chat template refuses that message or leaves out its text or one of its calls, under every data.truncation value, whatever its length. The check refuses such rows first, by row number, with the reason for the first, and says to train another base model or remove the rows. Under oldest with a maxSeqLength below 512, the analysis cannot say which rows fit once trimmed, so the check records not run with the reason where a row is longer than maxSeqLength.

A long row that carries the column named by loss.explicitLossMaskColumn cannot be trimmed, so Optimize refuses it under oldest and oldest_or_drop. The column is a setting of the run, and an analysis is shared by every run over the same examples, so the analysis does not record which rows carry it. Under oldest, where the run sets loss.explicitLossMaskColumn and a row is longer than maxSeqLength, the check still refuses the rows that cannot fit even trimmed. Otherwise it records not run with the reason, and the run refuses a long row that carries the column when it reads it.

If the analysis fails or does not finish within the time limit, the run continues without the check. When the run reads the rows, it still refuses a row longer than maxSeqLength under error, and under oldest a row that cannot fit even trimmed. lengthCheck in the run’s record states the check’s status: checking, passed, refused or not run. It also names the analysis that each dataset was checked against, and the reason where the check did not run. If you cancel a run while it checks, the check is not run with the reason "the run was cancelled".

Loss

loss specifies which tokens in a conversation contribute to the run’s loss. Without an explicit mask, system and user tokens are excluded from training regardless of these settings. A mask from explicitLossMaskColumn is used as given. It has the following fields:

Field Default Description

assistantTurns

last

Which assistant messages are trained on: all, or last alone. On a single-turn row, last is the whole completion.

trainOnToolCalls

true

Whether the tool-call tokens in a selected assistant message before the answer count. Text in the same message is trained either way. Where the chat template does not render that text as one run before the calls, a run with this set to false refuses the row. This setting works with assistantTurns. The answer, a row’s final message, is always trained whole, its text and its calls, so under last this setting changes nothing.

trainOnToolResults

false

Whether a tool result’s own tokens contribute to the loss.

explicitLossMaskColumn

none

The JSON key that holds an explicit loss mask in an example. For an example with a non-null mask, Optimize uses that mask instead of the other loss settings. An example without the key, or with a null value, is masked by the other settings. A mask is a non-empty list of integer 0 and 1 values, with one entry per token of the conversation and its tools. The base model’s tokenizer and the chat template the run trains with determine these tokens. 1 selects a token for training and 0 excludes it. The first entry is 0, because no earlier token predicts the first token. The key cannot be messages, tools, prompt, completion or images. Optimize refuses a training or validation dataset where no example has a non-null mask. It refuses an example whose mask has the wrong length, holds other values, marks the first token, or selects no token. Optimize refuses an explicit mask for a base model whose chat template moves the system prompt into the last user message. Such a model trains only under the default loss settings.

A supervised run on more than one device computes the model’s output layer only for tokens included in the loss. This saves memory and time on conversations where most tokens are excluded from training. The loss and gradients are the same as in the full computation. This applies when the deployment computes log-probabilities in chunks of tokens, as it does by default. Optimize refuses this computation for a base model it has not verified. A model that soft-caps its final logits, such as Gemma, retains that cap.

Optimize refuses a loss or data group on a reinforcement run. A reinforcement run takes its loss from the reward over a rollout and reads its dataset as rollout prompts. A group that contains only defaults is accepted. A branch to a reinforcement method inherits neither group, so a group on the supervised parent does not cause the branch to be rejected.

Reinforcement hyperparameters

rlHyperparams has the following fields and the same lora, optimizer, and scheduler groups as hyperparams. Three defaults differ for reinforcement training: optimizer.learningRate is 0.00001, optimizer.weightDecay is 0.01, and scheduler.kind is constant. The reinforcement engines train the adapter without dropout and follow a constant or cosine schedule. Optimize refuses a lora.dropout above zero or a linear schedule. The reinforcement engines do not render a row’s tools, so Optimize refuses a reinforcement run over a dataset whose rows offer tools. See Conversations and tool calls.

Field Default Description

epochs

1

The number of passes over the prompts.

maxSteps

None

A cap on optimizer updates. A positive value overrides epochs.

groupSize

4

The number of completions sampled per prompt. Optimize computes group statistics from these completions.

promptsPerStep

8

The number of prompts rolled out per step. A step generates promptsPerStep times groupSize completions. Set it to a multiple of execution.gpus so each device receives the same number of prompts.

klBeta

0.04

The KL penalty toward the reference policy. Zero disables the penalty.

temperature

0.7

The sampling temperature. Zero uses greedy sampling.

maxCompletionLength

256

The longest completion sampled, in tokens.

stalenessThreshold

0

Not supported. Optimize refuses a value above zero: every step trains on-policy, on rollouts generated from the current weights. Leave it out or set it to 0.

An omitted field takes its default, whether you omit rlHyperparams or only that field. Zero is valid for klBeta, temperature, maxSteps, and stalenessThreshold, so Optimize keeps a 0 you set for any of them. optimizer.weightDecay set to 0 trains without weight decay, and only an omitted value takes the default of 0.01. The other fields use their defaults when omitted or set to zero. akka training runs get --settings shows the temperature and the KL weight a run trains with on its Rollouts line.

Model settings

The model object has the following fields:

Field Default Description

precision

BF16

How Optimize loads the base model’s weights: BF16 or INT4. With INT4, the adapter trains against the quantized base, which is a different model from the one used with BF16. Only a run on one device trains at INT4. Optimize refuses INT4 for a run on more than one device when you submit the run.

gradientCheckpointing

true for a dpo run. Otherwise true if maxSeqLength is greater than 4096

Whether the backward pass recomputes activations instead of keeping them. This reduces memory use at the cost of more computation. If omitted, the value is derived from the method and maxSeqLength, and the derived value is part of the hash. A dpo run recomputes activations by default because each pair runs two answers through both the model it trains and the reference model. Set false to turn it off.

attentionImplementation

sdpa. flex_attention for a Gemma 4 run on H200 devices, see Default kernels by model type

The kernel used by the attention layers: sdpa, eager or flex_attention. sdpa is PyTorch’s fused implementation. It is faster and uses less memory where the architecture supports it. eager is the architecture’s own implementation and is always available. Use it when a model card requires it or when the fused kernel does not work correctly on your hardware. flex_attention is accepted only for a model it is verified on. See Attention in a packed pass.

weightPrecision

The deployment’s, usually fp32

The precision of the trained model’s weights: fp32 or bf16. At bf16, the weights take half the memory, but an update the size of a typical learning rate can round away and the run learns less. Use bf16 only when the fp32 weights do not fit the device. Part of the configuration hash. See Memory settings.

Attention in a packed pass

A supervised run on more than one device packs several conversations into one pass. Under sdpa and eager, each conversation attends only to its own tokens, so its log-probabilities do not depend on the other conversations in the pass.

flex_attention is offered only for a model type it is verified on: the types of Gemma 4 E2B, E4B and 12B. Before claiming devices, Optimize refuses a run that names it for a model of another type, and the error names the model. A dry run reports the same refusal.

A flex_attention run pads every pass to a multiple of engine.pad_to_length_bucket, 1024 tokens by default, because flex attention has no kernel for a pass shorter than 128 tokens. Optimize refuses a run that turns engine.pad_to_length off or sets the bucket below 128. Flex attention also needs a device with enough shared memory per block. On a smaller GPU, the run fails at startup and the error names the device.

Default kernels by model type

A supervised run under verl on H200 devices that does not set attentionImplementation or a liger.* setting takes its model type’s default. The defaults apply only on H200 devices. A run on any other device, or on a deployment whose catalog names no accelerator type, takes no defaults.

Model type attentionImplementation Liger kernels

Gemma 4 E2B and E4B

flex_attention

liger.rms_norm and liger.geglu

Gemma 4 12B

flex_attention

None

Any other model

sdpa

None

A default applies to every model of its type, so Gemma 4 E2B and E4B take the same one.

Flex attention trains more tokens a second than sdpa, more so on longer sequences. On Gemma 4 12B it can take more device memory than sdpa. Set attentionImplementation to sdpa for a run that does not fit a device under the default.

A run that sets attentionImplementation, including to sdpa, keeps its value. A run’s config, as training runs get -o json shows it, leaves attentionImplementation out where it is sdpa, so a plan written from it takes the defaults. A run that sets a liger.* setting, to true or false, keeps its value for that kernel. A run on one device trains under TRL and takes no defaults. A Liger default applies only where the deployment lets a run set that kernel. A run keeps sdpa where flex attention’s padding would refuse it: where it turns engine.pad_to_length off, sets the bucket below 128, or carries a pass that, padded to the bucket, exceeds the catalog entry’s limit.

The defaults are written into the run’s configuration, so they are part of the configuration hash, and training runs get --settings and training runs start --dry-run --show-resolved show them. A branch trains with the kernels its parent trained with.

Engine settings

Use engineSettings to set the training engine’s settings directly. Most settings affect only memory use and speed. Use them to fit your model to its devices. Precision settings change the arithmetic used during training. engineSettings has the following fields:

Field Description

engine

verl. Required. No other engine takes engine settings.

settings

Each setting’s value, by its name. Required.

A run can set the following settings, where its deployment allows each one:

Setting Default Description

engine.forward_prefetch

false

Fetches the next layer’s weights while the current layer computes. A step can be faster, and it needs more memory.

engine.reshard_after_forward

true

Frees each layer’s gathered weights after the forward pass. false keeps them for the backward pass, which can be faster and needs more memory.

engine.pad_to_length

false, true for a flex_attention run

Pads each pass up to a multiple of engine.pad_to_length_bucket tokens, so the kernels see fewer shapes. It can speed up a run whose sequence lengths vary. Padding requires additional computation.

engine.pad_to_length_bucket

1024

The multiple engine.pad_to_length pads to.

engine.use_no_sync_for_gradient_accumulation

true for a run that trains an adapter

Accumulates the gradients from each pass on each device and combines them across devices once, after the last pass. Each device holds the whole gradient until then, which is small for an adapter. false combines them after every pass, so every device waits at each pass for the device whose pass takes longest. That is slower when passes differ in length, such as one conversation per pass. With bf16 computation, a device adds up its passes in bf16 before the gradients are combined in fp32, so the two differ by rounding. A reinforcement run that trains the full model keeps false.

engine.mixed_precision.param_dtype

bf16

The data type used for computation in the forward and backward passes: bf16 or fp32. fp32 is slower and needs more memory. Supervised runs only.

engine.mixed_precision.reduce_dtype

fp32

The data type used to sum gradients across devices: bf16 or fp32. bf16 transfers half as many bytes but reduces the precision of the sum. Supervised runs only.

engine.mixed_precision.buffer_dtype

fp32

The data type used to store the model’s buffers, such as its rotary tables: bf16 or fp32. Supervised runs only.

compile.recompile_limit

8

The maximum number of compiled versions of a function that Torch compile keeps for one layer. For example, each sequence shape can have its own version. After this limit, the function runs without compilation. Supervised runs only.

compile.accumulated_recompile_limit

256

The maximum number of compiled versions of a function that Torch compile keeps across all layers that run it. Supervised runs only.

compile.fail_on_recompile_limit_hit

false

true fails the run when either recompile limit is reached. false runs the function without compilation from then on. This is slower and does not report that the limit was reached. Supervised runs only.

liger.rms_norm

false. true for a Gemma 4 E2B or E4B run on H200 devices, see Default kernels by model type

Uses Liger’s fused kernel for the model’s RMS normalization layers. This can improve speed and reduce memory use. On Gemma 4 E2B and E4B, the kernel applies to every RMS normalization layer in the language model except the attention value norm v_norm, post_per_layer_input_norm and per_layer_projection_norm. These keep their own forward methods. Not available for Gemma 4 12B. Supervised runs only, for model types verified for this kernel.

liger.geglu

false. true for a Gemma 4 E2B or E4B run on H200 devices, see Default kernels by model type

Uses Liger’s fused kernel for a GeGLU feed-forward layer, such as Gemma’s. This can improve speed and reduce memory use. Not available for Gemma 4 12B. Supervised runs only, for model types verified for this kernel.

checks.start

true

Checks the run’s numerics before its first update. false trains without the checks. Supervised runs only. See Checks before the first update.

checks.head.max_logprob_error

0.01

The head check’s bound on the largest difference in a log-probability. Supervised runs only. See Checks before the first update.

checks.head.max_gradient_error

0.005

The head check’s bound on the gradient’s difference relative to its size. Supervised runs only.

checks.isolation.max_logprob_error

0.0001

The isolation check’s bound on how far a conversation’s log-probabilities move. Supervised runs only, with data.use_dynamic_bsz left on.

checks.packing.max_loss_error

0.005

The packing check’s bound on the difference in the step’s mean loss. Supervised runs only, with data.use_dynamic_bsz left on.

checks.reference.max_logprob_error

1.0 for conversations of up to 4096 tokens, and longer ones scaled by the length

The reference check’s bound on the largest difference in a log-probability at the run’s precision. Supervised runs only. See Checks before the first update.

checks.reference.mean_logprob_error

Calculated from the run’s rounding, at least 0.03

The reference check’s bound on the mean difference in a log-probability at the run’s precision. Supervised runs only. See Checks before the first update.

checks.reference.max_nll_error

Calculated from the measured differences, at least 0.005

The reference check’s bound on the difference in the mean loss at the run’s precision. Supervised runs only. See Checks before the first update.

checks.reference.fp32_max_logprob_error

0.002 for conversations of up to 4096 tokens, and longer ones scaled by the length

The reference check’s bound on the largest difference in a log-probability at fp32. Supervised runs only. See Checks before the first update.

checks.reference.fp32_mean_logprob_error

0.0001

The reference check’s bound on the mean difference in a log-probability at fp32. Supervised runs only.

head.chunk_tokens

512

The number of positions the output head computes log-probabilities for at once. Each position in a chunk holds its logits over the whole vocabulary, so a smaller value needs less memory and launches more kernels. Any whole number above 0 is accepted, and the log-probabilities and gradients are the same at every value up to rounding. Applies when the deployment computes log-probabilities in chunks, which it does by default. Supervised runs only. A reinforcement run computes 512 at a time.

checkpoint.compare_replicas

true

With memory.sharding set to none, compares every device’s optimizer state with the first device’s before the first save of each attempt, and fails the run if any differs. Only the first device’s copy is saved. false skips the comparison. Supervised runs only. See [recovery].

rollout.enable_chunked_prefill

true

Splits a long prompt’s prefill into chunks. Reinforcement runs only.

rollout.enable_prefix_caching

true

Reuses the cache for prompts with a common prefix, such as the completions of one group. Reinforcement runs only.

rollout.enforce_eager

false

Runs the rollout engine without CUDA graphs. It starts faster and uses less memory, but generates more slowly. Reinforcement runs only.

rollout.max_num_seqs

The step’s rollouts

The most sequences the rollout engine generates at once. Reinforcement runs only.

rollout.max_num_batched_tokens

The larger of maxSeqLength and the step’s rollouts

The most tokens the rollout engine processes in one batch. Reinforcement runs only.

data.use_dynamic_bsz

true

Packs whole conversations into each pass, up to hyperparams.training.tokensPerGpu. false trains one conversation per pass, so a step takes one pass for each of its conversations. The loss is the same mean over the update’s trained tokens either way. With false, an unset hyperparams.training.microBatchPerGpu is 1. A run that sets it above 1, or sets tokensPerGpu, is refused. Supervised runs only.

The following example sets two of them:

"engineSettings": {
  "engine": "verl",
  "settings": {"engine.forward_prefetch": true, "rollout.enforce_eager": true}
}

The settings are part of the configuration hash. training runs get --settings and training runs start --dry-run --show-resolved show them after the engine, one line for each group of settings, such as compile or liger. Settings reported on other lines are omitted from this list. Explicit data types appear on the precision line, and data.use_dynamic_bsz set to false appears on the batch line. For a supervised run on more than one device, both commands also show the data types used for training. These include the parameter type from config.model.weightPrecision and the types used for computation, gradient reduction and buffers. A run that specifies engine settings uses the engine they name. Before claiming devices, Optimize refuses the run if the catalog lists no such engine for the device profile. An advisory catalog considers engines across the profile’s envelopes, regardless of the requested GPU count. A strict catalog uses only the matching capability’s engines. It also rejects a supervised run on one device that specifies them.

Optimize also refuses a run, naming the setting, when the setting:

  • is not one the deployment allows,

  • is for the rollout engine and the run is supervised,

  • applies only to supervised runs and the run is a reinforcement run, or

  • has a value of the wrong kind, such as a number for engine.forward_prefetch.

Before claiming devices, Optimize also refuses your run if you enable a Liger kernel for a model that is not verified for that kernel. The error names the kernel and the model. Liger’s kernels are verified for Gemma 4 E2B and E4B, and not for Gemma 4 12B, which takes flex attention alone. A supervised Gemma 4 E2B or E4B run under verl on H200 devices turns both kernels on by default. See Default kernels by model type.

A run cannot specify a setting already determined by its descriptor, a setting that names a path or code, or the seed.

A branch keeps its parent’s applicable engine settings. For example, a supervised branch drops rollout settings.

Execution settings

execution has the following fields:

Field Default Description

gpus

1

The number of devices that the run claims. A supervised run on one device and one on more than one device train differently, and several fields on this page apply to only one of them. A run that fits on one node can use part of it. A larger verl run must fill whole nodes. See Measured training envelopes.

rolloutGpus

0

Not supported. Optimize refuses a value above zero: the rollout engine runs on the training devices. Leave it out or set it to 0, and count every device in gpus. See Where a reinforcement run generates rollouts.

microBatchSequences

chosen by the job

The maximum number of sequences per device in one forward and backward pass. It changes only peak memory. Accumulation combines the same rollouts in the same optimizer step.

saveEvery

off

The periodic checkpoint interval, in optimizer updates. When it is off and saveEachEpoch is not set, the run writes only one checkpoint at the end. An interrupted run therefore has no checkpoint from which to restart.

keepAdapters

5

The number of unreferenced adapter checkpoints retained for the run. Adapter checkpoints saved at the end of an epoch do not count toward this limit.

keepRecoverable

1

The number of unreferenced saves that keep their training state, so that a resume can start from any of them. Must be at least 1. Each retained save uses storage for its training state. See [recovery].

resumableSaveEvery

10

When the run saves faster than it uploads, the maximum number of saves before the run chooses one to upload with its training state. Must be at least 1. See Saves that wait to upload.

resumableSaveMinutes

5

When the run saves faster than it uploads, the maximum number of minutes after a training state finishes uploading before the run chooses another save to upload with its training state. Must be at least 1. See Saves that wait to upload.

saveEachEpoch

false

Set this to true to also save at the end of each epoch in a supervised run. These adapters are retained without counting toward the keepAdapters limit. Each adapter is registered as a version of the run’s model. The save at the run’s last update is the run’s own model and is not registered twice. A deployment that stores checkpoints on a shared filesystem refuses this setting. See Epoch adapters.

deviceProfile

chosen by the catalog

The accelerator to train on, as the catalog names it. Omit it when a deployment has only one kind of device. The device must exist in the catalog. An advisory catalog admits an unmeasured request with a warning. A strict catalog requires a matching capability.

engine

chosen by the catalog

The training engine, trl or verl, among the engines that the catalog lists for the device. Use it to test an engine. A deployment can refuse it.

rollout.memoryShare

The deployment’s, usually 0.5

The share of each device that a reinforcement run’s rollout engine takes for its weights and its cache. See Memory settings.

memory.offload.weights

The deployment’s, usually false

Whether the trained model’s weights move to host memory between passes. See Memory settings.

memory.offload.optimizer

false

Whether the optimizer’s state moves to host memory between updates. See Memory settings.

memory.offload.activations

false

Whether activations saved during the forward pass stay in host memory until the backward pass needs them. See Memory settings.

memory.sharding

all

How your trained model is distributed across your run’s devices. all splits it across the devices. none keeps a complete copy on each device. See Memory settings.

hostMemoryGiB

none

The host memory required to load the run’s model. Optimize refuses the run at submission if this value exceeds the host memory listed for the named device. This field does not set the run’s host memory allocation. Omit it if the device lists no host memory, because the device then accepts any value.

Execution settings are recorded with the run and are not part of the configuration hash.

For example, to save every 50 updates, keep the three latest saves resumable, and retain each epoch’s adapter separately, set:

"execution": {"saveEvery": 50, "keepAdapters": 3, "keepRecoverable": 3, "saveEachEpoch": true}

Where the deployment stores checkpoints in object storage, each save is a snapshot. A save holds the adapter weights and the full training state: the optimizer, the scheduler, the random-number generators and the step reached. The run also saves at its last update. Each time a save is recorded, Optimize retains the following and deletes the rest:

  • The adapter weights of the keepAdapters latest saves that nothing references, the save at the last update included.

  • The training state of the keepRecoverable latest saves that nothing references. Earlier training state is deleted, so of the unreferenced saves only those can be resumed.

  • The training state of the save that a retry or a resume of the run would restore, whatever its rank.

  • The adapter weights of every epoch save, until you release its epoch hold with training snapshots release. Their training state follows the rule above.

  • Any part of a save that a branch or a pin references, for as long as the reference exists. A registered model version keeps its own copy of the weights and does not keep the save.

A deployment that stores checkpoints on a shared filesystem lists no snapshots, so these retention rules do not apply, and it refuses a keepRecoverable above 1.

A retry, and training runs resume without --step, restore the newest save. To resume a paused run from an older save that still holds its training state, name its step with training runs resume RUN_ID --step STEP. training runs snapshots lists the saves and their steps. A branch with --restore trainingState can start from any of them by its snapshot ID. See Resume a run.

Training state is the optimizer’s two moments for every trained parameter, plus the adapter, the scheduler and the random-number generators, for every device. The adapter and its moments are held at fp32, and the training state also holds an exported copy of the adapter. With either value of memory.sharding, a save’s training state is the same size regardless of the number of devices.

With none, before the first save of each attempt, the run compares a hash of every device’s optimizer state with the first device’s, and fails if any differs. The comparison takes longer the larger the optimizer state. Set the engine setting checkpoint.compare_replicas to false to skip it.

After a resume from an older save, the later saves from the earlier attempt remain until retention deletes them. Retention ranks the resumed attempt’s saves first. It deletes unreferenced saves from the earlier attempt before those from the resumed attempt.

Memory settings

Use these settings to adjust memory use, training speed, or learning. You cannot set them on a supervised run on one device. If you omit a setting with a deployment value, your run uses that value. execution.memory.offload.optimizer and execution.memory.offload.activations default to false. execution.memory.sharding defaults to all.

Setting Trade

execution.rollout.memoryShare

A larger share gives the rollout engine more room for its cache, so it generates more completions at once. It leaves less memory for training. A model that fills most of the device needs a larger share to load at all. A reinforcement run only, since a supervised run has no rollout engine.

config.model.weightPrecision

bf16 halves the memory that the weights take. A small update can round away, so the run can learn less than the same run at fp32.

execution.memory.offload.weights

true moves the weights to host memory between passes, which frees the device for the rollout engine. Each step takes longer because the weights move back and forth.

execution.memory.offload.optimizer

true keeps the optimizer state in host memory. It moves the state to the device only during each update. If your run trains an adapter, it keeps optimizer state only for the adapter. In that case, offloading optimizer state frees less memory than offloading weights. Each update takes longer. The state uses host memory.

execution.memory.offload.activations

true moves activations saved during the forward pass to host memory, one layer at a time. It returns each layer’s activations to the device when the backward pass needs them. This frees the most memory on long sequences, where activations fill device memory. Each step takes longer. The activations use host memory, which hostMemoryGiB does not count. You cannot use this setting with an image-text model, such as Gemma 4, whose vision layers do not run on text rows.

execution.memory.sharding

all splits the weights, gradients, and optimizer state across your run’s devices. Each pass gathers the parts it needs. none keeps all of them on each device. The devices combine their gradients once per update, or after each pass if engine.use_no_sync_for_gradient_accumulation is false. Use none when your model’s training state fits on one device. A step does not gather model state, so it can be faster. On one device, both settings have the same effect.

The deployment determines which settings a run can specify and the allowed range of each. Optimize refuses a run before it claims devices, and names the setting, when:

  • the deployment does not allow a run to specify the setting,

  • memoryShare is outside the deployment’s range, or the run is supervised,

  • weightPrecision is not one the deployment allows,

  • offload.activations is true for an image-text model, or

  • no engine that the catalog lists for the run can honor the setting.

If you set any of these settings, your run uses an engine that supports them. Optimize refuses it if the catalog lists no such engine for its device profile. A strict catalog also requires a matching measured capability.

A run records the value of each setting when it is created. A later change to the deployment’s values does not change the run, a branch of it, or a retry. training runs get --settings shows the recorded values for a run that takes these settings and marks the ones the run stated. training runs engine-config shows the values the job passed to the training engine.

Where a reinforcement run generates rollouts

A reinforcement run generates rollouts on the same devices used for training. During each step, every device generates its share of the rollouts and then takes part in the update. To run on four GPUs, set gpus to 4 and leave rolloutGpus out. Separate devices for generation, such as three for training and one for rollouts, are not supported.

The deployment states the most devices a run can use. It counts gpus and rolloutGpus together against that limit. Catalog envelopes record measured counts. In an advisory catalog, a run can request other counts with a warning. A model outside every training and serving envelope is admitted with an unmeasured warning where the engines support it. See Measured training envelopes.

Start a run

Validate a descriptor, start the run, and wait for it to finish:

akka training runs start -f run.json --dry-run
akka training runs start -f run.json -o json --jq .runId
akka training runs wait RUN_ID --exit-status

--dry-run validates the descriptor against the service without training. Without it, the command validates the descriptor and asks for confirmation. --force skips the prompt. Without a submissionId in the descriptor, every submission starts a new run, so submitting the same descriptor twice trains it twice. With one, a later submission reaches the run that the first started.

Add --show-resolved to --dry-run to see the configuration and execution settings that the run would use. The output includes every default and every resolved dataset and reward reference:

akka training runs start -f run.json --dry-run --show-resolved

This command reports the same values that training runs get --settings reports after the run exists. Use it to resolve a descriptor before starting the run. -o json includes the resolved configuration as structured data.

For a supervised run, dpo included, a dry run also shows a Lengths section for the training dataset and any validation dataset, with or without --show-resolved. Each figure names its source:

  • With a completed dataset analysis of the same examples, base model commit and Optimize version, Optimize names the analysis, the tokenizer and chat template it measured under, and the Optimize version. It shows the row count, the median, 90th percentile, 99th percentile and longest row lengths, the rows longer than the run’s maxSeqLength, and the suggested maxSeqLength. For a smoke check, Optimize counts only the training rows in its slice. It counts every validation row. If Optimize cannot read the count in time, it reports the other figures, marks the count as unknown and explains why.

    • Under data.truncation: oldest or oldest_or_drop, Optimize also shows how many of the long rows fit with their oldest turns removed, which the run trims, and how many cannot fit even trimmed, with the figure of the analysis that gives the count. The counts are of measured lengths. Optimize refuses these rows under oldest. Under oldest_or_drop, Optimize leaves them out unless their mask or their tool calls cause a refusal. A long row that carries the column named by loss.explicitLossMaskColumn is counted as trimmed, though the run refuses it. How many turns the run removes depends on the limit, so the analysis does not give it. The run reports it as dataFit.turnsRemoved.

    • Under every value, Optimize shows how many rows hold an assistant message that calls tools which the chat template cannot train as it was written. The run refuses these rows whatever their length.

  • When no analysis exists, Optimize shows the training datasets analyze DATASET --model BASE_MODEL command that starts one. If Optimize cannot read the analysis status, it explains why and shows no command. If an analysis is pending or running, Optimize names it and its status. If the last analysis failed, Optimize names it and gives the reason. Optimize shows the command only if another attempt could succeed. A failure that Optimize would report again for the same examples, base model commit and Optimize version gets no command. If the run’s base model resolves to no commit, Optimize states that an analysis needs one and shows no command. Optimize does not start an analysis during a dry run.

  • For prepared token rows, Optimize shows the longest row recorded by the dataset version, and the tokenizer that prepared the rows.

A reinforcement run shows no Lengths section. In the API, resolved.lengths carries the same figures. See [row-lengths].

Smoke check

A smoke check trains on the first rows of the dataset until the model has memorized them. It then checks that the model reproduces them. If a model cannot reproduce rows that it has seen repeatedly, the data, masking, or tokenizer has a problem.

A smoke check takes everything else from the descriptor. Add smoke to the descriptor to run one:

Field Default Description

sliceSize

200

The rows taken from the start of the dataset. These are also the rows scored afterwards.

maxSteps

2000

The cap on optimizer updates. Raise it with sliceSize, so the slice can still be memorized.

threshold

0.98

The reproduction score a working run reaches. It must not be above 1.

training runs get shows the verdict. training runs wait --exit-status exits with a non-zero status when the check fails.

Base models

A run can specify only a base model that the deployment allows and that someone has selected. To list the allowed models and select one, see Base models.

google/gemma-4-26B-A4B-it trains under TRL on one H200 and under verl on several, with sft, grpo and gspo. A run on several devices is not measured yet, so Optimize admits it with a warning. Its default adapter covers attention and the shared MLP, and leaves the experts and the router frozen.

If a descriptor specifies a model that the deployment does not allow, Optimize refuses the run. Specify a model from the list, or ask your operator to allow the model that you need.

Optimize checks the adapter rank and the engine’s rules before admitting a run. In advisory mode, a model outside every training envelope can train with a warning where the engine supports it. Strict mode refuses such a run.

Measured training envelopes

A training envelope records a measurement for a model size, method, precision, engine, device profile, GPU count and sequence length. It can also state an active parameter limit and a token limit per pass. To see the measurements, run:

akka training base-models get Qwen/Qwen3-8B
akka training base-models envelopes

The first lists one model’s measurements, and the second every measurement of the deployment. Both label them Measured training envelopes. training base-models envelopes also shows whether the catalog is advisory or strict, and so does training capacity.

Training envelopes are advisory by default. You can request an unmeasured configuration if the engine supports it. For example, a catalog measured at 4 GPUs and 32,768 tokens can admit a run at 6 GPUs and 65,536 tokens. Optimize warns that the run is unmeasured and may run out of device memory. A run within a measured envelope has no envelope warning.

A dry run shows the warning with or without --show-resolved. Starting a run also shows it. training runs get shows the warning without --settings. The run keeps its warning even if the catalog changes. In the API, dry run, start and get responses carry the top-level warnings list when it is non-empty. A warning does not change the configuration hash or run id.

An operator can set trainer.base-model-catalog.strict-envelopes = true in the applied catalog. A strict catalog refuses configurations outside its measurements. This includes an unmeasured device count, model size, active parameter count, method, precision, engine, sequence length or token count per pass.

Other limits apply in either mode:

  • The training engine must support the model’s architecture.

  • A run that names no device chooses a profile with training envelopes. In advisory mode, you can name a profile with serving envelopes alone and train with a warning. Strict mode refuses it.

  • A stated hostMemoryGiB must fit the profile’s host memory, where the profile states one.

  • TRL trains on one GPU. Supervised training under verl needs at least two GPUs.

  • A reinforcement run’s promptsPerStep must be divisible by gpus.

  • Device claims must fit the deployment’s GPU ceiling and capacity rules.

  • A verl run that fits on one node uses one node. A larger run must fill whole nodes. The runtime reads the largest GPU count on a live Ray node when the job starts. With 8 GPUs per node, 6 GPUs use one node and 16 use two. A request for 14 GPUs is refused. A split of 7 + 7 is not supported.

The engine must support every setting you request. See Execution settings for device and host memory settings.

Methods

Supervised fine-tuning

sft learns from the assistant messages in the dataset. Use it when every example has a known correct output, such as classification, extraction, or another task with a manually written or curated dataset.

Checks before the first update

A supervised run on more than one device checks its numerical results before its first update, using its model, devices, and first step’s rows. While the checks run, the run is PREPARING at the stage checking the run before its first update. Each attempt checks again.

Check What it compares Bounds, with their defaults

head

The output head that computes the log-probabilities, on fp32 copies of a few positions of the first pass. Its log-probabilities against the model’s own head, and its gradient against the exact gradient of the model’s own head

checks.head.max_logprob_error 0.01, the largest difference in a log-probability. checks.head.max_gradient_error 0.005, the gradient’s difference relative to its size

packing

The step’s mean loss with its conversations packed as the run packs them, against one conversation per pass

checks.packing.max_loss_error 0.005

isolation

Two conversations packed into one pass, with every token of each replaced in turn. The other conversation’s log-probabilities must not move

checks.isolation.max_logprob_error 0.0001

reference

Two conversations through the run’s training path, against plain Transformers on each conversation alone, at the run’s precision and at fp32

At the run’s precision: checks.reference.max_logprob_error, 1.0 or more as described below, checks.reference.mean_logprob_error, calculated from the run’s rounding as described below, at least 0.03, and checks.reference.max_nll_error, the difference in the mean loss, calculated as described below. At fp32: checks.reference.fp32_max_logprob_error, 0.002 or more as described below, and checks.reference.fp32_mean_logprob_error 0.0001

The difference in the mean loss is a mean over the positions the reference check compares, which are the supervised tokens of its two conversations. Conversations with few supervised tokens provide few positions to compare. A large rounding difference at one position can then have a large effect on the mean. Rounding can shift positions in either direction. A fault in the run’s computation shifts every position in the same direction. When you leave checks.reference.max_nll_error out, its bound is calculated from the differences the check measures: the largest difference divided by the number of positions, plus three standard errors of the mean of the other differences, and at least 0.005. For example, over 80 positions with one differing by 0.61 and the rest rounded by about 0.01, the bound is about 0.011, and the measured difference of 0.0078 passes. If every position is shifted by 0.015, the bound is 0.005, and the check passes with a warning. A shift of more than 0.025 fails it. training runs get -o json reports positions and the bound used, under startChecks.

How far your run’s precision moves log-probabilities depends on the model and the device. A small model in bf16 moves them further. The reference check computes its two conversations through plain Transformers at the run’s precision and at fp32, and reports the mean difference between the two as roundingError. When you leave checks.reference.mean_logprob_error out, its bound is three times roundingError, and at least 0.03. A sound run differs from plain Transformers by up to about twice the rounding, because each side rounds on its own. At fp32 there is no rounding, and the bound is 0.03. The packing bound stays 0.005. A small model can miss it while the fp32 figures are within theirs. In that case, state checks.packing.max_loss_error. The fp32 bounds are not calculated. Widen a bound at the run’s precision only when the fp32 figures are within theirs.

Past 4096 tokens, the largest differences are a coarse guard and the mean differences are the sensitive tests, at the run’s precision and at fp32. Deep in a long conversation a model can enlarge rounding at a few positions far past the rounding itself, while the means stay small. Computing the same passes at fp64 shows no difference. When you leave checks.reference.max_logprob_error or checks.reference.fp32_max_logprob_error out, its bound is 1.0 or 0.002 for conversations of up to 4096 tokens, and that times the longer conversation’s length divided by 4096 for longer ones: 4.0 and 0.008 at 16384 tokens, and 8.0 and 0.016 at 32768. The mean bounds do not grow with the length. training runs get -o json reports the longer conversation’s length as tokens, and how deep in its conversation each precision’s largest difference is as worstDepth and fp32WorstDepth.

For a model that routes each token to some of its experts, such as google/gemma-4-26B-A4B-it, a router score near a tie can choose different experts in the reference and in the run, and that moves the token’s log-prob by far more than rounding. The reference check then still measures and records logProbError, fp32LogProbError and fp32MeanLogProbError, but bounds them only where you state their bounds. training runs get -o json lists each one under leftOut, with the reason, and shows routed as 1. meanLogProbError and nllError keep their bounds, and the head, packing and isolation checks are unchanged.

Each bound is an engine setting that your deployment must allow. A bound you leave out takes its default. For example, to set the reference check’s maximum log-probability error at the run’s precision to 0.002:

"engineSettings": {
  "engine": "verl",
  "settings": {"checks.reference.max_logprob_error": 0.002}
}

A bound must be a number of at least 0. Optimize refuses a bound when checks.start is false, and a packing or isolation bound when data.use_dynamic_bsz is false, since those checks then do not run.

A check that does not apply is skipped, and the record says why. head runs only when the run computes log-probabilities with fused kernels. packing and isolation run only when a pass can hold more than one conversation. reference runs only on an attempt that starts from a fresh adapter, so a resumed attempt, a branch and a warm start skip it. It is also skipped when the first step has fewer than two conversations, or when the base model at fp32 does not fit beside the run on the first GPU.

If a head or isolation value exceeds its bound, the run fails before it trains. A packing or reference value over its bound but at most 5 times the bound passes with a warning, and the run trains. A value more than 5 times its bound fails the run. A bound you set scales that limit too. training runs failure shows the check, the measured value, the bound, the limit where it differs from the bound, and the setting that sets the bound. Whether the checks pass or fail, Optimize keeps their values and bounds, and training runs get shows them. With -o json, startChecks holds one entry for each attempt. Each check’s boundSettings names the setting of each bound, warned lists the values that passed with a warning, and refusalBounds holds the limit of each bound.

akka training runs get RUN shows one line for the checks of the last attempt: how many passed and how long they took, with each value that passed with a warning, or the first check that failed with each value over its limit. --checks adds one row for each check, with its verdict, each value against its bound and, for a value over its bound, its limit, and the setting that sets the bound. -o json includes the whole record under startChecks.

The checks add roughly the time needed to load the base model once, run two forward passes over the first step, and run three short passes over two of its conversations. To train without them, set the engine setting checks.start to false. Your deployment must allow the setting.

Direct preference optimization

dpo trains an adapter on preference pairs: a prompt with a chosen answer and a rejected answer. It makes the chosen answer more likely than the rejected one, relative to a reference model, and does not generate anything while it trains. Use it when you can say which of two answers is better but cannot write the best answer for every prompt, or to move a supervised model away from answers it gives that you do not want.

The training dataset, and the validation dataset where you name one, must be preference datasets. See Preference pairs. Optimize refuses a dataset of chat examples or prepared token rows for a dpo run, and refuses a preference dataset for a run of another method. The refusal names the dataset.

A dpo run takes hyperparams and preference. Its learning rate defaults to 0.000005. The following config trains from the base model:

"config": {
  "dataset": "triage/preferences",
  "method": "dpo",
  "preference": {"beta": 0.1, "lossType": "sigmoid"},
  "data": {"validationDataset": "triage/preferences-held-out"}
}

A dpo run recomputes activations in the backward pass unless its config sets model.gradientCheckpointing. Each pair runs two answers through both the model the run trains and the reference model, so a dpo run needs much more device memory than a supervised run at the same maxSeqLength. To keep the activations instead, set model.gradientCheckpointing to false. training runs get --settings and --dry-run show recomputing activations on the Sequence line when the setting is on.

A dpo job starts with the PyTorch allocator setting the deployment configures, expandable_segments:True by default. This setting keeps the device memory the job reserves close to the memory it allocates.

A dpo run computes its loss with Liger’s fused kernel where it can. TRL’s own loss holds the logits over the whole vocabulary for every pair in a pass. The fused loss holds them for one pair at a time, so it saves the logits of every pair in a pass but one. A pass of one pair holds the same logits with either loss. A pass holds more than one pair by default at a maxSeqLength of 1024 or less, or where the run states a microBatchPerGpu above 1, so the fused loss applies only there. The run uses TRL’s own loss when any of these holds:

  • preference.fusedLoss is false,

  • a pass holds one pair, where microBatchPerGpu is 1,

  • the run does not train on a CUDA device,

  • lossType is ipo, which the fused loss does not compute,

  • the base model is a mixture of experts,

  • the adapter targets lm_head,

  • the base model’s type is not one the fused loss is verified for. The job’s log names the type.

The two losses agree to within rounding, but a run with the fused loss logs fewer figures for each step. It logs no mean_token_accuracy and no entropy, so it records no token accuracy. Its logits/chosen and logits/rejected are averaged over every position and the whole vocabulary rather than over the answer tokens. The fused loss computes the log-softmax in fp32, while TRL’s own loss computes it in the logits' precision, bf16 in a bf16 run. So in a bf16 run the loss, the log-probabilities and the rewards agree with TRL’s to within bf16 rounding, not exactly, and the accuracy can differ on a pair whose margin is that close to zero. For a model that caps its logits, such as Gemma 4, the fused loss applies the same cap. The job’s log names the loss and the reason, and training runs get reports it on the Loss line.

The reference model

A dpo run measures each answer against a reference model:

  • A run that starts from the base model uses the base model.

  • A weights branch of a supervised or dpo run uses a frozen copy of the adapter it starts from, and continues training from that adapter.

  • A trainingState continuation of a dpo run keeps its parent’s reference. Optimize refuses trainingState for a dpo run that started from an inherited adapter.

The reference is not a setting. training runs get --settings names it on the Preference line: reference base model or reference inherited adapter.

To start from a supervised run’s adapter, branch from one of its weights snapshots with an override document that sets the method and the dataset:

method: dpo
dataset: triage/preferences
akka training runs branch SFT_RUN_ID --snapshot SNAPSHOT_ID --steps 200 -f to-dpo.yaml --dry-run

The child does not inherit the parent’s data group, and a branch cannot set one, so it has no validation dataset and reports no Validation preference line. It also does not inherit training.validateEachEpoch, training.validationBatchSize or training.tokensPerGpu. A child that does not set model.gradientCheckpointing recomputes activations, as a dpo run does, unless the parent turned the setting off at a maxSeqLength above 4096. That child keeps it off. See Preference training from a supervised run. warmStartModelId is for reinforcement runs, and Optimize refuses it on a dpo run.

What Optimize refuses

A dpo run trains on one GPU, with one sequence for each answer of a pair. Optimize refuses the run, and names the field, when it sets:

  • execution.gpus above 1,

  • rlHyperparams, reward, or a loss group that sets anything other than the defaults. The run takes its loss over the whole of each answer,

  • data.packing to true or data.shuffle to false,

  • data.truncation to right, oldest or oldest_or_drop. A pair is never cut. Under error, Optimize refuses a pair longer than maxSeqLength. Under drop, the run leaves it out and reports it on the Data fit line, and refuses the run only if no pair remains,

  • smoke. A smoke check reproduces answers, and a dpo run has no single answer to reproduce,

  • engineSettings, execution.rolloutGpus or execution.rollout.memoryShare, or

  • warmStartModelId.

A pair’s length is the longer of the prompt with the chosen answer and the prompt with the rejected answer. Under error, Optimize checks it against maxSeqLength before the run claims its GPU. See [row-lengths]. One pass holds both answers of every pair in it, so a pair counts as two sequences against the measured token limit per pass. Exceeding the measurement gives a warning in an advisory catalog and a refusal in a strict catalog.

A dpo run fails when it reads its pairs if the base model’s chat template moves the system prompt into the last user message, since its answers cannot then be told apart from its prompt.

training base-models get lists measured methods under Measured training envelopes. training base-models list shows them in the METHODS column. An advisory catalog admits an unmeasured dpo run with a warning if TRL is available for the device profile. A strict catalog requires a matching dpo measurement. A dpo run always trains on one GPU.

What to read

training runs get reports a Preference line in its results, and a Validation preference line once the run has recorded a validation reading:

Preference              accuracy 0.82, margin 1.4 (chosen -0.3, rejected -1.7)
Validation preference   accuracy 0.71, margin 0.9 (chosen -0.5, rejected -1.4)
Loss                    fused (Liger)

Loss is fused (Liger) where the run computed its loss with Liger’s fused kernel, and unfused where it used TRL’s own. A run that does not report it has no Loss line.

  • A reward is how much more likely the trained adapter makes an answer than the reference model does, as a log ratio scaled by beta. chosen and rejected are the average rewards of the chosen and rejected answers.

  • The margin is the average of the chosen reward less the rejected reward. It grows as the run separates the answers.

  • The accuracy is the share of pairs where the chosen answer’s reward is higher than the rejected answer’s. A tie counts as not preferred, so a run that still matches its reference, where every reward is 0, reads 0.

Each figure is the average of its batch readings. The training figures average the batches of the last step, and the validation figures average the batches of the whole validation dataset at the last validation. Read them together. A training accuracy near 1 with a validation accuracy that does not rise means the run is memorizing its pairs. Both rewards often fall while the margin grows, because the run lowers the likelihood of the rejected answer faster than that of the chosen one. A chosen reward that falls far below zero means the run is making the answers you prefer less likely too. A larger beta holds the adapter closer to the reference.

Training loss and Validation loss are the preference loss. The validation loss has no perplexity. Data fit counts pairs. While the run trains, its progress shows the loss, accuracy and margin of the latest step, and training runs metrics shows LOSS, ACCURACY and MARGIN columns, with VALIDATION ACCURACY and VALIDATION MARGIN columns for the validation readings. TensorBoard shows the same figures.

Reinforcement learning

grpo and gspo sample groupSize completions per prompt and score each completion with the reward. They update the model toward completions that score above the group mean. Use them when you can verify but cannot enumerate correct output, such as executable code, SQL that must return the correct rows, or an answer with a verifiable final value.

A reward with only a strict correctness term gives an untuned model no signal. Every completion in a group scores zero, so the advantage is zero. Add a format term so that the model has a gradient before it produces a correct answer. Alternatively, warm-start from a supervised adapter with warmStartModelId.

You can train an image-text model such as Gemma 4 E4B with grpo or gspo over a text dataset on one device. As in a supervised run, the adapter updates the language backbone and leaves the vision and audio towers unchanged. On more than one device, your deployment must enable grpo and gspo for image-text models. Otherwise, the model supports only sft there. Run base-models get with the model’s repository to see the methods that your deployment supports. Reinforcement runs cannot train on images. Optimize refuses a grpo or gspo run whose dataset contains images, regardless of the base model.

The following descriptor, from the GSM8K example, is a descriptor for a reinforcement run:

{
  "kind": "run",
  "project": "gsm8k",
  "configName": "grpo-100-steps",
  "baseModel": "Qwen/Qwen3-0.6B",
  "config": {
    "dataset": "gsm8k/train",
    "method": "grpo",
    "rlHyperparams": {
      "epochs": 1,
      "maxSteps": 100,
      "groupSize": 8,
      "klBeta": 0.001,
      "temperature": 1.0,
      "maxCompletionLength": 512,
      "lora": {
        "rank": 32
      },
      "optimizer": {
        "learningRate": 1e-5
      }
    },
    "reward": {
      "terms": [
        {"kind": "rule", "weight": 1.0, "bundle": "gsm8k/scoring", "module": "reward"}
      ]
    },
    "maxSeqLength": 1536
  },
  "execution": {"gpus": 1, "saveEvery": 10, "keepAdapters": 4}
}

Lifecycle

Run the following commands to read a run, list open runs, and cancel a run:

akka training runs get RUN_ID
akka training runs list --phase open
akka training runs cancel RUN_ID

training runs list --project NAME lists one training project’s runs, and combines with --phase.

The run detail reports the phase, progress, cost, timing, and any failure reason. observedStep counts the updates reported by the worker. savedStep is the selected restorable checkpoint. The difference between these values is the work that a restore repeats.

A run reports one of the following phases. A first attempt passes through them in order and skips phases that it does not need. A supervised run over a chat dataset also reports PREPARING before AWAITING_CAPACITY, while it checks its rows. A supervised run on more than one device also reports PREPARING before AWAITING_CAPACITY, while it prepares its rows without GPUs. A resumed run re-enters at AWAITING_BASE_MODEL, DEPLOYING_GRADER or AWAITING_CAPACITY. If the cluster stops a job, its run moves from TRAINING to PAUSING.

Phase What the run is doing

RESOLVING_DATASET

Reading the dataset’s manifest.

AWAITING_BASE_MODEL

Waiting for the run’s base model to finish preparing.

DEPLOYING_GRADER, AWAITING_GRADER

Starting the judge specified by a reinforcement run’s reward and waiting for it to respond.

AWAITING_CAPACITY

Waiting for the cluster to allocate the devices requested by the run.

STARTING_JOB

Submitting the training job.

QUEUED

Waiting for the cluster to schedule the submitted job. The wait depends on the cluster’s other work.

PREPARING

A run is preparing before its first reported step. Its stage names the current action and reports completed and total rows during tokenization. A supervised run over a chat dataset enters this phase first at the stage checking row lengths against maxSeqLength. See [row-lengths]. A supervised run on more than one device enters this phase before AWAITING_CAPACITY, while row preparation waits to be scheduled or runs. It requests GPUs after preparation succeeds. Cached rows need no rendering. For a prepared dataset, Optimize checks the rows without rendering them. You cannot pause the run during this preparation. After claiming devices, the run also uses this phase while reading data or starting training. A supervised run on one device tokenizes after claiming its device. A reinforcement run reads as PREPARING from the start of its job until its first step.

TRAINING

The job is running. runs metrics reads its steps.

PAUSING, PAUSED

Stopping the job and releasing its resources until the run resumes.

RECORDING

Registering the trained adapter as a model. While the registry does not answer, the run waits here and tries again. If the registry refuses the model, the run fails.

CLEANING_UP

Releasing the run’s resources before a terminal phase. A run also waits here, with its devices released, until its last saves are recorded and committed or their commit fails, for at most 15 minutes, so a finished run reports its last saved step.

COMPLETED, FAILED, CANCELLED

Terminal.

A completed run registers its final adapter as a model. training runs get shows the model. Cancelling a run releases its devices but retains its committed snapshots, which you can still register or use as branch sources.

For training runs pause and resume, see Checkpoints. For training runs branch, see Branches.