Observe a run

Run details, metrics, and TensorBoard show the progress and results of a training run. Use them to monitor a run while it trains or investigate it after it finishes.

Run detail

Read a run, or follow it to a terminal phase:

akka training runs get RUN_ID
akka training runs get RUN_ID --settings
akka training runs wait RUN_ID --exit-status

The run detail has three parts. State reports what the run is, its project, method, base model and dataset, then the phase, training progress, the saved step and any failure reason. Results reports the model the run registered, its validation and training loss, and its start checks. Cost reports the GPU time, the duration and the device memory. --settings, or -o wide, adds Settings: the configuration, hyperparameters, engine settings, runtime and run settings the run was started with. -o json is the whole run. When the job ends, the run detail reports the GPU seconds that the run consumed. Until then, the detail marks an estimate based on the training time and the devices requested by the run. The duration separates the time spent waiting for the cluster from the time spent training. The JSON output reports timing for each phase under timing.phaseMillis. A pipeline’s detail reports the cost of each train stage and the total. observedStep counts the updates reported by the worker. savedStep is the selected restorable checkpoint. A branch counts its own updates from zero.

A supervised run on more than one device reports PREPARING before it requests GPUs. Training and validation rows are tokenized without GPUs. This phase includes waiting for tokenization to be scheduled. The stage names the current action, with completed and total rows during tokenizing. On a large dataset of long conversations this takes minutes. While tokenized rows are cached for later runs, the stage is caching the tokenised rows, with no row counts. A job that reports no progress for the deployment’s startup timeout fails the run. Rows that an earlier run tokenized are not tokenized again, so the phase passes quickly. The training job reads the tokenized rows. If the stored rows are damaged, the training job tokenizes them again after it claims its GPUs. For a dataset of prepared token rows, the job tokenizes nothing and checks the rows against the dataset and the base model’s tokenizer and chat template. Where the base model’s revision does not resolve to a commit, nothing is cached, and the training job tokenizes the rows after it claims its GPUs.

After training job submission, the run is QUEUED until the cluster starts the job. A supervised run then reports PREPARING while reading data or starting workers. A supervised run on one device also tokenizes during this phase. The run holds its devices during training job preparation. runs metrics provides the live view of a training run. The run record updates less often, so its progress can lag behind the metrics.

wait streams status updates until the run reaches a terminal phase. With -o json, it streams line-delimited events. A terminal event with the failure reason ends the stream, followed by the full run record. --timeout limits the wait. The run continues after the timeout expires.

Run costs include all attempts and any work repeated after a restore.

Peak memory is the peak device memory for each device. For a supervised run, and a reinforcement run that trains in one process, Memory shows what held the device at the peak of the step that reserved the most: the weights, adapter, gradients or generation, activations, optimizer state, what the allocator and the driver held, and what was free. It also shows what the other phase of the step held. For a supervised run, Data fit reports how many rows reached maxSeqLength, how many of them likely lost the end of their target, how many were dropped with nothing left to train on, and the share of the batches that was padding.

Why a run failed

A failed run reports one of the following reasons:

Reason What to do

Reason	What to do
A refusal from the training runtime, such as "an example ends with a user message", or an adapter that does not load, which names the tensors	Correct the input that the refusal names, then submit the run again. The runtime read the run’s inputs and would not train on them.
"the training job ran out of device memory; a shorter sequence, a smaller micro batch or activation recomputation uses less"	A training step exceeded the device’s memory. Lower `config.maxSeqLength`, lower `execution.microBatchSequences` on a reinforcement run, or enable `config.model.gradientCheckpointing`. If the run still fails, select a device profile with more device memory.
"the training job’s rollout engine did not fit its share of device memory; a shorter sequence or a smaller model fits"	The rollout engine could not hold the model weights and the cache for one sequence in its share of device memory. Lower `config.maxSeqLength` or use a smaller model.
"the training job ran out of device memory handing its weights to the rollout engine; a smaller model or a device with more memory fits"	While the trainer updates the rollout engine, the device holds both copies of the model weights. Use a smaller model, or select a device profile with more device memory.
"the evaluation job ran out of device memory; a lower completion limit uses less"	Lower the evaluation’s completion limit, or ask your deployment’s operator for evaluation devices with more device memory.
"the evaluation job’s inference engine did not fit its share of device memory; a smaller model or a device with more memory fits"	The evaluation’s inference engine could not hold the model weights and its cache in its share of device memory. The completion limit does not change this. Evaluate a smaller model, or ask your deployment’s operator for evaluation devices with more device memory.
"the training job ran out of host memory; a smaller micro batch or sequence length uses less"	The worker ran out of host memory, so the operating system stopped the job. Lower `config.maxSeqLength` and, on a reinforcement run, `execution.microBatchSequences`. If the run still fails, select a device profile with more host memory.
"the evaluation job ran out of host memory; a device with more host memory fits it"	The evaluation worker ran out of host memory, so the operating system stopped the job. Ask your deployment’s operator for evaluation devices with more host memory.
"the training cluster could not be reached while this run waited for its devices"	The trainer granted the run its devices, and the training cluster did not answer before the wait for devices ran out. Submit the run again later. If it fails the same way, quote the run ID to your operator.
A sentence that names what the run could not do, such as "this run’s job could not be started on the training cluster"	A step of the run failed on every retry. The sentence names no cause. Submit the run again later. If it fails the same way, quote the run ID to your operator.
"the training job was stopped"	The cluster terminated the job. Start the run again, or resume it if it holds a checkpoint.
"the training job stopped before it finished"	The job ended without reporting what it did. Start the run again.
"the training job failed; the run ID is what support needs"	Nothing in the job reported what went wrong. Quote the run ID.

A refusal from the training runtime, such as "an example ends with a user message", or an adapter that does not load, which names the tensors

Correct the input that the refusal names, then submit the run again. The runtime read the run’s inputs and would not train on them.

"the training job ran out of device memory; a shorter sequence, a smaller micro batch or activation recomputation uses less"

A training step exceeded the device’s memory. Lower config.maxSeqLength, lower execution.microBatchSequences on a reinforcement run, or enable config.model.gradientCheckpointing. If the run still fails, select a device profile with more device memory.

"the training job’s rollout engine did not fit its share of device memory; a shorter sequence or a smaller model fits"

The rollout engine could not hold the model weights and the cache for one sequence in its share of device memory. Lower config.maxSeqLength or use a smaller model.

"the training job ran out of device memory handing its weights to the rollout engine; a smaller model or a device with more memory fits"

While the trainer updates the rollout engine, the device holds both copies of the model weights. Use a smaller model, or select a device profile with more device memory.

"the evaluation job ran out of device memory; a lower completion limit uses less"

Lower the evaluation’s completion limit, or ask your deployment’s operator for evaluation devices with more device memory.

"the evaluation job’s inference engine did not fit its share of device memory; a smaller model or a device with more memory fits"

The evaluation’s inference engine could not hold the model weights and its cache in its share of device memory. The completion limit does not change this. Evaluate a smaller model, or ask your deployment’s operator for evaluation devices with more device memory.

"the training job ran out of host memory; a smaller micro batch or sequence length uses less"

The worker ran out of host memory, so the operating system stopped the job. Lower config.maxSeqLength and, on a reinforcement run, execution.microBatchSequences. If the run still fails, select a device profile with more host memory.

"the evaluation job ran out of host memory; a device with more host memory fits it"

The evaluation worker ran out of host memory, so the operating system stopped the job. Ask your deployment’s operator for evaluation devices with more host memory.

"the training cluster could not be reached while this run waited for its devices"

The trainer granted the run its devices, and the training cluster did not answer before the wait for devices ran out. Submit the run again later. If it fails the same way, quote the run ID to your operator.

A sentence that names what the run could not do, such as "this run’s job could not be started on the training cluster"

A step of the run failed on every retry. The sentence names no cause. Submit the run again later. If it fails the same way, quote the run ID to your operator.

"the training job was stopped"

The cluster terminated the job. Start the run again, or resume it if it holds a checkpoint.

"the training job stopped before it finished"

The job ended without reporting what it did. Start the run again.

"the training job failed; the run ID is what support needs"

Nothing in the job reported what went wrong. Quote the run ID.

Except for refusals, failure reasons omit internal exception messages, compute backend messages, and module names.

Metrics

Read the metrics of a run:

akka training runs metrics RUN_ID
akka training runs metrics RUN_ID -o json

The command reports the metrics that the engine logged for each step. It retains every numeric field from the engine under measured. A reinforcement run also reports the contribution of each reward term under rewardTerms. The trainer removes metrics after a restored step, so the series reflects the run’s current progress.

Each step also reports how the optimizer updated the adapter. The field names are the same for every run:

Field Description

Field	Description
`grad_norm`	The gradient norm before clipping. A reinforcement run with a worker process for each device reports it as `actor/grad_norm`.
`grad_clip_factor`	The factor used to scale the gradient during clipping. It is `maxGradNorm` divided by the gradient norm, or 1 if the gradient norm does not exceed `maxGradNorm`.
`adapter_norm`	The norm of the adapter’s trained weights after the step.
`update_ratio`	The norm of the adapter update divided by the adapter norm before the step.

grad_norm

The gradient norm before clipping. A reinforcement run with a worker process for each device reports it as actor/grad_norm.

grad_clip_factor

The factor used to scale the gradient during clipping. It is maxGradNorm divided by the gradient norm, or 1 if the gradient norm does not exceed maxGradNorm.

adapter_norm

The norm of the adapter’s trained weights after the step.

update_ratio

The norm of the adapter update divided by the adapter norm before the step.

If clipping occurs at almost every step, the gradient norm alone says little about the size of each update. The update_ratio reports the update size. training runs metrics displays these values as columns.

Compare two runs

Compare the training loss of a supervised run with that of a baseline run:

akka training runs compare RUN_ID BASELINE_RUN_ID

The command accepts only supervised runs. Reinforcement runs record a reward at each step instead of a training loss.

Two runs with the same configuration train the same rows at every step. A difference in their loss at a step therefore comes from what they learned, not from which rows they drew. The command calculates the mean loss difference for each window and lists the steps where either run’s gradient norm spiked. It reports the first full window whose mean difference, in either direction, is greater than --margin. A shorter window, such as the last window of an active run, shows how many steps were compared but cannot be the window the command reports.

Option Default Description

Option	Default	Description
`--window`	10	The number of steps averaged into each row.
`--margin`	0.03	The mean loss difference over a window above which the command reports the window.
`--spike-factor`	10	How many times its run’s median a gradient norm must be to count as a spike.
`--across-plans`	off	Compare two runs whose configurations or base models differ.

--window

The number of steps averaged into each row.

--margin

0.03

The mean loss difference over a window above which the command reports the window.

--spike-factor

How many times its run’s median a gradient norm must be to count as a spike.

--across-plans

off

Compare two runs whose configurations or base models differ.

The backward pass is nondeterministic, so two runs with the same configuration can differ. A gradient spike can affect one run’s loss for many steps. Compare a reported window with repeated runs of the same configuration before you act on it.

By default, the command rejects runs that differ in dataset, seed, base model or its revision, sequences per update, or another configuration value. It reports the differences. The loss of such runs differs because of their plans, as well as because of what they learned. Use --across-plans to compare them anyway. Two runs that differ only in how they divide the same sequences per update across devices, passes, and accumulation train on the same rows. The command compares these runs and reports the division.

Grafana

Print a link to the run on the project’s training dashboard in Grafana:

akka training runs grafana RUN_ID

The link covers the run from its start to its end, or to now while it trains. It carries a token that lets anyone who has it view the project’s Grafana until the expiry that the command prints, so do not share it. The command reads the platform project from --platform-project, the AKKA_PROJECT environment variable, or the akka context, and prints the project that it used. --attempt selects a resubmitted attempt of the run.

Hyperparameters

Read the hyperparameters of a run:

akka training runs hparams RUN_ID

The command reports the resolved hyperparameters used for training, including every applied default. They become available shortly after the job starts, when the worker writes them.

Engine configuration

Read the configuration the training engine was given:

akka training runs engine-config RUN_ID
akka training runs engine-config RUN_ID -o json

The configuration is available before the first step, in two parts. requested is what the worker built from the run’s settings for the training engine. resolved is the engine’s whole configuration once the engine has filled in its own defaults, and the adapter as it was attached to the model. The table lists each requested setting and shows a second value where the engine resolved it differently. Use the JSON output to read a default the run did not set, for example --jq .resolved.config.adam_beta1.

TensorBoard

TensorBoard event files are available during training, along with reward-term curves and hyperparameters for the HParams dashboard.

To start TensorBoard for a run, run:

akka training runs tensorboard RUN_ID

The command uses your credentials to download the files and starts TensorBoard on your machine. TensorBoard has no authentication, so keep it local. Engine curves use the engine’s step counter. A full-state branch can start above zero on these curves, while the run detail counts from zero.

Rollouts

A reinforcement run keeps every rollout that it scores: the prompt, the completion, the score, and the weighted share of each reward term. To check that a rising reward comes from better answers and not from reward hacking, read the rollouts:

akka training runs rollouts RUN_ID
akka training runs rollouts RUN_ID --step 40 --worst 5
akka training runs rollouts RUN_ID --best 3 -o json

The command shows one step. By default, it shows the latest step that the run kept. To show a different step, use --step. --worst N and --best N sort the rollouts of the step by score and show the first N. --limit N shows the first N in scoring order. The table shows the last message of the prompt and truncates the prompt and the completion. -o json and --jq return both in full.

Rollouts are retained after each step is reported. If a training job stops or fails, the rollouts scored in its last step are retained.

Telemetry

The trainer and its training jobs publish metrics and spans through OpenTelemetry under the akka.optimize.training. prefix. For what they publish and how to use it across runs, see Observability.