Checkpoints

A snapshot is a verified, immutable save from one run attempt. Use snapshots to resume a paused run, start a branch, or register an earlier adapter. A snapshot contains adapter weights, full training state, or both. Full state includes the optimizer, scheduler, random-number generators, and data position. Lists contain only committed snapshots, so you cannot restore from an interrupted upload.

execution.saveEvery in the run descriptor sets the save interval in optimizer updates. Set execution.saveEachEpoch to true to also save at the end of each epoch. Without either, a run saves only at the end. See Epoch adapters.

List the snapshots of a run, then read one:

akka training runs snapshots RUN_ID
akka training snapshots get SNAPSHOT_ID

If the trainer cannot record a save when the job stops, it releases the run’s devices and tries again for up to 15 minutes. A run that still has saves it could not record then finishes, and training runs get names them by step on its Unrecorded line. You cannot resume, branch from or register an unrecorded save.

Each snapshot reports the total file size of each artifact kind. training runs snapshots shows the total of the kinds the snapshot holds, leaving out a kind whose deletion has begun, and training snapshots get shows each kind.

Pause a run

Pause a run, then read its state:

akka training runs pause RUN_ID --checkpoint save
akka training runs get RUN_ID

--checkpoint is required. It specifies which checkpoint the run stops at:

Policy Behavior

save

Saves at the next optimizer boundary, commits the snapshot, and then stops.

latest

Stops at the newest committed checkpoint. A resumed run repeats updates after that checkpoint. Optimize refuses this policy after a run has trained without any restorable state because the run would otherwise restart from the base model.

The command returns when the trainer accepts the pause. Saving and device release continue in the background. Follow the run detail until the phase is PAUSED and pause.resourcesReleased is true. The pause object reports the save progress, acknowledged snapshot, deadline, and any failure. If training finishes during the pause, the run completes. Cancellation takes precedence over a pending pause.

The trainer marks a pause as timed out when its deadline passes. It still waits for the job to confirm that it stopped. You can then resume a run that has restorable state. A run that made progress without restorable state fails instead of restarting from the base model.

Resume a run

Resume a paused run and follow it:

akka training runs resume RUN_ID
akka training runs wait RUN_ID --exit-status

Resume keeps the run’s identity and restores its newest full-state snapshot in a new attempt. The new attempt requires the same number of devices as the original allocation. To abort the run instead, use training runs cancel.

To resume from an earlier snapshot, name its step:

akka training runs resume RUN_ID --step 100

--step applies only to a PAUSED run. The snapshot at that step must still hold its training state. execution.keepRecoverable sets how many of the newest unreferenced snapshots keep it. If several attempts saved at the same step, Optimize selects the latest attempt’s snapshot. The run repeats the updates after that step. Optimize refuses a step with no such snapshot and names the step. A deployment that stores checkpoints on a shared filesystem lists no snapshots, so Optimize refuses every --step there.

Retention

Each time a snapshot is recorded, Optimize retains training state for the execution.keepRecoverable newest unreferenced snapshots, one by default. It also retains adapter weights for the execution.keepAdapters newest snapshots. The latest attempt’s snapshots count as the newest. It deletes the rest, except the training state of the snapshot that a retry or a resume of the run would restore. It can delete one artifact kind from a snapshot while retaining another. A branch or a pin retains the artifact kinds that it references for as long as it exists. Registering a snapshot’s adapter as a model version holds the weights only until the registry has its own copy.

Epoch adapters

A supervised run with execution.saveEachEpoch set to true also saves at the end of each epoch. These saves are retained separately from the periodic saves controlled by saveEvery:

  • The run retains the adapter weights of each epoch save. These weights do not count toward keepAdapters, and retention does not delete them. The training state of an epoch save follows the same retention rules as other saves.

  • Each epoch’s adapter is registered as a version of the run’s model. training snapshots candidate SNAPSHOT_ID reports the version, the epoch, and the step.

  • The save at the run’s last update is the run’s own model. It is not registered a second time. training runs snapshots marks it as the run’s model. training runs get reports the run’s model by its exact version. Versions are numbered in the order they are registered, so an epoch adapter registered after the run’s model can be the newest version. A reference without a version, such as triage/qwen3-0.6b, resolves to the newest version, which can be that epoch adapter. Name the run’s model by the version training runs get reports, or point an alias at it.

  • A full-state branch does not save at epoch boundaries. It continues from its parent’s position in the data. Its length is defined in updates rather than epochs.

Use training runs snapshots to see which epoch each save ended in the EPOCH column. The hold on an epoch’s weights is listed as epoch. Once the run has finished, release the hold with training snapshots release SNAPSHOT_ID RUN_ID. Those weights then become subject to keepAdapters. The registered versions retain their own copy of the weights.

Support by method

The following table lists what each method supports:

Method Support

Supervised

Periodic snapshots, epoch adapters, latest and save pauses, resume, and weights branches. Full-state branches require the same GPU count as the snapshot.

Reinforcement

Periodic snapshots, latest and save pauses, resume, and compatible weights branches. Full-state branches require the same method and GPU count as the snapshot.

For grpo and gspo with a positive klBeta, Optimize refuses a full-state branch if the run started from inherited adapter weights, through a warm start or a branch from weights. The restore does not reconstruct the frozen reference adapter. Optimize refuses an unsupported combination and never downgrades it.

A run that starts from another run’s adapter, through a weights branch or a warm start, can set a positive klBeta only on one GPU. On more than one GPU the reference policy is the base model, so a penalty would pull the run toward the base model rather than toward the adapter it started from, and Optimize refuses it.