akka optimize training runs continue
Continue a run’s training state from a snapshot as a child run
Synopsis
Continue PARENT’s training state from a committed snapshot, as a child run that trains --steps more optimizer updates.
The parent run is unchanged. The child restores the adapter weights, optimizer, RNG, data position and engine step the snapshot saved, and replaces the saved schedule with one spanning --steps at the child’s learning rate.
--snapshot selects the source snapshot. Without it, the command takes PARENT’s newest committed snapshot that holds its training state. Use "training runs snapshots" to list them.
Only the learning rate (hyperparams.optimizer.learningRate, or rlHyperparams.optimizer.learningRate for a reinforcement parent) and hyperparams.training.validationFrequencySteps may appear in --overrides. The service rejects every other field, even when its value equals the parent’s. A continuation trains on as many GPUs as the snapshot was saved across. For reinforcement learning with a positive klBeta, it is unavailable when the producer started from inherited adapter weights. Where the snapshot cannot continue, the command says why. "training runs branch" then trains a fresh run from the snapshot’s weights.
The child run inherits the parent run’s training project, base model, devices and grader. --overrides loads a configuration-change document. Omitted fields are inherited. Undeclared fields are rejected. The child trains the adapter it inherits, so the document cannot change lora.rank or lora.targetModules.
--dry-run asks the service to validate and resolve the request without creating a submission, retaining the source snapshot or claiming a device. The preview does not reserve the source.
A successful submission response means the operation is durable. Source retention and child-run creation happen during preparation. --wait follows preparation before it follows the child run.
The command generates a token for each invocation. Running the command again generates another one, which the service treats as a second run. Passing --submission-id with the token from an earlier attempt, and the same flags, reaches whatever that attempt created, instead of starting a second run. It is for retrying an attempt whose outcome was never seen. Without --snapshot, a retry reads the snapshot the earlier attempt wrote back from the service and sends it again, so it need not repeat --snapshot. Against a trainer that sends no snapshotReference, a retry of an attempt that named --snapshot repeats it as it was written. A retry under a token the service has no submission for is a first attempt. "training runs branches PARENT --submission TOKEN" reads that attempt’s state back instead, and is the cheaper check.
akka optimize training runs continue PARENT [--snapshot SNAPSHOT] --steps N [flags]
Examples
# Preview a continuation at a lower learning rate
akka optimize training runs continue 7d496e82c3a15f04 \
--steps 200 --overrides lower-lr.json --dry-run
# Submit it from a named snapshot and follow the run it starts
akka optimize training runs continue 7d496e82c3a15f04 \
--snapshot 3f2a9c71bd04 --steps 200 --overrides lower-lr.json --wait
Options
--config-name string Label for the child configuration in the training project score table
--description string Description for the child run
--dry-run Preview the resolved child run without submitting it
--exit-status Exit non-zero if preparation or the child run fails; requires --wait
-h, --help help for continue
-f, --overrides string Override document path, or '-' for standard input
--snapshot string Source snapshot ID or unambiguous prefix; the newest that holds what the command restores by default
--steps int Additional optimizer updates the child run trains
--submission-id string Submission token to reuse after an uncertain response
--wait Wait for preparation and child-run completion
Options inherited from parent commands
--cache-file string location of cache file (default "~/.akka/cache.yaml")
--config string location of config file (default "~/.akka/config.yaml")
--context string configuration context to use
--disable-prompt Disable all interactive prompts when running akka commands. If input is required, defaults will be used, or an error will be raised.
This is equivalent to setting the environment variable AKKA_DISABLE_PROMPTS to true.
--force Skip the confirmation prompt
--jq string Filter the JSON output with a jq expression
--json string Output JSON with only these fields; --json= lists the fields a command offers
--no-color Print without color or a redrawn progress line, as NO_COLOR does
-o, --output string set output format to one of [text,json,json-compact,go-template=] (default "text")
--page-mode string the mode for paging, either paged, buffered or auto. (default "auto")
-q, --quiet set quiet output (helpful when used as part of a script)
--region string Region to use if the project has more than one region
--timeout duration client command timeout (default 10s)
--trainer-url string Trainer service URL (default TRAINER_URL, then AKKA_OPTIMIZE_URL, then a local dev-mode service, then http://localhost:9001)
--use-grpc-web use grpc-web when talking to Akka APIs. This is useful when behind corporate firewalls that decrypt traffic but don't support HTTP/2.
--verbose set verbose output
SEE ALSO
-
akka optimize training runs - Manage training runs, snapshots and child runs