akka optimize training runs continue

Continue a run’s training state from a snapshot as a child run

Synopsis

Continue PARENT’s training state from a committed snapshot, as a child run that trains --steps more optimizer updates.

The parent run is unchanged. The child restores the adapter weights, optimizer, RNG, data position and engine step the snapshot saved, and replaces the saved schedule with one spanning --steps at the child’s learning rate.

--snapshot selects the source snapshot. Without it, the command takes PARENT’s newest committed snapshot that holds its training state. Use "training runs snapshots" to list them.

Only the learning rate (hyperparams.optimizer.learningRate, or rlHyperparams.optimizer.learningRate for a reinforcement parent) and hyperparams.training.validationFrequencySteps may appear in --overrides. The service rejects every other field, even when its value equals the parent’s. A continuation trains on as many GPUs as the snapshot was saved across. For reinforcement learning with a positive klBeta, it is unavailable when the producer started from inherited adapter weights. Where the snapshot cannot continue, the command says why. "training runs branch" then trains a fresh run from the snapshot’s weights.

The child run inherits the parent run’s training project, base model, devices and grader. --overrides loads a configuration-change document. Omitted fields are inherited. Undeclared fields are rejected. The child trains the adapter it inherits, so the document cannot change lora.rank or lora.targetModules.

--dry-run asks the service to validate and resolve the request without creating a submission, retaining the source snapshot or claiming a device. The preview does not reserve the source.

A successful submission response means the operation is durable. Source retention and child-run creation happen during preparation. --wait follows preparation before it follows the child run.

The command generates a token for each invocation. Running the command again generates another one, which the service treats as a second run. Passing --submission-id with the token from an earlier attempt, and the same flags, reaches whatever that attempt created, instead of starting a second run. It is for retrying an attempt whose outcome was never seen. Without --snapshot, a retry reads the snapshot the earlier attempt wrote back from the service and sends it again, so it need not repeat --snapshot. Against a trainer that sends no snapshotReference, a retry of an attempt that named --snapshot repeats it as it was written. A retry under a token the service has no submission for is a first attempt. "training runs branches PARENT --submission TOKEN" reads that attempt’s state back instead, and is the cheaper check.

akka optimize training runs continue PARENT [--snapshot SNAPSHOT] --steps N [flags]

Examples

  # Preview a continuation at a lower learning rate
  akka optimize training runs continue 7d496e82c3a15f04 \
    --steps 200 --overrides lower-lr.json --dry-run

  # Submit it from a named snapshot and follow the run it starts
  akka optimize training runs continue 7d496e82c3a15f04 \
    --snapshot 3f2a9c71bd04 --steps 200 --overrides lower-lr.json --wait

Options

      --config-name string     Label for the child configuration in the training project score table
      --description string     Description for the child run
      --dry-run                Preview the resolved child run without submitting it
      --exit-status            Exit non-zero if preparation or the child run fails; requires --wait
  -h, --help                   help for continue
  -f, --overrides string       Override document path, or '-' for standard input
      --snapshot string        Source snapshot ID or unambiguous prefix; the newest that holds what the command restores by default
      --steps int              Additional optimizer updates the child run trains
      --submission-id string   Submission token to reuse after an uncertain response
      --wait                   Wait for preparation and child-run completion

Options inherited from parent commands

      --cache-file string    location of cache file (default "~/.akka/cache.yaml")
      --config string        location of config file (default "~/.akka/config.yaml")
      --context string       configuration context to use
      --disable-prompt       Disable all interactive prompts when running akka commands. If input is required, defaults will be used, or an error will be raised.
                             This is equivalent to setting the environment variable AKKA_DISABLE_PROMPTS to true.
      --force                Skip the confirmation prompt
      --jq string            Filter the JSON output with a jq expression
      --json string          Output JSON with only these fields; --json= lists the fields a command offers
      --no-color             Print without color or a redrawn progress line, as NO_COLOR does
  -o, --output string        set output format to one of [text,json,json-compact,go-template=] (default "text")
      --page-mode string     the mode for paging, either paged, buffered or auto. (default "auto")
  -q, --quiet                set quiet output (helpful when used as part of a script)
      --region string        Region to use if the project has more than one region
      --timeout duration     client command timeout (default 10s)
      --trainer-url string   Trainer service URL (default TRAINER_URL, then AKKA_OPTIMIZE_URL, then a local dev-mode service, then http://localhost:9001)
      --use-grpc-web         use grpc-web when talking to Akka APIs. This is useful when behind corporate firewalls that decrypt traffic but don't support HTTP/2.
      --verbose              set verbose output

SEE ALSO