akka optimize training runs compare

Compare a supervised run’s training loss with a baseline, step by step

Synopsis

Compare the training loss of a supervised run with a baseline over windows of steps.

The command accepts only supervised runs. Reinforcement runs record a reward at each step instead of a training loss.

Runs with the same plan and seed train the same rows at every step. A difference in their loss at a step therefore comes from what they learned, not from which rows they drew. The command calculates the mean loss difference for windows of --window steps. It reports a gradient spike when either run’s gradient norm before clipping is at least --spike-factor times that run’s median. It also reports the first full window whose mean difference, in either direction, is greater than --margin.

The backward pass is nondeterministic, so two runs with the same plan can differ. A gradient spike can affect one run’s loss for many steps. Interpret a parting window by comparing it with repeated runs of the same plan.

By default, the command rejects runs with different plans, in dataset, seed, base model, sequences per update or any other setting of the configuration. The loss of such runs differs because of their plans, as well as because of what they learned. Use --across-plans to compare them and report the plan differences. Two runs that differ only in how they divide the same sequences per update across devices, passes, and accumulation train on the same rows. The command compares these runs and reports the division.

akka optimize training runs compare RUN BASELINE [flags]

Examples

  # Compare a run with a baseline that uses the same plan
  akka optimize training runs compare 7d496e82c3a15f04 be2a310245ea0c11

  # Wider windows and a stricter margin
  akka optimize training runs compare 7d496e82c3a15f04 be2a310245ea0c11 \
    --window 20 --margin 0.02

  # The first window where the two part, if any
  akka optimize training runs compare 7d496e82c3a15f04 be2a310245ea0c11 \
    --jq .parting

Options

      --across-plans         Compare runs with different plans and report the differences
  -h, --help                 help for compare
      --margin float         Mean loss difference over a full window, in either direction, above which the runs part (default 0.03)
      --spike-factor float   Minimum multiple of the run's median gradient norm that counts as a spike (default 10)
      --window int           Number of steps averaged into each row (default 10)

Options inherited from parent commands

      --cache-file string    location of cache file (default "~/.akka/cache.yaml")
      --config string        location of config file (default "~/.akka/config.yaml")
      --context string       configuration context to use
      --disable-prompt       Disable all interactive prompts when running akka commands. If input is required, defaults will be used, or an error will be raised.
                             This is equivalent to setting the environment variable AKKA_DISABLE_PROMPTS to true.
      --force                Skip the confirmation prompt
      --jq string            Filter the JSON output with a jq expression
      --json string          Output JSON with only these fields; --json= lists the fields a command offers
      --no-color             Print without color, as NO_COLOR does
  -o, --output string        set output format to one of [text,json,json-compact,go-template=] (default "text")
      --page-mode string     the mode for paging, either paged, buffered or auto. (default "auto")
  -q, --quiet                set quiet output (helpful when used as part of a script)
      --region string        Region to use if the project has more than one region
      --timeout duration     client command timeout (default 10s)
      --trainer-url string   Trainer service URL (default TRAINER_URL, then AKKA_OPTIMIZE_URL, then http://localhost:9001)
      --url string           Optimize service URL (default OPTIMIZE_URL, then AKKA_OPTIMIZE_URL, then http://localhost:9010)
      --use-grpc-web         use grpc-web when talking to Akka APIs. This is useful when behind corporate firewalls that decrypt traffic but don't support HTTP/2.
      --verbose              set verbose output

SEE ALSO