akka optimize training comparisons create

Start evaluations comparing a source snapshot with branch models

Synopsis

Start a durable branch comparison.

Specify branch models to evaluate using --candidate, --branch, or --all-branches. At least one of these flags is required, and they can be combined: --candidate: specifies a candidate model directly, by registry reference NS/NAME[:VERSION]. --branch: specifies a run ID, evaluating the model registered by that run. --all-branches: specifies a parent run ID, evaluating every completed child run that branched from --source. Child runs still in progress are omitted.

Child runs are resolved from an asynchronous projection, so a run that completed moments earlier may be omitted from the fan, and nothing adds it after the comparison is created. The Fan line reports how many child runs the listing contained; compare it against the number of branches the parent is expected to have.

Each snapshot and run identifier may be a complete ID or an unambiguous prefix. The dataset is a registry reference; the service resolves it once and records the exact version. The service accepts a snapshot of any run as a candidate. The command returns after the comparison is durable; use training comparisons get until it reaches a terminal phase.

akka optimize training comparisons create --source SNAPSHOT --branch RUN --dataset DATASET [flags]

Examples

  akka optimize training comparisons create \
    --source 3f2a9c71bd04f6504a2f9306b8e465c7d336e4ca12d27f8872944655d455aa66 \
    --branch 7d496e82c3a15f04b1c6d0e8a34f29b7e5c81d0a6f3b249e7c05a8d1f6b32e40 \
    --all-branches b7c2d1e0a94f3b6188aa0c5d7e2f4a9134e8c07b52d6193fa0c84e7b51d2960f \
    --dataset branch-eval --grader json-label-match

Options

      --all-branches string          Prefix or complete ID of a parent run whose completed children branched from --source will be evaluated
      --branch strings               Prefix or complete ID of a branch run whose registered model will be evaluated; repeat for multiple runs
      --candidate strings            Registry reference of a candidate model to evaluate, NS/NAME[:VERSION]; repeat for multiple models
      --dataset string               Evaluation dataset, by registry reference
      --grader string                Grader specification: json-label-match, model:NS/NAME[:VERSION], or http:URL
      --grader-instructions string   Instructions a model: grader is asked under, in place of the ones it would otherwise get
  -h, --help                         help for create
      --max-completion-tokens int    Maximum completion tokens per sample; zero leaves completions uncapped
      --source string                Prefix or complete ID of the source snapshot to compare against
      --submission-id string         Submission token to reuse after an uncertain response

Options inherited from parent commands

      --cache-file string    location of cache file (default "~/.akka/cache.yaml")
      --config string        location of config file (default "~/.akka/config.yaml")
      --context string       configuration context to use
      --disable-prompt       Disable all interactive prompts when running akka commands. If input is required, defaults will be used, or an error will be raised.
                             This is equivalent to setting the environment variable AKKA_DISABLE_PROMPTS to true.
      --force                Skip the confirmation prompt
      --jq string            Filter the JSON output with a jq expression
      --json string          Output JSON with only these fields; --json= lists the fields a command offers
      --no-color             Print without color, as NO_COLOR does
  -o, --output string        set output format to one of [text,json,json-compact,go-template=] (default "text")
      --page-mode string     the mode for paging, either paged, buffered or auto. (default "auto")
  -q, --quiet                set quiet output (helpful when used as part of a script)
      --region string        Region to use if the project has more than one region
      --timeout duration     client command timeout (default 10s)
      --trainer-url string   Trainer service URL (default TRAINER_URL, then AKKA_OPTIMIZE_URL, then http://localhost:9001)
      --url string           Optimize service URL (default OPTIMIZE_URL, then AKKA_OPTIMIZE_URL, then http://localhost:9010)
      --use-grpc-web         use grpc-web when talking to Akka APIs. This is useful when behind corporate firewalls that decrypt traffic but don't support HTTP/2.
      --verbose              set verbose output

SEE ALSO