akka optimize training workloads set-evaluation

Set evaluation dataset and grader for a workload

Synopsis

Configure the evaluation dataset and grader used to compare candidate models for a workload.

Score comparisons require matching dataset and grader pairs. Updating either value resets the default baseline comparison table. Previous evaluation scores remain accessible by supplying --dataset and --grader to the "scores" command.

akka optimize training workloads set-evaluation WORKLOAD --dataset HASH [flags]

Examples

  # Set the evaluation dataset
  akka-optimize training workloads set-evaluation commit-router --dataset eval-set

  # Set both evaluation dataset and grader
  akka-optimize training workloads set-evaluation commit-router \
    --dataset 4b1c2f9b2ae4 --grader model:commit-judge

Options

      --dataset string   Evaluation dataset name or hash
      --grader string    Grader specification: json-label-match, model:MODEL, or http:URL
  -h, --help             help for set-evaluation

Options inherited from parent commands

      --cache-file string    location of cache file (default "~/.akka/cache.yaml")
      --config string        location of config file (default "~/.akka/config.yaml")
      --context string       configuration context to use
      --disable-prompt       Disable all interactive prompts when running akka commands. If input is required, defaults will be used, or an error will be raised.
                             This is equivalent to setting the environment variable AKKA_DISABLE_PROMPTS to true.
      --force                Skip the confirmation prompt
      --jq string            Filter the JSON output with a jq expression
      --json string          Output JSON with only these fields; --json= lists the fields a command offers
      --no-color             Print without color, as NO_COLOR does
  -o, --output string        set output format to one of [text,json,json-compact,go-template=] (default "text")
      --page-mode string     the mode for paging, either paged, buffered or auto. (default "auto")
  -q, --quiet                set quiet output (helpful when used as part of a script)
      --region string        Region to use if the project has more than one region
      --timeout duration     client command timeout (default 10s)
      --trainer-url string   Trainer service URL (default http://localhost:9001, or TRAINER_URL)
      --url string           Optimize service URL (default http://localhost:9010, or OPTIMIZE_URL)
      --use-grpc-web         use grpc-web when talking to Akka APIs. This is useful when behind corporate firewalls that decrypt traffic but don't support HTTP/2.
      --verbose              set verbose output

SEE ALSO