Workloads

A workload represents one task. Use it to group the runs and models for that task. It specifies the evaluation dataset and grader for every candidate, and contains three model references.

Create a workload

Create a workload, then read and list workloads:

akka-optimize training workloads create triage --dataset triage-eval --grader json-label-match
akka-optimize training workloads get triage
akka-optimize training workloads list

A workload name must match [a-z0-9][a-z0-9-]{0,63} and must not have the form of an ID. Runs store the workload’s ID, so training workloads rename preserves its history and associations.

Evaluation settings

The evaluation dataset and grader define how candidates for the task are compared. training workloads scores tabulates evaluations that use both values. An evaluation specifies its own dataset and grader, so pass the workload’s values when you start one.

Set the evaluation settings when you create the workload, or set them later:

akka-optimize training workloads set-evaluation triage --dataset triage-eval --grader json-label-match

Score comparisons require the same dataset and grader. Changing either value resets the default comparison table in training workloads scores. To read scores from the previous settings, pass --dataset and --grader to scores.

For the grader kinds, see Graders and rewards.

References

A workload contains the following model references:

Reference Description

base

The model that a training request uses when it names no base model.

candidate

The model under consideration for promotion. Pipelines and operators move it.

champion

The model that serves the task.

To read and set a reference, run the following commands:

akka-optimize training workloads refs get triage base
akka-optimize training workloads refs set triage base MODEL_ID
akka-optimize training workloads refs set triage champion MODEL_ID

History

The trainer records every change to a workload’s name, evaluation settings, and references. Read the history:

akka-optimize training workloads history triage

Use the history to restore an earlier reference after an incorrect promotion.

Scores

Tabulate the scores of a workload:

akka-optimize training workloads scores triage

The command compares the newest evaluation of each configuration in the workload with the registered baselines. It uses the workload’s evaluation settings. --config-name on a run, or training runs annotate, labels the configuration’s row.