Evaluation
An evaluation scores a registered model against a dataset with a grader. Use evaluations to measure a model after training or to compare it with a baseline. Training and evaluation are separate operations: finishing a run registers its model, and you score that model in a separate step. A score belongs to the evaluation, so you can evaluate a model repeatedly.
Models
Models are versions in the artifact registry. A completed run registers its final adapter as a version of the run’s named model, such as triage/qwen3-0.6b:v3. Name a model by its registry reference: triage/qwen3-0.6b:v3, triage/qwen3-0.6b for the latest version, or an alias such as triage/qwen3-0.6b:champion. The trainer stores the exact version that a reference resolves to.
List a named model’s versions, then read one:
akka registry models versions triage/qwen3-0.6b
akka registry models get triage/qwen3-0.6b:v3The registry records each version’s base model, the model a reinforcement run started from, and the run that produced it.
To register a snapshot from an earlier point in a run, use the following command:
akka training snapshots candidate SNAPSHOT_IDRegistration retains the weights and prepares a serving copy before registering the version. The trainer accepts the request before registration finishes. candidate reports its progress. Wait for REGISTERED, when model names the version. Registering the same snapshot again returns the same version. You cannot register a snapshot that contains only training state.
Baselines
Compare a candidate with a baseline. Evaluate the base model on the same dataset with the same grader. Selecting a base model in the catalog registers it as a version of a named model identified by its lowercase repository name, so Qwen/Qwen3-0.6B becomes qwen/qwen3-0.6b:v1. training base-models list shows the version of each selected model.
A base model version pins a repository commit. A newer commit is a separate version, and a model trained over the earlier one continues to name the earlier one. Training and evaluation both use the pinned commit.
To score an adapter trained elsewhere as a baseline or judge, register it with registry models register.
A base model does not belong to a training project, so specify the project for its score when you evaluate it:
# Selecting the base model registered it as a version of the named model its repository
# lowercases to, so the baseline names that version.
BASE=$(akka registry models get unsloth/llama-3.2-1b-instruct:latest -o json --jq .reference)
BASELINE=$(akka training evaluations start --model "$BASE" \
--dataset support-triage/eval-frontier --project support-triage -o json --jq .evaluationId)
akka training evaluations wait "$BASELINE" --exit-statusA candidate records the training project that trained it, so it does not require --project. Optimize refuses a different project.
Evaluate a model
Start an evaluation, then read its detail, report, and predictions:
akka training evaluations start --model triage/qwen3-0.6b:v3 --dataset triage/eval --wait --exit-status
akka training evaluations get EVALUATION_ID
akka training evaluations report EVALUATION_ID
akka training evaluations predictions EVALUATION_ID
akka training evaluations list --model triage/qwen3-0.6b:v3training evaluations start takes the following options:
| Option | Default | Description |
|---|---|---|
|
none |
The model to score, by registry reference, such as |
|
none |
The evaluation dataset, by registry reference, such as |
|
|
The grader: |
|
none |
The instructions given to a |
|
The model’s training project |
The training project the score belongs to, by name or ID. A base model belongs to no training project, so set it to score a baseline. For a candidate, it must match the candidate’s training project. |
|
off |
Wait for the evaluation to finish. |
|
off |
Exit with a non-zero status if the evaluation fails or is canceled. Requires |
To cap the length of a scored answer, use an evaluate stage in a pipeline with maxCompletionTokens. See Pipelines.
When the training project uses a grader other than json-label-match, pass it with --grader. training projects scores tabulates only evaluations whose dataset and grader match the project’s evaluation settings.
The report shows the mean score, exact-match rate, and per-field accuracy, with the weakest fields first. It marks examples for which the grader returned no verdict as unscored and excludes them from the mean. predictions lists each example’s position in the split, prompt, expected answer, model answer, and score. It marks an unscored example instead of assigning it a zero.
An evaluation fails if it cannot reach the grader. training evaluations cancel stops an evaluation in progress.
Comparisons
A comparison evaluates one source snapshot and selected branch models with one pinned dataset, grader, and completion-token limit. Use a comparison to score branches under the same conditions. The comparison registers the source as a candidate if necessary.
Create a comparison, then read it:
akka training comparisons create \
--source SNAPSHOT_ID --candidate triage/qwen3-0.6b:v4 --candidate triage/qwen3-0.6b:v5 \
--dataset triage/eval --grader json-label-match --max-completion-tokens 64 \
-o json --jq .comparisonId
akka training comparisons get COMPARISON_IDUse the full snapshot ID. Creation starts durable preparation, and you can read the comparison while its phase is PREPARING or RUNNING. COMPLETED means that every target completed. PARTIAL shows completed results with failed, refused, or uncertain targets. FAILED means that no target completed. An incomplete comparison has no winner.
The contract fingerprint identifies the resolved dataset, grader, and completion limit. A zero limit is uncapped. Results include the configuration differences between targets and their training costs. The comparison counts a selected run once and excludes a selected ancestor from the ancestry total. These values are the costs of complete runs, rather than the cost up to the source checkpoint.
Promote a model
When a candidate scores better than the model that the task uses, move the named model’s champion alias to it:
akka registry models alias set triage/qwen3-0.6b:champion --version v5
akka registry models alias get triage/qwen3-0.6b:championThe registry records every move of an alias, so you can move it back to an earlier version.