0.8.0

Datasets, scoring bundles, and models are named, versioned artifacts in the artifact registry. Training projects group runs and models, and each project name is the registry namespace for its models. When your deployment allows it, you can choose how a run uses device memory. The trainer documentation describes every field in run and pipeline descriptors.

Headlines

Datasets live in the registry

Register a dataset with akka registry datasets register NS/NAME -f FILE, for example registry datasets register triage/train -f train.jsonl. Registering the same examples again returns the existing version, and --alias points an alias at the version. registry datasets list, registry datasets get, registry datasets versions and registry datasets object read what the registry holds. When an example references a local image by path, registration uploads the image. registry images push uploads images separately.

A run, a pipeline stage, an evaluation and a training project’s evaluation settings name a dataset by registry reference: triage/train resolves through latest, and triage/train:v3, triage/train:candidate and triage/train@sha256:…​ name a version directly. Optimize resolves the reference when it accepts the request, and the run records the exact version. A validation dataset is named and pinned the same way.

A dataset or a scoring bundle named without a namespace is in the training project’s namespace, so in a run descriptor for triage, train is triage/train. A reference is not case sensitive, so Triage/Train is triage/train.

A curation specifies the dataset name for its registered output, such as oasst1/curated. The judge’s dataset is registered beside it, with a -judge suffix.

Scoring bundles live in the registry

Register a scoring bundle with akka registry bundles register NS/NAME -f DIR, for example registry bundles register gsm8k/scoring -f scoring/. Registering the same files again returns the version the registry already holds. --alias points an alias at the version.

A reward rule and a bundle: grader name a bundle by registry reference: gsm8k/scoring resolves through latest, and gsm8k/scoring:v2, gsm8k/scoring:candidate and gsm8k/scoring@sha256:…​ name a version directly. Optimize resolves the reference when it accepts a run, a pipeline, or an evaluation, and the run records the exact version. Moving an alias later does not change accepted work.

Models live in the registry

A run registers its model in the artifact registry as a version of a named model in its training project’s namespace. The run’s modelName specifies the named model and defaults to the base model’s name, so a run trained on Qwen/Qwen3-0.6B in the triage project registers triage/qwen3-0.6b:v1, then triage/qwen3-0.6b:v2. The registry records each version’s base model, the model a reinforcement run started from, and the run that produced it. Read models with akka registry models get, registry models versions and registry models list.

A base model is a version of a named model identified by its repository name in lowercase, so Qwen/Qwen3-0.6B is qwen/qwen3-0.6b. Selecting a base model in the catalog registers it, and training base-models list names a selected model by its version, such as qwen/qwen3-0.6b:v2.

A run, an evaluation, an endpoint, a comparison and a model: grader name a model by registry reference: triage/qwen3-0.6b resolves through latest, and triage/qwen3-0.6b:v3, triage/qwen3-0.6b:champion and triage/qwen3-0.6b@sha256:…​ name a version directly. Optimize records the exact version, so moving an alias later does not change accepted work.

A model named without a namespace is in the training project’s namespace, so in a run descriptor for triage, "warmStartModelId": "qwen3-0.6b:v2" is triage/qwen3-0.6b:v2 and model:judge is the model triage/judge. An evaluation or endpoint resolves a model reference without a namespace in the project specified by the request.

candidate and champion are aliases on a named model. To promote a model, move its alias with akka registry models alias set triage/qwen3-0.6b:champion --version v3.

Workloads are training projects

A training project holds a task’s runs, models and evaluation settings. Use akka training projects in place of training workloads. The subcommands are unchanged, except refs and rename, which are removed. A workload in the assessment and the gateway router remains a slice of your traffic.

Train prepared token rows as given

A supervised dataset can hold rows that are already token IDs, each with a loss mask, in Parquet files beside a prepared.json descriptor. A run trains those tokens as given, without rendering a chat template or applying a loss policy. The descriptor names the tokenizer and chat template the rows were prepared with. Optimize refuses a run whose base model has another tokenizer, or whose maxSeqLength is shorter than the longest row, before it claims devices. The datasets page offers a script that writes the descriptor from the rows. See Prepared token rows.

Choose a run’s memory settings

You can set the following memory settings for your run:

  • execution.rollout.memoryShare is the share of each device that a reinforcement run’s rollout engine takes.

  • config.model.weightPrecision is the precision of the trained model’s weights, fp32 or bf16.

  • execution.memory.offload.weights moves the weights to host memory between passes.

  • execution.memory.offload.optimizer keeps optimizer state in host memory between updates.

  • execution.memory.offload.activations stores activations in host memory until the backward pass needs them.

  • execution.memory.sharding set to none keeps a complete copy of your model on each device. Use it when the model’s training state fits on one device.

A run that omits these settings uses the configured defaults. You cannot set them on a supervised run on one device. Before claiming devices, Optimize refuses a run when the setting or value is not allowed and names the setting. config.model.weightPrecision is part of the configuration hash. For the effects of each setting, see Memory settings.

A run records the value of each setting when you start it. training runs get --settings shows the values used for training and marks the ones specified by the run. A later change to the deployment does not change a run, its branches or its retries.

Set the training engine’s settings directly

Use config.engineSettings to set the training engine’s settings directly when your deployment allows them. Use these settings to fit your model to its devices. They include weight prefetching and padding, packing conversations into passes, and the rollout engine’s batch limits, prefix caching and CUDA graphs. You cannot set them on a supervised run on one device. Set data.use_dynamic_bsz to false on a supervised run to train one conversation per pass rather than packing each pass up to a token budget. An unset training.microBatchPerGpu is then 1. The loss is unchanged, and training runs get --settings and a dry run show one conversation per pass.

For a supervised run, you can also set the data types used for computation, gradient reduction and buffers. You can set Torch compile’s recompile limits and whether reaching either limit fails the run. You can set head.chunk_tokens, how many positions the output head computes log-probabilities for at once. The default is 512, and a smaller value needs less memory. You can enable Liger’s RMSNorm and GeGLU kernels for model types verified for those kernels: Gemma 4 E2B and E4B. Before claiming devices, Optimize refuses a run that enables one for any other model, Gemma 4 12B included, and the error names the setting and the model. A dry run reports the same refusal.

The settings are part of the configuration hash. training runs get --settings and training runs start --dry-run --show-resolved show them after the engine, one line per group. For a supervised run, they also show the data types used for training. Explicit data types and data.use_dynamic_bsz set to false appear on the relevant lines and are omitted from the settings list. For details about each setting, see Engine settings.

A supervised run checks itself before it trains

By default, a supervised run on more than one device checks its output head before its first update. It also checks the isolation of packed conversations, compares loss with and without packing, and compares its forward pass with plain Transformers. The comparison against Transformers runs only on an attempt that starts from a fresh adapter. If a check exceeds its bound, the run fails before it trains, and training runs failure names the check, the value and the bound. training runs get says whether the checks passed, and training runs get --checks shows each value against its bound. Set the engine setting checks.start to false to train without the checks. Each check’s bounds are also engine settings, such as checks.reference.max_logprob_error. Set a smaller bound for a stricter check or a larger one to allow a greater difference. The default bound on the difference in the mean loss depends on how many positions the check compares, so rounding does not fail a check on conversations with few supervised tokens. A bound is a number of at least 0. Optimize refuses a bound when checks.start is false, and a packing or isolation bound when data.use_dynamic_bsz is false. See Checks before the first update.

Packed passes keep conversations apart

In a supervised run on more than one device, each conversation in a packed pass attends only to its own tokens, so its log-probabilities do not depend on the other conversations in the pass. See Attention in a packed pass.

Flex attention for Gemma 4

A supervised run of Gemma 4 E2B, E4B or 12B on more than one device can set config.model.attentionImplementation to flex_attention. Optimize refuses flex_attention for a model of any other type before the run claims devices. A flex_attention run pads each pass to a multiple of engine.pad_to_length_bucket, 1024 tokens by default, and the padded size counts against the token limit the base model’s catalog entry was qualified at. Optimize refuses a flex_attention run that turns engine.pad_to_length off or sets the bucket below 128. Flex attention needs a GPU with enough shared memory per block. On a GPU without it, the run fails at startup and the error names the device. See Attention in a packed pass.

Resume from one of several saves

Set execution.keepRecoverable to retain training state for multiple saves. For example, 3 keeps the three newest saves resumable. Resume a paused run from any of them with akka training runs resume RUN_ID --step STEP. The run repeats the updates after that step. Each retained training state uses storage, so the default is 1. See Recovery.

A run with execution.memory.sharding set to none saves one copy of its training state rather than one copy for each device, so a save of Gemma 4 E4B on 8 devices uploads about 2.4 GB rather than about 15 GB. Before its first save, the run checks that the devices hold the same optimizer state. Set the engine setting checkpoint.compare_replicas to false to skip the check.

Keep an adapter for each epoch

A supervised run with execution.saveEachEpoch set to true also saves at the end of each epoch. Each epoch’s adapter is retained without counting toward the keepAdapters limit. Optimize registers it as a version of the run’s model with its epoch and step. The run continues to report its final model version. training runs snapshots shows which saves ended an epoch. See Epoch adapters.

Validate before training and at each epoch

A supervised run with hyperparams.training.validateEachEpoch set to true also evaluates on its whole validation dataset before the first update and at the end of each epoch, in addition to the evaluations every validationFrequencySteps updates. Each measurement reports the loss weighted by supervised tokens and its perplexity in training runs metrics and the run’s curves. See Validation.

On a supervised run on more than one device, hyperparams.training.validationBatchSize sets how many validation conversations each call evaluates. The default is the number of conversations in one update. Changing it does not change the validation loss, apart from rounding, and fewer, larger calls can make an evaluation faster. Optimize refuses a run that sets it without a validation dataset.

See how far each step moved the model

training runs metrics shows the gradient norm, gradient clip factor, adapter norm, and update ratio for each step. The update ratio measures the adapter update relative to the adapter norm before the step. These fields use the same names for supervised and reinforcement runs across supported training engines. See Metrics.

training runs compare RUN_ID BASELINE_RUN_ID compares the training loss of two supervised runs with the same configuration. It calculates the mean loss difference for each window, lists gradient spikes, and reports the first full window whose mean difference, in either direction, is greater than the margin. See Compare two runs.

Other changes

  • When a service cannot check your access with the platform, it answers 503 and asks you to retry after a few seconds, rather than a 403 that says you have no access, and akka retries once by itself.

  • training runs get shows a run in three parts: its state, its results and its cost. --settings, or -o wide, adds the settings the run was started with. -o json is unchanged.

  • A dataset can be as large as the deployment’s upload limit, 256 MiB unless the deployment sets another. The limit also applies to the files of a ZIP after extraction.

  • A scoring bundle can be as large as the deployment’s bundle limits: 256 MiB compressed, 2 GiB expanded and 10,000 files, unless the deployment sets others.

  • training runs snapshots and training snapshots get show how large a snapshot is, and the API reports the bytes of each artifact kind.

  • A finished run reports the last step it saved. training runs start --wait and the live progress line could report an earlier step, or that nothing was saved, for a run whose last save committed after its job ended. The run now finishes once its last saves have committed or their commit has failed, waiting at most 15 minutes.

  • A run you cancel ends CANCELLED when its cleanup fails after it has given its devices back. Its failure reason then says that its cleanup failed. A run that still holds its devices keeps retrying its cleanup, as before. A cancelled evaluation whose cleanup retries no longer reports a failed phase.

  • A run that finishes with saves it could not record names them by step in training runs get and in the run API’s unrecordedSaves.

  • A run’s validation loss is the mean over every supervised token in the validation set. A supervised run on one device weighted each batch’s loss by the examples in the batch rather than by their supervised tokens in earlier releases, so its validation losses are not comparable with those of earlier runs.

  • The comparison table in training projects scores has one row per base model version, such as qwen/qwen3-0.6b:v2, with the repository’s spelling beside it, such as Qwen/Qwen3-0.6B.

  • The trainer pages document every descriptor field, including kind, submissionId, smoke, execution.engine, the pipeline descriptor’s fields, the curate stage and the options of training evaluations start.

  • Saving, pausing, resuming and branching a run spend less time moving checkpoints.

  • A dry run reports dataset findings, such as exact and near duplicates, for the version the run would train on. training runs start shows these findings before it asks you to confirm.

  • Optimize refuses a run, a pipeline, or an evaluation whose dataset or scoring bundle its training jobs cannot read. The reason identifies the part of the location that did not match. The refusal occurs when you submit, before any GPU is claimed.

  • Set hyperparams.training.tokensPerGpu on a supervised run to limit the tokens in one pass on one device. This lets you fill each device with whole conversations without raising maxSeqLength. The budget cannot exceed the qualified limit in the base model’s catalog entry. Optimize refuses it on a run on one device. training runs get --settings and a dry run show it. See Supervised hyperparameters.

  • A supervised run on more than one device trains every conversation in every epoch. This includes the remaining conversations at the end of an epoch, even when their count is not a multiple of the device count. The reported updates for an epoch include the update that trains those conversations.

  • Optimize refuses a supervised run on more than one device that sets data.packing to true. The reason explains that the run already packs whole conversations into each pass, or trains one conversation per pass where data.use_dynamic_bsz is false.

  • A supervised run on more than one device tokenizes its rows before claiming GPUs. This avoids holding GPUs idle while tokenizing a new dataset. The run is in PREPARING while row preparation waits to be scheduled and while it runs. It reports the completed and total row counts. The run fails if preparation reports no progress within the deployment’s startup timeout. For prepared token rows, Optimize checks the rows against the dataset and the base model’s tokenizer and chat template. A mismatch fails the run before it claims GPUs. You cannot pause a run while it tokenizes its rows. Cancelling it requests that tokenization stop. A supervised run on one device tokenizes its rows after it claims its device.

  • By default, a supervised run on more than one device computes the model’s output layer only at the tokens it trains on, so a run over long conversations uses less memory and takes less time per step. The loss and gradients are unchanged. See Loss.

  • A supervised run of Gemma 4 E2B or E4B on more than one device uses less device memory per pass, about 8 GB less at 196,608 tokens on E4B. The loss and gradients are unchanged.

  • A run that trains an adapter on more than one device combines its gradients across devices once per update, after the last pass, rather than after every pass. Devices no longer wait for each other at every pass, so an update over passes of different lengths, such as one conversation per pass, takes less time. With bf16 computation the gradients differ by rounding. Set engine.use_no_sync_for_gradient_accumulation to false in config.engineSettings to combine them after every pass. See Engine settings.

  • If a request has no system or developer message, an endpoint uses the model’s training instructions. training endpoints get shows them. Use training endpoints send --instructions to provide different instructions. See Endpoints.

  • A base instruct model can be a model: judge. A judge uses the first available instructions: graderInstructions specified alongside the grader, the judge’s own training instructions, or the built-in judge instructions. The built-in instructions carry the scored dataset’s first system message, where it has one, as context. training evaluations start, training projects create, training projects set-evaluation and training comparisons create take --grader-instructions. training evaluations get and training runs get show the instructions given to the judge and their source. See The judge’s instructions.

  • A run checks that each saved adapter loads onto the base model. It fails if the adapter has missing, extra, or misnamed tensors, or tensors whose rank differs from its configuration. The reason states how many tensors and parameters the adapter holds and how many the base model expects. A save whose adapter fails the check keeps its training state but no weights, so no model can be registered from it, and the run fails at that save.

  • An evaluation of an adapter at a rank the inference engine does not load directly, such as 24, runs at the next rank the engine accepts.

  • A run or an evaluation no longer fails because the training cluster does not answer for a few minutes. It keeps its phase and retries for up to 15 minutes. After that, it ends, and training runs failure says why.

  • akka reaches every service at AKKA_OPTIMIZE_URL, the address of your project’s route, so you set one address for a deployed project rather than one for each service. A service’s own variable, such as TRAINER_URL or REGISTRY_URL, still takes precedence for that service. See CLI commands for training.

  • training endpoints info shows what an endpoint in your deployment can serve: the largest adapter rank it loads, how many adapters it loads at once, the devices it reserves and their type, the longest context it serves, how long an idle endpoint waits before it stops, and the tool-call and reasoning parsers for each architecture. See What an endpoint can serve.

  • training adapters import NS/NAME --base BASE_MODEL -d DIR imports a PEFT LoRA adapter trained outside Optimize, so an endpoint can serve it. Optimize checks the adapter’s files against the base model and the rank an endpoint loads, checks that the adapter loads, and registers it. It refuses a DoRA adapter and one with modules_to_save. A refused import registers nothing. training adapters get IMPORT_ID shows an import’s progress. See Serve an adapter trained elsewhere.

  • An endpoint on the vLLM engine returns tool calls and reasoning for a model whose architecture has a parser for them, such as Gemma 4. A request can offer the model tools, and the endpoint returns each tool call and the reasoning separately from the answer. training endpoints send --tools FILE offers the tools in a file, and training endpoints info shows the parsers for each architecture. An endpoint for a model with no tool-call parser refuses a request that offers tools, and the reason names the model’s architecture. See Tool calls and reasoning.

  • Text output names a training project by its name, such as gsm8k, in training runs list, training endpoints list, training capacity, live progress and training adapters import. JSON output retains the project ID.

  • Hints, errors and help examples name the command as you ran it: akka optimize under the Akka CLI, and akka-optimize as a standalone binary.

  • Optimize refuses an endpoint for an adapter above the rank training endpoints info shows when you create it, rather than failing at the first request.

  • A registry upload, such as registry datasets register, fails after 60 seconds without progress or 10 minutes without an answer, rather than after 30 seconds in total. --timeout still sets one limit for the whole upload.

  • training runs wait and training runs start --wait show the run’s final step count on the training line.

  • registry info shows the registry’s build version.

Breaking changes

  • History. 0.8.0 does not carry over what 0.7 held: runs, evaluations, pipelines and branch comparisons, workloads, trained models and base model selections, and the datasets and bundles pushed to it. Register your datasets and bundles again.

  • Dataset commands. Dataset commands are under registry datasets, beside registry bundles and registry models. datasets create, datasets name and datasets images push are removed. Use registry datasets register NS/NAME -f FILE, --alias, and registry images push.

  • Dataset names. A dataset is named by registry reference, such as triage/train, rather than by a content hash or a name such as triage-train. Register your datasets with the registry, then name them by reference in every run or pipeline descriptor, evaluation and training project.

  • datasetHash. Every request and response field named datasetHash is now dataset, and judgeDatasetHash is judgeDataset. Each holds a registry reference. Rename the fields, and replace each hash with a registry reference.

  • Curations. A curation request needs a dataset field naming the dataset to register its output under. Add one to each curation request, such as "dataset": "oasst1/curated".

  • Dataset findings. A dataset no longer records findings when you register it. Read them with training runs start -f PLAN --dry-run.

  • Dataset routes. POST /v1/training/datasets, GET /v1/training/datasets, GET /v1/training/datasets/REFERENCE, PUT /v1/training/datasets/names/NAME, POST /v1/training/datasets/images and POST /v1/training/datasets/images/missing are removed. Use the registry datasets and registry images commands, or the registry’s routes under /v1/registry.

  • Bundle commands. training bundles push is removed. Use registry bundles register NS/NAME -f DIR.

  • Bundle names. A bundle is named by registry reference, such as gsm8k/scoring, rather than by a content hash or a name such as gsm8k-scoring. Register your bundles with the registry, then name them by reference in your plans.

  • Grader specification. A bundle grader is bundle:REFERENCE#MODULE or bundle:REFERENCE#MODULE:FUNCTION, for example bundle:gsm8k/scoring#reward:compute_score. In 0.7, it was bundle:BUNDLE/MODULE:FUNCTION.

  • Bundle routes. POST /v1/training/bundles and POST /v1/training/bundles/missing are removed. Use registry bundles register, or the registry’s routes under /v1/registry.

  • training projects. training workloads is renamed training projects, and every --workload flag is --project.

  • Descriptors. A run or pipeline descriptor names its training project with "project", in place of "workload".

  • Project references and base model. A training project no longer holds references or a base model. training workloads refs and the routes under /v1/training/workloads/WORKLOAD/refs are removed. Every run and every pipeline train stage names its baseModel, and a descriptor that names none is refused.

  • Project routes. The routes under /v1/training/workloads are under /v1/training/projects. The fields workload, workloadId and workloadName are project, projectId and projectName, and GET /v1/training/capacity takes ?project=.

  • Capacity. training capacity groups claims and quotas under Training projects.

  • Grafana links. training runs grafana takes the platform project as --platform-project, because --project names a training project in every other training command.

  • Refused actions. A refused request names its action under optimize.training.projects, such as optimize.training.projects.create, where earlier releases used optimize.training.workloads. The permissions that grant each action are unchanged. Update a script that matches optimize.training.workloads in a refusal.

  • Model names. A model is named by registry reference, such as triage/qwen3-0.6b:v3, rather than by a 64-character ID or the start of one. A run’s modelId, an evaluation’s modelId, warmStartModelId and a model: grader hold a version reference. Replace each model ID with a registry reference.

  • Model commands. trained-models is removed. Use registry models get, registry models versions, registry models register and registry models annotate.

  • Model routes. POST /v1/training/models, GET /v1/training/models, GET /v1/training/models/MODEL_ID and PUT /v1/training/models/MODEL_ID/annotations are removed. Use the registry models commands, or the registry’s routes under /v1/registry.

  • Promotion. candidate and champion are aliases on a named model, set with registry models alias set.

  • Project names. A training project name follows the registry’s naming rules and cannot change. training projects rename and PUT /v1/training/projects/PROJECT/name are removed. Choose a project’s name when you create it.

  • Base model selection. Selecting a repository that the registry cannot name as a model is refused. training base-models list shows failed for a selection the registry did not register, and the runs that waited on it fail with the reason. Select a repository the registry can name, then submit the runs again.

  • Registered names. A run cannot register its model under a named model that holds base model versions. A run that names no modelName registers under its base model’s name. If your project is named for the base model’s owner, such as qwen for Qwen/Qwen3-0.6B, set modelName. Optimize refuses a run when that name holds versions trained on another base model, such as unsloth/Llama-3.2-1B-Instruct alongside meta-llama/Llama-3.2-1B-Instruct. Set modelName to give the models separate names.

  • Snapshot candidates. A candidate registration reports registrationId in place of modelId, and model names the version once it is REGISTERED. Read registrationId in place of modelId.

  • Base model listing. A base model entry’s modelId is model, which is the registered version or repository@commit if the entry has not been selected.

  • Comparison table. A row of GET /v1/training/projects/PROJECT/scores has baseModel, a base model version, in place of baseModelId and revision.

  • Endpoint detach. DELETE /v1/training/endpoints/ENDPOINT_ID/models/NAMESPACE/NAME detaches the version an endpoint serves by any registry reference to it, such as triage/qwen3-0.6b or triage/qwen3-0.6b:champion.