Datasets

A dataset is a set of training or evaluation examples. The artifact registry holds datasets as named, versioned artifacts. A named dataset, such as triage/train, holds immutable numbered versions and aliases that you can move between versions. Optimize resolves a dataset reference when it accepts a run, an evaluation, or a pipeline. It records the exact version, so moving an alias later does not change the training data.

Use datasets to provide examples for training and evaluation. Keep training and evaluation datasets separate. A training project scores every candidate with the evaluation dataset, so an example that appears in both datasets inflates the score.

Format

A dataset uses chat-format JSONL, holds preference pairs as Preference pairs describes, or holds prepared token rows as Prepared token rows describes. Each line of chat-format JSONL contains one JSON object with a messages array of system, user, and assistant messages. The following example shows one row:

{"messages":[
  {"role":"system","content":"Extract the invoice total as JSON."},
  {"role":"user","content":"Invoice 1042, 3 items, total due 1290.00 EUR"},
  {"role":"assistant","content":"{\"total\":1290}"}
]}

Supervised fine-tuning teaches the model to produce the assistant message. A reinforcement run samples its own completions and uses the assistant message as the gold answer when the reward requires one. During an evaluation, the grader compares the model’s answer with the assistant message.

Conversations and tool calls

A row is a whole conversation rather than one exchange. Roles are system, user, assistant, and tool. A row ends with an assistant message, the answer, which is what a run trains towards. An assistant message may carry tool_calls. Before the answer, each call is answered by a tool message naming the tool_call_id it answers. A row declares the tools it may call in a row-level tools list, which the chat template renders.

Any assistant message may carry text and tool_calls together, as an agent writes a sentence and calls a tool in the same message. The answer’s calls need no tool message after them. Such a row is what a distillation writes for a teacher that answers with a call. A message with calls and no text may omit content.

The following example is one three-turn row with a tool call:

{"tools":[{"type":"function","function":{
   "name":"get_weather",
   "description":"Look up the current weather for a city",
   "parameters":{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}}}],
 "messages":[
  {"role":"system","content":"You are a weather assistant."},
  {"role":"user","content":"What is the weather in Wellington?"},
  {"role":"assistant","content":"","tool_calls":[{"id":"call_1","type":"function",
     "function":{"name":"get_weather","arguments":{"city":"Wellington"}}}]},
  {"role":"tool","tool_call_id":"call_1","name":"get_weather","content":"18C and windy"},
  {"role":"assistant","content":"It is 18C and windy in Wellington."}
]}

A system message is optional. You can register a row without one, and a chat template renders the conversation without one.

Optimize refuses registration of a row it cannot render or mask. The refusal names the line and the reason:

  • a row that does not end with an assistant message

  • a row with no user or no assistant message

  • a role other than the four above

  • a system message that is not at the head of the conversation

  • a tool message that follows no assistant message carrying tool_calls

  • a tool message whose tool_call_id the preceding assistant message did not call

  • a tool call before the answer that no following tool message answers

  • an assistant message with two tool calls that share an id

  • an image part outside a user message

A run trains an answer that calls tools whole, its text and its calls. An earlier assistant message that calls tools is context under the default loss policy. Under loss.assistantTurns: all it trains whole, and with loss.trainOnToolCalls: false its text trains without its calls. See Training runs.

The chat template has to render every assistant message that calls tools as it was written, its text and each of its calls. A dataset analysis names a row where the template refuses such a message, or renders it without its text or one of its calls. The message can be the answer or an earlier one, and an earlier call is checked whether or not its message carries text. Optimize refuses a supervised or preference run over such a row before it claims a GPU, under every data.truncation value. A run also refuses a row where the template moves an earlier message’s text after its tool results, since the tokens of that message cannot be told apart. A dataset with images does not train an answer that calls tools, or an earlier message that carries text with its calls, and Optimize refuses such a row when the run reads it. Optimize trains the Gemma 4 models in the catalog under corrected chat templates that render a message’s text before its calls, the order the model writes them in.

Evaluation and reinforcement learning compare an answer’s text, so they refuse a row whose answer calls tools. A supervised run leaves such rows out of the answers its smoke check scores, and reports how many it left out and why. A smoke check whose slice holds only such rows reports no reproduction score and says why. A reinforcement run also refuses a row whose prompt the model’s chat template refuses, such as a tool call whose arguments are not a JSON object under the Gemma 4 templates. The refusal names the line and quotes the template’s reason, before training starts.

sft and dpo over text render each row’s tools and train the rows that offer them. A reinforcement run does not render a row’s tools, and neither does sft over a dataset with images, so training such rows would not match evaluation and serving, which render them. The registry counts the rows that offer tools when a version is uploaded, and akka registry datasets get shows the count on its Tools line. Optimize refuses a reinforcement run, and an sft run over images, when a dataset it reads has rows that offer tools, and names the engine and the reason. A dry run reports the same refusal. A version shows not counted where its content was first uploaded before the registry counted these rows, and Optimize does not refuse a run over it for its tools.

A dataset’s diagnostics report how many rows are multi-turn, how many tool calls the dataset makes, and how many rows declare tools. None of these diagnostics raises a finding. Use the run’s loss policy to specify how to train on multi-turn or tool-calling rows. See Training runs.

Preference pairs

A preference dataset holds pairs of answers to the same prompt: one to prefer and one to avoid. A dpo run trains on it. See Direct preference optimization. Runs of other methods, evaluations and distillations refuse it, and the refusal names the dataset.

Each line is one JSON object with a prompt array, a chosen answer, a rejected answer and an optional tools list. The messages take the same form as in a chat row. The following example is one pair whose chosen answer calls a tool:

{"tools":[{"type":"function","function":{
   "name":"get_weather",
   "description":"Look up the current weather for a city",
   "parameters":{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}}}],
 "prompt":[
  {"role":"system","content":"You are a weather assistant."},
  {"role":"user","content":"What is the weather in Wellington?"}
 ],
 "chosen":{"role":"assistant","content":"","tool_calls":[{"id":"call_1","type":"function",
    "function":{"name":"get_weather","arguments":{"city":"Wellington"}}}]},
 "rejected":{"role":"assistant","content":"It is sunny everywhere today."}}

The prompt is the conversation the answers reply to, so it ends with a user message or with the tool messages that answer its last calls. An assistant message in the prompt may carry text and tool_calls together. Each answer is one assistant message, and follows the rules of a chat row’s answer: it may carry text and tool_calls together, and its calls need no tool message.

Optimize refuses registration of a pair it cannot use. The refusal names the line and the reason:

  • a prompt that is empty, has no user message, or ends with an assistant message

  • a system message that is not at the head of the prompt

  • a tool call in the prompt that no following tool message answers

  • an assistant message, in the prompt or as an answer, that gives two of its calls the same ID

  • a chosen or rejected that is not one assistant message

  • an answer carrying neither tool_calls nor text

  • a chosen and a rejected that are the same: the same text with surrounding whitespace ignored, and the same calls in the same order. Call IDs do not count, and arguments are compared as JSON, so an object and the JSON string that encodes it are the same arguments

  • an image part anywhere in the line

A dataset holds chat rows or preference pairs, not both. Where you state no format, registration reads the first non-blank line to decide which: the first non-blank line of the file, or of examples-00000.jsonl in a ZIP file of shards. A line with chosen makes the dataset preference pairs, and any other line makes it chat. Registration then refuses each line of the other kind, naming it. To state the format, register with --format preference-jsonl-v1. registry datasets register reports how many pairs it registered.

Prepared token rows

A supervised dataset can hold rows that are already token IDs, each with a loss mask that marks the tokens to train on. A run trains these rows as given. It does not render a chat template and does not apply a loss policy.

Use prepared rows when you have decided exactly which tokens carry loss and need a run to train those tokens. With chat examples, Optimize tokenizes each conversation and applies the run’s loss policy. A different template revision, tokenizer, or masking rule can change the trained tokens without reporting an error.

Prepared rows are for the training and validation datasets of a supervised run. An evaluation, a reinforcement run and a smoke check read prompts, which prepared rows do not hold. An evaluation of a model trained on prepared rows names its dataset, since it cannot default to the training dataset. Such a model has no training instructions. An endpoint serves it without instructions. To use it as a model: grader, specify graderInstructions.

Rows files

The rows are in one or more Parquet files. Each row has an input_ids and a loss_mask list column of the same length. loss_mask[i] is 1 where token i is trained on and 0 elsewhere. The first token is never marked, because no token predicts it, and every row marks at least one token. Other columns are ignored.

Prepare the rows with the base model’s own tokenizer, at the commit the run trains, and under the chat template the run renders under.

The descriptor

A prepared version is a ZIP file containing prepared.json alongside the Parquet files it lists:

{"tokenizer":{"repository":"google/gemma-4-E4B-it",
              "revision":"ee0ef6023621cff504d758262d4e04895a5af4a2"},
 "tokenizerSha256":"bca34862f49c6a63155300a313b0d3f901ad1c16d619d19c2677ba963449085d",
 "chatTemplateSha256":"0a2c8073c878ab1da004bee933a998606537bbb62016310352c7285c3f01c5b5",
 "files":[{"path":"train.parquet",
           "sha256":"5f0c9e1d2a7b4c3e8f6a1d0b9c8e7f6a5b4c3d2e1f0a9b8c7d6e5f4a3b2c1d0e",
           "rows":2000,"tokens":3412880,"supervisedTokens":402117,"longestRow":8190}]}
  • tokenizer names the model repository and the full commit the tokenizer was loaded from.

  • tokenizerSha256 is the SHA-256 of the tokenizer’s serialized form, tokenizer.backend_tokenizer.to_str() in Hugging Face Transformers.

  • chatTemplateSha256 is the SHA-256 of the chat template the rows were rendered under. A run renders under the template the repository ships at that commit, or under the corrected template where Optimize corrects that commit.

  • files lists every Parquet file in training order, with the SHA-256 of its bytes, its row count, its token count, how many tokens its loss masks mark, and the token count of its longest row.

The describe_prepared.py script reads every row and writes the descriptor. It needs transformers and pyarrow, and it refuses a row that a run would refuse. With --zip, it writes the zip to register. To download it, see the descriptor script.

python describe_prepared.py --model google/gemma-4-E4B-it --zip train.zip train.parquet
akka registry datasets register gemma/train -f train.zip

--revision names a branch, tag or commit of the model. The descriptor records it as a commit.

Optimize trains these Gemma 4 commits under a corrected chat template:

Repository Commit Template

google/gemma-4-E2B-it

3e22461f65e89153144f8adb70e3b8c2cc9845a7

gemma-4-E2B-E4B-it.jinja

google/gemma-4-E4B-it

ee0ef6023621cff504d758262d4e04895a5af4a2

gemma-4-E2B-E4B-it.jinja

google/gemma-4-12B-it

707f0a3b8a3c7ad586ed01e27eafbad8a27dd0f7

gemma-4-12B-it.jinja

For one of these commits, render the rows under the corrected template, for example with tokenizer.apply_chat_template(messages, chat_template=Path("gemma-4-E2B-E4B-it.jinja").read_text(encoding="utf-8")) in Hugging Face Transformers. Then name the template’s file with --chat-template and the commit with --revision, so the descriptor records the template’s digest at that commit:

python describe_prepared.py --model google/gemma-4-E4B-it --revision ee0ef6023621cff504d758262d4e04895a5af4a2 --chat-template gemma-4-E2B-E4B-it.jinja --zip train.zip train.parquet

What is checked

Registration refuses a version whose zip holds a file the descriptor does not list, lacks a file it lists, or holds a file whose SHA-256 differs from the one stated.

Before claiming devices, Optimize refuses a run on a prepared dataset when:

  • the rows were prepared with a tokenizer other than the base model’s, at another repository or commit

  • the base model is a Gemma 4 commit that Optimize trains under a corrected chat template, and the rows were rendered under another template

  • the longest row is longer than the run’s maxSeqLength

  • the run is a reinforcement run, or asks for a smoke check

  • one of the training and validation datasets is prepared and the other holds chat examples

  • the run states a loss group with values other than the defaults, or data.truncation set to right

Before training, Optimize reads every row. It checks each file’s row count and the dataset’s total counts and longest row against the descriptor. It also checks the tokenizer digest against the base model’s tokenizer, and the chat template digest against the template the run renders under. A run on more than one device performs these checks before claiming GPUs. A dry run reports no dataset findings for prepared rows.

Register a dataset

Register a dataset under a namespace and a name, then read and list datasets:

akka registry datasets register triage/train -f train.jsonl
akka registry datasets register triage/train -f train.jsonl --alias candidate
akka registry datasets get triage/train
akka registry datasets versions triage/train
akka registry datasets list --namespace triage

register prints the registered version, such as triage/train:v1. Registering the same examples under the same name again returns the existing version. Changing the examples adds a version. The first registration creates the named dataset and its namespace. --alias points an alias at the registered version. registry datasets alias set moves an alias without another upload.

A ZIP file can be as large as the deployment’s upload limit, which is 256 MiB unless the deployment sets another. The limit also applies to the total size of the files in a ZIP after extraction. Registration refuses a larger upload and states the limit.

A JSONL file of up to 256 MiB is sent in one upload. register cuts a larger file into shards of up to 256 MiB on line boundaries, sends them one at a time, and registers them as one version. A version holds at most 1,000 shards. To cut a file at another size, or to send a smaller file in shards, add --shard-size:

akka registry datasets register triage/train -f train.jsonl --shard-size 64MiB

The cut depends only on the file and the shard size, so the same file cut at the same size registers as the same version on every deployment. register refuses, before it sends anything, a shard larger than the deployment takes, more shards than a version holds, or a file larger than one upload where the deployment takes less than 256 MiB in one. The refusal names the --shard-size that fits. akka registry info shows the deployment’s limits.

If a sharded upload stops, run the same command again within 7 days. It sends only the shards the registry does not already hold. A file that changed since is sent again in full.

A run descriptor, a pipeline stage, a training project’s evaluation settings, and an evaluation command name a dataset by registry reference:

  • triage/train resolves through the latest alias, which always points at the newest version.

  • triage/train:v3 names a version by number.

  • triage/train:candidate names the version an alias points at.

  • triage/train@sha256:…​ names a version by its digest.

A reference without a namespace names a dataset in the training project’s namespace, so in a run descriptor for the triage project, train is triage/train and train:v3 is triage/train:v3. A registry datasets command has no training project to read one from, so name the namespace there. References are case-insensitive: Triage/Train is triage/train.

When you submit the request, Optimize refuses a reference that does not exist in the registry. This happens before it claims any devices.

Analyze a dataset

An analysis measures a dataset under a base model’s tokenizer and chat template, without a GPU. Use it to choose maxSeqLength, the loss policy and the GPU count before a run claims any GPU:

akka training datasets analyze triage/train --model Qwen/Qwen3-0.6B --wait
akka training datasets analyze triage/train -f run.json --wait

The dataset is a registry reference that names its namespace. --model names the base model by repository. With -f, you can leave --model out, and the base model is the run file’s baseModel. Optimize refuses a --model that names another base model than the run file. Optimize measures the examples of the dataset version the reference resolves to, under the commit the base model pins, with the tokenizer and chat template a run uses. A dataset with images is measured as a run measures it, image tokens included. A dataset of preference pairs is measured as a dpo run reads it. Optimize refuses a dataset of prepared token rows, since those rows carry their lengths already.

Optimize keeps one analysis for each set of examples, base model commit and Optimize version. Running the command again for the same three returns that analysis. Two dataset versions or names that hold the same examples share one analysis. Changed examples, a new commit of the base model, or a new Optimize version is a new analysis. Optimize starts a failed analysis again on the next request. An analysis that refused the dataset is returned as it is, and no new attempt starts.

Without --wait, the command starts the analysis or reads it, and prints its status: pending, running, done or failed. With --wait, it waits until the analysis is done or failed, and then prints the report. --timeout limits the wait, and the analysis continues after the wait ends. During the wait, a read that fails with a server error or a dropped connection is tried again a few times, after which the wait ends with an error and the analysis continues. A refusal, such as a missing permission, ends the wait at once. The estimates are for the dataset version that the command’s first answer named, even where the reference has moved to another version during the wait. If the dataset or the base model resolves to another analysis by then, the command fails and asks you to run it again. A failed analysis prints its reason and exits with a non-zero status. -o json returns the whole report as a document.

What the report contains

A row’s length is the token count a run compares against maxSeqLength. For most chat templates, that is the whole conversation, with its tools. Rows are numbered from 0 among the dataset’s non-blank lines, the number a run’s refusal of a row names.

A row’s trimmed length is the shortest length data.truncation: oldest can bring the row to by removing its oldest turns, keeping the system prompt and the final turn. A run under oldest or oldest_or_drop fits a row exactly where its trimmed length is within maxSeqLength. A chat template can render a shorter conversation to more tokens, so Optimize measures every count of removed turns and takes the shortest. A single-turn row is its own trimmed length. A row of at most 512 tokens fits every common limit as it is, so Optimize does not render it again. Optimize records the whole length as an upper bound. This bound is sufficient to check limits of 512 tokens or more. A longer row is rendered once more for each turn after its first, so a dataset of long multi-turn rows takes longer to analyze, and that time counts against the time an analysis is allowed. A dataset with images has no trimmed lengths, since Optimize does not trim one. In -o json, trimmed holds the figures of the trimmed lengths: the longest, the rows longer than each common limit, and the longest rows.

The report contains the following figures:

Figure Meaning and source

Measured under

The repository and commit of the tokenizer, the first characters of the SHA-256 of the chat template the rows were rendered with, whether that template is a corrected copy or the one the repository ships, and the Optimize version. Every other measured figure comes from these.

Rows

How many rows the dataset holds.

Tokens

The median, 90th percentile, 99th percentile and longest row length, by nearest rank.

Over 512 to Over 32768

How many rows are longer than each of 512, 1024, 2048, 4096, 8192, 16384 and 32768 tokens, the common limits. Under data.truncation: error, a run at that maxSeqLength refuses these rows. Under drop, it leaves them out. Where any row is longer, the report also shows how many are still longer with their oldest turns removed. Under oldest, a run at that maxSeqLength trims the others and refuses these. Under oldest_or_drop, Optimize leaves these rows out unless their mask or their tool calls cause a refusal.

Longest

The longest rows, longest first, by row number and length. The text output names the first few. -o wide and -o json name every row the report holds.

Suggested maxSeqLength

The smallest common limit that no row is longer than, and the smallest common limit at or above the 99th percentile, with the number of rows longer than it. Either is absent where no common limit is large enough.

Default and All assistant turns

The tokens a run’s loss is taken over, summed across every row, under the default loss policy and under assistantTurns: all. The sums are counted before any row is cut or left out by data.truncation. A sum is absent where a row cannot be trained under that policy, and the report names the reason. See Training runs.

Preference pairs

In a dataset of preference pairs, each row is a pair, and the report says pairs where it says rows. A pair’s length is the longer of two lengths: the prompt with the chosen answer, and the prompt with the rejected answer. A dpo run compares that length against maxSeqLength. The figures are the same as for chat examples, counted in pairs. A dpo run removes no turns, so the report shows no trimmed lengths.

In place of the loss policy figures, Chosen and rejected answers is the tokens of both answers, summed across every pair. A dpo run takes its loss over these tokens. The sum is absent where a pair cannot be trained, for example where the chat template does not render an answer after its prompt, and the report names the first such pair as pair N. Optimize then refuses a dpo run over the dataset, under every data.truncation value. Optimize refuses the analysis where the base model’s chat template moves the system prompt, since a dpo run refuses such a template.

Estimates

Once an analysis is done, the report estimates the steps and the time of a supervised run over every analyzed row. For a dataset of preference pairs, the estimates are for a dpo run, and each row is a pair. The estimates do not leave out the rows that data.truncation: drop would leave out. -f reads a run file, as training runs start -f does, and the estimates are for a run of its config and execution. The file’s dataset is not read. Without -f, the estimates are for a run with the default settings. Optimize refuses a run file whose config trains by reinforcement, a dpo config over chat examples, and any other config over preference pairs. Optimize also refuses a dpo run file that a dpo run would be refused for, such as one that sets data.packing or asks for more than one GPU.

Steps is the estimated step count. Each value it is computed from is listed with its source. run file means the run file supplied the value the run uses. default means the run file left the value out, gave zero, or gave a value the run replaces, such as execution.gpus: -1. The values are the epochs, the maximum steps, maxSeqLength, microBatchPerGpu, gradientAccumulation and the GPUs.

Batch is the rows one step holds: microBatchPerGpu times gradientAccumulation times the GPUs, and no more than the row count rounded up to a multiple of the GPU count. The steps are the maximum steps where the run sets them, and otherwise the epochs times the steps one epoch takes to cover every row. The steps are absent, with the reason, where the dataset holds fewer rows than the run has GPUs, or where the run packs rows, since the analysis does not count how many rows fit each sequence.

Time comes from the newest completed supervised runs of the same base model, on the device profile the base model is qualified on for this run’s GPUs, under the engine this run would train under, including one the run file chooses with execution.engine, and at its precision. Smoke checks, branches and packed runs are not counted. Those runs can have trained on other GPU counts and other datasets. For each run, Optimize divides the GPU seconds it reported by its steps and by the rows each step held. The estimate scales the median of those figures to the estimated steps, batch and GPUs: the median times the steps and the batch gives the GPU seconds, and dividing by the GPUs gives the time. Measured from names the median, how many runs it was taken over, the base model, the device profile, the engine and the precision. A run’s GPU seconds include its start-up and its data preparation, and the estimate does not tell a long row from a short one. The time is absent, and the report names the reason, in these cases:

  • no such run recorded its GPU time and steps;

  • the base model is not qualified for supervised training at the run’s precision on its GPUs, or on its execution.deviceProfile where the run file names one;

  • Optimize cannot tell which engine the run would train under;

  • the steps are absent, so there is no step count to time;

  • the run is a dpo run. The times come from completed supervised runs, so a dpo run’s time is not estimated.

Images

A user message can carry text and image parts, as in the following example:

{"messages":[
  {"role":"system","content":"Extract the invoice total as JSON."},
  {"role":"user","content":[
    {"type":"text","text":"Read this invoice."},
    {"type":"image","path":"images/invoice-1042.png"}
  ]},
  {"role":"assistant","content":"{\"total\":1290}"}
]}

path is relative to the JSONL file. registry datasets register reads each image and uploads those that the registry does not already have. It replaces each image path with the image’s content hash when registering the examples, so a second version that uses the same images uploads only its text. --no-upload reports missing images without uploading or registering anything. The registry rejects an example with an image that it does not hold. To upload images separately, run akka registry images push PATH…​.

Training on images uses a vision model’s processor and adapts its language backbone. The base model must therefore accept images. Graders receive text. Only sft trains on images, and Optimize refuses it over rows that offer tools. Optimize refuses a reinforcement run over a dataset with images. See Conversations and tool calls.