Datasets

A dataset is a set of training or evaluation examples. The artifact registry holds datasets as named, versioned artifacts. A named dataset, such as triage/train, holds immutable numbered versions and aliases that you can move between versions. Optimize resolves a dataset reference when it accepts a run, an evaluation, or a pipeline. It records the exact version, so moving an alias later does not change the training data.

Use datasets to provide examples for training and evaluation. Keep training and evaluation datasets separate. A training project scores every candidate with the evaluation dataset, so an example that appears in both datasets inflates the score.

Format

A dataset uses chat-format JSONL, or holds prepared token rows as Prepared token rows describes. Each line of chat-format JSONL contains one JSON object with a messages array of system, user, and assistant messages. The following example shows one row:

{"messages":[
  {"role":"system","content":"Extract the invoice total as JSON."},
  {"role":"user","content":"Invoice 1042, 3 items, total due 1290.00 EUR"},
  {"role":"assistant","content":"{\"total\":1290}"}
]}

Supervised fine-tuning teaches the model to produce the assistant message. A reinforcement run samples its own completions and uses the assistant message as the gold answer when the reward requires one. During an evaluation, the grader compares the model’s answer with the assistant message.

Conversations and tool calls

A row is a whole conversation rather than one exchange. Roles are system, user, assistant, and tool. A row ends with an assistant message. An assistant message may carry tool_calls, and each call is answered by a tool message naming the tool_call_id it answers. A row declares the tools it may call in a row-level tools list, which the chat template renders.

The following example is one three-turn row with a tool call:

{"tools":[{"type":"function","function":{
   "name":"get_weather",
   "description":"Look up the current weather for a city",
   "parameters":{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}}}],
 "messages":[
  {"role":"system","content":"You are a weather assistant."},
  {"role":"user","content":"What is the weather in Wellington?"},
  {"role":"assistant","content":"","tool_calls":[{"id":"call_1","type":"function",
     "function":{"name":"get_weather","arguments":{"city":"Wellington"}}}]},
  {"role":"tool","tool_call_id":"call_1","name":"get_weather","content":"18C and windy"},
  {"role":"assistant","content":"It is 18C and windy in Wellington."}
]}

A system message is optional. You can register a row without one, and a chat template renders the conversation without one.

Optimize refuses registration of a row it cannot render or mask. The refusal names the line and the reason:

a row that does not end with an assistant message
a row with no user or no assistant message
a role other than the four above
a system message that is not at the head of the conversation
a tool message that follows no assistant message carrying tool_calls
a tool message whose tool_call_id the preceding assistant message did not call
a tool call that no following tool message answers
an assistant message carrying both tool_calls and text
an image part outside a user message

A chat template renders an assistant message that carries tool_calls as its calls, the results that answer them, and then its text, so the text moves when the result arrives and its tokens cannot be attributed to the message. Put the text in a separate assistant message before the call.

A dataset’s diagnostics report how many rows are multi-turn, how many tool calls the dataset makes, and how many rows declare tools. None of these diagnostics raises a finding. Use the run’s loss policy to specify how to train on multi-turn or tool-calling rows. See Training runs.

Prepared token rows

A supervised dataset can hold rows that are already token IDs, each with a loss mask that marks the tokens to train on. A run trains these rows as given. It does not render a chat template and does not apply a loss policy.

Use prepared rows when you have decided exactly which tokens carry loss and need a run to train those tokens. With chat examples, Optimize tokenizes each conversation and applies the run’s loss policy. A different template revision, tokenizer, or masking rule can change the trained tokens without reporting an error.

Prepared rows are for the training and validation datasets of a supervised run. An evaluation, a reinforcement run and a smoke check read prompts, which prepared rows do not hold. An evaluation of a model trained on prepared rows names its dataset, since it cannot default to the training dataset. Such a model has no training instructions. An endpoint serves it without instructions. To use it as a model: grader, specify graderInstructions.

Rows files

The rows are in one or more Parquet files. Each row has an input_ids and a loss_mask list column of the same length. loss_mask[i] is 1 where token i is trained on and 0 elsewhere. The first token is never marked, because no token predicts it, and every row marks at least one token. Other columns are ignored.

Prepare the rows with the base model’s own tokenizer and chat template, at the commit the run trains.

The descriptor

A prepared version is a ZIP file containing prepared.json alongside the Parquet files it lists:

{"tokenizer":{"repository":"google/gemma-4-E4B-it",
              "revision":"ee0ef6023621cff504d758262d4e04895a5af4a2"},
 "tokenizerSha256":"bca34862f49c6a63155300a313b0d3f901ad1c16d619d19c2677ba963449085d",
 "chatTemplateSha256":"0a2c8073c878ab1da004bee933a998606537bbb62016310352c7285c3f01c5b5",
 "files":[{"path":"train.parquet",
           "sha256":"5f0c9e1d2a7b4c3e8f6a1d0b9c8e7f6a5b4c3d2e1f0a9b8c7d6e5f4a3b2c1d0e",
           "rows":2000,"tokens":3412880,"supervisedTokens":402117,"longestRow":8190}]}

tokenizer names the model repository and the full commit the tokenizer was loaded from.
tokenizerSha256 is the SHA-256 of the tokenizer’s serialized form, tokenizer.backend_tokenizer.to_str() in Hugging Face Transformers.
chatTemplateSha256 is the SHA-256 of the chat template the repository ships at that commit.
files lists every Parquet file in training order, with the SHA-256 of its bytes, its row count, its token count, how many tokens its loss masks mark, and the token count of its longest row.

The describe_prepared.py script reads every row and writes the descriptor. It needs transformers and pyarrow, and it refuses a row that a run would refuse. With --zip, it writes the zip to register. To download it, see the descriptor script.

python describe_prepared.py --model google/gemma-4-E4B-it --zip train.zip train.parquet
akka registry datasets register gemma/train -f train.zip

--revision names a branch, tag or commit of the model. The descriptor records it as a commit.

What is checked

Registration refuses a version whose zip holds a file the descriptor does not list, lacks a file it lists, or holds a file whose SHA-256 differs from the one stated.

Before claiming devices, Optimize refuses a run on a prepared dataset when:

the rows were prepared with a tokenizer other than the base model’s, at another repository or commit
the longest row is longer than the run’s maxSeqLength
the run is a reinforcement run, or asks for a smoke check
one of the training and validation datasets is prepared and the other holds chat examples
the run states a loss group with values other than the defaults, or data.truncation set to right

Before training, Optimize reads every row. It checks each file’s row count and the dataset’s total counts and longest row against the descriptor. It also checks both digests against the base model’s tokenizer. A run on more than one device performs these checks before claiming GPUs. A dry run reports no dataset findings for prepared rows.

Register a dataset

akka registry datasets register triage/train -f train.jsonl
akka registry datasets register triage/train -f train.jsonl --alias candidate
akka registry datasets get triage/train
akka registry datasets versions triage/train
akka registry datasets list --namespace triage

register prints the registered version, such as triage/train:v1. Registering the same examples under the same name again returns the existing version. Changing the examples adds a version. The first registration creates the named dataset and its namespace. --alias points an alias at the registered version. registry datasets alias set moves an alias without another upload.

A dataset can be as large as the deployment’s upload limit, which is 256 MiB unless the deployment sets another. The limit also applies to the total size of the files in a ZIP after extraction. Registration refuses a larger upload and states the limit.

A run descriptor, a pipeline stage, a training project’s evaluation settings, and an evaluation command name a dataset by registry reference:

triage/train resolves through the latest alias, which always points at the newest version.
triage/train:v3 names a version by number.
triage/train:candidate names the version an alias points at.
triage/train@sha256:… names a version by its digest.

A reference without a namespace names a dataset in the training project’s namespace, so in a run descriptor for the triage project, train is triage/train and train:v3 is triage/train:v3. A registry datasets command has no training project to read one from, so name the namespace there. References are case-insensitive: Triage/Train is triage/train.

When you submit the request, Optimize refuses a reference that does not exist in the registry. This happens before it claims any devices.

Images

A user message can carry text and image parts, as in the following example:

{"messages":[
  {"role":"system","content":"Extract the invoice total as JSON."},
  {"role":"user","content":[
    {"type":"text","text":"Read this invoice."},
    {"type":"image","path":"images/invoice-1042.png"}
  ]},
  {"role":"assistant","content":"{\"total\":1290}"}
]}

path is relative to the JSONL file. registry datasets register reads each image and uploads those that the registry does not already have. It replaces each image path with the image’s content hash when registering the examples, so a second version that uses the same images uploads only its text. --no-upload reports missing images without uploading or registering anything. The registry rejects an example with an image that it does not hold. To upload images separately, run akka registry images push PATH….

Training on images uses a vision model’s processor and adapts its language backbone. The base model must therefore accept images. Graders receive text. Only sft trains on images. Optimize refuses a reinforcement run over a dataset with images.