akka optimize training datasets analyze

Measure a dataset under a base model’s tokenizer and chat template. The analysis claims no GPU.

DATASET is a registry reference that names its namespace, such as triage/train. --model names the base model by repository. Where it is not given, the base model is the run file’s baseModel, and a --model that names another base model than the run file is refused. The analysis measures the examples of the dataset version the reference resolves to, under the commit the base model pins. A dataset of prepared token rows is refused, since it carries its lengths already. A dataset of preference pairs is measured as a dpo run reads it.

Optimize keeps one analysis for each set of examples, base model commit and Optimize version. Two dataset versions or names that hold the same examples share one analysis. Running the command again for the same three returns that analysis. A failed analysis is started again, except one that refused the dataset, which is returned as it is.

A row’s length is the length a run compares against maxSeqLength. The report gives:

In a dataset of preference pairs, the report says pairs where it says rows. A pair’s length is the longer of the prompt with the chosen answer and the prompt with the rejected answer, which is what a dpo run compares against maxSeqLength. A dpo run removes no turns, so the report shows no trimmed lengths. Its trained tokens are those of both answers, or why a pair cannot be trained, such as a chat template that does not render the answer after its prompt.

Once the analysis is done, the report estimates the steps of a run over every analyzed row, including rows that data.truncation: drop would leave out: a supervised run over chat examples, and a dpo run over preference pairs. It also estimates the time, from the GPU seconds per row of completed supervised runs of the same base model, scaled to the estimated steps, batch and GPUs. Those runs can have trained on other GPU counts and other datasets. The times come from completed supervised runs, so a dpo run’s time is not estimated.

--file reads a run file, as "training runs start -f" does, and estimates a run of its config and execution. Without it, the estimate is for a run with the default settings. Each value the steps are computed from is marked "run file" where the run file supplied the value the run uses, and "default" where the run file left it out, or gave zero or a value the run replaces.

Without --wait, the command starts the analysis or reads it, and prints where it stands. With --wait, it follows the analysis until it is done or failed, then prints the report. --timeout bounds the wait, not the analysis, which carries on. A read that fails with a server error or a dropped connection is tried again a few times. A refusal, such as a missing permission, ends the wait at once. The estimates are asked for the dataset version the first answer named, so a reference that moves during the wait does not change what is reported.

A failed analysis prints its reason and exits with a non-zero status.

-o json prints the response as Optimize sends it.