Graders and rewards

A grader scores a finished model’s answer against an evaluation example. A reward scores every sampled completion at every step of a reinforcement run. Use graders to evaluate models and rewards to guide reinforcement training. A grader runs once per example, while a reward runs once per rollout. A scorer that is affordable as a grader can therefore be too expensive as a reward, so you configure them separately.

Graders

You configure a grader with a specification string. A training project’s evaluation settings specify one, and a run descriptor specifies one for its evaluations. training evaluations start --grader and a pipeline’s evaluate stage specify a grader for an individual evaluation. When no grader is specified, the trainer uses json-label-match.

The following table lists the grader specifications:

Specification Scores by

json-label-match

Per-field exact match of the JSON in the model’s answer against the JSON in the expected answer. Code fences and prose around the JSON are tolerated. This is the default.

model:NAMESPACE/NAME[:VERSION]

A model in the registry that the trainer provisions as a judge for the duration of the evaluation. The model can be one a run trained, or a base instruct model. See The judge’s instructions for the instructions given to the judge.

http:URL

An evaluator that you operate. For each example, it receives an envelope with input, output, and expected. It returns JSON with a score between 0 and 1, or a boolean ok or passed, and can include an explanation. An unreachable evaluator fails the evaluation.

bundle:REFERENCE#MODULE or bundle:REFERENCE#MODULE:FUNCTION

A function in a scoring bundle that the trainer serves for the duration of the evaluation. REFERENCE is the bundle’s registry reference, such as spider/scoring or spider/scoring:v2. FUNCTION defaults to compute_score. See Scoring bundles.

If the grader returns no verdict for an example, the example is unscored. training evaluations predictions marks it, and the mean excludes it.

json-label-match grades a structured answer. To grade free text, use a judge model, an HTTP evaluator, or a bundle function.

The judge’s instructions

A judge receives an envelope with input and output for each example, along with instructions. Optimize uses the first available source of instructions in this order:

  1. graderInstructions, specified alongside the grader. A run descriptor, a pipeline’s train and evaluate stages, a training project’s evaluation settings, an evaluation request, and a branch comparison each take one. training evaluations start --grader-instructions sets it for one evaluation.

  2. For a judge that a run trained, the system message of the first row of its training dataset. The evaluation or run fails when that dataset holds prepared token rows, is missing, is empty, or has no system message in its first row. It does not fall through to the next source, because the judge was trained under its own instructions.

  3. The system message of the first row of the dataset being scored: the evaluation’s dataset, or the dataset a run trains on. The answers were produced using that message. The judge receives the built-in instructions below, followed by the message as context.

  4. Built-in instructions that ask for JSON with a score between 0 and 1 and a one-sentence explanation.

The instructions given to the judge are recorded with the score. training evaluations get and training runs get show where they came from, their SHA-256 and their text. In the API, they are the judgeInstructions field, with source set to SPEC, TRAINING, DATASET or DEFAULT.

A judge given graderInstructions is a different grader from the same model without them. Its key, which training projects scores groups by, ends in +instructions: and the start of the instructions' SHA-256. Two evaluations of one model under two sets of instructions are therefore not compared. A run that states no reward takes the same key into its reward, so two runs judged with different instructions have different configurations.

A run’s judge endpoint records only instructions that come from the judge’s own training dataset. While the run holds its judge, endpoints create for the same base model adds your model to the judge’s endpoint, as it does for any endpoint already serving that base model. Requests to your model use your model’s own instructions. The run specifies its judge’s instructions only for requests to the judge.

Rewards

A reinforcement run’s config.reward.terms is a list of weighted terms. The reward for a completion is the weighted sum of these terms. The trainer does not normalize weights because rescaling them would change the advantage. A run with no declared reward uses its grader’s kind as the only term.

The following example declares a bundle function and a label match as reward terms:

"reward": {
  "terms": [
    {"kind": "rule", "weight": 1.0, "bundle": "spider/scoring", "module": "reward", "function": "compute_score"},
    {"kind": "json-label-match", "weight": 0.2}
  ]
}

The following table lists the reward term kinds:

Kind Scores by

json-label-match

A per-field match against the example’s assistant message, like the grader of the same name.

rule

A function in a scoring bundle that runs inside the training job. bundle is a registry reference, such as gsm8k/scoring, module is the module in import form, and function is the function. The function defaults to compute_score. This kind has the lowest cost per rollout.

model

A registered model that the trainer starts as a judge at an endpoint for the run’s training project. The endpoint remains available for the entire run. The run’s grader specifies the model to deploy. The endpoint reserves the number of devices shown by training endpoints info from the project’s GPU quota, in addition to the devices requested by the run. The project needs quota for both. The term identifies the model by configuration hash. A pipeline that trains its judge in an earlier stage therefore refers to it as model:@judge. A model term always runs at an endpoint, so its placement is fixed.

endpoint

A model served inside the cluster at url under the name model. The job posts each completion to this model.

grader

The run’s grader, whatever kind it is.

The bundle function

A rule term and a bundle: grader call the same Python function:

def compute_score(data_source, solution_str, ground_truth, extra_info=None) -> float:
    # Return a score for solution_str.

solution_str contains the completion, and ground_truth contains the example’s assistant message. When the trainer serves the function as a grader, extra_info["input"] contains the prompt. The function returns the score, to which the trainer applies the term’s weight. If the function raises an exception, the evaluation records an error for that example.

Format before correctness

An untuned model with a strict correctness reward can produce no correct completions. Every completion in a group then receives the same score, which makes the advantage zero and provides no training signal. Add a second term that rewards the answer format, such as a parseable JSON object or a final line in the expected shape. This term provides a gradient before the model produces correct answers. The correctness term dominates after the model starts producing them.