Evaluation

Evaluation measures agent behavior against defined quality criteria. It checks what the agent said and did, including its tool calls, rather than relying on one exact reply.

Where to evaluate

Use tests before release, replay to check a change against recorded interactions, batch experiments to assess recorded traffic, and live evaluators to monitor production. Adversarial cases exercise inputs intended to violate constraints.

An evaluator checks a property. A model judge assesses a criterion that cannot be checked exactly. A gate turns several results into a build or release decision. Because model judgments vary, assess representative cases and repeated attempts rather than treating one reply as a stable measurement.

A live evaluator runs after an interaction and cannot undo it. Use an inline guardrail when a check must determine whether an action or reply proceeds. Sampling reduces evaluation cost but leaves some interactions without a verdict.

Reading results

The APIs use different result types:

  • TestKit evaluators report pass, fail, or inconclusive. Setup and target failures identify attempts that could not run normally.

  • Guardrails report allow, deny, or failure of the check. Their decisions control execution.

  • Runtime evaluators record verdicts on completed interactions.

  • Specification checks report exit-condition states and a conformance verdict.

  • Training evaluations score a model against a dataset with a grader, rather than evaluating a complete agent process.

Read each result in its own context. Missing evidence is not a successful check, and an advisory result is not an enforced release decision. See Evaluation terminology for the terms used across sections.

Next steps