Evaluation
Evaluation runs an agent through a set of eval cases and checks each reply, and the tool calls behind it, with evaluators.
Overview
Evaluation is part of the Akka TestKit: the akka.javasdk.testkit.eval package in the akka-javasdk-testkit artifact.
An evaluation is a JUnit test that extends TestKitSupport.
The agent runs in the in-process runtime with its tools, session memory and guardrails.
The runtime traces every model call and tool call, and the evaluation reads that trace as evidence.
Nothing leaves the test JVM.
The run writes its report to a JSON file under target/eval-reports, see Reading the report.
Use evaluation for three purposes.
-
Measure the quality of an agent against a real model. Run a set of cases as one batch. A gate states what the case results of the batch must satisfy. A real model does not give the same reply twice, so set a gate that requires a minimum pass rate over all cases instead of the default gate, which requires every case to pass.
-
Check that an agent still behaves as it did in production.
EvalCaseParserturns recorded production interactions into cases. Each replayed case expects the agent to call the same tools, in the same order and with the same arguments as in the recording. When the recording carries the model call count, the tokens and the latency, the case also holds the agent to that spend. -
Attack an agent and check that it holds. The user message of the case is an attack, and its evaluators state what a breached reply looks like. See Adversarial tests.
Evaluation is not the place for the following.
-
The effect one command produces. Use Unit testing and the agent’s
TestKit. -
An endpoint’s HTTP contract, or the wiring between components without an agent. Use Integration testing.
How a case runs
ExperimentRunner runs the cases one after the other, in the calling thread.
-
The case’s recorded tool calls, if any, are loaded into the stubs through the runner’s
ToolBindings. -
The runner calls the agent’s command handler through the TestKit component client, in a fresh session, with the case’s command.
-
The runner reads the trace the runtime wrote for that session through
TelemetryReader: the tool calls with their arguments, results and errors, the model calls with their token counts, the guardrail evaluations and the latency. -
Every evaluator of the case reads the reply and the trace and reports a result: pass, fail, or inconclusive when the evidence it needs is absent.
-
The gate checks the aggregated results and the report is returned.
A case whose recorded calls fail to load fails with a result labeled setup.
A case whose agent call throws fails with a result labeled target.
It does not abort the batch.
The parts of an evaluation
| Type | Role |
|---|---|
|
One case: an id, the command sent to the agent, the recorded tool calls, and the evaluators. See Eval cases and evaluators. |
|
The built-in evaluators: tool calls, order and arguments, reply text, and budgets. See Eval cases and evaluators. |
|
A custom check over the reply and the tool calls. See Writing a custom evaluator. |
|
Builds the experiment step by step: the cases with their evaluators and tool bindings, the agent, the gate, then the run. See Writing the first case. |
|
What the batch must satisfy over all case results. Without a gate every case must pass. See Gates and reports. |
|
The outcome: whether the gate passed, the pass rate, every case result, and the report as text. See Gates and reports. |
|
A model that scores the reply against a criterion stated as a sentence. See Model judges. |
|
Cases derived from recorded interactions, and the stubs their recorded results are loaded into. See Replaying recorded interactions. |
All types are in the akka.javasdk.testkit.eval package.
The API documentation is at akka.javasdk.testkit.eval.