Getting started with evaluation
Why an agent needs evaluation tests
A unit test asserts that a call returns one expected value. An agent’s reply cannot be asserted that way. The model decides which tools to call, with which arguments, and how to formulate a reply, and a real model might not make the same decisions twice. Asserting on the exact reply fails on a harmless rewording.
An evaluation test sends the agent a fixed set of messages, the eval cases, and checks each turn with evaluators: which tools were called, in which order, with which arguments, what the reply must contain, and what a turn may cost. Criteria that have no exact answer, such as tone, go to a model judge that scores the reply. A real model does not give the same reply twice, so an evaluation does not require every case to pass. It counts the results over the whole set of cases and requires a minimum pass rate.
The EvalKit is a set of utilities in the Akka TestKit for testing agents with an ordinary JUnit test.
-
The agent runs in the TestKit runtime with its tools, session memory and guardrails.
-
The evidence is the trace the runtime records for the turn. The test does not instrument the agent.
-
With the model mocked by
TestModelProvider, the same cases are deterministic and run in every build. Without the mock, they run against the real model on demand, with a gate that requires a minimum pass rate. -
Cases can be written in Java or derived from recorded production interactions.
This page shows how to write an evaluation for the order agent of an online shop. It starts with the agent, then adds one eval case, then a batch of cases with a gate. The complete code is in the doc-snippets sample.
The agent under evaluation
The OrderAgent answers questions about orders.
It has two tools: one looks an order up, the other issues a refund.
Both delegate to an OrderService, which the agent receives through its constructor.
@Component(id = "order-agent")
public class OrderAgent extends Agent {
private static final String SYSTEM_MESSAGE =
"""
You are the support assistant of an online shop.
Look an order up with getOrder before saying anything about it.
Issue a refund with issueRefund only when the customer asks for one and the order
has been delivered. Refund the full order total unless the customer names a smaller
amount. Never invent an order, a status or an amount. Answer in one or two sentences.
""";
private final OrderService orders;
public OrderAgent(OrderService orders) { (1)
this.orders = orders;
}
public Effect<String> ask(String question) { (2)
return effects().systemMessage(SYSTEM_MESSAGE).userMessage(question).thenReply();
}
@FunctionTool(
name = "getOrder",
description = "Look up an order by its id, for example o_42."
)
Order getOrder(String orderId) { (3)
return orders.getOrder(orderId);
}
@FunctionTool(
name = "issueRefund",
description = "Refund an amount in cents to the customer of an order."
)
Refund issueRefund(String orderId, int amountCents) {
return orders.issueRefund(orderId, amountCents);
}
}| 1 | The OrderService is injected. Production provides a client for the real service through a DependencyProvider; the evaluation provides a stub. |
| 2 | The command handler the evaluation calls. It takes the case’s command, here a String, and replies with a String. A handler that takes its own type, such as a record. |
| 3 | The tools. Each tool sets an explicit name. Evaluators and recordings refer to tools by that name. |
/**
* The shop's order system as the agent sees it. Production wires a client for the real system,
* an evaluation wires a stub that answers with what a case loaded into it.
*/
public interface OrderService {
/** The order, or a thrown {@link java.util.NoSuchElementException} when the id is unknown. */
Order getOrder(String orderId);
Refund issueRefund(String orderId, int amountCents);
}Under test, a stub takes the place of the real order system.
The sample’s StubOrderService keeps a map of orders that the test fills before the cases run, and throws for an unknown order id the way the real client would.
The test registers the stub through a test specific DependencyProvider, as described in Custom dependency injection in tests.
The test class
The evaluation is a TestKitSupport test.
It registers the stub as the agent’s OrderService and leaves the model as configured in application.conf, so the agent answers with the model it uses in production.
@EnabledIfSystemProperty(named = "eval", matches = "true") (1)
public class OrderAgentQualityIntegrationTest extends TestKitSupport {
private final StubOrderService orders = new StubOrderService(); (2)
@Override
protected TestKit.Settings testKitSettings() {
return TestKit.Settings.DEFAULT.withDependencyProvider(dependencies()); (3)
}
private DependencyProvider dependencies() {
return new DependencyProvider() {
@SuppressWarnings("unchecked")
@Override
public <T> T getDependency(Class<T> type) {
if (type == OrderService.class) return (T) orders;
throw new IllegalArgumentException("no such dependency: " + type);
}
};
}
@BeforeEach
public void theOrders() {
orders.reset(); (4)
orders.addOrder(new Order("o_42", "shipped", 2599, "leave it with a neighbour"));
orders.addOrder(new Order("o_9", "delivered", 4999, ""));
}| 1 | The class runs only when the build sets the eval system property. Without it JUnit skips the class. |
| 2 | The stub the agent’s tools call. |
| 3 | The DependencyProvider that serves the stub. It replaces the one the service’s ServiceSetup creates. |
| 4 | The orders the cases ask about. The stub is refilled before each test, so every test starts from the same orders. |
Run it with the property set and the model’s API key in the environment:
mvn verify -Deval=true@EnabledIfSystemProperty is one way to keep an expensive test out of the regular build.
A JUnit @Tag with a Failsafe groups filter, a Maven profile, or a class name the plugin does not select work as well.
The first case
An EvalCase describes one turn with the agent.
It carries the command the agent receives and the evaluators that check what the agent did with it.
The first case asks where an order is and expects the agent to look the order up without refunding it.
private EvalCase<String> orderStatus() {
return EvalCase.of(
"order-status", (1)
"Where is order o_42?", (2)
Evaluators.shouldCallTool("getOrder"), (3)
Evaluators.shouldCallToolWith("getOrder", "orderId", "o_42"),
Evaluators.shouldNotCallTool("issueRefund"),
Evaluators.replyShouldContain("shipped")
);
}| 1 | The id names the case in the report. |
| 2 | The command the agent receives. |
| 3 | The evaluators. The agent must call getOrder with the order id from the question, must not issue a refund, and the reply must carry the order’s status. |
An experiment is a set of cases bound to an agent and a gate, run once for one report.
@Test
public void looksTheOrderUpBeforeAnswering() {
var report = new ExperimentRunner(testKit)
.cases(orderStatus()) (1)
.agent(OrderAgent::ask) (2)
.run(); (3)
assertThat(report.passed()).withFailMessage(report::render).isTrue(); (4)
}| 1 | The cases. One, several, or a list. |
| 2 | The agent’s command handler. The runner calls it through the component client in a fresh session per case. |
| 3 | Runs the cases and returns an EvalReport. |
| 4 | Without a gate the report passes only when every case passes. The rendered report is the failure message, so a failed case prints its evidence. |
A single case against a real model is not a reliable check. The model may formulate the reply differently on another run, or call a tool it did not call before, and the case fails on that run. The next step runs a set of cases and requires a minimum pass rate over all of them instead. Running every case several times names the cases that do not hold on every run.
A batch behind a gate
Several cases run as one batch.
A Gate decides what the batch must satisfy over all case results.
private List<EvalCase<String>> curated() {
return List.of(orderStatus(), fullRefund(), greeting());
}@Test
public void qualityGate() {
var report = new ExperimentRunner(testKit)
.cases(curated())
.agent(OrderAgent::ask)
.gate(
Gate.passRateShouldBeAtLeast(0.9) (1)
.and(Gate.passRateShouldBeAtLeast(Evaluators.ToolArgument.class, 1.0)) (2)
.and(Gate.targetShouldNotFail()) (3)
)
.run();
assertThat(report.passed()).withFailMessage(report::render).isTrue();
}| 1 | At least 90% of the cases must pass. |
| 2 | Every declared tool argument must arrive as declared, in every case that declares one. A wrong argument is a defect even when the reply reads well. |
| 3 | No case may fail in the agent call. An unavailable model fails the gate rather than lowering the pass rate. |
The gates are combined with and, so all three must hold.
A rendered report of this batch:
3 cases, 1 run: 3/3 attempts passed (100%) gate: passed — pass rate 1.00 over 3 attempts (3 cases, 1 run), required 0.90; tool-arguments rate 1.00 over 2 judged attempts, required 1.00; no target failures tools 1/1 tool-arguments 2/2 forbidden-tools 2/2 reply-contains 2/2 tool-order 1/1 reply-matches 1/1 tool-call-budget 1/1 model-call-budget 1/1 spend: 6 model calls, 123 tokens in, 321 out, 550 ms in total, slowest order-status run 1 at 489 ms, over 3/3 attempts with evidence
The second line is the gate’s verdict, with the detail of each of the three gates. The indented lines are the pass counts per evaluator, over the cases that activated it. The last line sums what the run spent. Reading the report explains every line.
See also
-
Eval cases and evaluators lists every built-in evaluator and when it is inconclusive.
-
Model judges scores criteria that have no exact answer.
-
Replaying recorded interactions turns recorded production traffic into cases.
-
Adversarial tests writes the cases as attacks.