Experiments and runs

Feature set: Evaluations Contact our support for access.
This functionality evolves quickly, the behavior and APIs might change between releases without further notice.

An experiment is a named set of eval cases that runs together against one target. A run is one execution of it. Each eval case becomes one EvalCaseResults in the result, carrying every attempt of that case. The report is rendered from those results rather than stored beside them.

The experiment setup

ExperimentSetup is a record. Build it with its constructor, then two methods return a setup with one thing changed.

var setup = new ExperimentSetup(
        "refund-flow",                          (1)
        cases,                                  (2)
        Lanes.of(4),                            (3)
        Rubric.load("case-judge", 3))           (4)
    .under(Policy.load("refund-desk", 1))       (5)
    .repeating(3);                              (6)
1 The name is the stable key reports and baselines join on. Lowercase, hyphen-separated by convention.
2 A List<EvalCase>. Read them from wherever the project keeps them. EvalSource is the interface for paging a large set, and a project supplies its own implementation.
3 How many cases run at once. See Lanes.
4 The rubric the judge scores against, loaded from eval/rubrics/case-judge-v3.txt on the classpath.
5 under(Policy) records the rules the system was given. Two runs under different rules are not comparable, and a reader cannot see that from the numbers.
6 repeating(N) runs each eval case N times. One attempt cannot tell a case the system meets from one it happened to meet. A case the system handles eight times in ten still passes five attempts about a third of the time.

A setup is checked before it runs. setup.check(target) returns Check.Ready or Check.Refused carrying the reasons. An experiment naming a fixture the target cannot build is refused in seconds rather than yielding one NotReached per case at the end of a long run.

Running an experiment

Local runs are ordinary JUnit tests gated on -Deval=true. mvn verify -Deval=true walks every experiment class under src/test/java/<pkg>/eval/.

var result = ExperimentRunner.run(setup, target, judge);

ExperimentRunner.Result carries the counts in report(), what the lanes sustained in utilisation(), one row per attempt in completed(), those rows grouped by eval case in cases(), and anything the run needs to say in notes().

To render the report, turn the result into the record a report is made from and print it. The run counts the cases, the policy, the lane count and the system’s own spend. Reporting carries what the run cannot know about itself, such as what it is called and how far the judge can be relied on.

var record = result.asRecord(setup, reporting);
Files.writeString(Path.of("target/evalkit/refund-flow.txt"), Panels.render(record));

Where the report is written is the caller’s choice. Nothing writes one on its own.

For dispatching an experiment to a running evaluation service instead of running it in-process, see Remote experiments.

Lanes

A lane is a worker. Lanes.of(4) runs four eval cases at once, over a shared queue rather than a fixed split. Transcripts vary from a couple of messages to eighty, and an unlucky partition leaves one worker running long after the rest are idle.

The configured number is not the interesting one. Akka will run thousands of these without noticing. The model provider will not. result.utilisation() reports what the run actually sustained against what was asked for. That separates "we asked for 64" from "we sustained 9 because we were being throttled".

The ceiling is 512, and it is not an Akka limit. A number that high against a rate-limited provider produces a queue of failing calls rather than throughput.

Fixtures

An eval case names the state it needs through its EvalSetup: None, a Replay of earlier turns, a Fixture the target builds, or a FailingTool. The target declares what it can build in SystemUnderTest.fixtures(), with a one-line description of each. Only the code that builds "authenticated with a claim open" knows what that contains.

An experiment setup naming a fixture the target never declared is refused by setup.check(target) before anything runs.

Recording and re-scoring

FileLedger reads and writes interaction records in a directory. FileLedger.save writes one. RecordedInteractions is a target that reads them back and calls nothing.

An experiment over RecordedInteractions scores runs evalkit did not execute. A dataset scored in CI and traffic scored in production reach the same report through the same metrics. The same recording can be scored again under a new evaluator or a new rubric without paying the target twice.

Nothing is recorded automatically. ExperimentRunner does not touch a ledger. A project that wants a recording writes one.

Failing a build on the result

ExperimentRunner.Result carries no assertions. An experiment is a JUnit test, so it asserts with whatever the project already uses.

assertThat(result.report().passRate()).isGreaterThan(0.9);

ExperimentReport counts passed, review and failed attempts and answers passRate(). It also says when it will not stand behind a figure. provesAnyReachability() is false when every attempt started from a seeded state, and isTrustworthy() is false when too little was measured to read a rate from.

For comparing a run against an earlier one, see Baselines and regression.

Best practices

  • Give each experiment a stable, lowercase, hyphen-separated name. The name is the key reports, baselines, and CI dashboards join on.

  • Use repeating(3) for experiments that use agentic evaluators. Model calls sample. Three repeats stabilise most verdicts and let the variance panel read the flips.

  • State a policy on every experiment with under(…​), so a policy change reads as a policy change in the report and not as a target change.

  • Keep the eval case count per experiment small enough that a local run finishes quickly. Long-running experiments belong on the offline-evals service.

See also