Class EvalCaseParser

Object
akka.javasdk.testkit.eval.EvalCaseParser

public final class EvalCaseParser extends Object
Turns recorded production interactions into cases.

The input is JSONL, one interaction per line:


 { "id": "c1",
   "input": "Why was I charged a late fee on loan_5001?",
   "toolCalls": [
     { "name": "getLoan", "arguments": {"loanId": "loan_5001"},
       "result": {"status": "overdue", "fee": 12.50} } ],
   "output": "You were charged because loan_5001 is overdue...",
   "modelCalls": 2,
   "tokens": { "input": 640, "output": 85 },
   "latencyMs": 1400 }
 

The case sends the recorded input, carries the recorded tool calls for the runner to load into the stubs through its ToolBindings, and expects the recorded tools, their order and their arguments. These evaluators describe what production did. They are a baseline, not a statement of correctness.

modelCalls, tokens and latencyMs are optional. When present they become budgets on the case: the model call count as recorded, tokens and latency multiplied by the tolerance. tokens is either a total or an object with input and output.

The runner sends input to the agent. Record it as text when the command handler takes a String. When the handler has its own command type, record it as the JSON of that command and pass the type to parse(Path, Class).

  • Field Details

    • DEFAULT_TOLERANCE

      public static final double DEFAULT_TOLERANCE
      Multiplies the recorded token and latency figures to set the budgets. A rerun almost never spends the same as the recording, so the budgets allow 50% more.
      See Also:
  • Method Details

    • parse

      public static List<EvalCase<String>> parse(Path canonicalJsonl)
      Reads the JSONL file into cases that send the recorded input as text, with DEFAULT_TOLERANCE.
      Throws:
      IllegalArgumentException - listing every line that does not parse, has no input, or names a tool call without a name
    • parse

      public static List<EvalCase<String>> parse(Path canonicalJsonl, double tolerance)
      Reads the JSONL file into cases that send the recorded input as text.
      Parameters:
      tolerance - multiplies the recorded token and latency figures to set the budgets. 1.0 holds a case to exactly what was recorded. Values below 1.0 are rejected
    • parse

      public static <C> List<EvalCase<C>> parse(Path canonicalJsonl, Class<C> commandType)
      Reads the JSONL file with DEFAULT_TOLERANCE.
      Parameters:
      commandType - the type the agent's command handler takes. The parser deserializes the recorded input to it
    • parse

      public static <C> List<EvalCase<C>> parse(Path canonicalJsonl, Class<C> commandType, double tolerance)
      Reads the JSONL file.
      Parameters:
      commandType - the type the agent's command handler takes. The parser deserializes the recorded input to it
      tolerance - multiplies the recorded token and latency figures to set the budgets. 1.0 holds a case to exactly what was recorded. Values below 1.0 are rejected