Class EvalCaseParser
The input is JSONL, one interaction per line:
{ "id": "c1",
"input": "Why was I charged a late fee on loan_5001?",
"toolCalls": [
{ "name": "getLoan", "arguments": {"loanId": "loan_5001"},
"result": {"status": "overdue", "fee": 12.50} } ],
"output": "You were charged because loan_5001 is overdue...",
"modelCalls": 2,
"tokens": { "input": 640, "output": 85 },
"latencyMs": 1400 }
The case sends the recorded input, carries the recorded tool calls for the runner to load into
the stubs through its ToolBindings, and expects the recorded tools, their order and their
arguments. These evaluators describe what production did. They are a baseline, not a statement of
correctness.
modelCalls, tokens and latencyMs are optional. When present they
become budgets on the case: the model call count as recorded, tokens and latency multiplied by
the tolerance. tokens is either a total or an object with input and
output.
The runner sends input to the agent. Record it as text when the command handler takes
a String. When the handler has its own command type, record it as the JSON of that command and
pass the type to parse(Path, Class).
-
Field Summary
FieldsModifier and TypeFieldDescriptionstatic final doubleMultiplies the recorded token and latency figures to set the budgets. -
Method Summary
Modifier and TypeMethodDescriptionReads the JSONL file into cases that send the recordedinputas text, withDEFAULT_TOLERANCE.Reads the JSONL file into cases that send the recordedinputas text.Reads the JSONL file withDEFAULT_TOLERANCE.Reads the JSONL file.
-
Field Details
-
DEFAULT_TOLERANCE
public static final double DEFAULT_TOLERANCEMultiplies the recorded token and latency figures to set the budgets. A rerun almost never spends the same as the recording, so the budgets allow 50% more.- See Also:
-
-
Method Details
-
parse
Reads the JSONL file into cases that send the recordedinputas text, withDEFAULT_TOLERANCE.- Throws:
IllegalArgumentException- listing every line that does not parse, has noinput, or names a tool call without a name
-
parse
Reads the JSONL file into cases that send the recordedinputas text.- Parameters:
tolerance- multiplies the recorded token and latency figures to set the budgets.1.0holds a case to exactly what was recorded. Values below1.0are rejected
-
parse
Reads the JSONL file withDEFAULT_TOLERANCE.- Parameters:
commandType- the type the agent's command handler takes. The parser deserializes the recordedinputto it
-
parse
public static <C> List<EvalCase<C>> parse(Path canonicalJsonl, Class<C> commandType, double tolerance) Reads the JSONL file.- Parameters:
commandType- the type the agent's command handler takes. The parser deserializes the recordedinputto ittolerance- multiplies the recorded token and latency figures to set the budgets.1.0holds a case to exactly what was recorded. Values below1.0are rejected
-