2 · Evals

The eval framework answers one question: how good is this agent? You define a dataset of cases, run them against any kernel agent, and (optionally) have an LLM judge score the outputs against named criteria. The result is a structured EvalReport with pass rates and per-criterion aggregates.

from substrate.fabric.evals import (
    EvalCase, EvalDataset, LLMJudge, EvalRunner, CORRECTNESS, HELPFULNESS,
)

The whole pipeline:

Defining cases — EvalCase / EvalDataset

An EvalCase is one test; an EvalDataset is a named collection.

dataset = EvalDataset(
    name="math-suite",
    cases=[
        EvalCase(input="What is 2+2?", expected_output="4", tags=["math", "easy"]),
        EvalCase(input="Capital of France?", expected_output="Paris", tags=["geo"]),
    ],
)

# or from plain dicts
dataset = EvalDataset.from_list(
    [{"input": "2+2?", "expected_output": "4"}], name="math-suite"
)

easy = dataset.filter_by_tag("easy")   # → a new EvalDataset

Criteria — what "good" means

A criterion is a frozen dataclass holding a judge prompt template, a raw score range (default 1–5), and a pass threshold (normalised 0.0–1.0). Six are built in (fabric/evals/criteria.py):

Write your own by constructing an EvalCriterion:

from substrate.fabric.evals import EvalCriterion

TONE = EvalCriterion(
    name="tone",
    description="Is the reply warm and professional?",
    prompt_template=(
        "Score the ACTUAL OUTPUT on tone.\n"
        "USER INPUT: {input}\nACTUAL OUTPUT: {actual_output}\n{context_section}\n"
        'Respond with ONLY JSON: {{"score": <1-5>, "reasoning": "<why>"}}'
    ),
    threshold=0.7,
)

Templates may use {input}, {expected_output}, {actual_output}, and {context_section} placeholders. Tool-trace criteria also get {expected_tools} (from the case's expected_tool_calls) and {actual_tools} (the tools the agent actually called, in order) — this is what TOOL_USAGE uses to grade tool behaviour rather than reply text.

Scoring — LLMJudge

LLMJudge calls an LLM once per criterion, parses a {"score", "reasoning"} JSON object, and normalises the raw score into 0.0–1.0. It is deliberately robust:

• Markdown-fence tolerant — strips ```json wrappers before parsing, and falls back to a regex that finds the first {... "score": …} object.
• Retries malformed output up to max_retries (default 2); on final failure it returns a score=0.0, passed=False with the error in reasoning rather than throwing.
• Parallel by default — all criteria for a case are judged concurrently (parallel=False to serialise).

judge = LLMJudge(
    model_client=strong_client,        # any kernel LLMClient — use a capable model
    criteria=[CORRECTNESS, HELPFULNESS, SAFETY],
)
scores = await judge.score(
    input_text="What is 2+2?",
    actual_output="4",
    expected_output="4",
)   # → list[EvalScore]

Running — EvalRunner

EvalRunner accepts any kernel agent (a ReActAgent, an OrchestratorAgent, or a flow). For each case it spins up a throwaway in-memory Runtime, submits the input, awaits the reply over the signal bus, then hands the output to the judge.

runner = EvalRunner(
    agent=my_agent,
    judge=judge,            # optional — omit to only capture outputs/latency
    concurrency=4,          # cases run in parallel (default 1 = sequential)
    timeout=60.0,           # per-case seconds; None = no limit
)

report = await runner.run(dataset)
print(report.summary())

A case that errors or times out is recorded with status="error" and is not sent to the judge — it simply counts against the pass rate. run_case() runs a single case with its own runtime if you want to drive one at a time.

Execution-trace capture

After a case replies, the runner reads back that run's event log and fills the trace fields on its EvalCaseResult:

The ordered list of actual tool calls is also handed to the judge so the TOOL_USAGE criterion can grade real tool traces (see below).

Results — EvalReport

EvalReport aggregates every EvalCaseResult and exposes computed metrics:

report = await runner.run(dataset)

print(f"pass rate: {report.pass_rate:.0%}")
for name, stats in report.scores_by_criterion().items():
    print(f"  {name}: mean={stats['mean']:.2f} pass={stats['pass_rate']:.0%}")

for failure in report.filter_failed():
    print(failure.case_id, failure.actual_output[:80])

Scope & caveats

• Trace metrics reflect one run. tokens_used, steps_used, and the tool_calls_* fields are read from the agent-under-test's own event log. For an OrchestratorAgent they cover the orchestrator's run only — a sub-agent's tokens and tool calls live in its child run and are not aggregated in.
• tokens_used is whatever the provider reported. It sums the tokens field journaled on each llm.call; a provider that omits usage contributes 0 for that step.
• TOOL_USAGE needs expected_tool_calls. Set them on the EvalCase, otherwise the criterion has nothing to compare the captured trace against.