TrustBench

A production-readiness harness for AI customer-support agents.

Most teams put a support agent in front of customers on the strength of a good demo and a deflection rate. Neither tells you whether the agent actually resolves problems, follows policy, or quietly got worse on one ticket category after the last prompt change.

TrustBench takes a support agent, runs it against a versioned golden set, scores it on six trust dimensions, catches regressions by ticket category before they ship, and packages the result with a 14-day onboarding playbook and an ROI writeup. It measures resolution, not deflection, and it traces failures back to retrieval, generation, policy, or reasoning so you can tell which layer actually broke. It is built around a fictional consumer neobank, "Northwind".

It is one engineer's answer to a single question: can you prove a support agent is good enough to put in front of real customers, and explain exactly why when it is not?

The loop

1. A support agent ("Sophie") runs on a hosted LLM (Gemini Flash by default, Groq's openai/gpt-oss-20b in the committed run), retrieves from Northwind's knowledge base, and takes real actions through tools: issue refunds, check KYC, freeze cards, pause subscriptions, open disputes, escalate to a human.
2. Every run emits a structured trace: what was retrieved, what the model reasoned, which tools were called with what result, and whether policy was in context.
3. The eval harness scores each response on the six trust metrics. The soft ones use an LLM judge from a different family than the agent (Gemini Pro by default, meta-llama/llama-4-scout-17b in the committed run); the hard ones are deterministic checks.
4. The judge can be calibrated against hand labels: trustbench.evals.calibration computes judge-vs-human agreement and Cohen's kappa. I have not hand-labeled the golden set yet, so there is no kappa in this repo.
5. Two agent versions are compared. A regression that only shows up on one ticket category is surfaced, tested with McNemar on the paired pass/fail outcomes, and traced to its root cause.

The six trust metrics

Trust dimension	How TrustBench measures it
Resolution Accuracy	LLM judge against a reference, plus deterministic tool-coverage
Escalation Intelligence	Deterministic: did it escalate exactly when it should have
Policy and Guardrail Adherence	Deterministic guardrails plus an LLM judge against the policy
Completeness	LLM judge: was every part of the request addressed
Tone and Empathy	LLM judge on a three-point rubric
Groundedness / Hallucination	LLM judge: is every claim supported by context or tools

The table is six dimensions; the harness writes eight metric keys, because Resolution Accuracy and Policy Adherence each get a deterministic check alongside the judged score.

The judge always reasons before scoring and runs at temperature zero. trustbench.evals.calibration computes judge-vs-human agreement and Cohen's kappa against hand labels. That calibration has not been run, so judge quality is not measured here and no kappa is reported.

The regression centerpiece

v2 of the agent is an improvement pass that makes it warmer and pushes it to resolve everything itself. It drops the paragraph of the v1 system prompt that told the agent to follow policy, to claim an action only when the tool result confirms it, and to escalate when policy says to. In the run committed under dashboard/data/, tone, completeness, tool coverage and overall resolution accuracy all rise, while policy adherence falls from 0.85 to 0.70 and groundedness from 0.90 to 0.80, concentrated on the card, fraud, dispute and transfer slices. Refund resolution accuracy is 1.000 in both runs; the only refund-slice movement is groundedness, 1.00 to 0.83. TrustBench:

• shows the per-intent slice where the aggregate hides the problem,
• runs McNemar's test on the paired pass/fail outcomes. In the committed run the strongest signal is policy adherence, regressed on 3 cases and improved on 0, exact p of 0.25. The direction is clean and 20 cases is not enough to call it significant,
• traces the cause back to the dropped prompt paragraph.

The result is a short incident report: symptom, slice, paired test, root cause, fix. docs/regression-incident.md is the one for the committed runs, and it regenerates from the committed JSON with no API key:

python -m trustbench.cli.compare_runs dashboard/data/sample-run.json dashboard/data/sample-run-v2.json

Repo map

src/trustbench/
  scenario/      Northwind: knowledge base, policy, tools, seeded state
  retrieval/     embedder protocol, knowledge index, Gemini embedder
  agent/         the support agent loop, prompts (v1, v2), trace models
  llm/           neutral LLM types, client protocol, Gemini adapter, fakes
  evals/         golden set, metrics, judge, calibration, runner, regression,
                 failure taxonomy, root-cause attribution
  cli/           run_ticket, run_eval, compare_runs
data/golden/     versioned golden set (JSONL)
docs/            run guide, playbook, ROI, outreach, the regression incident report
dashboard/       Next.js results dashboard; dashboard/data/ holds the committed run

Setup

macOS/Linux:

python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"
pytest                              # 82 tests, no network needed

Windows (PowerShell):

python -m venv .venv
.venv\Scripts\Activate.ps1
pip install -e ".[dev]"
pytest                              # 82 tests, no network needed

The regression report regenerates from the committed runs with no key:

python -m trustbench.cli.compare_runs dashboard/data/sample-run.json dashboard/data/sample-run-v2.json

To run the agent and judge yourself, see docs/RUN_LIVE.md. The CLI path needs a Gemini key:

cp .env.example .env                # add GEMINI_API_KEY (Windows: copy .env.example .env)
python -m trustbench.cli.run_ticket "I lost my card, what do I do?"
python -m trustbench.cli.run_eval --run-label v1-baseline --agent-version v1
python -m trustbench.cli.run_eval --run-label v2-candidate --agent-version v2

The committed run came from the Groq path instead, which needs GROQ_API_KEY and rewrites dashboard/data/:

python scripts/run_live_groq.py

Motivation

TrustBench was originally built to apply for the Founding AI Solutions Engineer role at Fini, and the six trust dimensions map to Fini's published "Trust Metrics" framework. That role is not pure eval research; it is forward-deployed solutions engineering: onboard an enterprise customer, tune the agent to a resolution target, catch problems, and show the ROI. TrustBench runs that entire loop end to end on one realistic customer. Fini's CEO has written that "companies think they've added AI to support; what they've really done is automate frustration," and that "the bottleneck in AI support isn't your model, it's your knowledge management." TrustBench takes those claims seriously.

Honest notes

The entire engine is unit-tested with fake LLM clients, so the test suite runs with zero API calls. Nothing is hard-coded: every number on this page and on the dashboard comes from one live run of the 20-case golden set on 2026-06-04, agent openai/gpt-oss-20b judged by meta-llama/llama-4-scout-17b-16e-instruct on Groq, committed under dashboard/data/ with the provider, model ids and date recorded in each file.

Three limits worth stating plainly:

• The judge is not calibrated. trustbench.evals.calibration computes agreement and Cohen's kappa, but the golden set has no hand labels, so no kappa exists. Roughly thirty hand-labeled cases would produce one.
• 20 cases is a small golden set. The clearest regression, policy adherence on 3 cases against 0 improvements, has an exact McNemar p of 0.25. Nothing here reaches p < 0.05, and getting there needs more cases, not a different test.
• The regression direction is deliberate: the v2 prompt drop is seeded. Which metrics and which slices it hits was not, and that part is what the harness measured.