field-agent

An enterprise AI support-agent deployment kit — a RAG-grounded Claude agent that resolves customer tickets, escalates the ones it shouldn't touch, and scores itself in CI on every change.

Most "AI support bot" demos are a single messages.create call wrapped in a chat box. The hard part of actually deploying one at a customer isn't the chat — it's proving it's grounded, safe, and won't regress. That's what this kit is about.

Built as a public re-implementation of patterns I shipped on an internal AI support agent at Krafton. No proprietary data or code — the knowledge base here is a fictional SaaS, "Nimbus."

---

What's in the box

Piece	Why it matters for a real deployment
RAG-grounded agent	Claude with a search_knowledge_base tool over a pluggable KB. It retrieves before it answers — no ungrounded claims.
Guardrails + human handoff	Deterministic backstops force escalation on refunds > $500, GDPR/data-deletion, security incidents, and legal — independent of what the model decides.
⭐ Eval harness	~15 support scenarios graded by exact-match (outcome + keywords) and an LLM judge. Prints a regression score; CI fails if it drops below threshold.
Cost + latency tracking	Every conversation reports exact USD cost (from token usage) and wall-clock latency.
Channels	Zero-setup CLI, an HTTP API, and a browser dashboard. The Retriever and channel seams are where real integrations (Gmail/Slack/pgvector) drop in.
FDE field notes	docs/ENGAGEMENT.md — the kit written up as a real customer deployment: discovery → scoping → rollout → measured results.

Architecture

                    ┌──────────────────────────────────────────┐
  customer ──▶ channel (CLI / HTTP / web) ──▶ guardrails ──┐    │
                                                           ▼    │
                                              ┌──────────────────────────┐
                                              │  agent loop (Claude)      │
                                              │  search_knowledge_base ───┼──▶ BM25 retriever ──▶ KB (markdown)
                                              │  escalate_to_human        │
                                              │  resolve_ticket           │
                                              └──────────────────────────┘
                                                           │
                          outcome + reply + citations + cost/latency metrics

Full write-up: docs/ARCHITECTURE.md.

Quick start

npm install
cp .env.example .env      # add your ANTHROPIC_API_KEY

npm run chat              # talk to the agent in your terminal
npm run serve             # dashboard + API at http://localhost:8080
npm run eval              # run the eval suite and print the score

Or with Docker:

ANTHROPIC_API_KEY=sk-ant-... docker compose up   # dashboard at :8080

Try these

• "I subscribed 3 days ago and want a refund" → resolved, cites the 14-day policy
• "I think my account was hacked" → escalated by the security guardrail
• "Refund me the $1,200 I was charged" → escalated (over the $500 agent limit)

The eval harness (the point)

$ npm run eval

Running 15 scenario(s) — agent=claude-opus-4-8, judge=claude-opus-4-8

  PASS  refund-within-window          resolved   $0.0182 5231ms
  PASS  security-breach-escalate      escalated  $0.0004 3ms
  ...
Score: 14/15 (93%)  total cost $0.24  avg latency 6100ms  threshold 80%

Each scenario is scored three ways: the outcome must match (resolved vs escalated), required keywords must appear, and an independent LLM judge grades the answer against a rubric. The run writes eval-report.json / eval-report.md and exits non-zero below threshold, so CI blocks regressions.

npm run eval -- --scenario refund-within-window   # one scenario
npm run eval -- --limit 5                          # first five
AGENT_MODEL=claude-haiku-4-5 npm run eval          # cheaper run

Configuration

Env var	Default	Notes
ANTHROPIC_API_KEY	—	Required.
AGENT_MODEL	claude-opus-4-8	Agent model. claude-haiku-4-5 for cheap runs.
JUDGE_MODEL	claude-opus-4-8	Eval judge model.
PORT	8080	Server port.
KB_DIR	kb	Knowledge-base folder.
EVAL_THRESHOLD	0.8	CI pass-rate gate.

Extending it

• Swap in embeddings retrieval: implement the Retriever interface in src/kb/retriever.ts (BM25 today → Voyage + pgvector for production).
• Add a real channel: implement a connector that turns inbound messages into runAgent(...) calls (Gmail/Slack/WhatsApp/ticketing webhook).
• Grow the eval set: add scenarios to evals/scenarios.json — coverage is the whole game.

Tech

TypeScript · @anthropic-ai/sdk (tool use + adaptive thinking) · Express · zero-dependency BM25 retrieval.

MIT © Lavya Tanotra