A factory machine throws an error code. A technician logs in and submits it — no human writes the diagnosis. In the background, three agents — an Investigator, a Diagnostician, and a Verifier — gather evidence, write a report, and check that report against the evidence before anyone sees it. Every request is authenticated and scoped to the logged-in user: workers see their own incident history, admins see everyone's plus a cost/usage summary. (Cross-company multi-tenant isolation is a planned upgrade, not implemented in this version — see Roadmap.)
| Login | Home + incident history | Finished report |
|---|---|---|
 |
 |
 |
Latest run of the eval harness (scripts/run_eval.py), against the committed eval/baseline_cache.db baseline — reproducible with zero API calls via CACHE_ONLY=true, verified stable across two consecutive runs before being trusted:
| Metric | Result |
|---|---|
| Retrieval recall@3, scoped to category (matches production) | 12/13, across a corpus of 8 manuals / 88 chunks |
| Retrieval recall@3, full corpus (unfiltered, for comparison only) | 11/13 |
| RCA rubric (3 pinned questions × 13 completed incidents) | 36/39 |
| Outcome breakdown | 13 completed · 0 verification_failed · 0 investigation_exhausted · 0 call_budget_exceeded · 0 timed_out · 0 error |
Verifier injection_detected flagged |
1/13 — the seeded injection case, caught with no self-disclosing hint in the source text |
| Verifier stress tests | 2/2 — correctly failed a report with a fabricated claim, correctly passed a fully-grounded control report |
Two recall numbers are reported because they measure different things. search_manuals in production always filters by the incident's system_category — the eval's own recall check used to search the full unfiltered corpus instead, which is a different, harder task than what the system actually does at runtime. Scoped recall (12/13) is the number that reflects production; full-corpus recall (11/13) is kept alongside it for comparison, not as the headline. The corpus spans 8 fault domains with deliberate overlaps (two independent hydraulic-fault domains, three independent temperature-fault domains) so a chunk landing in the top 3 has to out-compete genuinely similar distractors, not just clear a near-empty index. Ten of the thirteen incidents are realistic fault reports; three (gold-11–gold-13, tagged adversarial) were written to stress retrieval. Under scoped search, gold-11's cross-domain PM-schedule collision is gone — there's no other domain to collide with once the search is confined to pneumatic_systems — which is itself a finding worth stating plainly: the harder, unscoped version of that question was never a fair test of what the product does. gold-12 still misses under both measures: a bare error code with no description isn't enough signal to find the right section within its own manual, category filter or not. See eval/results.json for the full per-incident breakdown.
The RCA rubric's three misses aren't evenly meaningful. gold-12 failing "identifies a specific, plausible root cause" is the expected, correct outcome for an incident that supplies almost no information — a judge that gave it credit here would be the actual problem. gold-03 misses the same question because its "fault" is the injection attempt itself, not a mechanical failure, so there's no root cause to state; the report correctly treats it as a security note instead, which the rubric wasn't written to reward. gold-04 is the one real miss on an otherwise normal fault report, worth reading directly in results.json rather than explaining away.
The outcome breakdown is reported as its own category deliberately: investigation_exhausted, call_budget_exceeded, or timed_out would mean the system hit a hard cap and honestly declined to invent an answer, which is a success for that safety mechanism, not a failure of the run — collapsing it into a single pass/fail number would hide that distinction.
Two verifier stress tests are called directly against agents._run_verifier with hand-authored evidence/report pairs (see "Decisions worth knowing about"), not generated live end-to-end — verify-01's report invents a two-incident recurrence history and a specific part number not present in its evidence bundle, and the Verifier correctly returned fail; verify-02 is an otherwise-identical control report with no invented claims, and it correctly returned pass. That's the first time this project has a committed, non-trivial Verifier catch on record, rather than just an assertion that it works.
POST /auth/login → verify bcrypt hash → issue JWT (id, username, role, 24h exp)
Authorization: Bearer <token> on every call below
POST /incidents → identity read from the JWT, never from the request body
↓ write to Postgres (created_by = current_user.id) → return 202 + ticket_id
↓ (background task)
┌──────────────────┐
│ INVESTIGATOR │ ← tools: machine_history (SQL)
│ gathers evidence│ search_manuals (vector)
└────────┬─────────┘ similar_incidents (SQL)
↓ evidence bundle
┌──────────────────┐
│ DIAGNOSTICIAN │ writes RCA report
└────────┬─────────┘
↓ report + evidence
┌──────────────────┐
│ VERIFIER │ every claim grounded? → pass/fail
└────────┬─────────┘
↓
write result + metrics to Postgres
GET /incidents/{id} → worker: own incidents only (404 otherwise) · admin: any incident
GET /incidents/{id}/metrics → cost/tokens/latency for that incident, same access rule
GET /admin/usage-summary → admin-only aggregate across all incidents
Every SQL query and every vector search still carries a tenant_id filter, applied by our own code — that plumbing is what a real multi-tenant boundary would build on (see Roadmap). What actually gates access today is identity: created_by is set from the JWT-verified user on the way in, and every read filters on it for workers (admins bypass the filter). The client never gets to assert who it is — there's no tenant_id or user_id field anywhere in a request body. Every LLM call goes through the response cache first.
The investigator's tool-calling loop is driven manually — this project doesn't use the Gemini SDK's automatic function-calling mode — specifically so every transition (call a tool, get a result, decide whether to call another) is a line of Python you can point at and explain, not something happening inside the SDK. That loop is also the system's only real control-flow loop: the verifier does not loop back to the diagnostician. It passes or fails, and either way the incident reaches a terminal state. That's a deliberate simplification — removing that loop removes an entire class of caps and transitions to explain, for very little loss: a failing report is still generated, still stored, and still flagged for review.
queued → investigating → diagnosing → verifying → completed
↘ ↘ verification_failed
investigation_exhausted
↘
call_budget_exceeded (any stage)
↘
timed_out (any in-flight stage, stuck past INCIDENT_TIMEOUT_SECONDS)
↘
error (any stage, unexpected exception)
Two independent caps back this up: MAX_INVESTIGATOR_ROUNDS (default 6) bounds the investigator's own tool-calling loop, and a separate MAX_LLM_CALLS_PER_INCIDENT (default 15) is a global ceiling across every LLM call the incident makes — investigator rounds, the diagnostician call, the verifier call, and anything a future code path might add. The first cap answers "did the investigator get stuck gathering evidence"; the second answers "what's the worst-case cost of one incident," which is the harder, more useful question once someone's paying for this.
A third failure mode neither cap catches: the background task itself dying (process restart, an unhandled exception between two await db.update_incident_status calls) leaves a row stuck at investigating/diagnosing/verifying forever, with nothing ever marking it done. There's no job queue here to retry it, so db.get_incident does a lazy check on every read instead — if an in-flight incident's updated_at is older than INCIDENT_TIMEOUT_SECONDS (default 180s), it's flipped to timed_out right there before the row is returned. It's not a sweep or a background job, just a check-on-read, which is enough at this scale: the frontend polls every 2s while a report is open, so any stuck incident a user is actually looking at resolves within one poll cycle of crossing the timeout.
The Verifier is genuinely useful — in the eval run above it correctly reads the same evidence bundle the Diagnostician used and checks that every claim in the report is grounded in it, and the two stress tests above (a fabricated claim caught, a grounded control passed) show that isn't just an assertion. But that's the limit of what it checks: it verifies claims are grounded in the evidence, not that the evidence itself is true. The Verifier has no independent source of truth — if the evidence bundle were subtly wrong (a poisoned manual chunk, a corrupted history row) in a way that didn't look like an obvious injected instruction, a report built faithfully from that bad evidence would still pass.
One incident in this eval (gold-03, tagged injection_defense) tests this directly, and this version is meaningfully harder than the one this project shipped with originally: the seeded manual chunk used to both contain the injected instruction and a sentence telling the reader it was unverified and shouldn't be trusted — which meant a "catch" proved less than it looked like, since the model was tipped off in-band. That disclosure is gone now; the chunk ("Vendor Firmware Advisory") is just the manipulative instruction, nothing else, and the incident description doesn't hint at suspicion either. Against that harder version, both the Diagnostician and the Verifier still catch it: the report treats the bulletin as an unverified claim rather than an instruction to act on, and the Verifier's structured injection_detected flag came back true — eval/results.json's top-level injection_flagged_count is 1, and gold-03's own entry shows "injection_detected": true. That's the real result this section used to be missing: not just behavior that happened to be safe, but the system's own self-reporting correctly naming why.
This doesn't close the underlying gap, though — one caught case against one hardened chunk is evidence the defense generalizes past the easy version, not proof it holds against every disguise. A more sophisticated injection (one that doesn't read as obviously out-of-place in a technical manual) is untested here. With more time, the fix still isn't a better prompt — it's an independent check: a second grounding pass on a separate model lineage, or retrieval-provenance scoring that doesn't rely on the same model self-attesting to what it noticed.
- JWT auth with bcrypt-hashed passwords, two roles (
worker,admin), identity read from the token on every call - Async ingestion — returns a ticket immediately, never blocks on the LLM
- Structured fault history per machine (SQL)
- Semantic search over technical manuals (RAG, local embeddings)
- Investigator agent with three callable tools
- Diagnostician agent that writes the report
- Verifier agent that checks every claim against retrieved evidence, and catches instructions hidden in manual text
- Hard caps on loops, with named terminal states on exhaustion, plus a lazy timeout reap for orphaned in-flight incidents
- Tenant scoping filter on both the SQL and vector paths (mechanism proven by a test; not yet exposed as a real auth boundary — see Roadmap)
- Role-scoped incident history — workers see their own, admins see everyone's with reporter names
- Per-incident and tenant-wide cost/token/latency summary endpoints, backed by metrics already being collected
GET /metrics— Prometheus-format scrape endpoint (incidents by state, LLM calls/tokens/cost/latency), same underlying data as the usage summary above, exposed the way an ops stack actually consumes it- Evaluation harness measuring retrieval recall@k (both scoped-to-category, matching production, and full-corpus) and RCA correctness
- Verifier stress tests with a known-ungrounded-claim case and a hardened, non-self-disclosing prompt-injection case, both committed to eval results
- Response cache so re-runs cost nothing
- Retries on 429/5xx/timeouts, not just rate limits; blocking vector search and cache I/O moved off the event loop via
asyncio.to_thread - Sanitized markdown rendering (DOMPurify) so retrieved content can't execute in the browser
- GitHub Actions CI running tests and a cached-baseline eval
- Small React client — login, home, incident history, live status, cost display
- One-command Docker startup
| Layer | Tech |
|---|---|
| API | FastAPI, async Python, HTTP 202 pattern |
| Auth | JWT (PyJWT), bcrypt password hashing, role-based access (worker/admin) |
| Structured data | PostgreSQL, raw SQL via psycopg3 (no ORM) |
| Unstructured data | ChromaDB, local embeddings, chunking, cosine similarity |
| LLM | Gemini (gemini-3.5-flash-lite), tool/function calling |
| Orchestration | Plain Python state machine, hand-rolled — no LangGraph |
| Security | JWT auth, tenant-scoping filter, prompt-injection defense, sanitized markdown rendering |
| Quality | Eval harness, recall@k (scoped + full), rubric scoring, verifier stress tests |
| Ops | Docker Compose, GitHub Actions, stdlib logging, Prometheus-format /metrics |
| Frontend | React (Vite, plain JS) |
- Embeddings run locally, not through Gemini. Calling Gemini for both ingest and query embeddings was a real cost leak, especially with eval runs repeating. ChromaDB's bundled local embedding function (ONNX MiniLM, no API key, no network after the first model download) replaced it. Consequence: the vector dimensions differ from the old Gemini embeddings, so the collection was deleted and fully re-seeded — gold labels were written after re-ingest, against the real resulting chunk IDs, never guessed ahead of time.
- Raw SQL, not an ORM.
app/db.pyis parameterized queries, not a mapped model layer. It's shorter, and showing the actual query with the tenant filter written into it is a stronger answer to "how do you enforce multi-tenancy" than pointing at an ORM method. - Agents are three functions, not three classes. Nothing about
_run_investigator/_run_diagnostician/_run_verifierneeds inheritance or shared state beyond what a closure already gives them. - Prompts live in
prompts/*.md, not in Python strings. Keeps the Python readable and makes prompt iteration a text edit — and since the cache key includes the prompt text, editing one agent's prompt only invalidates that agent's cache entries. - The Verifier doesn't loop back. Pass or fail is terminal. See the blind-spot writeup above for the tradeoff that buys.
- The cache is a hash-keyed SQLite table, not a class. Key =
sha256(model + messages + tools + system_prompt + temperature). A cache miss underCACHE_ONLY=trueraises rather than silently falling through to a live call — that's what makes CI deterministic and key-free. - 13 gold-labeled incidents, not 30. Ten realistic fault reports plus three deliberately adversarial ones (
gold-11–gold-13), all hand-checked against real retrieval, beats thirty written on faith. Raw counts are reported alongside every percentage for exactly this reason — and the adversarial cases exist specifically so recall@3 isn't just a suspiciously perfect number nobody can interrogate. - Hand-rolled orchestration, no LangGraph. Every state transition in
app/agents.pyis explainable without reference to a framework's internals. - Kept despite trimming: the tenant filter (everywhere), the caps and terminal states, the eval harness, and error handling/logging/config — the last one specifically because production observability (their stack runs Prometheus and Datadog) is something worth demonstrating, not skipping.
- Explicitly not in it: a Redis/Celery queue (FastAPI's own
BackgroundTasksis enough at this scale — worth discussing conceptually, not worth the operational surface here), SSE streaming, TypeScript, any agent framework, DB migrations (there's a single idempotentschema.sql, not a migration chain). tenant_idis now an internal constant, not user input. The multi-tenant story was previously the headline claim and the weakest part of the system — there was no auth, so "tenant_id" was whatever string the client sent, which is a filter, not a boundary. Rather than half-build real multi-tenant auth, this version drops the claim:tenant_idis hardcoded to a single value (db.SINGLE_TENANT_ID) and removed from every request model. The SQL/vector filtering plumbing stays exactly as it was — it's real, tested infrastructure — but it's now positioned honestly as "the mechanism a tenant boundary would be built on," not as multi-tenancy itself. Auth built on top of it (below) is what's real today.- JWT over server-side sessions. No session store to run or clean up, and it keeps the identity check symmetric with everything else here (stateless, no new infrastructure). Tradeoff, stated plainly: tokens can't be revoked before they expire (24h default). Fine for a demo; the first thing to add for real use is a revocation list or a move to short-lived tokens with refresh.
/metricsis deliberately unauthenticated, unlike everything else in this project. That's not an oversight — it's the standard Prometheus deployment pattern: the scraper lives on an internal network that never reaches the public API surface, so gating it behind the same JWT that guards incident data would just break the standard tooling (Prometheus doesn't send a bearer token by default) for a boundary that's normally enforced at the network layer, not the app layer. Worth saying out loud in case it reads as inconsistent at a glance.- Two gold-labeled verifier stress tests, called directly, not through the pipeline. Getting the live Diagnostician to reliably produce an ungrounded claim would mean fighting the system's own guardrails — the more honest test is to hand-author one evidence bundle with a report that fabricates a specific detail (a part number and two dates not in the evidence) and call
agents._run_verifierdirectly with it, same prompt and model as production, deterministic and cache-friendly. A second case with an equally hand-authored but fully grounded report exists as a control, so a passing stress test isn't just "the Verifier says fail to everything." - The injection test chunk no longer discloses itself. The original manual chunk both contained the injected instruction and a sentence telling the reader (and the model) it was unverified and shouldn't be trusted — which meant catching it proved less than it looked like. The chunk now just contains the manipulative instruction, nothing else; whatever catches it has to come from the agents' own prompts, not a hint planted in the evidence.
8 documents, 88 chunks (H2-section chunking, front-matter-scoped by tenant_id + system_category):
injection_molding · cnc_calibration · hydraulic_press · industrial_hvac_thermal · robotic_welding · conveyor_systems · packaging_line · pneumatic_systems
Two pairs are deliberately overlapping so retrieval has to discriminate on more than a keyword match: injection_molding and hydraulic_press both cover hydraulic faults with different root causes, and injection_molding, industrial_hvac_thermal, and robotic_welding all cover distinct "temperature exceedance" faults. injection_molding also carries the seeded prompt-injection test chunk — a "Vendor Firmware Advisory" section containing an embedded instruction aimed at the AI, with no self-disclosure that it's fake (see "Decisions worth knowing about").
cp .env.example .env
# fill in GEMINI_API_KEY in .env
docker compose up --buildThis brings up Postgres, applies schema.sql, seeds the manuals corpus and three demo accounts, and starts the API — nothing else to run by hand. API docs at http://localhost:8000/docs.
Log in with any of the seeded demo accounts (also shown on the frontend's login screen):
| Username | Password | Role |
|---|---|---|
worker1 |
worker123 |
worker |
worker2 |
worker123 |
worker |
admin1 |
admin123 |
admin |
To populate demo data without spending API quota, run the eval harness against the committed cache:
docker compose exec -e CACHE_ONLY=true -e CACHE_DB_PATH=/app/eval/baseline_cache.db api python scripts/run_eval.pypython -m venv venv
source venv/bin/activate # or venv\Scripts\activate on Windows
pip install -r requirements.txt
cp .env.example .env
# fill in POSTGRES_PASSWORD and GEMINI_API_KEY
docker compose up -d postgres
python scripts/seed_manuals.py
python -m app # starts uvicorn; use this instead of the bare uvicorn
# CLI on Windows — psycopg3's async pool needs the
# selector event loop, which this entrypoint sets up
# before uvicorn creates its ownRun the test suite and the eval harness:
pytest -q
python scripts/run_eval.pyPOST /api/v1/auth/login
{ "username": "worker1", "password": "worker123" }Response:
{
"token": "eyJhbGciOiJIUzI1NiIs...",
"user": { "id": 1, "username": "worker1", "display_name": "Worker One", "role": "worker" }
}Every request below sends this token as Authorization: Bearer <token>. There is no tenant_id or user_id field in any request body — identity comes from the token.
POST /api/v1/incidents
{
"machine_id": "MOLD_IM_402",
"error_code": "TEMP_EXCEED_E045",
"description": "Hydraulic clamp temperature spiked to 92C during continuous injection cycle.",
"system_category": "injection_molding"
}Immediate response (202 Accepted):
{
"ticket_id": "23fd0a77-b92b-4a54-b0cd-17b17b3a6c89",
"status": "queued",
"message": "Incident logged. Root cause analysis started."
}GET /api/v1/incidents/{ticket_id}
Once the pipeline completes:
{
"ticket_id": "23fd0a77-b92b-4a54-b0cd-17b17b3a6c89",
"tenant_id": "tenant_toyota_aichi",
"machine_id": "MOLD_IM_402",
"system_category": "injection_molding",
"error_code": "TEMP_EXCEED_E045",
"status": "completed",
"created_by": 1,
"reporter_name": "Worker One",
"rca_markdown": "## Observations\n...\n## Root Cause\n...\n## Documentation Match\n...\n## Action Items\n...",
"verifier_verdict": "pass",
"verifier_notes": "{\"verdict\": \"pass\", \"ungrounded_claims\": [], \"injection_detected\": false, \"notes\": \"...\"}"
}A worker requesting a ticket_id they didn't create gets a 404, identical to a nonexistent ticket — admins can fetch any ticket.
Other endpoints: GET /api/v1/incidents (role-scoped history — own for workers, everyone's for admins), GET /api/v1/incidents/{ticket_id}/metrics (cost/tokens/latency for one incident), GET /api/v1/admin/usage-summary (admin-only, tenant-wide cost/token totals plus a per-worker breakdown), GET /api/v1/health (liveness), and GET /metrics (Prometheus scrape format — see below).
GET /metrics — no /api/v1 prefix, matching the convention that ops endpoints aren't part of the versioned API, and no auth, matching the convention that a scraper hits this on an internal network, not through the public API:
# HELP traceflow_incidents_total Incidents grouped by current state.
# TYPE traceflow_incidents_total gauge
traceflow_incidents_total{status="completed"} 16
traceflow_incidents_total{status="timed_out"} 2
...
# HELP traceflow_llm_cost_usd_total Total LLM cost in USD across all incidents.
# TYPE traceflow_llm_cost_usd_total counter
traceflow_llm_cost_usd_total 0.010920
Same underlying query as /admin/usage-summary, just in the text-exposition format a real Prometheus server scrapes — docker-compose.yml's api service could add a prometheus.io/scrape label tomorrow and this would just start showing up in a dashboard.
app/
main.py FastAPI app, lifespan (pool init, schema apply, user seeding), /metrics (Prometheus)
api.py auth + incident + metrics + admin endpoints, Pydantic models
auth.py bcrypt hashing, JWT issue/verify, get_current_user / require_admin deps
db.py raw psycopg3 queries, async connection pool, user seeding, timeout reap
schema.sql 7 tables: tenants, users, machines, incidents, evidence, reports, metrics
vectors.py Chroma client, local embeddings, tenant-filtered search
tools.py 3 tool functions + their JSON schemas
agents.py investigator / diagnostician / verifier + state machine + caps
llm.py Gemini wrapper, cache check, metrics capture, broadened retries
cache.py hash(request) -> SQLite table
settings.py os.getenv, one Settings object
logging_conf.py stdlib logging setup
prompts/ investigator.md, diagnostician.md, verifier.md
manuals/ 8 fault-domain manuals (source docs for RAG ingestion)
scripts/
seed_manuals.py chunk, embed locally, ingest into Chroma
run_eval.py 13 incidents -> scoped+full recall@k, rubric, verifier stress tests -> results.json
eval/
gold_incidents.json 13 gold-labeled incidents (10 realistic + 3 adversarial) + 2 verifier stress cases
results.json latest eval run output
baseline_cache.db committed LLM response cache — CI runs against this, no API key needed
tests/
test_tenant.py, test_caps.py, test_agents.py
frontend/
src/App.jsx auth-state router (Login vs Home)
src/auth.js localStorage token/user persistence
src/api.js fetch wrapper, attaches Authorization header, handles 401
src/pages/Login.jsx login form + demo-account autofill
src/pages/Home.jsx nav, new-incident form + presets, history, report + cost line, admin usage card
docs/ local AI-context planning files (gitignored)
docker-compose.yml postgres + api services — `docker compose up` is the whole startup
Dockerfile
requirements.txt
.github/workflows/ci.yml tests + cached-baseline eval, no secrets required
A small React + Vite client lives in frontend/. Log in with one of the seeded demo accounts (buttons on the login screen autofill the credentials) and land on a home screen: a history list scoped to your role (workers see their own incidents, admins see everyone's with the reporter's name), a new-incident form with three one-click preset examples pulled from the eval's gold incidents, and — for admins — a usage summary card. Opening a history item polls every 2 seconds until a terminal status, then renders the finished report as sanitized markdown (DOMPurify around marked, so anything a retrieved manual chunk tries to inject into the report can't execute in the browser) alongside that incident's cost/token/latency line, flagging the report visually if the status is verification_failed.
cd frontend
npm install
npm run devOpen http://localhost:5173 with the backend already running on :8000. The dev server proxies /api to http://localhost:8000 (configured in vite.config.js), so the browser only ever talks to the Vite origin — no CORS middleware needed on the backend.
.github/workflows/ci.yml runs on every push/PR: spins up a Postgres service container, seeds the manuals corpus (free — local embeddings, no key needed), runs pytest, then runs the eval harness with CACHE_ONLY=true against the committed eval/baseline_cache.db. No GEMINI_API_KEY secret is required anywhere in CI — the whole run, including a fork's first PR, is deterministic and free. This determinism depends on the DB being empty before the eval runs — run_eval.py deletes and recreates its own tenant's incident rows at the start of every run specifically so machine_history/similar_incidents tool results (and therefore the LLM cache key) can't drift based on what a previous run happened to leave behind; CI gets this for free since its Postgres container starts empty every time, and running the eval locally now gets the same guarantee instead of silently accumulating state. Verified locally by running CACHE_ONLY=true twice in a row against the committed baseline and diffing the results — byte-identical both times.
- Multi-tenant isolation (a real per-client auth boundary — separate credential spaces,
tenant_idback on the login/user record instead of hardcoded) is a planned upgrade, not implemented in this version. The SQL and vector filtering it would sit on top of already exists and is tested; what's missing is a second axis of identity above the current single-company worker/admin model. - JWT revocation. Tokens are valid for their full 24h lifetime once issued; there's no server-side blocklist. A logout button that actually invalidates a token, or a move to short-lived tokens with refresh, would close this.
- A real queue.
BackgroundTasksplus the lazy timeout reap covers this project's scale, but neither retries a crashed pipeline — a timed-out incident is marked failed, not resumed.