Drifty — Prompt Refinement & Drift Monitoring

You know what you want from a model but not how to phrase it. Drifty fixes that, then keeps it fixed.

You type a goal. Three models each write a prompt for it, each in a different style. The three then read each other's (shuffled, anonymized) attempts and rank them — forced ranking, not scores, because weak free-tier models compress scores into meaningless 7s and 8s but ranking is a task they're actually good at. The top two candidates by aggregated rank get executed for real, against your own example inputs, so you're picking based on output you've seen, not a prompt that just reads well.

Then the second half starts. You register the winning prompt with a few machine-checkable rules about what a correct output looks like. Every day, in the background, Drifty re-runs it and checks those rules. If a registered prompt's last DRIFT_WINDOW_RUNS runs pass fewer than DRIFT_THRESHOLD of the time, the dashboard flags it and — if you've set DRIFT_WEBHOOK_URL — a webhook fires the moment it happens, instead of you finding out three weeks later because something downstream looked wrong. This is a rolling pass-rate threshold check, not statistical anomaly detection — see Decisions worth knowing about for why that's the right amount of mechanism for what this actually needs to catch.

Screenshots

Refine — a finished round's top-ranked candidate: the highlighted "Top pick" card, the real generated prompt, real validation output against the example input, unanimous judge agreement, and the check-type register form:

Dashboard — six registered prompts in different real drift states side by side: no runs yet, healthy, and one genuinely drifting at 0% pass:

Results

Real numbers, pulled straight from this repo's own local database after actually using it — not projected, not synthetic. As of 2026-07-30:

• 15 refine rounds completed, 0 failed outright — 3 models generating, judging, and validating each other, end to end, every single time
• 42 individual judge votes cast across those rounds — 86% agreed with the eventual winner, and 9 of 15 rounds (60%) were fully unanimous
• 6 prompts registered and monitored, 28 monitor runs recorded automatically through the real queue, no shortcuts
• 1 of 4 actively-monitored prompts is drifting right now (0% pass over its last 5 runs) — caught by the exact same threshold check that would fire in production, not staged for this section
• Total spend so far: $0.0008 — Groq's calls are genuinely free-tier $0, Gemini's are priced and tracked per call, not estimated

This is a local dev database from building and testing the project, not a hosted service with outside users — the honest way to read these numbers is "the pipeline has actually run this many times and this is what happened," not "N people have used this."

How it works

REFINE (someone is waiting)
  POST /refine → round_id → enqueue 3 generation jobs → queue:refine, status=generating
       ↓ N workers race the queue, whichever picks up each job runs it
       ↓ Redis INCR on one shared `finished` counter tracks "2 of 3 done" — atomic, no lock needed
       ↓ partial-failure decision: proceed with ≥2 candidates, or fail the round
       ↓ any unexpected exception (not just the expected partial-failure case) fails the round explicitly
       ↓ a periodic sweep fails any round stuck in a non-terminal status for >10 minutes — nothing hangs invisibly
  3 candidates → cross-critique (shuffled order, forced ranking, 3 judges)
       ↓ a judge's reply must parse as a clean, complete permutation of the labels or it doesn't count — no guessing
       ↓ positional averaging across judges that returned a usable ranking
  top 2 by aggregate rank executed on your example inputs → results shown

REGISTER
  POST /prompts → save prompt text + strategy + model + checks → Postgres
       ↓ capped at MAX_REGISTERED_PROMPTS active prompts (409 past the cap)
       ↓ POST /prompts/{id}/deactivate frees a slot — the cap's actual recovery path, not just a mention of one

MONITOR (nobody is waiting)
  GH Actions cron → scripts/scheduler.py → query due prompts → enqueue → exit
       ↓ same worker pool, same queue, queue:monitor lane (lower priority)
  run prompt → apply its checks → write run + check_results
       ↓
  drift.check_drift() → pass rate over last N runs < threshold? → if it just flipped to true, POST to DRIFT_WEBHOOK_URL

Two Redis lists are the entire queue: queue:refine and queue:monitor. BLPOP checks the keys it's given in the order given and pops from whichever is non-empty first — that's the whole priority mechanism, no broker, no scheduler process of its own. A pool of worker coroutines (asyncio.gather over N worker_loops in worker.py) all block on the same BLPOP, so the 3 generation jobs for one refine round genuinely run concurrently if the pool has capacity, and refine jobs never wait behind a backlog of monitor jobs.

Fan-in — waiting for all 3 generation jobs, or deciding to proceed without the third — is a single Redis INCR per completed job, success or failure, on one shared finished counter (queue.py). INCR is atomic, so when two workers finish within the same millisecond, exactly one of their INCR calls will observe the counter reach its expected value. That worker, and only that worker, moves the round on to critique. No lock, no leader election, no polling — the atomicity of a single INCR is the whole mechanism. (An earlier version of this gated the decision on done + failed == expected, reading two separate counters non-atomically — two concurrent INCRs on different counters could each observe the sum reach expected and both close the barrier. Collapsing to one counter closes that.)

Decisions worth knowing about

• Hand-rolled Redis queue, not Celery or RQ. Celery is enormous and eats a day on configuration for something this small. RQ is fine but hides exactly the mechanism worth demonstrating. BLPOP on two Redis lists is ~100 lines and it's the single most interview-worthy code in this project.
• The fan-in barrier is gated on one counter, not two. mark_done/mark_failed both INCR a shared :finished key; the decision to move a round to critique compares finished == expected, never done + failed. :done and :failed still exist, but only to report how many succeeded once the barrier is already known to be closed. Reading two separate counters to make the closing decision has a real race — two INCRs on different counters can each observe the sum reach expected in the same instant. One counter, one INCR, no gap.
• The critique parser fails closed, not open. A judge's reply must — after trimming whitespace and one optional pair of wrapping quotes/trailing period — be nothing but a comma-separated permutation of every label exactly once, or it's rejected outright (app/refine.py::_parse_ranking). An earlier version scanned the whole reply for any A–Z letters, which meant a judge that answered "Based on my analysis: C,B,A" or refused outright ("I cannot rank these prompts") still produced a full, fabricated ranking, silently, with no way to detect it. MIN_JUDGES only means something once judges that didn't answer cleanly are actually excluded instead of padded.
• Few-shot candidates can't leak their own example into validation. The few-shot generation prompt used to let the model end its own output with a dangling, unanswered Input: ...\nOutput: pair — meant as a template slot, but at validation time that sat in the same system prompt as the real input, and models would sometimes complete the candidate's own invented example instead of the real one. Two fixes, not one: the generation template now explicitly forbids a trailing unanswered example, and validation prefixes the real input for few-shot candidates specifically, telling the model any Input/Output pairs above are reference-only.
• Nothing in the worker can fail silently into a hung round. Every path through _handle_generate and _finish_generation_stage — not just the expected PartialFailure case — ends in either finishing the round or calling fail_round with a reason. A periodic sweep (_sweep_stuck_rounds_loop, every 60s) additionally fails any round that's been in a non-terminal status for more than 10 minutes, which is the only thing that can catch a worker process dying mid-job — there's no exception to catch in that case, just a job that silently never finishes.
• Every LLM call has a timeout and a couple of retries. models._call_with_retries wraps every Groq/Gemini call in asyncio.wait_for plus up to 2 retries with backoff. Without this, one slow or flaky call could stall a whole worker — N_WORKERS is small by design (free-tier quota, not scale).
• Cost is real, not a placeholder. Groq's free tier is genuinely $0 for the models this project uses — that's not a stand-in, it's the actual price. Gemini 3.5 Flash-Lite is priced from its published per-token rate. Cache hits are labeled cached: true in the response so a repeated identical call's carried-through latency is never displayed as if it were freshly measured.
• Forced ranking, not scores out of 10. Weak free-tier models compress absolute scores into meaningless 7s and 8s. Ranking is an easier task for them and produces output you can actually aggregate.
• Randomised presentation order in critique. Position bias in LLM judges (favoring whichever candidate is shown first) is real and documented; randomising per-judge costs nothing and removes it as a confound. (Self-preference bias — a judge favoring output from its own model family — is a separate, real concern this doesn't address; see "Explicitly not in it" below.)
• Multi-model, not multi-provider. Only Groq and Gemini have free tiers generous enough to build against without a card on file. Groq alone serves multiple model families — that's genuine architectural diversity from one provider. Two of the three generation strategies run on different Groq-hosted models, the third on Gemini; the three critique judges reuse those same three models rather than adding a fourth.
• Generation strategies are fixed and different by design (prompts/gen_terse.md, gen_schema.md, gen_fewshot.md). If all three models got the same instruction they'd produce near-identical prompts, and the ranking step would be pure noise. Designed-in diversity is what makes cross-critique mean anything.
• The top 2 candidates get actually executed, not just ranked. Judging a prompt by how it reads is one step removed from what matters; running it against your own example inputs closes that gap.
• Four checks as functions in a dict, not a class hierarchy (checks.py). Adding a fifth check is one function and one dict entry. A class hierarchy would buy nothing here — none of the four share state beyond "take output + config, return pass/fail". All four are reachable from the Refine screen's register form, not just unit-tested.
• Drift is a threshold check on a rolling pass rate — not statistical anomaly detection, on purpose. drift.py is one SQL query (a pass/fail CTE, batched across every prompt in one round trip instead of one query per prompt) plus one threshold comparison, computed on read rather than written to a stored alerts table. It doesn't look at output distributions, embeddings, or trends beyond that single number, and doesn't claim to — that's the honest scope of what "the last N runs passed less than X% of the time" can tell you, and it's enough to catch the thing this project is actually about: a prompt that used to work and now doesn't.
• A webhook, not just a badge, on the healthy-to-drifting transition. app/notify.py fires a POST to DRIFT_WEBHOOK_URL the instant a monitor run causes a prompt's drift state to flip from false to true (tracked via prompts.last_drift_state, compared against the freshly computed result — not re-fired on every run a prompt happens to still be drifting). A dashboard nobody is watching isn't really monitoring; this is what makes "someone finds out" real instead of aspirational. A failed webhook delivery is logged and dropped, not retried — the dashboard badge is the source of truth regardless.
• Critique and validation are not queued a second time. Generation is fanned out across the worker pool via Redis because it's the naturally parallel, arbitrarily-sized part. Critique (3 judge calls) and validation (2 candidates × up to 3 examples) are a small, bounded number of concurrent calls made inline, by whichever worker's INCR closed the fan-in barrier, via plain asyncio.gather. Round-tripping those through Redis too would add a queue hop for no benefit at this scale — worth naming as a simplification, not an oversight.
• Raw SQL, not an ORM (db.py). Six tables, parameterized queries, no migration framework — schema.sql uses CREATE TABLE IF NOT EXISTS plus ALTER TABLE ... ADD COLUMN IF NOT EXISTS for the columns added after the first pass, applied idempotently on every pool open. That's a real limitation, not hidden: there's no down-migration and no framework, so a destructive schema change (dropping/renaming a column) would need a manual, hand-written statement, same as any single schema.sql setup without a migration tool.
• scripts/seed_demo.py drives the real system, not a shortcut around it. It makes real HTTP calls to a running API for refine and register (exercising FastAPI, auth, and the real queue/worker path), and pushes monitor-run jobs onto the same queue:monitor lane scheduler.py uses. The "goes from healthy to drifting" demo prompt swaps which model executes an unchanged prompt text partway through and records whatever that model actually outputs — a live simulation of a silent provider-side model swap, not a check engineered to always fail. That means the outcome is real, not scripted: in practice a capable model executing a schema-authored prompt sometimes still gets the contract mostly right after a swap, and the pass rate might land above the drift threshold on any given run rather than a guaranteed red badge. That's the honest behavior of a threshold-based system under a moderate-severity regression, not a bug in the demo.
• JavaScript first, TypeScript as a deliberately separate second pass. The frontend was built in plain JS, verified working end to end against the real API, then converted to TS once every line's behavior was already known — git log shows the JS commit and the TS conversion as two separate commits, not one. The types in api.ts are hand-written from the actual FastAPI response shapes, not inferred or any-escaped — npx tsc -b and npm run build both run clean with strict: true.
• No PATCH endpoint for registered prompts. Changing a registered prompt's text means refining again and registering the new winner. Deactivating one, however, is a real endpoint (POST /prompts/{id}/deactivate) — it exists specifically because the registry cap's own error message points to it as the recovery path, and there was no way to act on that pointer without it.
• Explicitly not in it: self-preference bias in LLM judges (the three judges are the same three models that wrote the candidates — real, undemonstrated, and harder to address than the position-bias mitigation that is in here), a revision/feedback loop back into generation, a fifth generation strategy, statistical drift detection beyond a threshold, multi-tenancy or per-user accounts, and any agent framework or orchestration library — every state transition in worker.py and refine.py is plain Python you can point at.

Guardrails

• MAX_REGISTERED_PROMPTS (default 5) is enforced in POST /prompts — a 409 once you're at the cap on active prompts. POST /prompts/{id}/deactivate frees a slot; this is a real endpoint, not just a mention in the error message. At the default MONITOR_INTERVAL_HOURS of 24, five registered prompts is at most 5 LLM calls/day from monitoring alone, comfortably inside Groq's and Gemini's free tiers.
• A single shared-secret API key on write endpoints. POST /refine and POST /prompts require an X-API-Key header matching API_KEY when it's set (app/auth.py); GET endpoints stay open. This is a single-client tool, not a multi-tenant product, so one key is the whole auth story by design — it stops drive-by abuse of endpoints that spend real Groq/Gemini quota, but it is not a defense against a determined attacker who can read the key out of the frontend's bundled JS. A real login system is out of scope at this size; see Security.
• CORS restricted to configured origins, not allow_origins=["*"]. Set ALLOWED_ORIGINS to your deployed frontend's origin(s); it defaults to the Vite dev server for local work.
• The drift demo simulates a real model swap, not a permanently-unsatisfiable check. scripts/seed_demo.py registers a prompt, lets it pass for real for a few monitor cycles, then swaps which model executes the same prompt text and keeps monitoring — standing in for what a silent provider-side model change looks like from the outside. See the note on this in Decisions above for why the outcome is real rather than guaranteed.

Security

This is built and scoped for a single trusted client, not a public multi-tenant product — the auth model matches that:

• Write endpoints require a shared API key; reads don't. Good enough to stop random internet traffic from spending your LLM quota; not good enough to stop someone who can read the key out of the deployed frontend's JS bundle (any client-side secret is inherently visible to whoever loads the page). If you need real per-user auth, that's a different, larger project than this one.
• No rate limiting beyond MAX_REGISTERED_PROMPTS. A malicious or buggy client with the API key could still spam POST /refine. Not addressed here; a reverse-proxy rate limit (Cloudflare, or your host's built-in one) is the practical fix if this goes further than a demo.
• CORS is restricted, not open, and should be set to your actual deployed frontend origin(s) — see ALLOWED_ORIGINS.
• No secrets are logged. logging_conf.py quiets httpx/httpcore/groq/google_genai to WARNING specifically so request/response bodies (which can include your API keys in headers) don't end up in application logs at INFO.
• A malformed regex check config can't crash monitoring, but a pathological one could still block a worker. checks.py::run_checks catches per-check exceptions (an invalid pattern, a missing config key) and fails that check with a note instead of raising — see Decisions. It does not guard against a catastrophically slow regex (ReDoS): registering one would run synchronously inside a worker's event loop and could stall that worker's other jobs. Registering a check requires the API key already, so this is self-inflicted by whoever holds it, not an open attack surface — not hardened further at this size.
• The frontend's dev-server dependency (esbuild, via Vite) has a known moderate CVE (GHSA-67mh-4wv8-2f99) with no fix available except a major Vite version bump (5→8). It only affects npm run dev — a malicious website could read responses from the dev server if you had it running and visited that site at the same time — and doesn't affect the production build. Left unpatched rather than risking a breaking major upgrade for a dev-only, low-real-world-risk issue; run npm audit yourself before relying on this if that tradeoff matters for your use.

Why this isn't deployed

Not everything that could be online should be. The engineering signal here — the queue, the fan-in logic, the fixes documented above — is fully demonstrable via docker compose up and a walkthrough; a public URL doesn't add to that, and a free-tier deployment (cold starts, spin-down, provider quotas) can go wrong in front of an interviewer in ways a local run under your own control never will. ALLOWED_ORIGINS, API_KEY, and DRIFT_WEBHOOK_URL are real, tested, and wired up regardless — they're just not pointed at anything public.

Complete feature list

• Goal intake with optional category and example inputs
• Three-strategy generation: terse directive, explicit schema, few-shot (with a no-dangling-example rule so the model can't hand back a template that later gets confused for the real input — see Decisions)
• Plain-English one-line description of each candidate, parsed from the same generation call
• Cross-critique with forced ranking, randomised presentation order, and a strict parser that rejects anything that isn't a clean, complete ranking rather than guessing from stray letters
• Rank aggregation by positional averaging across judges that returned a usable ranking
• Validation — top 2 candidates actually executed on your example inputs
• Live per-candidate generation progress and unranked-candidate preview while a round is still in flight — not a client-side timer, read from the same Redis state the fan-in barrier itself uses
• Registry: save prompt + task + checks, capped at MAX_REGISTERED_PROMPTS, with a working deactivate endpoint as the cap's actual recovery path
• Four check types, all reachable from the UI: contains, regex, json_schema, max_length
• Two-lane hand-rolled Redis job queue (queue:refine / queue:monitor)
• Worker pool consuming both lanes, fan-out via concurrent workers, fan-in via a single atomic Redis counter
• Exception handling around the full worker task lifecycle, plus a periodic sweep for rounds stuck in a non-terminal status — nothing hangs invisibly
• Per-call timeout and retry-with-backoff on every LLM call
• Scheduled daily monitoring via GitHub Actions cron
• Drift detection: pass rate over a rolling window vs. a threshold, computed on read — an honest threshold check, not anomaly detection
• A webhook fired the moment a registered prompt's drift state flips from healthy to drifting
• Per-run metrics: pass/fail per check, latency, tokens in/out, real cost (Groq's free tier is genuinely $0; Gemini priced from its published per-token rate) — with cached responses labeled as such so a cache hit's latency is never shown as a fresh measurement
• Single shared-secret API key on write endpoints; CORS restricted to configured origins
• Batched queries for prompt list drift and run-history check results — no N+1
• React (Vite, TypeScript) client: a landing screen with the pitch and a single CTA, a Refine screen, and a monitoring Dashboard with a pass/fail trend per prompt
• One-command Docker Compose startup; also runs as a single container via RUN_WORKER_INLINE (see app/main.py)

Tech stack

Layer	Tech
Frontend	React, Vite, TypeScript
API	FastAPI, async
Auth	Single shared-secret API key on write endpoints (see Security)
Queue	Redis lists, BLPOP, two priority lanes — hand-rolled
Concurrency	asyncio.gather, a single atomic Redis counter (INCR) for fan-in, partial-failure handling
Scheduling	GitHub Actions cron → scripts/scheduler.py
Storage	PostgreSQL, raw SQL via psycopg3 (no ORM)
Cache	Redis SETEX, hash-keyed on (model, system, user)
LLM	Groq (2 model families) + Gemini (1 model) — multi-model, not multi-provider
Deploy	Docker Compose (local)

Setup

Docker (one command)

cp .env.example .env
# fill in GROQ_API_KEY and GEMINI_API_KEY in .env — API_KEY can stay blank for local dev
docker compose up --build

This brings up Postgres, Redis, the API, and a separate worker container consuming both queue lanes. API docs at http://localhost:8000/docs.

To see something on first launch instead of an empty dashboard, seed a couple of real refine rounds + registered prompts + monitor history through the real API and queue (requires the stack above already running — this is an HTTP client of it, not a shortcut around it):

python scripts/seed_demo.py

Local development

python -m venv venv
source venv/bin/activate   # or venv\Scripts\activate on Windows
pip install -r requirements.txt

cp .env.example .env
# fill in POSTGRES_PASSWORD, GROQ_API_KEY, GEMINI_API_KEY

docker compose up -d postgres redis
python -m app          # API on :8000
python -m app.worker   # in a second terminal — the worker pool

Run the test suite (pure unit tests run with no services; the queue, drift, and db tests need Postgres/Redis reachable on localhost and skip themselves otherwise — use a dedicated database, not one you care about, since these tests write real rows):

pytest -q

Example: refine a goal

POST /api/v1/refine

{
  "goal": "Summarize a support ticket into a one-line Jira title",
  "category": "text-transformation",
  "examples": [
    "Customer says the export button on the billing page does nothing when clicked, happens on Chrome and Firefox, started after yesterday's release."
  ]
}

Immediate response: {"round_id": 7, "task_id": 4, "status": "generating"} — poll GET /api/v1/refine/7 until status is done (or failed, with an error field explaining which fan-in point didn't get enough successes). While generation is in flight, the response includes a generation_progress field (per-candidate generating/done/failed, reconstructed from the same Redis state the fan-in barrier itself uses); once generation finishes but critique/validation are still running, candidates is already populated (unranked) instead of staying empty for the several real seconds those stages take. A finished round returns candidates (all that succeeded), rankings (each judge's ranking, mapped back to original indices — only judges whose reply parsed as a clean, complete ranking are included), aggregate_rank (best to worst), and validation_results (real output from the top 2, against your example inputs).

Example: register the winner

POST /api/v1/prompts (requires X-API-Key if API_KEY is set)

{
  "round_id": 7,
  "candidate_index": 2,
  "checks": [
    { "type": "max_length", "config": { "max_chars": 80 } },
    { "type": "regex", "config": { "pattern": "^[A-Z]" } }
  ]
}

GET /api/v1/prompts then lists it with a live drift field ({"pass_rate": 1.0, "window": 3, "drifting": false}), recomputed on every request. POST /api/v1/prompts/{id}/deactivate frees its slot against MAX_REGISTERED_PROMPTS without deleting its history.

Project structure

app/
  main.py             FastAPI app, lifespan (pool init, CORS, API-key warning, optional inline worker)
  api.py               6 endpoints + Pydantic request models + generation-progress reconstruction
  auth.py                shared-secret API key dependency
  db.py                 raw psycopg3 queries, async connection pool, batched drift/check-results queries
  schema.sql              6 tables: tasks, rounds, prompts, checks, runs, check_results
  queue.py                 Redis lists, BLPOP, two lanes, single-counter atomic fan-in — hand-rolled
  worker.py                 consume loop, fan-in via one Redis INCR, full-lifecycle exception handling, stuck-round sweep
  models.py                  3 generation model clients (Groq x2, Gemini x1) + critique judges + retry/timeout wrapper
  refine.py                   generation → critique (strict parser) → aggregate → validation (few-shot-safe)
  checks.py                    4 check functions in a dict
  drift.py                      one batched SQL query + threshold, computed on read
  notify.py                      webhook fired on a healthy → drifting transition
  cache.py                       Redis SETEX, hash(model+system+user) -> cached LLM response, cached-flag passthrough
  settings.py                     os.getenv, one Settings object
  logging_conf.py                  stdlib logging setup, quiets HTTP client libraries to avoid logging secrets
prompts/                gen_terse.md, gen_schema.md, gen_fewshot.md (no-dangling-example rule), critique.md
scripts/
  scheduler.py           query due prompts -> enqueue onto queue:monitor -> exit (GH Actions cron entrypoint)
  seed_demo.py            drives the real API + queue + worker for first-launch demo data — not a shortcut around them
docs/screenshots/       refine.png, dashboard.png — referenced above
tests/
  test_checks.py, test_refine.py, test_models.py, test_worker.py, test_notify.py, test_auth.py   pure unit tests, no services needed
  test_queue.py, test_drift.py, test_db.py                                                          integration tests, skip themselves if Redis/Postgres aren't reachable
  conftest.py
frontend/
  src/App.tsx            tab switcher between Home, Refine, and Dashboard
  src/Home.tsx             landing screen: the pitch, then one CTA into Refine
  src/Refine.tsx            goal intake, live per-candidate progress, ranked candidates with a highlighted top pick and judge agreement, check-type selector, register button
  src/Dashboard.tsx         registered prompts, live drift badges, run history drill-down with a pass/fail trend, deactivate control
  src/api.ts                fetch wrapper + hand-written types for every request/response shape
  tsconfig.json
docker-compose.yml     postgres + redis + api + worker — docker compose up is the whole local startup
Dockerfile
.github/workflows/
  ci.yml               tests (Postgres + Redis service containers) + frontend build, on every push/PR
  monitor.yml           the GH Actions cron itself — needs DATABASE_URL/REDIS_URL secrets pointing at deployed infra

Frontend

A React + Vite + TypeScript client in frontend/ with three screens, switched by a tab:

cd frontend
npm install
cp .env.example .env   # set VITE_API_KEY to match the backend's API_KEY, if you set one
npm run dev

Open http://localhost:5173 with the backend already running on :8000. The dev server proxies /api to http://localhost:8000 (vite.config.ts), so the browser only ever talks to the Vite origin for local dev. npm run build runs tsc -b before vite build, so a type error fails the build the same way it fails CI.

Home: the pitch, a 4-step summary of how the two halves fit together, and a single call to action into Refine — not the tool itself.

Refine: submit a goal (+ optional category and example inputs) and watch it move through a real, backend-reported stage tracker (Generating → Ranking → Validating → Done) with per-candidate status while generation is still fanning out across the worker pool. Once ranked, the top pick is visually distinguished, judge agreement is shown ("3/3 judges picked this as their top choice" or a split decision), and each candidate can be registered with any of the four check types via a small type-aware form — not just a bare "contains" text box.

Dashboard: every registered prompt with a live drift badge (no runs yet / NN% pass (window) / ⚠ drifting · NN% pass), and a click-through to that prompt's run history — a pass/fail trend dot sequence across the visible runs, plus each run's output, latency, real cost, and per-check pass/fail. Registered prompts can be deactivated from the drill-down to free a slot against the registry cap.

CI

.github/workflows/ci.yml runs on every push/PR: spins up Postgres and Redis as service containers, runs pytest (the queue, drift, and db integration tests exercise the real fan-in counter, the real drift SQL query, and the real batched check-results query against those containers — no LLM API key needed anywhere in CI, since pytest never calls models.call_model), and separately builds the frontend. .github/workflows/monitor.yml is the actual cron — it isn't currently wired to a deployed instance; running this project is a local (Docker Compose) exercise, by design — see "Why this isn't deployed" above.