TL;DR — I ran merchant support at a payments company on an AI stack I built myself, instead of buying per-seat vendor add-ons, because what decides a good support answer (our SOPs, our voice, our definition of "resolved") is exactly what a vendor can't sell. Five compounding builds; a roughly six-person team carried first-line support past half a billion dollars in payments. The reasoning and the trade-offs are below.
A sanitized case study of a system I designed and built at ScanPay, a payments company for the trades. It is the reasoning and the architecture, not the code. I built all of it myself with Claude Code; it runs in production and is now maintained by an engineering team. I owned Merchant Support as employee #3 and Head of Product, a roughly six-person team fielding tickets, calls, and SMS.
Run first-line support at a deliberately smaller headcount by owning the AI layer, instead of buying a bundle whose headline value I was already building.
Every vendor's AI add-on (Freshdesk's Freddy, Gong on the calls, Intercom's Fin on chat) priced per seat, per resolution, or behind a tier upgrade, across several tools, so cost grew with the team and the volume rather than the value. This is not a knock on those products. They are strong and full-featured, and for a large team they earn it. They were priced and scoped for a breadth we did not need at our size and for our specific use cases, and none could carry the things that actually decide a good answer at a payments company: our SOPs, our house voice, and our own definition of "resolved." Those are exactly what a vendor cannot sell you. So I kept the vendor where the value was generic (Freshdesk for tickets, CloudTalk for calls, Freshchat for chat transport) and built only where owning the value was the point.
Five builds, written to compound. Each produced something the next reused, so no build started cold. The hardest one, a customer-facing agent on a money product, stood on the shared foundations (the KB, the leak gate, the audit-the-first-model pattern), even though its merchant-facing brain, external voice, and guardrails were a fresh build.
flowchart TD
KB["Master KB + SOPs"] --> Gate["Leak gate<br/>internal view vs merchant view"]
Gate --> Nico["Nico<br/>internal agent for support + sales"]
Gate --> Kai["Kai<br/>merchant-facing agent, guardrailed"]
Nico --> Dash["Support-analytics dashboard<br/>cross-channel data + gated automations"]
Kai --> Dash
CallQA["Call-QA pipeline<br/>scores every call against our SOPs"] --> Dash
Judge["Judge lineage<br/>gap judge -> honesty judge -> human calibration"] -. audits .-> Nico
Judge -. audits .-> Kai
Judge -. audits .-> CallQA
| # | Build | The constraint that forced it | What it produced, and what the next build reused |
|---|---|---|---|
| 1 | Nico, an internal agent | No tool carried our SOPs or voice; per-seat assistants bill by the agent and answer generically | The brain, the KB grounding, the house voice, and an async gap-judge. Serves support and the sales team (answers product and SOP questions mid-conversation, coaches reps live). |
| 2 | KB pipeline | Productionizing to customers is a content-safety problem, not just a code one | One master KB compiled into two audience views behind a leak gate. The shared, safe source both agents run on. |
| 3 | Kai, a merchant-facing agent | An all-hours merchant self-serve solver that shrinks support cost, where a wrong answer on a money product is a liability, not a style miss | A purpose-built merchant-facing brain, not the internal agent re-pointed. Talking to merchants on a payments product is a different problem from answering an internal rep, so the reasoning was rebuilt for it, behind an honesty judge, a safety floor, and a strict data boundary that shows Kai only a subset of what the internal agent sees. |
| 4 | Support-analytics dashboard | Three siloed channels; per-seat analytics behind a tier wall | A cross-channel data layer joined on the one key every channel shares, then a gated write-back automation layer. The measurement and automation hub. |
| 5 | Call-QA pipeline | No systematic call quality; commercial tools priced per seat against a generic rubric | KB-grounded scoring of every call against our real SOP set, at roughly seven cents a call. Feeds the dashboard as a sixth channel. |
One architectural choice runs under all of them: each agent reads the KB out of a prompt-cached block, so the knowledge is present on every answer but paid for once, and retrieval was deliberately deferred (at our KB size, caching the whole thing beat the complexity of a retrieval layer). That is why the marginal cost of an answer is near zero, which is what let the team hold flat as the business scaled.
The through-line is one idea, escalated: a second model audits the first, and a human audits the judge. Nothing shipped on vibes.
- LLM-as-judge: Nico's gap judge grades every answer for fact and style gaps; Kai's honesty judge hard-fails an answer that over-promises.
- Golden sets and a 100-scenario regression bank: call-QA scores against our real SOPs; every Kai change runs the bank, and each past failure stays in it.
- Shadow evaluation: Kai's tuning ran fully offline on real cases before it ever touched a merchant.
- Human calibration: in the call-QA reviewer, a human "wrong" vote is the primary signal that keeps the judge honest.
- Traceable, not black-box: every score and label is defensible back to a rule or a cited source, because "the model said so" is not an answer you give a merchant on a payments product.
- Eval-gated rollout: no lane went live until it cleared its bank; the gap judge held about 92% on a labeled set.
The number I don't have yet is verified live deflection. The ticket-only view is blind to it, and that is the next build.
On a payments product a leak or an over-promise is a liability, not a style miss, so the model is boxed in on three sides: a deterministic floor, an LLM check, and a human.
- Least-privilege data boundary: Kai only ever sees a merchant-safe slice; the risk and tier fields never leave the internal side, so a leak is structurally impossible, not filtered after the fact.
- A leak gate, not a filter: the KB pipeline blocks any merchant view that would expose internal detail; what can't be scrubbed is held as a safe stub for a human (zero leaks in the build).
- A deterministic safety floor: plain rules strip internal identifiers and pointers on the way out, before an answer reaches a merchant.
- Operational-honesty hard-fails: over-promising an outcome or a time, over-reaching on risk, or claiming a capability it lacks each count as a hard failure the honesty judge blocks on.
- Input hardening and auth: every turn is screened, the one public endpoint is shared-secret gated, and the internal KB never ships to it.
Three layers, each with a clear job: a rule catches what a rule can, the model catches what a rule can't express, and a human owns what only a human should.
- Status. Shipped and in production; handed off to an engineering team, maintained in org-level repositories.
- Context. The value that mattered (our SOPs, our voice, our definition of "resolved") is not something a vendor can carry, yet every vendor priced its AI add-on per seat or behind a tier wall across several tools, so cost tracked headcount instead of value.
| Criterion | Buy the bundle | Build everything | Buy transport, build the AI layer |
|---|---|---|---|
| Cost tracks value, not headcount | ✗ per seat | ✓ | ✓ |
| Owns our KB, voice, definition of "resolved" | ✗ | ✓ | ✓ |
| Skips rebuilding commodity (ticketing, telephony) | ✓ | ✗ | ✓ |
| Time to first value | ✓ fast | ✗ slow | medium |
| Maintenance + eval burden we carry | low | high | medium (accepted) |
| ← chosen |
- Consequences. Cost decoupled from headcount, and we own the assets a vendor can't sell: the KB, the voice, the definition of a good answer, and every past failure, locked as a regression the next release has to pass. The honest cost is that we now carry the maintenance and the eval burden a vendor would otherwise own. The right trade at this scale, and a real one.
The common pushback is that the modern autonomous agents are priced per session now, not per seat, so the per-seat critique is dated. Fair, for the autonomous tier. But my objection was never "per-seat is bad." It was the tier-and-bundling trap: to reach the AI at all I would buy the mid-tier plan, per agent, across several support surfaces, whose headline value is the ticketing automation I was building anyway.
The market frame I find clarifying: a vertical AI vendor charges the seat price while its actual model cost is pennies, because the moat lives in the harness around the model (the grounding, the guardrails, the judge, the data model). Once you can build that harness yourself, the vendor's premium is paying for a moat you now own.
The deepest reason to build here is not cost. It is that a bought agent decides for you what counts as resolved, and bills you on its own definition.
A tool like Intercom Fin bills per resolution. It's a clean, modern model, but it means the vendor decides and scores what counts as resolved, and charges you on its own verdict. When you own the stack, you own and verify your own definition of a good outcome, instrument it, and hold the agent to it. On a payments product, where "resolved" and "over-promised" are a hair apart, that is the whole game.
A roughly six-person team carried first-line support for a business processing over half a billion dollars in payments, on a stack whose marginal cost per answer is near zero. Call quality is scored on every call at about seven cents each, against a tool like Gong at twelve to eighteen thousand dollars a year. The whole thing was built to hand off, and has been.
Stated plainly, because a case study that only lists wins is not trustworthy: the strongest live evidence is operational and offline (tuning deltas, gold-set accuracy, cost per call). The one number that closes the argument, verified all-hours production deflection, is the one I am still instrumenting. A few pieces shipped ahead of their test suites and are tracked as such. The code is private; I am happy to walk it in a conversation.
The specific system is a support stack, but the judgment is portable: where to build versus buy on AI, where an LLM is dangerous and how to box it in, and why owning your own definition of quality beats renting someone else's. That call shows up in every AI product, not just this one.
Built by Ritwik Manan. More at github.com/mananritwik.