Tapat

A Python toolkit for evaluating RAG (Retrieval-Augmented Generation) pipelines — retrieval accuracy, answer faithfulness, and Taglish code-switching degradation.

Why

RAG systems fail silently. A pipeline can return a confident, well-written answer that's wrong in ways no one notices: the retriever misses the right document but the LLM covers the gap with a guess from its own training data, or the pipeline works fine in English but degrades for Taglish, which is how real Filipino users actually type. Tapat puts numbers on both failure modes instead of "it seems to work."

Given a test set of questions with known-correct source documents, Tapat runs each question through a target RAG system and scores it on:

1. Retrieval hit rate — did the retriever surface the correct source document(s)?
2. Faithfulness — is the generated answer grounded in what was retrieved, or did the model answer from parametric memory?
3. Taglish degradation — does answer quality drop when the same question is asked in Taglish instead of English?

The third is Tapat's original contribution — no existing open-source eval toolkit measures code-switching degradation for Filipino users. Full spec: tapat-prd-v0.md.

Status

v0, actively being built, week 1 of the milestone plan. Implemented so far:

• Core schemas (TestCase, RetrievedChunk, RAGResult, RAGTarget) — tapat/schemas.py
• MockAdapter for offline testing — tapat/adapters/mock.py
• Retrieval hit rate + MRR metrics — tapat/metrics/retrieval.py
• Faithfulness judge (LLM-as-judge, structured output)
• Taglish degradation deltas
• LasalliaAdapter
• CLI (tapat run / report / diff)
• CI (lint, type check, tests on push)
• Lasallia LRC test set (40–60 cases)

Not usable end-to-end yet — there's no CLI or run command. What exists is unit-tested and passes lint/type checks.

How it works

Tapat never imports a target RAG system directly. The target implements one interface:

class RAGTarget(Protocol):
    def query(self, question: str) -> RAGResult: ...

Tapat calls query(), gets back an answer plus the retrieved chunks, and scores the result against a test case's known-correct document IDs. This keeps all target-specific code isolated to a single adapter file — swapping in a different chatbot means writing one new adapter, not touching Tapat itself.

Repo structure

tapat/
  tapat/            # package: schemas.py, adapters/, metrics/, judges/ (planned: cli.py, runner.py)
  tests/            # mirrors the package structure
  testsets/         # versioned test sets + JSON Schema for the format
  judges/           # judge prompts, versioned (planned)
  tapat-prd-v0.md   # full spec, source of truth
  BUILDLOG.md       # what was delegated to Claude Code, what it got wrong, what was corrected

Development

pip install -e ".[dev]"
pytest tests/
ruff check tapat/ tests/
pyright tapat/ tests/

Tests never call real APIs — everything runs against MockAdapter and fixtures.

Build process

Built agent-directed: specs written first, Claude Code implements per chunk, human review on every commit. BUILDLOG.md tracks what was delegated, what the agent got wrong, and what was corrected — a record of the process, not just the output.

License

MIT