Skip to content
BranchesTags

Repository files navigation

RAG Evaluator

A tool for benchmarking and comparing RAG (Retrieval-Augmented Generation) pipeline configurations. Upload documents, define named pipelines with different chunking strategies, embedding models, and rerankers, then score responses with RAGAS metrics — all from a Streamlit UI backed by a FastAPI service.

Features

  • Document ingestion — upload PDF, TXT, Markdown, or DOCX files
  • Flexible chunking — sentence, fixed-size, token, semantic, or hierarchical splitting
  • Embedding model selection — HuggingFace (local), OpenAI, or Cohere per pipeline
  • Reranking — none, BGE cross-encoder (local), Cohere Rerank, or LLM reranker
  • Retrieval modes — cosine similarity or MMR (maximum marginal relevance)
  • RAGAS evaluation — faithfulness, answer relevancy, context precision/recall, answer correctness
  • Persistent results — all runs and evaluations stored as JSON; downloadable as CSV
  • Side-by-side comparison — bar charts, radar plots, and heatmaps across pipeline configs

Architecture

┌──────────────────────────────────────────────────────────┐
│                    Streamlit Frontend                    │
│  Upload │ Pipeline Config │ Evaluate │ Results           │
└─────────────────────┬────────────────────────────────────┘
                      │ HTTP (httpx)
┌─────────────────────▼────────────────────────────────────┐
│                    FastAPI Backend                       │
│                                                          │
│  Routes ──► Services ──► Repositories                    │
│                │                  │                      │
│                ▼                  ▼                      │
│         LlamaIndex          JSON files                   │
│              │              (runs, evals,                │
│              ▼              pipeline configs)            │
│          ChromaDB                                        │
│       (persistent vector                                 │
│            store)                                        │
└──────────────────────────────────────────────────────────┘

Service layer

Service Responsibility
document_service Async file upload and metadata persistence
chunking_service Wraps LlamaIndex node parsers for 5 strategies
embedding_service Resolves embedding model by provider/model-id; caches loaded weights
indexing_service Builds/loads LlamaIndex VectorStoreIndex backed by ChromaDB
reranker_service Returns a configured LlamaIndex BaseNodePostprocessor or None
generation_service Builds RetrieverQueryEngine, executes query, returns RunResult
evaluation_service Runs RAGAS evaluate() on a completed RunResult
pipeline_service CRUD for named pipeline configurations (JSON-backed)

Project structure

rag-evaluator/
├── pyproject.toml              # uv project — all dependencies declared here
├── .env.example                # API key template
│
├── backend/
│   ├── main.py                 # FastAPI app, CORS, exception handlers
│   ├── config.py               # Settings via pydantic-settings (.env aware)
│   ├── core/
│   │   ├── chroma.py           # ChromaDB persistent client (singleton)
│   │   └── exceptions.py       # Typed domain exceptions
│   ├── models/                 # Pydantic request/response schemas
│   │   ├── document.py
│   │   ├── pipeline.py         # ChunkingConfig, RerankerConfig, PipelineConfig, …
│   │   ├── run.py
│   │   └── evaluation.py       # RAGASMetrics, EvaluationResult, …
│   ├── repositories/
│   │   ├── results_repo.py     # JSON file persistence for runs & evaluations
│   │   └── vector_store_repo.py # (doc_id, pipeline_id) → ChromaDB collection map
│   ├── services/               # Business logic — one file per concern
│   └── api/
│       ├── dependencies.py     # FastAPI Depends wrappers
│       └── routes/             # documents / pipelines / runs / evaluations / results
│
├── frontend/
│   ├── app.py                  # Home page with live KPI summary
│   ├── api_client.py           # httpx wrappers for every backend endpoint
│   ├── components/
│   │   ├── sidebar.py          # Nav + backend health badge
│   │   └── charts.py           # Radar, bar, heatmap, latency helpers (Plotly)
│   └── pages/
│       ├── 1_Upload.py         # Upload doc; choose embedding model
│       ├── 2_Pipeline.py       # Create / view named pipeline configs
│       ├── 3_Evaluate.py       # Run query → poll → show RAGAS scores
│       └── 4_Results.py        # Compare evaluations; CSV export
│
├── data/                       # Runtime data (gitignored)
│   ├── uploads/                # Raw uploaded files + metadata JSON
│   ├── chroma/                 # ChromaDB persistent storage
│   ├── results/runs/           # RunResult JSON files
│   ├── results/evaluations/    # EvaluationResult JSON files
│   └── pipelines/              # PipelineConfig JSON files
│
└── tests/
    ├── conftest.py             # Fixtures: temp dirs, patched settings, TestClient
    ├── test_services/          # Unit tests for chunking strategies
    └── test_api/               # Integration tests for document CRUD

Setup

Prerequisites

  • Python 3.11+
  • uv — pip install uv
  • (Optional) OpenAI API key for OpenAI embeddings / LLM reranker / generation
  • (Optional) Cohere API key for Cohere embeddings / Cohere reranker

Install

git clone <repo-url>
cd rag-evaluator

# Create virtualenv and install all dependencies
uv sync

Configure

cp .env.example .env

Edit .env and fill in your keys:

OPENAI_API_KEY=sk-...
COHERE_API_KEY=...
OPENAI_MODEL=gpt-4o-mini

Leave blank if you only intend to use local HuggingFace models (no API key required).

Running

Open two terminals:

# Terminal 1 — FastAPI backend
uv run uvicorn backend.main:app --reload --port 8000

# Terminal 2 — Streamlit frontend
uv run streamlit run frontend/app.py
Interface URL
Streamlit UI http://localhost:8501
FastAPI docs (Swagger) http://localhost:8000/docs
FastAPI docs (ReDoc) http://localhost:8000/redoc

Usage walkthrough

1. Upload a document

Go to Upload, choose an embedding model, and upload a PDF or text file.

Local HuggingFace models (BAAI/bge-small-en-v1.5, all-MiniLM-L6-v2) work without any API key and are a good default for experimentation.

2. Create pipeline configs

Go to Pipeline and create two or more configs with different settings so you can compare them. Example configs to try:

Name Chunking Embedding Reranker
Baseline sentence / 512 BGE-Small none
Semantic + Rerank semantic / 512 BGE-Large BGE cross-encoder
OpenAI stack sentence / 256 text-embedding-3-small Cohere

3. Run a query and evaluate

Go to Evaluate, select a document and pipeline, type your question, optionally provide a ground-truth answer, and click Run Query & Evaluate.

Metrics that require a ground truth: context_recall, answer_correctness.

4. Compare results

Go to Results to view all evaluations in a table, compare pipelines with bar charts and heatmaps, and export to CSV.

Supported options

Chunking strategies

ID Description
sentence Split on sentence boundaries (default)
fixed_size Fixed token/character window
token Token-count based splitting
semantic Embedding-based semantic grouping (requires embed model at index time)
hierarchical Multi-granularity nested chunks (4×, 2×, 1× chunk size)

Embedding models

ID Provider Dimensions API key
huggingface/BAAI/bge-small-en-v1.5 HuggingFace 384 No
huggingface/BAAI/bge-large-en-v1.5 HuggingFace 1024 No
huggingface/sentence-transformers/all-MiniLM-L6-v2 HuggingFace 384 No
openai/text-embedding-3-small OpenAI 1536 Yes
openai/text-embedding-3-large OpenAI 3072 Yes
cohere/embed-english-v3.0 Cohere 1024 Yes

Rerankers

ID Description API key
none No reranking No
flag_embedding BGE cross-encoder (BAAI/bge-reranker-base) No
cohere Cohere Rerank v3 Yes
llm LLM-based reranker (OpenAI) Yes

RAGAS metrics

Metric Ground truth required
faithfulness No
answer_relevancy No
context_precision No
context_recall Yes
answer_correctness Yes
answer_similarity Yes

REST API

All endpoints are prefixed with /api/v1. Interactive docs at /docs.

POST   /documents/upload          Upload a document
GET    /documents/                List documents
GET    /documents/{id}            Get document
DELETE /documents/{id}            Delete document

POST   /pipelines/                Create pipeline config
GET    /pipelines/                List pipeline configs
GET    /pipelines/{id}            Get pipeline config
DELETE /pipelines/{id}            Delete pipeline config
GET    /pipelines/meta/embedding-models
GET    /pipelines/meta/rerankers
GET    /pipelines/meta/chunking-strategies

POST   /runs/                     Start a RAG run (async)
GET    /runs/                     List runs
GET    /runs/{id}                 Get run (poll status)
DELETE /runs/{id}                 Delete run

POST   /evaluations/              Start RAGAS evaluation (async)
GET    /evaluations/              List evaluations
GET    /evaluations/{id}          Get evaluation (poll status)
DELETE /evaluations/{id}          Delete evaluation

POST   /results/compare           Compare multiple evaluations
GET    /results/summary           Overall stats

Tests

uv run pytest                     # all tests
uv run pytest tests/test_api/     # API integration tests only
uv run pytest tests/test_services/ # service unit tests only
uv run pytest -v                  # verbose output

Adding a new embedding model

  1. Add an entry to AVAILABLE_EMBEDDING_MODELS in backend/services/embedding_service.py.
  2. Add the corresponding if provider == "..." branch in get_embed_model().
  3. Add the model ID to the EmbeddingModel literal type in backend/models/pipeline.py.

Adding a new reranker

  1. Add an entry to AVAILABLE_RERANKERS in backend/services/reranker_service.py.
  2. Add the corresponding if strategy == "..." branch in get_reranker().
  3. Add the strategy to the RerankerStrategy literal type in backend/models/pipeline.py.

run

uv run uvicorn backend.main:app --reload --reload-dir backend --port 8000 --log-level debug

About

No description, website, or topics provided.

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages