Multi-Model RAG Document Reader

A full-stack Retrieval-Augmented Generation (RAG) system that lets you upload documents and ask questions about them.

Technical RAG Architecture & Optimizations

Under the hood, this application uses a highly optimized, multi-stage RAG architecture to ensure accuracy, context preservation, and speed.

1. Multi-Model Hierarchical Structure

The system strategically divides tasks across a hierarchical multi-model architecture to balance speed and intelligence:

• Metadata LLM (llama3.2 local): A fast, local model used exclusively during document ingestion to read the raw text and generate structured metadata (summaries, tags, keywords).
• Embedding Model (nomic-embed-text local): A specialized, high-dimensional vector model used to encode text chunks into numbers.
• Generation LLM (llama-3.3-70b-versatile cloud): A massive, highly capable model hosted on Groq that handles the final user reasoning and answer generation.

2. Advanced Ingestion & Parsing

Instead of blindly reading text, the system uses format-specific parsers (like PyMuPDF for PDFs and python-docx for Word files) to extract both raw text and structured tabular data. Magic-byte validation prevents malicious file spoofing.

3. Hierarchical Semantic Chunking

A naive chunking strategy splits text at fixed token counts, breaking sentences and losing context. This system uses Semantic Chunking:

• Heading Awareness: It first splits documents by markdown headings (#), keeping sections intact.
• Paragraph & Sentence Boundaries: It recursively splits large sections by paragraphs and then sentences.
• Context Overlap: Consecutive chunks include an overlap of a few sentences (configurable via CHUNK_OVERLAP_SENTENCES) to guarantee that the context bridging two chunks is never lost.

4. LLM-Powered Metadata Extraction

When a document is uploaded, a local LLM (llama3.2) is immediately dispatched to read the content and generate rich metadata:

• Extracted Title and Summary
• Keyword extraction
• Categorization and tagging This metadata is attached to the vector payload in Qdrant, allowing users to apply strict pre-filters (e.g., "only search in the 'Research' category") before the vector similarity search even begins.

5. Hybrid Retrieval & Keyword Reranking

Standard vector search can sometimes surface conceptually similar but factually irrelevant results. To counter this:

• Dense Retrieval: We first retrieve Top K * 2 results using Cosine Similarity against the nomic-embed-text embeddings.
• Keyword Reranking: A custom scoring algorithm calculates the exact keyword overlap between the user's query and the retrieved chunks. The final score is a weighted blend (70% vector score + 30% keyword score). The results are then sorted and truncated to the exact Top K requested.

6. Context Assembly & Injection

Retrieved chunks are formatted with strict provenance tracking. The context injected into the prompt looks like this:

Document: [Title] | Page: [X] | Section: [Heading]
[Chunk Text]

This forces the generation LLM to "see" exactly where the information came from, drastically reducing hallucinations and enabling the backend to map the LLM's response to structured, clickable UI citations.

7. Streaming WebSocket Generation

To eliminate perceived latency, the final prompt is sent to high-speed inference providers (like Groq) and the response is streamed back to the frontend character-by-character over WebSockets.

Getting Started & Setup Guide

1. Prerequisites

• Python 3.11+
• Node.js 20+
• Docker (for running Qdrant)

2. Setting up Qdrant (Vector Database)

Qdrant is where the document chunks are stored. You can run it easily using Docker:

docker run -p 6333:6333 -p 6334:6334 -v $(pwd)/qdrant_storage:/qdrant/storage:z qdrant/qdrant

This will start Qdrant on localhost:6333.

3. Setting up Ollama (Local AI Models)

We use local models for turning text into numbers (embeddings) and generating document summaries.

1. Download and install Ollama.
2. Once installed, open your terminal and download the required models:

ollama run nomic-embed-text
ollama run llama3.2

Keep Ollama running in the background.

4. Setting up API Keys (Groq)

We use Groq's high-speed API to run large models like llama-3.3-70b-versatile for answering questions.

1. Go to the Groq Console and create an account.
2. Generate an API Key.
3. Open the .env file in the root of this project and add your key:

# --- Groq LLM (Required) ---
GROQ_API_KEY=your_groq_api_key_here

# --- Ollama (Local LLM & Embeddings) ---
OLLAMA_BASE_URL=http://localhost:11434
EMBEDDING_MODEL=nomic-embed-text
LOCAL_LLM_MODEL=llama3.2

# --- Groq Model ---
GROQ_MODEL=llama-3.3-70b-versatile

# --- Qdrant Vector Database ---
QDRANT_HOST=localhost
QDRANT_PORT=6333
QDRANT_COLLECTION=rag_documents

# --- Application ---
UPLOAD_DIR=data/uploads
METADATA_DIR=data/metadata

5. Running the Application

Start the Backend: Open a terminal in the root directory:

cd backend
python -m venv venv
venv\Scripts\activate  # On Windows
pip install -r requirements.txt
python run.py

The backend will run on http://localhost:8000.

Start the Frontend: Open a new terminal window:

cd frontend
npm install
npm run dev

The frontend will run on http://localhost:3000.

Pipeline Walkthrough (Detailed)

The complete pipeline from document ingestion to answer generation consists of 8 stages. Here is the technical breakdown of each step:

Stage 1: Document Ingestion

Triggered by POST /api/documents/upload (file) or POST /api/documents/url (remote download).

1. File Validation — Extension allowlist check → magic-byte content validation → size check (<50MB).
2. Storage — File saved to data/uploads/ with a UUID4-based filename.
3. Parsing — Format-specific parser extracts raw text and tables.
   • PyMuPDF for PDFs.
   • python-docx for Word documents.
   • BeautifulSoup for HTML/URLs.
4. Metadata Generation — A local LLM (llama3.2) analyzes the raw text to generate a title, summary, keywords, categories, and tags. This is cached in data/metadata/.
5. Semantic Chunking — Text is hierarchically split by markdown headings, then paragraphs, then sentences, aiming for ~300 tokens per chunk with an overlap of 2 sentences to preserve context.
6. Embedding — Each chunk is embedded into a 768-dimensional vector via Ollama (nomic-embed-text). Identical chunks are deduplicated via a disk-backed SHA256 cache.
7. Vector Indexing — Chunks and their embeddings are upserted into Qdrant in batches of 100.

Stage 2: Query Processing

1. Query is received via POST /api/chat or POST /api/search.
2. Search parameters are extracted: top_k, and filters (categories, tags, document_ids, domain).
3. The in-memory conversation history is loaded (for chat endpoints) to maintain multi-turn context.

Stage 3: Retrieval

1. The user's query is embedded using the exact same Ollama embedding model (nomic-embed-text).
2. A fast vector similarity search (Cosine Distance) is performed against the Qdrant database.
3. If reranking is enabled, the system retrieves top_k * 2 candidates.
4. Qdrant Filter conditions (must/must_not) are applied directly at the database level to exclude unmatched documents (e.g., filtering out deleted docs or specific domains).

Stage 4: Reranking

To improve retrieval precision beyond just dense vector similarity, a hybrid keyword-overlap reranker is applied:

• Vector score contributes 70% of the final score.
• Keyword overlap (matching tokens between the user's query and the chunk) contributes 30%.
• Results are re-sorted by this combined score and truncated back to exactly top_k.

Stage 5: Context Assembly

1. The retrieved chunks are sorted by their relevance score.
2. A context string is built, injecting provenance data into the text so the LLM knows exactly where it came from:

   Document: Title | Page: 3 | Section: Introduction
   [Chunk Text]
   

3. Total token count is estimated. If it exceeds the target context window, the lowest-scoring chunks are safely truncated.
4. UI Citations are tracked and structured for each included chunk.

Stage 6: Prompt Rendering

1. The selected prompt template (default: question_answering.md) is loaded from the filesystem.
2. Template variables are dynamically replaced:
   • {context} — The concatenated retrieved chunks.
   • {query} — The user's question.
   • {chat_history} — Previous messages in the session.
3. The final rendered prompt is prepared for the LLM.

Stage 7: LLM Generation

1. The prompt is sent to the active provider (e.g., Groq using the groq Python SDK).
2. The response is streamed token-by-token over a WebSocket connection to ensure ultra-low latency for the user.
3. Token usage metrics (input/output) are tracked for analytics.

Stage 8: Citation Assembly

1. The retrieved chunk sources are mapped to inline citations in the UI.
2. The final response is delivered to the frontend, allowing the user to click any citation chip to view the exact Document, Page, Section, and snippet where the AI found the answer.