🔍 RepeatRadar

PhD Manuscript Repetition Detector & Fixer

Stop repeating yourself — let AI find it for you

---

🎯 The Problem

When writing a PhD thesis over months or years, the same ideas inevitably appear in multiple chapters, sometimes word-for-word, sometimes paraphrased. RepeatRadar finds these repetitions semantically (not just exact matches), scores them by severity, and lets you fix them with one click using Gemini AI.

---

✨ Features

• 📄 Upload PDF or DOCX — supports any academic document format
• 🧠 Semantic detection — finds repetitions even when phrased differently
• 🎯 Smart scoring — 🔴 High / 🟡 Medium / 🟢 Low based on similarity, length, and signpost keywords
• 🛠 Three fix strategies — rewrite, shorten to recall sentence, or replace with cross-reference
• ✅ Accept / Reject / Edit — you stay in control, AI just suggests
• 📥 Export fixes — download a .txt changelog to apply to your Word/LaTeX document
• ⚡ Caching — embeddings saved to disk, analysis is instant on repeat runs
• 💰 100% free — uses Gemini free tier, no paid API needed

---

🏗 Architecture

PDF/DOCX Upload
    ↓
LangChain RecursiveCharacterTextSplitter → tagged chunks
    ↓
Gemini gemini-embedding-001 → FAISS FlatIP index (cached)
    ↓
FAISS search (top-K) → cosine similarity pairs
    ↓
Filter: skip preamble / references / math / adjacent chunks
    ↓
Score: similarity + length + signpost keywords → 🔴/🟡/🟢
    ↓
Streamlit UI — side-by-side pair view
    ↓
Fix on demand → Gemini Flash rewrite → Accept / Reject

---

🖥 Screenshots

Upload	Analyse	Review & Fix
Drag & drop PDF/DOCX	Real-time embedding progress bar	Side-by-side pair comparison with fix panel

---

🚀 Quick Start

Prerequisites

• Python 3.11+
• Free Gemini API key → Get one here

1. Clone the repository

git clone https://github.com/bazzal99/RepeatRadar.git
cd RepeatRadar

2. Set up environment

cp .env.example .env
# Open .env and add your Gemini API key

3. Install dependencies

pip install -r requirements.txt
pip install streamlit==1.40.0 requests==2.32.3

4. Run the backend

cd backend
uvicorn app.main:app --reload --port 8080

5. Run the frontend (new terminal)

cd frontend
streamlit run app.py

6. Open the app

http://localhost:8501

---

🤖 Use as an MCP Server (Claude Desktop, Claude Code, etc.)

RepeatRadar's duplicate-detection pipeline (ingest → embed → detect → score) is also exposed as a Model Context Protocol (MCP) tool (mcp_server/server.py), so any MCP-compatible client can call find_duplicates(file_path, doc_id=None) directly on a PDF or DOCX — no need to run the Streamlit UI. No changes were made to backend/app/services/*; the MCP server only imports and calls the functions that already exist there.

Tool exposed: find_duplicates(file_path, doc_id=None) → returns a JSON report: total pairs found, counts per severity (high/medium/low), and the pairs themselves (chunk text truncated for readability).

Setup

cd mcp_server
python -m venv venv
source venv/bin/activate        # Windows: venv\Scripts\activate
pip install -r requirements.txt
export GOOGLE_API_KEY="your-gemini-api-key"   # Windows: $env:GOOGLE_API_KEY="..."

Why a separate requirements.txt? The mcp SDK requires pydantic>=2.12.0, while the FastAPI backend above is pinned to pydantic==2.8.2. Merging them into one file would force an untested pydantic upgrade on the core app, so mcp_server/ keeps its own isolated environment on purpose.

Run standalone (sanity check)

python server.py

A silent hang with no error means it started correctly — it's waiting for a client over stdio. Ctrl+C to stop.

Connect to Claude Desktop

Add to your claude_desktop_config.json (Settings → Developer → Local MCP servers → Edit Config in Claude Desktop):

{
  "mcpServers": {
    "repeatradar": {
      "command": "/absolute/path/to/RepeatRadar/mcp_server/venv/bin/python",
      "args": ["/absolute/path/to/RepeatRadar/mcp_server/server.py"],
      "env": { "GOOGLE_API_KEY": "your-gemini-api-key" }
    }
  }
}

On Windows, use venv\Scripts\python.exe and escaped backslashes (\\) in the paths.

Restart Claude Desktop, then ask something like:

"Use repeatradar to find duplicates in C:\path\to\my_thesis.pdf"

Claude will call find_duplicates directly and report back the flagged pairs. First run on a new file embeds and caches it (mcp_server/faiss_store/<doc_id>.index); subsequent calls on the same file reuse the cache.

Troubleshooting

• Path errors on the local server: the server sets its own working directory (os.chdir to its own folder) at startup specifically because MCP clients may launch it from an arbitrary cwd — if you fork this, keep that line.
• Tool not visible in a chat: some Claude Desktop builds require enabling the connector per-conversation — check the "+" / Context panel near the message box.
• "File not found": the path must be a real path on the machine running the server, not an uploaded/attached file in the chat — the server can't reach files that only exist inside the conversation.

---

🐳 Docker (Optional)

cp .env.example .env
# Add your API key to .env

cd infra
docker compose up -d

• Frontend: http://localhost:8501
• API docs: http://localhost:8080/docs

---

⚙️ Configuration

Edit RepeatRadar/.env:

# Embedding key (gemini-embedding-001)
GOOGLE_API_KEY=your_key_here

# Optional: separate key for fix agent to avoid quota conflicts
GOOGLE_API_KEY_GENERATION=your_second_key_here

# Detection sensitivity (0.92 = recommended, lower = more pairs found)
SIMILARITY_THRESHOLD=0.92

# Chunk size for splitting (characters)
CHUNK_SIZE=400
CHUNK_OVERLAP=80

Tip: Use two separate Gemini API keys — one for embeddings and one for the fix agent. This prevents quota conflicts since embedding and generation share the same free tier limits.

---

📁 Project Structure

RepeatRadar/
├── backend/
│   ├── app/
│   │   ├── api/routes/
│   │   │   ├── documents.py      # Upload & chunk endpoints
│   │   │   └── analysis.py       # Embed, analyse, fix, accept endpoints
│   │   ├── core/
│   │   │   └── config.py         # Settings from .env
│   │   ├── services/
│   │   │   ├── ingestor.py       # PDF/DOCX → LangChain chunks
│   │   │   ├── embedder.py       # Gemini embeddings + FAISS cache
│   │   │   ├── detector.py       # FAISS search + pair filtering
│   │   │   ├── scorer.py         # 🔴/🟡/🟢 scoring logic
│   │   │   └── fixer.py          # Gemini Flash rewriting agent
│   │   └── main.py
│   └── Dockerfile
├── frontend/
│   ├── app.py                    # Full Streamlit UI
│   └── Dockerfile
├── mcp_server/
│   ├── server.py                 # MCP tool wrapping the detection pipeline
│   ├── manifest.json             # Desktop Extension (.mcpb) manifest
│   └── requirements.txt          # Separate env: mcp SDK needs pydantic>=2.12.0
├── infra/
│   └── docker-compose.yml
├── .env.example
├── requirements.txt
├── LICENSE
└── README.md

---

🔬 How Scoring Works

Each flagged pair is scored on three signals:

Signal	Criteria	Points
Similarity	≥ 0.95 (near-identical)	+2
	≥ 0.90 (very similar)	+1
	≥ 0.92 threshold	0
Length	Both chunks ≥ 200 chars	+2
	One chunk ≥ 200 chars	+1
Signpost	"as introduced in", "recall that", etc.	−2

Final score → severity:

• 🔴 High (4-5 pts) — fix strongly recommended
• 🟡 Medium (2-3 pts) — review recommended
• 🟢 Low (0-1 pts) — likely intentional, shown collapsed

---

🛠 Fix Strategies

Strategy	What it does	Best for
Rewrite	Different phrasing, same meaning	Same-chapter repetition
Shorten	Compress to one recall sentence	Intro repeated in Conclusion
Reference	Replace with pointer to original	When repetition adds no value

---

🧪 Tech Stack

Layer	Technology
LLM + Embeddings	Google Gemini (free tier)
Vector Index	FAISS FlatIP
Chunking	LangChain RecursiveCharacterTextSplitter
API	FastAPI + Uvicorn
Frontend	Streamlit
PDF parsing	pypdf
DOCX parsing	python-docx
Containers	Docker Compose

---

👤 Author

Mohammad Bazzal — ML Engineer | PhD Candidate in Telecommunications
IMT Atlantique, Lab-STICC, Brest, France

---

📄 License

This project is licensed under the MIT License — see the LICENSE file for details.

---

Built to solve a real PhD problem — writing the same thing twice without realizing it.