An MVP for algorithmic document search: upload documents, then search them with a hand-written TF-IDF + cosine similarity ranking engine. No external AI/LLM service is used anywhere in the backend — the ranking logic is plain Python, so there are no API keys to configure and nothing to run to get started besides the two servers below.
See docs/ARCHITECTURE.md for the full pipeline/scaling design and
docs/DECISIONS.md for why this project avoids generative AI and how
extraction is sequenced.
- Document upload —
.txtand.pdf(viapypdf), chunked paragraph-aware into ~600-word pieces for meaningful retrieval units. - Persistent storage — SQLite (
backend/idp.db), survives restarts. - Algorithmic search — term frequency / inverse document frequency vectors built fresh per query from the full corpus, ranked by cosine similarity. Results are a ranked list of matching chunks with a score and highlighted matched terms — not a generated "answer."
- Document management — list and delete uploaded documents.
Backend:
- Python 3 + FastAPI + Uvicorn
- SQLite (stdlib
sqlite3) for persistence pypdffor PDF text extractionnumpyfor vector math only — the TF-IDF/cosine-similarity algorithm itself is hand-written inbackend/tfidf.py
Frontend:
- Vite + React 18 + TypeScript
- MUI (
@mui/material) for components
cd backend
python -m venv venv
source venv/bin/activate # Windows: venv\Scripts\activate
pip install -r requirements.txtNo .env file or API key is required.
npm installOptionally set VITE_API_URL (defaults to http://localhost:8001) via a
.env.local file — see .env.example.
# Terminal 1
cd backend && python main.py # http://localhost:8001
# Terminal 2
npm run dev # http://localhost:3000GET /— API info + document countPOST /api/upload— upload a.txtor.pdffile (multipartfilefield)POST /api/search—{"query": "...", "top_k": 5}→ ranked chunk resultsGET /api/documents— list uploaded documentsDELETE /api/documents/{id}— remove a document and its chunksGET /docs— interactive API documentation
curl -X POST http://localhost:8001/api/upload -F "file=@document.pdf"
curl -X POST http://localhost:8001/api/search \
-H "Content-Type: application/json" \
-d '{"query": "your search terms"}'- Upload — text is extracted (
backend/extractors.py) and split into chunks (backend/chunking.py), then persisted to SQLite. - Search — every chunk in the corpus is tokenized, a TF-IDF matrix is
built fresh (
backend/tfidf.py), the query is projected into the same vector space, and chunks are ranked by cosine similarity. - Result — the top-k chunks are returned with filename, score, and the specific query terms that matched, for client-side highlighting.
Because TF-IDF's inverse-document-frequency term depends on the whole corpus, vectors are computed at query time rather than cached at upload time — this keeps rankings correct as new documents are added, at the cost of recomputing the matrix per search (fast at MVP scale).
Search-AI/
├── backend/
│ ├── main.py # FastAPI app — routes only
│ ├── db.py # SQLite persistence (documents, chunks)
│ ├── extractors.py # .txt/.pdf -> text
│ ├── chunking.py # paragraph-aware chunking
│ ├── tfidf.py # hand-written TF-IDF + cosine similarity
│ └── requirements.txt
├── src/
│ ├── components/
│ │ ├── Header.tsx
│ │ ├── DocumentPanel.tsx # upload, list, delete
│ │ └── AskPanel.tsx # query box + ranked results
│ ├── api.ts # typed fetch wrappers
│ ├── types.ts
│ ├── App.tsx
│ └── main.tsx
├── vite.config.ts
├── tsconfig.json
└── package.json
- OCR / scanned images, DOCX and formats beyond TXT/PDF
- Smarter ranking (BM25, n-grams, phrase matching) as the next iteration on top of this TF-IDF baseline
- Structured field/entity extraction, document classification
- Auth, multi-tenancy, rate limiting
- Production CORS lock-down (currently
allow_origins=["*"]for local MVP) - Deployment/hosting
MIT