Multimodal RAG API / 多模态 RAG API

English · 中文

---

English

Overview

Multimodal RAG API is a production-style implementation of a Retrieval-Augmented Generation (RAG) system for text + image documents (PDFs with charts, figures, and mixed layouts). It extends a course Streamlit prototype into a FastAPI service with an async Kafka worker for heavy PDF parsing.

Typical use cases: enterprise knowledge bases, course materials, reports with diagrams, and Q&A that must cite source file, page, and images.

Features

Feature	Endpoint / Module
Upload documents	POST /upload/document
List documents	GET /upload/documents?knowledge_base_id=
Delete documents	DELETE /upload/document/{file_id}
Multimodal Q&A	POST /chat
Serve images	GET /files/processed/..., /files/uploads/...
API key per KB	Header X-API-Key, POST /kb/api-keys
Offline indexing	worker/parse_document_worker.py
Batch evaluation	scripts/run_evaluation.py

Retrieval: BGE (text) + CLIP (text & image vectors), merged ranking.
Generation: Qwen (text) / Qwen-VL (when retrieved chunks contain images).
Parsing: MinerU (offline, via Kafka).

Architecture

flowchart LR
  Client[Client / UI]
  API[FastAPI]
  DB[(SQLite)]
  Kafka[Kafka]
  Worker[Parse Worker]
  MinerU[MinerU]
  Milvus[(Milvus)]
  LLM[DashScope Qwen / VL]

  Client --> API
  API --> DB
  API -->|upload| Kafka
  Kafka --> Worker
  Worker --> MinerU
  Worker --> Milvus
  API -->|chat: search| Milvus
  API -->|chat: generate| LLM
  API -->|static images| Client

Loading

Pipeline

1. Upload — PDF saved locally; metadata in SQLite; message to Kafka rag-data.
2. Worker — Consumes Kafka → MinerU → markdown + images → chunk → BGE + CLIP embeddings → Milvus.
3. Chat — Embed query → hybrid search (BGE + CLIP text + CLIP image) → build context → Qwen-VL answer with sources (file, page, image URLs).

Quick start (Mock, no GPU / Milvus / Kafka)

cd multimodal-rag-api
pip install -r requirements.txt
cp .env.example .env   # optional for mock
export RAG_MOCK_EXTERNAL=1   # Windows: set RAG_MOCK_EXTERNAL=1
pytest -q
uvicorn app.main:app --reload --port 8000

Open http://127.0.0.1:8000/docs for OpenAPI.

Production setup

1. Copy .env.example → .env and set:
   • MILVUS_URI, MILVUS_TOKEN
   • DASHSCOPE_API_KEY
   • BGE_MODEL_PATH, CLIP_MODEL_PATH
2. Start Kafka, MinerU (MINERU_BACKEND_URL, default http://127.0.0.1:30000).
3. python scripts/ensure_milvus_schema.py
4. uvicorn app.main:app --host 0.0.0.0 --port 8000
5. python -m worker.parse_document_worker
6. (Optional) python scripts/register_api_key.py <kb_id> <secret>

API examples

# Upload
curl -F "file=@doc.pdf" -F "knowledge_base_id=demo" \
  http://localhost:8000/upload/document

# Chat
curl -X POST http://localhost:8000/chat \
  -H "Content-Type: application/json" \
  -d '{"question":"When did product A sales decline?","knowledge_base_id":"demo","use_clip":true}'

Response sources[] includes file_name, page, image_urls (HTTP), retrieval_method (bge | clip_text | clip_image).

Evaluation (course rubric)

Weighted score: filename match (0.25) + page match (0.25) + answer Jaccard × 0.5.

python scripts/run_evaluation.py --dataset tests/fixtures/sample_eval.jsonl

Project layout

app/          FastAPI routes & services
worker/       Kafka consumer + MinerU pipeline
scripts/      Schema check, API key, evaluation
tests/        pytest (mock by default)

Tests

pytest -q                    # ~29 tests, mock mode
pytest -m integration        # needs RAG_RUN_INTEGRATION=1 + real credentials

License

Configure your own API keys and infrastructure before public deployment.

---

中文

项目简介

Multimodal RAG API（多模态检索增强问答 API） 面向 图文混排 PDF 知识库，实现「上传 → 离线解析 → 向量检索 → 多模态生成」完整链路。在课程 Streamlit 原型基础上，重构为 FastAPI + Kafka Worker，便于集成与扩展。

适用于：企业知识库、含图表的报告、需要 引用文件名、页码与图片 的智能问答场景。

功能一览

能力	接口 / 模块
文档上传	POST /upload/document
文档列表	GET /upload/documents?knowledge_base_id=
文档删除	DELETE /upload/document/{file_id}（含 processed 目录）
多模态问答	POST /chat
图片访问	GET /files/processed/...
知识库鉴权	X-API-Key；POST /kb/api-keys
离线解析	worker/parse_document_worker.py
批量评测	scripts/run_evaluation.py

检索： BGE 文本向量 + CLIP 文本/图像向量，三路合并排序。
生成： Qwen 文本模型；有图时走 Qwen-VL。
解析： MinerU 离线处理（Kafka 异步）。

系统架构

flowchart LR
  Client[客户端]
  API[FastAPI]
  DB[(SQLite)]
  Kafka[Kafka]
  Worker[解析 Worker]
  MinerU[MinerU]
  Milvus[(Milvus)]
  LLM[DashScope Qwen / VL]

  Client --> API
  API --> DB
  API -->|上传| Kafka
  Kafka --> Worker
  Worker --> MinerU
  Worker --> Milvus
  API -->|问答检索| Milvus
  API -->|问答生成| LLM
  API -->|图片 URL| Client

Loading

流程说明

1. 上传：PDF 落盘，元数据写入 SQLite，并向 Kafka 主题发送解析任务。
2. Worker：消费 Kafka → MinerU 转 Markdown/图片 → 分块 → BGE + CLIP 编码 → 写入 Milvus。
3. 问答：问题向量化 → 混合检索 → 拼装上下文 → Qwen-VL 生成答案，并返回 sources（来源文件、页码、图片链接）。

快速开始（Mock，无需 Kafka / Milvus / GPU）

cd multimodal-rag-api
pip install -r requirements.txt
copy .env.example .env
$env:RAG_MOCK_EXTERNAL="1"
python -m pytest tests -q
uvicorn app.main:app --reload --port 8000

浏览器打开 http://127.0.0.1:8000/docs 查看接口文档。

生产环境部署

1. 复制 .env.example 为 .env，配置 Milvus、DashScope、本地模型路径。
2. 启动 Kafka、MinerU 服务。
3. 执行 python scripts/ensure_milvus_schema.py 检查集合字段。
4. 启动 API：uvicorn app.main:app --host 0.0.0.0 --port 8000
5. 启动 Worker：python -m worker.parse_document_worker
6. 可选：注册 API Key — python scripts/register_api_key.py my_kb your-secret

接口示例

# 上传
curl -F "file=@doc.pdf" -F "knowledge_base_id=demo" http://localhost:8000/upload/document

# 问答
curl -X POST http://localhost:8000/chat -H "Content-Type: application/json" \
  -d '{"question":"产品A销售额何时下降?","knowledge_base_id":"demo","use_clip":true}'

返回的 sources[].image_urls 为可直接访问的 HTTP 地址（经 StaticFiles 挂载）。

评测方式（课程要求）

综合分 = 文件名匹配 (0.25) + 页码匹配 (0.25) + 答案 Jaccard 相似度 × 0.5。

python scripts/run_evaluation.py --dataset tests/fixtures/sample_eval.jsonl

目录结构

app/          路由与服务层
worker/       Kafka 消费者与解析流水线
scripts/      工具脚本
tests/        单元 / 集成测试

测试

python -m pytest tests -q
python -m pytest tests -m integration   # 需设置 RAG_RUN_INTEGRATION=1

说明

生产使用前请自行配置密钥与中间件, 此处删除了.env文件