Retrieval-Augmented Medical QA: RDF Knowledge Graphs and Claim-Level Verification for Faithful Answer Generation

RDF-grounded RAG pipeline with LLM-powered SPARQL generation and answer synthesis. Supports both English and Spanish medical Q&A.

Features

• 🤖 LLM-powered SPARQL generation - AI generates queries from natural language
• 🌐 Multilingual support - Works with Spanish and English
• 📊 RDF knowledge graph - Structured medical knowledge with SPARQL querying
• 🔄 Flexible backends - OpenAI API or HiPerGator open models
• 🎯 Grounded generation - Answers cite sources from knowledge graph

Prerequisites

• UV: https://docs.astral.sh/uv/getting-started/installation/
• Python 3.10+
• HiPerGator access (recommended - free open-source models) OR OpenAI API key (requires payment)

Quick Start

1. Sync submodules

   git submodule update --init --remote

2. Install dependencies

   uv sync
   python -m spacy download en_core_web_sm
   python -m spacy download es_core_news_sm

3. Configure LLM backend

   Copy the example environment file:

   cp .env.example .env

   Option A: HiPerGator (Recommended - Free!)

   Follow the HiPerGator Quick Start Guide (step-by-step instructions).

   TL;DR: Deploy Mixtral on HiPerGator, create SSH tunnel, .env already configured!

   Option B: OpenAI (Requires API Key)

   Edit .env:

   LLM_PROVIDER=openai
   OPENAI_API_KEY=your-api-key-here
   LLM_MODEL=gpt-4-turbo-preview

4. Ingest Spanish dataset (optional, if you want to use Spanish medical Q&A)

   uv run python ingest_spanish_qa.py data/dev_HEAD_converted.jsonl

5. Test the pipeline

   uv run python test_spanish_pipeline.py

Usage

Spanish Medical Q&A

from database.database import Database
from pipeline_02_retrieval.pipeline import run_pipeline

# Initialize database
db = Database()

# Ask a question in Spanish
result = run_pipeline(
    db=db.rdf,
    text="¿Qué es la diabetes?",
    use_llm_query=True,   # Use LLM to generate SPARQL
    use_llm_answer=True,  # Use LLM to generate answer
    lang="es"             # Spanish language
)

print(result.grounded_answer)
print(f"Sources: {len(result.sources)}")

English Medical Q&A

result = run_pipeline(
    db=db.rdf,
    text="What are the symptoms of hypertension?",
    use_llm_query=True,
    use_llm_answer=True,
    lang="en"
)

Rule-Based Mode (No LLM)

result = run_pipeline(
    db=db.rdf,
    text="What is diabetes?",
    use_llm_query=False,  # Use rule-based SPARQL generation
    use_llm_answer=False, # Use simple concatenation
    lang="en"
)

HiPerGator Setup

For production use with free open-source models, see HIPERGATOR_SETUP.md.

Recommended models:

• Mixtral 8x7B - Excellent for Spanish medical QA
• Llama 3.1 70B - Strong reasoning, multilingual
• BioMistral 7B - Biomedical specialist

Project Structure

betterai-project/
├── common/
│   ├── llm.py              # LLM abstraction layer
│   ├── tokenize.py          # NLP tokenization
│   └── get_source.py        # Source extraction
├── database/
│   ├── rdf/                 # RDF graph storage
│   └── redis/               # Redis caching
├── pipeline_02_retrieval/
│   ├── pipeline.py          # Main retrieval pipeline
│   ├── llm_query_generator.py  # LLM SPARQL generation
│   ├── generation.py        # LLM answer generation
│   └── schemas/             # Data schemas
├── evaluation/              # Evaluation metrics
├── hipergator/              # HiPerGator SLURM scripts
├── data/                    # Datasets (Spanish/English)
└── tests/                   # Unit tests

Helpful Commands

Add new dependencies:

uv add package-name

Run tests:

uv run pytest

Ingest Spanish Q&A dataset:

uv run python ingest_spanish_qa.py data/dev_HEAD_converted.jsonl

Test LLM client only:

uv run python test_spanish_pipeline.py --mode llm

Evaluation

Run a simple baseline on a PubMedQA-style JSONL:

uv run evaluation/run_eval.py --dataset path/to/dev.jsonl --output runs/pqa_eval.jsonl --model heuristic --with_retrieval

JSONL format (fields per line):

• id: sample id
• question: text
• answer or final_decision: yes/no (case-insensitive)
• context: optional text

The runner computes Accuracy and Macro-F1, and can optionally attach retrieval summaries from the current KG.

Compute additional metrics on the predictions JSONL:

python -m evaluation.compute_metrics --pred runs/pqa_eval.jsonl --k 5 \
  --gold-entities-key gold_mesh --gold-pmids-key gold_pmids

Heuristic claim judging for faithfulness (optional):

python -m evaluation.judge_claims --input runs/pqa_eval.jsonl --output runs/pqa_eval_judged.jsonl --answer-key pred
python -m evaluation.compute_metrics --pred runs/pqa_eval_judged.jsonl --k 5

Metrics Overview

• QA quality
  • Accuracy: fraction of correct yes/no answers.
  • Macro‑F1: average F1 over the "yes" and "no" classes (robust to imbalance).
• Retrieval quality
  • Coverage: fraction of samples with non‑empty retrieval_sources.
  • Precision@k / Recall@k: correctness and completeness among top‑k retrieved items.
  • MRR: how early the first correct item appears (higher is better).
  • NDCG: rank quality rewarding correct items near the top.
  • Note: to compute P@k/R@k/MRR/NDCG, include gold lists in your predictions JSONL, e.g. gold_mesh (MeSH IDs) and/or gold_pmids (PMIDs). Pass their keys via --gold-entities-key / --gold-pmids-key.
• Faithfulness (hallucination proxy)
  • Heuristic judge splits the answer into sentence‑like claims and checks word‑overlap against source titles/contents.
  • Outputs per‑claim verdicts: Supported or NEI (Not‑Enough‑Info).
  • Aggregates:
    • Hallucination Rate = (Contradicted + NEI) ÷ total claims (in heuristic, we only produce NEI/Supported).
    • Factual Precision = Supported ÷ total claims.

Typical Flow

1. Run evaluation to produce predictions JSONL (optionally with --with_retrieval).
2. Compute QA + retrieval metrics with evaluation.compute_metrics.
3. (Optional) Run evaluation.judge_claims to annotate claims, then re‑run evaluation.compute_metrics to include faithfulness scores.

Datasets Used

Below are the datasets used for evaluation and benchmarking hallucination reduction in RDF-grounded LMs.

PubMedQA - Link

---

Notes

• pubmedqa.jsonl → Main evaluation benchmark (factual accuracy, hallucination rate). - Biomedical yes/no QA dataset with verified human labels (1k samples).
• pubmedqa_artificial.jsonl → Optional training/calibration dataset. - Automatically labeled synthetic QA pairs (211k). Used for model calibration or pre-training.
• pubmedqa_unlabeled.jsonl → Retrieval stress-testing for RDF graph coverage. - Questions and contexts without gold labels (61k). Used for retrieval evaluation.
• medqa.jsonl → Complex reasoning benchmark (clinical multi-choice).
• medhalt.jsonl → Hallucination stress test for medical text generation.

---