RECAP

Training-free relation evidence calibration for detecting spatial relation hallucinations in vision-language models.

RECAP verifies a model's direct yes/no relation answer by comparing visual support for the claimed relation against a contradiction relation. It supports image-conditioned and text-only probes, confidence-gated selection, offline ablations, risk-coverage curves, bootstrap intervals, and probe-cost analysis.

Supported Setup

• Primary datasets: VSR and What'sUp controlled spatial relations
• External validation: GSR-Bench COCO/GQA two-object splits
• Relation families: left/right, vertical, topology, and depth
• Direct likelihood backends: Qwen2.5/3-VL, InternVL, and LLaVA-HF (including OneVision)
• Evaluation: selective accuracy, hallucination FPR, error AUROC, and hallucination AUROC

Repository Layout

recap/                         Core data, probing, scoring, and analysis package
scripts/run_qwen.sh            End-to-end Qwen3-VL experiment
scripts/run_qwen2b.sh          Qwen3-VL-2B scale-stress wrapper
scripts/run_internvl.sh        End-to-end InternVL experiment
scripts/run_llava.sh           End-to-end LLaVA experiment
scripts/run_prompt_sc.sh       Compute-matched prompt self-consistency baseline
scripts/run_prompt_robustness.sh
                                Fixed-template prompt robustness runs
scripts/build_recap_paper_artifacts.py
                                Rebuild paper tables and plots from run caches
scripts/summarize_answer_conditioned.py
                                Standalone answer-conditioned table builder
tests/test_core.py             Lightweight unit tests

Generated datasets and experiment outputs are intentionally excluded from Git.

Installation

Python 3.10 or newer is recommended.

pip install -e ".[all]"

For analysis without loading a model:

pip install -e ".[data,dev]"

Check the available commands:

python -m recap.cli --help

Quick Start

VSR uses COCO images. Set COCO_ROOT to a local COCO 2017 directory. Model variables can be either Hugging Face IDs or local checkpoint paths.

# Qwen3-VL-8B
COCO_ROOT=/path/to/COCO2017 \
QWEN_PATH=/path/to/Qwen3-VL-8B-Instruct \
LIMIT=20 bash scripts/run_qwen.sh

# Qwen3-VL-2B scale-stress run
COCO_ROOT=/path/to/COCO2017 \
QWEN_PATH=/path/to/Qwen3-VL-2B-Instruct \
LIMIT=20 bash scripts/run_qwen2b.sh

# InternVL
COCO_ROOT=/path/to/COCO2017 \
MODEL_PATH=OpenGVLab/InternVL3_5-8B-HF \
bash scripts/run_internvl.sh

# LLaVA-1.5
COCO_ROOT=/path/to/COCO2017 \
MODEL_PATH=llava-hf/llava-1.5-7b-hf \
bash scripts/run_llava.sh

# LLaVA-OneVision
COCO_ROOT=/path/to/COCO2017 \
MODEL_PATH=llava-hf/llava-onevision-qwen2-7b-ov-hf \
LLAVA_DTYPE=bfloat16 LLAVA_ATTN_IMPLEMENTATION=sdpa \
bash scripts/run_llava.sh

Use RUN_VSR=0 or RUN_WHATSUP=0 to run only one dataset. Set WHATSUP_ROOT=/path/to/whatsup when metadata/images are not under data/whatsup, and set WHATSUP_DATASET=controlled_a or WHATSUP_DATASET=controlled_b to run one controlled split instead of the default all_controlled. Set WHATSUP_DOWNLOAD=1 on the first What'sUp run if the dataset is not already available under WHATSUP_ROOT.

External GSR-Bench Extension

GSR-Bench adds grounding annotations to What'sUp. To avoid duplicating the controlled Subset A/B images used in the primary experiment, RECAP evaluates only COCO-Spatial-Two and GQA-Spatial-Two through the external_two_object group. Models receive RGB images and relation queries; boxes, masks, and depth maps are not passed to the scorer.

Prepare the canonical file from already-downloaded official assets:

python -m recap.cli prepare-gsrbench \
  --dataset external_two_object \
  --root-dir data/whatsup \
  --coco-root /path/to/COCO2017 \
  --relation-families left_right,vertical \
  --output data/recap_gsrbench_external_two_object.jsonl

With --coco-root, --download fetches the COCO-Spatial metadata but reuses the existing val2017 directory instead of downloading val2017.zip. Set GSRBENCH_GQA_ROOT or pass --gqa-root to reuse existing Visual Genome/GQA images as well; otherwise the GQA image archive is downloaded. To run only this optional extension:

RUN_VSR=0 RUN_WHATSUP=0 RUN_GSRBENCH=1 GSRBENCH_DOWNLOAD=1 \
bash scripts/run_internvl.sh

The same GSR-Bench flags work with the Qwen and LLaVA scripts. Results are written under runs/rice_v5/<model>_gsrbench_external_two_object and are kept out of the primary VSR/What'sUp table.

Core Pipeline

The shell scripts execute the same four stages exposed by the CLI:

1. Convert VSR or What'sUp examples to canonical relation records.
2. Build claim, contradiction, support, image, and text-only probes.
3. Score yes and no continuations with a frozen VLM.
4. Aggregate selective prediction and hallucination metrics.

Example offline aggregation:

python -m recap.cli aggregate \
  --scores runs/rice_v5/<run>/probe_scores.jsonl \
  --output runs/rice_v5/<run>/metrics.json \
  --samples-output runs/rice_v5/<run>/sample_scores.jsonl

Offline analyses do not rerun the model:

python -m recap.cli recap-ablation --help
python -m recap.cli recap-curves --help
python -m recap.cli recap-bootstrap-ci --help
python -m recap.cli recap-low-cost-check --help

LLaVA runs checkpoint every completed probe and pass --resume automatically, so rerunning the same model tag skips existing (sample_id, probe) records. The reviewer-facing deployment artifacts use a fixed-reference reliability test rather than model-specific cutoffs:

python scripts/build_problem_mitigation_artifacts.py

The generated principled_policy_multiseed.csv reports 20 group-disjoint splits. A confidence veto is eligible only when Bonferroni-adjusted grouped bootstrap bounds jointly support direct H-FPR below 0.5 and confidence hallucination AUROC above 0.5.

Compute-Matched Prompt Self-Consistency

The fixed Prompt-SC baseline uses the original relation question plus four predefined semantic paraphrases. Each prompt scores both yes and no, for exactly ten likelihood calls per sample. Its risk is the negative of the weakest answer-aligned margin across the five prompts; no prompt or weight is selected on test data.

BACKEND=qwen \
INPUT=data/recap_gsrbench_external_two_object.jsonl \
WORK_DIR=runs/rice_v5/qwen3_8b_prompt_sc_gsrbench_external_two_object \
bash scripts/run_prompt_sc.sh

Use BACKEND=internvl or BACKEND=llava with INTERNVL_PATH or LLAVA_PATH. Prompt-SC requires new inference and cannot be reconstructed from an existing RECAP probe cache.

VCD-Style Visual Contrast Baseline

The VCD-style baseline compares each relation probe under the clean image and a deterministically degraded image (vcd_perturbed). It is a diagnostic visual-contrast baseline, not the original decoding-time VCD algorithm. Because it needs perturbed-image likelihoods, it requires a new model run:

PROBE_MODE=recap_vcd \
MODEL_TAG=qwen3_8b_recap_vcd \
QWEN_PATH=/path/to/Qwen3-VL-8B-Instruct \
bash scripts/run_qwen.sh

When PROBE_MODE=recap_vcd, the run scripts automatically add vcd_visual_contrast and vcd_selector to metrics_compact.json. Override this with COMPACT_RISKS=... if a different compact report is needed.

Then include the baseline in offline reports:

python -m recap.cli recap-ablation \
  --scores runs/rice_v5/qwen3_8b_recap_vcd_vsr_other_relations/probe_scores.jsonl \
  --output runs/rice_v5/qwen3_8b_recap_vcd_vsr_other_relations/recap_ablation_vcd.json \
  --risks confidence,vcd_visual_contrast,vcd_selector,recap_evidence,rice_recap_selector

vcd_claim_contrast uses only the claim relation; vcd_visual_contrast compares claim contrast against contradiction contrast; vcd_selector combines confidence and VCD-style contrast by rank, matching the RECAP-Selector protocol.

Constraint-Aware Answer-Change Baseline

constraint_aware adapts bidirectional and combined bidirectional/transitive spatial prompting to selective prediction. It freezes the original yes/no answer and assigns high rejection risk when the constrained prompt gives that answer weak likelihood support. Binary answer flips are also exported as diagnostics, but the continuous answer-conditioned support is the headline risk.

USE_TF=0 TRANSFORMERS_NO_TF=1 \
PROBE_MODE=constraint_aware \
MODEL_TAG=qwen3_8b_constraint \
QWEN_PATH=/path/to/Qwen3-VL-8B-Instruct \
bash scripts/run_qwen.sh

The run contains orig, constraint_bidirectional, and constraint_combined probes. The Qwen runner resumes completed sample/probe pairs, and all model runners automatically include the two continuous risks and their flip diagnostics in metrics_compact.json.

Use the standard grouped bootstrap interface for marginal intervals:

python -m recap.cli recap-bootstrap-ci \
  --scores runs/rice_v5/qwen3_8b_constraint_vsr_other_relations/probe_scores.jsonl \
  --output runs/rice_v5/qwen3_8b_constraint_vsr_other_relations/constraint_bootstrap.json \
  --risks confidence,constraint_bidirectional,constraint_bidirectional_flip,constraint_combined,constraint_combined_flip \
  --n-bootstrap 1000 --seed 13

For a paired comparison with an aligned RECAP cache, including answer-flip bias and cost diagnostics:

python -m scripts.analyze_constraint_aware \
  --constraint-scores runs/rice_v5/qwen3_8b_constraint_vsr_other_relations/probe_scores.jsonl \
  --recap-scores runs/rice_v5/qwen3_8b_recap_vcd_vsr_other_relations/probe_scores.jsonl \
  --dataset VSR \
  --output runs/rice_v5/qwen3_8b_constraint_vsr_other_relations/constraint_paired_analysis.json \
  --csv-output runs/rice_v5/qwen3_8b_constraint_vsr_other_relations/constraint_paired_gains.csv

Answer-Conditioned RECAP

Answer-conditioned RECAP treats an anti-relation as an optional contradictory witness rather than the exhaustive meaning of not claim. For a predicted yes, it keeps the full claim--anti--support evidence margin. For a predicted no, it uses the strongest of three sufficient witnesses: missing claim support, positive anti-relation evidence, or failure of a necessary support relation. The experiment is computed entirely from an existing RECAP cache:

python -m recap.cli recap-ablation \
  --scores runs/rice_v5/qwen3_8b_recap_vsr_other_relations/probe_scores.jsonl \
  --output runs/rice_v5/qwen3_8b_recap_vsr_other_relations/answer_conditioned_ablation.json \
  --risks confidence,recap_evidence,recap_no_claim_only,recap_answer_conditioned,selector_answer_conditioned

The report always includes by_direct_answer, which separates predictions of yes and no. recap_no_claim_only is a diagnostic showing why discarding contradictory witnesses entirely is too conservative.

Prompt Robustness

Prompt robustness uses exactly four pre-specified templates (canonical, image, careful, and visible_only). It reports the mean and population standard deviation over all templates and every pairwise Spearman correlation between sample risk rankings. No best-template result is reported.

BACKEND=qwen \
INPUT=data/rice_vsr_random_test_other_relations.jsonl \
WORK_DIR_ROOT=runs/rice_v5/qwen3_8b_prompt_robustness_vsr \
CANONICAL_RUN_DIR=runs/rice_v5/qwen3_8b_recap_vsr_other_relations \
QWEN_PATH=/path/to/Qwen3-VL-8B-Instruct \
bash scripts/run_prompt_robustness.sh

This experiment requires new inference for three templates when CANONICAL_RUN_DIR points to an existing standard RECAP cache; the canonical template is then re-aggregated offline. Use SKIP_EXISTING=1 (the default) to resume an interrupted run. The final files are prompt_robustness.json and prompt_robustness.csv inside WORK_DIR_ROOT.

Paper tables and plots can be rebuilt from completed runs with:

python scripts/build_recap_paper_artifacts.py \
  --runs-root runs/rice_v5 \
  --output-dir runs/rice_v5/paper_artifacts

PYTHONPATH=. python scripts/summarize_core_mechanism.py \
  --runs-root runs/rice_v5 \
  --output-dir runs/rice_v5/paper_artifacts

PYTHONPATH=. python scripts/build_image_disjoint_gsr_artifacts.py \
  --runs-root runs/rice_v5 \
  --output-dir runs/rice_v5/image_disjoint_gsr_artifacts

The second command reconstructs the image-only claim, symmetric-pair, answer-conditioned, support, and confidence-veto risks from cached per-relation margins. It writes the six-setting Hall-AUC mechanism matrix and the threshold-free AURC summary used by the paper. The third command compares source-qualified image IDs across VSR, controlled What'sUp, and GSR-Bench, removes target images shared with either primary benchmark, and recomputes both target-ranked and frozen-threshold transfer metrics from cached probe scores.

When the full VCD-style run matrix exists, the main table automatically inserts VCD-Contrast and VCD-Selector rows after Confidence. The artifact script looks for the conventional MODEL_TAG=<base>_recap_vcd run directories, for example qwen3_8b_recap_vcd_vsr_other_relations and qwen3_8b_recap_vcd_whatsup_controlled_other_relations, with the same naming pattern for InternVL and LLaVA. If the matrix is incomplete, VCD rows are omitted and the standard RECAP table is still written.

If both qwen3_2b_recap_vsr_other_relations and qwen3_2b_recap_whatsup_controlled_other_relations exist, the artifact script also writes an optional scale-stress table: runs/rice_v5/paper_artifacts/qwen_scale_table.tex.

Tests

pytest -q

License

The code in this repository is released under the MIT License. External datasets, model weights, and third-party dependencies remain subject to their respective licenses and terms of use.

Reproducibility Notes

• The main method is fixed across models and datasets; diagnostic variants are not selected per test set.
• Probe scores are sufficient for all offline ablations and plots.
• Low-cost variants are verified against full-cache sample risks before their reduced call counts are reported.
• runs/, datasets, checkpoints, and paper working files are ignored by default and should be distributed separately when appropriate.