SEER

Self-grounded Evidence for Entity-Relation Reasoning.

Paper: arXiv:2608.03631 | PDF

This repository contains the code used to build SEER-style spatial-relation datasets, generate relation-blind subject/object boxes with frozen VLMs, render explicit evidence states, score closed-set answer choices, apply the training-free evidence rule, and reproduce analysis tables from cached runs.

SEER is an inference-time evidence interface. It does not train a new vision-language model and does not call a separate external object detector. A frozen VLM grounder first locates the queried subject/object pair with the target relation hidden; by default the grounder is the evaluated VLM itself, while explicit transfer settings may hold a separate VLM grounder fixed. The evaluated frozen model then scores answer candidates under Full and S/O-marker evidence, with high-confidence geometry used only for supported relations. The primary training-free decision selects Geometry when eligible and otherwise compares the Full/Marker score margins. An optional analysis swaps the queried roles and tests inverse-relation consistency. No controller is fit.

Repository Layout

recap/                 Shared data, image, scoring, and SEER utilities
scripts/               Dataset builders, VLM evaluators, evidence policies, and table summaries
tests/                 Lightweight unit tests for core utilities and SEER logic
README.md              Public reproduction entry point
REPRODUCIBILITY.md     Artifact and release checklist
RELEASE_MANIFEST_TEMPLATE.md  Fill-in template for anonymous artifact release
RELEASE_MANIFEST.md           Current paper-specific table-to-artifact map

Generated data, model outputs, raw images, local model checkpoints, paper sources, and internal review notes are intentionally excluded from Git. See .gitignore before publishing a repository snapshot.

Installation

Python 3.10 or newer is recommended. Install the package in editable mode:

python -m pip install -e ".[dev]"

Model evaluation additionally requires the correct CUDA-enabled PyTorch build and the model-specific dependencies used by the selected backbone. For Qwen-VL experiments, install a compatible torch, transformers, accelerate, and qwen-vl-utils stack in the environment where the model weights are available. If you do not install the package, run scripts from the repository root with PYTHONPATH=$PWD.

Run local tests without downloading models:

pytest -q

Before publishing a clean repository snapshot, run the release audit. It checks the Git-visible file set after .gitignore is applied and fails if raw data, model weights, paper sources, generated caches, internal notes, or private local paths are still present:

PYTHONPATH=$PWD python scripts/audit_release_package.py

Main SEER Pipeline

The full pipeline has seven stages. Paths below are examples; adjust them to local dataset and checkpoint locations.

1. Build Target-Complete Relation Data

GQA and Visual Genome spatial-relation pools are built from scene-graph relations with deterministic option shuffling. The paper uses a target-complete protocol: every displayed relation must also occur as a target within that option set. Release the final sample IDs and construction audit so this invariant can be checked independently.

python scripts/build_real_spatial_relation_mc_data.py \
  --source gqa \
  --gqa-root /path/to/gqa \
  --gqa-image-root /path/to/gqa/images \
  --output runs/data/gqa_val_spatial_relation_mc.jsonl \
  --hard-output runs/data/gqa_val_spatial_relation_mc_hard.jsonl \
  --summary-output runs/data/gqa_val_spatial_relation_mc.summary.json \
  --max-per-relation 1500 \
  --seed 23

python scripts/build_real_spatial_relation_mc_data.py \
  --source vg \
  --vg-root /path/to/visual_genome \
  --vg-image-root /path/to/VG_100K \
  --output runs/data/vg_spatial_relation_mc.jsonl \
  --hard-output runs/data/vg_spatial_relation_mc_hard.jsonl \
  --summary-output runs/data/vg_spatial_relation_mc.summary.json \
  --max-per-relation 1500 \
  --seed 23

Convert the base outputs into the target-complete protocol. The builder balances targets within each option family, excludes proximity because no matched reliable far targets exist, and role-swaps half of the inside examples to construct contains. --write-manifest records the minimal metadata needed to reconstruct the exact released split.

python scripts/build_target_complete_relation_mc_data.py \
  --input runs/data/gqa_val_spatial_relation_mc.jsonl \
  --output runs/data/gqa_val_spatial_relation_mc_targetcomplete_v2.jsonl \
  --dataset-tag gqa \
  --seed 2027 \
  --write-manifest artifacts/splits/gqa_targetcomplete_manifest.json

python scripts/build_target_complete_relation_mc_data.py \
  --input runs/data/vg_spatial_relation_mc.jsonl \
  --output runs/data/vg_spatial_relation_mc_targetcomplete_v2.jsonl \
  --dataset-tag vg \
  --seed 2027 \
  --write-manifest artifacts/splits/vg_targetcomplete_manifest.json

Then use deterministic balanced sampling, or the released manifest, to create the 900-example-per-dataset cross-model split:

python scripts/sample_balanced_jsonl.py \
  --input runs/data/gqa_val_spatial_relation_mc_targetcomplete_v2.jsonl \
  --output runs/data/gqa_targetcomplete_v2_balanced100.jsonl \
  --summary-output runs/data/gqa_targetcomplete_v2_balanced100.summary.json \
  --key target_relation \
  --max-per-key 100 \
  --seed 23

python scripts/sample_balanced_jsonl.py \
  --input runs/data/vg_spatial_relation_mc_targetcomplete_v2.jsonl \
  --output runs/data/vg_targetcomplete_v2_balanced100.jsonl \
  --summary-output runs/data/vg_targetcomplete_v2_balanced100.summary.json \
  --key target_relation \
  --max-per-key 100 \
  --seed 23 \
  --require-existing-image

For the frozen nine-relation test, first build the GQA-Train source pool, then exclude every image used by either development pool and sample one relation tuple per image. The freeze command writes the exact IDs, source checksums, timestamp, output SHA256, relation counts, and development-overlap count before any model scoring:

python scripts/freeze_image_disjoint_relation_mc.py \
  --input runs/data/gqa_train_spatial_relation_mc_targetcomplete_v2_seed20270723.jsonl \
  --development runs/data/gqa_val_spatial_relation_mc_targetcomplete_v2.jsonl \
                runs/data/vg_spatial_relation_mc_targetcomplete_v2.jsonl \
  --output runs/data/gqa_train900_image_disjoint_frozen20260723.jsonl \
  --manifest protocol_freeze/gqa_train900_image_disjoint_frozen20260723.json \
  --per-relation 100 \
  --seed 20270723

Set QWEN3I_MODEL, QWEN3T_MODEL, and optionally CONDA_BIN to local checkpoint/environment paths, then run the frozen Full and evidence branches:

bash scripts/run_gqa_train900_frozen.sh qwen3i all
bash scripts/run_gqa_train900_frozen.sh qwen3t all

python scripts/summarize_frozen_gqa_train900.py \
  --model-tag qwen3i \
  --scene-graphs /path/to/gqa/train_sceneGraphs.json

VSR, COCO-Geom, ARO, SugarCrepe, HallusionBench, and GSRBench conversion scripts live in scripts/build_*_data.py and recap/*. The complete VSR random-test protocol additionally uses scripts/build_vsr_reciprocal_inputs.py and scripts/summarize_vsr_relationblind_marker.py; its binary reciprocal query swaps the entities and replaces the relation with its inverse, so the statement truth value and expected yes/no answer remain unchanged.

2. Generate Relation-Blind Self-Grounded Boxes

python scripts/generate_self_grounded_boxes_qwen.py \
  --model /path/to/Qwen3-VL-8B-Instruct \
  --input runs/data/gqa_targetcomplete_v2_balanced100.jsonl \
  --output runs/data/gqa_targetcomplete_v2_balanced100_qwen3i_selfbox_pad070.jsonl \
  --grounding-prompt-mode entity_pair \
  --hide-target-relation \
  --crop-padding 0.70 \
  --id-suffix selfbox_pad070 \
  --device cuda \
  --attn-implementation eager

The grounding prompt names the subject and object but omits the answer set and target relation. Use scripts/revalidate_self_grounded_boxes.py and scripts/sanitize_self_grounded_boxes.py to filter malformed boxes.

3. Render Evidence States

python scripts/make_visual_view_jsonl.py \
  --input runs/data/gqa_targetcomplete_v2_balanced100_qwen3i_selfbox_pad070.jsonl \
  --output runs/data/gqa_targetcomplete_v2_balanced100_qwen3i_marker.jsonl \
  --view crop_boxed \
  --crop-padding 0.70

Related controls include Auto-Crop, text-bbox prompts, random/swap/wrong-pair box controls, and Graph-of-Mark-style views:

scripts/make_text_bbox_prompt_data.py
scripts/make_bbox_control_jsonl.py
scripts/make_graph_of_mark_view_jsonl.py

4. Score Answer Choices

Qwen choice scoring uses mean token negative log-likelihood for candidate continuations, so scores are length-normalized. InternVL and LLaVA variants use the same stored output schema.

python scripts/evaluate_qwen_choice_logprob.py \
  --model /path/to/Qwen3-VL-8B-Instruct \
  --data runs/data/gqa_targetcomplete_v2_balanced100.jsonl \
  --output runs/seer/qwen3i_gqa_targetcomplete_v2_balanced100_full \
  --prompt-style direct \
  --device cuda

Marker-view scoring uses the rendered evidence-state JSONL:

python scripts/evaluate_qwen_choice_logprob.py \
  --model /path/to/Qwen3-VL-8B-Instruct \
  --data runs/data/gqa_targetcomplete_v2_balanced100_qwen3i_selfbox_pad070.jsonl \
  --output runs/seer/qwen3i_gqa_targetcomplete_v2_balanced100_marker \
  --prompt-style marker_direct \
  --device cuda

5. Apply the SEER Evidence Decision

SEER uses a fixed training-free rule: high-confidence eligible geometry takes priority; otherwise the higher-margin Full or S/O-marker prediction is selected. This is the primary method reported in the paper.

python scripts/evaluate_training_free_seer.py \
  --metadata runs/data/gqa_targetcomplete_v2_balanced100.jsonl \
  --selfbox-data runs/data/gqa_targetcomplete_v2_balanced100_qwen3i_selfbox_pad070.jsonl \
  --full-run runs/seer/qwen3i_gqa_targetcomplete_v2_balanced100_full \
  --marker-run runs/seer/qwen3i_gqa_targetcomplete_v2_balanced100_marker \
  --output runs/seer/seer_fwd_qwen3i_gqa.json \
  --tau 0.5 \
  --strip-selfbox-suffix=-selfbox_pad070 \
  --strip-marker-suffix=-selfbox_pad070

6. Optional Reciprocal Analysis

For inverse-closed candidate sets, build role-swapped Full and Marker inputs. Option strings and order stay fixed; only the queried roles and visible S/O marker identities are exchanged.

python scripts/build_reciprocal_targetcomplete.py \
  --metadata runs/data/gqa_targetcomplete_v2_balanced100.jsonl \
  --marker-data runs/data/gqa_targetcomplete_v2_balanced100_qwen3i_selfbox_pad070.jsonl \
  --marker-suffix selfbox_pad070 \
  --forward-predictions runs/seer/seer_fwd_qwen3i_gqa.predictions.jsonl \
  --trigger-only \
  --output-prefix runs/data/reciprocal_qwen3i_gqa

Score the reciprocal JSONL files with the same frozen model and prompt styles used for the forward states, then apply the conservative decision:

python scripts/evaluate_reciprocal_seer.py \
  --forward runs/seer/seer_fwd_qwen3i_gqa.predictions.jsonl \
  --reverse-full-run runs/seer/reciprocal_qwen3i_gqa_reverse_full \
  --reverse-marker-run runs/seer/reciprocal_qwen3i_gqa_reverse_marker \
  --output runs/seer/reciprocal_qwen3i_gqa.json

The optional source change occurs only when exactly one visual state is inverse-consistent. Non-invertible relation families, invalid grounding, and non-discriminating reciprocal checks retain the primary SEER decision. The paper reports this analysis separately because it does not transfer reliably to the frozen SpatialSense binary test.

7. Rebuild Tables and Diagnostics

Most paper tables are rebuilt from cached generations.jsonl, sample_scores.jsonl, and policy prediction files:

python scripts/summarize_reciprocal_balanced.py --preset fullcanonical
python scripts/summarize_reciprocal_balanced.py --preset canonical
python scripts/summarize_targetcomplete_clustered_stats.py
python scripts/summarize_targetcomplete_mechanism_ablation.py
python scripts/summarize_efficiency_cost.py
python scripts/analyze_evidence_reliability.py

Each run directory stores a config.json, aggregate metrics, raw generations, and per-option scores. These files are sufficient for offline paired tests, threshold sweeps, and most table reconstruction without rerunning the VLM.

Blinded Human Audits

Two independent audits separate source-relation validity from predicted grounding quality. Their builders create image cards, two blank annotation sheets, a private aggregation key, and deterministic sampling summaries:

python scripts/build_relation_semantic_human_audit.py
python scripts/build_grounding_human_audit.py

Annotators complete sheets A and B independently without opening manifest_private.jsonl. After both sheets are complete:

python scripts/summarize_relation_semantic_human_audit.py \
  runs/human_audit/semantic_v1
python scripts/summarize_grounding_human_audit.py \
  runs/human_audit/grounding_v1
python scripts/analyze_human_audit_effects.py \
  --grounding-dir runs/human_audit/grounding_v1 \
  --semantic-dir runs/human_audit/semantic_v1 \
  --qwen3i-gqa runs/seer/training_free_qwen3i_gqa_targetcomplete_v2_tau050.predictions.jsonl \
  --qwen3i-vg runs/seer/training_free_qwen3i_vg_targetcomplete_v2_tau050.predictions.jsonl \
  --qwen3t-gqa runs/seer/training_free_qwen3t_gqa_targetcomplete_v2_tau050.predictions.jsonl \
  --qwen3t-vg runs/seer/training_free_qwen3t_vg_targetcomplete_v2_tau050.predictions.jsonl \
  --output runs/human_audit/human_audit_effects.json

The semantics audit asks whether the marked source pair is correct, whether one displayed answer is visually supported, and whether another option is also valid. The grounding audit estimates whether relation-blind predicted boxes select the intended instances, with explicit categories for same-class duplicates, role swaps, and unjudgeable cases. The final command measures SEER on the strict two-annotator semantic-consensus subset and stratifies accuracy by predicted-grounding consensus.

Public Release Scope

The intended public repository should include:

• source code in recap/, scripts/, and tests/;
• deterministic dataset-construction scripts;
• prompt templates embedded in the evaluators;
• table-summary scripts;
• small synthetic or smoke-test fixtures if available;
• instructions for where to place licensed benchmark images and model weights.

It should not include:

• raw benchmark images;
• model checkpoints;
• generated run caches under runs/;
• paper source, compiled PDFs, or submission metadata;
• internal review notes, reviewer simulations, or private experiment logs.

See REPRODUCIBILITY.md for the release checklist. Use RELEASE_MANIFEST.md as the current paper-specific table-to-artifact map. RELEASE_MANIFEST_TEMPLATE.md provides a reusable checklist for future releases with different split files or cached prediction artifacts.

Citation

If you use SEER, please cite:

@article{liu2026seer,
  title   = {SEER: A Self-Grounded Evidence Interface for Controlled Spatial Relation Classification},
  author  = {Liu, Feixiang and Wang, Likun and Qiu, Qiang and Xu, Hui and Shen, Huawei and Cheng, Xueqi},
  journal = {arXiv preprint arXiv:2608.03631},
  year    = {2026},
  doi     = {10.48550/arXiv.2608.03631}
}

Machine-readable citation metadata is available in CITATION.cff.

License

The code is released under the MIT License. Benchmark data, model weights, and third-party baselines remain subject to their respective licenses.