Visual Credit Audit

Official implementation and reproducibility materials for Visual Credit Audit for Multimodal Spatial Reasoning. This repository contains the VCA source code, data indices, and analysis-ready cached scores used in the paper. It excludes model weights, full licensed source datasets, raw annotation packages, and deprecated experiments.

Layout

• code/recap: probe construction, model scoring, and audit metrics.
• code/scripts: scripts used by the paper's current audit and robustness analyses.
• code/tests: lightweight unit tests.
• data_indices: public-dataset sample indices; source images remain under their original licenses.
• cached_scores: path-sanitized sample-level scores and aggregate summaries.
• controlled_validation: known-source manifests, probes, cached scores, and analysis outputs; deterministic images are regenerated by the packaged builder.
• evidence_factorial: the complete 3-by-3 visual--textual factorial manifests, probes, four-model scores, and analysis outputs.
• object_neutral_control: matched same-object relation-neutral manifests, four-model scores, and cluster-bootstrap analysis outputs.
• audited_edits: edited images and consensus labels for the object-swap audit.
• human_annotations: deidentified per-annotator object-swap labels, including the blinded within-annotator repeats.

Install with pip install -e code and run pytest code/tests. Model inference requires local copies of the public checkpoints and datasets. The cached score files allow the paper's audit statistics, prior-metric comparison, and held-out benchmark filtering experiment to be recomputed without redistributing those external assets. The filtering directory includes its four path-sanitized input score bundles and the exact group-level outputs. Full VSR and GSR-COCO Yes/No caches, True/False caches, and ensemble cases support replay of the surface-form analysis. The same cases reproduce the label-free surface-consensus coverage, abstention, and covered-item audit rates. The pair-centered GSR and VSR case files reproduce the PairCredit answer-interface stability analysis without packaging additional raw scores. The empirical-null summary and its replay script use those same Yes/No and True/False caches; no duplicate derived case file is included.

Controlled Evidence-Source Factorial

The complete 3-by-3 factorial reported in the paper is packaged under evidence_factorial. Each visual and textual source independently supports the query, supports its opposite, or supplies no queried relation. The directory contains the exact sample and probe manifests, four-model sample-level scores, and the reported aggregate analysis. To keep the repository compact, deterministically generated image copies are omitted; the packaged renderer, seed, and manifests reconstruct them exactly. Rebuild the inputs with:

python code/scripts/build_vca_evidence_factorial.py   --output-dir reproduced/evidence_factorial

Object-Preserving Relation-Neutral Control

The matched control under object_neutral_control preserves the factorial background, object identities, shapes, and sizes while aligning object centers on the queried axis. Its deterministic images are omitted and can be rebuilt from the packaged manifests and renderer:

python code/scripts/build_vca_object_neutral_control.py   --factorial-dir reproduced/evidence_factorial   --output-dir reproduced/object_neutral_control

Inference Protocol

Original and blank contexts provide one image object; text-only provides no image object and retains no visual placeholder. Within each backend, all three contexts use the same question, checkpoint-native chat template, assistant generation prefix, and space-prefixed yes/no verbalizers. The scoring code masks prompt and BOS tokens and averages loss only over candidate continuation tokens. Blank images preserve the source dimensions before native processor transforms. Cached rows retain the sanitized checkpoint identifier, exact question, verbalizers, continuation losses, and margins needed to audit this protocol.

Relation Interaction

The image-by-relation analysis is fully cache-replayable:

python code/scripts/analyze_vca_relation_interaction.py   --model-score Qwen=cached_scores/vca_shortcut_filtering_visnec_seed20260722/input_scores/qwen.jsonl   --model-score InternVL=cached_scores/vca_shortcut_filtering_visnec_seed20260722/input_scores/internvl.jsonl   --model-score LLaVA=cached_scores/vca_shortcut_filtering_visnec_seed20260722/input_scores/llava.jsonl   --model-score Ministral=cached_scores/vca_shortcut_filtering_visnec_seed20260722/input_scores/ministral.jsonl   --output-dir reproduced/relation_interaction

This reconstructs all 440 relation pairs per model, the original/text-only/blank relation gaps, the image-by-relation difference-in-differences, and 5,000-pair bootstrap intervals. The script also verifies that every reported Joint case passes the added interaction condition.

The independent Joint failure audit can then be replayed from the packaged source decisions and accepted edited-image outcomes:

python code/scripts/analyze_vca_joint_failure_audit.py   --swap-cases cached_scores/raw_gap_swap_4model_108audit_seed20260723/cases.jsonl   --joint-cases cached_scores/vca_relation_interaction_seed20260722/relation_interaction_cases.jsonl   --output-dir reproduced/joint_failure_audit

This replay first separates edited relation-pair consistency after the human visibility audit from the stricter target that additionally requires the model's two object-presence responses. It also reports the component-isolated comparison of Rel_o against Rel_o+PairD and its leave-one-edit-out increment beyond confidence, model, and relation controls.

The leave-one-edit-out incremental analysis uses only the headline original/text-only/blank VCA gap and can be replayed with:

python code/scripts/analyze_vca_swap_incremental_grouped.py   --cases cached_scores/raw_gap_swap_4model_108audit_seed20260723/cases.jsonl   --output-dir reproduced/swap_incremental
python code/scripts/analyze_vca_swap_incremental_sensitivity.py   --cases cached_scores/raw_gap_swap_4model_108audit_seed20260723/cases.jsonl   --output-dir reproduced/swap_incremental_sensitivity

Empirical Null Calibration

The selector, same-split image-permutation, and cross-interface diagnostics can be replayed directly from the packaged margins. Pass the four model--dataset score pairs to code/scripts/analyze_vca_empirical_null_calibration.py with repeated --run MODEL DATASET YES_NO_SCORES TRUE_FALSE_SCORES arguments. The packaged result is under cached_scores/vca_empirical_null_calibration_seed20260725; its summary.json records all input paths, definitions, 5,000-resample intervals, and the eight per-model estimates.

Reference Environment

Qwen, InternVL, and LLaVA scoring used Python 3.13.5, PyTorch 2.6.0+cu124, and Transformers 4.57.3. Ministral used Python 3.13.5, the same PyTorch build, and Transformers 5.15.0.dev0 for checkpoint support. Experiments ran on NVIDIA RTX A6000 48GB GPUs with native processors and bfloat16 weights; Ministral used its checkpoint-native automatic dtype. Qwen fixed both min_pixels and max_pixels to 50,176.

Checkpoint copies are fingerprinted by SHA256 of config.json and model.safetensors.index.json, respectively:

• Qwen3-VL-8B-Instruct: 5cd452860dc1e9c29dd71cc3cef7f39b338b7a40793f7a260655c2d3568f3661, 520b2e05079402e9468a8701d03d1154d14b2599593afb6effa7fb60c1bff070.
• InternVL3.5-8B-HF: 32a9a726c8fd4d2386038bb1eab9fd34c6e04cf89c070e44aab346a41d9454e6, b15fa5f358beeadf2901d180db961be52ec0c1a0f17fcdb064e4e9c304f77d1b.
• LLaVA-OneVision-Qwen2-7B-OV-HF: 69277c2c9ba8a4f61d2e72f79fbe02e043b8c3af670858a606818527f971a0c2, e389b969d3b9b7120f136fbf6592cbbc1c07326157cae6f863fb64559261dae9.
• Ministral-3-14B-Instruct-2512: fc51295bdc0b39c015c9c3ca5d649de183eae73a23b8cbab1a0e055be8d5fa04, 2b03cf0cf68dcb05010643e80470e52466db5c38e1361f9ca831e3d981108882.

MANIFEST.sha256 records every packaged file except the manifest itself.

Citation

@article{liu2026visual,
  title={Visual Credit Audit for Multimodal Spatial Reasoning},
  author={Liu, Feixiang and Qiu, Qiang and Sun, Lanbo and Wei, Nan and Shen, Huawei and Cheng, Xueqi},
  journal={arXiv preprint arXiv:2607.27069},
  year={2026}
}