Official implementation of Witness Evidence Portfolios: Single-Prefill Risk Detection for Closed Multimodal Answers.
Paper: arXiv:2607.27667 | PDF
WEP ranks the error risk of a multimodal model's current closed-answer prediction. It extracts question-bound visual witnesses and signed decision-evidence routes from the same instrumented prefill that produces the candidate logits, then fuses a sparse validation-selected route portfolio with candidate-margin confidence. WEP requires no crop, input perturbation, backward pass, decoding modification, or external verifier.
This repository contains the implementation, tests, and JSON/JSONL benchmark probes and image indexes. Raw images, model checkpoints, and generated experiment outputs are not redistributed.
Repository: https://github.com/SouthWinter/WEP
Released under the Apache License 2.0.
- Single-prefill risk scoring for finite candidate sets.
- Qwen3-VL, LLaVA-1.5, and InternVL3.5 backends.
- Provenance and concentration route families with grouped portfolio selection.
- Error AP, AUROC, AURC, review-budget, transfer, null, localization, and intervention analyses.
- Multi-token candidate prototype and protocol-matched internal baselines.
code/recap/: model hooks, witness/evidence extraction, risk portfolios, evaluation, localization, matched-layer patching, and WEP-Span.code/scripts/: primary experiment launchers and the CPU smoke test.data/: benchmark probe files and image indexes in JSON/JSONL form.tests/: focused unit tests for the main and supplementary mechanisms.
Image fields use public-dataset-relative or placeholder paths. Download the corresponding benchmark images separately and rewrite their roots locally.
conda create -n wep python=3.13 -y
conda activate wep
pip install -r requirements.txt
export PYTHONPATH=$PWD/code:$PYTHONPATH
export PYTHONDONTWRITEBYTECODE=1Full scoring requires a CUDA GPU and locally accessible Qwen3-VL-8B, LLaVA-1.5-7B, or InternVL3.5-8B weights. Offline portfolio analysis and unit tests do not require model checkpoints.
The reported runs were validated with Python 3.13.9, PyTorch 2.6.0 + CUDA
12.4, Transformers 4.57.1, and one NVIDIA A800 80GB GPU. The exact direct
dependencies are pinned in requirements.txt. Model identifiers are
Qwen/Qwen3-VL-8B-Instruct, llava-hf/llava-1.5-7b-hf, and
OpenGVLab/InternVL3_5-8B-HF; use a fixed local snapshot for reproducible
scoring.
python code/scripts/smoke_test_wep_portfolio.py \
--features data/smoke/wep_smoke_features.jsonl \
--output /tmp/wep_smoke.jsonThe test exercises held-out route selection, evidence/confidence fusion, and risk metrics on eight synthetic feature rows.
After resolving image paths in a probe JSONL:
python -m recap.qwdc_entity_cli score-qwen \
--probes data/amber/amber_d_probes.jsonl \
--output /tmp/amber_wep_features.jsonl \
--pretrained /path/to/Qwen3-VL-8B-Instruct \
--candidates yes,no --probe-name orig \
--layer-start-fraction 0.6666667 \
--feature-only --attn-implementation sdpa
python -m recap.qwdc_evidence_ensemble_portfolio_cli run \
--features /tmp/amber_wep_features.jsonl \
--output /tmp/amber_wep_report.json \
--top-k-rule select --folds 5 --seed 0 --include-scoresThe launchers in code/scripts/ cover the main Qwen benchmarks. Model and
image roots are environment variables or explicit placeholders; no private
server paths are embedded.
Multi-token candidates with one shared visual prefill:
python -m recap.qwdc_span_cli score-qwen \
--probes data/grounded_vsr_random_test.jsonl \
--output /tmp/wep_span_scores.jsonl \
--pretrained /path/to/Qwen3-VL-8B-Instruct
python -m recap.qwdc_span_cli analyze \
--scores /tmp/wep_span_scores.jsonl \
--output /tmp/wep_span_report.json --folds 5 --seed 0Cross-backbone witness localization and matched-layer value patching are launched with:
bash code/scripts/run_wep_cross_backbone_localization.sh
bash code/scripts/run_wep_cross_backbone_matched_patching.shSet QWEN_PATH, LLAVA_PATH, INTERNVL_PATH, PROBES, and image-root fields
for the local installation. The scripts support LIMIT for smoke-scale runs.
run_wep_matched_budget_hidden.sh can cache fixed projected hidden states and
layer-wise candidate margins from the same instrumented prefill. The offline
drivers then evaluate candidate-adapted Overthinking trajectories, all-layer
decision-state and explicit image--question--decision MLPs, projection-width
and five-seed robustness, an unprojected full-state linear probe, direct
sparse-route learners, label-permutation nulls,
and zero-target-label leave-dataset-out transfer:
bash code/scripts/run_wep_matched_budget_hidden.sh
QWEN_LLAVA_LIMIT=2000 INTERNVL_LIMIT=300 \
bash code/scripts/run_wep_multimodal_selector.sh
python code/scripts/build_wep_overthinking_baselines.py --help
python code/scripts/build_wep_hidden_mlp_transfer.py --help
python code/scripts/build_wep_hidden_mlp_robustness.py --help
python code/scripts/build_wep_multimodal_selector.py --help
python code/scripts/build_wep_full_state_probe.py --help
python code/scripts/build_wep_halp_baseline.py --help
python code/scripts/build_wep_matched_baseline_bootstrap.py --help
python code/scripts/build_wep_transfer_bootstrap.py --helpThe scripts expect the compact WEP reports named in their command-line
arguments; they do not require additional image paths after feature caching.
build_wep_halp_baseline.py reproduces the protocol-matched HALP-QT control
with its published 512--256--128 probe, fixed normalized build_wep_full_state_probe.py; the large
cache itself is intentionally excluded from this artifact.
The two *bootstrap.py drivers reproduce the paired image-cluster intervals
for WEP versus the matched hidden MLP and for zero-target-label transfer.
The same cached route features support grouped portfolio, route-reselection
null, and sparse-route controls. Attach an image_group field directly or pass
the original probe JSONL as sparse-baseline metadata:
python -m recap.qwdc_evidence_ensemble_portfolio_cli run \
--features /tmp/amber_wep_features.jsonl --output /tmp/wep_seed0.json \
--top-k-rule select --folds 5 --seed 0 --fold-key image_group
python -m recap.qwdc_evidence_ensemble_portfolio_cli sanity \
--features /tmp/amber_wep_features.jsonl --output /tmp/wep_label_null.json \
--top-k-rule select --folds 5 --seed 7 --fold-key image_group \
--modes label_shuffle --repeats 50 --reselect-null-routes
python -m recap.qwdc_sparse_route_baselines_cli \
--features /tmp/amber_wep_features.jsonl \
--metadata data/amber/amber_d_probes.jsonl \
--output /tmp/wep_sparse_routes.json --fold-key image_groupThe final controller selects between signed provenance and concentration families. The corresponding offline implementation and the question-residual removal audit are:
python code/scripts/build_wep_concentration_controls.py --help
python code/scripts/audit_wep_no_question_residual.py --help
python code/scripts/audit_vsr_question_only_spans.py \
--probes data/grounded_vsr_random_test.jsonl \
--output-dir /tmp/wep_vsr_span_auditThe VSR audit parses entities from the visible question only; it does not read captions, relation annotations, boxes, or target labels.
The scoring CLIs support --readout-fidelity-only, which keeps only sample
identifiers, candidate logits, and the three primitive per-layer tensors needed
to audit the source-token readout. It does not store full attention or token
maps. For example:
PYTHONPATH=code python -m recap.qwdc_entity_cli score-qwen \
--probes data/amber/amber_d_probes.jsonl \
--output /tmp/qwen_readout_fidelity.jsonl \
--pretrained /path/to/Qwen3-VL-8B-Instruct \
--candidates yes,no --probe-name orig --layer-start-fraction 0 \
--attn-implementation sdpa --readout-fidelity-only --limit 64
PYTHONPATH=code python code/scripts/audit_wep_readout_fidelity.py \
--scores /tmp/qwen_readout_fidelity.jsonl \
--model-name Qwen3-VL-8B \
--output /tmp/qwen_readout_fidelity_report.jsonThe same compact flag is available on score-llava-qwdc and
score-internvl-qwdc in recap.cli. The audit compares source-token closure
with the bias-free attention update and compares its linear readout with the
exact same-sublayer candidate-margin change.
Compact full-vocabulary uncertainty statistics are extracted from the same candidate-decision prefill. Only candidate logits and scalar summaries are stored, not full vocabulary vectors:
QWEN_PATH=/path/to/Qwen3-VL-8B-Instruct \
LLAVA_PATH=/path/to/llava-1.5-7b-hf \
INTERNVL_PATH=/path/to/InternVL3_5-8B-HF \
LIMIT=2000 bash code/scripts/run_wep_vocab_baselines.shBatch-one candidate-margin versus instrumented-WEP latency and incremental peak allocated GPU memory are measured with:
QWEN_PATH=/path/to/Qwen3-VL-8B-Instruct \
LLAVA_PATH=/path/to/llava-1.5-7b-hf \
INTERNVL_PATH=/path/to/InternVL3_5-8B-HF \
PROBES=/path/to/amber64_probes.jsonl \
bash code/scripts/run_wep_latency_memory.shThese scripts write generated reports outside the artifact by default when run from the repository root.
python -m pytest -qThe CPU unit suite does not require model checkpoints. GPU scoring tests are performed through the launchers above with locally installed checkpoints and benchmark images.
Please cite the accompanying paper if this repository is useful in your work:
@article{liu2026witness,
title = {Witness Evidence Portfolios: Single-Prefill Risk Detection for Closed Multimodal Answers},
author = {Liu, Feixiang and Wang, Shiye and Qiu, Qiang and Wang, Zheng},
journal = {arXiv preprint arXiv:2607.27667},
eprint = {2607.27667},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2607.27667},
year = {2026}
}Dataset annotations and indexes remain governed by their upstream licenses; see DATA_LICENSES.md. Model checkpoints must be obtained from their respective publishers.
The WEP source code is licensed under the Apache License 2.0. Dataset annotations, image indexes, and model assets remain subject to their respective upstream terms; see DATA_LICENSES.md.