Super Bowl LI referee-announced player-foul timeline task

[YashJayswal24]

Task: Super Bowl LI referee-announced player-foul timeline

Proposed and discussed in #60. Adds tasks/agentic_vbench_understanding/ne-atl-2017-super-bowl-li-penalty-timeline/.

Opened for design/code review — NOT yet calibration-complete, so it should not be merged until the calibration lands. Shared early per the proposal-first process, so the design, scorer, and ground truth can be reviewed before the (expensive) calibration stage.

What the task is

Reconstruct every referee-announced player foul from the full Super Bowl LI broadcast: quarter, game clock, infraction type, penalised player's jersey number, team. Cross-modal by construction — the jersey number is stated only by the referee's mic (audio), the game clock only by the on-screen score bug (video); neither channel alone yields a true positive. Ground truth = official NFL Game Book (machine-truth). Media pinned from archive.org (pin, not re-host), matching the gsw-cle worked example.

Verified so far (locally)

• Media: youtube-noLK78Hgq0A.mp4, 1920×1080, 2h23m; SHA256 pinned in the Dockerfile (ba2281d6…4ba998, computed by streaming the file).
• Scorer (judge.py): deterministic compound-match F1. Local pilot: oracle → 1.0, empty → 0.0, plausible guess → 0.0, partial (clock off 3 s) → 0.57. Clock tolerance and wrong-number rejection behave as designed.
• no_media gate (maintainer's step 1): indicative proxy run — a no-media, no-tools model reconstructed 0/13 penalties (F1 0.0). De-risks the fame concern; the official GPT 5.6 Sol run is still required.
• Audio intelligibility (design's single point of failure): the referee's announced jersey number transcribed cleanly on a small CPU ASR model for 4/4 sampled player fouls — No. 70 Matthews (Q3), No. 23 Alford (Q2), No. 34 Poole (Q2), No. 59 Campbell (OT PI).

Now complete (update)

• Full ground truth — DONE. Replaced the 4-foul pilot with 13 referee-announced player fouls parsed from the official NFL Game Book play-by-play; jersey numbers taken from the Game Book lineups (the 4 audio-verified numbers all match). judge.py GROUND_TRUTH and the oracle solve.sh are in lockstep — locally oracle 13/13 → 1.0, empty → 0.0, partial → 0.83. Scope rule + the 3 no-number exclusions (delay of game, illegal formation, illegal touching) documented in PROVENANCE.md, per @christine1729's guidance in [Task Proposal] Super Bowl LI referee-announced player-foul timeline #60.

• check_task.py — PASSES (partial). Ran the repo's own scripts/understanding/check_task.py: PASS on task structure, input video (143.1 min, 1080p — resolution now confirmed by its ffprobe, not assumed), oracle == 1.0, baseline == 0.0. The agent-reward / agent-turns / ablation inputs are reported SKIP (they need the strong-agent stack).

Still pending (needs the agent/Harbor stack — same dependency as calibration)

• Calibration. GPT 5.6 Sol / Codex / Antigravity rollouts + performance table + raw trajectories under calibration/. scores.md marks these _to run_.
• Ablations. single_frame, video_only, audio_only, frame_dump_no_tools — each is a strong-agent run on degraded media, so it needs the same stack as calibration.

Open design questions on the ground truth (see PROVENANCE.md)

• Declined fouls (3 of the 13) are included because the referee announces the number; accepted-only would give 10. Your call.
• Event count is 13 vs 22 in the NBA example; widening to replay reviews / measurements would need a schema addition, so it is left open rather than silently included.

Review asks

1. Is the scope rule (referee-announced player fouls; team fouls excluded because their fields are OCR-recoverable) the right boundary?
2. With type + team conceded to OCR, is the compound-match-on-number+clock cross-modal argument sufficient, or do you want a stricter formulation?
3. Anything you'd want changed in the task layout / scorer before the calibration spend?

Closes #60 once calibration lands (not with this draft).

[techgenmini]

Thanks for opening this pre-calibration. Please respond to the scope/clock audit request on #60 before spending on calibration. I’ve also left inline comments for the leaked examples and clock contract.

[YashJayswal24]

Pushed fixes for the two concrete bugs (leaked GT examples in instruction.md, allow_internet) plus the restored clock rule — commit 1bb2aaf. Full scope/clock/audibility audit posted on #60: #60 (comment)

[techgenmini]

Hi @YashJayswal24 — thanks, this audit is enough to settle the task shape before calibration.

• Keep the natural 13 referee-announced player-foul scope. Please do not pad it with reviews or measurements: they lack the audio/video jersey-number seam that makes this task valuable. A declined foul
  stays only if its number is individually audible.

• Raw F1 cannot remain: one exact row currently scores 0.1429. Please use recall-weighted F2 instead of an arbitrary two-hit cliff. It keeps continuous partial credit, gives one exact row 0.0943, two exact
  rows 0.1852, and oracle 1.0. Add null / one-hit / two-hit / oracle regressions.

• Complete a compact observability audit for all 13 rows before runs: the spoken-number timestamp and transcript, plus the last pre-announcement score-bug frame, displayed clock, Game Book clock, and delta.
  Drop any row that fails. Pin the exact official Game Book PDF URL, SHA256, and row references. This is best described as official-record-derived GT plus a broadcast audio/video audit.

• Two small prompt fixes: the prose example “Holding, number 70, offense” is still a real GT pattern; make it generic. Also replace “clock at the moment the foul occurred” with “clock defined below,” since
  the binding rule is the pre-announcement score-bug frame.

• The spoken number is load-bearing, but the image currently ships only ffmpeg while the evidence used local Whisper. Either bake/pin the intended ASR path, or demonstrate the permitted exact-image setup in
  the final traces. Keep allow_internet=true per the existing ruling; raw traces must distinguish dependency/model downloads from prohibited source lookups.

Once those land, run final-prompt no-media and modality ablations plus the required Codex CLI, Claude Code, and Antigravity traces. No n=3 campaign is needed unless a final result is near the gate.

[YashJayswal24]

Pushed the rest of the punch list from the latest review — F1→F2 in judge.py (with checked regressions), the remaining prose leak ("number 70" example), and faster-whisper baked into the Dockerfile (build + offline transcription verified locally with --network none). Commit d7e5d1d. Details on #60: #60 (comment)

[YashJayswal24]

@techgenmini can you check it out

[techgenmini]

Thanks — I checked d7e5d1d. The F2 implementation and expected regression values are coherent, the remaining prompt leak and clock wording are fixed, and the Dockerfile now includes the intended faster-
whisper setup. This was a static review; I did not independently rerun the Docker build or offline transcription.

I also see from your #60 update that the exact Game Book pin/references, full 13-row observability audit—including the three declined fouls—and final-prompt calibration are still in progress. Because that
audit may change the retained event set and scorer, I’ll treat this as a good progress checkpoint rather than a merge-ready re-review and wait for the completed batch.

One small reproducibility item to fold in before final runs: small.en is currently cached without an exact Hugging Face revision. Please pin its snapshot, or freeze and record the exact final image digest
used for calibration.