Add task: flightgear-atc-clearance-compliance-ledger

[JordanPeng]

Closes #95.

Summary

Adds flightgear-atc-clearance-compliance-ledger to agentic_vbench_understanding. The task reconstructs 65 ATC clearances and their visible aircraft-response chains across five independent 12-minute FlightGear C172P legs.

Data and scoring

• Pinned 60-minute, 1280x720 H.264/AAC media.
• Ground truth derived from command logs, controller traces, and FlightGear telemetry.
• Each row includes command, target, issue/execution/completion/end state, timing, status, supersession, progress, and overshoot.
• Deterministic reward: 0.9 * exact_leg_accuracy + 0.1 * clearance_chain_f1.
• Oracle 1.0; null 0.0.

Calibration

model	harness	score	tool calls
GPT-5.6 Sol	Codex CLI 0.147.0	0.000000	88
Claude Opus 4.8	VS Code Claude Agent SDK	0.010800	119
Gemini 3.6 Flash High	Antigravity CLI 1.1.12	0.000000	89

All required rows are below 0.10. All five degraded-input runs remain near the null baseline.

Validation

• scripts/understanding/check_task.py passes structure, media, score, and rollout-length checks.
• Harbor oracle: 1.000, 0 exceptions.
• 17 verifier and adversarial tests pass.
• Deterministic judge: oracle 1.0, null 0.0.
• Retained JSON and JSONL files parse successfully.

[techgenmini]

Hi @JordanPeng, thanks for the PR, ATC clearance compliance from sim footage is an interesting one! Generated FlightGear media on HF is fine with me; treat that as the #95 sign-off. The calibration hygiene here is genuinely good (paired events, coherent timestamps, all five ablations measured, network-isolated runs) — the remaining work is the scorer and one harness-parity problem, in priority order:

1. Observability first (feasibility gate)

• Show the tolerances are readable before more runs. Both non-Claude agents scored exactly 0 on full 65-entry answers, and audio-only (0.0031, 2/65 strict) beat every video-inclusive run — the signature of 25 ft / 2 deg / 2 kt snapshots being unrecoverable from 720p15 analog gauges, not of weak models. Extend the observability audit from 19 to all 65 events (or a demonstrated partial solve), or relax the state tolerances until an honest partial solve scores.

2. Then the scorer

• Rebalance the reward: 90% of it sits in exact_leg_accuracy, which pays 0.18 only when all ~13 clearances in a leg strict-match — effectively oracle-only, so real agents compete over the 0.1-weighted chain F1 alone (your Claude row's 0.0108 is exactly 0.1 x F1 0.108). Move to graded per-clearance credit: match on command_type + time window, then graded credit on the field groups — same shape as the partial-credit scheme in PR Add task: byu-wsu-2023-volleyball-block-timeline (#54) #59/Add RoboCup 2024 possession-chain ledger task #67. Re-verify null/guessing ~0 and the difficulty gate after the change; right now the gate passes because the metric can barely leave zero, which is not the same as the task being appropriately hard.
• Fix the leg-assignment boundary: the judge buckets by predicted issued_time_s // 720, so a prediction 1.5 s early on a clearance issued just after a cut — within your own +/-2 s issue tolerance — lands in the wrong leg and matches nothing. Match within the time tolerance instead of floor-bucketing by predicted time.

3. Harness parity

• Re-run the Claude row natively with claude -p on the shipped instruction only. The rollout's copilotUsage/AHP metadata shows Opus 4.8 ran through a VS Code Copilot agent-host, and its wrapper prompt adds solving guidance the other agents never saw ("validate the closed vocabularies", "ensure all expected chronological clearances are represented") — so the one nonzero row is also the one coached row. Credit where due: the wrapper did enforce --network none and a frozen image, and scores.md declares the harness honestly; parity, not isolation, is the issue.

Mechanical

• Retain the ablation trajectories per the rollout policy (release asset, LFS, or hash-pinned dataset revision), not just the numbers.
• Publish the replaced/failed-round trajectories the same way, or a one-line note per round on what changed; "outside the PR under the managed research Run" isn't reviewable.

[JordanPeng]

Maintainer feedback, item by item

1. Observability audit — all 65 events

calibration/observability/ now audits 65 events / 235 snapshots (was 19), and adds two arguments that need no video at all.

• measure_observability.py — 25/2/2 is satisfiable on 5 of 65 events; 100/8/3 on 62 of 65. There is a cliff between them and a plateau from 75/6/3 onward. 100/8/3 is chosen over 75/6/3 because it is the capture band instruction.md already uses to define completion, so the scorer and the task definition use one number instead of two.
• anchoring_bias.py (ground truth only) — over the 50 clearances that reach a stable hold, heading and airspeed land exactly on the commanded value (sd 0.000, n=20 and n=15), while altitude is systematically −27.14 ft ± 7.95 (range −42.91..−18.68). Nothing in the video reveals that offset, so a 25 ft budget was irreducibly unmeetable on the third instrument.
• tolerance_consistency.py (ground truth only) — the state tolerance and the timing tolerance contradicted each other on 47 of 65 clearances at 25/2/2.

Rather than widen bands to paper over that, each snapshot is now compared against the trajectory interpolated to the timestamp the answer itself reports. An exact timestamp still gets the strict band; a late-but-honest answer is read where it claims to be and pays for the shift out of the timing group instead of twice. Verified in the script: +1 s coherent shift → 260/260 state units, reward 1.0000; +2 s → 244/260, 0.5987; +4 s → 216/260, 0.5203. Under the old scorer all three scored 0.

Two caveats are stated in the audit README rather than glossed: telemetry.npz is the Opus 4.8 rollout's own reader, not an independent one (so section 1 is corroboration, and the reader-independent sections 2 and 3 are the basis), and the 3 residual events are a saturation artefact of that reader, not a measured property of the video.

2. Scorer — graded per-clearance credit

0.9 * exact_leg_accuracy + 0.1 * clearance_chain_f1 is gone. A prediction matches an expected clearance on command_type plus an issue time inside the 2 s window, aligned order-preservingly, then earns credit out of 20 units: target 4, status 4, instrument snapshots 4, timing 4, supersession chain 2 (which also requires the right overshoot_bucket), progress 2. Timing is graded — full within 1 s, half within 4 s.

Each clearance is discounted by what a transcript-only ledger could have earned on that same clearance (953 of 1300 units, leaving 60 of 65 gradable), the signed gain is clipped to [−1, 1] so a below-transcript reading subtracts rather than clipping to zero, and unmatched submissions are charged as padding.

The inversion the comment flagged is resolved. Audio-only went 0.0031 → 0.0000, and it no longer beats anything. It still earns 252/260 target units from the spoken numbers, but only 86/260 instrument-snapshot units — and those are exactly the values the transcript predicts, so they clear no headroom.

Leg-boundary fix. Matching is global on (command_type, |Δt| ≤ 2 s); there is no issued_time_s // 720 bucketing anywhere in the match path. Leg attribution is reported per leg for diagnostics only. test_prediction_early_across_a_cut_still_matches builds exactly the case in the comment — ground truth at 720.4, prediction at 718.9, i.e. 718.9 // 720 == 0 vs 720.4 // 720 == 1 — and it matches with reward 1.0. (The shipped ground truth has no clearance within 6 s of a cut, so the event is re-timed in the test on purpose; the mirrored late case is covered separately.)

Null / guessing / gate, re-verified:

	reward
oracle	1.0000
null / empty	0.0000
transcript-only anchor (constructed upper bound)	0.0000
no media / one frame / video only / audio only / frame dump	0.0000 (all five)

Adversarial probing beyond the required ablations is now a committed, reproducible script — python3 calibration/shortcut_probe.py runs 29,100 video-free probes (12 structural: all-null fields, padding to the 2000-entry limit, truncation, duplicating the highest-ceiling clearance 65×, keeping only zero-headroom clearances; a 27,648-point systematic-offset grid; 1,440 randomized jitter ledgers) and exits non-zero if any clears AGENT_MAX. All 29,100 return 0.0000.

scripts/understanding/check_task.py passes on all five checks, agent reward 0.0938 against AGENT_MAX = 0.10.

3. Harness parity — Claude re-run natively

The Opus 4.8 row is now a native claude -p rollout on the shipped instruction.md byte for byte (sha256 a970d16d…11e4, 7407 B) — no --append-system-prompt, no wrapper preamble, no copilotUsage/AHP path. Container has --network none; the model endpoint is reachable only through one bind-mounted unix socket. Launcher and isolation are recorded in calibration/rollouts/claude-opus-4.8-native.md.

Result: claude exited 0 after 147 tool calls, reward 0.0000. It wrote a bare JSON array instead of {"clearances": [...]}, which the judge rejects; hand-wrapping it as a diagnostic still gives 0.0000, with 8/260 state units and the first target misheard ("two niner three" → 93).

Fixing this surfaced a second parity gap I had not reported before: the Codex and Antigravity rows still run an earlier instruction.md. Codex cannot be re-run here (codex exec → 401, no credentials in the image); the Antigravity re-run was built and then abandoned because that harness's own sandbox-bash/bwrap isolation fails open in this container — curl https://example.com returned 200 under all three postures tried — and publishing a row with a weaker network posture than the one it replaces seemed worse than disclosing the gap. Exact hashes, the one-hunk diff (0963eb1a…c3e1 6887 B → a970d16d…11e4 7407 B, a single @@ inside ## Clearance fields), and the reason per row are in calibration/rollouts/README.md. The deviation runs against those rows: they were told to hit a band four times tighter than the one they are now scored against.

4. Mechanical — trajectories retained

All five ablation trajectories and both replaced rounds are hash-pinned release assets: https://github.com/JordanPeng/agentic-vbench/releases/tag/flightgear-calibration-20260827 — public, sha256s listed in calibration/ablations/README.md (13,085,971 B as shipped, 31,160,204 B uncompressed). The three required-harness trajectories stay in the repo under calibration/rollouts/, each with its submitted ledger, so python3 calibration/rescore_ledgers.py reproduces every number in calibration/scores.md from the repository alone with no replay and no network.

The native Claude trajectory is the one exception, and it uses the same mechanism. Raw, claude -p --include-partial-messages writes 13,395 lines / 15,343,222 B — 10× the repo's current largest blob. That raw file is now an eighth release asset (sha256 819b9585…8990), and the committed copy is condensed to 660 lines / 1,232,159 B by calibration/rollouts/condense_trajectory.py, which is in the PR so the transform is reproducible:

python3 calibration/rollouts/condense_trajectory.py FULL.jsonl out.jsonl
cmp out.jsonl calibration/rollouts/claude-opus-4.8-native.jsonl

It drops 12,735 stream_event records and 84 base64 image payloads. Neither is content: the deltas open 328 content blocks and the assembled assistant records carry the same 328 (102 thinking / 79 text / 147 tool_use) under the same 147 message ids, and ttft_ms — the one field living only on the delta stream — is copied onto the assembled record. All 147 tool calls, 147 tool results, and the final result record survive verbatim; each image becomes a placeholder carrying that payload's length and sha256.

ledgers/video-only.json is labelled honestly as the one reconstruction rather than the agent's own bytes, and why.

Independent review pass

An independent reviewer checked the branch against your comment and the repo's authoring rules before this update. Its one substantive finding is fixed: check_event accepted an arbitrarily large clearance_index / superseded_by_index, so an index like 10**400 survived the isinstance(..., int) test and then raised OverflowError from float() deeper in — outside the function's ValueError contract, which zeroed the entire submission with n_dropped_invalid: 0. Both indices are now bounded by MAX_PREDICTIONS, so such an entry costs one clearance instead of the answer: the reviewer's repro went from reward 0.0 / "int too large to convert to float" / dropped 0 to reward 0.9672 / ok / dropped 1, with the boundary checked at 2000 (accepted) and 2001 (dropped). Two regression tests cover it; the suite is 47 → 49.

The reviewer also independently re-verified the leg-boundary fix at every 720 s cut in both directions, confirmed all release assets download hash-identical, found no leaked host paths in the native trajectory, and confirmed instruction.md does not describe the scoring method. Its remaining findings were documentation accuracy — a wrong hunk count, a wrong asset size, a release body that said "one replaced round" while shipping two, and the trajectory size issue in item 4 above — all fixed here.

Things I did not resolve, flagged rather than buried

1. The gate passes partly because the strongest reader was removed. The replaced agent-host round scores 0.6060 on the same judge — 6× AGENT_MAX — on 242/260 state units. It is retained in the release so that number is checkable. Nothing was widened or narrowed to move a row across a threshold; Codex moved 0.1506 → 0.0938 from graded timing and interpolation, not a tolerance change. If a future native run reads gauges that well unprompted, this task exceeds the gate and should be re-examined, not re-tuned.
2. A residual pruning incentive. Because a below-ceiling clearance subtracts, an agent that could identify its own weak readings could score higher by withholding them: Codex 0.0938 → 0.3291 over 45 retained, replaced Claude 0.6060 → 0.6594 over 60. rescore_ledgers.py reports this per row. The instruction does not describe the scoring method, so this is not something an agent is told — but it is a real property of the metric.
3. Codex/Antigravity instruction parity, above.
4. check_task.py internal contradiction — its module docstring says "strong agent reward < 0.50" while the constant it executes is AGENT_MAX = 0.10. Both readings pass for the three required rows and both fail for the replaced round, so nothing here turns on it, but it is worth settling.
5. The release is hosted on the contributor fork, since that is where the branch lives. All assets are hash-pinned so content is verified rather than trusted, but if you would rather they live upstream or in LFS, say the word — the hashes will not change.

Verification

python3 -m unittest discover -s steps/solve/tests -p 'test_judge_unit.py'   # 49 tests, OK
python3 calibration/rescore_ledgers.py
python3 calibration/shortcut_probe.py                                      # 29,100 probes, max 0.0000
python3 calibration/observability/measure_observability.py --telemetry calibration/observability/telemetry.npz
python3 calibration/observability/anchoring_bias.py
python3 calibration/observability/tolerance_consistency.py
python3 scripts/understanding/check_task.py --task-dir tasks/.../flightgear-atc-clearance-compliance-ledger \
    --oracle-reward oracle.json --baseline-reward null.json --agent-reward agent.json --agent-turns 88

[techgenmini]

Hi @JordanPeng, thanks for the unusually thorough update. The observability audit, global time matching, native Claude rerun, and retained calibration artifacts address the original review concerns.

To make the maintainer calls explicit:

• The hash-pinned release assets may remain on the contributor release; no upstream/LFS migration is requested.

• Please do not attempt an unsafe Antigravity rerun or a Codex rerun without valid credentials. Keep those rows clearly labelled as earlier-instruction evidence; we will assess their retained outputs after
  the scorer correction.

• The replaced wrapped-Claude 0.606 row is useful solvability evidence, but is not a compliant calibration-gate row.

• The executed AGENT_MAX = 0.10 is the applicable contract; aligning the docstring is a small cleanup, not a new calibration request.

One scorer issue remains before we can rely on the gate: rescore_ledgers.py

gains = { ... for ... in pairs }
  credit = max(0, sum(gains.values()))
  chargeable = len(gradable) + spurious

shows that pruning weak rows raises the retained Codex answer from 0.0938 to 0.3291. Missing clearances avoid their negative gain, so a legal partial answer can outscore the complete one. Please add a coverage rule so omissions cannot improve the score, add a regression using that retained-row pruning case, and re-score the existing full-task and ablation artifacts. No fresh model campaign is requested unless those re-scored results cross a gate.