Add baddies-smp-pigs-can-fly-command-ledger (agentic omni understanding)

[wenkai-li]

Add baddies-smp-pigs-can-fly-command-ledger (agentic omni understanding)

Reconstruct the ledger of spoken requests in a 228-minute multiplayer building session
that were not acted on when asked, but were carried out later. For each one the
agent reports the request, who eventually did it, what happened, and the timestamps and
evidence window of the execution. Scoring is deterministic event-level F1 with
one-to-one bipartite matching (steps/solve/tests/judge.py); no LLM judge.

The reasoning is cross-modal by construction: the request exists only in the voice
chat, the fulfilment exists only on screen, and the two are separated by 20 s to 12 min
of unrelated activity. The image ships an offline ASR tool because text+vision
harnesses cannot hear — without it they would fail for lack of a sensor rather than for
lack of reasoning.

Calibration — this task does NOT clear the difficulty bar

I am opening this with the failing number rather than filing it away, because the
reason it fails turned out to be the most useful thing the build produced.

harness	version	model	reasoning	score	turns
Claude Code	2.1.220	Opus 5	xhigh	0.1875	115
Claude Code	2.1.220	Opus 4.8	default	0.1379	51
Codex CLI	0.145.0	GPT-5.6 Sol	xhigh	0.0741	112

scripts/understanding/check_task.py passes 9 of its 10 checks; the one failure is
strong agent < 0.1. Oracle 1.0, empty 0.0, > 50 turns for all three, and every
ablation scores exactly 0.0000 against a <= 0.15 gate.

The ablations are the part that worked

evidence given	reward	detection F1	what it could do
video only (audio stripped)	0.0000	0.00	could not even detect that a request occurred
audio only (no video)	0.0000	0.30	found 4 real requests, grounded none of them
one frame / 20 frames / no media	0.0000	0.00	submitted nothing
both channels (Codex)	0.0741	0.37	finds requests and begins to ground them
both channels (Opus 5)	0.1875	0.44	—

Detection F1 is unscored credit for merely locating the right utterance. The two
single-modality runs both score zero but fail differently, and that asymmetry is the
real evidence: speech carries the existence of the event, video carries its temporal
anchor, and neither half completes the chain. Audio-only located four genuine requests
and then died on exec_time accuracy 0.50 and evidence-window IoU 0.25 — the transcript
tells you who asked for what, never which second it got done.

(In an earlier 15-row / 180-min configuration audio-only scored 0.0952, nearly matching
the full-evidence agents. On the shipped ledger it is a clean zero, with no restriction
of the ground truth needed to make the cross-modal claim hold.)

Two repairs were measured and both failed

Enlarging the ledger does not work. Event F1 is 2·TP/(P+N); when predictions and
ground truth are the same order of magnitude the reward tracks the agent's solve rate,
not the ledger size. Opus 5 solves 3 of 21 rows with 11 predictions, so reaching 0.10
needs N > 49. I extended the window from 90 to 228 minutes to try exactly this: the
ledger grew 15 → 21 rows and Opus 5's score went up, 0.1053 → 0.1875, because the
extra footage handed it more solvable rows rather than diluting the ones it had.

Raising the deferral floor does not work either. The design assumed longer gaps are
harder. They are not — the filter removes ground-truth rows and agent true positives at
the same rate:

deferral floor	GT rows	Opus 5	Opus 4.8	Codex
15 s	18	0.2069	0.0769	0.0000
45 s	12	0.0870	0.1000	0.0000
90 s	10	0.0952	0.1111	0.0000
180 s	6	0.1176	0.1429	0.0000

No floor clears the gate for both Claude agents. Opus 5's three solved rows have gaps of
36 s, 20 s and 549 s — it handles the nine-minute deferral as comfortably as the
twenty-second one.

What the difficulty axis actually is

Field accuracy among located rows tells the story: object, executor and outcome
are 1.00 for every agent on every row they found. The loss is entirely in when —
execution timestamp and evidence-window IoU. Detection F1 (locating the right utterance,
unscored) is 0.37–0.44, roughly triple the official score.

So these agents hear what was asked and know who did it; they cannot say which second it
was done. And what separates a row they solve from one they miss is whether the
execution is visually conspicuous — a pig landing, a block placed into an empty
frame — not how long after the request it happened. That is the axis footage should be
selected on, and it is not the axis this task was built around.

Known gaps

• Difficulty gate fails (above). Everything else passes.
• The image has never been built end-to-end. No Docker on the build machine. Every
  remote fetch verified HTTP 200 and the media sha256 matches the remote LFS oid, but a
  real docker build is still required.
• Ground truth has one human reviewer, the contributor, not the 2+ independent
  annotators the family asks for. It was drafted by two vision models independently
  (Cohen's κ 0.916 on actionability), adjudicated from dense frame sampling, and every
  row cites the frames that settled it.
• Two harnesses, not three. No Antigravity/Gemini access; Opus 4.8 stands in.

Provenance

• A prompt leak was found — an earlier instruction.md used a real ground-truth row as
  its worked example and one agent copied it verbatim. Every affected run was
  discarded and re-run, not re-scored; the example is now fictional and flagged.
• A phantom third player (an on-screen tooltip for a custom-named pickaxe, read as a
  nameplate) entered the vocabulary and cost one run a 0.0. Caught, verified from
  cropped frames, removed.
• Contamination audit on the shipped trajectories: 0 of 278 tool calls across the three
  agents touch any answer-key path.
• Rows were never added to move a number; a sweep run specifically to enlarge the ledger
  had most of its candidates rejected on the evidence.

Full numbers and method: calibration/scores.md. Design card: SPEC.md. Build history
and rejected source alternatives: RUNBOOK.md.

[christine1729]

Hi @wenkai-li, thanks for the PR.

before merge:

• Complete the scores.md table by running the shipped task on the shipped 1-hour budget with exactly these rows, each with harness version, reasoning effort, score, tool-call turns, and one raw trajectory file committed in calibration/rollouts/: Codex CLI + GPT 5.6 Sol (done, keep it and add its turn count), Claude Code CLI + Fable 5 or Opus 4.8, Antigravity CLI + Gemini 3.5 Flash or 3.1 Pro, and for the Gemini row add the integrity audit line like the other tasks: confirm from the trajectory that there was no web grounding, no lookups, and no answer-file access. Move the reward and solution dumps out of the repo; the HF links can stay as extras, but the in-repo trajectory is the audit record.

[wenkai-li]

Thanks @christine1729 — all four items addressed.

scores.md table now carries the required rows, each with harness version, reasoning effort, score, tool-call turns, and a committed raw trajectory in calibration/rollouts/:

harness	version	model	reasoning	score	turns	trajectory
Codex CLI	0.145.0	GPT-5.6 Sol	xhigh	0.0741	112	codex-full.jsonl
Claude Code	2.1.220	Opus 4.8	default	0.1379	51	opus48-full.jsonl
Antigravity CLI	1.1.21	Gemini 3.5 Flash	high	0.0	98	antigravity-full.jsonl

(Opus 5 / xhigh / 0.1875 / 115 kept as a fourth row.)

• Codex turn count added (112).
• Antigravity + Gemini 3.5 Flash row added, with the integrity-audit line: the trajectory shows no web grounding, no lookup/search tool calls, no server-side grounding metadata, and no access to any answer-key path (98 tool-call turns, all clean — checked by scripts/understanding/audit_trajectory.py). This row scored 0.0; the full run provenance is in scores.md → "The Antigravity row" so the number is interpretable.
• Solution dumps moved out of the repo: the ground truth is now inlined into judge.py and the oracle answer into solve.sh (matching dualrunway/gsw/soccer); ground_truth.json and oracle_solution.json are removed. Oracle still scores 1.0, empty 0.0. The raw trajectories stay in-repo as the audit record.
• Agent budget set to 3600 s (1 h) to match SPEC.md and the calibration table.

Also rebased onto latest main to clear the conflict. Happy to adjust anything.

[techgenmini]

Hi @wenkai-li — thanks for completing the documentation and artifact portion of the previous checklist. I re-reviewed live head 9b0c4de.

I verified that the requested model/version/reasoning/turn-count rows and raw trajectories are present, the one-hour timeout is configured, and the loose solution/reward dumps were removed. Those parts of the
earlier checklist are closed.

The raw trajectories make the measurements auditable, but they also reveal that the calibration portion is not closed yet. I recommend moving the PR back to Draft for one bounded correction cycle.

1. The retained scores are not clean measurements of the shipped task. Codex starts with a populated workspace containing prior transcripts, diarization, frames and scripts; Opus 4.8 reads and merges a prior-
   session solution.json; and Opus 5 uses prior artifacts and reads output written by a concurrent run. The Antigravity run quota-terminated before analysis and produced no solution, so its 0.0 is the empty
   anchor rather than a completed one-hour run. These remain useful development records, but the current scores establish neither a pass nor a failure of the difficulty gate.

2. The new trajectory auditor fails open for the retained Codex and Claude formats. It reports zero calls and PASS for those three files, although their actual counts are 112, 51 and 115. Please add support
   for Codex item.completed command/file-change records and Claude assistant.message.content[].tool_use, and fail nonzero when a non-empty trajectory uses an unsupported format or yields no recognized calls.

3. The scorer has a concrete numeric bypass and an inconsistent output contract. I reproduced 1.0 after replacing every command and execution timestamp with JSON NaN; non-finite numeric values need to be
   rejected and covered by a regression test. I also reproduced 1.0 with every speaker and target wrong, although the prompt requires those fields and the judge documentation describes them as part of a true
   positive. Please either remove them as required fields or establish observable definitions and score them consistently. The prompt should also state the enforced evidence threshold (temporal IoU >= 0.5).

Before rerunning, please successfully build and freeze the exact image, including immutable media/ASR revisions and checksums, and remove the unnecessary HF_TOKEN exposure.

The next checkpoint should be just one clean Codex GPT-5.6 Sol pilot: fresh container and empty workspace, only shipped setup materials, one-hour budget, task networking disabled, and a native audited
trajectory. If it scores <0.10, freeze the task and proceed with the final Claude/Antigravity campaign, required ablations and second human review. If it scores >=0.10, that supports a substantive task/source
pivot rather than scorer tightening.

GitHub currently also reports a merge conflict with main; that can be resolved after this measurement checkpoint. I reproduced the existing oracle 1.0, the NaN and speaker/target cases, and the auditor’s
false zero-call passes. I did not run the full 3.09 GB Docker build because the scorer contract and environment still need the corrections above.