run_gap_tests.sh is red on pristine main and its snapshot has gone stale — the gate reports 9 false regressions

[proggeramlug]

Summary

scripts/run_gap_tests.sh reports 591 PASS / 14 FAIL on pristine b3f14e9cde, with 9 of those flagged as "regressions". They are not regressions, and the gate is currently non-functional as a signal for everyone.

Proof that the 9 are false

All 9 were compiled with a base compiler and a patched compiler in the same directory, and their outputs compared:

9/9 byte-identical output, identical exit codes.

Identical bytes cannot be one pass and one fail. So the failures are not produced by any change under test — the committed gap_snapshot.json has gone stale against the current expected outputs, and the suite is reporting the drift as regressions.

Why this matters more than 14 red tests

A gate that is red on arrival is CLAUDE.md hazard 2. Worse, this one is red in a way that manufactures attribution: it names 9 specific tests as "regressions" of whatever branch happens to be running, so the natural reading is that your change broke them. Anyone who trusts it wastes an afternoon; anyone who learns not to trust it stops reading the whole 603-test suite, which is the real cost — this is the suite that covers the compiled-output parity fixtures.

There is precedent for exactly this failure shape in this repo: #797's parity skip-list, where entries that had stopped being true cost nothing to keep and quietly went stale, and #9108, where an abort masked ~1,000 runtime tests. The pattern is the same — a signal that is wrong in a direction nobody audits.

What is wanted

1. Re-derive gap_snapshot.json against current main so the suite is green on arrival.
2. Make the drift self-invalidating, the way SUITE_EXCLUSIONS in scripts/ci_e2e_scope.py already is: an entry that stops being true should fail the job with instructions to delete it, rather than sitting there silently. A snapshot that can only rot in one direction will rot again.
3. If any of the 14 are genuine expected-failures, record each with its issue number rather than as an unexplained snapshot entry.

Found by

A registry-probe performance change (#9272) that ran the suite as a pre-merge check, saw 9 "regressions", and disproved every one by byte-comparing base against patched output rather than trusting the label.

[proggeramlug]

Five of the nine "false regressions" now have a concrete root cause, found during unrelated verification (#9315): missing npm dependencies, not snapshot drift.

backoff_options, cron_cronjob, dayjs_factory_arg, moment_methods, ratelimiter_memory are gap tests comparing perry's native package emulation against the real package under node. The packages are declared in package.json, but neither run_gap_tests.sh nor run_parity_tests.sh ever provisions node_modules — they silently assume a prior npm ci. On a fresh checkout the node oracle exits with an ERR_MODULE_NOT_FOUND stack trace, and the harness scores the mismatch against the branch under test.

Verified both directions: all five were byte-identical between base and patched binaries (so no branch caused them), and after npm ci all five pass inside the gate with the same binaries.

Two fixes wanted, the second mattering more:

1. Provision the oracle's dependencies (an npm ci --prefer-offline in the harness, or a preflight check).
2. The gate must fail loudly on a missing oracle dependency rather than scoring it. An oracle that cannot run is not a divergence — scoring it as one is how this issue's "regressions" manufactured attribution against whatever branch happened to be running. A one-line check that the node run didn't exit with ERR_MODULE_NOT_FOUND (or that the oracle produced any expected-shape output) converts a silent false-blame into an actionable setup error.

The remaining 4 of the 9 may still be genuine snapshot drift — the npm finding covers exactly five.