release blocker: `cc --help` segfaults on main (NULL deref in run_microtasks) — clean at 6c880be77b, broken at 42d0f45685 and still broken at tip

[proggeramlug]

claude --help on the natively compiled cli_2.1.112.js bundle segfaults deterministically on current main, with no GC knobs, no environment, and no flags. --version works; --help and -p crash.

This is not #9261 (that needs PERRY_GC_SCHEDULE_SEED/RATE; --help with no knobs is explicitly node-identical there). It is not #8770 (that was non-deterministic agent-loop init).

Repro

PERRY_NO_CACHE=1 PERRY_RUNTIME_DIR=<...>/target/release \
  perry compile --no-auto-optimize --enable-wasm-runtime cli_2.1.112.js -o app
HOME=/tmp/cc_home ./app --help

rc=139 (SIGSEGV)
app[...]: segfault at fffffffffffc8250 ip ... error 5

PERRY_DEBUG_SYMBOLS=1, x86-64 Linux (perrymaster). The compile itself is clean — CC_RC=0, only the usual deferred-API warnings.

Good/bad, six binaries, one box, one toolchain

Every arm below is the same host, same nightly, same cli_2.1.112.js, same compile command. Three consecutive runs each.

binary	source	--help
A	6c880be77b + #9272's content	rc=0, byte-identical to node (9,175 B)
B	A + an unrelated runtime perf change	rc=0, byte-identical
C	A + the same change, final form	rc=0, byte-identical
D	pristine 42d0f45685	139 139 139
E	D + that same unrelated change	139 139 139
F	pristine d20fb4fd25 (main tip)	139 139 139

So the regression is in 6c880be77b..42d0f45685, excluding #9272 (which arm A already carries), and it is still present on main tip. The five non-test, non-CI commits in that window are #9282, #9257, #9254-phase-1, #9263, #9274, #9279.

Where it crashes

perry_runtime::promise::microtasks::run_microtasks, at the same file offset every run and across ASLR:

c6fc808:  mov    0x38(%rsp),%rax
c6fc814:  movzbl -0x37db0(%rax),%eax     <-- faults, %rax == 0
c6fc81b:  test   %eax,%eax
c6fc81d:  jne    ...

Fault address 0xfffffffffffc8250 is exactly 0 - 0x37db0, i.e. a NULL base with a large negative displacement — the shape of a thread-local access whose base pointer is null. bt is #0 run_microtasks #1 main.

The slot holds a live pointer, then holds NULL — same frame

gdb (ASLR off), breakpoints on the store and the reload of that spill slot:

STORE   rax=0x7ffff7e97bc0  rsp=0x7fffffffe780     (run_microtasks+0xa35)
RELOAD  slot=(nil)          rsp=0x7fffffffe780     (run_microtasks+0xc58)
Program received signal SIGSEGV

Same %rsp, so the same frame and the same slot: it held a valid pointer and reads back as NULL. Between the two there is a call setjmp and an indirect call *%r12 into runtime/generated code.

Caveat, stated rather than glossed: objdump finds six writers of 0x38(%rsp) in this (heavily LLVM-merged, ~30 KB) function, so "a callee scribbled on the caller's frame" and "a different control-flow path legitimately stored 0 and the reload does not check it" are both consistent with the evidence above. Distinguishing them is the next step, and a watchpoint on the slot should do it in one run.

Why this matters beyond the crash

cc --help is the corpus this repo's whole perf campaign measures against — #9272, #9216, #9177 and every A/B in that family gate on "--help byte-identical to node, 9,175 bytes". On main today that gate cannot run at all. Any perf work that rebases onto main right now gets a baseline that segfaults, which is indistinguishable from "my change broke it" until you build a control. It also means the bundle is not exercising the microtask pump at all, so nothing downstream of it is covered.

[proggeramlug]

Resolved: not a clobber — a setjmp stack-slot-coloring hazard on the exception-return path

A hardware watchpoint on the slot settles the "callee scribbled" vs "a path stored 0" question from the caveat above. It is the second one, and the two writes are both run_microtasks's own:

Hardware watchpoint -location *(unsigned long*)($rsp+0x38)
  Old = 5609279237992      New = 140737352661952   at run_microtasks+2618   (0x7ffff7e97bc0, valid)
  Old = 140737352661952    New = 0                 at run_microtasks+...    (0xc6fd5d0)
then: RELOAD slot=(nil) -> SIGSEGV

The two writers share one stack slot, and one of them is on the longjmp path

The pointer is stored before perry's exception transport arms setjmp:

c6fc5e5:  mov    %rax,0x38(%rsp)          ; <-- the pointer, stored PRE-setjmp
...
c6fc72d:  call   perry_runtime::exception::try_push_with_kind
c6fc732:  mov    %rax,%rdi
c6fc735:  call   setjmp                   ; <-- SETJMP
c6fc73b:  test   %eax,%eax
c6fc742:  je     c6fc7f1                  ; ==0 -> first pass
          ; fallthrough = NON-ZERO return = arrived here via LONGJMP
c6fc748:  mov    %r14,0x30(%rsp)
c6fc74d:  mov    MICROTASK_PREV_CONTEXTS::SLOT,%r14d
...
c6fc808:  mov    0x38(%rsp),%rax          ; <-- reload, on the longjmp path
c6fc814:  movzbl -0x37db0(%rax),%eax      ; <-- FAULT, %rax == 0

The zero comes from a completely unrelated loop that LLVM gave the same slot, on the normal (post-setjmp) path — it is copying a 0x30-byte record out of an array:

c6fd5c6:  mov    0x18(%r8,%rcx,1),%rdx
c6fd5cb:  mov    %rdx,0x38(%rsp)          ; <-- unrelated temporary, SAME SLOT
c6fd5d0:  mov    0x20(%r8,%rcx,1),%rdx
c6fd5d5:  mov    %rdx,0x90(%rsp)

Traced with breakpoints, the order in a failing run is exactly:

SETJMP call
STORE_ZERO rdx=(nil)   x5      <- the loop, after setjmp
RELOAD slot=(nil)              <- the longjmp-return path
SIGSEGV

What this means

This is the textbook setjmp rule: a non-volatile local that is written before setjmp and read after a longjmp return has an indeterminate value. The compiler is entitled to color that stack slot for something else on the path between, and here it did. The crash therefore needs a JS throw inside a microtask to fire at all, which is why it is invisible to smaller corpora and why --version is fine.

Two consequences for the fix:

1. It is not specific to the value that happens to be in 0x38(%rsp) — any live-across-setjmp local in a function that uses the js_eh_try_push/setjmp transport has the same exposure. The commits in the bisect window did not create the hazard; they changed inlining/layout enough to make LLVM take the coloring. That means a "fix" that only perturbs this one function will make the symptom move rather than go away.
2. The durable fix is on perry's side of the transport: values that must survive a longjmp return have to be volatile (or re-loaded from memory after the setjmp, or held somewhere the transport itself restores). panic="abort" plus the -C force-unwind-tables=yes pairing documented in [profile.release] already makes this transport special; this is the other half of that contract and it is currently unenforced.

Suggested instrument

A regression that throws from inside a microtask and reads a pre-setjmp local afterwards, compiled at -O3, would pin the contract. Reproducing it reliably will likely need the same inlining pressure as the bundle, so a plain unit test may not be enough — the bundle itself, gated on --help being byte-identical to node, is the honest gate, and it is currently the one that is red.

[proggeramlug]

Independently confirmed, from a different session, using the preserved known-good/known-bad pair rather than a fresh build.

Same box, same HOME=/tmp/cc_home, three runs each:

binary	source	result
app_A	6c880be77b + #9272's content	rc=0, 9,175 bytes, 3/3 — byte-identical to node cli_2.1.112.js --help
app_D	pristine 42d0f45685	SIGSEGV 3/3, rc=139, 0 bytes (core dumped)

So it is deterministic, not a flake, and the good/bad boundary is where the report places it.

Why this is worse than one crashing binary

It takes the campaign's primary correctness gate offline. Every recent runtime performance PR — #9272, #9216, #9177, #9214, #9190 and the stack-map work in #9299/#9300 — gates its numbers on cc --help being byte-identical to node, on the grounds that a performance number from a binary producing wrong output is not a number. That gate cannot run on main today.

The second-order effect is the dangerous one: a segfaulting baseline is indistinguishable from "my change broke it" until someone builds a control pair. Anyone measuring a runtime change against main right now will see their own binary crash and reasonably conclude they caused it. That costs an afternoon per person, and it silently discourages exactly the parity check we want people running.

The root-cause analysis is the actionable part

The report's diagnosis — a non-volatile local written before setjmp (perry's try_push_with_kind exception transport) and read on the longjmp return path, with LLVM giving the same stack slot 0x38(%rsp) to an unrelated array-copy loop on the normal path — is the textbook setjmp indeterminacy rule, and it has a consequence worth stating loudly:

The commits in the bisect window did not create this hazard. They changed inlining enough for LLVM to take the coloring. So a fix that only perturbs this one function will move the symptom rather than remove it, and it will look fixed. The durable fix is at the transport: any local live across setjmp and read after a longjmp return must be volatile or spilled somewhere the optimiser cannot color, and that property wants a test that fails when it is violated — not a rebuild that happens to reassign the slot.

It also explains the shape of the report: it needs a JS throw inside a microtask to fire, which is why smaller corpora miss it entirely and --version is clean.

The known-good/known-bad pair is preserved at /root/claude-reg3/{app_A,app_D} on perrymaster with an OWNER file asking they be kept for this issue.