diff --git a/OWNER b/OWNER new file mode 100644 index 0000000000..baef411823 --- /dev/null +++ b/OWNER @@ -0,0 +1 @@ + OWNER: session 57b8a088 — train120 assembly, no build diff --git a/changelog.d/9728-dynamic-number-tostring.md b/changelog.d/9728-dynamic-number-tostring.md new file mode 100644 index 0000000000..143bbbebac --- /dev/null +++ b/changelog.d/9728-dynamic-number-tostring.md @@ -0,0 +1,52 @@ +**A dynamically dispatched `x["toString"]()` on a number now produces +`NumberToString`, not Rust's `f64` Display** (#9713). It printed `inf` for +`Infinity` and the full decimal expansion past the exponential thresholds, so +the same value stringified four static ways and once dynamically disagreed +inside one program: + +```ts +const a = 2.2e-308; +a.toString(); // 2.2e-308 (all four static forms) +((x: any, m: string) => x[m]())(a, "toString"); // 0.000…00022 — ~308 digits +``` + +Three arms of the native-method tower — the plain-number and boxed-`Number` +`toString` in `dispatch_common`, and the boxed-`Number` +`toString`/`toLocaleString` in `dispatch_primitive` — formatted with + +```rust +if n.fract() == 0.0 && n.abs() < INT_EXACT_FASTPATH_LIMIT { (n as i64).to_string() } else { n.to_string() } +``` + +`f64::to_string()` is Rust's shortest-round-trip Display, which never switches +to scientific notation and renders the infinities as `inf`. It is the exact +mistake `js_format_f64`'s doc comment already warns about — #3987 replaced the +same `format!("{}", n)` in the string-concat fast paths and these three arms +were not part of that sweep. They now call `js_number_to_string`, which carries +the spec's `|n| >= 1e21 || |n| < 1e-6` switch, the `Infinity` / `NaN` / `-0` +spellings, and its own (safer) integer fast path — `js_format_f64` cuts over to +the shortest-round-trip formatter at 1e15 rather than 2^53, so it also avoids +the `2**58` → `…744` vs `…740` divergence the local fast path could reach. + +Measured against node 26.5.1, previously wrong and now correct: `1e21`, +`1e-7`, `-2.5e-9`, `2.2e-308`, `Number.MAX_VALUE`, `Number.MIN_VALUE`, +`Number.EPSILON`, `±Infinity`, and every one of those again through +`new Number(x).toString()`. + +One neighbouring defect in the same arms rides along: a boxed receiver dropped +an explicit radix entirely, so `new Number(255).toString(16)` answered `"255"` +instead of `"ff"`. Both boxed arms now route an explicit radix through +`js_jsvalue_to_string_radix` the way the unboxed arm already did (which also +means an out-of-range radix throws `RangeError` there, as the spec requires). +`toLocaleString` keeps ignoring its argument — that one is a locale, not a +radix. + +`test-files/test_gap_9713_dynamic_number_tostring.ts` pins 18 values across the +thresholds in all seven renderings plus the radix and `toFixed` / +`toPrecision` / `toExponential` forms. Unpatched it differs from node on 12 +lines; patched it is byte-identical. + +Not fixed here, and filed separately: `toString(radix)` above 2^53 for a +non-power-of-two radix still emits exact digits rather than V8's shortest +round-trip (#9725) — that one reproduces from a plain static call and is a +different formatter. diff --git a/changelog.d/9729-lazy-inline-cache-slots.md b/changelog.d/9729-lazy-inline-cache-slots.md new file mode 100644 index 0000000000..b6e9c39803 --- /dev/null +++ b/changelog.d/9729-lazy-inline-cache-slots.md @@ -0,0 +1,70 @@ +### Inline caches are allocated per *used* site, not per emitted site (#9708) + +Every inline-cache site codegen emitted — the generic property read, the +static- and dynamic-key write ICs and their poly tail, the Symbol-keyed and +composed `o[sym].field` reads, the Array-subclass `length` / `[i]` caches, +the fused `if (a.f[i]) return a.f[i]` cache and the imported-object method +guard — owned a `[12 x i64] zeroinitializer` global: 96 B of `__bss` per +site whether or not the program ever executed it. On the Claude Code bundle +that was 262k caches, 25 MB of zero-fill, and 18.7 MB of it **dirty resident +memory at idle**, because a page is dirtied by the first cache touched on it +and the few thousand hot sites are scattered across all of them. + +A site now owns an 8-byte pointer **slot**, `@perry_ic_N = private global +ptr null`. The cache words live in a runtime arena +(`perry-runtime/src/object/field_get_set/ic_slot.rs`): the miss handler +resolves the slot with `pic_slot_resolve` the first time it actually +*primes* the site, bump-allocates the words from a 64 KiB zeroed chunk and +publishes them with a compare-and-swap (two `perry/thread` agents racing on +one site agree on one cache). A miss that cannot prime — proxy, string or +small-handle receiver, a missing key, an accessor, a frozen target — never +touches the slot, so such a site costs its 8 bytes and nothing else; the +write IC's poly tail is not allocated until a fifth shape arrives. Cache +layout and every prime/evict policy are unchanged: the runtime writes the +same words through the same `PicCache` type, and `pic_slot_resolve` sizes +the allocation from that type, so the width pairing test keeps its meaning. + +**Hot path.** Each inline hit path loads the slot (a load with no dependency +on the receiver, so it issues alongside the header loads) and folds `!= null` +into the receiver guard it already evaluates — one fused compare, no new +block — then reads the cache words through the loaded pointer; the runtime +entries take the slot's address. Where a site reads word 0 inside a flat +predicate (the dynamic-key write IC, the array-like index cache, the method +guard) it reads through `select(present, cache, slot)`: the slot's own 8 +bytes of null are exactly the zero token an empty global used to read as, so +the branch structure and the transition-IC reachability are untouched. +Measured on x86-64 (`perf stat -e instructions:u`, perry-dev builds): + +| program | base | lazy slots | delta | +|---|---:|---:|---:| +| all-generic-IC microbenchmark (95M IC ops) | 13.662 G | 14.196 G | +3.9 % (3 instr per hit: slot load, `test`, never-taken `je`) | +| `bench_object_property` | 250.9 M | 248.6 M | −0.9 % | +| `bench_json_readonly` | 2 263.2 M | 2 258.6 M | −0.2 % | +| `bench_dynamic_property_keys` | 1 129.1 M | 1 124.5 M | −0.4 % | +| `07_object_create`, `09_method_calls`, `12_binary_trees`, `14_closure` | | | ±0.00 % | + +The typed-feedback IC counters (`PERRY_TYPED_FEEDBACK_TRACE`) are identical +on both arms for the microbenchmark — 81 666 674 guard passes, 18 333 339 +guard failures, 18 333 339 fallback calls over 18 sites — so hit rates are +unchanged, not merely output. On the issue's target (macOS arm64) the fused +compare is a `ccmp`, so the hit-path cost there is the slot load plus one +instruction. + +**Footprint.** A generated probe with 16 000 read sites of which 1 604 prime +(every 10th function runs — the scattered-hot-sites shape from the issue), +Linux x86-64, 4 KiB pages: `.bss` 2 408 752 → 997 144 B, whole-process +anonymous `Private_Dirty` 1 660 → 632 kB. The `PERRY_GC_CENSUS` side table +gains an `ic.lazy_caches` row (resolved sites, arena bytes) so a run can +assert the subject was live; the issue's macOS `vmmap` numbers are the ones +to re-measure on a bundle build. + +Gap coverage: `test_gap_9708_lazy_inline_cache_slots.ts` exercises every +IC shape across the null → allocated transition — mono/poly/megamorphic +reads, a site that can never prime, a nullish first read, inherited +properties, static writes through the four ways and the poly tail, a frozen +target, rotating dynamic keys, Symbol and composed Symbol-then-field reads +with invalidation, Array-subclass `length`/index, the fused field-index +return, and hundreds of never-executed sites — and matches node byte for +byte. `array/subclass.rs` was at the 2 000-line cap, so +`js_packed_arraylike_index_get` and its cache types moved to the child module +`array/subclass_packed_index.rs`. diff --git a/changelog.d/9730-labeled-escape-nested-loops.md b/changelog.d/9730-labeled-escape-nested-loops.md new file mode 100644 index 0000000000..80f244a6e2 --- /dev/null +++ b/changelog.d/9730-labeled-escape-nested-loops.md @@ -0,0 +1,51 @@ +**A labeled `break`/`continue` that targets an outer loop from inside a nested +loop now works in generators, async generators and async functions** (#9199). +It previously threw `TypeError: Cannot read properties of undefined (reading +'done')` for `break`, and silently produced nothing for `continue`. + +```ts +async function* g() { + O: for (const x of [1, 2]) { I: for (const y of [0, 1]) { yield "b" + x + y; break O; } } +} +// node: b10 before: TypeError … reading 'done' +``` + +Generator linearization gives each loop a single `break` sentinel and a single +`continue` sentinel, so a completion can only name the loop it sits in. +`rewrite_labeled_bc_in_stmts` therefore converted `break label` / `continue +label` to plain completions **only at the labeled loop's own body level** and +stopped at nested loops — correctly, since a plain completion inside a nested +loop would bind to that loop. What was missing is what happens to the escape +that is left: it survived verbatim into a state body, where the dispatch +lowering has no sentinel for it and dropped it. The limitation was noted in the +code ("the single-sentinel scheme can't yet distinguish targets"). + +Rather than teach the state machine to name a distant target, the escape is now +unwound one loop at a time through a carrier local, so every completion the +linearizer sees is plain and binds to the loop it is in: + +``` +__esc = 0; +inner: while (…) { … __esc = 1; break; … } // was `break label` +if (__esc == 1) break; // in the labeled loop +if (__esc == 2) continue; +``` + +Deeper nesting reuses the same carrier and propagates outward with a bare +`if (__esc != 0) break;` after each intermediate loop. A `switch` that carries +an escape is desugared to `if`s first, since a plain `break` inside a switch +would bind to the switch. + +The hole was wider than the issue's own repro, which #9189 had already closed: +it reached sync generators and async functions as well as async generators, +and the `switch` in the report was incidental — a bare `break outer` in a +nested loop failed on its own, while the switch-wrapped form worked because +#9186's routing already handled it. + +`test-files/test_gap_9199_labeled_escape_nested_loops.ts` pins 13 shapes: +`break`/`continue` of an outer label from a nested loop in all three function +kinds, three-deep nesting, a `while` outer, an `await` before the escape, a +conditional escape, `try`/`finally` around it (finalizers still run in order), +a reused label name on a sibling loop, and the switch-wrapped form that already +worked. Unpatched the fixture throws on its first row and then hangs; patched +it is byte-identical to node 26.5.1. diff --git a/changelog.d/9731-arena-right-sizing.md b/changelog.d/9731-arena-right-sizing.md new file mode 100644 index 0000000000..78109a41f7 --- /dev/null +++ b/changelog.d/9731-arena-right-sizing.md @@ -0,0 +1,20 @@ +**Idle heaps now return arena capacity left behind by a burst.** General-arena +blocks deliberately need two full-GC observations before their mappings can be +released, but the idle reducer excluded its own collections from its activity +clock. A quiet heap therefore got one full and could stop forever with every +empty block only halfway through the page-return protocol (#9709). + +Two consecutive post-collection samples at or below 50% utilization, above a +32 MiB capacity floor, now open a bounded arena right-size episode. The existing +idle-reclaim and page-return paths supply only the full observations still +needed, stop early once live data reaches 60% of capacity, and remain subject to +the reducer's quiet-time, rate, wake, and work-budget gates. A completed episode +stays disarmed until utilization reaches 70% or capacity grows materially +(at least 25% and 8 MiB), so a retained low live set cannot turn the idle timer +into a periodic full-GC loop. + +On the compiled Claude Code 2.1.112 workload from the report, arena capacity +fell from 96.5 MiB before the episode to 36.7 MiB, then 35.7 MiB and 35.7 MiB +across a five-minute idle soak with about 23 MiB live. RSS fell from 401 MiB at +the first census to 132 MiB at the last, and exactly one idle full was attributed +to arena right-sizing. diff --git a/changelog.d/9732-followup-diag-counter-verdict.md b/changelog.d/9732-followup-diag-counter-verdict.md new file mode 100644 index 0000000000..632b0a90bc --- /dev/null +++ b/changelog.d/9732-followup-diag-counter-verdict.md @@ -0,0 +1,5 @@ +**Classify #9717's `forwarded_stub_recoveries` diagnostic counter.** The +`gc_runtime_root_holders` gate requires a written verdict for every new +core `perry_thread_local!` declaration; `FORWARDED_STUB_MEMBERSHIP_RECOVERIES` +is a `Cell` tally reported on the `PERRY_GC_DIAG` `[gc-incremental]` line +and holds no address, so it records as `not_a_gc_pointer`. diff --git a/changelog.d/9732-idle-reclaim-growth-stub-membership.md b/changelog.d/9732-idle-reclaim-growth-stub-membership.md new file mode 100644 index 0000000000..0a9bcc4d1a --- /dev/null +++ b/changelog.d/9732-idle-reclaim-growth-stub-membership.md @@ -0,0 +1,7 @@ +**Idle-time (budgeted) collections no longer sweep an array a live field reaches only through an array-growth forwarding stub (#9717).** A `#private` array pushed past its inline capacity leaves a *permanent* forwarding stub at the pre-grow address, and the reference pointing at it is never rewritten (#6228/#233), so a live slot — hono `SmartRouter`'s `#routes`, in the report — keeps naming the stub. A **synchronous** full trace is fine: its exact census (`ValidPointerSetBuilder::record_arena_header`) admits every arena object, stubs included, so `mark_field_into_worklist` marks the stub and `trace_one_worklist_header` follows it to the live array. + +A **budgeted** full trace — the one the idle-time reducer (`PERRY_GC_IDLE_RECLAIM`) runs when a server goes quiet between requests — resolves membership through the page-metadata classifier instead of a census. `classifier_valid_object_start` rejected every `GC_FLAG_FORWARDED` header by design (a dead metadata key's recycled bytes can set that bit, #8040), so the field→stub edge was silently dropped: the stub was never marked, the FORWARDED-follow never ran, and the array reachable *only* through the stub was swept. The field then resolved to reused memory — an empty array — and every route `match()` returned 404 for the life of the process. It reproduced only when the first request arrived ~10–20 s after startup while background work allocated: an early request built the router before any idle collection ran. + +**Fix.** The classifier is documented as a census *superset*; for growth stubs it was not. `classifier_valid_object_start` now admits a plausible forwarded arena stub (`GC_FLAG_ARENA` set, valid `obj_type`/size — the shape a real growth stub has, which separates it from off-heap bytes that coincidentally set the bit). The forwarding *target* is still validated where it always was, in `trace_one_worklist_header`'s follow, so a garbage target simply stops the walk. A `PERRY_GC_DIAG` counter (`forwarded_stub_recoveries=` on the `[gc-incremental]` line) reports how many such stubs a budgeted cycle recovered; it stays zero on a run with no such edge. + +Regression coverage: `gc::tests::forwarded_stub_membership` plants the edge, asserts the pre-fix census-superset gate would have rejected the stub, and drives a budgeted full cycle to completion — the array reached only through the stub survives with its contents intact, and a synchronous full cycle keeps it without needing the recovery path. diff --git a/crates/perry-codegen/src/codegen/closure.rs b/crates/perry-codegen/src/codegen/closure.rs index 57f15b9af3..e530881f23 100644 --- a/crates/perry-codegen/src/codegen/closure.rs +++ b/crates/perry-codegen/src/codegen/closure.rs @@ -1403,11 +1403,7 @@ pub(super) fn compile_closure( llmod.declare_function(&name, ret, ¶ms); } for ic_name in &ic_globals { - llmod.add_raw_global(format!( - "@{} = private global [{} x i64] zeroinitializer", - ic_name, - crate::expr::property_get::generic_dispatch::PIC_CACHE_WORDS - )); + llmod.add_raw_global(crate::expr::inline_cache_global_definition(ic_name)); } for raw in &typed_parse_rodata { llmod.add_raw_global(raw.clone()); diff --git a/crates/perry-codegen/src/codegen/entry.rs b/crates/perry-codegen/src/codegen/entry.rs index b063835dc1..4564eb9046 100644 --- a/crates/perry-codegen/src/codegen/entry.rs +++ b/crates/perry-codegen/src/codegen/entry.rs @@ -1426,11 +1426,7 @@ pub(super) fn compile_module_entry( llmod.declare_function(&name, ret, ¶ms); } for ic_name in &ic_globals { - llmod.add_raw_global(format!( - "@{} = private global [{} x i64] zeroinitializer", - ic_name, - crate::expr::property_get::generic_dispatch::PIC_CACHE_WORDS - )); + llmod.add_raw_global(crate::expr::inline_cache_global_definition(ic_name)); } for raw in &typed_parse_rodata { llmod.add_raw_global(raw.clone()); @@ -1959,11 +1955,7 @@ pub(super) fn compile_module_entry( // A dylib's top-level plugin exports live in its entry module, and the // three symbols must be defined exactly once per shared library. for ic_name in &ic_globals { - llmod.add_raw_global(format!( - "@{} = private global [{} x i64] zeroinitializer", - ic_name, - crate::expr::property_get::generic_dispatch::PIC_CACHE_WORDS - )); + llmod.add_raw_global(crate::expr::inline_cache_global_definition(ic_name)); } for raw in &typed_parse_rodata { llmod.add_raw_global(raw.clone()); diff --git a/crates/perry-codegen/src/codegen/function.rs b/crates/perry-codegen/src/codegen/function.rs index 8e58e0805b..21eb811fb3 100644 --- a/crates/perry-codegen/src/codegen/function.rs +++ b/crates/perry-codegen/src/codegen/function.rs @@ -1454,11 +1454,7 @@ pub(super) fn compile_function( llmod.declare_function(&name, ret, ¶ms); } for ic_name in &ic_globals { - llmod.add_raw_global(format!( - "@{} = private global [{} x i64] zeroinitializer", - ic_name, - crate::expr::property_get::generic_dispatch::PIC_CACHE_WORDS - )); + llmod.add_raw_global(crate::expr::inline_cache_global_definition(ic_name)); } for raw in &typed_parse_rodata { llmod.add_raw_global(raw.clone()); diff --git a/crates/perry-codegen/src/codegen/method.rs b/crates/perry-codegen/src/codegen/method.rs index e4464d2d8e..ff861e76f8 100644 --- a/crates/perry-codegen/src/codegen/method.rs +++ b/crates/perry-codegen/src/codegen/method.rs @@ -1336,11 +1336,7 @@ pub(super) fn compile_method( llmod.declare_function(&name, ret, ¶ms); } for ic_name in &ic_globals { - llmod.add_raw_global(format!( - "@{} = private global [{} x i64] zeroinitializer", - ic_name, - crate::expr::property_get::generic_dispatch::PIC_CACHE_WORDS - )); + llmod.add_raw_global(crate::expr::inline_cache_global_definition(ic_name)); } for raw in &typed_parse_rodata { llmod.add_raw_global(raw.clone()); @@ -1870,11 +1866,7 @@ pub(super) fn compile_static_method( llmod.declare_function(&name, ret, ¶ms); } for ic_name in &ic_globals { - llmod.add_raw_global(format!( - "@{} = private global [{} x i64] zeroinitializer", - ic_name, - crate::expr::property_get::generic_dispatch::PIC_CACHE_WORDS - )); + llmod.add_raw_global(crate::expr::inline_cache_global_definition(ic_name)); } for raw in &typed_parse_rodata { llmod.add_raw_global(raw.clone()); diff --git a/crates/perry-codegen/src/expr/index_get.rs b/crates/perry-codegen/src/expr/index_get.rs index ee04a90858..0297f1acc5 100644 --- a/crates/perry-codegen/src/expr/index_get.rs +++ b/crates/perry-codegen/src/expr/index_get.rs @@ -72,15 +72,27 @@ pub(crate) fn lower_symbol_property_get_ic( ctx.ic_site_counter += 1; let cache_name = super::inline_cache_global_name(ctx, site_id); ctx.ic_globals.push(cache_name.clone()); - let cache_ref = format!("@{cache_name}"); + let probe_idx = ctx.new_block("symic.probe"); let hit_idx = ctx.new_block("symic.hit"); let miss_idx = ctx.new_block("symic.miss"); let merge_idx = ctx.new_block("symic.merge"); + let probe_label = ctx.block_label(probe_idx); let hit_label = ctx.block_label(hit_idx); let miss_label = ctx.block_label(miss_idx); let merge_label = ctx.block_label(merge_idx); + // #9708: the cache sits behind a pointer slot that the miss handler fills + // on the first prime. The probe's three loads go through the pointer, so + // an absent cache branches straight to the miss — the edge a fresh + // (all-zero) global took anyway, since a zero epoch never matches. + let ic_slot = super::emit_inline_cache_slot(ctx, &cache_name); + let cache_ref = ic_slot.cache.clone(); + let cache_slot_ref = ic_slot.slot_ref.clone(); + ctx.block() + .cond_br(&ic_slot.present, &probe_label, &miss_label); + + ctx.current_block = probe_idx; let epoch = ctx .block() .load_atomic_acquire(I64, "@PERRY_SYMBOL_PROPERTY_IC_EPOCH", 8); @@ -110,7 +122,7 @@ pub(crate) fn lower_symbol_property_get_ic( let miss_value = ctx.block().call( DOUBLE, "js_object_get_symbol_property_ic_miss", - &[(DOUBLE, obj_box), (DOUBLE, sym_box), (PTR, &cache_ref)], + &[(DOUBLE, obj_box), (DOUBLE, sym_box), (PTR, &cache_slot_ref)], ); let miss_end = ctx.block().label.clone(); ctx.block().br(&merge_label); diff --git a/crates/perry-codegen/src/expr/index_get/inline_dyn_typed_array.rs b/crates/perry-codegen/src/expr/index_get/inline_dyn_typed_array.rs index 2113d0370b..1ca6db0f2b 100644 --- a/crates/perry-codegen/src/expr/index_get/inline_dyn_typed_array.rs +++ b/crates/perry-codegen/src/expr/index_get/inline_dyn_typed_array.rs @@ -340,7 +340,17 @@ pub(super) fn lower_inline_dyn_typed_array_get( ctx.ic_site_counter += 1; let cache_name = super::super::inline_cache_global_name(ctx, site_id); ctx.ic_globals.push(cache_name.clone()); - let cache_ref = format!("@{cache_name}"); + // #9708: the cache sits behind a pointer slot the runtime fills on the + // first shape-carried prime. `arrlike.ic.shape` reads word 0 inside a + // flat predicate, so it reads through `key_cache`: the real cache when + // present, else the slot itself — 8 bytes of null, i.e. a zero identity, + // which fails `key_nonzero` exactly as the all-zero global did. Every + // later word is read only past that edge, through the real pointer. + let ic_slot = crate::expr::emit_inline_cache_slot(ctx, &cache_name); + let cache_ref = ic_slot.cache.clone(); + let key_cache = ctx + .block() + .select(I1, &ic_slot.present, PTR, &cache_ref, &ic_slot.slot_ref); let object_header_idx = ctx.new_block("arrlike.ic.header"); let object_brand_idx = ctx.new_block("arrlike.ic.brand"); @@ -613,7 +623,7 @@ pub(super) fn lower_inline_dyn_typed_array_get( let shape64 = ctx.block().zext(I32, &shape_id, I64); let class_high = ctx.block().shl(I64, &class64, "32"); let live_key = ctx.block().or(I64, &class_high, &shape64); - let cached_key_ptr = ctx.block().gep(I64, &cache_ref, &[(I64, "0")]); + let cached_key_ptr = ctx.block().gep(I64, &key_cache, &[(I64, "0")]); let cached_key = ctx.block().load(I64, &cached_key_ptr); let key_nonzero = ctx.block().icmp_ne(I64, &cached_key, "0"); let object_ok = ctx.block().and(I1, &is_object, &key_nonzero); @@ -843,7 +853,7 @@ pub(super) fn lower_inline_dyn_typed_array_get( let slow_raw = ctx.block().call( DOUBLE, "js_packed_arraylike_index_get", - &[(DOUBLE, obj_box), (DOUBLE, idx_d), (PTR, &cache_ref)], + &[(DOUBLE, obj_box), (DOUBLE, idx_d), (PTR, &ic_slot.slot_ref)], ); // In a number context, coerce the (possibly boxed) slow result here so the // merge phi is uniformly a Number and the arithmetic caller skips its own diff --git a/crates/perry-codegen/src/expr/mod.rs b/crates/perry-codegen/src/expr/mod.rs index dc6f0e332d..a360d47452 100644 --- a/crates/perry-codegen/src/expr/mod.rs +++ b/crates/perry-codegen/src/expr/mod.rs @@ -1517,8 +1517,10 @@ pub(crate) struct FnCtx<'a> { pub ic_site_counter: u32, /// (Issue #51) Names of IC globals created during lowering. After - /// the function is emitted, the caller emits `@ = private - /// global [2 x i64] zeroinitializer` for each entry. + /// the function is emitted, the caller emits one + /// [`inline_cache_global_definition`] — `@ = private global ptr + /// null`, an 8-byte slot the runtime fills with an arena cache on the + /// site's first priming miss (#9708) — for each entry. pub ic_globals: Vec, /// Region-scoped cache selected by a guarded statement fusion. Generic @@ -2235,6 +2237,50 @@ pub(crate) fn inline_cache_global_name(ctx: &FnCtx<'_>, site_id: u32) -> String inline_cache_global_name_for_prefix(ctx.strings.module_prefix(), site_id) } +/// The definition emitted for every name in `FnCtx::ic_globals`. +/// +/// #9708: an inline-cache site owns an 8-byte pointer **slot**, not its cache +/// words. The slot is zero-initialised (so it lands in `__bss` and costs +/// nothing until touched) and stays null until the runtime's miss handler +/// primes the site, at which point it publishes a cache allocated from the +/// runtime's IC arena (`perry_runtime::object::pic_slot_resolve`). A site +/// the program never executes therefore costs 8 bytes of zero-fill instead +/// of a 96-byte cache that dirtied a resident page on first touch. Every +/// inline hit path loads the slot through [`emit_inline_cache_slot`] and +/// proves it non-null before reading a cache word; every runtime miss entry +/// takes the slot's address. +pub(crate) fn inline_cache_global_definition(name: &str) -> String { + format!("@{name} = private global ptr null") +} + +/// A site's inline-cache slot, loaded in the current block (#9708). +/// +/// `slot_ref` is the `@perry_ic_N` global — what the runtime miss entries +/// take. `cache` is the `ptr` loaded from it and `present` the `i1` proving +/// it non-null: a site must branch (or fold `present` into a guard that +/// dominates) before it emits any load through `cache`, because the slot is +/// null until the site's first priming miss. +pub(crate) struct InlineCacheSlot { + pub slot_ref: String, + pub cache: String, + pub present: String, +} + +/// Load `@`'s cache pointer in the current block and test it. +/// The load has no dependency on the receiver, so it is free to issue early; +/// folding `present` into the receiver guard the site already evaluates costs +/// a single fused compare on the hit path. +pub(crate) fn emit_inline_cache_slot(ctx: &mut FnCtx<'_>, cache_name: &str) -> InlineCacheSlot { + let slot_ref = format!("@{cache_name}"); + let cache = ctx.block().load(PTR, &slot_ref); + let present = ctx.block().icmp_ne(PTR, &cache, "null"); + InlineCacheSlot { + slot_ref, + cache, + present, + } +} + /// Record a cold-arm bailout for a compiler-private versioned-loop callback. /// The stack context is `[counter_slot_ptr, original_bound, resume_index]` as /// three i64 words. The first cold arm stores `counter + 1` and poisons the diff --git a/crates/perry-codegen/src/expr/property_get/composed_ics.rs b/crates/perry-codegen/src/expr/property_get/composed_ics.rs index 4171ce5130..38ecc9ecdb 100644 --- a/crates/perry-codegen/src/expr/property_get/composed_ics.rs +++ b/crates/perry-codegen/src/expr/property_get/composed_ics.rs @@ -42,27 +42,42 @@ pub(super) fn lower_symbol_then_named_property_ic( let symbol_site = ctx.ic_site_counter; ctx.ic_site_counter += 1; - let symbol_cache = super::super::inline_cache_global_name(ctx, symbol_site); - ctx.ic_globals.push(symbol_cache.clone()); - let symbol_cache = format!("@{symbol_cache}"); + let symbol_cache_name = super::super::inline_cache_global_name(ctx, symbol_site); + ctx.ic_globals.push(symbol_cache_name.clone()); let field_site = ctx.ic_site_counter; ctx.ic_site_counter += 1; - let field_cache = super::super::inline_cache_global_name(ctx, field_site); - ctx.ic_globals.push(field_cache.clone()); - let field_cache = format!("@{field_cache}"); + let field_cache_name = super::super::inline_cache_global_name(ctx, field_site); + ctx.ic_globals.push(field_cache_name.clone()); + let probe_idx = ctx.new_block("symfield.probe"); let identity_idx = ctx.new_block("symfield.identity"); let hit_idx = ctx.new_block("symfield.hit"); let live_idx = ctx.new_block("symfield.live"); let miss_idx = ctx.new_block("symfield.miss"); let merge_idx = ctx.new_block("symfield.merge"); + let probe_label = ctx.block_label(probe_idx); let identity_label = ctx.block_label(identity_idx); let hit_label = ctx.block_label(hit_idx); let live_label = ctx.block_label(live_idx); let miss_label = ctx.block_label(miss_idx); let merge_label = ctx.block_label(merge_idx); + // #9708: both caches sit behind pointer slots the miss handler fills + // on the first prime. The probe, identity and hit blocks read through + // the two pointers, so they are reached only once both are present; + // an absent cache is the miss edge a zero epoch always took. + let symbol_slot = crate::expr::emit_inline_cache_slot(ctx, &symbol_cache_name); + let field_slot = crate::expr::emit_inline_cache_slot(ctx, &field_cache_name); + let symbol_cache = symbol_slot.cache.clone(); + let field_cache = field_slot.cache.clone(); + let both_present = ctx + .block() + .and(I1, &symbol_slot.present, &field_slot.present); + ctx.block() + .cond_br(&both_present, &probe_label, &miss_label); + + ctx.current_block = probe_idx; let epoch = ctx .block() .load_atomic_acquire(I64, "@PERRY_SYMBOL_PROPERTY_IC_EPOCH", 8); @@ -144,8 +159,8 @@ pub(super) fn lower_symbol_then_named_property_ic( (DOUBLE, &symbol_box), (PTR, &key_ptr), (I64, &feedback_site_id), - (PTR, &symbol_cache), - (PTR, &field_cache), + (PTR, &symbol_slot.slot_ref), + (PTR, &field_slot.slot_ref), ], ); let miss_end = ctx.block().label.clone(); @@ -175,10 +190,10 @@ pub(super) fn emit_array_subclass_length_ic( ctx.ic_site_counter += 1; let cache_name = super::super::inline_cache_global_name(ctx, site_id); ctx.ic_globals.push(cache_name.clone()); - let cache_ref = format!("@{cache_name}"); let header_idx = ctx.new_block("plen.ic.header"); let shape_idx = ctx.new_block("plen.ic.shape"); + let shape_probe_idx = ctx.new_block("plen.ic.shape.probe"); let identity_idx = ctx.new_block("plen.ic.identity"); let exact_idx = ctx.new_block("plen.ic.exact"); let family_meta_idx = ctx.new_block("plen.ic.family_meta"); @@ -192,6 +207,7 @@ pub(super) fn emit_array_subclass_length_ic( let merge_idx = ctx.new_block("plen.ic.merge"); let header_label = ctx.block_label(header_idx); let shape_label = ctx.block_label(shape_idx); + let shape_probe_label = ctx.block_label(shape_probe_idx); let identity_label = ctx.block_label(identity_idx); let exact_label = ctx.block_label(exact_idx); let family_meta_label = ctx.block_label(family_meta_idx); @@ -219,6 +235,14 @@ pub(super) fn emit_array_subclass_length_ic( let below_ceiling = ctx.block().icmp_ult(I64, recv_handle, &heap_ceiling); let in_heap = ctx.block().and(I1, &pointer_tag, &above_floor); let in_heap = ctx.block().and(I1, &in_heap, &below_ceiling); + // #9708: the cache sits behind a pointer slot the runtime fills on the + // first shape-carried prime. It is loaded here, off the receiver's + // dependency chain, and tested at `plen.ic.shape` — NOT folded into the + // header guard, because the elements-backed arm between them serves + // `length` without ever publishing a cache, and a site whose receivers + // are all elements-backed must keep that arm with a slot that stays null. + let ic_slot = crate::expr::emit_inline_cache_slot(ctx, &cache_name); + let cache_ref = ic_slot.cache.clone(); ctx.block().cond_br(&in_heap, &header_label, &miss_label); ctx.current_block = header_idx; @@ -281,6 +305,10 @@ pub(super) fn emit_array_subclass_length_ic( ctx.block().br(&merge_label); ctx.current_block = shape_idx; + ctx.block() + .cond_br(&ic_slot.present, &shape_probe_label, &miss_label); + + ctx.current_block = shape_probe_idx; let object_ptr = ctx.block().inttoptr(I64, recv_handle); let class_id = ctx.block().load(I32, &object_ptr); let shape_addr = ctx.block().add(I64, recv_handle, "4"); @@ -395,7 +423,7 @@ pub(super) fn emit_array_subclass_length_ic( let miss_length = ctx.block().call( DOUBLE, "js_value_length_property_ic_f64", - &[(DOUBLE, recv_box), (PTR, &cache_ref)], + &[(DOUBLE, recv_box), (PTR, &ic_slot.slot_ref)], ); let miss_end = ctx.block().label.clone(); ctx.block().br(&merge_label); diff --git a/crates/perry-codegen/src/expr/property_get/generic_dispatch.rs b/crates/perry-codegen/src/expr/property_get/generic_dispatch.rs index 1cdb2c8424..29040b9c65 100644 --- a/crates/perry-codegen/src/expr/property_get/generic_dispatch.rs +++ b/crates/perry-codegen/src/expr/property_get/generic_dispatch.rs @@ -13,15 +13,19 @@ use perry_hir::Expr; use crate::nanbox::POINTER_MASK_I64; use crate::types::{DOUBLE, I1, I32, I64, I8, PTR}; -/// Words in a per-site `@perry_ic_N` property-read cache global. +/// Words in a per-site property-read cache. /// -/// **Must equal `perry_runtime::object::field_get_set::PIC_CACHE_WORDS`** — -/// the runtime writes this memory through a `*mut [i64; PIC_CACHE_WORDS]`, so a -/// smaller global here is an out-of-bounds store. perry-codegen does not depend -/// on perry-runtime (the same reason `INLINE_SLOT_FLOOR` is duplicated in +/// **Must equal `perry_runtime::object::field_get_set::PIC_CACHE_WORDS`.** +/// Since #9708 codegen emits only the 8-byte slot (`@perry_ic_N = private +/// global ptr null`) and the runtime allocates the words itself, sized from +/// its own `PicCache` — so the constant is no longer an emission width, but +/// the emitted way GEPs (`PIC_WAY_BASE + PIC_WAYS * 2` words) must still +/// land inside that allocation. perry-codegen does not depend on +/// perry-runtime (the same reason `INLINE_SLOT_FLOOR` is duplicated in /// `target_layout`), so the pairing is held by `pic_cache_layout_matches_runtime` /// here and `pic_cache_words_match_codegen` in the runtime: change one and both /// fail. +#[cfg(test)] pub(crate) const PIC_CACHE_WORDS: usize = 12; /// First word of the polymorphic way array (words 0..2 are the MRU entry and /// word 3 is the gate). Mirrors the runtime's `PIC_WAY_BASE`. @@ -136,7 +140,9 @@ pub(crate) fn lower_generic_property_get( // unchanged. let cache_name = overridden_cache_name(ctx, object, property) .unwrap_or_else(|| allocate_property_cache(ctx)); - let cache_ref = format!("@{}", cache_name); + // #9708: the helper takes the site's SLOT and resolves the cache + // itself; nothing is read inline here, so no load is emitted. + let cache_slot_ref = format!("@{}", cache_name); let key_handle = emit_key_handle(ctx, &key_handle_global); let val = ctx.block().call( DOUBLE, @@ -145,7 +151,7 @@ pub(crate) fn lower_generic_property_get( (I64, &obj_bits), (I64, &key_handle), (I64, &feedback_site_id), - (PTR, &cache_ref), + (PTR, &cache_slot_ref), ], ); return Ok(val); @@ -354,7 +360,6 @@ pub(crate) fn lower_generic_property_get( // // Threshold matches `js_native_call_method`'s small-handle // detection (raw_ptr < 0x100000). - let cache_ref = format!("@{}", cache_name); let is_real_ptr = ctx.block().icmp_ugt(I64, &obj_handle, "1048575"); // 0x100000 // #7883: the hit/miss/merge blocks are minted here so the guard chain @@ -462,7 +467,21 @@ pub(crate) fn lower_generic_property_get( let reserved = ctx.block().load(crate::types::I16, &reserved_ptr); let has_desc = ctx.block().and(crate::types::I16, &reserved, "2048"); // OBJ_FLAG_HAS_DESCRIPTORS (0x800) let no_desc = ctx.block().icmp_eq(crate::types::I16, &has_desc, "0"); + // #9708: the site's cache lives behind a pointer slot that is null until + // the first priming miss. The slot load does not depend on the receiver, + // so it issues alongside the header loads, and its non-null test joins + // the flat header predicate as one more fused compare. Both edges that + // read a cache word (`pic.token` and the descriptor prefix path) require + // `cache_present`; the slot itself is what the miss handler takes, so a + // fresh site goes straight to it. `cache_ref` is the LOADED pointer from + // here on, never the global: every GEP below goes through it, and only + // the runtime calls take `cache_slot_ref`. + let ic_slot = crate::expr::emit_inline_cache_slot(ctx, &cache_name); + let cache_ref = ic_slot.cache.clone(); + let cache_slot_ref = ic_slot.slot_ref.clone(); + let cache_present = ic_slot.present.clone(); let is_plain_object = ctx.block().and(I1, &is_object_kind, &no_desc); + let is_plain_object = ctx.block().and(I1, &is_plain_object, &cache_present); // #7883: first exit. The header predicates above are kept as one flat // `and` on purpose — they are loads from the same cache line and LLVM @@ -485,8 +504,15 @@ pub(crate) fn lower_generic_property_get( // is unrelated to all class-declared named fields and arm word 2. Keep // this classification off the ordinary descriptor-free hit path. ctx.current_block = desc_classify_idx; - ctx.block() - .cond_br(&is_object_kind, &desc_prefix_guard_label, &cold_label); + // #9708: the descriptor prefix path reads cache word 2, so it needs the + // same non-null proof `pic.token` has; without a cache the receiver is + // simply a cold miss. + let desc_object_with_cache = ctx.block().and(I1, &is_object_kind, &cache_present); + ctx.block().cond_br( + &desc_object_with_cache, + &desc_prefix_guard_label, + &cold_label, + ); ctx.current_block = tok_idx; // The receiver token is derived solely from its authoritative ShapeId. @@ -578,7 +604,7 @@ pub(crate) fn lower_generic_property_get( (I64, &obj_handle), (I64, &ovf_key_handle), (I32, &ovf_slot_i32), - (PTR, &cache_ref), + (PTR, &cache_slot_ref), ], ); let ovf_end_label = ctx.block().label.clone(); @@ -931,7 +957,7 @@ pub(crate) fn lower_generic_property_get( &[ (I64, &obj_handle), (I64, &miss_key_handle), - (PTR, &cache_ref), + (PTR, &cache_slot_ref), ], ); let miss_end_label = ctx.block().label.clone(); diff --git a/crates/perry-codegen/src/expr/property_get/tests.rs b/crates/perry-codegen/src/expr/property_get/tests.rs index 72542af871..5196d9c092 100644 --- a/crates/perry-codegen/src/expr/property_get/tests.rs +++ b/crates/perry-codegen/src/expr/property_get/tests.rs @@ -276,12 +276,13 @@ fn fs_parent_promises_property_installs_before_resolution() { /// #7753, paired with `pic_cache_words_match_codegen` in /// `perry-runtime/src/object/field_get_set/ic_miss.rs`. /// -/// The runtime writes a `@perry_ic_N` global through `*mut [i64; -/// PIC_CACHE_WORDS]`. If codegen emits a NARROWER global, `pic_prime_get`'s way -/// stores run past the end of it into whatever global the linker placed next — -/// silent memory corruption that no property-read test would notice. This pins -/// the emitted width to the constant both sides share, and pins the constant -/// itself so the runtime's copy cannot drift. +/// The runtime writes a site's cache through `*mut [i64; PIC_CACHE_WORDS]` +/// and the emitted ways read words up to `PIC_WAY_BASE + PIC_WAYS * 2`. Since +/// #9708 the cache words are allocated by the runtime (`pic_slot_resolve` +/// sizes them from its own `PicCache`), so the two constants pinned here are +/// what keeps the emitted way GEPs inside that allocation. The emitted global +/// itself is the 8-byte SLOT, never the words: a `[N x i64]` IC global would +/// be the pre-#9708 shape coming back, with its 96 B of zero-fill per site. #[test] fn pic_cache_layout_matches_runtime() { use crate::expr::property_get::generic_dispatch::{ @@ -301,11 +302,25 @@ fn pic_cache_layout_matches_runtime() { "runtime PicCache word 2 carries the Array-subclass named-prefix token" ); let ir = emit(false, None); + let ic_defs: Vec<&str> = ir + .lines() + .filter(|l| l.starts_with("@perry_ic_") && l.contains(" = ")) + .collect(); assert!( - ir.contains(&format!( - "= private global [{PIC_CACHE_WORDS} x i64] zeroinitializer" - )), - "every @perry_ic_N must be emitted at the width the runtime writes:\n{ir}" + !ic_defs.is_empty(), + "test premise: the generic read emits a per-site cache slot:\n{ir}" + ); + for def in &ic_defs { + assert!( + def.ends_with(" = private global ptr null"), + "every @perry_ic_N must be an 8-byte null pointer slot the runtime \ + fills on the first prime (#9708), got:\n{def}\n\nIR:\n{ir}" + ); + } + assert!( + ir.contains("load ptr, ptr @perry_ic_") && ir.contains("icmp ne ptr "), + "the hit path must load the slot and prove it non-null before reading \ + a cache word:\n{ir}" ); } diff --git a/crates/perry-codegen/src/expr/proxy_reflect.rs b/crates/perry-codegen/src/expr/proxy_reflect.rs index 23ea00e5a4..e4e84e19d7 100644 --- a/crates/perry-codegen/src/expr/proxy_reflect.rs +++ b/crates/perry-codegen/src/expr/proxy_reflect.rs @@ -510,13 +510,21 @@ fn lower_put_value_static_write_ic( ctx.pending_declares .push((format!("__ic_decl_{}", site_id), DOUBLE, vec![])); ctx.ic_globals.push(cache_name.clone()); - let cache_ref = format!("@{}", cache_name); // Keep the first four ways inline. Shapes 5–8 use a separate cache in a // compact outlined helper, avoiding four more copies of the generated // receiver guards while preventing the fourth inline way from thrashing. let tail_cache_name = format!("{}_poly_tail", cache_name); ctx.ic_globals.push(tail_cache_name.clone()); - let tail_cache_ref = format!("@{}", tail_cache_name); + // #9708: both caches sit behind pointer slots. The inline ways read the + // primary cache through the pointer loaded here, so its non-null test + // joins `heap_candidate` (every way guard is dominated by that edge); a + // site that has never primed goes to the way-0 miss, which allocates. The + // tail is only ever handed to the runtime, which resolves it itself — + // so a tail cache is not allocated until a fifth shape actually arrives. + let ic_slot = crate::expr::emit_inline_cache_slot(ctx, &cache_name); + let cache_ref = ic_slot.cache.clone(); + let cache_slot_ref = ic_slot.slot_ref.clone(); + let tail_cache_slot_ref = format!("@{}", tail_cache_name); // Branch before the first header load so primitives, forged non-pointer // bit patterns, and native handle ids can never be dereferenced by the @@ -525,6 +533,7 @@ fn lower_put_value_static_write_ic( let pointer_tag = ctx.block().icmp_eq(I64, &target_tag, "32765"); // 0x7FFD let above_handles = ctx.block().icmp_ugt(I64, &target_handle, "1048575"); // 0x100000 let heap_candidate = ctx.block().and(I1, &pointer_tag, &above_handles); + let heap_candidate = ctx.block().and(I1, &heap_candidate, &ic_slot.present); let guard_idx = ctx.new_block("put.pic.guard"); let guard2_idx = ctx.new_block("put.pic.guard2"); let guard3_idx = ctx.new_block("put.pic.guard3"); @@ -854,7 +863,8 @@ fn lower_put_value_static_write_ic( (I64, &key_handle), (DOUBLE, &stored_value), (I32, strict_i32), - (PTR, &cache_ref), + (PTR, &cache_slot_ref), + (I32, "0"), ], ); let deleted_end_label = ctx.block().label.clone(); @@ -869,7 +879,8 @@ fn lower_put_value_static_write_ic( (I64, &key_handle), (DOUBLE, &stored_value), (I32, strict_i32), - (PTR, &cache_ref), + (PTR, &cache_slot_ref), + (I32, "0"), ], ); let miss_end_label = ctx.block().label.clone(); @@ -884,7 +895,8 @@ fn lower_put_value_static_write_ic( (I64, &key_handle), (DOUBLE, &stored_value), (I32, strict_i32), - (PTR, &cached2_token_ptr), + (PTR, &cache_slot_ref), + (I32, "1"), ], ); let miss2_end_label = ctx.block().label.clone(); @@ -899,7 +911,8 @@ fn lower_put_value_static_write_ic( (I64, &key_handle), (DOUBLE, &stored_value), (I32, strict_i32), - (PTR, &cached3_token_ptr), + (PTR, &cache_slot_ref), + (I32, "2"), ], ); let miss3_end_label = ctx.block().label.clone(); @@ -914,7 +927,8 @@ fn lower_put_value_static_write_ic( (I64, &key_handle), (DOUBLE, &stored_value), (I32, strict_i32), - (PTR, &cached4_token_ptr), + (PTR, &cache_slot_ref), + (I32, "3"), ], ); let miss4_end_label = ctx.block().label.clone(); @@ -925,7 +939,7 @@ fn lower_put_value_static_write_ic( DOUBLE, "js_put_value_set_ic_poly_tail", &[ - (PTR, &tail_cache_ref), + (PTR, &tail_cache_slot_ref), (DOUBLE, &target_value), (I64, &key_handle), (DOUBLE, &stored_value), @@ -974,7 +988,25 @@ fn lower_put_value_dyn_ic_inline( ctx.ic_site_counter += 1; let cache_name = super::inline_cache_global_name(ctx, site_id); ctx.ic_globals.push(cache_name.clone()); - let cache_ref = format!("@{}", cache_name); + // #9708: the site cache sits behind a pointer slot that is null until the + // miss handler's first prime. The guard block below reads word 0 in a + // flat predicate, so it reads through `token_cache`: the real cache when + // present, else the SLOT ITSELF — an 8-byte null, i.e. a zero token, which + // is exactly what the all-zero global used to read as. A zero token fails + // `token_nonzero`, so the ways (which read words 1..6 through the real + // pointer) are unreachable for an absent cache, and the transition probe + // is reached on the same edge it always was. The outlined slow entry and + // the miss handler take the slot. + let ic_slot = crate::expr::emit_inline_cache_slot(ctx, &cache_name); + let cache_ref = ic_slot.cache.clone(); + let cache_slot_ref = ic_slot.slot_ref.clone(); + let token_cache = ctx.block().select( + I1, + &ic_slot.present, + crate::types::PTR, + &cache_ref, + &cache_slot_ref, + ); // #9287: this thread's transition-cache base, loaded once per function // (the table is thread-local; a link-time constant would alias one @@ -1094,7 +1126,7 @@ fn lower_put_value_dyn_ic_inline( let shape_token = ctx .block() .select(I1, &has_shape_id, I64, &shape_id_token, "0"); - let cached_token_ptr = ctx.block().gep(I64, &cache_ref, &[(I64, "0")]); + let cached_token_ptr = ctx.block().gep(I64, &token_cache, &[(I64, "0")]); let cached_token = ctx.block().load(I64, &cached_token_ptr); let token_match = ctx.block().icmp_eq(I64, &shape_token, &cached_token); let token_nonzero = ctx.block().icmp_ne(I64, &shape_token, "0"); @@ -1433,7 +1465,7 @@ fn lower_put_value_dyn_ic_inline( DOUBLE, "js_put_value_set_dyn_ic", &[ - (crate::types::PTR, &cache_ref), + (crate::types::PTR, &cache_slot_ref), (DOUBLE, t), (DOUBLE, k), (DOUBLE, v), @@ -2049,12 +2081,13 @@ pub(crate) fn lower(ctx: &mut FnCtx<'_>, expr: &Expr) -> Result { ctx.ic_site_counter += 1; let cache_name = super::inline_cache_global_name(ctx, site_id); ctx.ic_globals.push(cache_name.clone()); - let cache_ref = format!("@{}", cache_name); + // #9708: the outlined entry takes the site's slot. + let cache_slot_ref = format!("@{}", cache_name); Ok(ctx.block().call( DOUBLE, "js_put_value_set_dyn_ic", &[ - (crate::types::PTR, &cache_ref), + (crate::types::PTR, &cache_slot_ref), (DOUBLE, &t), (DOUBLE, &k), (DOUBLE, &v), diff --git a/crates/perry-codegen/src/lower_call/property_get/imported_object.rs b/crates/perry-codegen/src/lower_call/property_get/imported_object.rs index 058558d7af..925e5e5076 100644 --- a/crates/perry-codegen/src/lower_call/property_get/imported_object.rs +++ b/crates/perry-codegen/src/lower_call/property_get/imported_object.rs @@ -60,7 +60,6 @@ fn emit_cached_own_method_guard( ctx.ic_site_counter += 1; let cache_name = crate::expr::inline_cache_global_name(ctx, cache_site); ctx.ic_globals.push(cache_name.clone()); - let cache_ref = format!("@{cache_name}"); let deref_idx = ctx.new_block("object_method_cache.deref"); let fast_idx = ctx.new_block("object_method_cache.fast"); @@ -85,6 +84,15 @@ fn emit_cached_own_method_guard( let below_ceiling = ctx.block().icmp_ult(I64, &recv_handle, &heap_ceiling); let in_range = ctx.block().and(I1, &above_floor, &below_ceiling); let safe = ctx.block().and(I1, &tagged, &in_range); + // #9708: the token lives behind a pointer slot the miss handler fills on + // the first prime. `deref` reads it in a flat predicate, so it reads + // through `token_cache`: the real cache when present, else the slot itself + // — 8 bytes of null, a zero token, which fails `cache_populated` exactly as + // the all-zero global did and takes the same revalidate edge. + let ic_slot = crate::expr::emit_inline_cache_slot(ctx, &cache_name); + let token_cache = + ctx.block() + .select(I1, &ic_slot.present, PTR, &ic_slot.cache, &ic_slot.slot_ref); ctx.block().cond_br(&safe, &deref_label, miss_label); ctx.current_block = deref_idx; @@ -96,7 +104,7 @@ fn emit_cached_own_method_guard( .and(I32, &gc_header, GC_OBJECT_METHOD_GUARD_MASK_I32); let gc_header_ok = ctx.block().icmp_eq(I32, &guarded_gc_bits, GC_TYPE_OBJECT); let live_class_shape = ctx.block().load(I64, &object_ptr); - let cache_token_ptr = ctx.block().gep(I64, &cache_ref, &[(I64, "0")]); + let cache_token_ptr = ctx.block().gep(I64, &token_cache, &[(I64, "0")]); let cached_class_shape = ctx.block().load(I64, &cache_token_ptr); let cache_populated = ctx.block().icmp_ne(I64, &cached_class_shape, "0"); let shape_matches = ctx @@ -137,7 +145,7 @@ fn emit_cached_own_method_guard( (PTR, &bytes_global), (I64, &name_len), (PTR, &format!("@{closure_symbol}")), - (PTR, &cache_token_ptr), + (PTR, &ic_slot.slot_ref), ], ); let cold_passes = ctx.block().icmp_ne(I64, &cold_handle, "0"); diff --git a/crates/perry-codegen/src/module.rs b/crates/perry-codegen/src/module.rs index 8ed7f9c547..4839d3264e 100644 --- a/crates/perry-codegen/src/module.rs +++ b/crates/perry-codegen/src/module.rs @@ -1447,14 +1447,15 @@ mod tests { // `zeroinitializer` global that only ONE unit defines moves it out of // zerofill `__DATA,__bss` and writes its zeros into the file — 25.16 MB // (8.2%) of the Claude Code binary, all of it per-site inline caches - // (`[12 x i64] zeroinitializer`, one per property-access site, each - // referenced by exactly one function and so by exactly one unit). + // (`[12 x i64] zeroinitializer` then; an 8-byte `ptr null` slot since + // #9708 — one per property-access site either way, each referenced by + // exactly one function and so by exactly one unit). // Promote only what a link would otherwise see defined twice; ELF/COFF // are unaffected either way (their BSS choice ignores linkage). let mut m = LlModule::new("arm64-apple-macosx15.0.0"); m.declare_function("js_ic_touch", VOID, &[PTR]); - m.add_raw_global("@perry_ic_m__0 = private global [12 x i64] zeroinitializer".to_string()); - m.add_raw_global("@perry_ic_m__1 = private global [12 x i64] zeroinitializer".to_string()); + m.add_raw_global(crate::expr::inline_cache_global_definition("perry_ic_m__0")); + m.add_raw_global(crate::expr::inline_cache_global_definition("perry_ic_m__1")); m.add_internal_global("perry_class_keys_m__C", I64, "0"); m.add_global("perry_class_shape_id_m__C", I32, "0"); @@ -1481,9 +1482,7 @@ mod tests { for ic in ["@perry_ic_m__0", "@perry_ic_m__1"] { let defs: Vec<&String> = units .iter() - .filter(|u| { - u.contains(&format!("{ic} = private global [12 x i64] zeroinitializer")) - }) + .filter(|u| u.contains(&format!("{ic} = private global ptr null"))) .collect(); assert_eq!( defs.len(), diff --git a/crates/perry-codegen/src/runtime_decls/objects.rs b/crates/perry-codegen/src/runtime_decls/objects.rs index ef42376207..5a578f4c87 100644 --- a/crates/perry-codegen/src/runtime_decls/objects.rs +++ b/crates/perry-codegen/src/runtime_decls/objects.rs @@ -527,10 +527,11 @@ pub fn declare_phase_b_objects(module: &mut LlModule) { DOUBLE, &[PTR, DOUBLE, DOUBLE, DOUBLE, I32], ); + // #9708: takes the site's cache SLOT plus the way index to prime. module.declare_function( "js_put_value_set_ic_miss", DOUBLE, - &[DOUBLE, I64, DOUBLE, I32, PTR], + &[DOUBLE, I64, DOUBLE, I32, PTR, I32], ); // #9287: validate-and-store for a constant-key IC hit whose slot word // carries the overflow bit (property lives in the spill buffer). diff --git a/crates/perry-codegen/src/stmt/cached_field_index_return.rs b/crates/perry-codegen/src/stmt/cached_field_index_return.rs index 4cbd480574..709d085f83 100644 --- a/crates/perry-codegen/src/stmt/cached_field_index_return.rs +++ b/crates/perry-codegen/src/stmt/cached_field_index_return.rs @@ -119,7 +119,6 @@ pub(super) fn try_emit_cached_field_index_return( } let cache_name = allocate_shared_cache(ctx, &candidate); - let cache_ref = format!("@{cache_name}"); let base_box = lower_expr(ctx, &Expr::LocalGet(candidate.base_local_id))?; let index_i32 = ctx.block().load(I32, &index_slot); @@ -169,7 +168,15 @@ pub(super) fn try_emit_cached_field_index_return( let gc_flags = ctx.block().load(I8, &gc_flags_ptr); let forwarded_bits = ctx.block().and(I8, &gc_flags, "128"); let not_forwarded = ctx.block().icmp_eq(I8, &forwarded_bits, "0"); + // #9708: the shared cache sits behind a pointer slot that the generic + // property-get miss handler fills on the first prime. Every cache read + // below (`exact_token`, `prefix_meta`, `field_load`) is dominated by this + // edge, so the non-null test joins the header predicate; a site whose + // cache is still unallocated simply takes the normal lowering. + let ic_slot = crate::expr::emit_inline_cache_slot(ctx, &cache_name); + let cache_ref = ic_slot.cache.clone(); let object_ok = ctx.block().and(I1, &is_object, ¬_forwarded); + let object_ok = ctx.block().and(I1, &object_ok, &ic_slot.present); ctx.block() .cond_br(&object_ok, &exact_or_prefix_label, &normal_label); diff --git a/crates/perry-runtime/src/array/mod.rs b/crates/perry-runtime/src/array/mod.rs index 71ac00ae83..99359c9d92 100644 --- a/crates/perry-runtime/src/array/mod.rs +++ b/crates/perry-runtime/src/array/mod.rs @@ -201,7 +201,8 @@ pub(crate) use self::sort::object_prototype_has_index_prop; pub(crate) use self::sort::object_prototype_index_get as sort_object_prototype_index_get; pub(crate) use self::sort::object_prototype_index_get_with_receiver as sort_object_prototype_index_get_with_receiver; pub use self::subclass::{ - array_subclass_dense_snapshot, array_subclass_has_iterator_override, is_array_subclass_instance, + array_subclass_dense_snapshot, array_subclass_has_iterator_override, + is_array_subclass_instance, ArrayLikePicCache, ArrayLikePicCacheSlot, ARRAYLIKE_PIC_WORDS, }; #[cfg(test)] pub(crate) use indexing_support::test_swap_array_index_fast_path_invalidated; diff --git a/crates/perry-runtime/src/array/subclass.rs b/crates/perry-runtime/src/array/subclass.rs index 061309fe17..28074a6a12 100644 --- a/crates/perry-runtime/src/array/subclass.rs +++ b/crates/perry-runtime/src/array/subclass.rs @@ -16,6 +16,11 @@ use crate::value::JSValue; #[path = "subclass_loop_guard.rs"] pub(super) mod loop_guard; +#[path = "subclass_packed_index.rs"] +pub(super) mod packed_index; +#[cfg(test)] +pub(super) use packed_index::js_packed_arraylike_index_get; +pub use packed_index::{ArrayLikePicCache, ArrayLikePicCacheSlot, ARRAYLIKE_PIC_WORDS}; // The loop-guard entry points are exported C symbols; only the unit tests // reach them through Rust paths. #[cfg(test)] @@ -905,8 +910,15 @@ pub(crate) fn array_subclass_fast_length(value: f64) -> Option { /// Array-subclass named-prefix token used by the dense indexed-read IC. The /// payload is published before the identity, and no managed pointer escapes /// into the cache, so moving GC needs neither a root nor a rewrite hook. +/// +/// `cache_slot` is the site's slot (#9708); the cache is allocated only on +/// the publishing path below, so the elements-backed early return never +/// allocates one. #[inline] -pub(crate) fn array_subclass_fast_length_with_ic(value: f64, cache: *mut u64) -> Option { +pub(crate) fn array_subclass_fast_length_with_ic( + value: f64, + cache_slot: *mut crate::value::LengthPicCacheSlot, +) -> Option { if let Some(elements) = validated_object_receiver_for_value(value) .and_then(|r| super::subclass_elements::elements_for_validated(&r)) { @@ -916,7 +928,10 @@ pub(crate) fn array_subclass_fast_length_with_ic(value: f64, cache: *mut u64) -> } let (obj, layout) = dense_layout_for_value(value)?; let result = f64::from_bits(layout_length_value(obj, layout).bits()); - if !cache.is_null() { + if !cache_slot.is_null() { + // SAFETY: a non-null slot is the emitted pointer global or a test's + // stack slot; resolving it allocates the cache on the first prime. + let cache = unsafe { crate::object::pic_slot_resolve(cache_slot) } as *mut u64; let family_token = if crate::object::object_spill_enabled() { unsafe { array_subclass_named_prefix_token_for_slot(obj, layout.length_slot as usize) } } else { @@ -1531,94 +1546,6 @@ fn canonical_u32_index(value: f64) -> Option { .then_some(value as u32) } -/// Unknown-receiver numeric read used by codegen's guarded typed-array miss -/// block. Stable real arrays and Array subclasses terminate here; every other -/// receiver/key keeps the established tag-aware dispatcher as a cold side -/// exit. Keeping that call behind this ABI boundary removes `js_dyn_index_get` -/// from the emitted hot-loop artifact without weakening its semantics. -#[no_mangle] -/// The five optional IC words are -/// scalar layout facts, never heap pointers: -/// `(class_id, ShapeId)`, length slot, element base, dense prefix, inline bound. -/// The emitted hit path reloads the live object/meta/spill pointers, so moving -/// GC never has to trace or rewrite this cache. -pub extern "C" fn js_packed_arraylike_index_get(receiver: f64, index: f64, cache: *mut u64) -> f64 { - if let Some(index_u32) = canonical_u32_index(index) { - let js = JSValue::from_bits(receiver.to_bits()); - if js.is_pointer() { - let raw = js.as_pointer::(); - if let Some(header) = - unsafe { crate::value::addr_class::try_read_gc_header(raw as usize) } - { - if matches!( - header.obj_type, - crate::gc::GC_TYPE_ARRAY | crate::gc::GC_TYPE_LAZY_ARRAY - ) { - return crate::array::js_array_get_f64( - raw as *const crate::array::ArrayHeader, - index_u32, - ); - } - if header.obj_type == crate::gc::GC_TYPE_OBJECT - && header.gc_flags & crate::gc::GC_FLAG_FORWARDED == 0 - { - let obj = raw.cast::(); - // Elements-backed instance: an in-bounds non-hole element - // answers directly; a hole continues to the complete - // dispatcher (prototype chain). - let elements = unsafe { super::subclass_elements::elements_of(obj) }; - if !elements.is_null() { - if let Some(value) = - super::subclass_elements::elements_index_get(elements, index_u32) - { - return value; - } - } else if let Some(layout) = dense_layout_for_validated_object(obj) { - // The codegen hit path handles both inline and - // object-owned spill slots. In spill mode, publish a - // class-wide dense-tail identity when the owner has - // proved one. Exact push/pop transitions preserve - // that move-stable token, so a lifecycle loop does not - // miss once for every historical tail ShapeId. The - // cached dense-prefix word remains the admitted high - // water mark: a generic `length` grow beyond it still - // side-exits and re-establishes the complete proof. - if !cache.is_null() && crate::object::object_spill_enabled() { - let family_token = unsafe { - array_subclass_named_prefix_token_for_slot( - obj, - layout.length_slot as usize, - ) - }; - unsafe { - // GC_STORE_AUDIT(POINTER_FREE): generated IC - // words are scalar layout facts, not heap edges. - cache.add(1).write(layout.length_slot as u64); - cache.add(2).write(layout.element_base as u64); - cache.add(3).write(layout.dense_prefix_len as u64); - cache.add(4).write(layout.live_inline_slots as u64); - cache.write(if family_token != 0 { - family_token - } else { - dense_cache_key((*obj).class_id, (*obj).parent_class_id) - }); - } - } - if let Some(value) = dense_index_get_with_layout(obj, layout, index_u32) { - return value; - } - } - } - } - } - } - crate::value::js_dyn_index_get(receiver, index) -} -#[cfg(feature = "keepalive-anchors")] -#[used] -static KEEP_JS_PACKED_ARRAYLIKE_INDEX_GET: extern "C" fn(f64, f64, *mut u64) -> f64 = - js_packed_arraylike_index_get; - /// True when `class_id` is a user class that extends `Array` (the reserved /// parent id `0xFFFF0024` appears in its class chain), i.e. `class X extends /// Array`. Such instances are plain `ObjectHeader`s, so the array-like engines diff --git a/crates/perry-runtime/src/array/subclass_packed_index.rs b/crates/perry-runtime/src/array/subclass_packed_index.rs new file mode 100644 index 0000000000..2d50e3eef1 --- /dev/null +++ b/crates/perry-runtime/src/array/subclass_packed_index.rs @@ -0,0 +1,121 @@ +//! The packed array-like numeric read codegen's guarded typed-array miss +//! block calls, with its per-site index cache (#9708: allocated on the first +//! prime, behind an emitted pointer slot). +//! +//! Child module of `subclass.rs`, split out to stay under the 2,000-line file +//! gate; `use super::*` keeps the parent's private layout helpers reachable. + +use super::*; + +/// Words in a per-site packed array-like index cache: `(identity, length +/// slot, element base, dense prefix, inline bound)`. +pub const ARRAYLIKE_PIC_WORDS: usize = 5; +/// A per-site packed array-like index cache, as the emitted slot resolves it. +pub type ArrayLikePicCache = [u64; ARRAYLIKE_PIC_WORDS]; +/// The emitted `@perry_ic_N = private global ptr null` for such a site: null +/// until the site's first priming read (#9708). +pub type ArrayLikePicCacheSlot = *mut ArrayLikePicCache; + +/// Unknown-receiver numeric read used by codegen's guarded typed-array miss +/// block. Stable real arrays and Array subclasses terminate here; every other +/// receiver/key keeps the established tag-aware dispatcher as a cold side +/// exit. Keeping that call behind this ABI boundary removes `js_dyn_index_get` +/// from the emitted hot-loop artifact without weakening its semantics. +/// +/// The five optional IC words are +/// scalar layout facts, never heap pointers: +/// `(class_id, ShapeId)`, length slot, element base, dense prefix, inline bound. +/// The emitted hit path reloads the live object/meta/spill pointers, so moving +/// GC never has to trace or rewrite this cache. +/// +/// `cache_slot` is the site's [`ArrayLikePicCacheSlot`] address; the cache is +/// allocated on the first prime (#9708), so a site whose receivers are plain +/// Arrays or elements-backed instances never allocates one. +#[no_mangle] +pub extern "C" fn js_packed_arraylike_index_get( + receiver: f64, + index: f64, + cache_slot: *mut ArrayLikePicCacheSlot, +) -> f64 { + if let Some(index_u32) = canonical_u32_index(index) { + let js = JSValue::from_bits(receiver.to_bits()); + if js.is_pointer() { + let raw = js.as_pointer::(); + if let Some(header) = + unsafe { crate::value::addr_class::try_read_gc_header(raw as usize) } + { + if matches!( + header.obj_type, + crate::gc::GC_TYPE_ARRAY | crate::gc::GC_TYPE_LAZY_ARRAY + ) { + return crate::array::js_array_get_f64( + raw as *const crate::array::ArrayHeader, + index_u32, + ); + } + if header.obj_type == crate::gc::GC_TYPE_OBJECT + && header.gc_flags & crate::gc::GC_FLAG_FORWARDED == 0 + { + let obj = raw.cast::(); + // Elements-backed instance: an in-bounds non-hole element + // answers directly; a hole continues to the complete + // dispatcher (prototype chain). + let elements = unsafe { crate::array::subclass_elements::elements_of(obj) }; + if !elements.is_null() { + if let Some(value) = + crate::array::subclass_elements::elements_index_get(elements, index_u32) + { + return value; + } + } else if let Some(layout) = dense_layout_for_validated_object(obj) { + // The codegen hit path handles both inline and + // object-owned spill slots. In spill mode, publish a + // class-wide dense-tail identity when the owner has + // proved one. Exact push/pop transitions preserve + // that move-stable token, so a lifecycle loop does not + // miss once for every historical tail ShapeId. The + // cached dense-prefix word remains the admitted high + // water mark: a generic `length` grow beyond it still + // side-exits and re-establishes the complete proof. + if !cache_slot.is_null() && crate::object::object_spill_enabled() { + // SAFETY: a non-null slot is the emitted pointer + // global or a test's stack slot. + let cache = + unsafe { crate::object::pic_slot_resolve(cache_slot) } as *mut u64; + let family_token = unsafe { + array_subclass_named_prefix_token_for_slot( + obj, + layout.length_slot as usize, + ) + }; + unsafe { + // GC_STORE_AUDIT(POINTER_FREE): generated IC + // words are scalar layout facts, not heap edges. + cache.add(1).write(layout.length_slot as u64); + cache.add(2).write(layout.element_base as u64); + cache.add(3).write(layout.dense_prefix_len as u64); + cache.add(4).write(layout.live_inline_slots as u64); + cache.write(if family_token != 0 { + family_token + } else { + dense_cache_key((*obj).class_id, (*obj).parent_class_id) + }); + } + } + if let Some(value) = dense_index_get_with_layout(obj, layout, index_u32) { + return value; + } + } + } + } + } + } + crate::value::js_dyn_index_get(receiver, index) +} +#[cfg(feature = "keepalive-anchors")] +#[used] +static KEEP_JS_PACKED_ARRAYLIKE_INDEX_GET: extern "C" fn( + f64, + f64, + *mut ArrayLikePicCacheSlot, +) -> f64 = js_packed_arraylike_index_get; diff --git a/crates/perry-runtime/src/array/subclass_tests.rs b/crates/perry-runtime/src/array/subclass_tests.rs index fc2f186f7d..b540f8ff4c 100644 --- a/crates/perry-runtime/src/array/subclass_tests.rs +++ b/crates/perry-runtime/src/array/subclass_tests.rs @@ -366,9 +366,10 @@ fn array_subclass_length_ic_publishes_only_scalar_exact_or_family_facts() { let receiver = crate::value::js_nanbox_pointer(obj as i64); crate::node_stream::js_array_subclass_init(receiver, 0.0); - let mut cache = [0_u64; 3]; + let mut cache: crate::value::LengthPicCache = [0; crate::value::LENGTH_PIC_WORDS]; + let mut cache_slot: crate::value::LengthPicCacheSlot = &mut cache; assert_eq!( - array_subclass_fast_length_with_ic(receiver, cache.as_mut_ptr()), + array_subclass_fast_length_with_ic(receiver, &mut cache_slot), Some(0.0) ); if crate::object::object_spill_enabled() { @@ -392,7 +393,7 @@ fn array_subclass_length_ic_publishes_only_scalar_exact_or_family_facts() { js_array_push_f64(obj as *mut ArrayHeader, 11.0); assert_eq!( - array_subclass_fast_length_with_ic(receiver, cache.as_mut_ptr()), + array_subclass_fast_length_with_ic(receiver, &mut cache_slot), Some(1.0) ); if crate::object::object_spill_enabled() { @@ -677,7 +678,8 @@ fn array_subclass_named_prefix_token_survives_only_exact_numeric_tail_transition js_array_push_f64(obj as *mut ArrayHeader, 22.0); let mask_key = crate::string::js_string_from_bytes(b"mask".as_ptr(), 4); let mut cache = [0i64; crate::object::PIC_CACHE_WORDS]; - crate::object::js_object_get_field_ic_miss(obj, mask_key, &mut cache); + let mut cache_slot: crate::object::PicCacheSlot = &mut cache; + crate::object::js_object_get_field_ic_miss(obj, mask_key, &mut cache_slot); let token = cache[2] as u64; assert_ne!(token, 0); assert!(unsafe { array_subclass_named_prefix_token_matches_class(obj, class_id) }); @@ -691,9 +693,10 @@ fn array_subclass_named_prefix_token_survives_only_exact_numeric_tail_transition "the IC miss must publish the same owner-side token it caches" ); - let mut index_ic = [0u64; 5]; + let mut index_ic: super::ArrayLikePicCache = [0; super::ARRAYLIKE_PIC_WORDS]; + let mut index_ic_slot: super::ArrayLikePicCacheSlot = &mut index_ic; assert_eq!( - js_packed_arraylike_index_get(receiver, 0.0, index_ic.as_mut_ptr()), + js_packed_arraylike_index_get(receiver, 0.0, &mut index_ic_slot), 11.0 ); assert_eq!( @@ -711,8 +714,9 @@ fn array_subclass_named_prefix_token_survives_only_exact_numeric_tail_transition let zero_key = crate::string::js_string_from_bytes(b"0".as_ptr(), 1); let mut element_cache = [0i64; crate::object::PIC_CACHE_WORDS]; + let mut element_cache_slot: crate::object::PicCacheSlot = &mut element_cache; assert_eq!( - crate::object::js_object_get_field_ic_miss(obj, zero_key, &mut element_cache), + crate::object::js_object_get_field_ic_miss(obj, zero_key, &mut element_cache_slot), 11.0 ); assert_eq!( @@ -857,7 +861,8 @@ fn empty_array_subclass_named_prefix_token_survives_warm_tail_cycle() { let change_key = crate::string::js_string_from_bytes(b"change".as_ptr(), 6); let mut cache = [0i64; crate::object::PIC_CACHE_WORDS]; - let via_ic = crate::object::js_object_get_field_ic_miss(obj, change_key, &mut cache); + let mut cache_slot: crate::object::PicCacheSlot = &mut cache; + let via_ic = crate::object::js_object_get_field_ic_miss(obj, change_key, &mut cache_slot); let via_ladder = crate::object::js_object_get_field_by_name_f64(obj, change_key); assert_eq!(via_ic.to_bits(), via_ladder.to_bits()); let token = cache[2] as u64; diff --git a/crates/perry-runtime/src/gc/arena_right_size.rs b/crates/perry-runtime/src/gc/arena_right_size.rs new file mode 100644 index 0000000000..adcacd15c5 --- /dev/null +++ b/crates/perry-runtime/src/gc/arena_right_size.rs @@ -0,0 +1,316 @@ +//! Arena capacity right-sizing for idle heaps. +//! +//! Reclaiming dead objects and returning arena capacity are deliberately two +//! different operations. General-arena blocks are released only after two +//! full collection observations: the first resets a proven-empty block and +//! the second proves that the mutator did not reuse it. The idle reducer used +//! to subtract its own fulls from its activity clock, so a burst followed by +//! complete silence received one full and then parked forever. Empty blocks +//! stopped at `dead_cycles == 1`, even when live bytes occupied a small +//! fraction of reserved arena capacity. +//! +//! This module turns sustained post-collection slack into a bounded debt that +//! [`super::idle_reclaim`] may service without new mutator activity: +//! +//! * capacity must be above [`ARENA_RIGHT_SIZE_MIN_CAPACITY_BYTES`]; +//! * live bytes must be at or below [`ARENA_RIGHT_SIZE_TRIGGER_PCT`] percent +//! for [`ARENA_RIGHT_SIZE_LOW_COLLECTIONS`] consecutive collections; +//! * the episode asks for enough idle fulls to reach +//! [`ARENA_RIGHT_SIZE_FULL_OBSERVATIONS`] full observations, counting fulls +//! already present in that low-utilization streak; +//! * it stops early in the target band and is bounded even when fragmentation +//! prevents a block release. +//! +//! The start and stop bands are intentionally different. After an episode +//! stops below the re-arm watermark, another cannot begin until utilization +//! rises to [`ARENA_RIGHT_SIZE_REARM_PCT`] percent or reserved capacity grows +//! materially. That hysteresis is the burst-then-idle-then-burst protection: +//! stable low occupancy cannot buy a full every timer interval, while a real +//! new peak can earn another right-size episode. + +use super::*; +use std::sync::atomic::{AtomicU64, Ordering}; + +/// Small heaps do not buy extra whole-heap work merely to save a few blocks. +pub const ARENA_RIGHT_SIZE_MIN_CAPACITY_BYTES: usize = 32 * 1024 * 1024; + +/// Open a right-size episode at or below this live/capacity percentage. +pub const ARENA_RIGHT_SIZE_TRIGGER_PCT: usize = 50; + +/// Stop an episode once live bytes reach this share of reserved capacity. +pub const ARENA_RIGHT_SIZE_TARGET_PCT: usize = 60; + +/// A completed episode below this utilization stays disarmed until the heap +/// grows materially. This gap from the trigger is the utilization hysteresis. +pub const ARENA_RIGHT_SIZE_REARM_PCT: usize = 70; + +/// Number of consecutive low-utilization collection results required. +pub const ARENA_RIGHT_SIZE_LOW_COLLECTIONS: u32 = 2; + +/// Full observations needed for the arena's two-cycle empty-block release. +pub const ARENA_RIGHT_SIZE_FULL_OBSERVATIONS: u8 = 2; + +/// Capacity growth that re-arms a disarmed episode is at least this many +/// bytes, as well as at least [`ARENA_RIGHT_SIZE_REARM_GROWTH_PCT`] percent of +/// the capacity at which it disarmed. +pub const ARENA_RIGHT_SIZE_REARM_GROWTH_MIN_BYTES: usize = 8 * 1024 * 1024; + +/// See [`ARENA_RIGHT_SIZE_REARM_GROWTH_MIN_BYTES`]. +pub const ARENA_RIGHT_SIZE_REARM_GROWTH_PCT: usize = 25; + +#[derive(Clone, Copy, Debug, Default, Eq, PartialEq)] +pub(super) struct ArenaUsage { + pub(super) live_bytes: usize, + pub(super) capacity_bytes: usize, +} + +#[derive(Default)] +struct ArenaRightSizeState { + /// Consecutive post-collection samples in the trigger band. + low_collections: u32, + /// Full samples among `low_collections`, capped at the required count. + low_full_observations: u8, + /// Full collections still owed by the active episode. + fulls_remaining: u8, + /// A bounded episode finished without returning to the re-arm band. + disarmed: bool, + /// Capacity at disarm, used to recognize a later, material new peak. + disarmed_capacity_bytes: usize, + /// Capacity at the beginning of an active episode. + episode_start_capacity_bytes: usize, + /// Last post-collection sample, for diagnostics. + last_usage: ArenaUsage, +} + +thread_local! { + static STATE: RefCell = + RefCell::new(ArenaRightSizeState::default()); + #[cfg(test)] + static TEST_USAGE: Cell> = const { Cell::new(None) }; +} + +static EPISODES: AtomicU64 = AtomicU64::new(0); +static STARTS: AtomicU64 = AtomicU64::new(0); +static RELEASED_CAPACITY_BYTES: AtomicU64 = AtomicU64::new(0); + +/// Sustained-low-utilization episodes opened in this process. +pub fn arena_right_size_episodes() -> u64 { + EPISODES.load(Ordering::Relaxed) +} + +/// Idle fulls started specifically to service arena right-size debt. +pub fn arena_right_size_starts() -> u64 { + STARTS.load(Ordering::Relaxed) +} + +/// Reserved arena bytes removed over completed right-size episodes. +pub fn arena_right_size_released_capacity_bytes() -> u64 { + RELEASED_CAPACITY_BYTES.load(Ordering::Relaxed) +} + +#[inline] +fn utilization_at_most(usage: ArenaUsage, pct: usize) -> bool { + (usage.live_bytes as u128) * 100 <= (usage.capacity_bytes as u128) * (pct as u128) +} + +#[inline] +fn utilization_at_least(usage: ArenaUsage, pct: usize) -> bool { + (usage.live_bytes as u128) * 100 >= (usage.capacity_bytes as u128) * (pct as u128) +} + +fn in_trigger_band(usage: ArenaUsage) -> bool { + usage.capacity_bytes > ARENA_RIGHT_SIZE_MIN_CAPACITY_BYTES + && utilization_at_most(usage, ARENA_RIGHT_SIZE_TRIGGER_PCT) +} + +fn at_target(usage: ArenaUsage) -> bool { + usage.capacity_bytes <= ARENA_RIGHT_SIZE_MIN_CAPACITY_BYTES + || utilization_at_least(usage, ARENA_RIGHT_SIZE_TARGET_PCT) +} + +fn materially_regrew(usage: ArenaUsage, disarmed_capacity_bytes: usize) -> bool { + let relative = disarmed_capacity_bytes.saturating_mul(ARENA_RIGHT_SIZE_REARM_GROWTH_PCT) / 100; + let required = relative.max(ARENA_RIGHT_SIZE_REARM_GROWTH_MIN_BYTES); + usage.capacity_bytes.saturating_sub(disarmed_capacity_bytes) >= required +} + +fn rearm_reached(usage: ArenaUsage, disarmed_capacity_bytes: usize) -> bool { + utilization_at_least(usage, ARENA_RIGHT_SIZE_REARM_PCT) + || materially_regrew(usage, disarmed_capacity_bytes) +} + +fn current_usage(live_bytes: usize) -> ArenaUsage { + #[cfg(test)] + if let Some(usage) = TEST_USAGE.with(Cell::get) { + return usage; + } + ArenaUsage { + live_bytes, + capacity_bytes: crate::arena::arena_total_bytes(), + } +} + +fn reset_low_streak(st: &mut ArenaRightSizeState) { + st.low_collections = 0; + st.low_full_observations = 0; +} + +fn finish_episode(st: &mut ArenaRightSizeState, usage: ArenaUsage) { + let released = st + .episode_start_capacity_bytes + .saturating_sub(usage.capacity_bytes); + RELEASED_CAPACITY_BYTES.fetch_add(released as u64, Ordering::Relaxed); + st.fulls_remaining = 0; + st.episode_start_capacity_bytes = 0; + reset_low_streak(st); + st.disarmed = !utilization_at_least(usage, ARENA_RIGHT_SIZE_REARM_PCT); + st.disarmed_capacity_bytes = if st.disarmed { usage.capacity_bytes } else { 0 }; +} + +/// Observe the exact post-collection live census and current reserved block +/// capacity. `full` is true only when this collection supplied a whole-heap +/// sweep observation; copied and non-moving minors are still utilization +/// samples but cannot satisfy the two full observations by themselves. +pub(super) fn note_collection_finished(live_bytes: usize, full: bool) { + let usage = current_usage(live_bytes); + STATE.with(|state| { + let mut st = state.borrow_mut(); + st.last_usage = usage; + + if st.disarmed { + if !rearm_reached(usage, st.disarmed_capacity_bytes) { + return; + } + st.disarmed = false; + st.disarmed_capacity_bytes = 0; + reset_low_streak(&mut st); + } + + if st.fulls_remaining != 0 { + if at_target(usage) { + finish_episode(&mut st, usage); + return; + } + if full { + st.fulls_remaining = st.fulls_remaining.saturating_sub(1); + if st.fulls_remaining == 0 { + // Fragmentation or the protected recent-block window can + // make two full observations unable to hit the target. + // End the bounded episode instead of collecting forever. + finish_episode(&mut st, usage); + } + } + return; + } + + if !in_trigger_band(usage) { + reset_low_streak(&mut st); + return; + } + + st.low_collections = st + .low_collections + .saturating_add(1) + .min(ARENA_RIGHT_SIZE_LOW_COLLECTIONS); + if full { + st.low_full_observations = st + .low_full_observations + .saturating_add(1) + .min(ARENA_RIGHT_SIZE_FULL_OBSERVATIONS); + } + if st.low_collections < ARENA_RIGHT_SIZE_LOW_COLLECTIONS { + return; + } + + st.episode_start_capacity_bytes = usage.capacity_bytes; + st.fulls_remaining = + ARENA_RIGHT_SIZE_FULL_OBSERVATIONS.saturating_sub(st.low_full_observations); + reset_low_streak(&mut st); + EPISODES.fetch_add(1, Ordering::Relaxed); + if st.fulls_remaining == 0 { + finish_episode(&mut st, usage); + } + }); +} + +/// Whether the idle reducer may start a full without new mutator collection +/// activity. Quiet-time and rate gates remain the reducer's responsibility. +pub(super) fn owed() -> bool { + STATE.with(|state| state.borrow().fulls_remaining != 0) +} + +/// Record that the idle reducer successfully opened a full for this debt. +pub(super) fn note_started() { + STARTS.fetch_add(1, Ordering::Relaxed); +} + +pub(super) fn snapshot() -> (u32, u8, bool, ArenaUsage) { + STATE.with(|state| { + let st = state.borrow(); + ( + st.low_collections, + st.fulls_remaining, + st.disarmed, + st.last_usage, + ) + }) +} + +/// `PERRY_GC_DIAG=1` exit line. +pub(super) fn emit_diag() { + let (low_collections, fulls_remaining, disarmed, usage) = snapshot(); + eprintln!( + "[gc-arena-right-size] episodes={} starts={} released_capacity_bytes={} \ + low_collections={} fulls_remaining={} disarmed={} arena_live={} arena_capacity={}", + arena_right_size_episodes(), + arena_right_size_starts(), + arena_right_size_released_capacity_bytes(), + low_collections, + fulls_remaining, + disarmed, + usage.live_bytes, + usage.capacity_bytes, + ); +} + +#[cfg(test)] +pub(super) mod test_support { + use super::*; + + pub(crate) fn set_test_usage(usage: Option) { + TEST_USAGE.with(|cell| cell.set(usage)); + } + + pub(crate) fn observe(usage: ArenaUsage, full: bool) { + set_test_usage(Some(usage)); + note_collection_finished(usage.live_bytes, full); + } + + pub(crate) fn reset_state() { + STATE.with(|state| *state.borrow_mut() = ArenaRightSizeState::default()); + } + + pub(crate) fn state_snapshot() -> (u32, u8, bool, ArenaUsage) { + snapshot() + } + + pub(crate) struct ArenaRightSizeTestGuard; + + impl ArenaRightSizeTestGuard { + pub(crate) fn new() -> Self { + reset_state(); + set_test_usage(Some(ArenaUsage { + live_bytes: ARENA_RIGHT_SIZE_MIN_CAPACITY_BYTES, + capacity_bytes: ARENA_RIGHT_SIZE_MIN_CAPACITY_BYTES, + })); + Self + } + } + + impl Drop for ArenaRightSizeTestGuard { + fn drop(&mut self) { + set_test_usage(None); + reset_state(); + } + } +} diff --git a/crates/perry-runtime/src/gc/barrier/mod.rs b/crates/perry-runtime/src/gc/barrier/mod.rs index 34f058e629..35c37c0454 100644 --- a/crates/perry-runtime/src/gc/barrier/mod.rs +++ b/crates/perry-runtime/src/gc/barrier/mod.rs @@ -960,6 +960,44 @@ pub(super) unsafe fn plausible_arena_user_ptr_header( } } +/// A plausible ARENA object whose header carries `GC_FLAG_FORWARDED` — i.e. an +/// array-growth forwarding stub (#6228: growth installs a PERMANENT stub at the +/// pre-grow address, and a live slot can keep pointing directly at it because +/// references are never rewritten). +/// +/// This is [`plausible_arena_user_ptr_header`] with the FORWARDED test +/// inverted: same alignment / `obj_type` / size / `GC_FLAG_ARENA` gate, but the +/// header MUST be forwarded. `GC_FLAG_ARENA` is what separates a genuine stub +/// from an off-heap region whose bytes coincidentally set FORWARDED (#8040) — +/// the census (`ValidPointerSetBuilder::record_arena_header`) admits exactly +/// these real arena allocations, so the classifier must too (#9717). The +/// forwarding TARGET is deliberately NOT validated here: the caller follows it +/// through `trace_one_worklist_header`, which re-checks membership before +/// marking it, so a garbage target simply stops the walk. +#[inline] +pub(super) unsafe fn plausible_forwarded_arena_stub( + header: *mut GcHeader, +) -> Option<*mut GcHeader> { + if header.is_null() { + return None; + } + if !(header as usize).is_multiple_of(std::mem::align_of::()) { + return None; + } + let obj_type = (*header).obj_type; + let size = (*header).size as usize; + if gc_type_info(obj_type).is_none() + || size < GC_HEADER_SIZE + || size as u64 > (1u64 << 34) + || (*header).gc_flags & GC_FLAG_ARENA == 0 + || (*header).gc_flags & GC_FLAG_FORWARDED == 0 + { + None + } else { + Some(header) + } +} + pub(super) fn current_heap_header_for_user_ptr( user_ptr: usize, valid_ptrs: Option<&ValidPointerSet>, diff --git a/crates/perry-runtime/src/gc/census.rs b/crates/perry-runtime/src/gc/census.rs index 5aeaad82a0..891035533f 100644 --- a/crates/perry-runtime/src/gc/census.rs +++ b/crates/perry-runtime/src/gc/census.rs @@ -559,6 +559,7 @@ fn side_tables() -> Vec { rows.extend(crate::object::shapes::shape_table_census()); rows.extend(crate::object::class_registry_census()); rows.extend(crate::object::object_tables_census()); + rows.extend(crate::object::pic_slot_census()); rows.extend(super::roots::stack_map_index_census()); let (slots, bytes) = crate::string::intern_table_census(); rows.push(("string.intern_table(fixed)", slots, bytes)); diff --git a/crates/perry-runtime/src/gc/copying.rs b/crates/perry-runtime/src/gc/copying.rs index 05b42a59f6..6c53d857e8 100644 --- a/crates/perry-runtime/src/gc/copying.rs +++ b/crates/perry-runtime/src/gc/copying.rs @@ -1739,7 +1739,7 @@ pub(super) fn run_copied_minor_attempt( // flip and the new active survivor holds only the copies, so from-space // live == from-space high-water. crate::arena::record_arena_live_census(arena_live_bytes, None); - note_collection_finished_arena_occupancy(); + note_collection_finished_arena_occupancy(false); // The same argument one trigger over: a young generation that did not die // is a heap growing by LIVE data, so arena-growth pacing must not read that // growth as garbage accumulating. Fed after publishing the census so the diff --git a/crates/perry-runtime/src/gc/cycle.rs b/crates/perry-runtime/src/gc/cycle.rs index 7e4a8ce01d..33e0252f8c 100644 --- a/crates/perry-runtime/src/gc/cycle.rs +++ b/crates/perry-runtime/src/gc/cycle.rs @@ -1710,7 +1710,10 @@ impl GcCycleState { // per-object finalizers. Minor traces never mark the old // generation, so deadness there is only trusted for // untenured nursery headers. - .with_dead_collection_finalize(full_trace), + .with_dead_collection_finalize( + full_trace, + full_trace && !self.progress_kind.is_budgeted(), + ), ); } let done = self @@ -1956,7 +1959,7 @@ impl GcCycleState { // #7865: arena-growth pacing tests a POST-collection occupancy, which // is the same kind of quantity as its post-full baseline. Recorded here // rather than per-kind because this is the one site both kinds reach. - super::policy::note_collection_finished_arena_occupancy(); + super::policy::note_collection_finished_arena_occupancy(self.minor.is_none()); if self.minor.is_none() { finish_full_old_reclaim_baseline(); } diff --git a/crates/perry-runtime/src/gc/dead_owner.rs b/crates/perry-runtime/src/gc/dead_owner.rs index 35e5688de4..2c617264af 100644 --- a/crates/perry-runtime/src/gc/dead_owner.rs +++ b/crates/perry-runtime/src/gc/dead_owner.rs @@ -203,17 +203,27 @@ fn owner_type_matches(header: &GcHeader, expected_obj_type: Option) -> bool /// Post-trace fan-out (full mark-sweep + fallback minor). Runs at sweep /// entry, before any header is finalized or freed, so deadness probes read /// intact headers. -pub(super) fn prune_dead_owner_side_tables_post_trace(full_trace: bool) { +pub(super) fn prune_dead_owner_side_tables_post_trace( + full_trace: bool, + synchronous_full_trace: bool, +) { + debug_assert!(!synchronous_full_trace || full_trace); if full_trace { + // Rebuild every restamping table's ownership before consulting the + // complete receiver census. An evicted cache entry releases its id in + // this same post-trace window. + crate::object::shape_carriers::recompute_after_full_trace(); + if synchronous_full_trace { + crate::object::shapes::prune_uncarried_shape_descriptors_after_full_trace(); + } // #8112: a full trace enumerated every live object, so the old-carrier // notes it accumulated are exactly the shapes old objects still carry. // Adopting them here is what lets the gate SHED a shape — minors only // ever add notes, so without this the table's root set would grow // monotonically and no keys array would ever be reclaimed again. + // #9726: this also clears the all-generation carried note after the + // synchronous prune consumed it. Budgeted traces only clear the note. crate::object::shapes::rotate_old_carrier_epoch_after_full_trace(); - // The same rule for the array-tail transition caches: their carrier - // bits are exact only when rebuilt from live occupancy. - crate::object::array_tail_transition::recompute_cache_carriers_after_full_trace(); } let probe = PostTraceProbe::new(full_trace); fan_out( diff --git a/crates/perry-runtime/src/gc/idle_reclaim.rs b/crates/perry-runtime/src/gc/idle_reclaim.rs index 709658c86b..66d800961b 100644 --- a/crates/perry-runtime/src/gc/idle_reclaim.rs +++ b/crates/perry-runtime/src/gc/idle_reclaim.rs @@ -48,10 +48,12 @@ //! //! Three gates, all O(1), evaluated at every park: //! -//! 1. **Activity.** At least `2^backoff` collections the reducer did not start -//! itself have completed since its last full. A collection is the one -//! signal that the mutator allocated enough to matter; a heap nobody has -//! touched since the last idle full has nothing new for another to find. +//! 1. **Activity or arena debt.** Normally at least `2^backoff` collections +//! the reducer did not start itself have completed since its last full. A +//! collection is the signal that the mutator allocated enough to matter. +//! The exception is a bounded [`super::arena_right_size`] episode: arena +//! blocks need two full observations before their mappings can be returned, +//! and an idle heap cannot create the second through mutator activity. //! 2. **Quiet.** At least [`IDLE_RECLAIM_QUIET_MS`] since the last such //! collection was observed — a burst still in progress collects every few //! hundred milliseconds and must not be interleaved with a whole-heap mark. @@ -137,6 +139,25 @@ pub(crate) enum ParkVerdict { Park(u64), } +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +enum StartReason { + /// The existing memory-reducer signal: the mutator completed enough + /// collections and then went quiet. + Activity, + /// Sustained arena slack still needs full observations before empty blocks + /// can be returned, even though the mutator has done nothing new. + ArenaRightSize, +} + +impl StartReason { + fn as_str(self) -> &'static str { + match self { + StartReason::Activity => "activity", + StartReason::ArenaRightSize => "arena_right_size", + } + } +} + #[derive(Default)] struct IdleReclaimState { /// Collections not started by the reducer, as of the last observation. @@ -319,7 +340,7 @@ fn old_gen_occupancy() -> usize { /// Observe the external-collection counter and decide whether a reducer full /// is owed right now. Pure bookkeeping — no collector state is touched. -fn should_start(now: u64) -> bool { +fn start_reason(now: u64) -> Option { // Read the counters BEFORE taking the state borrow: `external_collections` // borrows the same cell. let external = external_collections(); @@ -330,22 +351,29 @@ fn should_start(now: u64) -> bool { st.last_seen_external = external; st.last_external_change_ms = now; } - let since_attempt = external.saturating_sub(st.external_at_last_attempt); - if since_attempt < (1u64 << st.backoff_shift) { - return false; - } if now.saturating_sub(st.last_external_change_ms) < IDLE_RECLAIM_QUIET_MS { - return false; + return None; } if st.attempts > 0 && now.saturating_sub(st.last_attempt_ms) < IDLE_RECLAIM_MIN_INTERVAL_MS { - return false; + return None; + } + // Arena capacity release itself needs multiple full observations. Once + // sustained low utilization has created that bounded debt, requiring + // another mutator collection here recreates #9709's deadlock: the + // mutator is idle precisely because there is no more activity. + if super::arena_right_size::owed() { + return Some(StartReason::ArenaRightSize); + } + let since_attempt = external.saturating_sub(st.external_at_last_attempt); + if since_attempt < (1u64 << st.backoff_shift) { + return None; } - true + Some(StartReason::Activity) }) } -fn note_started(now: u64) { +fn note_started(now: u64, reason: StartReason) { STATE.with(|s| { let mut st = s.borrow_mut(); st.attempts += 1; @@ -353,10 +381,23 @@ fn note_started(now: u64) { st.external_at_last_attempt = st.last_seen_external; st.old_in_use_at_start = old_gen_occupancy(); ATTEMPTS.fetch_add(1, Ordering::Relaxed); + if reason == StartReason::ArenaRightSize { + super::arena_right_size::note_started(); + } if gc_diag_enabled() { + let (_, right_size_fulls_remaining, _, usage) = super::arena_right_size::snapshot(); eprintln!( - "[gc-idle-reclaim] start attempt={} external_collections={} backoff_shift={} old_in_use={}", - st.attempts, st.last_seen_external, st.backoff_shift, st.old_in_use_at_start + "[gc-idle-reclaim] start attempt={} reason={} external_collections={} \ + backoff_shift={} old_in_use={} arena_live={} arena_capacity={} \ + right_size_fulls_remaining={}", + st.attempts, + reason.as_str(), + st.last_seen_external, + st.backoff_shift, + st.old_in_use_at_start, + usage.live_bytes, + usage.capacity_bytes, + right_size_fulls_remaining, ); } }); @@ -491,14 +532,14 @@ pub(crate) fn park_hook(budget_ms: u64) -> ParkVerdict { if super::idle_compact::maybe_compact(now) { return ParkVerdict::Resume; } - if !should_start(now) { + let Some(reason) = start_reason(now) else { return ParkVerdict::Park(budget_ms); - } + }; if !policy::gc_idle_reclaim_try_start() { START_BLOCKED.fetch_add(1, Ordering::Relaxed); return ParkVerdict::Park(budget_ms); } - note_started(now); + note_started(now, reason); drive_active_cycle(deadline) } @@ -580,6 +621,14 @@ pub(super) mod test_support { // before the hook runs. crate::event_pump::clear_main_thread_notified_for_test(); reset_state(); + super::super::arena_right_size::test_support::reset_state(); + super::super::arena_right_size::test_support::set_test_usage(Some( + super::super::arena_right_size::ArenaUsage { + live_bytes: super::super::arena_right_size::ARENA_RIGHT_SIZE_MIN_CAPACITY_BYTES, + capacity_bytes: + super::super::arena_right_size::ARENA_RIGHT_SIZE_MIN_CAPACITY_BYTES, + }, + )); set_test_now_ms(Some(now_ms)); set_test_enabled(Some(true)); set_test_max_slices(None); @@ -597,6 +646,8 @@ pub(super) mod test_support { set_test_slice_us(None); set_test_work_charge_ms(None); reset_state(); + super::super::arena_right_size::test_support::set_test_usage(None); + super::super::arena_right_size::test_support::reset_state(); } } } diff --git a/crates/perry-runtime/src/gc/layout_slot_visit.rs b/crates/perry-runtime/src/gc/layout_slot_visit.rs index c86d890569..6673909569 100644 --- a/crates/perry-runtime/src/gc/layout_slot_visit.rs +++ b/crates/perry-runtime/src/gc/layout_slot_visit.rs @@ -51,6 +51,12 @@ pub(super) unsafe fn visit_gc_layout_slot_descriptors( // needed: without it the verifier aborts on a `slot_page_ever_dirty=false` // old→young edge through this word. let shape_keys_edge = if (*header).obj_type == GC_TYPE_OBJECT { + // #9726: unlike the minor-rooting gate below, full-trace descriptor + // liveness is generation-blind. Every reachable shaped receiver must + // note the exact id it carries before synchronous-full pruning. + if full_trace_active() { + crate::object::shapes::note_full_trace_carrier(child_slots.object_shape); + } // A receiver the minor will not enumerate for itself arms the table's // ephemeron gate. The test is "not in the nursery", not "in old-gen": // a `gc_malloc`'d large object and an immortal bootstrap resident are diff --git a/crates/perry-runtime/src/gc/mod.rs b/crates/perry-runtime/src/gc/mod.rs index 9041e56360..81c6aa9ff7 100644 --- a/crates/perry-runtime/src/gc/mod.rs +++ b/crates/perry-runtime/src/gc/mod.rs @@ -47,6 +47,10 @@ mod heap_budget; pub(crate) use heap_budget::*; mod pressure; pub use pressure::*; +mod arena_right_size; +pub use arena_right_size::{ + arena_right_size_episodes, arena_right_size_released_capacity_bytes, arena_right_size_starts, +}; mod idle_compact; mod idle_reclaim; pub use idle_compact::{ @@ -1355,7 +1359,7 @@ fn emit_incremental_liveness_diag() { safepoints_blocked(in_alloc={blocked_alloc} unsafe_zone={blocked_unsafe_zone} \ root_lock={blocked_root_lock}) \ copying_minors={} loop_polls={} poll_arm_events={} \ - poll_armed_at_exit={}", + poll_armed_at_exit={} forwarded_stub_recoveries={}", instruments::incremental_cycle_starts(), instruments::incremental_steps(), instruments::incremental_completions(), @@ -1368,9 +1372,11 @@ fn emit_incremental_liveness_diag() { instruments::loop_polls_reached(), poll_arm::poll_arm_events(), poll_arm::poll_armed_count(), + trace::forwarded_stub_membership_recoveries(), ); idle_reclaim::emit_diag(); idle_compact::emit_diag(); + arena_right_size::emit_diag(); emit_step_bounds_diag(); emit_gc_time_share_diag(); } diff --git a/crates/perry-runtime/src/gc/oldgen.rs b/crates/perry-runtime/src/gc/oldgen.rs index 601fce3c07..3dea59757f 100644 --- a/crates/perry-runtime/src/gc/oldgen.rs +++ b/crates/perry-runtime/src/gc/oldgen.rs @@ -1200,12 +1200,19 @@ impl IncrementalSweepState { /// 2026-07-09 audit: buffers and typed arrays joined the same pattern — /// their registry/side-table entries are pruned when the owner is /// genuinely dead (full traces only; they are all tenured old residents). - pub(super) fn with_dead_collection_finalize(mut self, full_trace: bool) -> Self { + pub(super) fn with_dead_collection_finalize( + mut self, + full_trace: bool, + synchronous_full_trace: bool, + ) -> Self { // 2026-07-09 GC audit wave 2: death-prune the object-address-keyed // side tables in the same marks-fresh window. Cheap (one flag-check // walk over tables the root scanners already walk every cycle), so // it runs eagerly here rather than budget-chunked. - super::dead_owner::prune_dead_owner_side_tables_post_trace(full_trace); + super::dead_owner::prune_dead_owner_side_tables_post_trace( + full_trace, + synchronous_full_trace, + ); self.dead_maps = crate::map::collect_dead_registered_maps_post_trace(full_trace); self.dead_sets = crate::set::collect_dead_registered_sets_post_trace(full_trace); self.dead_buffers = crate::buffer::collect_dead_registered_buffers_post_trace(full_trace); diff --git a/crates/perry-runtime/src/gc/policy.rs b/crates/perry-runtime/src/gc/policy.rs index a81a6e3531..0977d5e70a 100644 --- a/crates/perry-runtime/src/gc/policy.rs +++ b/crates/perry-runtime/src/gc/policy.rs @@ -1904,12 +1904,14 @@ pub(super) fn pacing_arena_in_use_bytes() -> usize { } /// Record the post-collection live arena bytes arena-growth pacing tests -/// against. Called once at the end of every cycle, minor and full alike. The -/// copying fast path publishes directly; non-copying cycles publish from +/// against, and feed the same exact census to the idle arena right-sizer. +/// Called once at the end of every cycle, minor and full alike. The copying +/// fast path publishes directly; non-copying cycles publish from /// `GcCycle::publish_reclaim_outcome` after their sweep census. -pub(super) fn note_collection_finished_arena_occupancy() { +pub(super) fn note_collection_finished_arena_occupancy(full: bool) { let bytes = pacing_arena_in_use_bytes(); GC_LAST_COLLECTION_POST_IN_USE_BYTES.with(|cell| cell.set(bytes)); + super::arena_right_size::note_collection_finished(bytes, full); } /// The arena reading [`arena_growth_full_escalation_due`] tests — see diff --git a/crates/perry-runtime/src/gc/tests/arena_right_size.rs b/crates/perry-runtime/src/gc/tests/arena_right_size.rs new file mode 100644 index 0000000000..07b36d67c1 --- /dev/null +++ b/crates/perry-runtime/src/gc/tests/arena_right_size.rs @@ -0,0 +1,170 @@ +//! Arena right-sizing policy: sustained-low-utilization detection, full-pass +//! accounting, and the hysteresis that bounds idle work (#9709). + +use super::super::arena_right_size::test_support::*; +use super::super::arena_right_size::*; + +const MIB: usize = 1024 * 1024; + +fn usage(live_bytes: usize, capacity_bytes: usize) -> ArenaUsage { + ArenaUsage { + live_bytes, + capacity_bytes, + } +} + +fn low_usage(capacity_bytes: usize) -> ArenaUsage { + usage( + capacity_bytes * ARENA_RIGHT_SIZE_TRIGGER_PCT / 100, + capacity_bytes, + ) +} + +#[test] +fn low_utilization_must_persist_and_the_capacity_floor_is_strict() { + let _guard = ArenaRightSizeTestGuard::new(); + let capacity = 100 * MIB; + + observe(low_usage(capacity), false); + assert!(!owed(), "one low collection is only a transient"); + assert_eq!(state_snapshot().0, 1); + + observe(low_usage(capacity), false); + assert!(owed(), "two consecutive low collections open an episode"); + assert_eq!(state_snapshot().1, ARENA_RIGHT_SIZE_FULL_OBSERVATIONS); + + reset_state(); + let floor = ARENA_RIGHT_SIZE_MIN_CAPACITY_BYTES; + observe(low_usage(floor), false); + observe(low_usage(floor), false); + assert!(!owed(), "capacity exactly at the floor is left alone"); + + reset_state(); + let one_over_trigger = usage(capacity * ARENA_RIGHT_SIZE_TRIGGER_PCT / 100 + 1, capacity); + observe(one_over_trigger, false); + observe(one_over_trigger, false); + assert!( + !owed(), + "one byte above the utilization trigger resets the streak" + ); +} + +#[test] +fn fulls_already_in_the_low_streak_count_toward_block_release() { + let _guard = ArenaRightSizeTestGuard::new(); + let low = low_usage(100 * MIB); + + observe(low, false); + observe(low, true); + assert_eq!( + state_snapshot().1, + ARENA_RIGHT_SIZE_FULL_OBSERVATIONS - 1, + "the full that established sustained slack is the first release observation" + ); + + reset_state(); + observe(low, false); + observe(low, false); + assert_eq!( + state_snapshot().1, + ARENA_RIGHT_SIZE_FULL_OBSERVATIONS, + "minor samples establish utilization but do not impersonate full sweeps" + ); + + reset_state(); + observe(low, true); + observe(low, true); + assert!( + !owed(), + "two full observations already paid the bounded episode" + ); + assert!(state_snapshot().2, "unchanged low utilization disarms it"); +} + +#[test] +fn an_episode_is_bounded_and_utilization_hysteresis_rearms_it() { + let _guard = ArenaRightSizeTestGuard::new(); + let capacity = 100 * MIB; + let low = low_usage(capacity); + + observe(low, false); + observe(low, false); + assert_eq!(state_snapshot().1, 2); + observe(low, true); + assert_eq!(state_snapshot().1, 1); + observe(low, true); + assert!(!owed()); + assert!(state_snapshot().2, "two fulls are the hard work bound"); + + for _ in 0..8 { + observe(low, false); + } + assert!( + !owed(), + "a stable low heap must not buy another episode on periodic minors" + ); + + let rearmed = usage(capacity * ARENA_RIGHT_SIZE_REARM_PCT / 100, capacity); + observe(rearmed, false); + assert!(!state_snapshot().2, "the re-arm watermark ends hysteresis"); + observe(low, false); + observe(low, false); + assert!(owed(), "a later low-utilization epoch gets its own episode"); +} + +#[test] +fn material_capacity_regrowth_rearms_without_requiring_a_large_live_set() { + let _guard = ArenaRightSizeTestGuard::new(); + let capacity = 100 * MIB; + let low = low_usage(capacity); + + // Two low FULL samples consume the episode immediately and disarm it at + // `capacity`, without needing to start a synthetic collector in this pure + // policy test. + observe(low, true); + observe(low, true); + assert!(state_snapshot().2); + + let growth = (capacity * ARENA_RIGHT_SIZE_REARM_GROWTH_PCT / 100) + .max(ARENA_RIGHT_SIZE_REARM_GROWTH_MIN_BYTES); + observe(low_usage(capacity + growth - 1), false); + assert!(state_snapshot().2, "one byte short of material growth"); + observe(low_usage(capacity + growth), false); + assert!(!state_snapshot().2, "a real new capacity peak re-arms"); + assert_eq!( + state_snapshot().0, + 1, + "the re-arm sample starts the new streak" + ); + observe(low_usage(capacity + growth), false); + assert!(owed()); +} + +#[test] +fn reaching_the_target_stops_early_and_records_capacity_released() { + let _guard = ArenaRightSizeTestGuard::new(); + let start_capacity = 100 * MIB; + let low = low_usage(start_capacity); + let released_before = arena_right_size_released_capacity_bytes(); + + observe(low, false); + observe(low, false); + assert!(owed()); + + let target_capacity = 70 * MIB; + let target = usage( + target_capacity * ARENA_RIGHT_SIZE_TARGET_PCT / 100, + target_capacity, + ); + observe(target, true); + + assert!(!owed(), "the target band cancels the remaining full"); + assert_eq!( + arena_right_size_released_capacity_bytes(), + released_before + (start_capacity - target_capacity) as u64 + ); + assert!( + state_snapshot().2, + "target is deliberately below the higher re-arm watermark" + ); +} diff --git a/crates/perry-runtime/src/gc/tests/dead_owner_side_tables.rs b/crates/perry-runtime/src/gc/tests/dead_owner_side_tables.rs index 78aca9929c..3b38346f5d 100644 --- a/crates/perry-runtime/src/gc/tests/dead_owner_side_tables.rs +++ b/crates/perry-runtime/src/gc/tests/dead_owner_side_tables.rs @@ -1578,7 +1578,9 @@ fn test_live_transition_cache_entry_survives_full_gc() { let next_keys = crate::arena::arena_alloc_gc_old(64, 8, GC_TYPE_ARRAY) as usize; let live_key = crate::arena::arena_alloc_gc(64, 8, GC_TYPE_STRING) as usize; - let prev_shape_id = crate::object::shapes::shape_id_for_keys_ensure(std::ptr::null(), 0); + // This predecessor can recur from a generated module global; without such + // an owner, retiring it and the now-unusable cache entry is intentional. + let prev_shape_id = crate::object::shapes::js_object_shape_id_for_keys(0, 0); js_shadow_slot_set(0, ptr_bits(next_keys)); js_shadow_slot_set(1, ptr_bits(live_key)); diff --git a/crates/perry-runtime/src/gc/tests/forwarded_stub_membership.rs b/crates/perry-runtime/src/gc/tests/forwarded_stub_membership.rs new file mode 100644 index 0000000000..d40bfcb2cf --- /dev/null +++ b/crates/perry-runtime/src/gc/tests/forwarded_stub_membership.rs @@ -0,0 +1,159 @@ +//! #9717 — a budgeted (classifier-mode) full trace must keep alive an array a +//! live heap slot reaches only through an array-growth forwarding stub. +//! +//! Array growth leaves a PERMANENT forwarding stub at the pre-grow address and +//! never rewrites the references pointing at it (#6228 / #233), so a live field +//! — hono's `SmartRouter.#routes`, in the reported case — can keep naming the +//! stub. A SYNCHRONOUS full trace is fine: its census (`record_arena_header`) +//! admits every arena object, stubs included, so `mark_field_into_worklist` +//! marks the stub and `trace_one_worklist_header` follows it to the live array. +//! +//! A BUDGETED full trace (what the idle-time reducer runs) resolves membership +//! through `classifier_valid_object_start` instead, and that gate rejected +//! FORWARDED headers by design (a dead metadata key's recycled bytes can set +//! the bit, #8040). So the field -> stub edge was dropped: the stub was never +//! marked, the FORWARDED-follow never ran, and the array reachable ONLY through +//! it was swept — the private-field array that "is still an array, but empty" +//! ten seconds into a loaded server. +//! +//! The classifier is documented as a census SUPERSET; for stubs it was not. +//! These tests plant the exact edge and assert the budgeted trace keeps the +//! target alive. Each first asserts the PREMISE — the pre-#9717 gate rejects +//! the stub — so a green run says the recovery works, not that nothing was +//! tried. + +use super::super::*; +use super::support::*; + +fn reset_old_reclaim_pressure() { + let old_in_use = crate::arena::old_gen_in_use_bytes(); + GC_LAST_OLD_RECLAIM_IN_USE_BYTES.with(|bytes| bytes.set(old_in_use)); + GC_OLD_RECLAIM_PENDING.with(|pending| pending.set(false)); +} + +/// Grow a fresh array past its inline capacity and return `(stub, grown, first +/// child)`: `stub` is the pre-grow head, now a forwarding stub; `grown` is the +/// current head; `first_child` is the value at index 0, reachable only through +/// the array. +fn grow_array_leaving_stub() -> (usize, usize, usize) { + let stub = crate::array::js_array_alloc(0); + let mut current = stub; + let mut first_child = 0usize; + for i in 0..64 { + let child = young_leaf(); + if i == 0 { + first_child = child; + } + current = crate::array::js_array_push_f64(current, f64::from_bits(ptr_bits(child))); + } + assert_ne!( + stub, current, + "setup must grow the array and leave a forwarding stub" + ); + unsafe { + let stub_hdr = header_from_user_ptr(stub as *const u8) as *mut GcHeader; + assert_ne!( + (*stub_hdr).gc_flags & GC_FLAG_FORWARDED, + 0, + "the pre-grow head must be a forwarding stub" + ); + // THE PREMISE: the census-superset gate a budgeted classifier used + // rejects this stub, so a trace consulting only it drops the edge. + assert!( + crate::gc::barrier::plausible_arena_user_ptr_header(stub_hdr).is_none(), + "a forwarded stub must fail the pre-#9717 gate, or recovery proves nothing" + ); + } + (stub as usize, current as usize, first_child) +} + +#[test] +fn a_budgeted_full_cycle_keeps_an_array_a_live_field_reaches_through_a_growth_stub() { + let _guard = CopyingNurseryTestGuard::new(2); + let _trigger_guard = GcTriggerThresholdTestGuard::suppress_automatic_triggers(); + reset_old_reclaim_pressure(); + reset_global_roots(); + let _root_reset = ShadowAndGlobalRootResetGuard; + + let (stub, _grown, first_child) = grow_array_leaving_stub(); + + // The "#routes field": a heap holder whose slot points DIRECTLY at the + // stub (references are never rewritten), rooted so the trace reaches it. + let holder = crate::array::js_array_alloc(1); + let holder = crate::array::js_array_push_f64(holder, f64::from_bits(ptr_bits(stub))); + js_shadow_slot_set(0, ptr_bits(holder as usize)); + + let recoveries_before = crate::gc::forwarded_stub_membership_recoveries(); + + // Drive a budgeted FULL cycle — the idle reclaimer's path — to completion. + GC_OLD_RECLAIM_PENDING.with(|pending| pending.set(true)); + let mut result = JsGcStepResult::default(); + assert_eq!( + js_gc_step_work_units(1, &mut result), + JS_GC_STEP_STATUS_ACTIVE + ); + assert_eq!(result.collection_kind, GcCollectionKind::Full.ffi_code()); + let completed = complete_budgeted_gc_cycle(); + assert_eq!(completed.status, JS_GC_STEP_STATUS_COMPLETED); + + assert!( + crate::gc::forwarded_stub_membership_recoveries() > recoveries_before, + "#9717: the budgeted trace must recover the growth stub the live field \ + points at; without it the field -> stub edge is dropped and the array \ + reachable only through it is swept" + ); + + // The array survives with its contents: resolve holder[0] -> stub -> grown + // and read the first element back through the stub (clean_arr_ptr follows + // the forward). A swept target would read as an empty/undefined array here. + let holder_after = (js_shadow_slot_get(0) & POINTER_MASK) as usize; + let stub_bits = + crate::array::js_array_get_f64(holder_after as *const crate::array::ArrayHeader, 0) + .to_bits(); + let stub_after = (stub_bits & POINTER_MASK) as *const crate::array::ArrayHeader; + let child_bits = crate::array::js_array_get_f64(stub_after, 0).to_bits(); + let child_after = (child_bits & POINTER_MASK) as usize; + assert_eq!( + child_after, first_child, + "the array element reachable only through the field -> stub edge must \ + survive the budgeted full cycle unchanged" + ); +} + +/// Negative control: a SYNCHRONOUS full mark-sweep already handles the same +/// edge through the exact census (`record_arena_header`), so it needs no +/// recovery. This keeps the assertion above a statement about the BUDGETED +/// path specifically, not about stubs in general. +#[test] +fn a_synchronous_full_cycle_keeps_the_same_array_without_needing_recovery() { + let _guard = CopyingNurseryTestGuard::new(2); + let _trigger_guard = GcTriggerThresholdTestGuard::suppress_automatic_triggers(); + reset_global_roots(); + let _root_reset = ShadowAndGlobalRootResetGuard; + + let (stub, _grown, first_child) = grow_array_leaving_stub(); + let holder = crate::array::js_array_alloc(1); + let holder = crate::array::js_array_push_f64(holder, f64::from_bits(ptr_bits(stub))); + js_shadow_slot_set(0, ptr_bits(holder as usize)); + + let recoveries_before = crate::gc::forwarded_stub_membership_recoveries(); + gc_collect_full_mark_sweep_with_trigger(GcTriggerSnapshot::capture(GcTriggerKind::Manual)); + assert_eq!( + crate::gc::forwarded_stub_membership_recoveries(), + recoveries_before, + "the synchronous census admits stubs directly; the classifier recovery \ + path must not run on this path" + ); + + let holder_after = (js_shadow_slot_get(0) & POINTER_MASK) as usize; + let stub_bits = + crate::array::js_array_get_f64(holder_after as *const crate::array::ArrayHeader, 0) + .to_bits(); + let stub_after = (stub_bits & POINTER_MASK) as *const crate::array::ArrayHeader; + let child_bits = crate::array::js_array_get_f64(stub_after, 0).to_bits(); + assert_eq!( + (child_bits & POINTER_MASK) as usize, + first_child, + "a synchronous full cycle keeps the stub-reached array (unchanged behaviour)" + ); +} diff --git a/crates/perry-runtime/src/gc/tests/handle_bound_method_name.rs b/crates/perry-runtime/src/gc/tests/handle_bound_method_name.rs index 0fa6c7e051..a299a8d6ff 100644 --- a/crates/perry-runtime/src/gc/tests/handle_bound_method_name.rs +++ b/crates/perry-runtime/src/gc/tests/handle_bound_method_name.rs @@ -184,11 +184,11 @@ fn a_bound_timer_method_from_the_ic_miss_path_never_captures_the_key() { let id = live_timer(); let (key, key_interior) = heap_key("hasRef"); let mut cache = crate::object::PicCache::default(); - + let mut cache_slot: crate::object::PicCacheSlot = &mut cache; let bits = crate::object::js_object_get_field_ic_miss( id as *const crate::ObjectHeader, key, - &mut cache, + &mut cache_slot, ); assert_names_the_literal( crate::value::JSValue::from_bits(bits.to_bits()), diff --git a/crates/perry-runtime/src/gc/tests/idle_reclaim.rs b/crates/perry-runtime/src/gc/tests/idle_reclaim.rs index 4b37368bac..eacee0ba1f 100644 --- a/crates/perry-runtime/src/gc/tests/idle_reclaim.rs +++ b/crates/perry-runtime/src/gc/tests/idle_reclaim.rs @@ -120,7 +120,8 @@ fn idle_reclaim_runs_a_full_at_the_park_when_owed() { "rooted survivor intact and unmoved" ); - // Gate 1: activity. Nothing collected since — no second attempt, ever. + // Gate 1: activity. Nothing collected since, and this test guard keeps the + // arena right-size gate below its capacity floor, so no second attempt. set_test_now_ms(Some( IDLE_RECLAIM_QUIET_MS + IDLE_RECLAIM_MIN_INTERVAL_MS + 1, )); @@ -147,6 +148,81 @@ fn idle_reclaim_runs_a_full_at_the_park_when_owed() { drive_until_idle(t + IDLE_RECLAIM_QUIET_MS, 1000); } +/// #9709: general-arena block release deliberately needs two full collection +/// observations. One external collection plus the ordinary idle-reducer full +/// establishes sustained low utilization; that full is observation one, and +/// the right-sizer must grant observation two without demanding new mutator +/// activity. The episode then disarms, so the bypass cannot become a periodic +/// full-GC loop. +#[test] +fn sustained_arena_slack_gets_one_bounded_followup_without_mutator_activity() { + let _guard = CopyingNurseryTestGuard::new(1); + let _triggers = GcTriggerThresholdTestGuard::suppress_automatic_triggers(); + let _reducer = IdleReclaimTestGuard::new(0); + let capacity = 100 * 1024 * 1024; + super::super::arena_right_size::test_support::set_test_usage(Some( + super::super::arena_right_size::ArenaUsage { + live_bytes: capacity / 2, + capacity_bytes: capacity, + }, + )); + let right_size_starts_before = arena_right_size_starts(); + + // First low sample, and the only mutator-driven collection in this test. + external_collection_observed_at(0); + assert_eq!( + super::super::arena_right_size::test_support::state_snapshot().0, + 1 + ); + + // The ordinary activity arm starts the first full. Its post-collection + // sample opens the episode and counts as the first full observation. + set_test_now_ms(Some(IDLE_RECLAIM_QUIET_MS)); + assert!(resumes(idle_reclaim_park_hook(1000))); + drive_until_idle(IDLE_RECLAIM_QUIET_MS, 1000); + assert_eq!(thread_attempts(), 1); + assert_eq!( + super::super::arena_right_size::test_support::state_snapshot().1, + 1, + "one full observation remains" + ); + + // The normal activity gate is not re-armed. The capacity debt alone must + // cross the same rate floor and start the second full. + set_test_now_ms(Some( + IDLE_RECLAIM_QUIET_MS + IDLE_RECLAIM_MIN_INTERVAL_MS - 1, + )); + assert!(parks(idle_reclaim_park_hook(1000))); + assert_eq!(thread_attempts(), 1, "the rate floor still applies"); + + let followup_at = IDLE_RECLAIM_QUIET_MS + IDLE_RECLAIM_MIN_INTERVAL_MS; + set_test_now_ms(Some(followup_at)); + assert!(resumes(idle_reclaim_park_hook(1000))); + assert_eq!(thread_attempts(), 2); + assert_eq!( + arena_right_size_starts(), + right_size_starts_before + 1, + "LIVE SUBJECT: the follow-up must identify the capacity debt as its reason" + ); + drive_until_idle(followup_at, 1000); + + let (_, fulls_remaining, disarmed, _) = + super::super::arena_right_size::test_support::state_snapshot(); + assert_eq!(fulls_remaining, 0); + assert!( + disarmed, + "unchanged low utilization ends the bounded episode" + ); + + set_test_now_ms(Some(followup_at + IDLE_RECLAIM_MIN_INTERVAL_MS + 1)); + assert!(parks(idle_reclaim_park_hook(1000))); + assert_eq!( + thread_attempts(), + 2, + "no third full without utilization or material-capacity hysteresis" + ); +} + #[test] fn idle_reclaim_rate_floor_holds_between_two_owed_fulls() { let _guard = CopyingNurseryTestGuard::new(1); diff --git a/crates/perry-runtime/src/gc/tests/mod.rs b/crates/perry-runtime/src/gc/tests/mod.rs index 82a2f95b78..ad0ae26bcc 100644 --- a/crates/perry-runtime/src/gc/tests/mod.rs +++ b/crates/perry-runtime/src/gc/tests/mod.rs @@ -1,4 +1,5 @@ mod alloc; +mod arena_right_size; mod array_pointer_slot_enumeration; mod barrier; mod barrier_arming; @@ -19,6 +20,7 @@ mod dirty_page_cache; mod env_knob_parse; mod error_side_tables; mod evacuation; +mod forwarded_stub_membership; mod forwarding_target_validation; mod fromspace_protect; mod fromspace_scan; diff --git a/crates/perry-runtime/src/gc/tests/shape_keys_descriptor_edge.rs b/crates/perry-runtime/src/gc/tests/shape_keys_descriptor_edge.rs index 856a21dc62..12f011388a 100644 --- a/crates/perry-runtime/src/gc/tests/shape_keys_descriptor_edge.rs +++ b/crates/perry-runtime/src/gc/tests/shape_keys_descriptor_edge.rs @@ -24,7 +24,10 @@ //! alive by something else entirely". use super::super::*; -use super::support::{collect_minor_trace, init_test_closure, ptr_bits, CopyingNurseryTestGuard}; +use super::support::{ + collect_minor_trace, complete_budgeted_gc_cycle, init_test_closure, ptr_bits, + CopyingNurseryTestGuard, GcTriggerThresholdTestGuard, +}; use crate::object::shapes; /// Facts the assertions compare, read exclusively through the descriptor. @@ -488,3 +491,180 @@ fn metadata_rewrite_validates_the_post_visit_non_array_address() { ); shapes::test_clear_shape_table(); } + +fn collect_synchronous_full_trace() { + let _ = + gc_collect_full_mark_sweep_with_trigger(GcTriggerSnapshot::capture(GcTriggerKind::Direct)); +} + +fn build_unrooted_keyless_semantic_shape(slot: u32) -> u32 { + js_shadow_slot_set( + slot, + ptr_bits(crate::object::js_object_alloc(0, 0) as usize), + ); + let obj = (js_shadow_slot_get(slot) & POINTER_MASK) as *mut crate::ObjectHeader; + let predecessor = unsafe { shapes::object_shape_stamp(obj) }; + let shape_id = unsafe { shapes::transition_object_shape_semantics(obj) }; + assert_ne!(shape_id, predecessor, "semantic transition must mint an id"); + assert_ne!( + shapes::shape_descriptor_by_id(shape_id) + .expect("semantic descriptor") + .semantic_generation, + 0, + "test premise: this must be a per-object semantic generation" + ); + js_shadow_slot_set(slot, crate::value::TAG_UNDEFINED); + shape_id +} + +/// #9726: keyless semantic generations used to be immortal because dead-key +/// pruning asks about address zero, which is never a dead GC owner. A complete +/// receiver census must retire that descriptor while preserving the inverse: +/// the same kind of generation stays authoritative when its object is live. +#[test] +fn synchronous_full_trace_retires_only_uncarried_semantic_shapes() { + let _guard = CopyingNurseryTestGuard::new(2); + shapes::test_clear_shape_table(); + crate::arena::arena_reset_all_blocks_to_zero(); + gc_register_mutable_root_scanner(shapes::scan_shape_table_rekey_mut); + + let dead_shape = build_unrooted_keyless_semantic_shape(0); + collect_synchronous_full_trace(); + assert!( + shapes::shape_descriptor_by_id(dead_shape).is_none(), + "#9726: a keyless per-object generation with no live carrier survived a complete full trace" + ); + + build_two_key_object(0, b"e9726_live_"); + let live_before = (js_shadow_slot_get(0) & POINTER_MASK) as *mut crate::ObjectHeader; + let live_shape = unsafe { shapes::transition_object_shape_semantics(live_before) }; + assert_ne!( + shapes::shape_descriptor_by_id(live_shape) + .expect("live semantic descriptor before collection") + .semantic_generation, + 0 + ); + + collect_synchronous_full_trace(); + + let live_after = (js_shadow_slot_get(0) & POINTER_MASK) as *mut crate::ObjectHeader; + assert_eq!( + unsafe { shapes::object_shape_stamp(live_after) }, + live_shape + ); + assert!( + shapes::shape_descriptor_by_id(live_shape).is_some(), + "#9726/#9200: pruning must not leave a live receiver stamped with an unresolved id" + ); + let own_keys = crate::object::js_object_keys(live_after); + assert_eq!( + unsafe { (*own_keys).length }, + 2, + "#9726/#9200: Object.keys() lost the live receiver's descriptor facts" + ); + + js_shadow_slot_set(0, crate::value::TAG_UNDEFINED); + shapes::test_clear_shape_table(); +} + +/// Incremental full marking is sliced across mutator turns, so its receiver +/// notes are deliberately not an exact liveness census. It may rotate the +/// epoch, but it must leave uncarried retirement to a synchronous full trace. +#[test] +fn budgeted_full_trace_does_not_retire_from_a_partial_carrier_census() { + let _guard = CopyingNurseryTestGuard::new(1); + let _trigger_guard = GcTriggerThresholdTestGuard::suppress_automatic_triggers(); + shapes::test_clear_shape_table(); + crate::arena::arena_reset_all_blocks_to_zero(); + + let shape_id = build_unrooted_keyless_semantic_shape(0); + GC_OLD_RECLAIM_PENDING.with(|pending| pending.set(true)); + let mut first = JsGcStepResult::default(); + assert_eq!( + js_gc_step_work_units(1, &mut first), + JS_GC_STEP_STATUS_ACTIVE + ); + assert_eq!(first.collection_kind, GcCollectionKind::Full.ffi_code()); + let completed = complete_budgeted_gc_cycle(); + assert_eq!(completed.status, JS_GC_STEP_STATUS_COMPLETED); + assert!( + shapes::shape_descriptor_by_id(shape_id).is_some(), + "#9726: a budgeted trace must not retire from its partial carrier notes" + ); + + collect_synchronous_full_trace(); + assert!( + shapes::shape_descriptor_by_id(shape_id).is_none(), + "the next complete full trace must retire the same uncarried descriptor" + ); + shapes::test_clear_shape_table(); +} + +fn build_transition_cache_target_then_drop(slot: u32) -> (u32, u32) { + // Model the process-lifetime module global that can birth the predecessor + // again after no receiver currently carries it. + let predecessor = shapes::js_object_shape_id_for_keys(0, 0); + js_shadow_slot_set( + slot, + ptr_bits(crate::object::js_object_alloc(0, 0) as usize), + ); + let key = crate::string::js_string_from_bytes(b"cache6".as_ptr(), 6); + let obj = (js_shadow_slot_get(slot) & POINTER_MASK) as *mut crate::ObjectHeader; + assert_eq!(unsafe { shapes::object_shape_stamp(obj) }, predecessor); + crate::object::js_object_set_field_by_name(obj, key, 9726.0); + let obj = (js_shadow_slot_get(slot) & POINTER_MASK) as *mut crate::ObjectHeader; + let target = unsafe { shapes::object_shape_stamp(obj) }; + assert_ne!(target, predecessor); + js_shadow_slot_set(slot, crate::value::TAG_UNDEFINED); + (predecessor, target) +} + +/// The transition table is a real ShapeId publisher: generated write sites +/// read its target id and stamp it directly. Park a target with no receiver, +/// collect, then prove a later receiver can still take the cached transition. +#[test] +fn transition_cache_target_survives_and_can_restamp_after_full_trace() { + let _guard = CopyingNurseryTestGuard::new(1); + shapes::test_clear_shape_table(); + crate::arena::arena_reset_all_blocks_to_zero(); + gc_register_mutable_root_scanner(shapes::scan_shape_table_rekey_mut); + gc_register_mutable_root_scanner(crate::object::scan_transition_cache_roots_mut); + + let (predecessor, target) = build_transition_cache_target_then_drop(0); + collect_synchronous_full_trace(); + assert!( + shapes::shape_descriptor_by_id(predecessor).is_some(), + "a process-lifetime generated-code id must remain installed" + ); + let cached = shapes::shape_descriptor_by_id(target) + .expect("a live transition-cache entry must retain its target descriptor"); + assert!( + cached.cache_carrier, + "the transition target must own its id" + ); + + js_shadow_slot_set(0, ptr_bits(crate::object::js_object_alloc(0, 0) as usize)); + let key = crate::string::js_string_from_bytes(b"cache6".as_ptr(), 6); + let consumer = (js_shadow_slot_get(0) & POINTER_MASK) as *mut crate::ObjectHeader; + shapes::test_watch_cached_transition_stamps(consumer as usize); + crate::object::js_object_set_field_by_name(consumer, key, 26.0); + assert_eq!( + shapes::test_cached_transition_stamps(), + 1, + "the cache path must perform the stamp" + ); + let consumer = (js_shadow_slot_get(0) & POINTER_MASK) as *mut crate::ObjectHeader; + assert_eq!( + unsafe { shapes::object_shape_stamp(consumer) }, + target, + "the post-GC transition hit must stamp the retained target id" + ); + assert_eq!( + unsafe { (*crate::object::js_object_keys(consumer)).length }, + 1 + ); + + shapes::test_reset_cached_transition_stamps(); + js_shadow_slot_set(0, crate::value::TAG_UNDEFINED); + shapes::test_clear_shape_table(); +} diff --git a/crates/perry-runtime/src/gc/trace.rs b/crates/perry-runtime/src/gc/trace.rs index 8a1606c121..fb350daa45 100644 --- a/crates/perry-runtime/src/gc/trace.rs +++ b/crates/perry-runtime/src/gc/trace.rs @@ -9,6 +9,29 @@ thread_local! { const { std::cell::Cell::new(false) }; } +crate::perry_thread_local! { + /// #9717: array-growth forwarding stubs the classifier admitted into a + /// budgeted-cycle valid-pointer set that `plausible_arena_user_ptr_header` + /// would have rejected. A non-zero count is a POSITIVE report that a live + /// slot pointed at a growth stub during a budgeted full trace — the exact + /// edge whose loss swept a private-field array on the idle reclaim. Zero on + /// a run with no such edge, so it never perturbs a log a gate parses. + static FORWARDED_STUB_MEMBERSHIP_RECOVERIES: std::cell::Cell = + const { std::cell::Cell::new(0) }; +} + +#[cold] +fn note_forwarded_stub_membership_recovery() { + FORWARDED_STUB_MEMBERSHIP_RECOVERIES.with(|c| c.set(c.get().saturating_add(1))); +} + +/// Running count of array-growth forwarding stubs the classifier recovered into +/// a budgeted valid-pointer set (#9717). A test that plants a stub-only- +/// referenced array can assert this moved. +pub(crate) fn forwarded_stub_membership_recoveries() -> u64 { + FORWARDED_STUB_MEMBERSHIP_RECOVERIES.with(std::cell::Cell::get) +} + /// #6179 membership classifier: is `addr` a plausible live GC object start? /// UNION of the two backends — exact membership in the malloc registry OR a /// plausible arena header on an arena-classified page — deliberately NOT the @@ -25,10 +48,32 @@ pub(super) fn classifier_valid_object_start(addr: usize) -> bool { if super::gc_malloc_header_is_tracked(header) { return true; } - !matches!( + if matches!( crate::arena::classify_heap_generation(addr), crate::arena::HeapGeneration::Unknown - ) && unsafe { super::barrier::plausible_arena_user_ptr_header(header).is_some() } + ) { + return false; + } + if unsafe { super::barrier::plausible_arena_user_ptr_header(header).is_some() } { + return true; + } + // #9717: an array-growth forwarding stub is a real censused arena object a + // live slot can still point directly at (references are never rewritten, + // #6228). The census path admits it (record_arena_header pushes every arena + // object), so this classifier -- which contains() uses for a budgeted, + // non-moving cycle and which must be a census SUPERSET -- has to admit it + // too. plausible_arena_user_ptr_header rejects FORWARDED headers by design + // (a metadata key whose object may have died and been recycled with the bit + // set), so the stub was silently dropped: mark_field_into_worklist failed + // membership, never marked the stub, and the FORWARDED-follow in + // trace_one_worklist_header never ran -- so the live post-growth array, + // reachable only through the field to stub edge, was swept. That is the + // idle-time (budgeted full) reclaim turning a private-field array empty. + if unsafe { super::barrier::plausible_forwarded_arena_stub(header).is_some() } { + note_forwarded_stub_membership_recovery(); + return true; + } + false } /// #6179: differential-verification mode for the page-metadata classifier. diff --git a/crates/perry-runtime/src/node_submodules/tests.rs b/crates/perry-runtime/src/node_submodules/tests.rs index 30523768b1..a6abfdede0 100644 --- a/crates/perry-runtime/src/node_submodules/tests.rs +++ b/crates/perry-runtime/src/node_submodules/tests.rs @@ -468,8 +468,12 @@ fn test_default_and_named_exports_share_the_self_alias() { crate::object::js_object_get_field_by_name_f64(closure as *const ObjectHeader, key); assert_eq!(property.to_bits(), default.to_bits()); let mut cache = [0i64; crate::object::PIC_CACHE_WORDS]; - let property = - crate::object::js_object_get_field_ic_miss(closure as *const ObjectHeader, key, &mut cache); + let mut cache_slot: crate::object::PicCacheSlot = &mut cache; + let property = crate::object::js_object_get_field_ic_miss( + closure as *const ObjectHeader, + key, + &mut cache_slot, + ); assert_eq!(property.to_bits(), default.to_bits()); } diff --git a/crates/perry-runtime/src/object/field_get_set.rs b/crates/perry-runtime/src/object/field_get_set.rs index 5a9a2bb98c..088f9d70d1 100644 --- a/crates/perry-runtime/src/object/field_get_set.rs +++ b/crates/perry-runtime/src/object/field_get_set.rs @@ -216,6 +216,7 @@ mod ic_miss; #[cfg(test)] #[path = "field_get_set/ic_miss_array_length_tests.rs"] mod ic_miss_array_length_tests; +mod ic_slot; mod map_set_receiver; mod probe_dispatch; /// #9131: per-instance `[[Prototype]]` override lookup, split out of @@ -304,8 +305,10 @@ pub use ic_miss::{ js_class_field_add, js_object_get_field_by_name_f64, js_object_get_field_by_property_id_f64, js_object_get_field_ic, js_object_get_field_ic_miss, js_object_set_field_by_property_id, js_private_brand_add, js_private_brand_check, js_private_field_add, js_private_guard, PicCache, - PIC_CACHE_WORDS, + PicCacheSlot, PIC_CACHE_WORDS, }; +pub(crate) use ic_slot::pic_slot_census; +pub use ic_slot::{pic_arena_bytes, pic_slot_peek, pic_slot_resolve, pic_slots_resolved}; #[cfg(test)] mod buffer_ic_miss_tests { @@ -337,24 +340,25 @@ mod buffer_ic_miss_tests { for len in [16usize, 24, 32] { let buf = secret_buffer(len); let mut cache = [0i64; crate::object::PIC_CACHE_WORDS]; - + let mut cache_slot: crate::object::PicCacheSlot = &mut cache; let ty = js_object_get_field_ic_miss( buf as *const ObjectHeader, key(b"type"), - &mut cache, + &mut cache_slot, ); assert_eq!(string_value_bytes(ty), b"secret"); let size = js_object_get_field_ic_miss( buf as *const ObjectHeader, key(b"symmetricKeySize"), - &mut cache, + &mut cache_slot, ); assert_eq!(size, len as f64); let raw = dispatch_buffer_method(buf as usize, "export", std::ptr::null(), 0); let raw_addr = (raw.to_bits() & 0x0000_FFFF_FFFF_FFFF) as *const ObjectHeader; - let raw_len = js_object_get_field_ic_miss(raw_addr, key(b"length"), &mut cache); + let raw_len = + js_object_get_field_ic_miss(raw_addr, key(b"length"), &mut cache_slot); assert_eq!(raw_len, len as f64); } } diff --git a/crates/perry-runtime/src/object/field_get_set/ic_miss.rs b/crates/perry-runtime/src/object/field_get_set/ic_miss.rs index fdd60bce19..be3027adb6 100644 --- a/crates/perry-runtime/src/object/field_get_set/ic_miss.rs +++ b/crates/perry-runtime/src/object/field_get_set/ic_miss.rs @@ -239,6 +239,13 @@ pub const PIC_CACHE_WORDS: usize = 12; /// | 11 | round-robin victim index for the ways | pub type PicCache = [i64; PIC_CACHE_WORDS]; +/// The per-site slot codegen emits for a property-read cache — `@perry_ic_N = +/// private global ptr null` — holding null until the site's first priming +/// miss, then the arena cache `pic_slot_resolve` published (#9708). The +/// emitted hit path reads the cache words through this pointer; every runtime +/// entry that primes takes the slot's address and resolves it here. +pub type PicCacheSlot = *mut PicCache; + /// First word of the polymorphic way array. /// /// The ways start at 4, not 3, so that [`PIC_WAY_STATE`] can sit at word 3 — @@ -434,7 +441,7 @@ pub extern "C" fn js_object_get_field_ic_overflow_load( obj: *const ObjectHeader, key: *const crate::StringHeader, slot: i32, - cache: *mut PicCache, + cache_slot: *mut PicCacheSlot, ) -> f64 { let idx = (slot as u32 & !crate::proxy::IC_SLOT_OVERFLOW_BIT) as usize; if !obj.is_null() { @@ -444,7 +451,7 @@ pub extern "C" fn js_object_get_field_ic_overflow_load( } } } - js_object_get_field_ic_miss(obj, key, cache) + js_object_get_field_ic_miss(obj, key, cache_slot) } /// Monomorphic inline cache miss handler (issue #51). @@ -454,9 +461,13 @@ pub extern "C" fn js_object_get_field_ic_overflow_load( /// then populates the per-site cache so subsequent calls with the same shape /// hit the inline fast path (no function call, direct field load). /// -/// `cache` layout: see [`PicCache`]. Words 0..1 are the ShapeId-token MRU entry; -/// word 2 is reserved scratch, and words 3.. are the polymorphic ways filled by -/// [`pic_prime_get`] (#7753). +/// `cache_slot` is the address of the site's [`PicCacheSlot`]; the cache it +/// resolves to (allocated on the first priming miss, #9708) has the layout in +/// [`PicCache`]. Words 0..1 are the ShapeId-token MRU entry; word 2 is +/// reserved scratch, and words 3.. are the polymorphic ways filled by +/// [`pic_prime_get`] (#7753). A miss that cannot prime — SSO or proxy +/// receiver, a missing key, an accessor — never touches the slot, so a site +/// that only ever sees such receivers costs its 8-byte slot and nothing else. /// /// Only caches when: /// - obj is a valid ObjectHeader (not null, not handle, not string/array/etc.) @@ -469,7 +480,7 @@ pub extern "C" fn js_object_get_field_ic_overflow_load( pub extern "C" fn js_object_get_field_ic_miss( obj: *const ObjectHeader, key: *const crate::StringHeader, - cache: *mut PicCache, + cache_slot: *mut PicCacheSlot, ) -> f64 { // SSO receiver — never cacheable. Route through the SSO-aware // `js_object_get_field_by_name` which handles `.length` inline @@ -795,6 +806,7 @@ pub extern "C" fn js_object_get_field_ic_miss( // the prefix paths compute inline // addresses and must never fire from an // overflow-primed entry. + let cache = pic_slot_resolve(cache_slot); (*cache)[2] = 0; pic_prime_get( cache, @@ -844,6 +856,7 @@ pub extern "C" fn js_object_get_field_ic_miss( if has_own_descriptors && named_prefix_token == 0 { break; } + let cache = pic_slot_resolve(cache_slot); (*cache)[2] = named_prefix_token; pic_prime_get(cache, token, i as i64); let field_ptr = (obj as *const u8) @@ -877,13 +890,14 @@ pub extern "C" fn js_object_get_field_ic_miss( /// - `obj_bits`: the receiver's full (unmasked) NaN-box bits /// - `key`: the property-name `StringHeader`, already masked to a raw pointer /// - `site_id`: the typed-feedback site id -/// - `cache`: the per-site monomorphic IC cache global (primed by `..._ic_miss`) +/// - `cache_slot`: the per-site [`PicCacheSlot`] (resolved and primed by +/// `..._ic_miss`) #[no_mangle] pub extern "C" fn js_object_get_field_ic( obj_bits: i64, key: *const crate::StringHeader, site_id: u64, - cache: *mut PicCache, + cache_slot: *mut PicCacheSlot, ) -> f64 { // POINTER_MASK: lower 48 bits — strips the NaN-box tag to a raw heap pointer. const POINTER_MASK: u64 = 0x0000_FFFF_FFFF_FFFF; @@ -917,7 +931,7 @@ pub extern "C" fn js_object_get_field_ic( // is primed for any future inline sites sharing this global). if (tag & 0xFFFD) == 0x7FFD { crate::typed_feedback::js_typed_feedback_observe_property_get(site_id, obj_handle, key); - return js_object_get_field_ic_miss(obj_handle, key, cache); + return js_object_get_field_ic_miss(obj_handle, key, cache_slot); } // Invalid (non-pointer) receiver. `undefined`/`null` throw a TypeError (#462 — // matches the inline nullish path, which aborts with a node-shaped message); diff --git a/crates/perry-runtime/src/object/field_get_set/ic_miss/c3c_pic_tests.rs b/crates/perry-runtime/src/object/field_get_set/ic_miss/c3c_pic_tests.rs index 05efbcfcbe..605e09471d 100644 --- a/crates/perry-runtime/src/object/field_get_set/ic_miss/c3c_pic_tests.rs +++ b/crates/perry-runtime/src/object/field_get_set/ic_miss/c3c_pic_tests.rs @@ -31,9 +31,11 @@ fn unrelated_accessor_does_not_poison_plain_receiver_pic() { }); let mut cache = [0i64; super::PIC_CACHE_WORDS]; + + let mut cache_slot: crate::object::PicCacheSlot = &mut cache; assert_eq!( obj.with_mut_ptr( - |o| key.with_const_ptr(|k| super::js_object_get_field_ic_miss(o, k, &mut cache)) + |o| key.with_const_ptr(|k| super::js_object_get_field_ic_miss(o, k, &mut cache_slot)) ), 42.0 ); @@ -66,8 +68,10 @@ fn accessor_bearing_receiver_does_not_prime_plain_data_pic() { ); let mut cache = [0i64; super::PIC_CACHE_WORDS]; + + let mut cache_slot: crate::object::PicCacheSlot = &mut cache; let via_pic = obj.with_mut_ptr(|o| { - key.with_const_ptr(|k| super::js_object_get_field_ic_miss(o, k, &mut cache)) + key.with_const_ptr(|k| super::js_object_get_field_ic_miss(o, k, &mut cache_slot)) }); let via_ladder = obj.with_mut_ptr(|o| key.with_const_ptr(|k| super::js_object_get_field_by_name_f64(o, k))); @@ -96,7 +100,9 @@ fn a_class_instance_primes_an_id_token_after_rung1() { assert_eq!((*obj).class_id, 0x6080, "test premise: a class instance"); let mut cache = [0i64; super::PIC_CACHE_WORDS]; - let v = super::js_object_get_field_ic_miss(obj, key, &mut cache); + + let mut cache_slot: crate::object::PicCacheSlot = &mut cache; + let v = super::js_object_get_field_ic_miss(obj, key, &mut cache_slot); assert_eq!(v, 7.0); let stamp = crate::object::shapes::object_shape_stamp(obj); @@ -169,11 +175,17 @@ fn a_compacted_class_instance_primes_a_token_a_pristine_sibling_cannot_match() { ); let mut c_pristine = [0i64; super::PIC_CACHE_WORDS]; - let vp = super::js_object_get_field_ic_miss(pristine, key("picdel_c"), &mut c_pristine); + + let mut c_pristine_slot: crate::object::PicCacheSlot = &mut c_pristine; + let vp = + super::js_object_get_field_ic_miss(pristine, key("picdel_c"), &mut c_pristine_slot); assert_eq!(vp, 3.0, "pristine `c` is slot 2"); let mut c_compacted = [0i64; super::PIC_CACHE_WORDS]; - let vc = super::js_object_get_field_ic_miss(compacted, key("picdel_c"), &mut c_compacted); + + let mut c_compacted_slot: crate::object::PicCacheSlot = &mut c_compacted; + let vc = + super::js_object_get_field_ic_miss(compacted, key("picdel_c"), &mut c_compacted_slot); assert_eq!( vc, 3.0, "compacted `c` shifted to slot 1 and must still read 3" @@ -254,8 +266,10 @@ fn a_fresh_class_instance_computes_the_token_the_miss_handler_primed() { ); let mut cache = [0i64; super::PIC_CACHE_WORDS]; + + let mut cache_slot: crate::object::PicCacheSlot = &mut cache; assert_eq!( - super::js_object_get_field_ic_miss(primed_from, key, &mut cache), + super::js_object_get_field_ic_miss(primed_from, key, &mut cache_slot), 5.0, "test premise: the miss handler resolved the field" ); @@ -279,7 +293,8 @@ fn a_fresh_class_instance_computes_the_token_the_miss_handler_primed() { // And the same must hold once the fresh one has itself resolved: // priming from either instance is interchangeable. let mut cache2 = [0i64; super::PIC_CACHE_WORDS]; - super::js_object_get_field_ic_miss(fresh, key, &mut cache2); + let mut cache2_slot: crate::object::PicCacheSlot = &mut cache2; + super::js_object_get_field_ic_miss(fresh, key, &mut cache2_slot); assert_eq!( cache2[0], cache[0], "two instances of one class primed two different tokens — the \ diff --git a/crates/perry-runtime/src/object/field_get_set/ic_miss_array_length_tests.rs b/crates/perry-runtime/src/object/field_get_set/ic_miss_array_length_tests.rs index 24156b1ab8..271a8df9c2 100644 --- a/crates/perry-runtime/src/object/field_get_set/ic_miss_array_length_tests.rs +++ b/crates/perry-runtime/src/object/field_get_set/ic_miss_array_length_tests.rs @@ -21,7 +21,8 @@ fn array_length_short_circuit_agrees_with_the_full_ladder() { } let obj = arr as *const super::ObjectHeader; let mut cache = [0i64; super::PIC_CACHE_WORDS]; - let via_ic = super::js_object_get_field_ic_miss(obj, len_key, &mut cache); + let mut cache_slot: crate::object::PicCacheSlot = &mut cache; + let via_ic = super::js_object_get_field_ic_miss(obj, len_key, &mut cache_slot); let via_ladder = super::js_object_get_field_by_name_f64(obj, len_key); assert_eq!( via_ic.to_bits(), @@ -32,7 +33,7 @@ fn array_length_short_circuit_agrees_with_the_full_ladder() { // A same-length key that is not `length` must not be captured // by the fast path. assert_eq!( - super::js_object_get_field_ic_miss(obj, other_key, &mut cache).to_bits(), + super::js_object_get_field_ic_miss(obj, other_key, &mut cache_slot).to_bits(), super::js_object_get_field_by_name_f64(obj, other_key).to_bits(), "a non-`length` key on an array must take the normal path" ); @@ -41,8 +42,9 @@ fn array_length_short_circuit_agrees_with_the_full_ladder() { // short-circuit — it is an ordinary (absent) property there. let plain = crate::object::js_object_alloc(0, 0); let mut cache = [0i64; super::PIC_CACHE_WORDS]; + let mut cache_slot: crate::object::PicCacheSlot = &mut cache; assert_eq!( - super::js_object_get_field_ic_miss(plain, len_key, &mut cache).to_bits(), + super::js_object_get_field_ic_miss(plain, len_key, &mut cache_slot).to_bits(), super::js_object_get_field_by_name_f64(plain, len_key).to_bits(), "`length` on a plain object must keep its normal answer" ); @@ -70,7 +72,8 @@ fn array_subclass_length_short_circuit_preserves_object_semantics() { let len_key = crate::string::js_string_from_bytes(b"length".as_ptr(), 6); let mut cache = [0i64; super::PIC_CACHE_WORDS]; - let via_ic = super::js_object_get_field_ic_miss(obj, len_key, &mut cache); + let mut cache_slot: crate::object::PicCacheSlot = &mut cache; + let via_ic = super::js_object_get_field_ic_miss(obj, len_key, &mut cache_slot); let via_ladder = super::js_object_get_field_by_name_f64(obj, len_key); assert_eq!(via_ic.to_bits(), via_ladder.to_bits()); assert_eq!(via_ic, 3.0, "the fast path must observe the live length"); @@ -78,8 +81,9 @@ fn array_subclass_length_short_circuit_preserves_object_semantics() { let plain = crate::object::js_object_alloc(0, 1); crate::object::js_object_set_field_by_name(plain, len_key, 123.0); let mut plain_cache = [0i64; super::PIC_CACHE_WORDS]; + let mut plain_cache_slot: crate::object::PicCacheSlot = &mut plain_cache; assert_eq!( - super::js_object_get_field_ic_miss(plain, len_key, &mut plain_cache), + super::js_object_get_field_ic_miss(plain, len_key, &mut plain_cache_slot), 123.0, "ordinary objects must retain their own `length` property semantics" ); diff --git a/crates/perry-runtime/src/object/field_get_set/ic_slot.rs b/crates/perry-runtime/src/object/field_get_set/ic_slot.rs new file mode 100644 index 0000000000..040e2e3b6a --- /dev/null +++ b/crates/perry-runtime/src/object/field_get_set/ic_slot.rs @@ -0,0 +1,281 @@ +//! #9708: per-site inline caches allocated on first miss, not per emitted site. +//! +//! Codegen used to emit every inline-cache site as its own +//! `[PIC_CACHE_WORDS x i64] zeroinitializer` global — 96 B per property-access +//! site, whether or not the program ever executes it. On a large bundle that +//! is hundreds of thousands of sites and tens of megabytes of `__bss` that +//! turn into dirty resident pages as soon as a neighbour on the same page is +//! touched: the `cc` build carried 262k caches (25 MB) and dirtied 18.7 MB of +//! them at idle while running a few thousand distinct hot sites. +//! +//! Each site now owns an 8-byte **slot** — `@perry_ic_N = private global ptr +//! null` — and the cache words live in a runtime-owned arena, allocated the +//! first time the site's miss handler runs. A site that never executes costs +//! its 8 bytes of zero-fill and nothing else; a site that does costs the same +//! `PIC_CACHE_WORDS` words it always did, packed next to the other caches +//! that were touched around the same time. +//! +//! The emitted hit path loads the slot, folds `!= null` into the receiver +//! guard it already evaluates, and reads the cache words through the loaded +//! pointer; the miss path hands the *slot* to the runtime, which resolves or +//! allocates the cache here. Only the words the runtime writes ever live in +//! the arena, so a cache's layout and every prime/evict policy are unchanged. +//! +//! Publication is a compare-and-swap on the slot so two `perry/thread` agents +//! missing the same site for the first time agree on one cache; the loser's +//! block stays in the arena unused (bounded by the number of such races, and +//! counted). The arena is never freed: the sites reference their caches for +//! the life of the process, exactly as the globals were. + +use std::alloc::{alloc_zeroed, handle_alloc_error, Layout}; +use std::ptr::null_mut; +use std::sync::atomic::{AtomicPtr, AtomicUsize, Ordering}; +use std::sync::Mutex; + +/// Bytes requested from the system allocator per arena refill. Holds 682 +/// twelve-word caches; a program that primes fewer sites than that pays one +/// zeroed allocation in total. +const PIC_ARENA_CHUNK_BYTES: usize = 64 * 1024; + +/// Every cache starts on a 32-byte boundary, so the four words the MRU hit +/// path and the way-state gate read (`tok0`, `slot0`, scratch, state) never +/// straddle a 64-byte line — the property the pre-#9708 layout relied on +/// when it parked the gate at word 3 (see `PIC_WAY_BASE`'s doc). +const PIC_CACHE_ALIGN: usize = 32; + +/// Bump-pointer state over the current chunk. Addresses are kept as `usize` +/// on purpose: the arena hands out cache words (tokens and slot indices), it +/// never holds a GC heap pointer, and `usize` keeps the static `Send` without +/// an `unsafe impl`. +struct PicArena { + cur: usize, + end: usize, +} + +static PIC_ARENA: Mutex = Mutex::new(PicArena { cur: 0, end: 0 }); + +/// Caches published into a slot (one per site that has missed at least once). +static PIC_SLOTS_RESOLVED: AtomicUsize = AtomicUsize::new(0); +/// Bytes requested from the system allocator for arena chunks. +static PIC_ARENA_BYTES: AtomicUsize = AtomicUsize::new(0); +/// Allocations that lost the publication race to another thread. +static PIC_SLOT_RACES: AtomicUsize = AtomicUsize::new(0); + +/// Bump-allocate `bytes` of zeroed, `PIC_CACHE_ALIGN`-aligned memory. +fn pic_arena_alloc(bytes: usize) -> *mut u8 { + let bytes = bytes.div_ceil(PIC_CACHE_ALIGN) * PIC_CACHE_ALIGN; + debug_assert!(bytes <= PIC_ARENA_CHUNK_BYTES); + let mut arena = PIC_ARENA.lock().unwrap_or_else(|e| e.into_inner()); + if arena.cur + bytes > arena.end { + // SAFETY: the layout is non-zero-sized and its alignment is a power of + // two; a zeroed chunk is exactly the initial state every cache expects. + let layout = Layout::from_size_align(PIC_ARENA_CHUNK_BYTES, 64) + .expect("PIC arena chunk layout is a constant"); + let chunk = unsafe { alloc_zeroed(layout) }; + if chunk.is_null() { + handle_alloc_error(layout); + } + PIC_ARENA_BYTES.fetch_add(PIC_ARENA_CHUNK_BYTES, Ordering::Relaxed); + arena.cur = chunk as usize; + arena.end = arena.cur + PIC_ARENA_CHUNK_BYTES; + } + let out = arena.cur; + arena.cur += bytes; + out as *mut u8 +} + +/// Resolve a per-site cache slot to its cache, allocating a zeroed one and +/// publishing it into the slot on the first miss. +/// +/// `slot` is the address of the emitted `@perry_ic_N` pointer global (or a +/// stack `*mut T` in tests); `T` is the cache's word array — `PicCache` for a +/// property read, the eight-word write cache, the four-word Symbol cache — +/// and decides how many bytes the arena hands out. A null `slot` resolves to +/// a null cache: the write PIC's poly tail passes that to mean "run `[[Set]]` +/// and prime nothing" and every consumer already checks for it. +/// +/// # Safety +/// `slot` must be null or point at a live, 8-byte-aligned pointer-sized +/// location that holds either null or a cache previously returned by this +/// function (or any live `T`). All-zero bytes must be a valid `T`. +#[inline] +pub unsafe fn pic_slot_resolve(slot: *mut *mut T) -> *mut T { + if slot.is_null() { + return null_mut(); + } + let atomic = &*(slot as *const AtomicPtr); + let cur = atomic.load(Ordering::Acquire); + if !cur.is_null() { + return cur; + } + pic_slot_publish(atomic) +} + +/// The cache `slot` currently holds, without allocating: null for a site +/// that has never primed. Runtime entries that only *read* a site's cache +/// (the outlined write IC's hit check, a Symbol miss's epoch reset) go +/// through here so a concurrent first prime on another thread is an atomic +/// publication rather than a torn read. +/// +/// # Safety +/// `slot` must be null or a live cache slot (see [`pic_slot_resolve`]). +#[inline] +pub unsafe fn pic_slot_peek(slot: *mut *mut T) -> *mut T { + if slot.is_null() { + return null_mut(); + } + (*(slot as *const AtomicPtr)).load(Ordering::Acquire) +} + +#[cold] +#[inline(never)] +unsafe fn pic_slot_publish(atomic: &AtomicPtr) -> *mut T { + let fresh = pic_arena_alloc(std::mem::size_of::()) as *mut T; + match atomic.compare_exchange(null_mut(), fresh, Ordering::AcqRel, Ordering::Acquire) { + Ok(_) => { + PIC_SLOTS_RESOLVED.fetch_add(1, Ordering::Relaxed); + fresh + } + Err(existing) => { + PIC_SLOT_RACES.fetch_add(1, Ordering::Relaxed); + existing + } + } +} + +/// Number of sites whose slot holds an arena cache. +pub fn pic_slots_resolved() -> usize { + PIC_SLOTS_RESOLVED.load(Ordering::Relaxed) +} + +/// Bytes the arena has requested from the system allocator. +pub fn pic_arena_bytes() -> usize { + PIC_ARENA_BYTES.load(Ordering::Relaxed) +} + +/// `PERRY_GC_CENSUS`: the lazily allocated inline caches as one side-table +/// row — `entries` is the number of resolved sites, `bytes` the arena's +/// footprint (chunk granularity, so it over-reports by at most one chunk). +pub(crate) fn pic_slot_census() -> Vec { + vec![("ic.lazy_caches", pic_slots_resolved(), pic_arena_bytes())] +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn null_slot_resolves_to_null_cache() { + let cache = unsafe { pic_slot_resolve::<[i64; 12]>(null_mut()) }; + assert!(cache.is_null()); + } + + #[test] + fn first_resolve_allocates_a_zeroed_cache_and_publishes_it() { + let mut slot: *mut [i64; 12] = null_mut(); + let before = pic_slots_resolved(); + let cache = unsafe { pic_slot_resolve(&mut slot) }; + assert!(!cache.is_null(), "a fresh slot must resolve to a cache"); + assert_eq!(slot, cache, "the cache must be published into the slot"); + assert_eq!( + cache as usize % PIC_CACHE_ALIGN, + 0, + "caches are 32-byte aligned" + ); + assert!( + unsafe { (*cache).iter().all(|w| *w == 0) }, + "a fresh cache is all zeros" + ); + assert!( + pic_slots_resolved() > before, + "the census counter must observe the allocation" + ); + } + + #[test] + fn resolving_a_populated_slot_returns_the_same_cache_without_allocating() { + let mut slot: *mut [i64; 12] = null_mut(); + let first = unsafe { pic_slot_resolve(&mut slot) }; + unsafe { (*first)[0] = 0x4000_0000_0000_0001 }; + let resolved_before = pic_slots_resolved(); + let again = unsafe { pic_slot_resolve(&mut slot) }; + assert_eq!(first, again); + assert_eq!( + unsafe { (*again)[0] }, + 0x4000_0000_0000_0001, + "primed words survive" + ); + assert_eq!( + pic_slots_resolved(), + resolved_before, + "a second resolve of the same slot allocates nothing" + ); + } + + #[test] + fn peek_never_allocates() { + let mut slot: *mut [i64; 12] = null_mut(); + let before = pic_slots_resolved(); + assert!(unsafe { pic_slot_peek(&mut slot) }.is_null()); + assert!(slot.is_null(), "peek must not publish anything"); + assert_eq!(pic_slots_resolved(), before); + let cache = unsafe { pic_slot_resolve(&mut slot) }; + assert_eq!(unsafe { pic_slot_peek(&mut slot) }, cache); + assert!(unsafe { pic_slot_peek::<[i64; 12]>(null_mut()) }.is_null()); + } + + #[test] + fn a_stack_cache_pre_seeded_into_a_slot_is_honoured() { + let mut cache = [7i64; 8]; + let mut slot: *mut [i64; 8] = &mut cache; + let resolved = unsafe { pic_slot_resolve(&mut slot) }; + assert_eq!(resolved, &mut cache as *mut [i64; 8]); + } + + #[test] + fn consecutive_caches_are_packed_and_distinct() { + let mut a: *mut [i64; 12] = null_mut(); + let mut b: *mut [i64; 12] = null_mut(); + let ca = unsafe { pic_slot_resolve(&mut a) }; + let cb = unsafe { pic_slot_resolve(&mut b) }; + assert_ne!(ca, cb); + // Other tests allocate concurrently, so only the lower bound is exact: + // two caches can never overlap. + let (lo, hi) = if (ca as usize) < (cb as usize) { + (ca, cb) + } else { + (cb, ca) + }; + assert!(hi as usize - lo as usize >= 96, "caches must not overlap"); + } + + #[test] + fn eight_word_caches_fit_the_same_arena() { + let mut slot: *mut [i64; 8] = null_mut(); + let cache = unsafe { pic_slot_resolve(&mut slot) }; + assert!(!cache.is_null()); + assert_eq!(cache as usize % PIC_CACHE_ALIGN, 0); + unsafe { (*cache)[7] = 1 }; + assert_eq!(unsafe { (*slot)[7] }, 1); + } + + #[test] + fn concurrent_first_misses_agree_on_one_cache() { + use std::sync::atomic::AtomicUsize as Shared; + static SLOT: Shared = Shared::new(0); + let slot_addr = &SLOT as *const Shared as usize; + let handles: Vec<_> = (0..8) + .map(|_| { + std::thread::spawn(move || unsafe { + pic_slot_resolve(slot_addr as *mut *mut [i64; 12]) as usize + }) + }) + .collect(); + let results: Vec = handles.into_iter().map(|h| h.join().unwrap()).collect(); + assert!( + results.iter().all(|r| *r == results[0]), + "every thread must see one cache" + ); + assert_eq!(SLOT.load(Ordering::Relaxed), results[0]); + } +} diff --git a/crates/perry-runtime/src/object/mod.rs b/crates/perry-runtime/src/object/mod.rs index 2699829b45..3d8bc8563a 100644 --- a/crates/perry-runtime/src/object/mod.rs +++ b/crates/perry-runtime/src/object/mod.rs @@ -141,6 +141,7 @@ mod polymorphic_index_symbol_tests; mod primitive_proto_thunks; mod property_key; pub(crate) mod prototype_chain; +pub(crate) mod shape_carriers; pub(crate) mod shapes; pub(crate) use shapes::ShapeTable; mod prototype_helpers; @@ -687,6 +688,7 @@ fn shape_cache_insert(shape_id: u32, keys_array: *mut ArrayHeader) { .borrow_mut() .insert(shape_id, (keys_array, runtime_shape_id)); crate::gc::runtime_write_barrier_root_raw_ptr(keys_array); + shape_carriers::note_shape_id(runtime_shape_id); } /// Thread-local shape-transition cache for the dynamic-key write path @@ -988,6 +990,10 @@ fn transition_cache_lookup( return None; } } + // A weak, unstabilized entry must not publish a retired id. + if !shape_carriers::unstable_target_resolves(entry) { + return None; + } Some((entry.next_keys, entry_slot_idx, entry.target_shape_id)) } else { None @@ -1048,6 +1054,9 @@ fn transition_cache_insert( entry.slot_idx = slot_idx | (len_marker << 24); entry.target_len = target_len; }); + if target_len != 0 { + shape_carriers::note_shape_id(target_shape_id); + } if !array_tail_owner.is_null() { array_tail_transition::record_numeric_tail_transition( array_tail_owner, diff --git a/crates/perry-runtime/src/object/native_call_method/common_methods.rs b/crates/perry-runtime/src/object/native_call_method/common_methods.rs index c7782d7d60..12a9e001eb 100644 --- a/crates/perry-runtime/src/object/native_call_method/common_methods.rs +++ b/crates/perry-runtime/src/object/native_call_method/common_methods.rs @@ -802,15 +802,28 @@ pub(super) unsafe fn dispatch_common( } else { payload }; - let s = if n.fract() == 0.0 - && n.abs() < crate::builtins::INT_EXACT_FASTPATH_LIMIT - { - (n as i64).to_string() - } else { - n.to_string() - }; - let str_ptr = - crate::string::js_string_from_bytes(s.as_ptr(), s.len() as u32); + // #9713: `NumberToString`, not Rust's `{}`. A bare + // `f64::to_string()` prints `inf` for Infinity and the + // full decimal expansion past the exponential + // thresholds (`1e21` → `1000000000000000000000`, + // `Number.EPSILON` → `0.000…0002220446049250313`). + // `js_number_to_string` carries the spec's + // `|n| >= 1e21 || |n| < 1e-6` switch and its own + // integer fast path, so the local one is redundant too. + // + // A boxed receiver takes a radix like an unboxed one + // (`new Number(255).toString(16)` is "ff"); this arm + // dropped the argument entirely and answered "255". + // `js_jsvalue_to_string_radix` already accepts a boxed + // Number receiver, so hand it the box, not the payload. + let radix_arg = refreshed_args().first().copied(); + if let Some(r) = radix_arg { + if !JSValue::from_bits(r.to_bits()).is_undefined() { + let str_ptr = crate::value::js_jsvalue_to_string_radix(object, r); + return Some(f64::from_bits(JSValue::string_ptr(str_ptr).bits())); + } + } + let str_ptr = crate::string::js_number_to_string(n); return Some(f64::from_bits(JSValue::string_ptr(str_ptr).bits())); } Some("Boolean") => { @@ -861,12 +874,10 @@ pub(super) unsafe fn dispatch_common( crate::value::js_jsvalue_to_string_radix(object, radix_arg.unwrap()); return Some(f64::from_bits(JSValue::string_ptr(str_ptr).bits())); } - let s = if n.fract() == 0.0 && n.abs() < crate::builtins::INT_EXACT_FASTPATH_LIMIT { - (n as i64).to_string() - } else { - n.to_string() - }; - let str_ptr = crate::string::js_string_from_bytes(s.as_ptr(), s.len() as u32); + // #9713: same as the boxed-Number arm above — `NumberToString`, + // not Rust's `f64` Display. This is the arm a dynamic + // `x["toString"]()` on a plain number reaches. + let str_ptr = crate::string::js_number_to_string(n); return Some(f64::from_bits(JSValue::string_ptr(str_ptr).bits())); } else if jsval.is_bool() { let s = if jsval.as_bool() { "true" } else { "false" }; diff --git a/crates/perry-runtime/src/object/native_call_method/primitive_methods.rs b/crates/perry-runtime/src/object/native_call_method/primitive_methods.rs index 0a082e013d..39f7e6d8bb 100644 --- a/crates/perry-runtime/src/object/native_call_method/primitive_methods.rs +++ b/crates/perry-runtime/src/object/native_call_method/primitive_methods.rs @@ -222,15 +222,34 @@ pub(super) unsafe fn dispatch_primitive( } else { payload }; - let s = if n.fract() == 0.0 - && n.abs() < crate::builtins::INT_EXACT_FASTPATH_LIMIT - { - (n as i64).to_string() + // #9713: `NumberToString`, not Rust's `{}`. A bare + // `f64::to_string()` prints `inf` for Infinity and the + // full decimal expansion past the exponential + // thresholds (`1e21` → `1000000000000000000000`, + // `Number.EPSILON` → `0.000…0002220446049250313`). + // `js_number_to_string` carries the spec's + // `|n| >= 1e21 || |n| < 1e-6` switch and its own + // integer fast path, so the local one is redundant too. + // + // A boxed receiver takes a radix like an unboxed one + // (`new Number(255).toString(16)` is "ff"); this arm + // dropped the argument entirely and answered "255". + // `js_jsvalue_to_string_radix` already accepts a boxed + // Number receiver, so hand it the box, not the payload. + // `toLocaleString`'s argument is a locale, not a + // radix, so only `toString` consumes it here. + let radix_arg = if method_name == "toString" { + refreshed_args().first().copied() } else { - n.to_string() + None }; - let str_ptr = - crate::string::js_string_from_bytes(s.as_ptr(), s.len() as u32); + if let Some(r) = radix_arg { + if !JSValue::from_bits(r.to_bits()).is_undefined() { + let str_ptr = crate::value::js_jsvalue_to_string_radix(object, r); + return Some(f64::from_bits(JSValue::string_ptr(str_ptr).bits())); + } + } + let str_ptr = crate::string::js_number_to_string(n); return Some(f64::from_bits(JSValue::string_ptr(str_ptr).bits())); } Some("Boolean") => { diff --git a/crates/perry-runtime/src/object/shape_carriers.rs b/crates/perry-runtime/src/object/shape_carriers.rs new file mode 100644 index 0000000000..4ceb57b2bd --- /dev/null +++ b/crates/perry-runtime/src/object/shape_carriers.rs @@ -0,0 +1,83 @@ +//! ShapeId owners that may stamp an id after its last receiver dies (#9726). +//! +//! The descriptor table is weak with respect to receivers, but these runtime +//! caches are active metadata owners. Their bits are set on insertion and +//! rebuilt from exact table occupancy after every full trace. Generated module +//! globals are process-lifetime owners and use the record's separate external +//! carrier bit instead. + +use super::*; + +#[inline] +pub(crate) fn note_shape_id(shape_id: u32) { + unsafe { shapes::note_cache_carrier(shapes::shape_descriptor_by_id(shape_id)) }; +} + +#[inline] +fn target_descriptor_resolves(entry: TransitionEntry, expected_len: u32) -> bool { + shapes::shape_descriptor_by_id(entry.target_shape_id).is_some_and(|descriptor| { + descriptor.keys == entry.next_keys as u64 && descriptor.logical_key_count == expected_len + }) +} + +/// The runtime-only transition path revalidates weak, unstabilized entries +/// before publishing their target id. Generated probes reject these entries. +#[inline] +pub(crate) fn unstable_target_resolves(entry: TransitionEntry) -> bool { + let expected_len = (entry.slot_idx & TRANSITION_SLOT_IDX_MASK).wrapping_add(1); + target_descriptor_resolves(entry, expected_len) +} + +/// A stabilized entry is a call-free generated-code publisher only while its +/// exact target array and descriptor facts still agree with the cached edge. +#[inline] +fn stable_target_resolves(entry: TransitionEntry) -> bool { + let expected_len = (entry.slot_idx & TRANSITION_SLOT_IDX_MASK).wrapping_add(1); + if entry.target_len != expected_len { + return false; + } + let Some(header) = + (unsafe { crate::value::addr_class::try_read_tracked_gc_header(entry.next_keys) }) + else { + return false; + }; + unsafe { + if (*header.as_ptr()).obj_type != crate::gc::GC_TYPE_ARRAY { + return false; + } + let keys = entry.next_keys as *const ArrayHeader; + if (*keys).length != expected_len || (*keys).length > (*keys).capacity { + return false; + } + } + target_descriptor_resolves(entry, expected_len) +} + +/// Rebuild transient cache ownership after a full trace. This runs before a +/// synchronous trace's uncarried-descriptor prune, so every surviving table +/// entry that can publish a ShapeId has already claimed it. +pub(crate) fn recompute_after_full_trace() { + // Clears every transient bit, then re-notes both directions of the + // Array-subclass cache. Permanent external owners use a different bit. + array_tail_transition::recompute_cache_carriers_after_full_trace(); + + with_transition_cache(|table| unsafe { + for entry in (*table).iter() { + if stable_target_resolves(*entry) { + note_shape_id(entry.target_shape_id); + } + } + }); + + let state = crate::state::state(); + unsafe { + for entry in (&*state.object_hot.shape_inline_cache.get()).iter() { + note_shape_id(entry.runtime_shape_id); + } + } + for &(keys, runtime_shape_id) in state.object_hot.shape_cache_overflow.borrow().values() { + if !keys.is_null() { + note_shape_id(runtime_shape_id); + } + } +} diff --git a/crates/perry-runtime/src/object/shapes.rs b/crates/perry-runtime/src/object/shapes.rs index 5ec883968c..049f72295f 100644 --- a/crates/perry-runtime/src/object/shapes.rs +++ b/crates/perry-runtime/src/object/shapes.rs @@ -48,8 +48,9 @@ pub(crate) use shapes_slot_list::{ try_update_stable_tombstone_shape, try_update_stable_tombstone_shape_cached, SlotList, }; use shapes_store::{ - IdList, ShapeRecord, ShapeSlab, RECORD_FLAG_CACHE_CARRIER, RECORD_FLAG_FACTS_INDEXED, - RECORD_FLAG_OLD_CARRIER, RECORD_FLAG_OLD_CARRIER_SEEN, + IdList, ShapeRecord, ShapeSlab, RECORD_FLAG_CACHE_CARRIER, RECORD_FLAG_CARRIED_SEEN, + RECORD_FLAG_EXTERNAL_CARRIER, RECORD_FLAG_FACTS_INDEXED, RECORD_FLAG_OLD_CARRIER, + RECORD_FLAG_OLD_CARRIER_SEEN, }; #[derive(Clone)] @@ -634,6 +635,29 @@ pub(crate) unsafe fn note_old_generation_carrier(descriptor: Option) { + let Some(descriptor) = descriptor else { + return; + }; + if descriptor.record != 0 { + (*(descriptor.record as *mut ShapeRecord)).set(RECORD_FLAG_CARRIED_SEEN, true); + } +} + +#[inline] +pub(crate) unsafe fn note_external_shape_carrier(descriptor: Option) { + let Some(descriptor) = descriptor else { + return; + }; + if descriptor.record != 0 { + (*(descriptor.record as *mut ShapeRecord)).set(RECORD_FLAG_EXTERNAL_CARRIER, true); + } +} + /// Retain a descriptor while an agent-local optimization cache can reinstall /// its ShapeId. Cache tables live with `RuntimeState`; the bit is recomputed /// from live table occupancy after every full trace @@ -700,10 +724,11 @@ pub(crate) fn clear_all_cache_carriers() { /// Recompute the old-carrier gate from the trace that just finished. /// /// A FULL trace enumerates every live object, so the notes it accumulated are -/// exactly the shapes old objects still carry; adopt them and clear the -/// accumulator. Minors only ever ADD notes, which is why the gate needs a full -/// trace to shed a shape whose last old carrier died — the same rule that -/// governs every other old-generation reclamation. +/// exactly the shapes old objects still carry; adopt them and clear both the +/// old-carrier accumulator and the all-generation carried note. The latter is +/// consumed by synchronous-full descriptor retirement immediately before this +/// rotation. Budgeted full cycles clear it without retiring because their +/// sliced trace is not a complete carrier census. pub(crate) fn rotate_old_carrier_epoch_after_full_trace() { crate::state::state().shapes.slab().for_each(|_, record| { // SAFETY: live slab record, single-threaded agent. @@ -711,6 +736,7 @@ pub(crate) fn rotate_old_carrier_epoch_after_full_trace() { let seen = (*record).has(RECORD_FLAG_OLD_CARRIER_SEEN); (*record).set(RECORD_FLAG_OLD_CARRIER, seen); (*record).set(RECORD_FLAG_OLD_CARRIER_SEEN, false); + (*record).set(RECORD_FLAG_CARRIED_SEEN, false); } }); } @@ -724,7 +750,10 @@ pub(crate) fn rotate_old_carrier_epoch_after_full_trace() { /// rooted keys global as an integer heap word on every target. #[no_mangle] pub extern "C" fn js_object_shape_id_for_keys(keys: u64, key_count: u32) -> u32 { - shape_id_for_keys_ensure(keys as usize as *const ArrayHeader, key_count) + let id = shape_id_for_keys_ensure(keys as usize as *const ArrayHeader, key_count); + // SAFETY: `id` was resolved from this agent's live slab record above. + unsafe { note_external_shape_carrier(shape_descriptor_by_id(id)) }; + id } /// Mint a process-global ShapeId for a codegen-registered typed layout and @@ -1323,10 +1352,9 @@ fn retire_owned_shape_siblings(keys: u64, keep: u32) { .copied() .filter(|&id| { id != keep - && table - .slab() - .get(id) - .is_some_and(|record| !record.has(RECORD_FLAG_CACHE_CARRIER)) + && table.slab().get(id).is_some_and(|record| { + !record.has(RECORD_FLAG_CACHE_CARRIER | RECORD_FLAG_EXTERNAL_CARRIER) + }) }) .collect() }) @@ -1753,6 +1781,29 @@ fn shape_keys_address_is_recycled(addr: usize) -> bool { } } +/// Retire descriptors no live receiver carried during the just-completed +/// synchronous full trace and no runtime metadata owner can reinstall. +/// +/// The caller must run this while the full trace's `CARRIED_SEEN` notes are +/// intact and only after cache-carrier bits have been rebuilt from live table +/// occupancy. Minor and budgeted cycles are deliberately ineligible: neither +/// provides an exact, stop-the-world enumeration of every live receiver. +pub(crate) fn prune_uncarried_shape_descriptors_after_full_trace() { + let table = &crate::state::state().shapes; + let mut inner = table.inner.borrow_mut(); + let mut stale = Vec::new(); + table.slab().for_each(|id, record| { + // SAFETY: live slab record, read immediately under agent ownership. + let record = unsafe { &*record }; + if !record.has(RECORD_FLAG_CARRIED_SEEN) && !record.cache_carrier() { + stale.push(id); + } + }); + for id in stale { + remove_descriptor_and_reverse_indices(&mut inner, id); + } +} + /// Post-trace weak-table prune: drop slot indices and by-id descriptors whose /// keys array is dead. A live object has already traced its authoritative /// header edge and synchronized the descriptor named by its ShapeId, so a @@ -2066,7 +2117,7 @@ pub(crate) fn shape_table_liveness_census( uncarried += 1; // SAFETY: live slab record, read immediately. let record = unsafe { *record }; - if record.has(RECORD_FLAG_CACHE_CARRIER) { + if record.cache_carrier() { uncarried_cache += 1; } else if record.has(RECORD_FLAG_OLD_CARRIER) { uncarried_old += 1; diff --git a/crates/perry-runtime/src/object/shapes_slot_list.rs b/crates/perry-runtime/src/object/shapes_slot_list.rs index 0523e4df43..2dd6d74518 100644 --- a/crates/perry-runtime/src/object/shapes_slot_list.rs +++ b/crates/perry-runtime/src/object/shapes_slot_list.rs @@ -517,7 +517,7 @@ pub(super) fn install_external_shape_id( return false; } let keys = keys as usize as u64; - let record = ShapeRecord::new( + let mut record = ShapeRecord::new( keys, logical_key_count, live_inline_slot_count, @@ -525,10 +525,12 @@ pub(super) fn install_external_shape_id( super::ShapeObjectKind::Ordinary, 0, ); + record.set(super::shapes_store::RECORD_FLAG_EXTERNAL_CARRIER, true); let table = &crate::state::state().shapes; let mut inner = table.inner.borrow_mut(); - if let Some(existing) = table.slab().get(id) { - return existing.facts_match( + if let Some(existing) = table.slab().record_ptr(id) { + // SAFETY: live slab record, single-threaded agent. + let matches = unsafe { &*existing }.facts_match( keys, logical_key_count, live_inline_slot_count, @@ -536,6 +538,11 @@ pub(super) fn install_external_shape_id( super::ShapeObjectKind::Ordinary, 0, ); + if matches { + // SAFETY: same record and agent discipline as above. + unsafe { (*existing).set(super::shapes_store::RECORD_FLAG_EXTERNAL_CARRIER, true) }; + } + return matches; } // A worker can have minted an equivalent local descriptor before module // initialization installs the process-global codegen id. Keep both id diff --git a/crates/perry-runtime/src/object/shapes_store.rs b/crates/perry-runtime/src/object/shapes_store.rs index b305741f0f..ec9d0b812a 100644 --- a/crates/perry-runtime/src/object/shapes_store.rs +++ b/crates/perry-runtime/src/object/shapes_store.rs @@ -41,6 +41,8 @@ pub(super) const RECORD_FLAG_OLD_CARRIER: u8 = 1 << 2; pub(super) const RECORD_FLAG_OLD_CARRIER_SEEN: u8 = 1 << 3; pub(super) const RECORD_FLAG_CACHE_CARRIER: u8 = 1 << 4; pub(super) const RECORD_FLAG_KIND_CLASS: u8 = 1 << 5; +pub(super) const RECORD_FLAG_CARRIED_SEEN: u8 = 1 << 6; +pub(super) const RECORD_FLAG_EXTERNAL_CARRIER: u8 = 1 << 7; /// The table-owned record of one ShapeId. `keys` is first and 8-aligned: it /// is the word the collector marks through and rewrites in place. @@ -91,6 +93,13 @@ impl ShapeRecord { } } + /// A runtime table or process-lifetime generated-code global may reinstall + /// this id even while no object currently carries it. + #[inline] + pub(super) fn cache_carrier(&self) -> bool { + self.has(RECORD_FLAG_CACHE_CARRIER | RECORD_FLAG_EXTERNAL_CARRIER) + } + #[inline] pub(super) fn object_kind(&self) -> ShapeObjectKind { if self.has(RECORD_FLAG_KIND_CLASS) { @@ -171,7 +180,7 @@ impl ShapeRecord { keys: self.keys, record: record as usize, old_carrier: self.has(RECORD_FLAG_OLD_CARRIER), - cache_carrier: self.has(RECORD_FLAG_CACHE_CARRIER), + cache_carrier: self.cache_carrier(), logical_key_count: self.logical_key_count, live_inline_slot_count: self.live_inline_slot_count, semantic_generation: self.semantic_generation, diff --git a/crates/perry-runtime/src/object/with_env.rs b/crates/perry-runtime/src/object/with_env.rs index a7e2f03396..175765f3aa 100644 --- a/crates/perry-runtime/src/object/with_env.rs +++ b/crates/perry-runtime/src/object/with_env.rs @@ -101,7 +101,7 @@ pub extern "C" fn js_with_set_binding( if strict != 0 && !has_property(coerced, key) { crate::error::js_throw_reference_error_unresolvable_assignment(key_as_value(key)); } - crate::proxy::js_put_value_set_ic_miss(coerced, key, value, strict, std::ptr::null_mut()) + crate::proxy::js_put_value_set_ic_miss(coerced, key, value, strict, std::ptr::null_mut(), 0) } #[no_mangle] diff --git a/crates/perry-runtime/src/proxy.rs b/crates/perry-runtime/src/proxy.rs index a4e8213aeb..1975eedc69 100644 --- a/crates/perry-runtime/src/proxy.rs +++ b/crates/perry-runtime/src/proxy.rs @@ -34,6 +34,7 @@ pub use put_value::{js_proxy_set, js_put_value_set}; pub(crate) use put_value::{ js_put_value_set_ic_miss, proxy_set_with_receiver, IC_SLOT_OVERFLOW_BIT, }; +pub use put_value::{write_pic_way_entry, WritePicCache, WritePicCacheSlot, WRITE_PIC_WORDS}; mod json; mod metadata; mod own_keys; @@ -3280,8 +3281,10 @@ mod tests { let key_ptr = crate::string::js_string_from_bytes(b"n".as_ptr(), 1); let key_ptr = crate::string::js_string_intern(key_ptr, fnv1a(b"n")); - let mut cache = [0i64; 2]; - let stored = put_value::js_put_value_set_ic_miss(target, key_ptr, 7.0, 0, &mut cache); + let mut cache: put_value::WritePicCache = [0; put_value::WRITE_PIC_WORDS]; + let mut cache_slot: put_value::WritePicCacheSlot = &mut cache; + let stored = + put_value::js_put_value_set_ic_miss(target, key_ptr, 7.0, 0, &mut cache_slot, 0); assert_eq!(stored, 7.0); let expected_token = unsafe { crate::object::shapes::PIC_ID_TOKEN_BIT @@ -3303,13 +3306,15 @@ mod tests { (unmarked as *mut u8).sub(crate::gc::GC_HEADER_SIZE) as *mut crate::gc::GcHeader; (*header)._reserved &= !crate::gc::OBJ_FLAG_PLAIN_ORDINARY; } - let mut cache2 = [0i64; 2]; + let mut cache2: put_value::WritePicCache = [0; put_value::WRITE_PIC_WORDS]; + let mut cache2_slot: put_value::WritePicCacheSlot = &mut cache2; let stored = put_value::js_put_value_set_ic_miss( f64::from_bits(value.bits()), key_ptr, 9.0, 0, - &mut cache2, + &mut cache2_slot, + 0, ); assert_eq!(stored, 9.0, "the write itself still succeeds"); assert_eq!( diff --git a/crates/perry-runtime/src/proxy/put_value.rs b/crates/perry-runtime/src/proxy/put_value.rs index c41a52232b..1f4e8acdbb 100644 --- a/crates/perry-runtime/src/proxy/put_value.rs +++ b/crates/perry-runtime/src/proxy/put_value.rs @@ -328,13 +328,56 @@ pub extern "C" fn js_put_value_set( } value_handle.get_nanbox_f64() } + +/// Words in a per-site static-key write cache (`[shape_token, slot]` × the +/// four inline ways) and in its outlined poly tail. Both are views of the +/// same `PicCacheSlot` family the read PIC uses; the arena hands out exactly +/// this many words for them (#9708). +pub const WRITE_PIC_WORDS: usize = 8; + +/// A per-site write cache, as the emitted slot resolves it. +pub type WritePicCache = [i64; WRITE_PIC_WORDS]; +/// The emitted `@perry_ic_N = private global ptr null` for a write site: +/// null until the site's first priming miss (#9708). +pub type WritePicCacheSlot = *mut WritePicCache; + +/// Address of way `way` (a `[token, slot]` pair) inside the cache `slot` +/// resolves to, allocating the cache on first use. A null `slot` yields null: +/// the poly tail passes that to mean "prime nothing". +/// +/// # Safety +/// `slot` must be null or a live write-cache slot; `way` must index a pair +/// inside `WRITE_PIC_WORDS`. +#[inline] +pub unsafe fn write_pic_way_entry(slot: *mut WritePicCacheSlot, way: i32) -> *mut [i64; 2] { + if slot.is_null() { + return std::ptr::null_mut(); + } + debug_assert!( + (0..(WRITE_PIC_WORDS / 2) as i32).contains(&way), + "way {way} is outside the {WRITE_PIC_WORDS}-word write cache" + ); + let cache = crate::object::pic_slot_resolve(slot); + (cache as *mut i64).add(way as usize * 2) as *mut [i64; 2] +} + +/// Miss path for one way of the codegen-emitted polymorphic PutValue cache. +/// +/// The full strict/sloppy `[[Set]]` semantics run first. Only a successful +/// ordinary own-data overwrite may prime `[shape_token, slot]`; every +/// exotic, descriptor-bearing, frozen, or typed-layout-intact receiver +/// remains on the miss path. `cache_slot` is the site's +/// [`WritePicCacheSlot`] address and `way` the pair to prime; the cache is +/// allocated only when a prime actually happens (#9708), so a site that never +/// sees a primeable receiver never allocates. #[no_mangle] pub extern "C" fn js_put_value_set_ic_miss( target: f64, key: *const crate::StringHeader, value: f64, strict: i32, - cache: *mut [i64; 2], + cache_slot: *mut WritePicCacheSlot, + way: i32, ) -> f64 { let scope = crate::gc::RuntimeHandleScope::new(); let target_handle = scope.root_nanbox_f64(target); @@ -357,7 +400,7 @@ pub extern "C" fn js_put_value_set_ic_miss( ) }); - if cache.is_null() { + if cache_slot.is_null() { return result; } @@ -464,8 +507,9 @@ pub extern "C" fn js_put_value_set_ic_miss( // Publish the token last conceptually: a zero-initialized or stale // token cannot hit this slot until it matches this receiver's current // discriminated shape token. Perry's read PIC uses the same format. - (*cache)[1] = slot_word as i64; - (*cache)[0] = shape_token as i64; + let entry = write_pic_way_entry(cache_slot, way); + (*entry)[1] = slot_word as i64; + (*entry)[0] = shape_token as i64; // Rotating-key sites overflow the single-slot site cache immediately; // the global stub is what lets them hit. Key bits from the rooted key. // `key_handle` here roots a raw STRING pointer (this entry's key arrives @@ -497,23 +541,33 @@ const STATIC_PIC_TAIL_WAYS: usize = 4; /// semantics. Empty ways are filled in order and a full cache is never /// overwritten, so a stable eight-shape site settles instead of continuously /// replacing its fourth entry. +/// +/// `tail_slot` is the tail's own [`WritePicCacheSlot`]: a site reaches this +/// helper only once its four inline ways are full, so the tail cache is not +/// allocated before a fifth shape actually shows up (#9708). An unallocated +/// tail reads as four empty ways. #[no_mangle] pub extern "C" fn js_put_value_set_ic_poly_tail( - cache: *mut [i64; 8], + tail_slot: *mut WritePicCacheSlot, target: f64, key: *const crate::StringHeader, value: f64, strict: i32, ) -> f64 { - if !cache.is_null() { + if !tail_slot.is_null() { unsafe { + let cache = crate::object::pic_slot_peek(tail_slot); + if cache.is_null() { + return js_put_value_set_ic_miss(target, key, value, strict, tail_slot, 0); + } let c = &mut *cache; for way in 0..STATIC_PIC_TAIL_WAYS { let word = way * 2; let token = c[word] as u64; if token == 0 { - let entry = c.as_mut_ptr().add(word) as *mut [i64; 2]; - return js_put_value_set_ic_miss(target, key, value, strict, entry); + return js_put_value_set_ic_miss( + target, key, value, strict, tail_slot, way as i32, + ); } if let Some(result) = dyn_ic_try_store(target, token, c[word + 1] as u32, value) { return result; @@ -524,7 +578,7 @@ pub extern "C" fn js_put_value_set_ic_poly_tail( // More than eight stable shapes remain bounded and semantically correct: // execute the ordinary write without evicting a useful settled entry. - js_put_value_set_ic_miss(target, key, value, strict, std::ptr::null_mut()) + js_put_value_set_ic_miss(target, key, value, strict, std::ptr::null_mut(), 0) } // --------------------------------------------------------------------------- @@ -547,14 +601,21 @@ const DYN_IC_WAYS: usize = 3; /// Outlined dynamic-key PutValue with per-site cache. Fast path: shape token /// + key-bits match -> validated own-slot overwrite. Everything else falls /// through to the full `[[Set]]` semantics and re-primes. +/// +/// `cache_slot` is the site's [`WritePicCacheSlot`] address; the cache is read +/// through it here and allocated by the miss handler on the first prime +/// (#9708). #[no_mangle] pub extern "C" fn js_put_value_set_dyn_ic( - cache: *mut [i64; 8], + cache_slot: *mut WritePicCacheSlot, target: f64, key: f64, value: f64, strict: i32, ) -> f64 { + // SAFETY: a non-null slot is the emitted pointer global (or a test's + // stack slot); reading it is the same load the inline hit path performs. + let cache = unsafe { crate::object::pic_slot_peek(cache_slot) }; if !cache.is_null() { let hit = unsafe { let c = &*cache; @@ -585,7 +646,7 @@ pub extern "C" fn js_put_value_set_dyn_ic( return ret; } } - js_put_value_set_dyn_ic_miss(cache, target, key, value, strict) + js_put_value_set_dyn_ic_miss(cache_slot, target, key, value, strict) } /// Megamorphic stub cache for dynamic string-keyed WRITES — V8's answer to a @@ -832,8 +893,13 @@ unsafe fn dyn_ic_try_store(target: f64, token: u64, slot: u32, value: f64) -> Op // link would otherwise dead-strip the IC entry. #[cfg(feature = "keepalive-anchors")] #[used] -static KEEP_JS_PUT_VALUE_SET_DYN_IC: extern "C" fn(*mut [i64; 8], f64, f64, f64, i32) -> f64 = - js_put_value_set_dyn_ic; +static KEEP_JS_PUT_VALUE_SET_DYN_IC: extern "C" fn( + *mut WritePicCacheSlot, + f64, + f64, + f64, + i32, +) -> f64 = js_put_value_set_dyn_ic; /// #9287 transition-IC: the emitted hit's SPILL-APPEND arm. /// @@ -923,12 +989,16 @@ extern "C" fn transition_ic_report_shim() { /// its VALUE (SSO or heap) instead of requiring an interned pointer. #[no_mangle] pub extern "C" fn js_put_value_set_dyn_ic_miss( - cache: *mut [i64; 8], + cache_slot: *mut WritePicCacheSlot, target: f64, key: f64, value: f64, strict: i32, ) -> f64 { + // The slot is read once here; a null cache means the site has never + // primed, which the stub probe below treats as "no site token" exactly as + // it treated an all-zero global (#9708). + let cache = unsafe { crate::object::pic_slot_peek(cache_slot) }; // The compiled inline IC (`lower_put_value_dyn_ic_inline`) walks its three // ways in GENERATED code and calls straight here on a way miss — it never // enters `js_put_value_set_dyn_ic` above. A rotating-key site therefore @@ -1026,7 +1096,7 @@ pub extern "C" fn js_put_value_set_dyn_ic_miss( target_handle.get_nanbox_f64(), strict, ); - if cache.is_null() { + if cache_slot.is_null() { return result; } unsafe { @@ -1121,7 +1191,7 @@ pub extern "C" fn js_put_value_set_dyn_ic_miss( if idx >= alloc_limit { return result; } - let c = &mut *cache; + let c = &mut *crate::object::pic_slot_resolve(cache_slot); if c[0] as u64 != shape_token { // New shape at this site: restart the way set. *c = [0; 8]; @@ -1173,13 +1243,14 @@ mod tests { scope.root_raw_mut_ptr(crate::object::js_object_alloc(ANON_CLASS_ID, 0)); let user_class_handle = scope.root_raw_mut_ptr(crate::object::js_object_alloc(USER_CLASS_ID, 0)); - let mut cache = [0i64; 8]; + let mut cache: WritePicCache = [0; WRITE_PIC_WORDS]; + let mut cache_slot: WritePicCacheSlot = &mut cache; let first = first_handle.get_raw_mut_ptr::(); let learned_predecessor = unsafe { crate::object::shapes::object_shape_stamp(first) }; assert_eq!( js_put_value_set_dyn_ic_miss( - &mut cache, + &mut cache_slot, crate::value::js_nanbox_pointer(first as i64), key_handle.get_nanbox_f64(), 11.0, @@ -1209,7 +1280,7 @@ mod tests { ); assert_eq!( js_put_value_set_dyn_ic_miss( - &mut cache, + &mut cache_slot, crate::value::js_nanbox_pointer(second as i64), key_handle.get_nanbox_f64(), 29.0, @@ -1238,7 +1309,7 @@ mod tests { let user_class = user_class_handle.get_raw_mut_ptr::(); assert_eq!( js_put_value_set_dyn_ic_miss( - &mut cache, + &mut cache_slot, crate::value::js_nanbox_pointer(user_class as i64), key_handle.get_nanbox_f64(), 47.0, diff --git a/crates/perry-runtime/src/symbol.rs b/crates/perry-runtime/src/symbol.rs index c7ff66a031..3abb55867f 100644 --- a/crates/perry-runtime/src/symbol.rs +++ b/crates/perry-runtime/src/symbol.rs @@ -58,7 +58,7 @@ pub use properties::{ pub(crate) use get::{has_own_symbol_property, inherited_symbol_property, own_symbol_property}; pub use get::{ js_object_get_symbol_property, js_object_get_symbol_property_ic_miss, - js_object_get_symbol_then_field_ic_miss, + js_object_get_symbol_then_field_ic_miss, SymbolPicCache, SymbolPicCacheSlot, SYMBOL_PIC_WORDS, }; // Iterator protocol, getOwnPropertySymbols, ToPrimitive. diff --git a/crates/perry-runtime/src/symbol/get.rs b/crates/perry-runtime/src/symbol/get.rs index ba6aa95351..503d041ccf 100644 --- a/crates/perry-runtime/src/symbol/get.rs +++ b/crates/perry-runtime/src/symbol/get.rs @@ -88,6 +88,26 @@ pub(crate) unsafe fn own_symbol_property(obj_f64: f64, sym_f64: f64) -> Option *mut u64 { + crate::object::pic_slot_peek(slot) as *mut u64 +} + /// Miss path for the generated weak own-Symbol-property IC. /// /// Cache layout (all `u64`): epoch, receiver bits, symbol bits, value bits. @@ -99,14 +119,17 @@ pub(crate) unsafe fn own_symbol_property(obj_f64: f64, sym_f64: f64) -> Option f64 { let obj_bits = obj_f64.to_bits(); - if !cache.is_null() && (obj_bits >> 48) == 0x7FFD { + if !cache_slot.is_null() && (obj_bits >> 48) == 0x7FFD { let obj_key = (obj_bits & POINTER_MASK) as usize; if let Some(header) = crate::value::addr_class::try_read_gc_header(obj_key) { if header.obj_type == crate::gc::GC_TYPE_OBJECT { @@ -128,6 +151,10 @@ pub unsafe extern "C" fn js_object_get_symbol_property_ic_miss( if let Some(value_bits) = value_bits { let epoch = PERRY_SYMBOL_PROPERTY_IC_EPOCH .load(std::sync::atomic::Ordering::Acquire); + // #9708: the cache is allocated here, on the first + // successful prime, never for a miss that cannot + // prime. + let cache = crate::object::pic_slot_resolve(cache_slot) as *mut u64; // Publish identity/value first and epoch last. The // generated hit path acquire-loads this first word. *cache.add(1) = obj_bits; @@ -160,9 +187,12 @@ pub unsafe extern "C" fn js_object_get_symbol_then_field_ic_miss( sym_f64: f64, key: *const crate::StringHeader, feedback_site_id: u64, - symbol_cache: *mut u64, - field_cache: *mut crate::object::PicCache, + symbol_cache_slot: *mut SymbolPicCacheSlot, + field_cache_slot: *mut crate::object::PicCacheSlot, ) -> f64 { + // #9708: both caches sit behind slots. Each is re-read after the miss + // that may allocate it; before that, a null cache is simply "not primed". + let symbol_cache = symbol_cache_of(symbol_cache_slot); if key.is_null() { if !symbol_cache.is_null() { (&*(symbol_cache as *const std::sync::atomic::AtomicU64)) @@ -175,8 +205,10 @@ pub unsafe extern "C" fn js_object_get_symbol_then_field_ic_miss( // whose accessor/prototype fallback may allocate before the named read. let scope = crate::gc::RuntimeHandleScope::new(); let key_handle = scope.root_nanbox_f64(crate::value::js_nanbox_string(key as i64)); - let intermediate = js_object_get_symbol_property_ic_miss(obj_f64, sym_f64, symbol_cache); + let intermediate = js_object_get_symbol_property_ic_miss(obj_f64, sym_f64, symbol_cache_slot); let intermediate_handle = scope.root_nanbox_f64(intermediate); + let symbol_cache = symbol_cache_of(symbol_cache_slot); + let field_cache = crate::object::pic_slot_peek(field_cache_slot); // Do not mistake an entry retained from the previously cached // intermediate object for a prime performed by this miss. @@ -192,9 +224,11 @@ pub unsafe extern "C" fn js_object_get_symbol_then_field_ic_miss( current.to_bits() as i64, key_now, feedback_site_id, - field_cache, + field_cache_slot, ) }); + // The named read may have been the site's first prime, which allocates. + let field_cache = crate::object::pic_slot_peek(field_cache_slot); // `cache[3]` is dereferenced by generated code only after cache[0]'s // acquire-load succeeds. Leave that epoch published solely when both miss @@ -1352,8 +1386,9 @@ mod own_data_ic_tests { let first = 41.0_f64; super::properties::js_object_set_symbol_property(obj, sym, first); - let mut cache = [0_u64; 12]; - let got = js_object_get_symbol_property_ic_miss(obj, sym, cache.as_mut_ptr()); + let mut cache: SymbolPicCache = [0; SYMBOL_PIC_WORDS]; + let mut cache_slot: SymbolPicCacheSlot = &mut cache; + let got = js_object_get_symbol_property_ic_miss(obj, sym, &mut cache_slot); assert_eq!(got.to_bits(), first.to_bits()); assert_eq!(cache[1], obj.to_bits()); assert_eq!(cache[2], sym.to_bits()); @@ -1371,7 +1406,7 @@ mod own_data_ic_tests { PERRY_SYMBOL_PROPERTY_IC_EPOCH.load(Ordering::Acquire), "a Symbol data write must make the generated hit guard fail" ); - let got = js_object_get_symbol_property_ic_miss(obj, sym, cache.as_mut_ptr()); + let got = js_object_get_symbol_property_ic_miss(obj, sym, &mut cache_slot); assert_eq!(got.to_bits(), second.to_bits()); assert_eq!(cache[3], second.to_bits()); } @@ -1382,10 +1417,11 @@ mod own_data_ic_tests { let _global = crate::gc::global_side_table_test_lock(); unsafe { let sym = super::constructors::js_symbol_new_empty(); - let mut cache = [0_u64; 12]; - let got = js_object_get_symbol_property_ic_miss(7.0, sym, cache.as_mut_ptr()); + let mut cache: SymbolPicCache = [0; SYMBOL_PIC_WORDS]; + let mut cache_slot: SymbolPicCacheSlot = &mut cache; + let got = js_object_get_symbol_property_ic_miss(7.0, sym, &mut cache_slot); assert_eq!(got.to_bits(), TAG_UNDEFINED); - assert_eq!(cache, [0_u64; 12]); + assert_eq!(cache, [0; SYMBOL_PIC_WORDS]); } } @@ -1404,15 +1440,17 @@ mod own_data_ic_tests { crate::object::js_object_set_field_by_name(metadata_ptr, key, 41.0); super::properties::js_object_set_symbol_property(owner, sym, metadata); - let mut symbol_cache = [0_u64; 12]; + let mut symbol_cache: SymbolPicCache = [0; SYMBOL_PIC_WORDS]; + let mut symbol_cache_slot: SymbolPicCacheSlot = &mut symbol_cache; let mut field_cache: crate::object::PicCache = [0; crate::object::PIC_CACHE_WORDS]; + let mut field_cache_slot: crate::object::PicCacheSlot = &mut field_cache; let first = js_object_get_symbol_then_field_ic_miss( owner, sym, key, 0, - symbol_cache.as_mut_ptr(), - &mut field_cache, + &mut symbol_cache_slot, + &mut field_cache_slot, ); let epoch_before_named_write = symbol_cache[0]; let slot = field_cache[1] as usize; @@ -1430,8 +1468,8 @@ mod own_data_ic_tests { sym, key, 0, - symbol_cache.as_mut_ptr(), - &mut field_cache, + &mut symbol_cache_slot, + &mut field_cache_slot, ); let epoch_after_named_write = PERRY_SYMBOL_PROPERTY_IC_EPOCH.load(Ordering::Acquire); crate::gc::gc_unsuppress(); @@ -1459,15 +1497,17 @@ mod own_data_ic_tests { let sym = super::constructors::js_symbol_new_empty(); let key = js_string_from_bytes(b"id".as_ptr(), 2); super::properties::js_object_set_symbol_property(owner, sym, 7.0); - let mut symbol_cache = [0_u64; 12]; + let mut symbol_cache: SymbolPicCache = [0; SYMBOL_PIC_WORDS]; + let mut symbol_cache_slot: SymbolPicCacheSlot = &mut symbol_cache; let mut field_cache: crate::object::PicCache = [0; crate::object::PIC_CACHE_WORDS]; + let mut field_cache_slot: crate::object::PicCacheSlot = &mut field_cache; let got = js_object_get_symbol_then_field_ic_miss( owner, sym, key, 0, - symbol_cache.as_mut_ptr(), - &mut field_cache, + &mut symbol_cache_slot, + &mut field_cache_slot, ); crate::gc::gc_unsuppress(); diff --git a/crates/perry-runtime/src/typed_feedback.rs b/crates/perry-runtime/src/typed_feedback.rs index ebdac4c887..9535f9cad8 100644 --- a/crates/perry-runtime/src/typed_feedback.rs +++ b/crates/perry-runtime/src/typed_feedback.rs @@ -1153,6 +1153,7 @@ pub use guards::{ js_typed_feedback_class_field_get_guard, js_typed_feedback_class_field_set_guard, js_typed_feedback_closure_direct_call_guard, js_typed_feedback_method_direct_call_guard, js_typed_feedback_native_call_method, js_typed_feedback_native_call_method_apply, + MethodPicCache, MethodPicCacheSlot, METHOD_PIC_WORDS, }; #[path = "typed_feedback/trace.rs"] diff --git a/crates/perry-runtime/src/typed_feedback/guards.rs b/crates/perry-runtime/src/typed_feedback/guards.rs index 53f0121659..339bc3ccff 100644 --- a/crates/perry-runtime/src/typed_feedback/guards.rs +++ b/crates/perry-runtime/src/typed_feedback/guards.rs @@ -1213,6 +1213,15 @@ pub extern "C" fn js_closure_exact_func_guard( } } +/// Words in a per-site imported-object own-method cache: the one +/// `(ShapeId << 32 | class_id)` token the emitted guard compares. +pub const METHOD_PIC_WORDS: usize = 1; +/// A per-site own-method cache, as the emitted slot resolves it. +pub type MethodPicCache = [u64; METHOD_PIC_WORDS]; +/// The emitted `@perry_ic_N = private global ptr null` for such a site: null +/// until the site's first priming miss (#9708). +pub type MethodPicCacheSlot = *mut MethodPicCache; + /// Revalidate and prime the shape token for an own object-literal method. /// /// The exported adapter object may append ordinary state fields during @@ -1228,6 +1237,10 @@ pub extern "C" fn js_closure_exact_func_guard( /// or mints a semantic successor. A spill-only metadata record is allowed: /// appending past the object's inline birth width creates one even though the /// original method slot and its lookup semantics remain unchanged. +/// +/// `cache_slot` is the site's [`MethodPicCacheSlot`] address (#9708). A miss +/// that cannot prime clears an existing cache's token but never allocates +/// one; only the publishing tail below resolves the slot. #[no_mangle] pub unsafe extern "C" fn js_object_own_method_cache_miss( receiver: f64, @@ -1236,12 +1249,15 @@ pub unsafe extern "C" fn js_object_own_method_cache_miss( method_name_ptr: *const i8, method_name_len: usize, expected_func_ptr: *const u8, - cache_token: *mut u64, + cache_slot: *mut MethodPicCacheSlot, ) -> u64 { - if !cache_token.is_null() { - *cache_token = 0; + { + let cache = crate::object::pic_slot_peek(cache_slot); + if !cache.is_null() { + (*cache)[0] = 0; + } } - if expected_class_id == 0 || expected_func_ptr.is_null() || cache_token.is_null() { + if expected_class_id == 0 || expected_func_ptr.is_null() || cache_slot.is_null() { return 0; } let Some(method_bytes) = method_name_bytes(method_name_ptr, method_name_len) else { @@ -1303,7 +1319,8 @@ pub unsafe extern "C" fn js_object_own_method_cache_miss( if !crate::object::shapes::is_shape_id(shape_id) { return 0; } - *cache_token = ((shape_id as u64) << 32) | expected_class_id as u64; + let cache = crate::object::pic_slot_resolve(cache_slot); + (*cache)[0] = ((shape_id as u64) << 32) | expected_class_id as u64; closure as u64 } @@ -1339,7 +1356,7 @@ mod keep_guard_symbols { #[cfg(feature = "keepalive-anchors")] #[used] static G3B: extern "C" fn(f64, *const u8) -> u64 = js_closure_exact_func_guard; #[cfg(feature = "keepalive-anchors")] - #[used] static G3C: unsafe extern "C" fn(f64, u32, u32, *const i8, usize, *const u8, *mut u64) -> u64 = js_object_own_method_cache_miss; + #[used] static G3C: unsafe extern "C" fn(f64, u32, u32, *const i8, usize, *const u8, *mut MethodPicCacheSlot) -> u64 = js_object_own_method_cache_miss; #[cfg(feature = "keepalive-anchors")] #[used] static G4: unsafe extern "C" fn(f64, u32, u32, u32) -> i32 = js_method_direct_shape_guard; #[cfg(feature = "keepalive-anchors")] diff --git a/crates/perry-runtime/src/typed_feedback/tests.rs b/crates/perry-runtime/src/typed_feedback/tests.rs index 935caf01dd..c12ed2c0c5 100644 --- a/crates/perry-runtime/src/typed_feedback/tests.rs +++ b/crates/perry-runtime/src/typed_feedback/tests.rs @@ -1391,7 +1391,7 @@ fn representation_lowering_helpers_have_lto_keepalive_anchors() { ( guards, "static G3C", - "static G3C: unsafe extern \"C\" fn(f64, u32, u32, *const i8, usize, *const u8, *mut u64) -> u64", + "static G3C: unsafe extern \"C\" fn(f64, u32, u32, *const i8, usize, *const u8, *mut MethodPicCacheSlot) -> u64", "js_object_own_method_cache_miss", ), ( @@ -2517,7 +2517,8 @@ fn own_method_cache_accepts_appends_and_rejects_live_method_mutation() { let closure_value = crate::value::js_nanbox_pointer(closure as i64); crate::object::js_object_set_field_by_name(object, method_key, closure_value); let receiver = crate::value::js_nanbox_pointer(object as i64); - let mut cache = 0; + let mut cache: MethodPicCache = [0]; + let mut cache_slot: MethodPicCacheSlot = &mut cache; let first = unsafe { js_object_own_method_cache_miss( @@ -2527,12 +2528,12 @@ fn own_method_cache_accepts_appends_and_rejects_live_method_mutation() { b"method".as_ptr() as *const i8, 6, fn_ptr, - &mut cache, + &mut cache_slot, ) }; assert_eq!(first, closure as u64); - assert_ne!(cache, 0); - let initial_shape_token = cache; + assert_ne!(cache[0], 0); + let initial_shape_token = cache[0]; crate::object::js_object_set_field_by_name(object, extra_key, 42.0); let after_append = unsafe { @@ -2543,12 +2544,12 @@ fn own_method_cache_accepts_appends_and_rejects_live_method_mutation() { b"method".as_ptr() as *const i8, 6, fn_ptr, - &mut cache, + &mut cache_slot, ) }; assert_eq!(after_append, closure as u64); assert_ne!( - cache, initial_shape_token, + cache[0], initial_shape_token, "append must publish the live successor shape" ); @@ -2565,11 +2566,11 @@ fn own_method_cache_accepts_appends_and_rejects_live_method_mutation() { b"method".as_ptr() as *const i8, 6, fn_ptr, - &mut cache, + &mut cache_slot, ) }; assert_eq!(after_spilled_append, closure as u64); - assert_ne!(cache, 0); + assert_ne!(cache[0], 0); let replacement = crate::closure::js_closure_alloc_singleton(test_direct_method_ptr()); crate::object::js_object_set_field_by_name( @@ -2585,11 +2586,11 @@ fn own_method_cache_accepts_appends_and_rejects_live_method_mutation() { b"method".as_ptr() as *const i8, 6, fn_ptr, - &mut cache, + &mut cache_slot, ) }; assert_eq!(replaced, 0); - assert_eq!(cache, 0); + assert_eq!(cache[0], 0); crate::object::js_object_set_field_by_name(object, method_key, closure_value); crate::object::js_object_delete_field(object, method_key); @@ -2601,11 +2602,11 @@ fn own_method_cache_accepts_appends_and_rejects_live_method_mutation() { b"method".as_ptr() as *const i8, 6, fn_ptr, - &mut cache, + &mut cache_slot, ) }; assert_eq!(deleted, 0); - assert_eq!(cache, 0); + assert_eq!(cache[0], 0); } #[test] diff --git a/crates/perry-runtime/src/value/dynamic_object.rs b/crates/perry-runtime/src/value/dynamic_object.rs index 11b214975d..e8bc1203eb 100644 --- a/crates/perry-runtime/src/value/dynamic_object.rs +++ b/crates/perry-runtime/src/value/dynamic_object.rs @@ -255,15 +255,31 @@ pub extern "C" fn js_value_length_property_f64(value: f64) -> f64 { value_length_property_with_cache(value, std::ptr::null_mut()) } +/// Words in a per-site Array-subclass `length` cache: `(identity, length +/// slot, inline bound)`. See `array_subclass_fast_length_with_ic`. +pub const LENGTH_PIC_WORDS: usize = 3; +/// A per-site `length` cache, as the emitted slot resolves it. +pub type LengthPicCache = [u64; LENGTH_PIC_WORDS]; +/// The emitted `@perry_ic_N = private global ptr null` for a `length` site: +/// null until the site's first priming read (#9708). +pub type LengthPicCacheSlot = *mut LengthPicCache; + /// Property-semantic `.length` read whose Array-subclass arm primes the /// generated scalar IC. All non-subclass cases deliberately share the exact /// implementation used by `js_value_length_property_f64`. +/// +/// `cache_slot` is the site's [`LengthPicCacheSlot`] address; the cache is +/// allocated on the first prime (#9708), so an elements-backed receiver — +/// which never publishes shape words — never allocates one. #[no_mangle] -pub extern "C" fn js_value_length_property_ic_f64(value: f64, cache: *mut u64) -> f64 { - value_length_property_with_cache(value, cache) +pub extern "C" fn js_value_length_property_ic_f64( + value: f64, + cache_slot: *mut LengthPicCacheSlot, +) -> f64 { + value_length_property_with_cache(value, cache_slot) } -fn value_length_property_with_cache(value: f64, cache: *mut u64) -> f64 { +fn value_length_property_with_cache(value: f64, cache_slot: *mut LengthPicCacheSlot) -> f64 { let jsval = JSValue::from_bits(value.to_bits()); if jsval.is_undefined() || jsval.is_null() { crate::error::js_throw_type_error_property_access( @@ -278,7 +294,7 @@ fn value_length_property_with_cache(value: f64, cache: *mut u64) -> f64 { crate::builtins::boxed_primitive_to_string_tag(value), Some("String") ) { - return value_length_property_with_cache(payload, cache); + return value_length_property_with_cache(payload, cache_slot); } } @@ -287,7 +303,7 @@ fn value_length_property_with_cache(value: f64, cache: *mut u64) -> f64 { return crate::string::js_string_length(string) as f64; } - if let Some(length) = crate::array::array_subclass_fast_length_with_ic(value, cache) { + if let Some(length) = crate::array::array_subclass_fast_length_with_ic(value, cache_slot) { return length; } @@ -854,7 +870,7 @@ mod length_handle_band_tests { let console_ptr = crate::value::js_nanbox_get_pointer(console_ctor) as usize; let length_key = crate::string::js_string_from_bytes(b"length".as_ptr(), 6); let mut cache = [0_i64; crate::object::PIC_CACHE_WORDS]; - + let mut cache_slot: crate::object::PicCacheSlot = &mut cache; assert_eq!(js_value_length_f64(console_ctor), 1.0); assert_eq!( crate::closure::closure_get_dynamic_prop(console_ptr, "length"), @@ -864,7 +880,7 @@ mod length_handle_band_tests { crate::object::js_object_get_field_ic_miss( console_ptr as *const crate::ObjectHeader, length_key, - &mut cache, + &mut cache_slot, ), 1.0 ); @@ -872,9 +888,10 @@ mod length_handle_band_tests { #[test] fn property_length_preserves_missing_and_non_numeric_values() { - let mut cache = [0_u64; 3]; + let mut cache: LengthPicCache = [0; LENGTH_PIC_WORDS]; + let mut cache_slot: LengthPicCacheSlot = &mut cache; assert_eq!( - js_value_length_property_ic_f64(42.0, cache.as_mut_ptr()).to_bits(), + js_value_length_property_ic_f64(42.0, &mut cache_slot).to_bits(), crate::value::TAG_UNDEFINED, "a number has no length property" ); @@ -888,7 +905,7 @@ mod length_handle_band_tests { let boxed_obj = crate::value::js_nanbox_pointer(obj as i64); assert_eq!( - js_value_length_property_ic_f64(boxed_obj, cache.as_mut_ptr()).to_bits(), + js_value_length_property_ic_f64(boxed_obj, &mut cache_slot).to_bits(), seven_value.to_bits(), "a source-level property read must not coerce its value" ); diff --git a/crates/perry-runtime/src/value/mod.rs b/crates/perry-runtime/src/value/mod.rs index 44525e6c59..fe050ebc5f 100644 --- a/crates/perry-runtime/src/value/mod.rs +++ b/crates/perry-runtime/src/value/mod.rs @@ -150,5 +150,5 @@ pub use dynamic_array::{ pub use dynamic_object::{ js_collection_method_dispatch, js_dynamic_object_get_property, js_dynamic_object_keys, js_get_property, js_value_length_f64, js_value_length_property_f64, - js_value_length_property_ic_f64, + js_value_length_property_ic_f64, LengthPicCache, LengthPicCacheSlot, LENGTH_PIC_WORDS, }; diff --git a/crates/perry-transform/src/generator/break_continue.rs b/crates/perry-transform/src/generator/break_continue.rs index ff3963d58d..0536466a39 100644 --- a/crates/perry-transform/src/generator/break_continue.rs +++ b/crates/perry-transform/src/generator/break_continue.rs @@ -817,3 +817,288 @@ pub fn stmts_have_continue_inside_try_finally(stmts: &[Stmt]) -> bool { _ => false, }) } + +// ── #9199: labeled break/continue that escapes a NESTED loop ────────────── +// +// `rewrite_labeled_bc_in_stmts` converts `break label` / `continue label` to +// plain completions only at the labeled loop's OWN body level, and stops at +// nested loops (a plain completion there would bind to the nested loop). The +// linearizer's single break/continue sentinel per loop then has no way to name +// an outer loop's target, so a labeled completion crossing a loop boundary +// survived verbatim into a state body and the dispatch lowering dropped it: +// `break` produced a malformed iterator result ("Cannot read properties of +// undefined (reading 'done')"), `continue` silently produced nothing at all. +// +// The fix is to stop asking the state machine to name a distant target. A +// carrier local unwinds the escape one loop at a time, so every completion the +// linearizer sees is plain and binds to the loop it sits in: +// +// __esc = 0; +// inner: while (…) { … __esc = 1; break; … } // `break label` +// if (__esc == 1) break; // in the labeled loop +// if (__esc == 2) continue; +// +// Deeper nesting reuses the same carrier and propagates with a bare +// `if (__esc != 0) break;` after each intermediate loop, so an escape from any +// depth walks out to the labeled loop without the linearizer ever seeing a +// labeled completion. + +/// Carrier value for a `break