perf(ic): allocate inline caches per used site behind an 8-byte slot (#9708)

changelog.d/9729-lazy-inline-cache-slots.md

@@ -0,0 +1,70 @@
+### Inline caches are allocated per *used* site, not per emitted site (#9708)
+
+Every inline-cache site codegen emitted — the generic property read, the
+static- and dynamic-key write ICs and their poly tail, the Symbol-keyed and
+composed `o[sym].field` reads, the Array-subclass `length` / `[i]` caches,
+the fused `if (a.f[i]) return a.f[i]` cache and the imported-object method
+guard — owned a `[12 x i64] zeroinitializer` global: 96 B of `__bss` per
+site whether or not the program ever executed it. On the Claude Code bundle
+that was 262k caches, 25 MB of zero-fill, and 18.7 MB of it **dirty resident
+memory at idle**, because a page is dirtied by the first cache touched on it
+and the few thousand hot sites are scattered across all of them.
+
+A site now owns an 8-byte pointer **slot**, `@perry_ic_N = private global
+ptr null`. The cache words live in a runtime arena
+(`perry-runtime/src/object/field_get_set/ic_slot.rs`): the miss handler
+resolves the slot with `pic_slot_resolve` the first time it actually
+*primes* the site, bump-allocates the words from a 64 KiB zeroed chunk and
+publishes them with a compare-and-swap (two `perry/thread` agents racing on
+one site agree on one cache). A miss that cannot prime — proxy, string or
+small-handle receiver, a missing key, an accessor, a frozen target — never
+touches the slot, so such a site costs its 8 bytes and nothing else; the
+write IC's poly tail is not allocated until a fifth shape arrives. Cache
+layout and every prime/evict policy are unchanged: the runtime writes the
+same words through the same `PicCache` type, and `pic_slot_resolve` sizes
+the allocation from that type, so the width pairing test keeps its meaning.
+
+**Hot path.** Each inline hit path loads the slot (a load with no dependency
+on the receiver, so it issues alongside the header loads) and folds `!= null`
+into the receiver guard it already evaluates — one fused compare, no new
+block — then reads the cache words through the loaded pointer; the runtime
+entries take the slot's address. Where a site reads word 0 inside a flat
+predicate (the dynamic-key write IC, the array-like index cache, the method
+guard) it reads through `select(present, cache, slot)`: the slot's own 8
+bytes of null are exactly the zero token an empty global used to read as, so
+the branch structure and the transition-IC reachability are untouched.
+Measured on x86-64 (`perf stat -e instructions:u`, perry-dev builds):
+
+| program | base | lazy slots | delta |
+|---|---:|---:|---:|
+| all-generic-IC microbenchmark (95M IC ops) | 13.662 G | 14.196 G | +3.9 % (3 instr per hit: slot load, `test`, never-taken `je`) |
+| `bench_object_property` | 250.9 M | 248.6 M | −0.9 % |
+| `bench_json_readonly` | 2 263.2 M | 2 258.6 M | −0.2 % |
+| `bench_dynamic_property_keys` | 1 129.1 M | 1 124.5 M | −0.4 % |
+| `07_object_create`, `09_method_calls`, `12_binary_trees`, `14_closure` | | | ±0.00 % |
+
+The typed-feedback IC counters (`PERRY_TYPED_FEEDBACK_TRACE`) are identical
+on both arms for the microbenchmark — 81 666 674 guard passes, 18 333 339
+guard failures, 18 333 339 fallback calls over 18 sites — so hit rates are
+unchanged, not merely output. On the issue's target (macOS arm64) the fused
+compare is a `ccmp`, so the hit-path cost there is the slot load plus one
+instruction.
+
+**Footprint.** A generated probe with 16 000 read sites of which 1 604 prime
+(every 10th function runs — the scattered-hot-sites shape from the issue),
+Linux x86-64, 4 KiB pages: `.bss` 2 408 752 → 997 144 B, whole-process
+anonymous `Private_Dirty` 1 660 → 632 kB. The `PERRY_GC_CENSUS` side table
+gains an `ic.lazy_caches` row (resolved sites, arena bytes) so a run can
+assert the subject was live; the issue's macOS `vmmap` numbers are the ones
+to re-measure on a bundle build.
+
+Gap coverage: `test_gap_9708_lazy_inline_cache_slots.ts` exercises every
+IC shape across the null → allocated transition — mono/poly/megamorphic
+reads, a site that can never prime, a nullish first read, inherited
+properties, static writes through the four ways and the poly tail, a frozen
+target, rotating dynamic keys, Symbol and composed Symbol-then-field reads
+with invalidation, Array-subclass `length`/index, the fused field-index
+return, and hundreds of never-executed sites — and matches node byte for
+byte. `array/subclass.rs` was at the 2 000-line cap, so
+`js_packed_arraylike_index_get` and its cache types moved to the child module
+`array/subclass_packed_index.rs`.

crates/perry-codegen/src/codegen/closure.rs

@@ -1408,11 +1408,7 @@ pub(super) fn compile_closure(
         llmod.declare_function(&name, ret, &params);
     }
     for ic_name in &ic_globals {
-        llmod.add_raw_global(format!(
-            "@{} = private global [{} x i64] zeroinitializer",
-            ic_name,
-            crate::expr::property_get::generic_dispatch::PIC_CACHE_WORDS
-        ));
+        llmod.add_raw_global(crate::expr::inline_cache_global_definition(ic_name));
     }
     for raw in &typed_parse_rodata {
         llmod.add_raw_global(raw.clone());

crates/perry-codegen/src/codegen/entry.rs

@@ -1431,11 +1431,7 @@ pub(super) fn compile_module_entry(
             llmod.declare_function(&name, ret, &params);
         }
         for ic_name in &ic_globals {
-            llmod.add_raw_global(format!(
-                "@{} = private global [{} x i64] zeroinitializer",
-                ic_name,
-                crate::expr::property_get::generic_dispatch::PIC_CACHE_WORDS
-            ));
+            llmod.add_raw_global(crate::expr::inline_cache_global_definition(ic_name));
         }
         for raw in &typed_parse_rodata {
             llmod.add_raw_global(raw.clone());
@@ -1969,11 +1965,7 @@ pub(super) fn compile_module_entry(
         // A dylib's top-level plugin exports live in its entry module, and the
         // three symbols must be defined exactly once per shared library.
         for ic_name in &ic_globals {
-            llmod.add_raw_global(format!(
-                "@{} = private global [{} x i64] zeroinitializer",
-                ic_name,
-                crate::expr::property_get::generic_dispatch::PIC_CACHE_WORDS
-            ));
+            llmod.add_raw_global(crate::expr::inline_cache_global_definition(ic_name));
         }
         for raw in &typed_parse_rodata {
             llmod.add_raw_global(raw.clone());

crates/perry-codegen/src/codegen/function.rs

@@ -1459,11 +1459,7 @@ pub(super) fn compile_function(
         llmod.declare_function(&name, ret, &params);
     }
     for ic_name in &ic_globals {
-        llmod.add_raw_global(format!(
-            "@{} = private global [{} x i64] zeroinitializer",
-            ic_name,
-            crate::expr::property_get::generic_dispatch::PIC_CACHE_WORDS
-        ));
+        llmod.add_raw_global(crate::expr::inline_cache_global_definition(ic_name));
     }
     for raw in &typed_parse_rodata {
         llmod.add_raw_global(raw.clone());

crates/perry-codegen/src/codegen/method.rs

@@ -1341,11 +1341,7 @@ pub(super) fn compile_method(
         llmod.declare_function(&name, ret, &params);
     }
     for ic_name in &ic_globals {
-        llmod.add_raw_global(format!(
-            "@{} = private global [{} x i64] zeroinitializer",
-            ic_name,
-            crate::expr::property_get::generic_dispatch::PIC_CACHE_WORDS
-        ));
+        llmod.add_raw_global(crate::expr::inline_cache_global_definition(ic_name));
     }
     for raw in &typed_parse_rodata {
         llmod.add_raw_global(raw.clone());
@@ -1880,11 +1876,7 @@ pub(super) fn compile_static_method(
         llmod.declare_function(&name, ret, &params);
     }
     for ic_name in &ic_globals {
-        llmod.add_raw_global(format!(
-            "@{} = private global [{} x i64] zeroinitializer",
-            ic_name,
-            crate::expr::property_get::generic_dispatch::PIC_CACHE_WORDS
-        ));
+        llmod.add_raw_global(crate::expr::inline_cache_global_definition(ic_name));
     }
     for raw in &typed_parse_rodata {
         llmod.add_raw_global(raw.clone());

crates/perry-codegen/src/expr/index_get.rs

@@ -71,15 +71,27 @@ pub(crate) fn lower_symbol_property_get_ic(
     ctx.ic_site_counter += 1;
     let cache_name = super::inline_cache_global_name(ctx, site_id);
     ctx.ic_globals.push(cache_name.clone());
-    let cache_ref = format!("@{cache_name}");
 
+    let probe_idx = ctx.new_block("symic.probe");
     let hit_idx = ctx.new_block("symic.hit");
     let miss_idx = ctx.new_block("symic.miss");
     let merge_idx = ctx.new_block("symic.merge");
+    let probe_label = ctx.block_label(probe_idx);
     let hit_label = ctx.block_label(hit_idx);
     let miss_label = ctx.block_label(miss_idx);
     let merge_label = ctx.block_label(merge_idx);
 
+    // #9708: the cache sits behind a pointer slot that the miss handler fills
+    // on the first prime. The probe's three loads go through the pointer, so
+    // an absent cache branches straight to the miss — the edge a fresh
+    // (all-zero) global took anyway, since a zero epoch never matches.
+    let ic_slot = super::emit_inline_cache_slot(ctx, &cache_name);
+    let cache_ref = ic_slot.cache.clone();
+    let cache_slot_ref = ic_slot.slot_ref.clone();
+    ctx.block()
+        .cond_br(&ic_slot.present, &probe_label, &miss_label);
+
+    ctx.current_block = probe_idx;
     let epoch = ctx
         .block()
         .load_atomic_acquire(I64, "@PERRY_SYMBOL_PROPERTY_IC_EPOCH", 8);
@@ -109,7 +121,7 @@ pub(crate) fn lower_symbol_property_get_ic(
     let miss_value = ctx.block().call(
         DOUBLE,
         "js_object_get_symbol_property_ic_miss",
-        &[(DOUBLE, obj_box), (DOUBLE, sym_box), (PTR, &cache_ref)],
+        &[(DOUBLE, obj_box), (DOUBLE, sym_box), (PTR, &cache_slot_ref)],
     );
     let miss_end = ctx.block().label.clone();
     ctx.block().br(&merge_label);

crates/perry-codegen/src/expr/index_get/inline_dyn_typed_array.rs

@@ -340,7 +340,17 @@ pub(super) fn lower_inline_dyn_typed_array_get(
     ctx.ic_site_counter += 1;
     let cache_name = super::super::inline_cache_global_name(ctx, site_id);
     ctx.ic_globals.push(cache_name.clone());
-    let cache_ref = format!("@{cache_name}");
+    // #9708: the cache sits behind a pointer slot the runtime fills on the
+    // first shape-carried prime. `arrlike.ic.shape` reads word 0 inside a
+    // flat predicate, so it reads through `key_cache`: the real cache when
+    // present, else the slot itself — 8 bytes of null, i.e. a zero identity,
+    // which fails `key_nonzero` exactly as the all-zero global did. Every
+    // later word is read only past that edge, through the real pointer.
+    let ic_slot = crate::expr::emit_inline_cache_slot(ctx, &cache_name);
+    let cache_ref = ic_slot.cache.clone();
+    let key_cache = ctx
+        .block()
+        .select(I1, &ic_slot.present, PTR, &cache_ref, &ic_slot.slot_ref);
 
     let object_header_idx = ctx.new_block("arrlike.ic.header");
     let object_brand_idx = ctx.new_block("arrlike.ic.brand");
@@ -613,7 +623,7 @@ pub(super) fn lower_inline_dyn_typed_array_get(
     let shape64 = ctx.block().zext(I32, &shape_id, I64);
     let class_high = ctx.block().shl(I64, &class64, "32");
     let live_key = ctx.block().or(I64, &class_high, &shape64);
-    let cached_key_ptr = ctx.block().gep(I64, &cache_ref, &[(I64, "0")]);
+    let cached_key_ptr = ctx.block().gep(I64, &key_cache, &[(I64, "0")]);
     let cached_key = ctx.block().load(I64, &cached_key_ptr);
     let key_nonzero = ctx.block().icmp_ne(I64, &cached_key, "0");
     let object_ok = ctx.block().and(I1, &is_object, &key_nonzero);
@@ -843,7 +853,7 @@ pub(super) fn lower_inline_dyn_typed_array_get(
     let slow_raw = ctx.block().call(
         DOUBLE,
         "js_packed_arraylike_index_get",
-        &[(DOUBLE, obj_box), (DOUBLE, idx_d), (PTR, &cache_ref)],
+        &[(DOUBLE, obj_box), (DOUBLE, idx_d), (PTR, &ic_slot.slot_ref)],
     );
     // In a number context, coerce the (possibly boxed) slow result here so the
     // merge phi is uniformly a Number and the arithmetic caller skips its own

crates/perry-codegen/src/expr/mod.rs

@@ -1556,8 +1556,10 @@ pub(crate) struct FnCtx<'a> {
     pub ic_site_counter: u32,
 
     /// (Issue #51) Names of IC globals created during lowering. After
-    /// the function is emitted, the caller emits `@<name> = private
-    /// global [2 x i64] zeroinitializer` for each entry.
+    /// the function is emitted, the caller emits one
+    /// [`inline_cache_global_definition`] — `@<name> = private global ptr
+    /// null`, an 8-byte slot the runtime fills with an arena cache on the
+    /// site's first priming miss (#9708) — for each entry.
     pub ic_globals: Vec<String>,
 
     /// Region-scoped cache selected by a guarded statement fusion. Generic
@@ -2286,6 +2288,50 @@ pub(crate) fn inline_cache_global_name(ctx: &FnCtx<'_>, site_id: u32) -> String
     inline_cache_global_name_for_prefix(ctx.strings.module_prefix(), site_id)
 }
 
+/// The definition emitted for every name in `FnCtx::ic_globals`.
+///
+/// #9708: an inline-cache site owns an 8-byte pointer **slot**, not its cache
+/// words. The slot is zero-initialised (so it lands in `__bss` and costs
+/// nothing until touched) and stays null until the runtime's miss handler
+/// primes the site, at which point it publishes a cache allocated from the
+/// runtime's IC arena (`perry_runtime::object::pic_slot_resolve`). A site
+/// the program never executes therefore costs 8 bytes of zero-fill instead
+/// of a 96-byte cache that dirtied a resident page on first touch. Every
+/// inline hit path loads the slot through [`emit_inline_cache_slot`] and
+/// proves it non-null before reading a cache word; every runtime miss entry
+/// takes the slot's address.
+pub(crate) fn inline_cache_global_definition(name: &str) -> String {
+    format!("@{name} = private global ptr null")
+}
+
+/// A site's inline-cache slot, loaded in the current block (#9708).
+///
+/// `slot_ref` is the `@perry_ic_N` global — what the runtime miss entries
+/// take. `cache` is the `ptr` loaded from it and `present` the `i1` proving
+/// it non-null: a site must branch (or fold `present` into a guard that
+/// dominates) before it emits any load through `cache`, because the slot is
+/// null until the site's first priming miss.
+pub(crate) struct InlineCacheSlot {
+    pub slot_ref: String,
+    pub cache: String,
+    pub present: String,
+}
+
+/// Load `@<cache_name>`'s cache pointer in the current block and test it.
+/// The load has no dependency on the receiver, so it is free to issue early;
+/// folding `present` into the receiver guard the site already evaluates costs
+/// a single fused compare on the hit path.
+pub(crate) fn emit_inline_cache_slot(ctx: &mut FnCtx<'_>, cache_name: &str) -> InlineCacheSlot {
+    let slot_ref = format!("@{cache_name}");
+    let cache = ctx.block().load(PTR, &slot_ref);
+    let present = ctx.block().icmp_ne(PTR, &cache, "null");
+    InlineCacheSlot {
+        slot_ref,
+        cache,
+        present,
+    }
+}
+
 /// Record a cold-arm bailout for a compiler-private versioned-loop callback.
 /// The stack context is `[counter_slot_ptr, original_bound, resume_index]` as
 /// three i64 words. The first cold arm stores `counter + 1` and poisons the