From bae92b170df2a52fc1d128cfa8e94ce28e46efe1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ralph=20K=C3=BCpper?= Date: Fri, 4 Sep 2026 14:34:34 +0200 Subject: [PATCH 1/3] perf(runtime): store shape descriptors in an id-indexed slab and retire owned growth history MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes #9706. The agent-local shape table kept a PtrHashMap> beside two Vec-valued reverse maps (exact facts, keys address). On the compiled claude-code TUI at idle that was ~330 bytes per live descriptor: a 56-byte record in a 64-byte bin, a map entry at 25% load, a 57-byte facts bucket plus a Vec buffer, a keys bucket, and a per-scan probe memo. * ShapeSlab (object/shapes_store.rs): a ShapeId indexes a chunked, paged slab directly — packed 32-byte #[repr(C)] records (keys first, so the record address is the collector's rewritable keys slot, #8112), stable addresses, 32-record chunks under a two-level page directory, all-dead chunks and pages released at major GC. The lookup-way cache and its epoch are gone. * IdList: a 16-byte id list that is the value of both remaining reverse indices — by_facts (64-bit fold of the six facts, every hit re-validates the record) and families (keys address). ShapeFacts, ids_by_facts, ids_by_keys, indexed_keys, sync_descriptor_reverse_indices, PROBE_MEMO and shapes_reverse_indices.rs are deleted; the metadata scan probes each keys address once per family. * publish_object_shape_from retires an OWNED keys array's same-address growth history behind the version its single owner carries, after the successor is stamped and armed (#9200's order), keeping cache-carried versions. Array-subclass receivers keep the old behaviour: their tail-transition cache learns the predecessor after the publish and reinstalls it on pop. * PERRY_GC_CENSUS: shapes.by_facts / shapes.families replace the old rows; new shapes.ids_minted, shapes.descriptors.carried/.uncarried and shapes.families.multi/.largest rows. * scripts/shape_descriptor_census.py pins the slab and the retirement contract, with sabotage self-tests. Measured on the claude-code TUI (same objects relinked against both runtimes, third census after shrink): descriptors 68,661 -> 43,724, shape tables 22.22 MB -> 8.11 MB, RSS at census 441 -> 419 MB. A 150,000-key dictionary built by appends 11.7 s -> 0.23 s (retain_key_count_versions was O(N) per append); the three existing shape benchmarks are flat to slightly faster. Claude-Session: https://claude.ai/code/session_016TiA2Y98uX79JSsY3eV1DS --- changelog.d/9724-shape-descriptor-slab.md | 102 ++ crates/perry-runtime/src/fast_hash.rs | 14 +- crates/perry-runtime/src/gc/census.rs | 18 +- .../gc/tests/shape_keys_descriptor_edge.rs | 50 + crates/perry-runtime/src/object/shapes.rs | 952 ++++++++++-------- .../src/object/shapes_reverse_indices.rs | 119 --- .../src/object/shapes_slot_list.rs | 261 +++-- .../perry-runtime/src/object/shapes_store.rs | 776 ++++++++++++++ .../src/object/shapes_test_support.rs | 73 +- .../perry-runtime/src/object/shapes_tests.rs | 207 ++-- scripts/shape_descriptor_census.py | 115 ++- 11 files changed, 1854 insertions(+), 833 deletions(-) create mode 100644 changelog.d/9724-shape-descriptor-slab.md delete mode 100644 crates/perry-runtime/src/object/shapes_reverse_indices.rs create mode 100644 crates/perry-runtime/src/object/shapes_store.rs diff --git a/changelog.d/9724-shape-descriptor-slab.md b/changelog.d/9724-shape-descriptor-slab.md new file mode 100644 index 0000000000..84271dd2d2 --- /dev/null +++ b/changelog.d/9724-shape-descriptor-slab.md @@ -0,0 +1,102 @@ +### Performance + +- **Shape descriptors: an id-indexed slab and two compact reverse indices + instead of a boxed hash map and two `Vec`-valued ones; owned growth history + is retired** (#9706). `PERRY_GC_CENSUS` on the compiled + claude-code TUI at idle put the agent-local shape table at ~330 bytes per + live descriptor. The bytes were the STORAGE, not the facts: a 56-byte + `Box` in a 64-byte allocator bin, a 16-byte + `PtrHashMap>` entry sitting at ~25% load after + `shrink_to(2 * len)`, a 57-byte bucket in the exact-facts reverse map plus a + 16-byte `Vec` buffer per entry, and a 33-byte bucket in the keys-address + reverse map — four allocations and three hash tables saying the same thing. + + `crates/perry-runtime/src/object/shapes_store.rs` (new): + + - **`ShapeSlab`** — the by-id store. A ShapeId is `SHAPE_ID_BASE + n` from a + process-global monotonic counter, so `n` indexes a chunked slab directly: + no hash, no per-record allocation, and a record address that never moves + for the record's lifetime, which is the property the collector relies on + when it enumerates the record's `keys` word as a rewritable slot (#8112) + and retains that address across budgeted resumptions. Chunks (32 + records, behind a two-level page directory) are allocated lazily — a + worker's ids interleave with the main thread's, and the claude-code TUI + mints a million ids at startup for 44 k survivors — and an all-dead chunk + or page is released at the same cadence as the reverse-index shrink + (`shrink_shape_tables`, once per major collection). + The direct-mapped lookup-way cache that fronted the hash map is gone: a + slab probe IS "shift, index, deref", and it needs no invalidation epoch. + - **`ShapeRecord`** — the packed 32-byte `#[repr(C)]` table record (`keys` + first, so the record address is the keys slot). `ShapeDescriptor` stays + the by-value copy the rest of the runtime consumes, lifted from the + record; the copy's `record` field is the slab address. + - **`IdList`** — a 16-byte id list (three inline, a spilled `Vec` beyond) + that is the value of both remaining reverse indices: `by_facts`, keyed by + a 64-bit FNV fold of the six identity facts (every hit re-validates the + record, so a collision costs a second record read, never a wrong answer), + and `families`, keyed by keys address. The `ShapeFacts`-keyed map, + `indexed_keys` (the address a record was last indexed under) and the + per-scan `PROBE_MEMO` map are gone: the family index is the memo, and + `scan_shape_table_rekey_mut` now probes each keys address ONCE per + family — with the marking visit when any member is an old/cache + carrier, which is exactly the #8112 rooting duty. + + A family is small by construction. A SHARED keys array is immutable + (mutation forks a private clone), so its descriptors differ only in the + birth bound, a semantic generation, the class kind, or a tombstone count. + An OWNED array grows in place, and until now every same-address append left + the predecessor descriptor alive until the array itself died + (`retain_key_count_versions`): a dictionary built by ten thousand appends + kept ten thousand prefix descriptors. **`publish_object_shape_from` now + retires that history behind the version its single owner carries** + (`retire_owned_shape_siblings`), after the successor is stamped and armed + (#9200's order), keeping any version an optimization cache permanently owns + (`cache_carrier`). Sound because `GC_FLAG_SHAPE_SHARED` is sticky: an + unflagged array has had exactly one owner for its whole life, a stale IC + token already misses on the stamp compare, and `shape_descriptor_by_id` of + a retired id is `None`. The one owner whose history IS reinstalled — an + Array-subclass receiver, whose tail-transition cache learns the + (predecessor, successor) pair right after the publish and stamps the + predecessor back on `pop` — keeps the old behaviour, gated on the receiver + class the learner itself is scoped to. + + `PERRY_GC_CENSUS` rows: `shapes.ids_by_facts` / `shapes.ids_by_keys` are + replaced by `shapes.by_facts` / `shapes.families`; `shapes.descriptors` now reports the slab's + real bytes; new `shapes.ids_minted(process)`, + `shapes.descriptors.carried(live objects)` / `.uncarried` (with the + `cache_carrier` / `old_carrier` split) and `shapes.families.multi` / + `.largest` rows say how the population relates to the live heap. + + `scripts/shape_descriptor_census.py` pins the new invariants (slab chunks + individually boxed and never reallocated; `keys` first in the record; + owned-history retirement scoped to the family, keeping cache carriers, and + ordered after the stamp) with sabotage self-tests for each. + +Measured on the compiled claude-code TUI (`cli_2.1.112.js`, + `PERRY_GC_CENSUS` via `SIGUSR2` at idle, the same compiled objects relinked + against the two runtimes, third census = after `shrink_shape_tables`): + + | | before | after | + |---|---|---| + | live descriptors | 68,661 | 43,724 | + | `shapes.descriptors` | 9.40 MB | 3.82 MB | + | facts reverse index | 8.39 MB | 1.64 MB | + | keys-address reverse index | 2.67 MB | 0.89 MB | + | shape tables total | 22.22 MB (29.86 at the first census) | 8.11 MB (8.45) | + | RSS at census | 441 MB | 419 MB | + + Of the remaining 43.7 k descriptors 8,664 are carried by a live object; + the rest are per-object semantic generations and shared-array prefix + versions whose keys array is still alive, which nothing prunes yet. + + Benchmarks (min of 5): `bench_dynamic_property_keys` 38/12 → 37/12 ms, + `bench_populated_delete` 62 → 58 ms, `bench_shared_shape_delete` 45 → + 40 ms; a 150,000-key dictionary built by appends 11.7 s → 0.23 s (on + `main`, `retain_key_count_versions` rebuilt the whole same-address id + list on every append), 2,000 × 200-key dictionaries 491 → 246 ms, and + 200 k `{a,b,c}` literals with 20 hot read passes 103 → 47 ms. + + Validation: `cargo test -p perry-runtime` 3113 passed (dev and release, + single-threaded); `scripts/run_lint_gates.sh` 64/64; a 233-test + shape/object/class/GC/JSON gap subset gives identical verdicts on both + arms (224 PASS, 2 pre-existing PARITY_FAIL). diff --git a/crates/perry-runtime/src/fast_hash.rs b/crates/perry-runtime/src/fast_hash.rs index f020387b82..3d021d1f17 100644 --- a/crates/perry-runtime/src/fast_hash.rs +++ b/crates/perry-runtime/src/fast_hash.rs @@ -197,13 +197,13 @@ impl Hasher for FastKeyHasherImpl { // Integer writes fold one word per call instead of falling into `Hasher`'s // default `write_uN` -> `write(&n.to_ne_bytes())` byte loop. // - // This is what the shape table's `ids_by_facts` key pays for: `ShapeFacts` - // is six integer fields (two `u64`, three `u32`, one enum discriminant, an - // `isize`), so the derived `Hash` fed ~36 bytes -- ~36 serial multiplies -- - // through the byte loop for a key that six folds mix just as well. That - // lookup runs on every shape publish (`shape_descriptor_ensure_with_holes` - // probes `ids_by_facts` before minting an id), i.e. on every object - // property add/delete that transitions a shape. + // This was written for the shape table's `ShapeFacts` key (six integer + // fields: two `u64`, three `u32`, one enum discriminant, an `isize`), whose + // derived `Hash` fed ~36 bytes -- ~36 serial multiplies -- through the byte + // loop for a key that six folds mix just as well. #9706 replaced that map + // with a pre-folded `u64` key (`object/shapes_store.rs::facts_key`), but + // the same shape of key remains on this hasher: `RegisteredTypedShapeKey` + // (`gc/layout/typed_shape.rs`) and the `(usize, String)` descriptor keys. // // `write_u8` is deliberately included even though it is exactly equivalent // to the byte path for a single byte: routing it here keeps every integer diff --git a/crates/perry-runtime/src/gc/census.rs b/crates/perry-runtime/src/gc/census.rs index 63b8b5c8ac..5aeaad82a0 100644 --- a/crates/perry-runtime/src/gc/census.rs +++ b/crates/perry-runtime/src/gc/census.rs @@ -346,6 +346,10 @@ struct Census { clo_captures: u64, // objects obj_meta: u64, + /// ShapeId stamp of every live shaped object (duplicates included; sorted + /// and deduplicated once the walk is over). Feeds the shape table's + /// "carried by a live object" rows (#9706). + live_shape_ids: Vec, } const SPACE_NAMES: [&str; 6] = [ @@ -432,7 +436,11 @@ impl Census { self.obj_meta += 1; } let live = match crate::object::shapes::object_shape_descriptor(obj) { - Some(d) => (d.live_inline_slot_count as usize).min(slot_capacity), + Some(d) => { + self.live_shape_ids + .push(crate::object::shapes::object_shape_stamp(obj)); + (d.live_inline_slot_count as usize).min(slot_capacity) + } None => { entry.unshaped += 1; 0 @@ -764,7 +772,13 @@ fn take_census(label: &str, pass1: Option>) { }) .collect(); - let side: Vec = side_tables() + c.live_shape_ids.sort_unstable(); + c.live_shape_ids.dedup(); + let mut side_rows = side_tables(); + side_rows.extend(crate::object::shapes::shape_table_liveness_census( + &c.live_shape_ids, + )); + let side: Vec = side_rows .into_iter() .map(|(n, e, b)| serde_json::json!({"table": n, "entries": e, "bytes": b})) .collect(); diff --git a/crates/perry-runtime/src/gc/tests/shape_keys_descriptor_edge.rs b/crates/perry-runtime/src/gc/tests/shape_keys_descriptor_edge.rs index ac7adaf2c1..856a21dc62 100644 --- a/crates/perry-runtime/src/gc/tests/shape_keys_descriptor_edge.rs +++ b/crates/perry-runtime/src/gc/tests/shape_keys_descriptor_edge.rs @@ -212,6 +212,56 @@ fn a_keys_array_reachable_only_through_the_descriptor_survives_and_is_rewritten( ); } +/// #9706: the reverse indices are keyed by the keys ADDRESS. After a copying +/// minor moves the keys array, the metadata scan must re-key the family and +/// the exact-facts accelerator, so that interning the moved facts answers the +/// SAME id (a fresh id would be a duplicate descriptor per collection) and +/// the stale address answers nothing. +#[test] +fn the_reverse_indices_follow_a_moved_keys_array() { + let _guard = CopyingNurseryTestGuard::new(2); + // The record rewrite comes from the receiver's own edge; the re-keying + // is the metadata scanner's job, which production registers at gc init + // and a unit test must register itself (as the recycled-keys fixtures do). + gc_register_mutable_root_scanner(shapes::scan_shape_table_rekey_mut); + let (before, after) = collect_and_report(false) + .expect("#9706: the receiver must move for this cycle to be discriminating"); + assert_ne!( + after.keys, before.keys, + "test premise: the keys array moved" + ); + let obj = (js_shadow_slot_get(0) & POINTER_MASK) as *mut crate::ObjectHeader; + let id = unsafe { shapes::object_shape_stamp(obj) }; + assert!(shapes::is_shape_id(id), "the receiver stays stamped"); + assert_eq!( + shapes::test_shape_ids_for_keys(after.keys as usize), + vec![id], + "#9706: the family index must be re-keyed to the forwarded address \ + (before={:#x} after={:#x} under-before={:?} record-keys={:#x})", + before.keys, + after.keys, + shapes::test_shape_ids_for_keys(before.keys as usize), + unsafe { shapes::object_shape_descriptor(obj) } + .map(|d| d.keys) + .unwrap_or(0) + ); + assert!( + shapes::test_shape_ids_for_keys(before.keys as usize).is_empty(), + "#9706: nothing may stay indexed under the from-space address" + ); + let descriptor = unsafe { shapes::object_shape_descriptor(obj) }.expect("published"); + assert_eq!(descriptor.keys, after.keys); + assert_eq!( + shapes::shape_descriptor_ensure( + after.keys as usize as *const crate::ArrayHeader, + descriptor.logical_key_count, + descriptor.live_inline_slot_count, + ), + Ok(id), + "#9706: interning the moved facts must answer the existing id, not mint a duplicate" + ); +} + #[test] fn keys_edge_sabotage_is_detected() { let _guard = CopyingNurseryTestGuard::new(2); diff --git a/crates/perry-runtime/src/object/shapes.rs b/crates/perry-runtime/src/object/shapes.rs index 95b6754bd5..5ec883968c 100644 --- a/crates/perry-runtime/src/object/shapes.rs +++ b/crates/perry-runtime/src/object/shapes.rs @@ -34,23 +34,23 @@ use crate::array::ArrayHeader; use std::cell::RefCell; -#[path = "shapes_reverse_indices.rs"] -mod shapes_reverse_indices; #[path = "shapes_slot_list.rs"] mod shapes_slot_list; -use shapes_reverse_indices::{ - descriptor_facts, insert_descriptor_id_sorted, remove_descriptor_and_reverse_indices, - remove_descriptor_id_from_facts_index, sync_descriptor_reverse_indices, -}; +#[path = "shapes_store.rs"] +mod shapes_store; #[cfg(test)] pub(crate) use shapes_slot_list::shape_descriptor_keys_slot; pub(crate) use shapes_slot_list::shape_id_owns_keys_slot; pub(crate) use shapes_slot_list::{ - object_shape_hole_count, publish_object_shape_holes, record_shape_scan_outcome, + object_shape_hole_count, publish_object_shape_holes, rekey_stable_tombstone_shape_after_squeeze, retire_owned_shape_history, shape_index_migrate_after_delete, shape_index_shift_in_place, try_update_stable_tombstone_shape, try_update_stable_tombstone_shape_cached, SlotList, }; +use shapes_store::{ + IdList, ShapeRecord, ShapeSlab, RECORD_FLAG_CACHE_CARRIER, RECORD_FLAG_FACTS_INDEXED, + RECORD_FLAG_OLD_CARRIER, RECORD_FLAG_OLD_CARRIER_SEEN, +}; #[derive(Clone)] pub(crate) struct ShapeIndex { @@ -70,39 +70,27 @@ pub(crate) struct ShapeIndex { slots: crate::fast_hash::PtrHashMap, } -/// Immutable facts named by one ShapeId. +/// Immutable facts named by one ShapeId, copied out of the table. /// -/// #8112: `keys` is the AUTHORITATIVE ordered-keys edge — the collector marks -/// it and rewrites it in place. Before #8112 the header word was the sole -/// strong edge and this field a weak copy that a post-visit callback repaired. -/// The inversion is what #8047 needs, because deleting the header word must -/// not unroot anything. +/// #8112: the table record's `keys` is the AUTHORITATIVE ordered-keys edge — +/// the collector marks it and rewrites it in place. Before #8112 the header +/// word was the sole strong edge and this field a weak copy that a post-visit +/// callback repaired. The inversion is what #8047 needs, because deleting the +/// header word must not unroot anything. /// -/// The table is a rehashing `PtrHashMap`, so the bucket address is NOT stable -/// across descriptor insertion — and the incremental collector retains -/// enumerated slot addresses across budgeted resumptions. Descriptors are -/// therefore BOXED (`ShapeTableInner::descriptors`), which makes each record's -/// address fixed for its lifetime, and `record` carries the address of THIS -/// boxed descriptor so a traced receiver can hand the collector a rewritable +/// The table stores a packed [`ShapeRecord`] in a chunked slab whose record +/// addresses never move (#9706, `shapes_store.rs`); the incremental collector +/// retains enumerated slot addresses across budgeted resumptions, so that +/// stability is load-bearing. This value is the UNPACKED copy the rest of the +/// runtime consumes, and `record` carries the address of the slab record it +/// was lifted from so a traced receiver can hand the collector a rewritable /// `keys` location without a second table probe (#8122's one-probe rule). #[derive(Clone, Copy, Debug)] pub(crate) struct ShapeDescriptor { /// Raw ArrayHeader address in Perry's fixed-width heap-word ABI. Keeping /// this u64 preserves identical representation on ILP32/LP64. pub(crate) keys: u64, - /// The keys address currently represented in the reverse indices. The - /// collector may rewrite `keys` directly through a raw - /// slot before the metadata scanner runs; retaining the indexed address - /// lets that scanner repair exactly this descriptor instead of rebuilding - /// and sorting both reverse maps for every shape in the agent. - /// Never part of shape identity. - indexed_keys: u64, - /// Whether this descriptor participates in exact-facts interning. A - /// private stable-tombstone epoch mutates its counts in place and detaches - /// from `ids_by_facts`; it remains in `ids_by_keys` for GC relocation and - /// deterministic retirement at squeeze. - facts_indexed: bool, - /// Address of the BOXED record this value was lifted from, or 0 for a + /// Address of the slab record this value was lifted from, or 0 for a /// descriptor built outside the table (equality comparisons, tests). /// Never part of shape IDENTITY — see the hand-written `PartialEq` below. pub(crate) record: usize, @@ -116,10 +104,12 @@ pub(crate) struct ShapeDescriptor { /// on. It is sticky within an epoch and recomputed by every full trace, so /// it over-approximates by at most one full collection: exactly the /// generational contract, and never unconditional rooting. + /// + /// The record also keeps the notes accumulated since the last full trace + /// (`RECORD_FLAG_OLD_CARRIER_SEEN`), adopted into this bit by + /// [`rotate_old_carrier_epoch_after_full_trace`]; the copy carries only + /// the adopted gate. pub(crate) old_carrier: bool, - /// Notes accumulated since the last full trace; adopted into `old_carrier` - /// by [`rotate_old_carrier_epoch_after_full_trace`]. - pub(crate) old_carrier_seen: bool, /// A runtime optimization cache can reinstall this historical shape even /// while no live object currently carries it. Such a cache is an explicit /// strong metadata owner, so collection must root and rewrite `keys` before @@ -144,22 +134,30 @@ pub(crate) struct ShapeDescriptor { } /// Shape identity is the FACTS, never the storage address. A descriptor value -/// lifted out of the table compares equal to the boxed record it came from. +/// lifted out of the table compares equal to the record it came from. impl ShapeDescriptor { /// The one `keys` word the collector rewrites for this shape, or `None` /// for a descriptor value that was never lifted out of the table. + /// + /// `keys` is the first field of the `#[repr(C)]` slab record, so the + /// record address IS the slot address. #[inline] pub(crate) fn keys_slot(&self) -> Option<*mut u64> { if self.record == 0 { return None; } - Some(unsafe { std::ptr::addr_of_mut!((*(self.record as *mut ShapeDescriptor)).keys) }) + Some(self.record as *mut u64) } } impl PartialEq for ShapeDescriptor { fn eq(&self, other: &Self) -> bool { - descriptor_facts(*self) == descriptor_facts(*other) + self.keys == other.keys + && self.logical_key_count == other.logical_key_count + && self.live_inline_slot_count == other.live_inline_slot_count + && self.semantic_generation == other.semantic_generation + && self.object_kind == other.object_kind + && self.hole_count == other.hole_count } } @@ -226,121 +224,128 @@ fn clear_shape_object_kind_cache() { cache.fill(0); } -#[derive(Clone, Copy, Debug, Eq, Hash, PartialEq)] -struct ShapeFacts { - keys: u64, - logical_key_count: u32, - live_inline_slot_count: u32, - semantic_generation: u64, - object_kind: ShapeObjectKind, - /// Tombstoned key slots in the keys array (`TAG_HOLE` markers left by - /// O(1) deletes). Ordinarily part of identity. A private ordinary receiver - /// in the stable-tombstone epoch updates this fact in place; its ICs - /// validate the cached value slot against `TAG_HOLE` instead of relying on - /// token churn. - hole_count: u32, -} - struct ShapeTableInner { indices: crate::fast_hash::PtrHashMap, - /// #8125: `PtrHashMap`, not the SipHash default. - /// - /// This is the map `shape_descriptor_by_id` probes, and that probe is the - /// single hottest runtime lookup_ways in the object model: `object_is_regular` - /// runs it once per array element-shape test (3 M times on the `retain` - /// bench, 20 M on `churn`) and, since #8113 deleted - /// `ObjectHeader::field_count`, `object_live_slot_count` runs it on every - /// indexed field get/set. A symbol profile of the `shapes` bench - /// (`PERRY_DEBUG_SYMBOLS=1` + `sample`) put `RandomState::hash_one` at the - /// TOP of self time with `shape_descriptor_by_id` fourth — together ~22% of - /// the program, nearly all of it SipHash on a bare `u32`. - /// - /// The key is a ShapeId minted by this process from a monotonic counter. - /// No external input reaches it, so hash-flooding resistance buys nothing - /// here for the same reason it buys nothing on the pointer-keyed - /// registries `fast_hash` already serves. - /// BOXED (#8112): the collector enumerates `&mut record.keys` as an - /// ordinary GC slot, so the record's address must survive every descriptor - /// insertion that can happen while a budgeted scan holds it. A `Box` keeps - /// the payload put when the map rehashes; the map only ever moves the - /// eight-byte owning pointer. - descriptors: crate::fast_hash::PtrHashMap>, - /// Exact-facts reverse index. More than one id is legal when a worker - /// minted a local descriptor before a process-global module id arrived. + /// Exact-facts accelerator (#9706): the 64-bit fold of a descriptor's six + /// identity facts (`shapes_store::facts_key`) -> the ids carrying those + /// facts. Almost always one id; more than one is legal when a worker + /// minted a local descriptor before a process-global module id arrived, + /// or on a 64-bit collision — every hit re-validates the slab record, so + /// a collision costs a second record read, never a wrong answer. This + /// replaces the `ShapeFacts`-keyed map whose 32-byte key and 24-byte + /// `Vec` value made it the largest of the old reverse indices. /// - /// Deliberately NOT a `PtrHashMap`: `PtrHasher`'s `write_*` methods - /// OVERWRITE the accumulator instead of folding it, which is exactly right - /// for a single-word key and wrong for this five-field one — every - /// `ShapeFacts` would hash to its last field alone. + /// The key is a fold of internal shape state (never program input), so + /// `PtrHasher` (#8125) is the right hasher: the word is already mixed. + by_facts: crate::fast_hash::PtrHashMap, + /// Keys-array address -> every descriptor id currently indexed under it. + /// Same-address retirement, squeeze rekeying and GC relocation all work + /// per family instead of per descriptor, and the family is what the + /// metadata scan probes ONCE per keys array. /// - /// It is a `FastKeyHashMap` rather than the SipHash default, though: that - /// objection is to `PtrHasher` specifically, and leaving std's - /// `RandomState` here made this the only SipHash map left on the shape - /// path. Profiling `claude -p` showed `RandomState::hash_one` at 17 - /// self-samples inside `shapes::` alone (57 across the process) — pure - /// hashing overhead on a lookup_ways that runs on every descriptor - /// install/retire. + /// A family is small by construction for a SHARED keys array, which is + /// immutable (mutation forks a private clone): its descriptors differ only + /// in the birth bound, a semantic generation, the class kind, or a + /// tombstone count. An OWNED array grows in place, and every same-address + /// publish retires the predecessor it just superseded + /// (`retire_owned_shape_siblings`), so its family holds the current + /// version plus at most the cache-carried ones. Without that retirement a + /// dictionary built by ten thousand appends kept ten thousand prefix + /// descriptors alive until the array died. /// - /// `FastKeyHasher` is the right third option: it implements only `write`, - /// so every `write_u32` / `write_u64` from the derived `Hash` forwards - /// there and FOLDS with FNV-1a. All five fields reach the accumulator, - /// which is exactly the property `PtrHasher` lacks. The key is built from - /// internal shape state (never program input), so DoS-resistant hashing - /// buys nothing here — the same rationale already applied to the - /// descriptor side tables and to `indices` (#8125). - ids_by_facts: crate::fast_hash::FastKeyHashMap>, - /// Keys-array address -> every descriptor id that currently names it. - /// Same-address key-count retirement uses this index instead of scanning - /// every shape ever observed by the agent. Single-word key, so `PtrHasher` - /// (#8125). - ids_by_keys: crate::fast_hash::PtrHashMap>, -} - -/// Ways in the direct-mapped shape-descriptor lookup_ways cache. Power of two so -/// the index is a mask. 256 x 16 bytes = 4 KiB per thread. -const SHAPE_LOOKUP_WAYS: usize = 256; - -/// One way: `(shape_id, boxed record address, epoch)`. `shape_id == 0` is the -/// empty sentinel — a real id is always >= `SHAPE_ID_BASE`. -type ShapeLookupWay = std::cell::Cell<(u32, usize, u32)>; + /// Single-word key, so `PtrHasher` (#8125). + families: crate::fast_hash::PtrHashMap, +} + +impl ShapeTableInner { + #[inline] + fn family_push_back(&mut self, keys: u64, id: u32) { + self.families.entry(keys).or_default().push_back(id); + } + + #[inline] + fn family_push_front(&mut self, keys: u64, id: u32) { + self.families.entry(keys).or_default().push_front(id); + } + + /// Drop `id` from the family under `keys`, removing an emptied family. + #[inline] + fn family_remove(&mut self, keys: u64, id: u32) -> bool { + let Some(ids) = self.families.get_mut(&keys) else { + return false; + }; + let removed = ids.remove(id); + if ids.is_empty() { + self.families.remove(&keys); + } + removed + } + + #[inline] + fn facts_push_back(&mut self, facts: u64, id: u32) { + self.by_facts.entry(facts).or_default().push_back(id); + } + + #[inline] + fn facts_push_front(&mut self, facts: u64, id: u32) { + self.by_facts.entry(facts).or_default().push_front(id); + } + + /// Drop `id` from the accelerator bucket `facts`, removing it if emptied. + #[inline] + fn facts_remove(&mut self, facts: u64, id: u32) -> bool { + let Some(ids) = self.by_facts.get_mut(&facts) else { + return false; + }; + let removed = ids.remove(id); + if ids.is_empty() { + self.by_facts.remove(&facts); + } + removed + } +} pub(crate) struct ShapeTable { + /// The by-id store, outside the `RefCell` on purpose: `shape_descriptor_by_id` + /// is on the hot property path (profiling a dynamic-property loop put it + /// and `shape_descriptor_ensure_with_generation` at ~13% of main-thread + /// samples between them), and the collector reads and writes records + /// through raw pointers from inside walks that hold `inner` borrowed. + /// Records are cells; every access goes through a short-lived pointer. + slab: std::cell::UnsafeCell, inner: RefCell, - /// Direct-mapped cache in front of `inner.descriptors`. - /// - /// `shape_descriptor_by_id` is on the hot property path — profiling a - /// dynamic-property loop put it and `shape_descriptor_ensure_with_generation` - /// at ~13% of main-thread samples between them — and each call paid a - /// `RefCell` borrow plus a hash probe to reach a record whose address never - /// moves. `Box` is stable across rehash, so a way can hold - /// the record's address directly and a hit is: mask, compare, deref. - /// - /// Deliberately NOT holding a copy of the descriptor. The record is mutated - /// in place (`old_carrier`, `cache_carrier`, `keys` after evacuation), and a - /// cached copy would go quietly stale. Holding the address means a hit - /// always reads current data. - lookup_ways: [ShapeLookupWay; SHAPE_LOOKUP_WAYS], - /// Bumped whenever a record's ADDRESS can change under an id that is still - /// in use: removal, and the one insert path that can replace an existing id - /// with a fresh `Box`. A fresh-id insert cannot invalidate an existing way, - /// so it deliberately does not bump — otherwise ordinary shape creation - /// would flush the cache continuously. - lookup_epoch: std::cell::Cell, } impl ShapeTable { pub(crate) fn new() -> Self { ShapeTable { - lookup_ways: std::array::from_fn(|_| std::cell::Cell::new((0, 0, 0))), - lookup_epoch: std::cell::Cell::new(1), + slab: std::cell::UnsafeCell::new(ShapeSlab::new()), inner: RefCell::new(ShapeTableInner { indices: crate::fast_hash::new_ptr_hash_map(), - descriptors: crate::fast_hash::new_ptr_hash_map(), - ids_by_facts: crate::fast_hash::new_fast_key_hash_map(), - ids_by_keys: crate::fast_hash::new_ptr_hash_map(), + by_facts: crate::fast_hash::new_ptr_hash_map(), + families: crate::fast_hash::new_ptr_hash_map(), }), } } + + /// Shared view of the slab. Sound under the single-threaded agent + /// discipline the whole table relies on; mutation happens only through + /// [`Self::slab_mut`] in code that holds no other slab reference. + #[inline] + fn slab(&self) -> &ShapeSlab { + // SAFETY: see the field docs — one agent, one thread, no reference + // held across a call that can insert or remove. + unsafe { &*self.slab.get() } + } + + /// # Safety + /// + /// The caller holds no other reference into the slab for the duration. + #[inline] + #[allow(clippy::mut_from_ref)] + unsafe fn slab_mut(&self) -> &mut ShapeSlab { + &mut *self.slab.get() + } } /// #6759 C3c: ShapeIds live in their own u32 range, disjoint from every @@ -450,47 +455,59 @@ pub(crate) fn shape_descriptor_ensure_with_holes( object_kind: ShapeObjectKind, hole_count: u32, ) -> Result { - let keys_id = keys as usize; + let keys_id = keys as usize as u64; if keys_id == 0 && logical_key_count != 0 { return Err(ShapeDescriptorError::InvalidFacts); } - let facts = ShapeFacts { - keys: keys_id as u64, + let facts = shapes_store::facts_key( + keys_id, logical_key_count, live_inline_slot_count, semantic_generation, object_kind, hole_count, - }; - let mut inner = crate::state::state().shapes.inner.borrow_mut(); - if let Some(id) = inner - .ids_by_facts - .get(&facts) - .and_then(|ids| ids.first().copied()) - { - return Ok(id); + ); + let table = &crate::state::state().shapes; + let mut inner = table.inner.borrow_mut(); + if let Some(ids) = inner.by_facts.get(&facts) { + let slab = table.slab(); + for &id in ids.as_slice() { + let Some(record) = slab.record_ptr(id) else { + continue; + }; + // SAFETY: live slab record, read immediately. + let record = unsafe { *record }; + // The bucket is a 64-bit fold: validate the facts on every hit. + if record.has(RECORD_FLAG_FACTS_INDEXED) + && record.facts_match( + keys_id, + logical_key_count, + live_inline_slot_count, + semantic_generation, + object_kind, + hole_count, + ) + { + return Ok(id); + } + } } let id = alloc_shape_id().map_err(|_| ShapeDescriptorError::IdExhausted)?; - let descriptor = ShapeDescriptor { - keys: keys_id as u64, - indexed_keys: keys_id as u64, - facts_indexed: true, - record: 0, - old_carrier: false, - old_carrier_seen: false, - cache_carrier: false, + let record = ShapeRecord::new( + keys_id, logical_key_count, live_inline_slot_count, semantic_generation, object_kind, hole_count, - }; - // Publish by-id first, then the reverse accelerator. An ObjectHeader is + ); + // Publish by-id first, then the reverse accelerators. An ObjectHeader is // stamped only after this function returns, so a visible id always has a // complete descriptor. - inner.descriptors.insert(id, box_descriptor(descriptor)); - inner.ids_by_facts.entry(facts).or_default().push(id); - inner.ids_by_keys.entry(facts.keys).or_default().push(id); + // SAFETY: no slab reference is held; `slab()` above went out of scope. + unsafe { table.slab_mut().insert(id, record) }; + inner.facts_push_back(facts, id); + inner.family_push_back(keys_id, id); Ok(id) } @@ -547,34 +564,15 @@ pub(crate) fn shape_id_for_keys_ensure(keys: *const ArrayHeader, key_count: u32) /// One FIELD of a shape's descriptor, without lifting the whole record. /// /// [`shape_descriptor_by_id`] returns `ShapeDescriptor` **by value**, so every -/// caller that wants a single `u32` still copies the entire ~48-byte record -/// out of the table. That is most of them: `object_live_slot_count` — the slot -/// bound consulted on essentially every property read and write — throws away -/// all of it but `live_inline_slot_count`. -/// -/// This shares the way-cache probe with `shape_descriptor_by_id` and reads the -/// field through the record pointer instead. Same lookup, same validation, -/// four bytes instead of forty-eight. +/// caller that wants a single `u32` still copies the entire record out of the +/// table. That is most of them: `object_live_slot_count` — the slot bound +/// consulted on essentially every property read and write — throws away all +/// of it but `live_inline_slot_count`. #[inline] -fn shape_descriptor_field_by_id( - shape_id: u32, - read: impl Fn(&ShapeDescriptor) -> T, -) -> Option { - if !is_shape_id(shape_id) { - return None; - } - let table = &crate::state::state().shapes; - let epoch = table.lookup_epoch.get(); - let way = &table.lookup_ways[(shape_id as usize) & (SHAPE_LOOKUP_WAYS - 1)]; - let (cached_id, record, cached_epoch) = way.get(); - if cached_id == shape_id && cached_epoch == epoch && record != 0 { - // SAFETY: identical to `shape_descriptor_by_id`'s hit arm — the way is - // only filled from a live `Box` and the epoch is - // bumped whenever a record's address can change under an id still in - // use, so a matching epoch means this address is the table's record. - return Some(read(unsafe { &*(record as *const ShapeDescriptor) })); - } - shape_descriptor_by_id(shape_id).map(|d| read(&d)) +fn shape_descriptor_field_by_id(shape_id: u32, read: impl Fn(&ShapeRecord) -> T) -> Option { + let record = crate::state::state().shapes.slab().record_ptr(shape_id)?; + // SAFETY: `record_ptr` only returns a live slab record. + Some(read(unsafe { &*record })) } /// The live inline-slot bound for `shape_id`, without copying its descriptor. @@ -582,45 +580,16 @@ pub(crate) fn shape_live_inline_slot_count_by_id(shape_id: u32) -> Option { shape_descriptor_field_by_id(shape_id, |d| d.live_inline_slot_count) } -pub(crate) fn shape_descriptor_by_id(shape_id: u32) -> Option { - if !is_shape_id(shape_id) { - return None; - } - let table = &crate::state::state().shapes; - let epoch = table.lookup_epoch.get(); - let way = &table.lookup_ways[(shape_id as usize) & (SHAPE_LOOKUP_WAYS - 1)]; - - // Hit: mask, compare, deref. No RefCell borrow, no hash probe. - let (cached_id, record, cached_epoch) = way.get(); - if cached_id == shape_id && cached_epoch == epoch && record != 0 { - // SAFETY: the way is only filled from a live `Box`, - // and the epoch is bumped whenever a record's address can change under - // an id still in use, so a matching epoch means this address is the - // one the table holds for `shape_id`. - return Some(unsafe { *(record as *const ShapeDescriptor) }); - } - - let inner = table.inner.borrow(); - let record = inner.descriptors.get(&shape_id)?; - // `descriptor.record` is the box's own address (self-referential, #8112), - // so it is exactly the stable pointer the cache wants. - way.set((shape_id, record.record, epoch)); - Some(lift_descriptor(record)) -} - -/// Invalidate the whole lookup_ways cache. +/// The descriptor named by `shape_id`, or `None` when the id names no +/// descriptor in this agent. /// -/// Called where a record's ADDRESS can change while its id stays in use: -/// removal, and the insert path that can replace an existing id with a fresh -/// `Box`. A fresh-id insert deliberately does NOT bump — it cannot invalidate -/// an existing way, and bumping there would flush the cache on every shape -/// creation, which is precisely the workload that has one. +/// #9706: a slab probe — range check, chunk index, record — with no hash, +/// no `RefCell` borrow and no invalidation epoch. The direct-mapped way cache +/// that used to front the hash map is gone because the slab IS that cache: +/// a hit was "mask, compare, deref" and a probe is "shift, index, deref". #[inline] -fn invalidate_shape_lookup_cache() { - let table = &crate::state::state().shapes; - table - .lookup_epoch - .set(table.lookup_epoch.get().wrapping_add(1)); +pub(crate) fn shape_descriptor_by_id(shape_id: u32) -> Option { + crate::state::state().shapes.slab().lift(shape_id) } /// Immutable ordinary-vs-class fact with a pointer-free, per-agent direct @@ -636,31 +605,6 @@ pub(crate) fn shape_object_kind_by_id(shape_id: u32) -> Option Some(kind) } -/// Box a descriptor and stamp the record with its OWN address (#8112). -/// -/// Self-referential on purpose. The alternative — deriving the address in -/// `lift_descriptor` from the `&ShapeDescriptor` a shared table borrow yields — -/// would hand the collector a pointer with SHARED provenance and then write -/// through it. Taking it from the box while it is still uniquely owned keeps -/// the write well-formed. -fn box_descriptor(descriptor: ShapeDescriptor) -> Box { - let mut boxed = Box::new(descriptor); - boxed.record = std::ptr::addr_of_mut!(*boxed) as usize; - boxed -} - -/// Copy a boxed record out of the table (#8112). -/// -/// The copy's `keys` is a snapshot; `record` — stamped by [`box_descriptor`] — -/// names the one storage the collector rewrites. A caller that only reads facts -/// uses the snapshot; the GC hands `keys_slot()` to the slot visitor, so a -/// moved keys array lands back in the table with no second probe and no -/// write-back callback. -#[inline] -fn lift_descriptor(record: &ShapeDescriptor) -> ShapeDescriptor { - *record -} - /// Record that a shape is carried by an OLD-generation receiver. /// /// Called from the collector's slot visitor, which resolved the descriptor for @@ -672,10 +616,11 @@ fn lift_descriptor(record: &ShapeDescriptor) -> ShapeDescriptor { /// /// # Safety /// -/// `descriptor.record`, when non-zero, is the address of a live boxed record -/// owned by this agent's shape table. Records are freed only by -/// `prune_dead_shape_keys`, which runs at sweep — after every enumeration of -/// the cycle that produced this descriptor. +/// `descriptor.record`, when non-zero, is the address of a live slab record +/// owned by this agent's shape table. Records are retired only by the table's +/// own retirement paths, and their chunk is released by +/// `shrink_shape_tables` at the end of a major collection — after every +/// enumeration of the cycle that produced this descriptor. #[inline] pub(crate) unsafe fn note_old_generation_carrier(descriptor: Option) { let Some(descriptor) = descriptor else { @@ -684,10 +629,9 @@ pub(crate) unsafe fn note_old_generation_carrier(descriptor: Option) { if descriptor.record == 0 { return; } - let record = descriptor.record as *mut ShapeDescriptor; - // GC_STORE_AUDIT(POINTER_FREE): liveness bookkeeping byte, never a heap reference. - (*record).cache_carrier = true; + let record = descriptor.record as *mut ShapeRecord; + // GC_STORE_AUDIT(POINTER_FREE): liveness bookkeeping bit, never a heap reference. + (*record).set(RECORD_FLAG_CACHE_CARRIER, true); } /// The post-birth publication point for a ShapeId into a receiver's header @@ -747,10 +691,10 @@ pub(crate) unsafe fn stamp_object_shape_id_with_carrier_note( /// Clear every `cache_carrier` bit ahead of the post-full-trace recompute. pub(crate) fn clear_all_cache_carriers() { - let mut inner = crate::state::state().shapes.inner.borrow_mut(); - for record in inner.descriptors.values_mut() { - record.cache_carrier = false; - } + crate::state::state().shapes.slab().for_each(|_, record| { + // SAFETY: live slab record, single-threaded agent. + unsafe { (*record).set(RECORD_FLAG_CACHE_CARRIER, false) }; + }); } /// Recompute the old-carrier gate from the trace that just finished. @@ -761,11 +705,14 @@ pub(crate) fn clear_all_cache_carriers() { /// trace to shed a shape whose last old carrier died — the same rule that /// governs every other old-generation reclamation. pub(crate) fn rotate_old_carrier_epoch_after_full_trace() { - let mut inner = crate::state::state().shapes.inner.borrow_mut(); - for record in inner.descriptors.values_mut() { - record.old_carrier = record.old_carrier_seen; - record.old_carrier_seen = false; - } + crate::state::state().shapes.slab().for_each(|_, record| { + // SAFETY: live slab record, single-threaded agent. + unsafe { + let seen = (*record).has(RECORD_FLAG_OLD_CARRIER_SEEN); + (*record).set(RECORD_FLAG_OLD_CARRIER, seen); + (*record).set(RECORD_FLAG_OLD_CARRIER_SEEN, false); + } + }); } /// Mint (or retrieve) the ShapeId paired with canonical keys and equal @@ -1264,6 +1211,7 @@ pub(crate) unsafe fn publish_object_shape_from( // shared array must have cloned before push; otherwise siblings already // observe mutated bytes and no descriptor can make that state sound. let old_id = object_shape_stamp(obj); + let mut retire_owned_history = false; if let Some(old) = shape_descriptor_by_id(old_id) { // #9064: an owned ordinary receiver that already entered stable- // tombstone mode keeps its id across same-allocation tail appends and @@ -1301,7 +1249,14 @@ pub(crate) unsafe fn publish_object_shape_from( if shared { return old_id; } - retain_key_count_versions(keys as u64); + // An Array-subclass receiver is the one owner whose history IS + // reinstalled: `array_tail_transition` learns the (predecessor, + // successor) pair right after this publish returns and its + // reverse edge stamps the predecessor back on `pop`. That cache + // takes ownership through `cache_carrier`, but only once the + // learner has run, so the gate here is the receiver kind the + // learner is scoped to (`record_array_tail` in the append tail). + retire_owned_history = !crate::array::is_array_subclass_class_id((*obj).class_id); } } @@ -1332,10 +1287,55 @@ pub(crate) unsafe fn publish_object_shape_from( hole_count, )); stamp_object_shape_id_with_carrier_note(obj, id); + if retire_owned_history { + // #9706: the array is OWNED, so this receiver was the only carrier of + // every earlier same-address version, and the stamp above just + // superseded the last of them. Retire the growth history now rather + // than leaving one prefix descriptor per append alive until the + // array itself dies: on the compiled claude-code TUI that history was + // most of the descriptor table. Ordered after the stamp for the same + // reason as the tombstone publish (#9200) — the successor must be + // armed before the armed predecessor goes. + retire_owned_shape_siblings(keys as u64, id); + } debug_assert_object_shape_parity_for_keys(obj, keys); id } +/// Retire every descriptor of an OWNED keys array other than `keep`. +/// +/// Sound because `GC_FLAG_SHAPE_SHARED` is sticky: an array without it has +/// had exactly one owner for its whole life, and that owner now carries +/// `keep`. A stale IC token already misses on the stamp compare and +/// `shape_descriptor_by_id` of a retired id is `None`, so nothing can observe +/// the retired versions — with one exception: a descriptor an optimization +/// cache permanently owns (`cache_carrier`) may be reinstalled by that cache +/// while no live object carries it, so it stays. +fn retire_owned_shape_siblings(keys: u64, keep: u32) { + let table = &crate::state::state().shapes; + let mut inner = table.inner.borrow_mut(); + let stale: Vec = inner + .families + .get(&keys) + .map(|ids| { + ids.as_slice() + .iter() + .copied() + .filter(|&id| { + id != keep + && table + .slab() + .get(id) + .is_some_and(|record| !record.has(RECORD_FLAG_CACHE_CARRIER)) + }) + .collect() + }) + .unwrap_or_default(); + for id in stale { + remove_descriptor_and_reverse_indices(&mut inner, id); + } +} + /// Mint an exact successor for a descriptor/prototype semantic transition. /// The structural facts remain unchanged, but the process-unique generation /// prevents a cache trained before the transition from comparing equal after @@ -1433,37 +1433,36 @@ pub(crate) unsafe fn object_shape_id(obj: *const crate::object::ObjectHeader) -> .unwrap_or(0) } -fn retain_key_count_versions(keys: u64) { - let mut inner = crate::state::state().shapes.inner.borrow_mut(); - let Some(ids) = inner.ids_by_keys.remove(&keys) else { +/// Retire `id` from the by-id store and the family index (#9706). +/// +/// The family index is keyed by the address the descriptor was indexed under. +/// Between a live receiver rewriting the record's `keys` word and the +/// metadata scan moving the family, the two can name different addresses; a +/// removal in that window leaves the id in the stale family, where every +/// walk skips it (`record_ptr` is `None`) and the next scan drops it. +fn remove_descriptor_and_reverse_indices(inner: &mut ShapeTableInner, id: u32) { + let table = &crate::state::state().shapes; + let Some(indexed) = table.slab().get(id).map(|record| record.keys) else { return; }; - let mut current_ids = Vec::with_capacity(ids.len()); - for id in ids { - let Some(descriptor) = inner.descriptors.get(&id).map(|record| **record) else { - continue; - }; - debug_assert_eq!( - descriptor.keys, keys, - "keys index contains a foreign descriptor" - ); - if descriptor.keys != keys { - let correct_ids = inner.ids_by_keys.entry(descriptor.keys).or_default(); - if !correct_ids.contains(&id) { - correct_ids.push(id); - } - } else { - // Keep immutable historical descriptors addressable by id. An - // append under an owned keys allocation preserves the old prefix, - // and a stale cache/object may still carry either a local or an - // equivalent external id. Dead-key pruning reclaims the whole - // lineage once no live owner reaches the keys allocation. - current_ids.push(id); - } - } - if !current_ids.is_empty() { - inner.ids_by_keys.insert(keys, current_ids); + remove_descriptor_indexed_under(inner, id, indexed); +} + +/// [`remove_descriptor_and_reverse_indices`] for a caller that knows the +/// address the id is indexed under — the metadata scan, which retires a +/// family whose keys address was recycled while a live edge may already have +/// rewritten the records to the forwarded address. +fn remove_descriptor_indexed_under(inner: &mut ShapeTableInner, id: u32, indexed: u64) { + let table = &crate::state::state().shapes; + // SAFETY: no slab reference is held by the caller across this call. + let Some(record) = (unsafe { table.slab_mut().remove(id) }) else { + return; + }; + retire_cached_shape_object_kind(id); + if record.has(RECORD_FLAG_FACTS_INDEXED) { + inner.facts_remove(record.facts_key_with_keys(indexed), id); } + inner.family_remove(indexed, id); } /// Exact-facts test for a candidate id against the receiver's authoritative @@ -1760,7 +1759,8 @@ fn shape_keys_address_is_recycled(addr: usize) -> bool { /// descriptor removed here cannot be named by a live object. Correctness fails /// closed on a missing lookup_ways, independently of pruning. pub(crate) fn prune_dead_shape_keys(is_dead_owner: &dyn Fn(usize) -> bool) { - let mut inner = crate::state::state().shapes.inner.borrow_mut(); + let table = &crate::state::state().shapes; + let mut inner = table.inner.borrow_mut(); // A shape keys entry is keyed by the address of its keys array — a // `GC_TYPE_ARRAY` (or `GC_TYPE_LAZY_ARRAY`). When the keys array dies // and the arena recycles its address for a different object type @@ -1776,132 +1776,128 @@ pub(crate) fn prune_dead_shape_keys(is_dead_owner: &dyn Fn(usize) -> bool) { !is_dead_owner(*keys_id) && !shape_keys_address_is_recycled(*keys_id) }); } - let stale: Vec = inner - .descriptors - .iter() - .filter_map(|(&id, descriptor)| { - let keys = descriptor.keys as usize; - (is_dead_owner(descriptor.keys as usize) || shape_keys_address_is_recycled(keys)) - .then_some(id) - }) - .collect(); - if !stale.is_empty() { - for id in stale { - remove_descriptor_and_reverse_indices(&mut inner, id); + let mut stale: Vec = Vec::new(); + table.slab().for_each(|id, record| { + // SAFETY: live slab record, read immediately. + let descriptor = unsafe { *record }; + let keys = descriptor.keys as usize; + if is_dead_owner(descriptor.keys as usize) || shape_keys_address_is_recycled(keys) { + stale.push(id); } + }); + for id in stale { + remove_descriptor_and_reverse_indices(&mut inner, id); } } -crate::perry_thread_local! { - /// Scratch memo for [`scan_shape_table_rekey_mut`]'s per-address probe, - /// reused across collections so the scan allocates nothing. - /// `PtrHashMap`, NOT std's SipHash default: perf on the dynamic-property - /// benchmark put `RandomState::hash_one::<&(usize, bool)>` at **7.0% of - /// total samples** — pure hashing overhead inside the GC scan this memo - /// exists to make cheaper. The key is folded to one word (`addr ^ carrier` - /// in bit 0; addresses are >= 8-aligned so bit 0 is free), which is the - /// single-word shape `PtrHasher` is built for. - static PROBE_MEMO: std::cell::RefCell> = - std::cell::RefCell::new(crate::fast_hash::new_ptr_hash_map()); -} - /// Metadata-only forwarding repair for the weak descriptor table and /// pointer-keyed slot indices. Mark/copy mode does not root anything; live /// object scans provide descriptor reachability, and post-copy rewrite follows /// only forwarding records those live edges already created. +/// +/// #9706: the walk is per keys-array FAMILY, not per descriptor. Every +/// descriptor of a family shares one keys address, so one probe answers for +/// all of them — the per-address memo the descriptor walk used to keep +/// (`PROBE_MEMO`, a persistent map sized to every distinct address in the +/// table) is now simply the family index itself. A family is probed with the +/// MARKING visit when any of its descriptors is a carrier, which is exactly +/// the rooting duty the #8112 gate assigns: the keys array must survive while +/// an old receiver or a cache still names one of its shapes. pub(crate) fn scan_shape_table_rekey_mut(visitor: &mut crate::gc::RuntimeRootVisitor<'_>) { - let mut inner = crate::state::state().shapes.inner.borrow_mut(); - // TEMPORARY (#6759 phase 2 measurement): this scanner is 53.3% of all - // root-scanner time on `claude -p`. Report what it is actually walking so - // the fix targets the real term instead of a guess. - let mut descriptor_rekeys: Vec = Vec::new(); - let mut dead_descriptor_ids: Vec = Vec::new(); - - // #6759 phase 2: probe each distinct keys-array address ONCE. - // - // The per-descriptor probe is the expensive part of this scanner — 89.6% of - // its time is the rewrite phase, and each probe runs - // `classify_heap_space_in_range` and then reads the GC header at a - // scattered address (two likely cache misses). Shapes share keys arrays at - // a measured, stable 2.5:1, so the unmemoised loop paid that ~2.5 times per - // distinct address. - // - // Memoising is sound by construction: the same addresses are visited, just - // once each, and forwarding is a pure function of the address within one - // pass. Carriers take a different visit (`visit_usize_slot`, which MARKS in - // mark modes) than non-carriers, so the carrier flag is part of the key — - // otherwise a non-carrier hit could satisfy a carrier's marking duty. - // Reused across collections rather than allocated per scan: at ~300k - // entries a fresh map every GC is exactly the kind of churn the - // memory-parity work is trying to remove. `clear()` keeps the capacity. - PROBE_MEMO.with(|memo| { - let mut probe_memo = memo.borrow_mut(); - probe_memo.clear(); - - for (id, descriptor) in inner.descriptors.iter_mut() { - let mut addr = descriptor.keys as usize; - // #8112 ephemeron gate. A shape with an OLD carrier is rooted here: - // the minor that has to keep its keys array alive never enumerates the - // object that carries it. A shape with only young carriers is NOT — - // those receivers are traced, and each one emits the edge itself, so - // rooting them from the table would make every keys array ever minted - // immortal and turn `prune_dead_shape_keys`'s "is the keys array - // dead?" into a question it asks of itself. - let is_carrier = descriptor.old_carrier || descriptor.cache_carrier; - // Addresses are 8-aligned, so bit 0 is free to carry the carrier - // duty (carriers use a MARKING visit; the answers must not mix). - let memo_key = addr | usize::from(is_carrier); - if let Some(&(prev_moved, prev_addr)) = probe_memo.get(&memo_key) { - // Already probed this exact (address, carrier-duty) pair in this - // pass — reuse the answer instead of paying the walk again. - let moved = prev_moved; - addr = prev_addr; - record_shape_scan_outcome( - visitor, - id, - descriptor, - addr, - moved, - &mut dead_descriptor_ids, - &mut descriptor_rekeys, - ); - continue; + let table = &crate::state::state().shapes; + let mut inner = table.inner.borrow_mut(); + let rewrite_phase = visitor.is_metadata_rewrite_phase(); + let mut moved_families: Vec<(u64, u64)> = Vec::new(); + let mut dead_descriptor_ids: Vec<(u32, u64)> = Vec::new(); + // The shared slab view is scoped to the probe loop: retirement below + // takes the slab mutably, and nothing after the loop may still hold it. + let slab = table.slab(); + for (&indexed, ids) in inner.families.iter() { + if indexed == 0 { + // Keyless shapes hold no edge. + continue; + } + // #8112 ephemeron gate. A shape with an OLD carrier is rooted here: + // the minor that has to keep its keys array alive never enumerates the + // object that carries it. A shape with only young carriers is NOT — + // those receivers are traced, and each one emits the edge itself, so + // rooting them from the table would make every keys array ever minted + // immortal and turn `prune_dead_shape_keys`'s "is the keys array + // dead?" into a question it asks of itself. + // + // One descriptor stands for the family: a carrier if the family has + // one (its duty is the strongest), else any present member. + let mut descriptor: Option = None; + for &id in ids.as_slice() { + if let Some(lifted) = slab.lift(id) { + if lifted.old_carrier || lifted.cache_carrier { + descriptor = Some(lifted); + break; + } + descriptor.get_or_insert(lifted); } - let probe_addr = addr; - // Written out rather than reusing `is_carrier` on purpose: the - // census gate (`scripts/shape_descriptor_census.py`) pins this exact - // two-armed expression so that a sabotage which widens the gate or - // swaps the arms is red, and its own self-test sabotages this very - // literal. `is_carrier` above is the same predicate, and is what - // keys the memo. - let moved = if descriptor.old_carrier || descriptor.cache_carrier { - visitor.visit_usize_slot(&mut addr) - } else { - visitor.visit_metadata_usize_slot(&mut addr) - }; - probe_memo.insert(probe_addr | usize::from(is_carrier), (moved, addr)); - record_shape_scan_outcome( - visitor, - id, - descriptor, - addr, - moved, - &mut dead_descriptor_ids, - &mut descriptor_rekeys, - ); } - }); - // Remove descriptors whose keys array was recycled. - if !dead_descriptor_ids.is_empty() { - for id in &dead_descriptor_ids { - remove_descriptor_and_reverse_indices(&mut inner, *id); + let Some(descriptor) = descriptor else { + // Every id retired under a stale address; the family is empty. + moved_families.push((indexed, 0)); + continue; + }; + let mut addr = indexed as usize; + // The census gate (`scripts/shape_descriptor_census.py`) pins this + // exact two-armed expression so that a sabotage which widens the gate + // or swaps the arms is red, and its own self-test sabotages this very + // literal. + let moved = if descriptor.old_carrier || descriptor.cache_carrier { + visitor.visit_usize_slot(&mut addr) + } else { + visitor.visit_metadata_usize_slot(&mut addr) + }; + // Validate the POST-visit address. A stale shape key can follow the + // forwarding record of the non-array tenant that recycled its address; + // checking only an unmoved old address misses that case. + if rewrite_phase && shape_keys_address_is_recycled(addr) { + dead_descriptor_ids.extend(ids.as_slice().iter().map(|&id| (id, indexed))); + continue; + } + if moved { + for &id in ids.as_slice() { + if let Some(record) = slab.record_ptr(id) { + // SAFETY: live slab record, single-threaded agent. A live + // receiver's edge may already have written the same + // forwarded address here; the store is idempotent. + unsafe { (*record).keys = addr as u64 }; + } + } } + if addr as u64 != indexed { + moved_families.push((indexed, addr as u64)); + } + } + for (id, indexed) in dead_descriptor_ids { + remove_descriptor_indexed_under(&mut inner, id, indexed); } - for id in descriptor_rekeys { - sync_descriptor_reverse_indices(&mut inner, id); + for (old, new) in moved_families { + let Some(ids) = inner.families.remove(&old) else { + continue; + }; + if new == 0 { + continue; + } + for &id in ids.as_slice() { + let Some(record) = table.slab().get(id) else { + continue; + }; + // The accelerator was keyed with the OLD address; the other five + // facts never change under the collector. + if record.has(RECORD_FLAG_FACTS_INDEXED) { + inner.facts_remove(record.facts_key_with_keys(old), id); + inner.facts_push_back(record.facts_key_with_keys(new), id); + } + inner.family_push_back(new, id); + } } - if !visitor.is_metadata_rewrite_phase() || inner.indices.is_empty() { + if !rewrite_phase || inner.indices.is_empty() { return; } let moved: Vec<(usize, usize)> = inner @@ -1973,47 +1969,43 @@ mod shapes_tests; /// `prune_dead_shape_keys` had already discarded. /// /// Called once per MAJOR collection, right after the prune, where a rehash is -/// already amortized against a full heap walk. `shrink_to(2 * len)` rather -/// than `shrink_to_fit()` keeps one doubling of headroom so a table that is -/// merely oscillating does not re-grow on the next insert. +/// already amortized against a full heap walk. #9706: the by-id store is a +/// slab now, so this also releases its all-dead chunks; the family and slot +/// index maps are shrunk to `len + len / 4`, one growth step of headroom. pub(crate) fn shrink_shape_tables() { - fn worth_shrinking(len: usize, capacity: usize, _: &T) -> bool { - // Only when the table is holding at least 1 MB-ish of slack and is - // less than half used; a small or well-packed table is left alone. + fn worth_shrinking(len: usize, capacity: usize) -> bool { + // Only when the table is holding real slack and is less than half + // used; a small or well-packed table is left alone. capacity > 4096 && capacity > len.saturating_mul(2) } - let mut inner = crate::state::state().shapes.inner.borrow_mut(); - if worth_shrinking(inner.descriptors.len(), inner.descriptors.capacity(), &()) { - let target = inner.descriptors.len().saturating_mul(2); - inner.descriptors.shrink_to(target); - } - if worth_shrinking(inner.indices.len(), inner.indices.capacity(), &()) { - let target = inner.indices.len().saturating_mul(2); + let table = &crate::state::state().shapes; + let mut inner = table.inner.borrow_mut(); + if worth_shrinking(inner.indices.len(), inner.indices.capacity()) { + let target = inner.indices.len() + inner.indices.len() / 4; inner.indices.shrink_to(target); } - if worth_shrinking(inner.ids_by_facts.len(), inner.ids_by_facts.capacity(), &()) { - let target = inner.ids_by_facts.len().saturating_mul(2); - inner.ids_by_facts.shrink_to(target); + if worth_shrinking(inner.by_facts.len(), inner.by_facts.capacity()) { + let target = inner.by_facts.len() + inner.by_facts.len() / 4; + inner.by_facts.shrink_to(target); } - if worth_shrinking(inner.ids_by_keys.len(), inner.ids_by_keys.capacity(), &()) { - let target = inner.ids_by_keys.len().saturating_mul(2); - inner.ids_by_keys.shrink_to(target); + if worth_shrinking(inner.families.len(), inner.families.capacity()) { + let target = inner.families.len() + inner.families.len() / 4; + inner.families.shrink_to(target); } + // SAFETY: the prune that precedes this call holds no slab reference, and + // neither does anything else while the major collection owns the agent. + unsafe { table.slab_mut().release_empty_chunks() }; } -/// `PERRY_GC_CENSUS`: the shape table's four maps plus the boxed -/// descriptors and per-shape key indices they own. +/// `PERRY_GC_CENSUS`: the by-id slab, the per-shape key indices, the +/// exact-facts accelerator and the keys-address family index. pub(crate) fn shape_table_census() -> Vec { - use crate::gc::census::{hash_table_bytes, map_bytes, vec_bytes}; + use crate::gc::census::{hash_table_bytes, map_bytes}; let table = &crate::state::state().shapes; let inner = table.inner.borrow(); + let slab = table.slab(); let mut rows = Vec::new(); - rows.push(( - "shapes.descriptors", - inner.descriptors.len(), - map_bytes(&inner.descriptors) - + inner.descriptors.len() * (std::mem::size_of::() + 16), - )); + rows.push(("shapes.descriptors", slab.len(), slab.estimated_bytes())); let index_inner: usize = inner .indices .values() @@ -2024,17 +2016,87 @@ pub(crate) fn shape_table_census() -> Vec { inner.indices.len(), map_bytes(&inner.indices) + index_inner, )); - let facts_inner: usize = inner.ids_by_facts.values().map(vec_bytes).sum(); + let facts_inner: usize = inner.by_facts.values().map(IdList::heap_bytes).sum(); rows.push(( - "shapes.ids_by_facts", - inner.ids_by_facts.len(), - map_bytes(&inner.ids_by_facts) + facts_inner, + "shapes.by_facts", + inner.by_facts.len(), + map_bytes(&inner.by_facts) + facts_inner, )); - let keys_inner: usize = inner.ids_by_keys.values().map(vec_bytes).sum(); + let families_inner: usize = inner.families.values().map(IdList::heap_bytes).sum(); rows.push(( - "shapes.ids_by_keys", - inner.ids_by_keys.len(), - map_bytes(&inner.ids_by_keys) + keys_inner, + "shapes.families", + inner.families.len(), + map_bytes(&inner.families) + families_inner, )); + // Ids ever minted by this process: the slab is indexed by id, so the gap + // between this and `shapes.descriptors` is what chunk release reclaims. + let minted = SHAPE_ID_NEXT.load(std::sync::atomic::Ordering::Relaxed) - SHAPE_ID_BASE; + rows.push(("shapes.ids_minted(process)", minted as usize, 0)); rows } + +/// `PERRY_GC_CENSUS`: how the descriptor population relates to the live heap +/// (#9706). `live_ids` is the sorted, deduplicated set of ShapeIds stamped on +/// live shaped objects, collected by the census walk. +/// +/// * `shapes.descriptors.carried` — descriptors some live object is stamped +/// with: the population V8's "object shape" bucket corresponds to. +/// * `shapes.descriptors.uncarried` — descriptors no live object carries: +/// transition history a cache may reinstall (`cache_carrier`), versions +/// kept for an old receiver since the last full trace, and shapes whose +/// keys array is still alive on some other descriptor. +/// * `shapes.families.multi` — keys arrays with more than one descriptor, +/// and the descriptors they hold beyond the first: the duplication the +/// family walk pays for. +pub(crate) fn shape_table_liveness_census( + live_ids: &[u32], +) -> Vec { + let table = &crate::state::state().shapes; + let inner = table.inner.borrow(); + let slab = table.slab(); + let mut carried = 0usize; + let mut uncarried = 0usize; + let mut uncarried_cache = 0usize; + let mut uncarried_old = 0usize; + slab.for_each(|id, record| { + if live_ids.binary_search(&id).is_ok() { + carried += 1; + return; + } + uncarried += 1; + // SAFETY: live slab record, read immediately. + let record = unsafe { *record }; + if record.has(RECORD_FLAG_CACHE_CARRIER) { + uncarried_cache += 1; + } else if record.has(RECORD_FLAG_OLD_CARRIER) { + uncarried_old += 1; + } + }); + let mut multi_families = 0usize; + let mut multi_extra = 0usize; + let mut largest = 0usize; + for ids in inner.families.values() { + let n = ids.len(); + largest = largest.max(n); + if n > 1 { + multi_families += 1; + multi_extra += n - 1; + } + } + vec![ + ("shapes.descriptors.carried(live objects)", carried, 0), + ("shapes.descriptors.uncarried", uncarried, 0), + ( + "shapes.descriptors.uncarried.cache_carrier", + uncarried_cache, + 0, + ), + ("shapes.descriptors.uncarried.old_carrier", uncarried_old, 0), + ( + "shapes.families.multi(families,extra descriptors)", + multi_families, + multi_extra, + ), + ("shapes.families.largest", largest, 0), + ] +} diff --git a/crates/perry-runtime/src/object/shapes_reverse_indices.rs b/crates/perry-runtime/src/object/shapes_reverse_indices.rs deleted file mode 100644 index 59b475fc16..0000000000 --- a/crates/perry-runtime/src/object/shapes_reverse_indices.rs +++ /dev/null @@ -1,119 +0,0 @@ -//! Reverse-index maintenance for the shape descriptor table. -//! -//! Descriptors are indexed both by exact semantic facts and by their keys -//! allocation. Stable-tombstone epochs deliberately leave exact-facts -//! interning while retaining the keys index, so all insertion, relocation, -//! and retirement bookkeeping lives together here. - -use super::{ - invalidate_shape_lookup_cache, retire_cached_shape_object_kind, ShapeDescriptor, ShapeFacts, - ShapeTableInner, -}; - -#[inline] -pub(super) fn descriptor_facts(descriptor: ShapeDescriptor) -> ShapeFacts { - ShapeFacts { - keys: descriptor.keys, - logical_key_count: descriptor.logical_key_count, - live_inline_slot_count: descriptor.live_inline_slot_count, - semantic_generation: descriptor.semantic_generation, - object_kind: descriptor.object_kind, - hole_count: descriptor.hole_count, - } -} - -fn descriptor_facts_with_keys(descriptor: ShapeDescriptor, keys: u64) -> ShapeFacts { - ShapeFacts { - keys, - logical_key_count: descriptor.logical_key_count, - live_inline_slot_count: descriptor.live_inline_slot_count, - semantic_generation: descriptor.semantic_generation, - object_kind: descriptor.object_kind, - hole_count: descriptor.hole_count, - } -} - -pub(super) fn remove_descriptor_id_from_facts_index( - inner: &mut ShapeTableInner, - facts: ShapeFacts, - id: u32, -) { - let remove_entry = if let Some(ids) = inner.ids_by_facts.get_mut(&facts) { - if let Ok(index) = ids.binary_search(&id) { - ids.remove(index); - } else { - ids.retain(|&candidate| candidate != id); - } - ids.is_empty() - } else { - false - }; - if remove_entry { - inner.ids_by_facts.remove(&facts); - } -} - -fn remove_descriptor_id_from_keys_index(inner: &mut ShapeTableInner, keys: u64, id: u32) { - let remove_entry = if let Some(ids) = inner.ids_by_keys.get_mut(&keys) { - if let Ok(index) = ids.binary_search(&id) { - ids.remove(index); - } else { - ids.retain(|&candidate| candidate != id); - } - ids.is_empty() - } else { - false - }; - if remove_entry { - inner.ids_by_keys.remove(&keys); - } -} - -#[inline] -pub(super) fn insert_descriptor_id_sorted(ids: &mut Vec, id: u32) { - if let Err(index) = ids.binary_search(&id) { - ids.insert(index, id); - } -} - -/// Repair one descriptor after its collector-owned `keys` slot moved. -/// -/// `indexed_keys` records the address under which the id is still indexed, so -/// this is O(population sharing the old/new facts) rather than O(all shapes). -pub(super) fn sync_descriptor_reverse_indices(inner: &mut ShapeTableInner, id: u32) { - let Some(descriptor) = inner.descriptors.get(&id).map(|record| **record) else { - return; - }; - if descriptor.indexed_keys == descriptor.keys { - return; - } - - let old_facts = descriptor_facts_with_keys(descriptor, descriptor.indexed_keys); - let new_facts = descriptor_facts(descriptor); - if descriptor.facts_indexed { - remove_descriptor_id_from_facts_index(inner, old_facts, id); - } - remove_descriptor_id_from_keys_index(inner, descriptor.indexed_keys, id); - if descriptor.facts_indexed { - insert_descriptor_id_sorted(inner.ids_by_facts.entry(new_facts).or_default(), id); - } - insert_descriptor_id_sorted(inner.ids_by_keys.entry(descriptor.keys).or_default(), id); - if let Some(record) = inner.descriptors.get_mut(&id) { - record.indexed_keys = descriptor.keys; - } -} - -pub(super) fn remove_descriptor_and_reverse_indices(inner: &mut ShapeTableInner, id: u32) { - // The record's box is about to be dropped; any cached way naming it must - // stop matching. - invalidate_shape_lookup_cache(); - let Some(descriptor) = inner.descriptors.remove(&id) else { - return; - }; - retire_cached_shape_object_kind(id); - let facts = descriptor_facts_with_keys(*descriptor, descriptor.indexed_keys); - if descriptor.facts_indexed { - remove_descriptor_id_from_facts_index(inner, facts, id); - } - remove_descriptor_id_from_keys_index(inner, descriptor.indexed_keys, id); -} diff --git a/crates/perry-runtime/src/object/shapes_slot_list.rs b/crates/perry-runtime/src/object/shapes_slot_list.rs index 04fe163224..0523e4df43 100644 --- a/crates/perry-runtime/src/object/shapes_slot_list.rs +++ b/crates/perry-runtime/src/object/shapes_slot_list.rs @@ -79,38 +79,7 @@ impl SlotList { } } -use super::{shape_keys_address_is_recycled, ShapeDescriptor}; - -/// Per-descriptor bookkeeping after its keys address has been probed. -/// -/// Lifted out of `scan_shape_table_rekey_mut`'s loop so the memoised path and -/// the probing path cannot drift apart — the probe is what is deduplicated, -/// never the bookkeeping, which still runs once per descriptor. -#[inline] -pub(crate) fn record_shape_scan_outcome( - visitor: &mut crate::gc::RuntimeRootVisitor<'_>, - id: &u32, - descriptor: &mut ShapeDescriptor, - addr: usize, - moved: bool, - dead_descriptor_ids: &mut Vec, - descriptor_rekeys: &mut Vec, -) { - // Validate the POST-visit address. A stale shape key can follow the - // forwarding record of the non-array tenant that recycled its address; - // checking only an unmoved old address misses that case. - if visitor.is_metadata_rewrite_phase() && shape_keys_address_is_recycled(addr) { - dead_descriptor_ids.push(*id); - } else if moved { - descriptor.keys = addr as u64; - } - // A live-object edge can rewrite the boxed `keys` slot before this metadata - // pass. Comparing against the address represented in the reverse maps - // catches both that ordering and a move observed here. - if descriptor.keys != descriptor.indexed_keys { - descriptor_rekeys.push(*id); - } -} +use super::shapes_store::{ShapeRecord, RECORD_FLAG_FACTS_INDEXED}; /// Shift a key index in place after an IN-PLACE delete. /// @@ -233,9 +202,15 @@ pub(crate) unsafe fn retire_owned_shape_history( let keys_addr = keys as u64; let mut inner = crate::state::state().shapes.inner.borrow_mut(); let stale: Vec = inner - .ids_by_keys + .families .get(&keys_addr) - .map(|ids| ids.iter().copied().filter(|&id| id != current).collect()) + .map(|ids| { + ids.as_slice() + .iter() + .copied() + .filter(|&id| id != current) + .collect() + }) .unwrap_or_default(); for id in stale { super::remove_descriptor_and_reverse_indices(&mut inner, id); @@ -253,10 +228,10 @@ pub(crate) unsafe fn retire_owned_shape_history( /// mint-then-stamp path. /// /// A mutable private epoch must not participate in exact-facts interning. -/// Detach it from `ids_by_facts` on entry and leave it in `ids_by_keys`, which -/// keeps GC relocation and squeeze-time retirement exact without hashing six -/// changing facts on every delete and re-add. The boxed descriptor address is -/// stable, so the direct lookup cache observes updated counts immediately. +/// Detach it on entry and leave it in the keys-address family, which keeps GC +/// relocation and squeeze-time retirement exact without re-indexing six +/// changing facts on every delete and re-add. The slab record address is +/// stable, so every later lookup observes the updated counts immediately. pub(crate) unsafe fn try_update_stable_tombstone_shape( obj: *mut crate::object::ObjectHeader, keys: *mut super::ArrayHeader, @@ -278,12 +253,14 @@ pub(crate) unsafe fn try_update_stable_tombstone_shape( return None; } - let mut inner = crate::state::state().shapes.inner.borrow_mut(); - super::sync_descriptor_reverse_indices(&mut inner, id); - let current = **inner.descriptors.get(&id)?; + let table = &crate::state::state().shapes; + let record = table.slab().record_ptr(id)?; + // SAFETY: live slab record, single-threaded agent; read then written + // through the same pointer with nothing else holding a reference. + let current = unsafe { *record }; // A stable id may never silently retarget its collector-owned keys edge. // Array growth that reallocates falls back to a fresh descriptor. - if current.keys != keys as u64 || current.object_kind != super::ShapeObjectKind::Ordinary { + if current.keys != keys as u64 || current.object_kind() != super::ShapeObjectKind::Ordinary { return None; } if current.logical_key_count == logical_key_count @@ -293,21 +270,21 @@ pub(crate) unsafe fn try_update_stable_tombstone_shape( return Some(id); } - if current.facts_indexed { - let old_facts = super::descriptor_facts(current); - super::remove_descriptor_id_from_facts_index(&mut inner, old_facts, id); + // Detach from exact-facts interning, so a mutable private epoch is never + // handed to a second receiver. It stays in the family for GC relocation + // and squeeze retirement. The accelerator was keyed with the address the + // record is indexed under, which is `keys` (the caller proved the edge + // did not move). + if current.has(RECORD_FLAG_FACTS_INDEXED) { + let mut inner = table.inner.borrow_mut(); + inner.facts_remove(current.facts_key_with_keys(keys as u64), id); + } + unsafe { + (*record).logical_key_count = logical_key_count; + (*record).live_inline_slot_count = live_inline_slot_count; + (*record).hole_count = hole_count; + (*record).set(RECORD_FLAG_FACTS_INDEXED, false); } - { - let record = inner - .descriptors - .get_mut(&id) - .expect("stable tombstone descriptor disappeared while borrowed"); - record.logical_key_count = logical_key_count; - record.live_inline_slot_count = live_inline_slot_count; - record.hole_count = hole_count; - record.facts_indexed = false; - } - drop(inner); super::debug_assert_object_shape_parity(obj); Some(id) } @@ -316,10 +293,9 @@ pub(crate) unsafe fn try_update_stable_tombstone_shape( /// record address returned by `shape_descriptor_by_id`. /// /// The first stable mutation must use `try_update_stable_tombstone_shape` to -/// detach exact-facts interning, and a collector-relocated keys edge must use -/// it to repair the reverse index. Between those events the record address is -/// stable, its mutable epoch is deliberately absent from `ids_by_facts`, and -/// no table borrow or hash lookup is needed for a counter-only update. +/// detach exact-facts interning. Between those events the record address is +/// stable, its mutable epoch is deliberately invisible to interning, and no +/// table borrow is needed for a counter-only update. pub(crate) unsafe fn try_update_stable_tombstone_shape_cached( obj: *mut crate::object::ObjectHeader, current: super::ShapeDescriptor, @@ -341,12 +317,17 @@ pub(crate) unsafe fn try_update_stable_tombstone_shape_cached( return None; } - let record = &mut *(current.record as *mut super::ShapeDescriptor); - if record.record != current.record - || record.keys != current.keys - || record.indexed_keys != record.keys - || record.facts_indexed - || record.object_kind != super::ShapeObjectKind::Ordinary + // The caller's copy must still name the live record of THIS id: a + // retired id resolves to nothing, and a record reused under another id + // (never — ids are not recycled) would resolve to a different address. + let live = crate::state::state().shapes.slab().record_ptr(id)?; + if live as usize != current.record { + return None; + } + let record = &mut *live; + if record.keys != current.keys + || record.has(RECORD_FLAG_FACTS_INDEXED) + || record.object_kind() != super::ShapeObjectKind::Ordinary { return None; } @@ -357,11 +338,11 @@ pub(crate) unsafe fn try_update_stable_tombstone_shape_cached( Some(id) } -/// Retire the token of a detached private epoch while reusing its boxed -/// descriptor record. This is the stable-tombstone squeeze counterpart to a -/// full mint: generated caches must observe a new id after slots are -/// compacted, but no exact-facts interning or new descriptor allocation is -/// needed for a record that cannot be shared by another receiver. +/// Retire the token of a detached private epoch while reusing its descriptor +/// record. This is the stable-tombstone squeeze counterpart to a full mint: +/// generated caches must observe a new id after slots are compacted, but no +/// exact-facts interning is needed for a record that cannot be shared by +/// another receiver. pub(crate) unsafe fn rekey_stable_tombstone_shape_after_squeeze( obj: *mut crate::object::ObjectHeader, current: super::ShapeDescriptor, @@ -388,36 +369,41 @@ pub(crate) unsafe fn rekey_stable_tombstone_shape_after_squeeze( super::shape_id_exhausted_abort(); } - let mut inner = crate::state::state().shapes.inner.borrow_mut(); - super::sync_descriptor_reverse_indices(&mut inner, old_id); - let live = **inner.descriptors.get(&old_id)?; - if live.record != current.record - || live.keys != current.keys - || live.indexed_keys != live.keys - || live.facts_indexed - || live.object_kind != super::ShapeObjectKind::Ordinary + let table = &crate::state::state().shapes; + let mut inner = table.inner.borrow_mut(); + let live_ptr = table.slab().record_ptr(old_id)?; + if live_ptr as usize != current.record { + return None; + } + // SAFETY: live slab record, read immediately. + let live = unsafe { *live_ptr }; + if live.keys != current.keys + || live.has(RECORD_FLAG_FACTS_INDEXED) + || live.object_kind() != super::ShapeObjectKind::Ordinary { return None; } - super::invalidate_shape_lookup_cache(); - let mut record = inner.descriptors.remove(&old_id)?; + // Move the record to its new id in place of the old one. The family entry + // is replaced where it stands; a family still keyed under a stale address + // (a rewrite the metadata scan has not yet repaired) simply gains the new + // id under the current one and sheds the old id on that scan. + // SAFETY: no slab reference is held across these two calls. + let mut record = unsafe { table.slab_mut().remove(old_id)? }; record.logical_key_count = logical_key_count; record.live_inline_slot_count = live_inline_slot_count; record.semantic_generation = generation; record.hole_count = hole_count; - if let Some(ids) = inner.ids_by_keys.get_mut(&record.indexed_keys) { - if let Some(pos) = ids.iter().position(|&id| id == old_id) { - ids[pos] = new_id; - ids.sort_unstable(); - } else { - super::insert_descriptor_id_sorted(ids, new_id); - } - } else { - inner.ids_by_keys.insert(record.indexed_keys, vec![new_id]); + super::retire_cached_shape_object_kind(old_id); + unsafe { table.slab_mut().insert(new_id, record) }; + let replaced = inner + .families + .get_mut(&record.keys) + .is_some_and(|ids| ids.replace(old_id, new_id)); + if !replaced { + inner.family_push_back(record.keys, new_id); } inner.indices.remove(&(record.keys as usize)); - inner.descriptors.insert(new_id, record); drop(inner); // #9200: the funnel re-arms the preserved record for a non-nursery @@ -498,9 +484,15 @@ pub(crate) unsafe fn publish_object_shape_holes( // (53.6 s → 25.1 s on the churn benchmark) but ids still accumulated // one per iteration from the append publish. let stale: Vec = inner - .ids_by_keys + .families .get(&(current.keys)) - .map(|ids| ids.iter().copied().filter(|&other| other != id).collect()) + .map(|ids| { + ids.as_slice() + .iter() + .copied() + .filter(|&other| other != id) + .collect() + }) .unwrap_or_default(); for other in stale { super::remove_descriptor_and_reverse_indices(&mut inner, other); @@ -524,42 +516,37 @@ pub(super) fn install_external_shape_id( if !super::is_shape_id(id) || (keys.is_null() && logical_key_count != 0) { return false; } - let descriptor = super::ShapeDescriptor { - keys: keys as usize as u64, - indexed_keys: keys as usize as u64, - facts_indexed: true, - record: 0, - old_carrier: false, - old_carrier_seen: false, - cache_carrier: false, + let keys = keys as usize as u64; + let record = ShapeRecord::new( + keys, logical_key_count, live_inline_slot_count, - semantic_generation: 0, - object_kind: super::ShapeObjectKind::Ordinary, - hole_count: 0, - }; - let facts = super::descriptor_facts(descriptor); - let mut inner = crate::state::state().shapes.inner.borrow_mut(); - if let Some(existing) = inner.descriptors.get(&id) { - return **existing == descriptor; + 0, + super::ShapeObjectKind::Ordinary, + 0, + ); + let table = &crate::state::state().shapes; + let mut inner = table.inner.borrow_mut(); + if let Some(existing) = table.slab().get(id) { + return existing.facts_match( + keys, + logical_key_count, + live_inline_slot_count, + 0, + super::ShapeObjectKind::Ordinary, + 0, + ); } // A worker can have minted an equivalent local descriptor before module // initialization installs the process-global codegen id. Keep both id // descriptors valid for already-published objects and make the external - // id canonical for subsequent births in this agent. - // - // This is the one insert that can REPLACE a live id with a fresh box, so - // the lookup_ways cache has to be invalidated here (the fresh-id insert in - // `intern_shape_descriptor` cannot, and deliberately does not). - super::invalidate_shape_lookup_cache(); - inner - .descriptors - .insert(id, super::box_descriptor(descriptor)); - // An equivalent local descriptor can predate module initialization. Keep - // both reverse-index entries and prefer the external id for subsequent - // births in this agent; already-published local ids remain resolvable. - inner.ids_by_facts.entry(facts).or_default().insert(0, id); - super::insert_descriptor_id_sorted(inner.ids_by_keys.entry(descriptor.keys).or_default(), id); + // id canonical for subsequent births in this agent: it goes to the FRONT + // of its accelerator bucket, which is the order exact-facts interning + // walks. + // SAFETY: no slab reference is held; `slab().get` above returned a copy. + unsafe { table.slab_mut().insert(id, record) }; + inner.facts_push_front(record.facts_key_with_keys(keys), id); + inner.family_push_front(keys, id); true } @@ -573,9 +560,10 @@ pub(super) fn install_external_shape_id( /// descriptor holding the edge, the slot visitor writes the record directly /// and there is nothing left to reconcile. /// -/// The returned address belongs to a BOXED record, so it is stable across -/// descriptor insertion; only `prune_dead_shape_keys` frees one, and that runs -/// at sweep, after every enumeration of the cycle that produced it. +/// The returned address belongs to a slab record, so it is stable across +/// descriptor insertion; a record is only cleared by the table's own +/// retirement paths, and its chunk released at the end of a major +/// collection, after every enumeration of the cycle that produced it. #[cfg(test)] #[inline] pub(crate) fn shape_descriptor_keys_slot(shape_id: u32) -> Option<*mut u64> { @@ -584,11 +572,9 @@ pub(crate) fn shape_descriptor_keys_slot(shape_id: u32) -> Option<*mut u64> { } crate::state::state() .shapes - .inner - .borrow_mut() - .descriptors - .get_mut(&shape_id) - .map(|record| std::ptr::addr_of_mut!(record.keys)) + .slab() + .record_ptr(shape_id) + .map(|record| record as *mut u64) } /// Is `slot` the shared `keys` word of `shape_id`'s descriptor record? @@ -604,15 +590,14 @@ pub(crate) fn shape_id_owns_keys_slot(shape_id: u32, slot: *mut u64) -> bool { if !super::is_shape_id(shape_id) { return false; } - // Immutable borrow on purpose: this runs inside collector walks, and a - // `borrow_mut` here would make the predicate itself a re-entrancy hazard. + // No table borrow at all: this runs inside collector walks, and a + // `RefCell` borrow here would make the predicate itself a re-entrancy + // hazard. The slab is read through a raw pointer. crate::state::state() .shapes - .inner - .borrow() - .descriptors - .get(&shape_id) - .is_some_and(|record| std::ptr::addr_of!(record.keys) as *mut u64 == slot) + .slab() + .record_ptr(shape_id) + .is_some_and(|record| record as *mut u64 == slot) } #[cfg(test)] diff --git a/crates/perry-runtime/src/object/shapes_store.rs b/crates/perry-runtime/src/object/shapes_store.rs new file mode 100644 index 0000000000..b305741f0f --- /dev/null +++ b/crates/perry-runtime/src/object/shapes_store.rs @@ -0,0 +1,776 @@ +//! Storage for the agent-local shape descriptor table (#9706). +//! +//! Two structures, both owned by [`super::ShapeTable`]: +//! +//! * [`ShapeSlab`] — the by-id store. A ShapeId is a process-global monotonic +//! counter (`SHAPE_ID_BASE + n`), so `n` indexes a chunked slab directly: no +//! hash, no per-record heap allocation, and a record address that never +//! moves for the record's lifetime — the property the collector relies on +//! when it enumerates a descriptor's `keys` word as a rewritable slot +//! (#8112) and retains that address across budgeted resumptions. Chunks +//! (32 records) hang off a two-level page directory, are allocated lazily +//! (a worker's ids interleave with the main thread's), and an all-dead chunk +//! is released by [`ShapeSlab::release_empty_chunks`] at the same cadence as +//! the reverse-index shrink (once per major collection). +//! +//! * [`IdList`] — the value of the per-keys-address family index +//! (`ShapeTableInner::families`). One entry per keys array names every +//! descriptor id currently indexed under that address. Exact-facts interning +//! walks the family and compares the remaining facts against the slab +//! record, which is what lets the table drop the second, facts-keyed reverse +//! map it used to carry: a family is small by construction — a SHARED keys +//! array is immutable, so its descriptors differ only in the birth bound or +//! a semantic generation, and an OWNED array retires its growth history +//! eagerly (`retire_owned_shape_siblings`). +//! +//! Measured on the compiled claude-code TUI at idle (`PERRY_GC_CENSUS`), the +//! previous layout — a `PtrHashMap>` beside two +//! `Vec`-valued reverse maps — cost ~330 bytes per live descriptor: +//! a 56-byte record in a 64-byte allocator bin, a 16-byte map entry at 25% +//! load after `shrink_to(2 * len)`, a 57-byte facts-map bucket, and a 33-byte +//! keys-map bucket, plus a 16-byte `Vec` buffer per reverse entry. A packed +//! 32-byte slab record with one 24-byte family bucket per keys array is the +//! same information at a fraction of the bytes. + +use super::{ShapeDescriptor, ShapeObjectKind, SHAPE_ID_BASE}; +use std::cell::UnsafeCell; + +pub(super) const RECORD_FLAG_PRESENT: u8 = 1 << 0; +pub(super) const RECORD_FLAG_FACTS_INDEXED: u8 = 1 << 1; +pub(super) const RECORD_FLAG_OLD_CARRIER: u8 = 1 << 2; +pub(super) const RECORD_FLAG_OLD_CARRIER_SEEN: u8 = 1 << 3; +pub(super) const RECORD_FLAG_CACHE_CARRIER: u8 = 1 << 4; +pub(super) const RECORD_FLAG_KIND_CLASS: u8 = 1 << 5; + +/// The table-owned record of one ShapeId. `keys` is first and 8-aligned: it +/// is the word the collector marks through and rewrites in place. +#[repr(C)] +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(crate) struct ShapeRecord { + /// Raw ArrayHeader address in Perry's fixed-width heap-word ABI (0 for a + /// keyless shape). + pub(super) keys: u64, + pub(super) semantic_generation: u64, + pub(super) logical_key_count: u32, + pub(super) live_inline_slot_count: u32, + pub(super) hole_count: u32, + pub(super) flags: u8, + _pad: [u8; 3], +} + +const _: () = assert!(std::mem::size_of::() == 32); +const _: () = assert!(std::mem::align_of::() == 8); + +impl ShapeRecord { + const EMPTY: ShapeRecord = ShapeRecord { + keys: 0, + semantic_generation: 0, + logical_key_count: 0, + live_inline_slot_count: 0, + hole_count: 0, + flags: 0, + _pad: [0; 3], + }; + + #[inline] + pub(super) fn present(&self) -> bool { + self.flags & RECORD_FLAG_PRESENT != 0 + } + + #[inline] + pub(super) fn has(&self, flag: u8) -> bool { + self.flags & flag != 0 + } + + #[inline] + pub(super) fn set(&mut self, flag: u8, on: bool) { + if on { + self.flags |= flag; + } else { + self.flags &= !flag; + } + } + + #[inline] + pub(super) fn object_kind(&self) -> ShapeObjectKind { + if self.has(RECORD_FLAG_KIND_CLASS) { + ShapeObjectKind::Class + } else { + ShapeObjectKind::Ordinary + } + } + + /// A fresh, facts-indexed record with every liveness bit clear. + pub(super) fn new( + keys: u64, + logical_key_count: u32, + live_inline_slot_count: u32, + semantic_generation: u64, + object_kind: ShapeObjectKind, + hole_count: u32, + ) -> ShapeRecord { + let mut flags = RECORD_FLAG_PRESENT | RECORD_FLAG_FACTS_INDEXED; + if object_kind == ShapeObjectKind::Class { + flags |= RECORD_FLAG_KIND_CLASS; + } + ShapeRecord { + keys, + semantic_generation, + logical_key_count, + live_inline_slot_count, + hole_count, + flags, + _pad: [0; 3], + } + } + + /// Exact-facts identity test (#8067): keys edge, both counts, generation, + /// kind, tombstones. Liveness bits and the facts-indexed bit are storage + /// state, never identity. + #[inline] + pub(super) fn facts_match( + &self, + keys: u64, + logical_key_count: u32, + live_inline_slot_count: u32, + semantic_generation: u64, + object_kind: ShapeObjectKind, + hole_count: u32, + ) -> bool { + self.keys == keys + && self.logical_key_count == logical_key_count + && self.live_inline_slot_count == live_inline_slot_count + && self.semantic_generation == semantic_generation + && self.hole_count == hole_count + && self.object_kind() == object_kind + } + + /// The 64-bit fold of the six identity facts, with `keys` supplied by + /// the caller: the collector rewrites a record's `keys` in place, so the + /// address the record was INDEXED under (its family key) is what the + /// exact-facts accelerator must be probed with until the metadata scan + /// re-indexes it. + #[inline] + pub(super) fn facts_key_with_keys(&self, keys: u64) -> u64 { + facts_key( + keys, + self.logical_key_count, + self.live_inline_slot_count, + self.semantic_generation, + self.object_kind(), + self.hole_count, + ) + } + + /// Copy the record out as the by-value [`ShapeDescriptor`] the rest of the + /// runtime consumes. `record` is the slab address of THIS record, which is + /// what `keys_slot()` and the tombstone fast paths hand back to the table. + #[inline] + pub(super) fn lift(&self, record: *mut ShapeRecord) -> ShapeDescriptor { + ShapeDescriptor { + keys: self.keys, + record: record as usize, + old_carrier: self.has(RECORD_FLAG_OLD_CARRIER), + cache_carrier: self.has(RECORD_FLAG_CACHE_CARRIER), + logical_key_count: self.logical_key_count, + live_inline_slot_count: self.live_inline_slot_count, + semantic_generation: self.semantic_generation, + object_kind: self.object_kind(), + hole_count: self.hole_count, + } + } +} + +/// FNV-1a fold of the six identity facts into the single word the +/// exact-facts accelerator is keyed by. Every field reaches the accumulator +/// (fold, never overwrite — the property `PtrHasher` lacks and the reason the +/// old `ShapeFacts` map could not use it); a 64-bit collision between two +/// live shapes is resolved by the per-hit `facts_match` on the record, so a +/// collision only costs a second record read, never a wrong answer. +#[inline] +pub(super) fn facts_key( + keys: u64, + logical_key_count: u32, + live_inline_slot_count: u32, + semantic_generation: u64, + object_kind: ShapeObjectKind, + hole_count: u32, +) -> u64 { + const FNV_OFFSET_BASIS: u64 = 0xcbf2_9ce4_8422_2325; + const FNV_PRIME: u64 = 0x0000_0100_0000_01b3; + let fold = |acc: u64, word: u64| (acc ^ word).wrapping_mul(FNV_PRIME); + let mut h = fold(FNV_OFFSET_BASIS, keys); + h = fold(h, u64::from(logical_key_count)); + h = fold(h, u64::from(live_inline_slot_count)); + h = fold(h, semantic_generation); + h = fold(h, u64::from(hole_count)); + h = fold(h, u64::from(object_kind == ShapeObjectKind::Class)); + // Final avalanche: FNV keeps most of its entropy in the high bits and + // hashbrown's probe sequence starts from the LOW bits. + h ^ (h >> 32) +} + +/// Records per chunk. Ids are minted far faster than they survive — the +/// compiled claude-code TUI mints ~1.05 M ShapeIds during startup and keeps +/// ~44 k, scattered over the whole range — so a chunk is deliberately SMALL +/// (32 records, 1 KB): an all-dead chunk is released whole, and the smaller +/// the chunk the less of a survivor's neighbourhood it drags along. Measured +/// on that TUI, 256-record chunks held 7.15 MB for those 44 k records and +/// 32-record chunks 4.0 MB. +const CHUNK_SHIFT: usize = 5; +const CHUNK_LEN: usize = 1 << CHUNK_SHIFT; +const CHUNK_MASK: usize = CHUNK_LEN - 1; + +/// Chunk pointers per directory page. The directory is two-level so its +/// size follows the LIVE id range, not the minted one: a long-running server +/// minting a billion ids over its life would otherwise carry a flat +/// `Vec>` of 250 MB at 32 records per chunk. A page is 8 KB and +/// covers 32 K ids; a page whose chunks have all been released is dropped. +const PAGE_SHIFT: usize = 10; +const PAGE_LEN: usize = 1 << PAGE_SHIFT; +const PAGE_MASK: usize = PAGE_LEN - 1; + +/// One lazily allocated run of `CHUNK_LEN` consecutive ids. The cells give +/// the table interior mutability through a shared slab reference: the +/// collector writes liveness bits and the `keys` word through raw record +/// pointers while other code holds only copies (`ShapeDescriptor`). +type Chunk = Box<[UnsafeCell; CHUNK_LEN]>; + +/// One directory page: `PAGE_LEN` chunk slots. +type Page = Box<[Option; PAGE_LEN]>; + +fn new_chunk() -> Chunk { + let mut v: Vec> = Vec::with_capacity(CHUNK_LEN); + v.resize_with(CHUNK_LEN, || UnsafeCell::new(ShapeRecord::EMPTY)); + // Exact length by construction; the conversion moves the allocation. + v.into_boxed_slice() + .try_into() + .unwrap_or_else(|_| unreachable!("chunk vector has CHUNK_LEN cells")) +} + +fn new_page() -> Page { + let mut v: Vec> = Vec::with_capacity(PAGE_LEN); + v.resize_with(PAGE_LEN, || None); + v.into_boxed_slice() + .try_into() + .unwrap_or_else(|_| unreachable!("page vector has PAGE_LEN slots")) +} + +/// The by-id descriptor store. See the module docs. +pub(crate) struct ShapeSlab { + pages: Vec>, + /// Present records. + len: usize, +} + +impl ShapeSlab { + pub(super) fn new() -> Self { + ShapeSlab { + pages: Vec::new(), + len: 0, + } + } + + #[inline] + fn index_of(id: u32) -> Option { + super::is_shape_id(id).then(|| (id - SHAPE_ID_BASE) as usize) + } + + #[inline] + fn id_of(index: usize) -> u32 { + SHAPE_ID_BASE + index as u32 + } + + /// `(page, chunk within page, record within chunk)` of a slab index. + #[inline] + fn split(index: usize) -> (usize, usize, usize) { + ( + index >> (CHUNK_SHIFT + PAGE_SHIFT), + (index >> CHUNK_SHIFT) & PAGE_MASK, + index & CHUNK_MASK, + ) + } + + /// Present records. + #[inline] + pub(super) fn len(&self) -> usize { + self.len + } + + /// The record for `id`, or `None` when the id names no descriptor in this + /// agent. The pointer stays valid until the record is removed; a removal + /// only ever happens through the table's own retirement paths. + #[inline] + pub(super) fn record_ptr(&self, id: u32) -> Option<*mut ShapeRecord> { + let index = Self::index_of(id)?; + let (page, chunk, slot) = Self::split(index); + let chunk = self.pages.get(page)?.as_ref()?[chunk].as_ref()?; + let cell = chunk[slot].get(); + // SAFETY: the cell belongs to a live chunk owned by this slab; reads + // and writes are serialized by the single-threaded agent discipline + // every other shape-table access already relies on. + if unsafe { (*cell).present() } { + Some(cell) + } else { + None + } + } + + /// A copy of the record for `id`. + #[inline] + pub(super) fn get(&self, id: u32) -> Option { + // SAFETY: `record_ptr` only returns a cell of a live chunk. + self.record_ptr(id).map(|p| unsafe { *p }) + } + + /// Lift `id` to the by-value descriptor. + #[inline] + pub(super) fn lift(&self, id: u32) -> Option { + // SAFETY: as in `get`. + self.record_ptr(id).map(|p| unsafe { (*p).lift(p) }) + } + + /// Install `record` under `id`, allocating the page and chunk on first + /// touch. Returns the record it replaced, if the id was already present. + pub(super) fn insert(&mut self, id: u32, mut record: ShapeRecord) -> Option { + let index = Self::index_of(id).expect("ShapeSlab::insert: id outside the ShapeId range"); + record.flags |= RECORD_FLAG_PRESENT; + let (page, chunk, slot) = Self::split(index); + if page >= self.pages.len() { + self.pages.resize_with(page + 1, || None); + } + let page = self.pages[page].get_or_insert_with(new_page); + let chunk = page[chunk].get_or_insert_with(new_chunk); + let cell = chunk[slot].get_mut(); + let previous = cell.present().then_some(*cell); + *cell = record; + if previous.is_none() { + self.len += 1; + } + previous + } + + /// Clear the record under `id`, returning it if it was present. + pub(super) fn remove(&mut self, id: u32) -> Option { + let index = Self::index_of(id)?; + let (page, chunk, slot) = Self::split(index); + let chunk = self.pages.get_mut(page)?.as_mut()?[chunk].as_mut()?; + let cell = chunk[slot].get_mut(); + if !cell.present() { + return None; + } + let previous = *cell; + *cell = ShapeRecord::EMPTY; + self.len -= 1; + Some(previous) + } + + /// Visit every present record in id order. The callback may write + /// through the record pointer; it must not insert or remove. + pub(super) fn for_each(&self, mut f: impl FnMut(u32, *mut ShapeRecord)) { + for (page_index, page) in self.pages.iter().enumerate() { + let Some(page) = page else { + continue; + }; + for (chunk_index, chunk) in page.iter().enumerate() { + let Some(chunk) = chunk else { + continue; + }; + let base = ((page_index << PAGE_SHIFT) | chunk_index) << CHUNK_SHIFT; + for (slot, cell) in chunk.iter().enumerate() { + let p = cell.get(); + // SAFETY: live chunk, single-threaded agent. + if unsafe { (*p).present() } { + f(Self::id_of(base | slot), p); + } + } + } + } + } + + /// Every present id, in id order. + #[cfg(test)] + pub(super) fn ids(&self) -> Vec { + let mut ids = Vec::with_capacity(self.len); + self.for_each(|id, _| ids.push(id)); + ids + } + + /// Free chunks that hold no present record, and pages that hold no + /// chunk. Called once per major collection, after dead-key pruning: + /// retirement is monotonic in id order for the common workload, so the + /// oldest chunks empty first. + pub(super) fn release_empty_chunks(&mut self) { + for page in self.pages.iter_mut() { + let Some(chunks) = page.as_mut() else { + continue; + }; + let mut live_chunks = 0usize; + for chunk in chunks.iter_mut() { + let empty = chunk + .as_ref() + .is_some_and(|c| c.iter().all(|cell| !unsafe { (*cell.get()).present() })); + if empty { + *chunk = None; + } + if chunk.is_some() { + live_chunks += 1; + } + } + if live_chunks == 0 { + *page = None; + } + } + while self.pages.last().is_some_and(Option::is_none) { + self.pages.pop(); + } + self.pages.shrink_to_fit(); + } + + #[cfg(test)] + pub(super) fn clear(&mut self) { + self.pages.clear(); + self.len = 0; + } + + /// Bytes held: the page directory, every allocated page and every + /// allocated chunk. + pub(super) fn estimated_bytes(&self) -> usize { + let mut pages = 0usize; + let mut chunks = 0usize; + for page in self.pages.iter().flatten() { + pages += 1; + chunks += page.iter().filter(|c| c.is_some()).count(); + } + self.pages.capacity() * std::mem::size_of::>() + + pages * PAGE_LEN * std::mem::size_of::>() + + chunks * CHUNK_LEN * std::mem::size_of::() + } + + /// Allocated chunks (diagnostics). + #[cfg(test)] + pub(super) fn chunk_count(&self) -> usize { + self.pages + .iter() + .flatten() + .map(|page| page.iter().filter(|c| c.is_some()).count()) + .sum() + } +} + +/// A compact list of descriptor ids: up to three inline, then a spilled +/// `Vec`. Sized so a family-index bucket is `(u64, IdList)` = 24 bytes. +/// +/// Order is meaningful: [`IdList::push_front`] is how an installed +/// process-global id becomes the canonical answer for exact-facts interning +/// ahead of an equivalent local id (`install_external_shape_id`). +#[derive(Clone, Debug)] +pub(super) enum IdList { + Inline { + len: u8, + ids: [u32; 3], + }, + // The `Box` is the point: an inline `Vec` is 24 bytes and would make every + // bucket 32; the spill is the rare case, so its extra indirection is + // cheaper than eight bytes on every family. + #[allow(clippy::box_collection)] + Spill(Box>), +} + +const _: () = assert!(std::mem::size_of::() == 16); + +impl Default for IdList { + fn default() -> Self { + IdList::Inline { + len: 0, + ids: [0; 3], + } + } +} + +impl IdList { + #[inline] + pub(super) fn as_slice(&self) -> &[u32] { + match self { + IdList::Inline { len, ids } => &ids[..*len as usize], + IdList::Spill(v) => v.as_slice(), + } + } + + #[inline] + pub(super) fn len(&self) -> usize { + self.as_slice().len() + } + + #[inline] + pub(super) fn is_empty(&self) -> bool { + self.len() == 0 + } + + #[inline] + pub(super) fn contains(&self, id: u32) -> bool { + self.as_slice().contains(&id) + } + + fn spill(&mut self) -> &mut Vec { + if let IdList::Inline { len, ids } = self { + let v = ids[..*len as usize].to_vec(); + *self = IdList::Spill(Box::new(v)); + } + match self { + IdList::Spill(v) => v, + IdList::Inline { .. } => unreachable!(), + } + } + + /// Append `id` unless already present. + pub(super) fn push_back(&mut self, id: u32) { + if self.contains(id) { + return; + } + match self { + IdList::Inline { len, ids } if (*len as usize) < ids.len() => { + ids[*len as usize] = id; + *len += 1; + } + _ => self.spill().push(id), + } + } + + /// Prepend `id` unless already present. + pub(super) fn push_front(&mut self, id: u32) { + if self.contains(id) { + return; + } + match self { + IdList::Inline { len, ids } if (*len as usize) < ids.len() => { + ids.copy_within(0..*len as usize, 1); + ids[0] = id; + *len += 1; + } + _ => self.spill().insert(0, id), + } + } + + /// Drop `id` if present; returns whether it was. + pub(super) fn remove(&mut self, id: u32) -> bool { + match self { + IdList::Inline { len, ids } => { + let n = *len as usize; + let Some(pos) = ids[..n].iter().position(|&x| x == id) else { + return false; + }; + ids.copy_within(pos + 1..n, pos); + ids[n - 1] = 0; + *len -= 1; + true + } + IdList::Spill(v) => { + let Some(pos) = v.iter().position(|&x| x == id) else { + return false; + }; + v.remove(pos); + true + } + } + } + + /// Replace `old` with `new` in place (keeps its position); returns + /// whether `old` was present. + pub(super) fn replace(&mut self, old: u32, new: u32) -> bool { + match self { + IdList::Inline { len, ids } => { + let n = *len as usize; + match ids[..n].iter().position(|&x| x == old) { + Some(pos) => { + ids[pos] = new; + true + } + None => false, + } + } + IdList::Spill(v) => match v.iter().position(|&x| x == old) { + Some(pos) => { + v[pos] = new; + true + } + None => false, + }, + } + } + + /// Bytes held outside the containing bucket. + pub(super) fn heap_bytes(&self) -> usize { + match self { + IdList::Inline { .. } => 0, + IdList::Spill(v) => std::mem::size_of::>() + v.capacity() * 4, + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn slab_records_are_addressed_by_id_and_keep_their_address() { + let mut slab = ShapeSlab::new(); + let id_a = SHAPE_ID_BASE + 5; + let id_b = SHAPE_ID_BASE + 5 + (CHUNK_LEN * PAGE_LEN) as u32 * 3; + assert_eq!(slab.get(id_a), None); + assert_eq!( + slab.insert( + id_a, + ShapeRecord::new(0x1000, 1, 1, 0, ShapeObjectKind::Ordinary, 0) + ) + .map(|r| r.keys), + None + ); + let a_ptr = slab.record_ptr(id_a).expect("present"); + // A later insert into another chunk must not move the first record. + slab.insert( + id_b, + ShapeRecord::new(0x2000, 2, 2, 7, ShapeObjectKind::Class, 1), + ); + assert_eq!(slab.record_ptr(id_a), Some(a_ptr)); + assert_eq!(slab.len(), 2); + assert_eq!(slab.chunk_count(), 2); + let b = slab.get(id_b).unwrap(); + assert_eq!(b.object_kind(), ShapeObjectKind::Class); + assert_eq!(b.semantic_generation, 7); + assert_eq!(b.hole_count, 1); + assert!(b.facts_match(0x2000, 2, 2, 7, ShapeObjectKind::Class, 1)); + assert!(!b.facts_match(0x2000, 2, 2, 7, ShapeObjectKind::Ordinary, 1)); + // Ids outside the range and never-minted ids resolve to nothing. + assert_eq!(slab.get(0), None); + assert_eq!(slab.get(SHAPE_ID_BASE + 6), None); + assert_eq!(slab.get(super::super::SHAPE_ID_END - 1), None); + assert_eq!(slab.ids(), vec![id_a, id_b]); + // Removal clears the record and, once a chunk is empty, the chunk. + assert_eq!(slab.remove(id_a).map(|r| r.keys), Some(0x1000)); + assert_eq!(slab.remove(id_a), None); + assert_eq!(slab.len(), 1); + slab.release_empty_chunks(); + assert_eq!(slab.chunk_count(), 1); + assert_eq!(slab.get(id_b).map(|r| r.keys), Some(0x2000)); + assert_eq!(slab.remove(id_b).map(|r| r.keys), Some(0x2000)); + slab.release_empty_chunks(); + assert_eq!(slab.chunk_count(), 0); + assert_eq!(slab.estimated_bytes(), 0); + } + + #[test] + fn lifted_descriptor_mirrors_the_record_and_names_its_address() { + let mut slab = ShapeSlab::new(); + let id = SHAPE_ID_BASE + 42; + let mut record = ShapeRecord::new(0x3000, 4, 6, 9, ShapeObjectKind::Ordinary, 2); + record.set(RECORD_FLAG_OLD_CARRIER, true); + record.set(RECORD_FLAG_CACHE_CARRIER, true); + record.set(RECORD_FLAG_FACTS_INDEXED, false); + slab.insert(id, record); + let ptr = slab.record_ptr(id).unwrap(); + let lifted = slab.lift(id).unwrap(); + assert_eq!(lifted.record, ptr as usize); + assert_eq!(lifted.keys, 0x3000); + assert_eq!(lifted.logical_key_count, 4); + assert_eq!(lifted.live_inline_slot_count, 6); + assert_eq!(lifted.semantic_generation, 9); + assert_eq!(lifted.hole_count, 2); + assert!(lifted.old_carrier); + assert!(lifted.cache_carrier); + assert!(!slab.get(id).unwrap().has(RECORD_FLAG_FACTS_INDEXED)); + assert_eq!(lifted.keys_slot(), Some(ptr as *mut u64)); + // Writing through the slot is what an evacuating visitor does. + unsafe { *lifted.keys_slot().unwrap() = 0x4000 }; + assert_eq!(slab.get(id).unwrap().keys, 0x4000); + } + + /// Varying any ONE fact must change the key: a fold that dropped a field + /// would send two different shapes to one bucket for every value of it. + #[test] + fn facts_key_folds_every_field() { + let base = facts_key(0x1111_2222_3333_4444, 7, 3, 9, ShapeObjectKind::Ordinary, 0); + let variants = [ + ( + "keys", + facts_key(0x5555_6666_7777_8888, 7, 3, 9, ShapeObjectKind::Ordinary, 0), + ), + ( + "logical", + facts_key(0x1111_2222_3333_4444, 8, 3, 9, ShapeObjectKind::Ordinary, 0), + ), + ( + "live", + facts_key(0x1111_2222_3333_4444, 7, 4, 9, ShapeObjectKind::Ordinary, 0), + ), + ( + "generation", + facts_key( + 0x1111_2222_3333_4444, + 7, + 3, + 10, + ShapeObjectKind::Ordinary, + 0, + ), + ), + ( + "kind", + facts_key(0x1111_2222_3333_4444, 7, 3, 9, ShapeObjectKind::Class, 0), + ), + ( + "holes", + facts_key(0x1111_2222_3333_4444, 7, 3, 9, ShapeObjectKind::Ordinary, 1), + ), + ]; + for (field, key) in variants { + assert_ne!( + key, base, + "changing `{field}` alone must change the facts key" + ); + } + let record = ShapeRecord::new(0x1111_2222_3333_4444, 7, 3, 9, ShapeObjectKind::Ordinary, 0); + assert_eq!(record.facts_key_with_keys(0x1111_2222_3333_4444), base); + assert_eq!( + record.facts_key_with_keys(0x5555_6666_7777_8888), + variants[0].1 + ); + } + + #[test] + fn id_list_keeps_order_across_the_inline_to_spill_boundary() { + let mut list = IdList::default(); + assert!(list.is_empty()); + list.push_back(2); + list.push_back(3); + list.push_front(1); + list.push_back(2); // duplicate ignored + assert_eq!(list.as_slice(), &[1, 2, 3]); + assert!(matches!(list, IdList::Inline { .. })); + list.push_back(4); + assert!(matches!(list, IdList::Spill(_))); + assert_eq!(list.as_slice(), &[1, 2, 3, 4]); + list.push_front(0); + assert_eq!(list.as_slice(), &[0, 1, 2, 3, 4]); + assert!(list.remove(2)); + assert!(!list.remove(2)); + assert_eq!(list.as_slice(), &[0, 1, 3, 4]); + assert!(list.replace(3, 30)); + assert!(!list.replace(3, 300)); + assert_eq!(list.as_slice(), &[0, 1, 30, 4]); + assert!(list.heap_bytes() >= 4 * 4); + + let mut inline = IdList::default(); + inline.push_back(7); + inline.push_back(8); + inline.push_back(9); + assert!(inline.remove(8)); + assert_eq!(inline.as_slice(), &[7, 9]); + assert!(inline.replace(9, 10)); + assert_eq!(inline.as_slice(), &[7, 10]); + assert!(inline.remove(7)); + assert!(inline.remove(10)); + assert!(inline.is_empty()); + assert_eq!(inline.heap_bytes(), 0); + } +} diff --git a/crates/perry-runtime/src/object/shapes_test_support.rs b/crates/perry-runtime/src/object/shapes_test_support.rs index f012820f5b..311bb982e5 100644 --- a/crates/perry-runtime/src/object/shapes_test_support.rs +++ b/crates/perry-runtime/src/object/shapes_test_support.rs @@ -1,8 +1,8 @@ //! Test-only shape-table helpers, in a sibling file. //! -//! Extracted from `shapes.rs` to keep it under the repo's 2000-line cap; the -//! lookup-way cache pushed it over. A child module, so these keep reaching the -//! parent's private items through `super::`. Moved verbatim. +//! Extracted from `shapes.rs` to keep it under the repo's 2000-line cap. A +//! child module, so these keep reaching the parent's private items through +//! `super::`. use super::*; @@ -73,25 +73,18 @@ pub(crate) fn test_shape_entry_exists(keys_id: usize) -> bool { #[cfg(test)] pub(crate) fn test_shape_descriptor_count() -> usize { - crate::state::state() - .shapes - .inner - .borrow() - .descriptors - .len() + crate::state::state().shapes.slab().len() } #[cfg(test)] pub(crate) fn test_clear_shape_table() { - let mut inner = crate::state::state().shapes.inner.borrow_mut(); - // Every descriptor box is about to be dropped, so every cached way naming - // one has to stop matching. Without this the cache holds dangling - // `Box` addresses and the next hit derefs freed memory. - invalidate_shape_lookup_cache(); + let table = &crate::state::state().shapes; + let mut inner = table.inner.borrow_mut(); inner.indices.clear(); - inner.descriptors.clear(); - inner.ids_by_facts.clear(); - inner.ids_by_keys.clear(); + inner.by_facts.clear(); + inner.families.clear(); + // SAFETY: test-only reset with no slab reference held. + unsafe { table.slab_mut().clear() }; drop(inner); clear_shape_object_kind_cache(); } @@ -99,15 +92,49 @@ pub(crate) fn test_clear_shape_table() { #[cfg(test)] pub(crate) fn test_drop_shape_descriptors(keys_id: usize) { let mut inner = crate::state::state().shapes.inner.borrow_mut(); - let stale = inner - .ids_by_keys - .remove(&(keys_id as u64)) + let stale: Vec = inner + .families + .get(&(keys_id as u64)) + .map(|ids| ids.as_slice().to_vec()) .unwrap_or_default(); for id in stale { remove_descriptor_and_reverse_indices(&mut inner, id); } } +/// Move the family indexed under `old` to `new`, exactly as the metadata +/// scan does after the collector forwarded that keys array. +#[cfg(test)] +pub(crate) fn test_rekey_shape_family(old: usize, new: usize) { + let table = &crate::state::state().shapes; + let mut inner = table.inner.borrow_mut(); + if let Some(ids) = inner.families.remove(&(old as u64)) { + for &id in ids.as_slice() { + let Some(record) = table.slab().get(id) else { + continue; + }; + if record.has(shapes_store::RECORD_FLAG_FACTS_INDEXED) { + inner.facts_remove(record.facts_key_with_keys(old as u64), id); + inner.facts_push_back(record.facts_key_with_keys(new as u64), id); + } + inner.family_push_back(new as u64, id); + } + } +} + +/// The ids currently indexed under `keys_id`, in family order. +#[cfg(test)] +pub(crate) fn test_shape_ids_for_keys(keys_id: usize) -> Vec { + crate::state::state() + .shapes + .inner + .borrow() + .families + .get(&(keys_id as u64)) + .map(|ids| ids.as_slice().to_vec()) + .unwrap_or_default() +} + #[cfg(test)] pub(crate) fn test_seed_shape_entry(keys_id: usize) { crate::state::state() @@ -128,9 +155,5 @@ pub(crate) fn test_seed_shape_entry(keys_id: usize) { #[cfg(test)] pub(crate) fn test_shape_id_for_keys(keys_id: usize) -> Option { - let inner = crate::state::state().shapes.inner.borrow(); - inner - .ids_by_keys - .get(&(keys_id as u64)) - .and_then(|ids| ids.first().copied()) + test_shape_ids_for_keys(keys_id).first().copied() } diff --git a/crates/perry-runtime/src/object/shapes_tests.rs b/crates/perry-runtime/src/object/shapes_tests.rs index ac708f34d1..43475708a2 100644 --- a/crates/perry-runtime/src/object/shapes_tests.rs +++ b/crates/perry-runtime/src/object/shapes_tests.rs @@ -540,7 +540,11 @@ mod descriptor_tests_8067 { external, "the process-global id should be preferred for later births" ); - retain_key_count_versions(keys as u64); + assert_eq!( + test_shape_ids_for_keys(keys), + vec![external, local], + "the external id must lead the family so interning prefers it" + ); assert!(shape_descriptor_by_id(local).is_some()); assert!(shape_descriptor_by_id(external).is_some()); @@ -638,9 +642,9 @@ mod descriptor_tests_8067 { ); assert_eq!(unsafe { *slot }, keys as u64); assert_eq!( - shape_descriptor_by_id(id).unwrap().indexed_keys, - keys as u64, - "newly minted descriptor must record its indexed keys address" + test_shape_ids_for_keys(keys), + vec![id], + "newly minted descriptor must be indexed under its keys address" ); // Writing THROUGH the slot is what an evacuating visitor does. The @@ -649,18 +653,20 @@ mod descriptor_tests_8067 { unsafe { *slot = moved_keys }; assert_eq!(shape_descriptor_by_id(id).unwrap().keys, moved_keys); assert_eq!( - shape_descriptor_by_id(id).unwrap().indexed_keys, - keys as u64, - "an object-edge rewrite must retain the old indexed address until metadata repair" + test_shape_ids_for_keys(keys), + vec![id], + "an object-edge rewrite must leave the family under the old address until metadata repair" + ); + assert!( + test_shape_ids_for_keys(moved_keys as usize).is_empty(), + "the store alone must not re-index the family" ); - // The keys-address reverse index is repaired incrementally by the - // metadata pass, not by the store; force the same one-id repair here. - { - let mut inner = crate::state::state().shapes.inner.borrow_mut(); - sync_descriptor_reverse_indices(&mut inner, id); - } - assert_eq!(shape_descriptor_by_id(id).unwrap().indexed_keys, moved_keys); + // The keys-address family index is repaired by the metadata pass, not + // by the store; force the same one-family repair here. + test_rekey_shape_family(keys, moved_keys as usize); + assert_eq!(test_shape_ids_for_keys(moved_keys as usize), vec![id]); + assert!(test_shape_ids_for_keys(keys).is_empty()); assert_eq!( shape_descriptor_ensure(moved_keys as *const ArrayHeader, 3, 2), Ok(id), @@ -670,7 +676,7 @@ mod descriptor_tests_8067 { .expect("shape range unexpectedly exhausted"); assert_ne!( old_address_id, id, - "incremental repair must remove the stale old-address facts entry" + "incremental repair must remove the stale old-address family entry" ); test_drop_shape_descriptors(moved_keys as usize); assert_eq!( @@ -679,15 +685,17 @@ mod descriptor_tests_8067 { "descriptor rekey did not update the keys-address index" ); test_drop_shape_descriptors(keys); + test_drop_shape_descriptors(keys); } #[test] fn a_boxed_record_keeps_its_keys_slot_across_table_growth() { let _lock = crate::gc::global_side_table_test_lock(); // The prohibition #8067 recorded — "descriptor insertion can reallocate - // the table" — is what BOXING answers. Mint one descriptor, take its - // slot, then mint enough siblings to force several rehashes and assert - // the address never moved. Without the box this fails. + // the table" — is what a stable-address record store answers (a Box + // per record before #9706, a chunked slab since). Mint one descriptor, + // take its slot, then mint enough siblings to grow the store across + // several chunks and assert the address never moved. let keys = 0x8112_0000_0000_1000usize; let id = shape_descriptor_ensure(keys as *const ArrayHeader, 1, 1) .expect("shape range unexpectedly exhausted"); @@ -714,8 +722,11 @@ mod descriptor_tests_8067 { } } + /// #9706: an OWNED keys array's growth history is retired behind the + /// version its single owner now carries, except for a version an + /// optimization cache permanently owns. #[test] - fn key_count_versions_remain_resolvable_until_the_keys_die() { + fn owned_key_count_versions_are_retired_behind_the_current_one() { let _lock = crate::gc::global_side_table_test_lock(); let keys = 0x8067_0000_0000_2100usize; let unrelated_keys = 0x8067_0000_0000_2200usize; @@ -723,29 +734,83 @@ mod descriptor_tests_8067 { .expect("shape range unexpectedly exhausted"); let stale_b = shape_descriptor_ensure(keys as *const ArrayHeader, 1, 2) .expect("shape range unexpectedly exhausted"); + let cached = shape_descriptor_ensure(keys as *const ArrayHeader, 1, 3) + .expect("shape range unexpectedly exhausted"); let current = shape_descriptor_ensure(keys as *const ArrayHeader, 2, 2) .expect("shape range unexpectedly exhausted"); let unrelated = shape_descriptor_ensure(unrelated_keys as *const ArrayHeader, 1, 1) .expect("shape range unexpectedly exhausted"); + // Before retirement every version is resolvable and the family lists + // them in mint order. + assert_eq!( + test_shape_ids_for_keys(keys), + vec![stale_a, stale_b, cached, current] + ); + unsafe { note_cache_carrier(shape_descriptor_by_id(cached)) }; - retain_key_count_versions(keys as u64); + retire_owned_shape_siblings(keys as u64, current); - assert!(shape_descriptor_by_id(stale_a).is_some()); - assert!(shape_descriptor_by_id(stale_b).is_some()); + assert_eq!(shape_descriptor_by_id(stale_a), None); + assert_eq!(shape_descriptor_by_id(stale_b), None); + assert!( + shape_descriptor_by_id(cached).is_some(), + "a cache-carried version must survive same-address retirement" + ); assert!(shape_descriptor_by_id(current).is_some()); assert!(shape_descriptor_by_id(unrelated).is_some()); - let inner = crate::state::state().shapes.inner.borrow(); - let current_ids = inner - .ids_by_keys - .get(&(keys as u64)) - .expect("keys identity disappeared from descriptor index"); - assert_eq!(current_ids.as_slice(), &[stale_a, stale_b, current]); - drop(inner); + assert_eq!(test_shape_ids_for_keys(keys), vec![cached, current]); + // Retired facts re-intern as FRESH ids: nothing can resolve the old ones. + let reminted = shape_descriptor_ensure(keys as *const ArrayHeader, 1, 1) + .expect("shape range unexpectedly exhausted"); + assert_ne!(reminted, stale_a); test_drop_shape_descriptors(keys); test_drop_shape_descriptors(unrelated_keys); } + /// The retirement above is wired to the publish funnel: an in-place + /// append on an OWNED keys array must leave exactly one structural + /// descriptor under that address. + #[test] + fn in_place_owned_append_leaves_one_descriptor_per_keys_address() { + let _lock = crate::gc::global_side_table_test_lock(); + unsafe { + let obj = crate::object::js_object_alloc(0, 0); + let mut keys_before = 0usize; + let mut first_addr_count = 0usize; + for i in 0..96u32 { + let name = format!("owned9706_{i:03}"); + let key = crate::string::js_string_from_bytes(name.as_ptr(), name.len() as u32); + crate::object::js_object_set_field_by_name(obj, key, i as f64); + let keys = crate::object::object_keys_array(obj) as usize; + let stamp = object_shape_stamp(obj); + assert!(is_shape_id(stamp), "receiver must stay stamped"); + let family = test_shape_ids_for_keys(keys); + assert!( + family.contains(&stamp), + "the current stamp must be indexed under the current keys address" + ); + if keys == keys_before { + first_addr_count += 1; + let shared = crate::value::addr_class::try_read_gc_header(keys) + .is_some_and(|h| h.gc_flags & crate::gc::GC_FLAG_SHAPE_SHARED != 0); + if !shared { + assert_eq!( + family.len(), + 1, + "an owned in-place append left growth history alive: {family:?}" + ); + } + } + keys_before = keys; + } + assert!( + first_addr_count > 0, + "fixture premise: some appends must grow the owned array in place" + ); + } + } + #[test] fn shape_drop_does_not_delete_a_potential_siblings_descriptor() { let _lock = crate::gc::global_side_table_test_lock(); @@ -836,23 +901,35 @@ mod descriptor_tests_8067 { } } -/// `ids_by_facts` moved from std's SipHash `RandomState` to `FastKeyHasher`. +/// A multi-field shape key hashed with `FastKeyHasher` must fold every field. /// -/// The hazard that motivated the original "deliberately NOT a `PtrHashMap`" -/// note is real: `PtrHasher`'s `write_*` methods OVERWRITE the accumulator, so -/// a five-field `ShapeFacts` would collapse to its last field and every -/// descriptor sharing that field would collide into one bucket. +/// The shape table's facts-keyed reverse map is gone (#9706 interns through +/// the keys-address family instead), but the hazard this pinned is still +/// live for `gc/layout/typed_shape.rs`'s `RegisteredTypedShapeKey`: +/// `PtrHasher`'s `write_*` methods OVERWRITE the accumulator, so a multi-field +/// key would collapse to its last field and every entry sharing that field +/// would collide into one bucket. /// /// `FastKeyHasher` avoids this by implementing only `write` — the derived /// `Hash`'s `write_u32`/`write_u64` calls all forward there and FOLD with -/// FNV-1a. This test pins that property directly: vary ONE field at a time and -/// require a distinct hash each time. It fails loudly against any hasher that -/// overwrites instead of folding. +/// FNV-1a. This test pins that property directly on the old facts layout: +/// vary ONE field at a time and require a distinct hash each time. It fails +/// loudly against any hasher that overwrites instead of folding. #[test] fn shape_facts_hash_folds_every_field() { use crate::fast_hash::FastKeyHasher; use std::hash::{BuildHasher, Hash, Hasher}; + #[derive(Clone, Copy, Hash)] + struct ShapeFacts { + keys: u64, + logical_key_count: u32, + live_inline_slot_count: u32, + semantic_generation: u64, + object_kind: ShapeObjectKind, + hole_count: u32, + } + fn h(f: &ShapeFacts) -> u64 { let mut hasher = FastKeyHasher.build_hasher(); f.hash(&mut hasher); @@ -928,14 +1005,12 @@ fn shape_facts_hash_folds_every_field() { assert_eq!(h(&base), h(&base.clone()), "hashing must be deterministic"); } -/// The shape lookup cache holds a record's ADDRESS, so it must stop matching -/// the moment that address can change under an id still in use. +/// A removed id must stop resolving at once, and nothing may hand out its +/// record address afterwards. /// -/// A stale way would hand out a pointer to a dropped `Box` — -/// a use-after-free reachable from the hot property path, not a wrong answer. -/// Removal is the funnel that frees a record, so it bumps the epoch; this pins -/// that. Deleting the `invalidate_shape_lookup_cache()` call in -/// `remove_descriptor_and_reverse_indices` fails this test. +/// Before #9706 this pinned the lookup-way cache's invalidation epoch; the +/// slab has no cache in front of it, so the property is asserted directly: +/// removal clears the record and both by-id entry points report `None`. #[test] fn shape_lookup_cache_is_invalidated_when_a_record_is_removed() { let _lock = crate::gc::global_side_table_test_lock(); @@ -945,35 +1020,38 @@ fn shape_lookup_cache_is_invalidated_when_a_record_is_removed() { let id = test_shape_id_for_keys(keys as usize) .expect("a fresh object must have a registered shape"); - // Populate the way. assert!( shape_descriptor_by_id(id).is_some(), "the descriptor must resolve before removal" ); - let epoch_before = crate::state::state().shapes.lookup_epoch.get(); + let record = shape_descriptor_by_id(id).unwrap().record; + assert_ne!(record, 0); - // Drop it through the funnel that frees the box. + // Drop it through the funnel that retires a record. { let mut inner = crate::state::state().shapes.inner.borrow_mut(); remove_descriptor_and_reverse_indices(&mut inner, id); } - assert_ne!( - crate::state::state().shapes.lookup_epoch.get(), - epoch_before, - "removing a record must bump the lookup epoch — a way still naming \ - the freed box would hand out a dangling ShapeDescriptor pointer" - ); assert!( shape_descriptor_by_id(id).is_none(), - "a removed id must not resolve from the cache" + "a removed id must not resolve" + ); + assert_eq!( + shape_live_inline_slot_count_by_id(id), + None, + "the field reader must not read a retired record" + ); + assert_eq!(shape_descriptor_keys_slot(id), None); + assert!( + !shape_id_owns_keys_slot(id, record as *mut u64), + "a retired id must not claim its old record address" ); } } -/// A fresh-id insert must NOT invalidate the cache: it cannot make any existing -/// way wrong, and flushing on every shape creation would defeat the cache in -/// exactly the workloads that build shapes. +/// Minting fresh ids must not move any existing record: the collector may +/// hold a record address across the mint. #[test] fn fresh_shape_creation_does_not_flush_the_lookup_cache() { let _lock = crate::gc::global_side_table_test_lock(); @@ -981,8 +1059,8 @@ fn fresh_shape_creation_does_not_flush_the_lookup_cache() { let a = crate::object::js_object_alloc(0, 0); let keys_a = crate::object::object_keys_array(a); let id_a = test_shape_id_for_keys(keys_a as usize).expect("shape for a"); - assert!(shape_descriptor_by_id(id_a).is_some()); - let epoch = crate::state::state().shapes.lookup_epoch.get(); + let record_a = shape_descriptor_by_id(id_a).expect("resolves").record; + assert_ne!(record_a, 0); // Create more objects — each mints shapes through the fresh-id path. for _ in 0..8 { @@ -991,14 +1069,9 @@ fn fresh_shape_creation_does_not_flush_the_lookup_cache() { } assert_eq!( - crate::state::state().shapes.lookup_epoch.get(), - epoch, - "minting fresh shape ids must not bump the epoch; only removal and \ - the replacing insert may" - ); - assert!( - shape_descriptor_by_id(id_a).is_some(), - "the earlier descriptor must still resolve" + shape_descriptor_by_id(id_a).map(|d| d.record), + Some(record_a), + "minting fresh shape ids must not move an existing record" ); } } diff --git a/scripts/shape_descriptor_census.py b/scripts/shape_descriptor_census.py index a3bcaa6fe2..511863bfea 100644 --- a/scripts/shape_descriptor_census.py +++ b/scripts/shape_descriptor_census.py @@ -223,6 +223,7 @@ def assert_authority_surfaces(sources: dict[str, str]) -> None: authority_paths = ( "crates/perry-runtime/src/object/shapes.rs", "crates/perry-runtime/src/object/shapes_slot_list.rs", + "crates/perry-runtime/src/object/shapes_store.rs", "crates/perry-runtime/src/object/mod.rs", "crates/perry-runtime/src/object/live_slots.rs", "crates/perry-codegen/src/lower_call/new_alloc.rs", @@ -248,15 +249,18 @@ def assert_authority_surfaces(sources: dict[str, str]) -> None: ) clean = stripped_sources({path: sources[path] for path in authority_paths}) # `shapes.rs` sits against the repo's 2000-line cap, so helpers keep being - # split into the `shapes_slot_list.rs` sibling as it grows. Read the two as - # ONE logical unit: every `function_body(shapes, ...)` below then finds its - # target wherever it currently lives, instead of silently matching nothing - # the next time a pinned function crosses the split — #8918's exact failure - # mode, where a census inspecting an empty body reports success. + # split into siblings as it grows (`shapes_slot_list.rs`, and since #9706 + # the record store `shapes_store.rs`). Read them as ONE logical unit: every + # `function_body(shapes, ...)` below then finds its target wherever it + # currently lives, instead of silently matching nothing the next time a + # pinned function crosses the split — #8918's exact failure mode, where a + # census inspecting an empty body reports success. shapes = ( clean["crates/perry-runtime/src/object/shapes.rs"] + "\n" + clean["crates/perry-runtime/src/object/shapes_slot_list.rs"] + + "\n" + + clean["crates/perry-runtime/src/object/shapes_store.rs"] ) object_mod = clean["crates/perry-runtime/src/object/mod.rs"] live_slots = clean["crates/perry-runtime/src/object/live_slots.rs"] @@ -301,13 +305,21 @@ def assert_authority_surfaces(sources: dict[str, str]) -> None: raw_write_pics = sources["crates/perry-codegen/src/expr/proxy_reflect.rs"] for pattern, label in ( - # `PtrHashMap` since #8157 (SipHash on a bare u32 was 25% of self time in - # `shapes`). The hasher is free; the BOX is not. Since #8112 the - # collector enumerates `&mut record.keys` as an ordinary GC slot, and a - # budgeted dirty scan can hold that address across mutator resumptions - # that insert descriptors. Un-boxing the value puts the record back in - # the bucket, where a rehash moves it under the collector's feet. - (r"descriptors\s*:\s*(?:[\w:]+::)?(?:Ptr)?HashMap\s*<\s*u32\s*,\s*Box\s*<\s*ShapeDescriptor\s*>", "by-id descriptor table, boxed for a stable keys slot"), + # #9706: the by-id store is a chunked slab indexed by ShapeId. Since + # #8112 the collector enumerates the record's `keys` word as an + # ordinary GC slot, and a budgeted dirty scan can hold that address + # across mutator resumptions that insert descriptors — so a record's + # address must never move for its lifetime. Chunks are individually + # boxed and never reallocated; only the directory of chunk pointers + # grows. Putting records into one flat `Vec` (or back into a rehashing + # bucket) moves them under the collector's feet. + (r"slab\s*:\s*(?:std::cell::)?UnsafeCell\s*<\s*ShapeSlab\s*>", "by-id descriptor slab with stable record addresses"), + (r"type\s+Chunk\s*=\s*Box\s*<\s*\[\s*UnsafeCell\s*<\s*ShapeRecord\s*>\s*;\s*CHUNK_LEN\s*\]\s*>", "slab chunks individually boxed, never reallocated"), + (r"type\s+Page\s*=\s*Box\s*<\s*\[\s*Option\s*<\s*Chunk\s*>\s*;\s*PAGE_LEN\s*\]\s*>", "slab directory pages hold chunk pointers, not records"), + (r"pages\s*:\s*Vec\s*<\s*Option\s*<\s*Page\s*>\s*>", "slab directory is a vector of page pointers"), + # `keys` must stay the FIRST field of the `#[repr(C)]` record: the + # record address IS the rewritable keys slot (`keys_slot`). + (r"#\[repr\(C\)\]\s*(?:#\[[^\]]*\]\s*)*pub\(crate\)\s+struct\s+ShapeRecord\s*\{\s*(?://[^\n]*\n\s*)*pub\(super\)\s+keys\s*:\s*u64", "slab record is repr(C) with the keys word first"), (r"logical_key_count\s*:\s*u32", "exact logical-key fact"), (r"live_inline_slot_count\s*:\s*u32", "exact live-slot fact"), (r"semantic_generation\s*:\s*u64", "semantic transition fact"), @@ -393,8 +405,8 @@ def assert_authority_surfaces(sources: dict[str, str]) -> None: ensure = function_body(shapes, "shape_descriptor_ensure_with_holes") assert_before( ensure, - "inner.descriptors.insert", - "inner.ids_by_facts.entry", + "slab_mut().insert", + "family_push_back", "by-id descriptor before reverse accelerator", ) sync = function_body(shapes, "publish_object_shape_from") @@ -433,18 +445,37 @@ def assert_authority_surfaces(sources: dict[str, str]) -> None: shapes, ): raise CensusError("clear_object_shape_stamp escaped its #[cfg(test)] gate") - retirement = function_body(shapes, "retain_key_count_versions") + # #9706: an OWNED keys array's same-address growth history is retired + # behind the version its single owner now carries. The retirement must be + # scoped to that array's family (never a scan of the whole table), must + # keep the cache-carried versions an optimization cache can reinstall, + # and must run AFTER the successor is stamped and armed (#9200's order). + retirement = function_body(shapes, "retire_owned_shape_siblings") require_code( retirement, - r"ids_by_keys\s*\.\s*remove\s*\(\s*&keys\s*\)", + r"families\s*\.\s*get\s*\(\s*&keys\s*\)", "keys-scoped descriptor lineage index", ) - if re.search(r"descriptors\s*\.\s*(?:iter|values|keys)\s*\(", retirement): - raise CensusError("shape descriptor lineage repair scans the global descriptor table") - if "descriptors.remove" in retirement: - raise CensusError("live-key lineage repair eagerly deletes published descriptors") + if re.search(r"slab\(\)\s*\.\s*for_each\s*\(", retirement): + raise CensusError("owned-history retirement scans the global descriptor table") + require_code( + retirement, + r"RECORD_FLAG_CACHE_CARRIER", + "cache-carried versions survive same-address retirement", + ) + assert_before( + sync, + "stamp_object_shape_id_with_carrier_note", + "retire_owned_shape_siblings", + "successor stamped and armed before the owned history is retired", + ) + # A SHARED array's versions are immutable prefixes other objects may still + # carry; growth and drop of the slot index must not touch them. for name in ("shape_keys_grown", "shape_drop"): - if "descriptors.remove" in function_body(shapes, name): + if re.search( + r"remove_descriptor_(?:and_reverse_indices|indexed_under)", + function_body(shapes, name), + ): raise CensusError(f"{name} eagerly deletes a sibling descriptor") require_code( @@ -747,15 +778,26 @@ def run_sabotage_selftests(sources: dict[str, str], baseline: dict[str, object]) ) shapes_path = "crates/perry-runtime/src/object/shapes.rs" - unboxed_table = dict(sources) - unboxed_table[shapes_path] = unboxed_table[shapes_path].replace( - "PtrHashMap>", - "PtrHashMap", + store_path = "crates/perry-runtime/src/object/shapes_store.rs" + flat_slab = dict(sources) + flat_slab[store_path] = flat_slab[store_path].replace( + "type Chunk = Box<[UnsafeCell; CHUNK_LEN]>;", + "type Chunk = Vec>;", 1, ) expect_rejected( - "descriptor record un-boxed back into a rehashing bucket", - lambda: assert_authority_surfaces(unboxed_table), + "slab chunk turned into a reallocating Vec", + lambda: assert_authority_surfaces(flat_slab), + ) + keys_not_first = dict(sources) + keys_not_first[store_path] = keys_not_first[store_path].replace( + " pub(super) keys: u64,\n pub(super) semantic_generation: u64,", + " pub(super) semantic_generation: u64,\n pub(super) keys: u64,", + 1, + ) + expect_rejected( + "keys word moved off the front of the slab record", + lambda: assert_authority_surfaces(keys_not_first), ) ungated_root = dict(sources) @@ -804,11 +846,11 @@ def run_sabotage_selftests(sources: dict[str, str], baseline: dict[str, object]) unscoped_retirement = dict(sources) path = "crates/perry-runtime/src/object/shapes.rs" retirement_body = function_body( - unscoped_retirement[path], "retain_key_count_versions" + unscoped_retirement[path], "retire_owned_shape_siblings" ) unscoped_body, substitutions = re.subn( - r"ids_by_keys\s*\.\s*remove\s*\(\s*&keys\s*\)", - "ids_by_keys.get(&keys).cloned()", + r"families\s*\.\s*get\s*\(\s*&keys\s*\)", + "families.get(&0)", retirement_body, count=1, ) @@ -822,6 +864,19 @@ def run_sabotage_selftests(sources: dict[str, str], baseline: dict[str, object]) lambda: assert_authority_surfaces(unscoped_retirement), ) + early_retirement = dict(sources) + publish_body = function_body(early_retirement[path], "publish_object_shape_from") + early_body = swap_once( + publish_body, + "stamp_object_shape_id_with_carrier_note", + "retire_owned_shape_siblings", + ) + early_retirement[path] = early_retirement[path].replace(publish_body, early_body, 1) + expect_rejected( + "owned history retired before the successor is stamped", + lambda: assert_authority_surfaces(early_retirement), + ) + legacy_ir = dict(sources) path = "crates/perry-codegen/src/expr/property_get/generic_dispatch.rs" legacy_body, substitutions = re.subn( From a5c3b13a5048e14ea82d4a0cdf97ff7c1e3ff6ee Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ralph=20K=C3=BCpper?= Date: Fri, 4 Sep 2026 13:52:23 +0000 Subject: [PATCH 2/3] fix(gc): retire uncarried shape descriptors (#9726) --- crates/perry-runtime/src/gc/cycle.rs | 5 +- crates/perry-runtime/src/gc/dead_owner.rs | 18 +- .../perry-runtime/src/gc/layout_slot_visit.rs | 6 + crates/perry-runtime/src/gc/oldgen.rs | 11 +- .../src/gc/tests/dead_owner_side_tables.rs | 4 +- .../gc/tests/shape_keys_descriptor_edge.rs | 182 +++++++++++++++++- crates/perry-runtime/src/object/mod.rs | 9 + .../src/object/shape_carriers.rs | 83 ++++++++ crates/perry-runtime/src/object/shapes.rs | 75 ++++++-- .../src/object/shapes_slot_list.rs | 13 +- .../perry-runtime/src/object/shapes_store.rs | 11 +- 11 files changed, 392 insertions(+), 25 deletions(-) create mode 100644 crates/perry-runtime/src/object/shape_carriers.rs diff --git a/crates/perry-runtime/src/gc/cycle.rs b/crates/perry-runtime/src/gc/cycle.rs index 7e4a8ce01d..106efed33f 100644 --- a/crates/perry-runtime/src/gc/cycle.rs +++ b/crates/perry-runtime/src/gc/cycle.rs @@ -1710,7 +1710,10 @@ impl GcCycleState { // per-object finalizers. Minor traces never mark the old // generation, so deadness there is only trusted for // untenured nursery headers. - .with_dead_collection_finalize(full_trace), + .with_dead_collection_finalize( + full_trace, + full_trace && !self.progress_kind.is_budgeted(), + ), ); } let done = self diff --git a/crates/perry-runtime/src/gc/dead_owner.rs b/crates/perry-runtime/src/gc/dead_owner.rs index 35e5688de4..2c617264af 100644 --- a/crates/perry-runtime/src/gc/dead_owner.rs +++ b/crates/perry-runtime/src/gc/dead_owner.rs @@ -203,17 +203,27 @@ fn owner_type_matches(header: &GcHeader, expected_obj_type: Option) -> bool /// Post-trace fan-out (full mark-sweep + fallback minor). Runs at sweep /// entry, before any header is finalized or freed, so deadness probes read /// intact headers. -pub(super) fn prune_dead_owner_side_tables_post_trace(full_trace: bool) { +pub(super) fn prune_dead_owner_side_tables_post_trace( + full_trace: bool, + synchronous_full_trace: bool, +) { + debug_assert!(!synchronous_full_trace || full_trace); if full_trace { + // Rebuild every restamping table's ownership before consulting the + // complete receiver census. An evicted cache entry releases its id in + // this same post-trace window. + crate::object::shape_carriers::recompute_after_full_trace(); + if synchronous_full_trace { + crate::object::shapes::prune_uncarried_shape_descriptors_after_full_trace(); + } // #8112: a full trace enumerated every live object, so the old-carrier // notes it accumulated are exactly the shapes old objects still carry. // Adopting them here is what lets the gate SHED a shape — minors only // ever add notes, so without this the table's root set would grow // monotonically and no keys array would ever be reclaimed again. + // #9726: this also clears the all-generation carried note after the + // synchronous prune consumed it. Budgeted traces only clear the note. crate::object::shapes::rotate_old_carrier_epoch_after_full_trace(); - // The same rule for the array-tail transition caches: their carrier - // bits are exact only when rebuilt from live occupancy. - crate::object::array_tail_transition::recompute_cache_carriers_after_full_trace(); } let probe = PostTraceProbe::new(full_trace); fan_out( diff --git a/crates/perry-runtime/src/gc/layout_slot_visit.rs b/crates/perry-runtime/src/gc/layout_slot_visit.rs index c86d890569..6673909569 100644 --- a/crates/perry-runtime/src/gc/layout_slot_visit.rs +++ b/crates/perry-runtime/src/gc/layout_slot_visit.rs @@ -51,6 +51,12 @@ pub(super) unsafe fn visit_gc_layout_slot_descriptors( // needed: without it the verifier aborts on a `slot_page_ever_dirty=false` // old→young edge through this word. let shape_keys_edge = if (*header).obj_type == GC_TYPE_OBJECT { + // #9726: unlike the minor-rooting gate below, full-trace descriptor + // liveness is generation-blind. Every reachable shaped receiver must + // note the exact id it carries before synchronous-full pruning. + if full_trace_active() { + crate::object::shapes::note_full_trace_carrier(child_slots.object_shape); + } // A receiver the minor will not enumerate for itself arms the table's // ephemeron gate. The test is "not in the nursery", not "in old-gen": // a `gc_malloc`'d large object and an immortal bootstrap resident are diff --git a/crates/perry-runtime/src/gc/oldgen.rs b/crates/perry-runtime/src/gc/oldgen.rs index 601fce3c07..3dea59757f 100644 --- a/crates/perry-runtime/src/gc/oldgen.rs +++ b/crates/perry-runtime/src/gc/oldgen.rs @@ -1200,12 +1200,19 @@ impl IncrementalSweepState { /// 2026-07-09 audit: buffers and typed arrays joined the same pattern — /// their registry/side-table entries are pruned when the owner is /// genuinely dead (full traces only; they are all tenured old residents). - pub(super) fn with_dead_collection_finalize(mut self, full_trace: bool) -> Self { + pub(super) fn with_dead_collection_finalize( + mut self, + full_trace: bool, + synchronous_full_trace: bool, + ) -> Self { // 2026-07-09 GC audit wave 2: death-prune the object-address-keyed // side tables in the same marks-fresh window. Cheap (one flag-check // walk over tables the root scanners already walk every cycle), so // it runs eagerly here rather than budget-chunked. - super::dead_owner::prune_dead_owner_side_tables_post_trace(full_trace); + super::dead_owner::prune_dead_owner_side_tables_post_trace( + full_trace, + synchronous_full_trace, + ); self.dead_maps = crate::map::collect_dead_registered_maps_post_trace(full_trace); self.dead_sets = crate::set::collect_dead_registered_sets_post_trace(full_trace); self.dead_buffers = crate::buffer::collect_dead_registered_buffers_post_trace(full_trace); diff --git a/crates/perry-runtime/src/gc/tests/dead_owner_side_tables.rs b/crates/perry-runtime/src/gc/tests/dead_owner_side_tables.rs index 78aca9929c..3b38346f5d 100644 --- a/crates/perry-runtime/src/gc/tests/dead_owner_side_tables.rs +++ b/crates/perry-runtime/src/gc/tests/dead_owner_side_tables.rs @@ -1578,7 +1578,9 @@ fn test_live_transition_cache_entry_survives_full_gc() { let next_keys = crate::arena::arena_alloc_gc_old(64, 8, GC_TYPE_ARRAY) as usize; let live_key = crate::arena::arena_alloc_gc(64, 8, GC_TYPE_STRING) as usize; - let prev_shape_id = crate::object::shapes::shape_id_for_keys_ensure(std::ptr::null(), 0); + // This predecessor can recur from a generated module global; without such + // an owner, retiring it and the now-unusable cache entry is intentional. + let prev_shape_id = crate::object::shapes::js_object_shape_id_for_keys(0, 0); js_shadow_slot_set(0, ptr_bits(next_keys)); js_shadow_slot_set(1, ptr_bits(live_key)); diff --git a/crates/perry-runtime/src/gc/tests/shape_keys_descriptor_edge.rs b/crates/perry-runtime/src/gc/tests/shape_keys_descriptor_edge.rs index 856a21dc62..12f011388a 100644 --- a/crates/perry-runtime/src/gc/tests/shape_keys_descriptor_edge.rs +++ b/crates/perry-runtime/src/gc/tests/shape_keys_descriptor_edge.rs @@ -24,7 +24,10 @@ //! alive by something else entirely". use super::super::*; -use super::support::{collect_minor_trace, init_test_closure, ptr_bits, CopyingNurseryTestGuard}; +use super::support::{ + collect_minor_trace, complete_budgeted_gc_cycle, init_test_closure, ptr_bits, + CopyingNurseryTestGuard, GcTriggerThresholdTestGuard, +}; use crate::object::shapes; /// Facts the assertions compare, read exclusively through the descriptor. @@ -488,3 +491,180 @@ fn metadata_rewrite_validates_the_post_visit_non_array_address() { ); shapes::test_clear_shape_table(); } + +fn collect_synchronous_full_trace() { + let _ = + gc_collect_full_mark_sweep_with_trigger(GcTriggerSnapshot::capture(GcTriggerKind::Direct)); +} + +fn build_unrooted_keyless_semantic_shape(slot: u32) -> u32 { + js_shadow_slot_set( + slot, + ptr_bits(crate::object::js_object_alloc(0, 0) as usize), + ); + let obj = (js_shadow_slot_get(slot) & POINTER_MASK) as *mut crate::ObjectHeader; + let predecessor = unsafe { shapes::object_shape_stamp(obj) }; + let shape_id = unsafe { shapes::transition_object_shape_semantics(obj) }; + assert_ne!(shape_id, predecessor, "semantic transition must mint an id"); + assert_ne!( + shapes::shape_descriptor_by_id(shape_id) + .expect("semantic descriptor") + .semantic_generation, + 0, + "test premise: this must be a per-object semantic generation" + ); + js_shadow_slot_set(slot, crate::value::TAG_UNDEFINED); + shape_id +} + +/// #9726: keyless semantic generations used to be immortal because dead-key +/// pruning asks about address zero, which is never a dead GC owner. A complete +/// receiver census must retire that descriptor while preserving the inverse: +/// the same kind of generation stays authoritative when its object is live. +#[test] +fn synchronous_full_trace_retires_only_uncarried_semantic_shapes() { + let _guard = CopyingNurseryTestGuard::new(2); + shapes::test_clear_shape_table(); + crate::arena::arena_reset_all_blocks_to_zero(); + gc_register_mutable_root_scanner(shapes::scan_shape_table_rekey_mut); + + let dead_shape = build_unrooted_keyless_semantic_shape(0); + collect_synchronous_full_trace(); + assert!( + shapes::shape_descriptor_by_id(dead_shape).is_none(), + "#9726: a keyless per-object generation with no live carrier survived a complete full trace" + ); + + build_two_key_object(0, b"e9726_live_"); + let live_before = (js_shadow_slot_get(0) & POINTER_MASK) as *mut crate::ObjectHeader; + let live_shape = unsafe { shapes::transition_object_shape_semantics(live_before) }; + assert_ne!( + shapes::shape_descriptor_by_id(live_shape) + .expect("live semantic descriptor before collection") + .semantic_generation, + 0 + ); + + collect_synchronous_full_trace(); + + let live_after = (js_shadow_slot_get(0) & POINTER_MASK) as *mut crate::ObjectHeader; + assert_eq!( + unsafe { shapes::object_shape_stamp(live_after) }, + live_shape + ); + assert!( + shapes::shape_descriptor_by_id(live_shape).is_some(), + "#9726/#9200: pruning must not leave a live receiver stamped with an unresolved id" + ); + let own_keys = crate::object::js_object_keys(live_after); + assert_eq!( + unsafe { (*own_keys).length }, + 2, + "#9726/#9200: Object.keys() lost the live receiver's descriptor facts" + ); + + js_shadow_slot_set(0, crate::value::TAG_UNDEFINED); + shapes::test_clear_shape_table(); +} + +/// Incremental full marking is sliced across mutator turns, so its receiver +/// notes are deliberately not an exact liveness census. It may rotate the +/// epoch, but it must leave uncarried retirement to a synchronous full trace. +#[test] +fn budgeted_full_trace_does_not_retire_from_a_partial_carrier_census() { + let _guard = CopyingNurseryTestGuard::new(1); + let _trigger_guard = GcTriggerThresholdTestGuard::suppress_automatic_triggers(); + shapes::test_clear_shape_table(); + crate::arena::arena_reset_all_blocks_to_zero(); + + let shape_id = build_unrooted_keyless_semantic_shape(0); + GC_OLD_RECLAIM_PENDING.with(|pending| pending.set(true)); + let mut first = JsGcStepResult::default(); + assert_eq!( + js_gc_step_work_units(1, &mut first), + JS_GC_STEP_STATUS_ACTIVE + ); + assert_eq!(first.collection_kind, GcCollectionKind::Full.ffi_code()); + let completed = complete_budgeted_gc_cycle(); + assert_eq!(completed.status, JS_GC_STEP_STATUS_COMPLETED); + assert!( + shapes::shape_descriptor_by_id(shape_id).is_some(), + "#9726: a budgeted trace must not retire from its partial carrier notes" + ); + + collect_synchronous_full_trace(); + assert!( + shapes::shape_descriptor_by_id(shape_id).is_none(), + "the next complete full trace must retire the same uncarried descriptor" + ); + shapes::test_clear_shape_table(); +} + +fn build_transition_cache_target_then_drop(slot: u32) -> (u32, u32) { + // Model the process-lifetime module global that can birth the predecessor + // again after no receiver currently carries it. + let predecessor = shapes::js_object_shape_id_for_keys(0, 0); + js_shadow_slot_set( + slot, + ptr_bits(crate::object::js_object_alloc(0, 0) as usize), + ); + let key = crate::string::js_string_from_bytes(b"cache6".as_ptr(), 6); + let obj = (js_shadow_slot_get(slot) & POINTER_MASK) as *mut crate::ObjectHeader; + assert_eq!(unsafe { shapes::object_shape_stamp(obj) }, predecessor); + crate::object::js_object_set_field_by_name(obj, key, 9726.0); + let obj = (js_shadow_slot_get(slot) & POINTER_MASK) as *mut crate::ObjectHeader; + let target = unsafe { shapes::object_shape_stamp(obj) }; + assert_ne!(target, predecessor); + js_shadow_slot_set(slot, crate::value::TAG_UNDEFINED); + (predecessor, target) +} + +/// The transition table is a real ShapeId publisher: generated write sites +/// read its target id and stamp it directly. Park a target with no receiver, +/// collect, then prove a later receiver can still take the cached transition. +#[test] +fn transition_cache_target_survives_and_can_restamp_after_full_trace() { + let _guard = CopyingNurseryTestGuard::new(1); + shapes::test_clear_shape_table(); + crate::arena::arena_reset_all_blocks_to_zero(); + gc_register_mutable_root_scanner(shapes::scan_shape_table_rekey_mut); + gc_register_mutable_root_scanner(crate::object::scan_transition_cache_roots_mut); + + let (predecessor, target) = build_transition_cache_target_then_drop(0); + collect_synchronous_full_trace(); + assert!( + shapes::shape_descriptor_by_id(predecessor).is_some(), + "a process-lifetime generated-code id must remain installed" + ); + let cached = shapes::shape_descriptor_by_id(target) + .expect("a live transition-cache entry must retain its target descriptor"); + assert!( + cached.cache_carrier, + "the transition target must own its id" + ); + + js_shadow_slot_set(0, ptr_bits(crate::object::js_object_alloc(0, 0) as usize)); + let key = crate::string::js_string_from_bytes(b"cache6".as_ptr(), 6); + let consumer = (js_shadow_slot_get(0) & POINTER_MASK) as *mut crate::ObjectHeader; + shapes::test_watch_cached_transition_stamps(consumer as usize); + crate::object::js_object_set_field_by_name(consumer, key, 26.0); + assert_eq!( + shapes::test_cached_transition_stamps(), + 1, + "the cache path must perform the stamp" + ); + let consumer = (js_shadow_slot_get(0) & POINTER_MASK) as *mut crate::ObjectHeader; + assert_eq!( + unsafe { shapes::object_shape_stamp(consumer) }, + target, + "the post-GC transition hit must stamp the retained target id" + ); + assert_eq!( + unsafe { (*crate::object::js_object_keys(consumer)).length }, + 1 + ); + + shapes::test_reset_cached_transition_stamps(); + js_shadow_slot_set(0, crate::value::TAG_UNDEFINED); + shapes::test_clear_shape_table(); +} diff --git a/crates/perry-runtime/src/object/mod.rs b/crates/perry-runtime/src/object/mod.rs index 2699829b45..3d8bc8563a 100644 --- a/crates/perry-runtime/src/object/mod.rs +++ b/crates/perry-runtime/src/object/mod.rs @@ -141,6 +141,7 @@ mod polymorphic_index_symbol_tests; mod primitive_proto_thunks; mod property_key; pub(crate) mod prototype_chain; +pub(crate) mod shape_carriers; pub(crate) mod shapes; pub(crate) use shapes::ShapeTable; mod prototype_helpers; @@ -687,6 +688,7 @@ fn shape_cache_insert(shape_id: u32, keys_array: *mut ArrayHeader) { .borrow_mut() .insert(shape_id, (keys_array, runtime_shape_id)); crate::gc::runtime_write_barrier_root_raw_ptr(keys_array); + shape_carriers::note_shape_id(runtime_shape_id); } /// Thread-local shape-transition cache for the dynamic-key write path @@ -988,6 +990,10 @@ fn transition_cache_lookup( return None; } } + // A weak, unstabilized entry must not publish a retired id. + if !shape_carriers::unstable_target_resolves(entry) { + return None; + } Some((entry.next_keys, entry_slot_idx, entry.target_shape_id)) } else { None @@ -1048,6 +1054,9 @@ fn transition_cache_insert( entry.slot_idx = slot_idx | (len_marker << 24); entry.target_len = target_len; }); + if target_len != 0 { + shape_carriers::note_shape_id(target_shape_id); + } if !array_tail_owner.is_null() { array_tail_transition::record_numeric_tail_transition( array_tail_owner, diff --git a/crates/perry-runtime/src/object/shape_carriers.rs b/crates/perry-runtime/src/object/shape_carriers.rs new file mode 100644 index 0000000000..4ceb57b2bd --- /dev/null +++ b/crates/perry-runtime/src/object/shape_carriers.rs @@ -0,0 +1,83 @@ +//! ShapeId owners that may stamp an id after its last receiver dies (#9726). +//! +//! The descriptor table is weak with respect to receivers, but these runtime +//! caches are active metadata owners. Their bits are set on insertion and +//! rebuilt from exact table occupancy after every full trace. Generated module +//! globals are process-lifetime owners and use the record's separate external +//! carrier bit instead. + +use super::*; + +#[inline] +pub(crate) fn note_shape_id(shape_id: u32) { + unsafe { shapes::note_cache_carrier(shapes::shape_descriptor_by_id(shape_id)) }; +} + +#[inline] +fn target_descriptor_resolves(entry: TransitionEntry, expected_len: u32) -> bool { + shapes::shape_descriptor_by_id(entry.target_shape_id).is_some_and(|descriptor| { + descriptor.keys == entry.next_keys as u64 && descriptor.logical_key_count == expected_len + }) +} + +/// The runtime-only transition path revalidates weak, unstabilized entries +/// before publishing their target id. Generated probes reject these entries. +#[inline] +pub(crate) fn unstable_target_resolves(entry: TransitionEntry) -> bool { + let expected_len = (entry.slot_idx & TRANSITION_SLOT_IDX_MASK).wrapping_add(1); + target_descriptor_resolves(entry, expected_len) +} + +/// A stabilized entry is a call-free generated-code publisher only while its +/// exact target array and descriptor facts still agree with the cached edge. +#[inline] +fn stable_target_resolves(entry: TransitionEntry) -> bool { + let expected_len = (entry.slot_idx & TRANSITION_SLOT_IDX_MASK).wrapping_add(1); + if entry.target_len != expected_len { + return false; + } + let Some(header) = + (unsafe { crate::value::addr_class::try_read_tracked_gc_header(entry.next_keys) }) + else { + return false; + }; + unsafe { + if (*header.as_ptr()).obj_type != crate::gc::GC_TYPE_ARRAY { + return false; + } + let keys = entry.next_keys as *const ArrayHeader; + if (*keys).length != expected_len || (*keys).length > (*keys).capacity { + return false; + } + } + target_descriptor_resolves(entry, expected_len) +} + +/// Rebuild transient cache ownership after a full trace. This runs before a +/// synchronous trace's uncarried-descriptor prune, so every surviving table +/// entry that can publish a ShapeId has already claimed it. +pub(crate) fn recompute_after_full_trace() { + // Clears every transient bit, then re-notes both directions of the + // Array-subclass cache. Permanent external owners use a different bit. + array_tail_transition::recompute_cache_carriers_after_full_trace(); + + with_transition_cache(|table| unsafe { + for entry in (*table).iter() { + if stable_target_resolves(*entry) { + note_shape_id(entry.target_shape_id); + } + } + }); + + let state = crate::state::state(); + unsafe { + for entry in (&*state.object_hot.shape_inline_cache.get()).iter() { + note_shape_id(entry.runtime_shape_id); + } + } + for &(keys, runtime_shape_id) in state.object_hot.shape_cache_overflow.borrow().values() { + if !keys.is_null() { + note_shape_id(runtime_shape_id); + } + } +} diff --git a/crates/perry-runtime/src/object/shapes.rs b/crates/perry-runtime/src/object/shapes.rs index 5ec883968c..049f72295f 100644 --- a/crates/perry-runtime/src/object/shapes.rs +++ b/crates/perry-runtime/src/object/shapes.rs @@ -48,8 +48,9 @@ pub(crate) use shapes_slot_list::{ try_update_stable_tombstone_shape, try_update_stable_tombstone_shape_cached, SlotList, }; use shapes_store::{ - IdList, ShapeRecord, ShapeSlab, RECORD_FLAG_CACHE_CARRIER, RECORD_FLAG_FACTS_INDEXED, - RECORD_FLAG_OLD_CARRIER, RECORD_FLAG_OLD_CARRIER_SEEN, + IdList, ShapeRecord, ShapeSlab, RECORD_FLAG_CACHE_CARRIER, RECORD_FLAG_CARRIED_SEEN, + RECORD_FLAG_EXTERNAL_CARRIER, RECORD_FLAG_FACTS_INDEXED, RECORD_FLAG_OLD_CARRIER, + RECORD_FLAG_OLD_CARRIER_SEEN, }; #[derive(Clone)] @@ -634,6 +635,29 @@ pub(crate) unsafe fn note_old_generation_carrier(descriptor: Option) { + let Some(descriptor) = descriptor else { + return; + }; + if descriptor.record != 0 { + (*(descriptor.record as *mut ShapeRecord)).set(RECORD_FLAG_CARRIED_SEEN, true); + } +} + +#[inline] +pub(crate) unsafe fn note_external_shape_carrier(descriptor: Option) { + let Some(descriptor) = descriptor else { + return; + }; + if descriptor.record != 0 { + (*(descriptor.record as *mut ShapeRecord)).set(RECORD_FLAG_EXTERNAL_CARRIER, true); + } +} + /// Retain a descriptor while an agent-local optimization cache can reinstall /// its ShapeId. Cache tables live with `RuntimeState`; the bit is recomputed /// from live table occupancy after every full trace @@ -700,10 +724,11 @@ pub(crate) fn clear_all_cache_carriers() { /// Recompute the old-carrier gate from the trace that just finished. /// /// A FULL trace enumerates every live object, so the notes it accumulated are -/// exactly the shapes old objects still carry; adopt them and clear the -/// accumulator. Minors only ever ADD notes, which is why the gate needs a full -/// trace to shed a shape whose last old carrier died — the same rule that -/// governs every other old-generation reclamation. +/// exactly the shapes old objects still carry; adopt them and clear both the +/// old-carrier accumulator and the all-generation carried note. The latter is +/// consumed by synchronous-full descriptor retirement immediately before this +/// rotation. Budgeted full cycles clear it without retiring because their +/// sliced trace is not a complete carrier census. pub(crate) fn rotate_old_carrier_epoch_after_full_trace() { crate::state::state().shapes.slab().for_each(|_, record| { // SAFETY: live slab record, single-threaded agent. @@ -711,6 +736,7 @@ pub(crate) fn rotate_old_carrier_epoch_after_full_trace() { let seen = (*record).has(RECORD_FLAG_OLD_CARRIER_SEEN); (*record).set(RECORD_FLAG_OLD_CARRIER, seen); (*record).set(RECORD_FLAG_OLD_CARRIER_SEEN, false); + (*record).set(RECORD_FLAG_CARRIED_SEEN, false); } }); } @@ -724,7 +750,10 @@ pub(crate) fn rotate_old_carrier_epoch_after_full_trace() { /// rooted keys global as an integer heap word on every target. #[no_mangle] pub extern "C" fn js_object_shape_id_for_keys(keys: u64, key_count: u32) -> u32 { - shape_id_for_keys_ensure(keys as usize as *const ArrayHeader, key_count) + let id = shape_id_for_keys_ensure(keys as usize as *const ArrayHeader, key_count); + // SAFETY: `id` was resolved from this agent's live slab record above. + unsafe { note_external_shape_carrier(shape_descriptor_by_id(id)) }; + id } /// Mint a process-global ShapeId for a codegen-registered typed layout and @@ -1323,10 +1352,9 @@ fn retire_owned_shape_siblings(keys: u64, keep: u32) { .copied() .filter(|&id| { id != keep - && table - .slab() - .get(id) - .is_some_and(|record| !record.has(RECORD_FLAG_CACHE_CARRIER)) + && table.slab().get(id).is_some_and(|record| { + !record.has(RECORD_FLAG_CACHE_CARRIER | RECORD_FLAG_EXTERNAL_CARRIER) + }) }) .collect() }) @@ -1753,6 +1781,29 @@ fn shape_keys_address_is_recycled(addr: usize) -> bool { } } +/// Retire descriptors no live receiver carried during the just-completed +/// synchronous full trace and no runtime metadata owner can reinstall. +/// +/// The caller must run this while the full trace's `CARRIED_SEEN` notes are +/// intact and only after cache-carrier bits have been rebuilt from live table +/// occupancy. Minor and budgeted cycles are deliberately ineligible: neither +/// provides an exact, stop-the-world enumeration of every live receiver. +pub(crate) fn prune_uncarried_shape_descriptors_after_full_trace() { + let table = &crate::state::state().shapes; + let mut inner = table.inner.borrow_mut(); + let mut stale = Vec::new(); + table.slab().for_each(|id, record| { + // SAFETY: live slab record, read immediately under agent ownership. + let record = unsafe { &*record }; + if !record.has(RECORD_FLAG_CARRIED_SEEN) && !record.cache_carrier() { + stale.push(id); + } + }); + for id in stale { + remove_descriptor_and_reverse_indices(&mut inner, id); + } +} + /// Post-trace weak-table prune: drop slot indices and by-id descriptors whose /// keys array is dead. A live object has already traced its authoritative /// header edge and synchronized the descriptor named by its ShapeId, so a @@ -2066,7 +2117,7 @@ pub(crate) fn shape_table_liveness_census( uncarried += 1; // SAFETY: live slab record, read immediately. let record = unsafe { *record }; - if record.has(RECORD_FLAG_CACHE_CARRIER) { + if record.cache_carrier() { uncarried_cache += 1; } else if record.has(RECORD_FLAG_OLD_CARRIER) { uncarried_old += 1; diff --git a/crates/perry-runtime/src/object/shapes_slot_list.rs b/crates/perry-runtime/src/object/shapes_slot_list.rs index 0523e4df43..2dd6d74518 100644 --- a/crates/perry-runtime/src/object/shapes_slot_list.rs +++ b/crates/perry-runtime/src/object/shapes_slot_list.rs @@ -517,7 +517,7 @@ pub(super) fn install_external_shape_id( return false; } let keys = keys as usize as u64; - let record = ShapeRecord::new( + let mut record = ShapeRecord::new( keys, logical_key_count, live_inline_slot_count, @@ -525,10 +525,12 @@ pub(super) fn install_external_shape_id( super::ShapeObjectKind::Ordinary, 0, ); + record.set(super::shapes_store::RECORD_FLAG_EXTERNAL_CARRIER, true); let table = &crate::state::state().shapes; let mut inner = table.inner.borrow_mut(); - if let Some(existing) = table.slab().get(id) { - return existing.facts_match( + if let Some(existing) = table.slab().record_ptr(id) { + // SAFETY: live slab record, single-threaded agent. + let matches = unsafe { &*existing }.facts_match( keys, logical_key_count, live_inline_slot_count, @@ -536,6 +538,11 @@ pub(super) fn install_external_shape_id( super::ShapeObjectKind::Ordinary, 0, ); + if matches { + // SAFETY: same record and agent discipline as above. + unsafe { (*existing).set(super::shapes_store::RECORD_FLAG_EXTERNAL_CARRIER, true) }; + } + return matches; } // A worker can have minted an equivalent local descriptor before module // initialization installs the process-global codegen id. Keep both id diff --git a/crates/perry-runtime/src/object/shapes_store.rs b/crates/perry-runtime/src/object/shapes_store.rs index b305741f0f..ec9d0b812a 100644 --- a/crates/perry-runtime/src/object/shapes_store.rs +++ b/crates/perry-runtime/src/object/shapes_store.rs @@ -41,6 +41,8 @@ pub(super) const RECORD_FLAG_OLD_CARRIER: u8 = 1 << 2; pub(super) const RECORD_FLAG_OLD_CARRIER_SEEN: u8 = 1 << 3; pub(super) const RECORD_FLAG_CACHE_CARRIER: u8 = 1 << 4; pub(super) const RECORD_FLAG_KIND_CLASS: u8 = 1 << 5; +pub(super) const RECORD_FLAG_CARRIED_SEEN: u8 = 1 << 6; +pub(super) const RECORD_FLAG_EXTERNAL_CARRIER: u8 = 1 << 7; /// The table-owned record of one ShapeId. `keys` is first and 8-aligned: it /// is the word the collector marks through and rewrites in place. @@ -91,6 +93,13 @@ impl ShapeRecord { } } + /// A runtime table or process-lifetime generated-code global may reinstall + /// this id even while no object currently carries it. + #[inline] + pub(super) fn cache_carrier(&self) -> bool { + self.has(RECORD_FLAG_CACHE_CARRIER | RECORD_FLAG_EXTERNAL_CARRIER) + } + #[inline] pub(super) fn object_kind(&self) -> ShapeObjectKind { if self.has(RECORD_FLAG_KIND_CLASS) { @@ -171,7 +180,7 @@ impl ShapeRecord { keys: self.keys, record: record as usize, old_carrier: self.has(RECORD_FLAG_OLD_CARRIER), - cache_carrier: self.has(RECORD_FLAG_CACHE_CARRIER), + cache_carrier: self.cache_carrier(), logical_key_count: self.logical_key_count, live_inline_slot_count: self.live_inline_slot_count, semantic_generation: self.semantic_generation, From 9a7ac196dba15649c3e0598046b26470604c5959 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ralph=20K=C3=BCpper?= Date: Fri, 4 Sep 2026 13:58:33 +0000 Subject: [PATCH 3/3] docs(changelog): record shape descriptor pruning --- changelog.d/9733-uncarried-shape-descriptors.md | 7 +++++++ 1 file changed, 7 insertions(+) create mode 100644 changelog.d/9733-uncarried-shape-descriptors.md diff --git a/changelog.d/9733-uncarried-shape-descriptors.md b/changelog.d/9733-uncarried-shape-descriptors.md new file mode 100644 index 0000000000..3f57ae5025 --- /dev/null +++ b/changelog.d/9733-uncarried-shape-descriptors.md @@ -0,0 +1,7 @@ +**Full collections now retire shape descriptors that no live object or +restamping cache owns.** A full trace records every shaped receiver, and a +synchronous sweep prunes only records absent from that complete census. +Minor and budgeted cycles remain conservative. Generated-module ids and +shape/transition caches retain exact ownership, while unstable transition +entries validate their target before stamping. In the claude-code census, +uncarried descriptors without an owner fell from 34,501 to zero.