diff --git a/crates/rusty_alloc/CHANGELOG.md b/crates/rusty_alloc/CHANGELOG.md index 804c651..f743bcd 100644 --- a/crates/rusty_alloc/CHANGELOG.md +++ b/crates/rusty_alloc/CHANGELOG.md @@ -7,6 +7,30 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Fixed + +- **The small profile extended every page ONE BLOCK AT A TIME, so 100 % of + allocations took the slow path.** `page_extend` bounds its batch at 4 KiB of + payload and computed that bound with a hardcoded shift whose constant term is + really `SEGMENT_SLICE_SIZE / 4096`. At the shipped 64 KiB slice the literal + was right; under `--cfg ra_small_profile` the slice is 4 KiB, so the bound + was **256 bytes instead of 4 KiB** — sixteen times too small. For a 512-byte + class the batch computed to 0 and was clamped to 1, leaving every page with + `capacity == 1` and no second block for the fast path to find. Counted over + 100,000 alloc+free pairs at the small profile, entries into `malloc_generic` + per op: **512 B 1.0000 -> 0.1250, 513 B 1.0000 -> 0.1667, 1 KiB 1.0000 -> + 0.2500**, with page carve-and-retire churn falling from 195 per 100,000 to + 24/32/49. On a 32-bit host the 512-vs-513 step inverts from +8.6 % (slower) + to −36 % (faster), about **1.9× faster at 512**. **Measured on silicon** too + — a XIAO ESP32-S3, `main` against the fix on one board with identical + checksums and floor: **13.0 % faster** on 32 B ping-pong, **15.7 %** on a + 64-block mixed batch, **15.3 %** on 8-512 B churn, and 1.1 % at 2,048 B, + which is on the bin route this does not touch. Found by the Kairos RTOS + report (`docs/plans/finished/fixed-prim-small-step.md` §8.7). + **The default geometry is unchanged** — the derived constant equals the old + literal there, and the all-features x86-64 assembly diff moves no executable + function. + ### Added - **`--cfg ra_generic_collect="64" | "4096" | "65536"`**, so a bare-metal diff --git a/crates/rusty_alloc/src/page.rs b/crates/rusty_alloc/src/page.rs index 651418a..fa10315 100644 --- a/crates/rusty_alloc/src/page.rs +++ b/crates/rusty_alloc/src/page.rs @@ -1029,6 +1029,17 @@ unsafe fn page_collect_impl(page: *mut Page, flag: usize) } } +/// The constant term of [`page_extend`]'s batch shift: `SEGMENT_SLICE_SIZE` +/// expressed in 4 KiB pages, as a shift. 4 at the shipped 64 KiB slice, 0 at +/// the small profile's 4 KiB one. See the note in `page_extend`. +const EXTEND_SHIFT_BASE: u32 = { + assert!( + crate::types::SEGMENT_SLICE_SIZE >= 4096, + "the extend bound assumes a slice of at least one 4 KiB page" + ); + crate::types::SEGMENT_SLICE_SIZE.trailing_zeros() - 12 +}; + /// Lazily extend the free list into never-used capacity (`mi_page_extend_free`). /// /// # Safety @@ -1130,7 +1141,24 @@ pub unsafe fn page_extend(page: *mut Page, area: *mut u8) { "extend bound assumes a power-of-two span, got {}", (*page).slice_count ); - let span_shift = 4 + (*page).slice_count.trailing_zeros(); + // The `16` in that identity is `SEGMENT_SLICE_SIZE / 4096`, so the + // shift's constant term is the GEOMETRY's, not a literal 4. + // + // DEFECT (found 2026-09-10, `docs/plans/finished/fixed-prim-small-step.md` + // §8.7): it was written as a literal `4`, which is right only for the + // shipped 64 KiB slice. Under `ra_small_profile` the slice is 4 KiB, so + // the bound this computes was 256 BYTES of payload rather than 4 KiB -- + // sixteen times too small. For a 512-byte class `reserved >> shift` is + // then 0, `.max(1)` rescues it to ONE BLOCK, and every page on the + // profile firmware actually uses was extended one block at a time: + // `capacity == 1` on every page, and `malloc_generic` on 100 % of + // allocations instead of one in eight. Measured on a host at the small + // profile before the fix: `generic` exactly 1.0000/op at 512, 640 and + // 1024 bytes. + // + // Derived, so it is correct at every geometry and byte-identical at the + // default (65536 >> 12 == 16, whose log2 is the old 4). + let span_shift = EXTEND_SHIFT_BASE + (*page).slice_count.trailing_zeros(); let take = ((reserved >> span_shift).max(1)).min(reserved - capacity); let start = area.add(capacity * bsize); // Link the fresh blocks in address order. diff --git a/docs/LEDGER.md b/docs/LEDGER.md index 611dac5..0a81053 100644 --- a/docs/LEDGER.md +++ b/docs/LEDGER.md @@ -48,6 +48,39 @@ slice from `dedicated_segments` and the sizing test goes red); the reproduction is a permanent property-based test against the real extent allocator. Unsafe +4, all `#[cfg(test)]` — the fix is arithmetic and adds none to shipped code. +## SMALL-PROFILE EXTEND — every page carved one block at a time; 100% slow path -> 12.5% (2026-09-10) + +`docs/plans/finished/fixed-prim-small-step.md` §8.6-8.7, pulled out of the +Kairos step report rather than reported directly. + +`page_extend` bounds its batch at 4 KiB of payload via +`span_shift = 4 + slice_count.trailing_zeros()`. The identity is +`4096/bsize == reserved / (slice_count * 16)` and the `16` is +`SEGMENT_SLICE_SIZE / 4096` -- so the literal `4` holds only at the 64 KiB +slice. Under `ra_small_profile` (4 KiB slice) the bound was **256 bytes**, the +batch for a 512 B class computed to 0, `.max(1)` clamped it to ONE, and every +page carried `capacity == 1`. A page with one block has no second block for the +fast path, which is why `generic` read exactly 1.0000/op at every binned size. + +**Found by tracing `collect_inner`'s reclaim** while chasing the reported +512-byte step: every reclaimed page printed `cap=1` with `resv=8`, and reserved +being right while capacity was 1 named the extend immediately. + +**Fixed** by deriving the term (`SEGMENT_SLICE_SIZE.trailing_zeros() - 12`). +Counted, 100k pairs, small profile: generic/op **1.0000 -> 0.1250** (512 B), +0.1667 (513), 0.2500 (1024); churn 195 -> 24/32/49 per 100k. Timing, 32-bit +host, ABBA, three reproductions: the 512-vs-513 step inverts +8.6% -> -36%, +and 512 absolute 390,200 -> ~200,000 ns, ~1.9x. Default geometry byte-identical +(the derived value IS the old literal; all-features asm moves one debug blob). + +**Measured on silicon** (XIAO ESP32-S3, main vs fix, one board, one session, +identical floor 166 ns and identical checksums): pingpong 32 B 595 -> 518 +(13.0%), batch 64-mixed 833 -> 702 (15.7%), churn 8-512 B 1,011 -> 856 (15.3%), +large 2,048 B 1,134 -> 1,121 (1.1%). 13-16% on every binned row, the same +magnitude as the reported step; 2,048 B barely moving is the tell, since it is +on the bin route this does not touch. Still open: the heap_4 A/B row is the +consumer's to re-run, and the bin route enters generic on every op even now. + ## SMALL-PATH STEP — not the prim, the POINTER WIDTH; the heartbeat knob bare metal could not reach (2026-09-10) `docs/plans/finished/fixed-prim-small-step.md`: the Kairos RTOS measured one diff --git a/docs/plans/finished/fixed-prim-small-step.md b/docs/plans/finished/fixed-prim-small-step.md index 321a429..3229c6b 100644 --- a/docs/plans/finished/fixed-prim-small-step.md +++ b/docs/plans/finished/fixed-prim-small-step.md @@ -1,6 +1,7 @@ # The small-path step — a 16% cliff at 512 bytes that only exists on bare metal -**Status:** RESOLVED 2026-09-10 — see §8. The attribution was wrong and the +**Status:** RESOLVED 2026-09-10 — see §8, and §8.7 for the DEFECT the +report led to: the small profile extended every page one block at a time. The attribution was wrong and the correction is the finding. Originally: measured, nothing implemented · **Date:** 2026-09-10 · **Build:** `=2.1.0` from crates.io, unmodified · **Cfgs:** `ra_single_threaded`, `ra_small_profile` · **Boxes:** ESP32-S3 DevKit (Xtensa LX7, 32-bit, @@ -377,3 +378,90 @@ Recorded rather than chased because it is a hot-path change (`page_extend` and `page_fresh`) that needs the full instruction-count battery, not a session's tail. The reproducer is three lines of `eprintln` in `collect_inner` and the counter probe in §8.2. + +--- + +## 8.7 The defect, found by pulling §8.6 — the extend bound was sixteen times too small + +`page_extend` links a batch of blocks and bounds it by 4 KiB of payload +("one OS page seems to work well"). It computed that bound as + +```rust +let span_shift = 4 + (*page).slice_count.trailing_zeros(); +let take = ((reserved >> span_shift).max(1)).min(reserved - capacity); +``` + +The identity behind it is `4096 / bsize == reserved / (slice_count * 16)`, and +**the `16` is `SEGMENT_SLICE_SIZE / 4096`** — so the literal `4` is `log2(16)` +and is correct only for the shipped 64 KiB slice. Under `ra_small_profile` the +slice is 4 KiB, the true factor is 1, and the bound this computed was **256 +bytes of payload instead of 4 KiB**. + +For a 512-byte class `reserved >> shift` is then `8 >> 4 == 0`, `.max(1)` +rescues it to one, and **every page on the profile firmware actually uses was +extended ONE BLOCK AT A TIME**. That is why §8.6 saw `capacity = 1` on every +reclaimed page, and why `generic` read exactly 1.0000 per op in §8.2: a page +with one block has no second block for the fast path to find, so every single +allocation took the slow path. + +**Fixed** by deriving the constant term from the geometry +(`SEGMENT_SLICE_SIZE.trailing_zeros() - 12`), which is 4 at the default slice +and 0 at the small profile. Counted, 100,000 alloc+free pairs, small profile: + +| size | `generic`/op before | after | blocks per extend | +|---|---:|---:|---:| +| 512 | 1.0000 | **0.1250** | 8 | +| 513 | 1.0000 | **0.1667** | 6 | +| 1024 | 1.0000 | **0.2500** | 4 | + +Page churn falls with it, 195 carve-and-retire cycles per 100,000 becoming 24, +32 and 49 respectively. + +**The step inverts.** On the 32-bit host, ABBA-interleaved, reproduced three +times: 512 against 513 goes from **+8.6 % (slower)** to **−36 % (faster)**, and +512 in absolute terms from 390,200 ns to ~200,000 ns for the same 50,000 pairs +— about **1.9× faster**. The `direct[]` route is now the fast route, which is +what its name always claimed. + +**The default geometry does not move.** `65536.trailing_zeros() - 12 == 4`, the +old literal, so the constant is identical there; the all-features x86-64 +assembly diff against `main` changes exactly one symbol, the debug-record blob, +with no executable function touched. + +### Measured on silicon after all (2026-09-10) + +The "not measured on silicon" caveat above is withdrawn: it is measured, on our +own XIAO ESP32-S3, `main` against this fix in one session on one board. Same +192 KiB region, same 166 ns/op no-allocator floor, and **identical checksums on +every row**, so the two arms did identical work. + +| workload | `main` | with the fix | | +|---|---:|---:|---| +| pingpong, 32 B | 595 | **518** | **13.0 % faster** | +| batch, 64 mixed 8-512 B | 833 | **702** | **15.7 % faster** | +| churn, 64 live 8-512 B | 1,011 | **856** | **15.3 % faster** | +| large, 2,048 B | 1,134 | 1,121 | 1.1 % | + +ns per alloc/free pair, net of the floor. **13-16 % on every binned workload**, +which is the same magnitude as the 16 % step this report opened with — and +2,048 B barely moving is the tell that it is the same mechanism, because that +size is on the bin route, which still enters `malloc_generic` on every +operation and is untouched by this fix. + +What this does NOT settle is §7's `heap_4` row. That is a different harness +(direct calls, 64 KiB, one live block) and a different question; whether +256-512 stops being the only range `rusty_alloc` loses is still the consumer's +run to make. + +### What this does and does not settle for the report + +It closes the 512-byte step and it should take a large bite out of §0's 24× +host-versus-device gap, because the device was paying `malloc_generic` on 100 % +of allocations where it should pay it on one in eight. **It is not measured on +silicon** — that rig is Kairos's, and §7's `heap_4` A/B at 256–512 is the row +to re-run. The prediction to falsify: the 256–512 range stops being the only +one `rusty_alloc` loses. + +What it does NOT explain is why the bin route enters `malloc_generic` on every +op even now (`generic` still 1.0000/op at 1025 and 2048, unchanged by this +fix). That is a separate thread, and the trace in §8.6 is where to pick it up.