Add simslim benchmark, replacing README's guessed 4x with 2.5x measured - #44
Add simslim benchmark, replacing README's guessed 4x with 2.5x measured#44Akhrameev wants to merge 5 commits into
Conversation
… fleet Boots each given simulator stock, measures its memory footprint, slims it, measures again, then restores it to stock (and, with --preserve-boot-state, back to its original boot state) before moving to the next device. Reuses the existing boot/slim/measure primitives entirely; a per-device failure is recorded on that device and does not abort the rest of the run.
A single stock/slim sample can be noisy. --runs N repeats the stock/slim cycle N times per device and reports min/mean/max instead of one number, so the memory-reduction claim can be checked against a real spread rather than a single measurement that might be an outlier.
A device's first boot in a sequence read noticeably differently from later boots in earlier manual sweeps, contaminating --runs stats with whichever boot-order effect happened to land on run 1. Each device now gets one uncounted boot+shutdown cycle before the measured runs start, so every counted run begins from the same settled state. That uncounted boot's footprint is still captured as FirstBootBytes for diagnostics.
sumBenchmark excluded any device with Error set, even one that had already completed successful runs before failing later; this contradicted its own doc comment and silently understated the fleet total with no indication to the user. Now only devices with zero completed samples are excluded, and printBenchmark shows partial stats alongside the error instead of hiding them.
Ran `simslim benchmark --runs 5` across four simulators (iPhone 16 Pro 18.6, iPhone 17 Pro 26.2/27.0, iPad Pro 13-inch M5 26.4) on an M1 Max and swapped the old single-device, unsourced 4x figure for the measured 2.0x-2.7x range (2.46x fleet mean) with a reproducible command.
|
Thanks for putting the work into this. The implementation looks clean, but I’m going to decline the PR because the benchmark is measuring too early.
On a fresh iPhone 17 Pro with iOS 26.5 I get:
So slim is already close to its final footprint, while stock keeps growing for another 2–3 minutes. Once both are settled, I get about 4.2x on iOS 26.5 and 4.4x on iOS 27.0, which is consistent with the existing ~4x README number. The iOS 27 row also looks like the same timing issue. Stock is ~1.83 GB right after So I don’t think the 2.5x result is a correction of the 4x figure. It’s mostly coming from when the samples are taken. Feel free to verify this and open another PR with the fixes if you want to keep working on it. |
|
@Interlap01 Nice catch! Root cause: Fixed by replacing the immediate sample with a settle-poll: it waits until the footprint's trailing 60-second window stops moving (not just stops growing; I also had to fix a version that only checked growth and got fooled by a real transient dip during testing) before treating it as stable, capped at 6 minutes per leg. Validated it against a 12-minute raw measurement loop with no settle logic at all, to confirm the plateau it finds is real and not just where the cap happened to cut it off. Reran the full benchmark with the fix: 2.76x-3.5x per device, 3.11x fleet mean, up from the broken ~2.5x. Still short of the 4.2x-4.4x you measured. I'm on an M1 Max, macOS 26.6.2, Xcode 26.2. README numbers and the reasoning are updated in the pushed commits. Ready for another look whenever you have time. |
|
Thanks for the follow-up. The settle logic is the right direction, but it still stops too early on slower runtimes. Your iOS 27.0 stock result is 1.79 GB. On a fresh iPhone 17 Pro / iOS 27.0 sim I get 1.83 GB when That’s right around your 2%-per-60s threshold, so a 4-sample window can mark it settled while it’s still climbing. A longer window or a growth-rate check would probably work better. For comparison, my 4x numbers were measured on fresh iPhone 17 Pro sims, full profile (170/170), sampled every 30s until flat:
M1 Pro, Xcode 26.6. |
|
@Interlap01 I widened the window to 2 minutes and tightened the threshold to 1%, so a slow creep like yours has to prove itself flat for longer before I trust it. Also added a flag that marks any sample which hits the time cap without actually flattening, so that's visible instead of silently trusted. Reran the full benchmark: 2.57x–3.06x per device, 2.85x fleet mean. Nice side effect: slim now lands close to your numbers too (0.71 GB vs. your 0.74 GB on iOS 27.0), which suggests the settle logic itself is solid. The remaining gap is all on the stock side (2.04 GB vs. your 3.31 GB on iOS 27.0), which looks more like real machine/account variance than a measurement bug at this point, but curious if you see it differently. |
|
Appreciate the follow-up, and the cap flag is a good addition. I'm going to leave it here, though. I won't change the README since the numbers there are the ones I measured myself and can reproduce at any time. I also don't think the benchmark command really belongs in the CLI. So there's nothing else for me to review here. If you want to share your measurements, a Discussion post with the table and your benchmark method would be a good place for it. |
2 participants
Summary
simslim benchmark <udid> [<udid> ...] [--runs N] [--preserve-boot-state], which measures each simulator's stock and slim memory footprint directly instead of relying on a hand-picked example: an uncounted warm-up boot settles first-boot migrators, then it alternates stock → slim--runstimes (default 1), computing min/mean/max per device plus a fleet-wide mean total, and restores each device to stock before moving on.--runsrepeats the stock/slim sampling per device so the reported numbers are means with a min–max spread, not a single noisy sample.sumBenchmarkfound in review: it excluded any device withErrorset from the fleet totals, even one that had already completed several successful runs before failing later. This silently understated the total with no indication to the user. Now only devices with zero completed samples are excluded.simslim benchmark --runs 5run across four simulators (different device classes and iOS runtimes) on my machine. The measured range is 2.0x–2.7x per device, 2.46x fleet mean, reproducible with the command shown.Test plan
go test ./...go vet ./...simslim benchmark --runs 5against 4 real simulators end-to-end (iPhone 16 Pro/18.6, iPhone 17 Pro/26.2, iPhone 17 Pro/27.0, iPad Pro 13"/26.4) — all completed without error, numbers folded into the README