Document the test-binary-size memory multiplier and the strip mitigation - #3053
Closed
erneestoc wants to merge 2 commits into
Closed
Document the test-binary-size memory multiplier and the strip mitigation#3053erneestoc wants to merge 2 commits into
erneestoc wants to merge 2 commits into
Conversation
`xcodebuild test-without-building` buffers two classes of test payload in the client's memory, and no xcodebuild flag avoids either (measured against -resultBundlePath, -resultStreamPath, -collect-test-diagnostics, and attachment lifetimes on Xcode 26.2): - XCTAttachment payloads transit xcodebuild at ~1.4x their size - even under an attachment lifetime of "keepNever", which discards them only at finalization, after the full transfer. Measured: a hosted test adding 500MB of attachments drives xcodebuild from its 231 MiB baseline to 936 MiB. - The test process's console output is captured into xcodebuild's structured session log at ~26x per byte, independent of line length. Measured: 300MB of stdout drives xcodebuild to 7.9 GiB and the test to 27.3s mean. For attachment- or log-heavy suites this dominates harness memory and limits how many simulators a machine can run in parallel. Three opt-in features, following the apple.swizzle_absolute_ xcttestsourcelocation precedent (feature-gated force_load into the test bundle, plus the activating variable in the simulator test environment): - apple.test_drop_attachment_payloads: payloads over 4KB are replaced with a short note before reaching XCTest. For keepNever suites, where the payloads were going to be discarded after transfer anyway. Measured: 936 -> 232 MiB. - apple.test_spill_attachment_payloads: payloads are written to $TEST_UNDECLARED_OUTPUTS_DIR/spilled_attachments/ (delivered in Bazel's test outputs zip) and the attachment carries a note naming the file. Measured: 936 -> 232 MiB. - apple.test_redirect_stdout: the test process's standard output is dup2()d at load time to $TEST_UNDECLARED_OUTPUTS_DIR/ test_process_output.log. Only stdout: XCTest reports results on stderr, and the runner's tests-ran detection depends on those lines (verified both ways). Measured: 7.9 GiB -> 232 MiB, and the 300MB-stdout test drops from 27.3s mean to 2.5s mean over 3 trials - identical to an empty test - while trivial tests are unaffected. The attachment shim intercepts XCTAttachment's public designated initializer (unchanged since Xcode 9); both shims fail open with the install or degradation logged (linked-but-inactive measured identical to control, 935 vs 936 MiB). Runner-suite canary tests exercise both features end-to-end - linking and environment - so an Xcode release that restructures either seam turns a named test red instead of silently restoring the memory cost.
At session start xcodebuild eagerly builds an out-of-process symbolication service that reads the test bundle and every session binary into memory, plus other whole-binary reads while constructing the session. Measured on Xcode 26.2 with a size-controlled test host, xcodebuild's peak grows linearly at ~9x the host binary size (100MB host -> +903MB, 300MB host -> +2677MB, against a 231MB no-op baseline), before any test runs and on green runs included. No switch disables the eager pass - `-collect-test-diagnostics never`, attachment lifetimes, `XCTDisableAggressiveSymbolication`, and `XCT_IMAGE_NAMES_FOR_SYMBOLICATION` were each measured to have no effect on it - so document the lever that works: stripping local symbols and the debug map at link time (`--linkopt=-Wl,-x --linkopt=-Wl,-S`) on CI configurations, which shrinks what the harness reads. Field-measured on a 500+ dep app: host 673MB -> 500MB, xcodebuild peak -26%, an 8s relink with no recompile. Trade-offs documented (crash-report symbolication degrades to exported symbols; XCTest failure locations unaffected). Extends the "Reducing test harness memory" section added with the apple.test_* features, completing the harness peak-memory model: ~230MB fixed + 9x host binary + 1.4x attachments + 26x console output.
erneestoc
requested review from
aaronsky,
adincebic,
brentleyjones,
keith and
luispadron
as code owners
August 22, 2026 22:38
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked on #3051 (contains its commit; only the last commit is new). Documents a third — and for large apps, dominant — test-harness memory class, plus the mitigation that works.
At session start,
xcodebuild test-without-buildingeagerly builds an out-of-process symbolication service (XCTOutOfProcessSymbolicationService) that reads the test bundle and every session binary into memory — before any test runs, green runs included. Profiled on a large modular app (500+ deps), the two top allocation sites were the symbolication service's whole-binary/dSYM reads (~1.39GB) and test-session construction (~705MB).Measurements
Size-controlled experiment (Xcode 26.2): a hosted test whose host binary carries an inert
__TEXTblob, so binary size is the only variable.Linear at ~9× host size. Harness peak model, combining with #3051's measurements:
No off-switch exists. Each of these was measured against the 300MB fixture with no effect:
-collect-test-diagnostics never, attachment lifetimes,XCTDisableAggressiveSymbolication,XCT_IMAGE_NAMES_FOR_SYMBOLICATION(the latter two found by string-mining XCTestCore/XCTHarness; the in-process gates don't govern the harness-side eager pass).The working lever is shrinking what gets read:
--linkopt=-Wl,-x --linkopt=-Wl,-Son CI configs (relink-only). Field-measured on the 500+ dep app: host 673MB → 500MB (__LINKEDIT318MB → 146MB), xcodebuild peak −26%, 8s relink. Trade-offs documented in the section: crash-report backtraces degrade to exported symbols; XCTest assertion locations are unaffected (compiled into test code).Steps to reproduce the 9× multiplier
.section __TEXT,__const/.globl _blob/.no_dead_strip _blob/.space 314572800, in analwayslinkobjc_library.while :; do ps -axo rss=,comm= | grep xcodebuild; sleep 0.5; done--nocache_test_results; compare peak against the same test without the blob; vary.spaceto see the slope.