Cut Metro's memory and CPU footprint during harness runs - #173
Open
V3RON wants to merge 2 commits into
Open
Conversation
Metro's transform pool is sized for bundling a whole app, but a harness run transforms far fewer modules and reuses a persistent transform cache. Run the pool as worker threads instead of child processes and halve Metro's own default worker count, which cuts peak RSS for a cold bundle from ~3.5 GB to ~1.8 GB on a 12-core host at no cost to build time. On a 3-vCPU runner the device-core reservation now lands on a single worker, taking Metro's in-band path so nothing competes with the simulator. Stop discarding `resolver.blockList`. It was cleared outright because Metro's stock value reduces to `__tests__`, which would hide every harness test, but that also silently dropped whatever the project had configured. Inherit the project's patterns and carve the protected paths out of them instead, which works even for `exclusionList([...])`, where the project's own exclusions and the `__tests__` rule are fused into a single regex. Harness adds only `.harness/cache` on its own behalf -- scoped to the cache rather than all of `.harness`, since `.harness/manifest.js` is served as a polyfill and must stay in the file map. The playground declares its own `.nx` exclusion, as befits an artifact of this repo rather than of Harness. Also drop Metro's stdin hotkey, and only start the file watcher when Jest is actually in watch mode.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is this?
Metro is the largest memory consumer in a Harness run, and it was sized for the wrong job. Its transform pool is dimensioned for bundling a whole app, but a Harness run transforms far fewer modules and reuses a persistent transform cache between runs — so on a 12-core host it spawned 8 child processes, each carrying its own Node runtime and Babel instance, and peaked at ~3.5 GB of RSS for a cold bundle. This tunes the pool to the work actually being done, bringing that peak to ~1.8 GB with no increase in build time.
It also fixes an unrelated problem in the same config:
resolver.blockListwas being cleared outright, silently discarding whatever the project had configured in its ownmetro.config.js.How does it work?
getCappedMaxWorkersnow starts from Metro's resolved default and takes the lower of two reductions: the existing device-core reservation, and halving. It still only ever lowers the value, so an explicit lowermaxWorkersis respected. The halving is what makes the cap bite on large hosts — previously, at 12 cores the reservation computed 8 and Metro's own default was also 8, so it never engaged at all; at 16 cores the cap was 12 against a default of 10. On a 3-vCPU runner the reservation alone lands on a single worker, which takes Metro's in-band transform path and spawns no child process, so nothing competes with the simulator.The pool also runs as worker threads rather than child processes (
transformer.unstable_workerThreads). Threads share one Node runtime and one Babel instance instead of duplicating both per worker.RN_HARNESS_DEBUG_WORKER_THREADS=0restores child processes, following the existingRN_HARNESS_DEBUG_USE_WATCHMANconvention.blockListwas cleared because Metro's stock value reduces to__tests__, which would hide every Harness test. Rather than dropping the project's patterns to achieve that, Harness now inherits them and carves the protected paths back out, expressed as one regex because metro-file-map rebuilds whatever it is given vianew RegExp(source, flags). This matters for the commonexclusionList([...])case, where the project's own exclusions and the__tests__rule are fused into a single alternation and cannot be separated by inspecting the source. Combining also lets a flag conflict drop one pattern instead of throwing, which is what Metro's own array handling does.On its own behalf Harness excludes exactly one directory:
.harness/cache. The scoping is load-bearing — excluding all of.harnessfails every build withFailed to get the SHA-1 for: .harness/manifest.js, because the manifest is served as a polyfill and must stay in the file map. Nothing else is added; excluding build output or tool caches is the project's call. Accordingly the playground declares its own.nxexclusion, sincewithNxMetroputs the monorepo root inwatchFoldersand that cache is an artifact of this repo rather than of Harness.Two smaller trims: Metro's stdin hotkey is off, and the file watcher now starts only when Jest is actually in watch mode (
--watch/--watchAll), which previously ran on every local one-shot run. The file-map cache still persists either way, since it is written onchangedSinceCacheReadindependently of the watch-only auto-save.Worth flagging for review: worker threads share a V8 heap with the Metro server, so a leaky Babel plugin now leaks into the long-lived process rather than an isolable child, and a native crash is not contained. Unit and integration coverage is green, but the device e2e suite has not been run against this branch — that is the check worth having before relying on threads as the default, hence the escape hatch.
Why is this useful?
Halving Metro's peak memory makes Harness runs materially cheaper to host, and it matters most where headroom is smallest: constrained CI runners were exactly the case the previous cap failed to cover, and a 3-vCPU macOS runner now leaves its cores to the simulator instead of to eight Babel processes. Projects also get the Metro configuration they actually wrote — a
blockListnarrowing a large monorepo's crawl is now honoured instead of being silently thrown away.