Skip to content

runtime: reserve 16GB of heap address space on 64-bit unix - #5644

Open
0pcom wants to merge 1 commit into
tinygo-org:devfrom
0magnet:unix-heap-reserve
Open

runtime: reserve 16GB of heap address space on 64-bit unix#5644
0pcom wants to merge 1 commit into
tinygo-org:devfrom
0magnet:unix-heap-reserve

Conversation

@0pcom

@0pcom 0pcom commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

allocateHeap caps the heap at 1GB for all targets, so with -gc=conservative or -gc=precise any single allocation approaching 1GB fails with out of memory regardless of available system RAM. The real case that hit this: a scrypt key-derivation buffer (N=1<<20, r=8 → 1GiB) in the skycoin wallet, on a machine with 16GB free.

Reserve 16GB of virtual address space on 64-bit targets instead. The mmap is a reservation, not a commitment: pages cost physical memory only when first touched, and the existing halve-on-failure loop still adapts when the map is refused. 32-bit targets keep the 1GB cap unchanged. This is the direction the growHeap comment already points at: "If we run out of memory, we should consider increasing heapMaxSize on 64-bit systems."

Verified on linux/amd64 with -gc=conservative:

  • before: make([]byte, 1536<<20)fatal error: out of memory (15GB RAM free)
  • after: the same allocation succeeds and is fully usable (touched and read back across the whole range)

One behavioral note, stated for the record: under the blocks GC on 64-bit hosts the practical limit becomes actual system memory — matching the boehm default and big Go — so true exhaustion surfaces as OS memory pressure rather than a fatal error at an arbitrary 1GB.

🤖 Generated with Claude Code

https://claude.ai/code/session_01C3X3gvq8ZggMRvzVzWm66i

@0pcom
0pcom force-pushed the unix-heap-reserve branch from 68058ae to ae3954c Compare September 2, 2026 08:14
@0pcom

0pcom commented Sep 2, 2026

Copy link
Copy Markdown
Contributor Author

Fixed the CI failures: the 16GB literal did not compile on 32-bit targets (ARM/MIPS) — Go type-checks the dead branch too. The reservation size is now a constant derived from TargetBits (1 << (30 + 4*(TargetBits/64))): 16GB on 64-bit, the previous 1GB on 32-bit. Verified: linux/amd64 still allocates and uses a 1.5GiB buffer under -gc=conservative, and linux/arm + linux/mips cross-compile cleanly.

allocateHeap capped the heap at 1GB for all targets, so with -gc=conservative
or -gc=precise any single allocation approaching 1GB (for example a scrypt
key-derivation buffer with N=1<<20, r=8) failed with out of memory regardless
of available system RAM.

Reserve 16GB of virtual address space on 64-bit targets instead. The mmap is a
reservation, not a commitment: pages cost physical memory only when first
touched, and the existing halve-on-failure loop still adapts when the map is
refused. 32-bit targets keep the 1GB cap; the size is derived from TargetBits as a
shifted constant, since a plain 16GB literal does not compile on 32-bit
targets even in a dead branch. This is the direction the growHeap
comment already points at: "If we run out of memory, we should consider
increasing heapMaxSize on 64-bit systems."

With this, the practical limit under the blocks GC on 64-bit hosts becomes
actual system memory, matching the boehm default and big Go; true exhaustion
surfaces as the OS's memory pressure handling rather than a fatal error at an
arbitrary 1GB.
@0pcom
0pcom force-pushed the unix-heap-reserve branch from ae3954c to 5572fc4 Compare September 2, 2026 09:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant