Reduce peak memory of the JIT dependency analysis

[kwonjeomsim]

Cuts the peak memory that JITCompiler::buildVariables() needs while it resolves
which value each stack slot carries at a control-flow merge, and returns what is
left over to the operating system when compilation finishes.

Two commits, independent of each other:

• Store small JIT dependency sets inline instead of std::set — the real fix.
• Return the JIT analysis scratch memory with malloc_trim — releases what the
  analysis already freed.

Why the analysis dominates peak memory

DependencyGenContext holds one dependency set per (label, stack slot) cell. The
matrix is dense: every label gets requiredStackSize cells whether or not the
slot is live there, so both dimensions grow with function size and the cell count
grows quadratically. bzip2.wasm produces 413,652 cells, and with std::set each
one cost 48 bytes while empty plus a separately allocated 40-byte node per element.

(*bzip2.wasm is external benchmark - from human writer)

The sets are tiny. Counted over every cell of that module:

max elements	cells	share
0	120	0.0%
1	152,031	36.8%
2	228,507	55.2%
3	20,628	5.0%
4 or more	12,366	3.0%

So DependencyList now keeps two refs inline in the object and spills to a
malloc'd block only past that. A cell is 16 bytes instead of 48, and 92% of them
never allocate at all. The low two bits of the first word separate the two states:
dependency refs are either label pointers (low bits 0) or variable refs (low bits
1), and 0 marks an empty slot, so tag 2 is free to mark a spilled block. Elements
stay sorted and deduplicated, which preserves the std::set iteration order that
mergeVariables depends on.

Measurements

valgrind --tool=massif --time-unit=B over test/wasmBenchmarker/ctests/wasm,
run as walrus --jit <module>. Both binaries come from the same tree; "before"
only has Analysis.cpp reverted to its current upstream state. Numbers are
useful-heap at the peak snapshot. The "analysis" columns attribute the peak
tree to Analysis.cpp.

benchmark	peak before	peak after	analysis before	analysis after
binary-trees	1.10 MB	0.49 MB	0.79 MB (72%)	0.18 MB (38%)
fasta	1.09 MB	0.52 MB	0.79 MB (72%)	0.14 MB (26%)
huffman	1.10 MB	0.50 MB	0.78 MB (71%)	0.18 MB (37%)
kNucleotide	1.07 MB	0.47 MB	0.78 MB (73%)	0.18 MB (39%)
rsa	1.04 MB	0.43 MB	0.78 MB (75%)	0.18 MB (42%)
gaussian	1.03 MB	0.44 MB	0.75 MB (73%)	0.18 MB (40%)
nbody	1.01 MB	0.45 MB	0.73 MB (72%)	0.17 MB (38%)
simdMatrixMultiply	1.01 MB	0.44 MB	0.73 MB (72%)	0.17 MB (39%)

The top of the peak tree changes shape. Before, the largest single contributor is
red-black tree node allocation:

->45.5%  insert (stl_set.h:514) -> _M_insert_unique -> _M_get_node
    ->37.6% (495,680B) DependencyGenContext::update (Analysis.cpp:85)

After, that chain is gone and only the cell array itself is left:

->22.0%  _M_allocate (stl_vector.h:381)
    ->22.0% (108,288B) DependencyGenContext::DependencyGenContext (Analysis.cpp)

These modules are small (7-20 KB of wasm), so the absolute numbers are small. The
effect scales with the square of function size. Compiling every function of
bzip2.wasm peaks at 34.51 MB before and 7.42 MB after, split as:

	before	after
cell array (resize)	12.74 MB	4.25 MB
elements (insert)	19.37 MB	0.78 MB
module total	34.51 MB	7.42 MB

Across ten larger benchmarks, peak RSS over the interpreter drops from +29.35 MB
worst case to +5.58 MB. 483,954 node allocations become roughly 30,000 block
allocations, which also takes JIT compile time on that set from 5.14 s to 4.83 s.

The one behavioural subtlety

The transitive-closure loop in buildVariables() can pull from the very cell it
inserts into, when a label refers to itself. A std::set never moves its nodes,
so iterating while inserting was safe; this container may move its storage. The
loop now indexes rather than holding an iterator. (Please check lines 709 to 717 at Analysis.cpp)

malloc_trim

The analysis frees everything it allocates -- massif puts the heap at 0.41 MB in
its final snapshot -- but resident memory does not go back down, because the
allocations are many small chunks spread through the glibc main arena.
MALLOC_MMAP_THRESHOLD_ and MALLOC_TRIM_THRESHOLD_ do not help, since those
only govern large single allocations.

Splitting /proc/<pid>/smaps by mapping shows malloc_trim(0) touches only
[heap] and leaves the generated machine code alone. On a JIT-compiled sqlite the
heap mapping goes from 27.3 MB to 6.6 MB while the executable mapping stays at
3.4 MB.

jitCompile() runs once per module while the module is parsed, so this is one
call per module load. Over the 32 wasmBenchmarker programs it costs no measurable
time: 14.71 s and 14.71 s for the first two of four alternating passes, and within
0.012 s on the rest.

Verification

• Builds clean on this base, no new warnings.
• tools/run-tests.py passes 130/130 across basic-tests, jit, wasi,
  wasm-test-core, wasm-test-extended, wasm-test-web-assembly3.
• All 32 wasmBenchmarker programs produce byte-identical output before and after.
  (redBlack.wasm fails with undefined element on both, unrelated to this change.)
• An assert-enabled debug build compiles every benchmark module without tripping
  an assertion.

🤖 Generated with Claude Code

[zherczeg]

Is this an AI only contribution?

[kwonjeomsim]

Most of work is done by AI including problem finding, implementation, commit and PR message, test.
I did was just review the generated contents and remove some messy comment in code.

Is there any restriction in AI contribution? If it is, please let me know.

[zherczeg]

I don't think so. But AI usually means extra reviewer work.

Anyway, in my experiences the biggest memory consumer is Memory:
https://github.com/Samsung/walrus/blob/main/src/runtime/Memory.cpp#L66

If an application asks for 32MB, or 1GB memory, we must allocate that. The default is 16MB usually. The complication memory consumption is often negligible compared to that. That is why I spent little effort on it so far.

The new code has a lot of memove, memcpy operations. Is this makes a logarithmic search to polynomial?

[kwonjeomsim]

In program which allocating lot of memory in execution time, this change does not reduce max RSS as you said. But this change works in smaller memory allocation at execution time. In that case, max RSS depends on JIT compiling largest function. So this change will work for that.

And the terms of time complexity, this new structure's insertion time is O(n). But max value of n is under 2,000 in my tests, and most of cells has elements two or fewer (>90%), so memory allocation or memmove is not required in that case. As a result, actual execution time does not become bigger, rather become smaller in most case.

[zherczeg]

If I understand correctly, the key optimization is that the number of items are usually < 2.
What happens, if we would keep the set for more than 2 elements? Or an inline vector / normal vector / std::set cases

[zherczeg]

Why these functions are needed?

[kwonjeomsim]

I didn't checked that these libraries are already included in Walrus.h. My fault.

[zherczeg]

Is this better with vector with inline storage:
https://github.com/Samsung/walrus/blob/main/src/util/Vector.h#L552
Probably a bit better, but is it worth the code duplication?

[kwonjeomsim]

I checked whether this class can be replaced with inline storage vector, but inline storage vector used more memory than std::set. The size of one cell of std::set is 48 bytes while vector is 56 bytes.

And wall time becomes worse either. So I think this structure can not replaced with inline storage vector.

[zherczeg]

Thank you. I assumed it, but good to have confirmation.

[kwonjeomsim]

I measured maxRSS and wall time about three conditions you mentioned. std::set gets more wall time, but that is small part(around +0.1s), and other ways also has almost same result.

There is no meaningful difference in maxRSS, because the almost every cell's (over 90%) number of elements is less than 3.
So IMO using std::set when elements are larger than 2 is good choice if we should keep time complexity.

[zherczeg]

I agree. I remember I used a vector before, but changed to std::set after some big programs have runtime issues.

[kwonjeomsim]

I have replaced array to std::set when elements over 2. But some wrapper iterator is added to address different two data structure.