perf(memtrack): reduce serialization bottleneck

[not-matthias]

Summary

• replace derived flattened MemtrackEvent serialization with a byte-identical manual serializer
• make MemtrackWriter::finish return encoded frame bytes for batching
• encode contiguous memtrack event batches on worker threads and write sequence-numbered zstd frames in order

Verification

• cargo test -p runner-shared artifacts::memtrack
• cargo test -p runner-shared
• cargo check -p runner-shared
• cargo test --manifest-path crates/memtrack/Cargo.toml via Nix shell with libclang/build deps; eBPF integration cases compiled but were ignored because GITHUB_ACTIONS was unset
• cargo bench -p runner-shared --bench memtrack_writer

Notes

• memtrack_writer bench measures the Phase A single-writer encoder path only, not the full parallel memtrack pipeline.
• End-to-end memory-mode speedup was not measured in this environment.

[codspeed-hq]

Merging this PR will improve performance by 44.86%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 4 improved benchmarks
✅ 3 untouched benchmarks
🆕 10 new benchmarks

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
⚡	Simulation	write_events[1000000]	2.1 s	1.4 s	+47.8%
⚡	Simulation	write_events[100000]	207.3 ms	141.3 ms	+46.74%
⚡	Simulation	write_events[10000]	20 ms	13.9 ms	+43.69%
⚡	Simulation	write_events[500000]	1,036.6 ms	733.6 ms	+41.31%
🆕	WallTime	encode_events_realistic[16]	N/A	121.9 ms	N/A
🆕	WallTime	encode_events_realistic[4]	N/A	216.1 ms	N/A
🆕	WallTime	encode_events_realistic[8]	N/A	144.5 ms	N/A
🆕	WallTime	write_events[10000]	N/A	7 ms	N/A
🆕	WallTime	write_events[100000]	N/A	72 ms	N/A
🆕	WallTime	write_events[1000000]	N/A	771.5 ms	N/A
🆕	WallTime	write_events[500000]	N/A	387.4 ms	N/A
🆕	Simulation	encode_events_realistic[16]	N/A	1.2 s	N/A
🆕	Simulation	encode_events_realistic[8]	N/A	1.2 s	N/A
🆕	Simulation	encode_events_realistic[4]	N/A	1.2 s	N/A

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing cod-3071-fix-serialization-bottleneck (d06d42e) with main (705245d)}

[greptile-apps]

Greptile Summary

This PR replaces the single-threaded, single-zstd-frame memtrack encoding pipeline with a parallel one: events are batched into 64 K-event windows, each window is compressed as independent zstd frames across a Rayon worker pool, and frames are written in order so the output is fully deterministic. It also migrates the eBPF ring-buffer channel from std::sync::mpsc to crossbeam_channel and simplifies the Tracker ownership model so the poller's lifetime is tied to the struct rather than a detached relay thread.

• pipeline.rs — new encode_events<S, W> function: collects up to FRAME_EVENTS × WINDOW_FRAMES (1 M) events per window, compresses frames in parallel via rayon, writes in input order, and always emits at least one valid frame for an empty source.
• writer.rs — MemtrackWriter::finish() now returns the compressed Vec<u8> instead of (); compression level lowered to -5 (ultra-fast); BufWriter now buffers serializer output before the encoder rather than after.
• tracker.rs / poller.rs — Tracker owns the RingBufferPoller; stop_polling() drops it, which drains ring-buffer stragglers and closes the event channel; the old detached relay thread is gone.

Confidence Score: 5/5

Safe to merge. The core encoding logic is correct, multi-frame decoding works with zstd 0.13.3's default multiple-members behavior, and ordering is verified by the new tests.

The parallel pipeline, ownership changes, and channel migration are all well-reasoned and covered by tests. The two findings are style and benchmark-accuracy nits that don't affect production correctness.

No files require special attention. The benchmark in crates/runner-shared/benches/memtrack_writer.rs has a thread-pool-per-iteration overhead issue, but it only affects benchmark result accuracy, not production behavior.

Important Files Changed

Filename	Overview
crates/runner-shared/src/artifacts/memtrack/pipeline.rs	New encode_events function: batches events into FRAME_EVENTS-sized windows, compresses frames in parallel via Rayon, writes in order. Logic is correct; order-preservation and empty-stream tests are thorough.
crates/runner-shared/src/artifacts/memtrack/writer.rs	MemtrackWriter refactored to return compressed bytes from finish(); BufWriter and encoder layers swapped; compression level changed to -5; stale comment noted as P2.
crates/runner-shared/src/artifacts/memtrack/mod.rs	Split into mod.rs + pipeline.rs + writer.rs; decode_streamed unchanged; zstd 0.13.3 enables multiple_members by default so multi-frame decoding works correctly.
crates/memtrack/src/main.rs	Old two-stage drain+write pipeline replaced with a single encode_events call; poller lifecycle now tied to Tracker; shutdown sequence (disable → stop_polling → join) is correct.
crates/memtrack/src/ebpf/tracker.rs	Tracker now owns the RingBufferPoller; stop_polling() drops it cleanly; old relay-thread workaround removed; cleaner ownership.
crates/runner-shared/benches/memtrack_writer.rs	Adds encode_events_realistic bench comparing 4/8/16 workers; Rayon thread pool is created inside bench_local on every iteration, skewing worker-count comparisons with thread lifecycle overhead (P2).

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant eBPF as eBPF ring buffer
    participant Poller as RingBufferPoller (thread)
    participant Chan as crossbeam channel (unbounded)
    participant Pipeline as encode_events (main thread)
    participant Rayon as Rayon worker pool
    participant Disk as Output file

    eBPF->>Poller: events (ring buffer)
    Poller->>Chan: tx.send(event) xN
    Note over Pipeline: window.extend(take(1M))
    Chan-->>Pipeline: Receiver iterator (blocking)
    Pipeline->>Rayon: par_chunks(64K) to encode_frame
    Rayon-->>Pipeline: Vec of frames in order
    Pipeline->>Disk: write_all(frame) xWINDOW_FRAMES
    Note over Pipeline: repeat until channel closed
    Note over Poller: stop_polling() shutdown ringbuf.consume()
    Poller->>Chan: tx dropped channel closed
    Chan-->>Pipeline: iterator returns None
    Pipeline->>Disk: flush()

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant eBPF as eBPF ring buffer
    participant Poller as RingBufferPoller (thread)
    participant Chan as crossbeam channel (unbounded)
    participant Pipeline as encode_events (main thread)
    participant Rayon as Rayon worker pool
    participant Disk as Output file

    eBPF->>Poller: events (ring buffer)
    Poller->>Chan: tx.send(event) xN
    Note over Pipeline: window.extend(take(1M))
    Chan-->>Pipeline: Receiver iterator (blocking)
    Pipeline->>Rayon: par_chunks(64K) to encode_frame
    Rayon-->>Pipeline: Vec of frames in order
    Pipeline->>Disk: write_all(frame) xWINDOW_FRAMES
    Note over Pipeline: repeat until channel closed
    Note over Poller: stop_polling() shutdown ringbuf.consume()
    Poller->>Chan: tx dropped channel closed
    Chan-->>Pipeline: iterator returns None
    Pipeline->>Disk: flush()

Loading

_{Reviews (11): Last reviewed commit: "ci: install rust toolchain by explicit c..." | Re-trigger Greptile}

[GuillaumeLagrange]

lgtm