[architecture][memory] Make 32 concurrent tool-recall sessions protocol-safe and memory-bounded

[lidge-jun]

Summary

OpenCodex should remain TypeScript/Bun-based, but it needs an explicit concurrency and memory architecture for the real workload we must support:

• 32 sustained independent agent sessions
• 64-session burst
• repeated model stream -> 1..N parallel tool/MCP calls -> tool results -> API recall
• no OpenCodex-imposed serialization below that target
• no loss of Codex / Responses / Chat Completions / Anthropic / MCP compatibility

This is not a proposal to raw-passthrough provider bytes. OpenCodex is a semantic protocol translator. The required change is to keep that translation while replacing unbounded or duplicated retention with bounded state, byte accounting, and stream-lifetime leases.

Audit baseline: dev@18545f87e92b856be180ba01e8b2c44d67163879.

This issue is related to the field symptoms in #314 and #509, but is not a duplicate. Those issues discuss observed Windows memory behavior and imperfect Bun counters. This issue defines the architecture, hypotheses, implementation slices, and acceptance harness needed to distinguish OpenCodex-owned retention from Bun allocator/runtime retention under multi-session tool recall.

Hard requirements

Concurrency contract

sustained independent sessions: 32
burst independent sessions:     64
global emergency active cap:     96
same-session active turns:        1
same-session queued turns:        0 by default

Different sessions must remain parallel. Only overlapping turns for the same logical session/thread may be rejected or serialized. Tool execution performed by the external client must not keep an OpenCodex model-turn lease occupied after the model stream reaches a terminal.

Protocol invariants that must not be removed

Any memory fix must preserve all of the following:

1. Codex namespace/MCP tools are flattened for chat providers and restored as { namespace, name } on the return path.
2. Codex custom/freeform tools such as apply_patch are exposed to chat models through a temporary { input: string } function schema, then restored as custom_tool_call, not function_call.
3. tool_search remains distinct from an ordinary function call.
4. call_id, Responses item_id, output_index, and Chat Completions tool indexes remain stable and correctly correlated.
5. The complete lifecycle remains valid:
   output_item.added -> arguments.delta* -> arguments.done -> output_item.done -> response terminal.
6. No-argument calls finalize as {} rather than an empty invalid JSON string.
7. Malformed or truncated arguments must not be reported as a completed usable call.
8. Reasoning signatures, redacted thinking, tool results, image content, continuation metadata, and provider-specific replay behavior remain intact.

Raw provider SSE passthrough would violate these contracts.

Important correction: the existing parallel-tool contract is deliberate

The repository already solved provider-interleaved OpenAI Chat tool deltas in devlog/_fin/260709_parallel_tool_calls/ and tests/openai-chat-parallel-stream.test.ts.

The current design is:

1. openai-chat keys provider fragments by index/id and buffers all pending calls while text/reasoning continues.
2. At a valid terminal boundary, it emits each completed call as one non-overlapping:
   tool_call_start -> one full tool_call_delta -> tool_call_end sequence.
3. bridgeToResponsesSSE() intentionally holds one currentToolCall, because the canonical AdapterEvent contract is sequential and non-overlapping.
4. Finish-less EOF with pending calls fails closed rather than fabricating a completed call.

Therefore a keyed, live, multi-call bridge is not a required P0 change and must not be introduced merely to reduce memory. It would reopen previously closed correctness hazards: text/reasoning barriers, item ordering, partial-call failure, and custom-tool unwrapping.

The measured optimization path should be:

• preserve the sequential canonical contract;
• replace fragment-by-fragment string concatenation with chunk arrays and byte counters where it matters;
• clear each adapter call as soon as its atomic sequence has been consumed;
• add per-call, per-turn, and process-wide argument-byte metrics/limits;
• consider an optional atomic tool_call_complete event only if profiling proves material duplicate reassembly between an adapter that already has a final snapshot and the bridge.

A keyed-live event redesign remains a separately reviewed alternative, not the default plan.

Current confirmed pressure points

1. tee() still creates two independently paced consumers

The native Responses passthrough path still has a client branch and a background inspection branch created from upstreamResponse.body.tee() in src/server/responses/core.ts.

The existing macOS RSS plan measured a large external-memory difference between rate-mismatched tee and one-reader variants. The remaining work is to make the one-reader path protocol-safe, runtime-qualified, compatible with required payload rewrites, and valid for the 32-session workload.

2. The inspector is not fully bounded on current dev

createSseInspector() currently:

• appends decoded chunks into an unterminated string without a frame-byte cap;
• parses the same payload in multiple consumers;
• retains every reconstructed response.output_item.done entry until completion;
• does not clear the reconstruction map immediately after the completed response is copied.

The repository already contains a concrete Phase 2 plan for a 1 MiB frame cap, 256 reconstructed items, single parse, and post-callback cleanup. Add a reconstructed-byte cap as well.

3. Tool conversion is correct, but its retained bytes are not explicitly bounded

Confirmed current behavior:

• OpenAI Chat buffers all parallel calls until a valid flush boundary.
• PendingToolCall.args is grown by string concatenation.
• The Responses bridge accumulates the active call again to create required .done and final item snapshots.
• custom/freeform input may be repeatedly decoded from a growing wrapper.
• Responses -> Chat outbound retains per-index argument strings until output_item.done/terminal to avoid duplicate append-only deltas.

These states are protocol-motivated. The open question is their measured byte cost, not whether they can simply be deleted.

4. Continuation bytes have a single-entry escape hatch

src/responses/state.ts has a 64 MiB aggregate accounting cap, but eviction currently stops while states.size > 1. One oversized final entry can remain above the cap. Forced store:false expansion also retains full expanded input for each response ID, which can grow approximately quadratically by chain length.

A final design needs a hard-cap policy plus explicit oversize/tombstone behavior, and likely parent-linked deltas and/or content-addressed large payloads so continuity never fails silently.

5. Cursor blobs are count-bounded, not byte-bounded

src/adapters/cursor/native-exec.ts stores content-addressed Uint8Array values in a process-global map with:

• 15-minute TTL
• 4,096-entry cap
• no per-blob byte cap
• no process-wide byte cap
• no active-request pin/lease accounting

Count-only eviction does not bound memory when one or more tool/history blobs are large. Selection/budgeting should happen before insertion, and live blobs need pins so byte eviction cannot remove data still needed by an active Cursor turn.

6. The eager relay is bounded per stream, but not globally or precisely

src/server/relay-eager.ts currently defaults to:

• 8 MiB client queue per stream;
• 15-second post-cancel drain;
• 32 MiB post-cancel drain bytes per stream;
• approximate queue accounting that resets queuedBytes to zero on pull().

At 32 sessions, configured per-stream limits can multiply into a large process-wide envelope. The target design needs process-wide byte accounting and either pull-paced reading or exact queued-byte release.

7. Active-turn tracking is not a coordinator

src/server/lifecycle.ts tracks only a Set<AbortController>. It does not model:

• logical session lanes;
• admitted/bootstrap/committed phases;
• global and weighted byte permits;
• provider/account active load;
• queue age;
• stream-lifetime ownership.

A permit must be released on terminal/cancel/error/shutdown, not when the handler returns a Response object.

8. Account selection is not active-load-aware

src/codex/pool-rotation.ts implements sticky/smooth weighted selection but does not include current active turns, TTFT EWMA, or transient-failure pressure. Under synchronized recall bursts, many sessions can select the same account before outcomes update.

Affinity should remain authoritative when continuation is account-scoped, but portable requests should prefer an eligible target with lower active saturation.

9. Bundled Bun 1.3.14 cannot be assumed safe for the target relay

The package pins Bun 1.3.14. Two upstream fixes directly affect the proposed shape:

• http: couple fetch() receive backpressure to JS body consumption (h1/h2/h3) oven-sh/bun#29831: fetch receive backpressure coupled to JS body consumption; merged after 1.3.14.
• Fix crash when a client aborts a streaming response under backpressure oven-sh/bun#32120: crash when a client aborts an async streaming response under backpressure; merged after 1.3.14, with a 20-concurrent-client regression test.

Also, oven-sh/bun#33368 remains open and documents why process.memoryUsage().heapUsed can exceed heapTotal dramatically on current Bun. heapUsed alone must not be used as the leak discriminator.

The release must pin an exact verified Bun.revision, not trust a semver guess or moving canary.

Expanded hypotheses and discriminating measurements

ID	Hypothesis	Expected signature	Discriminating test
H1	tee() branch-rate mismatch amplifies external/native buffers	external/arrayBuffers rise with slow or inspection-skewed clients	same fixture under legacy tee vs qualified one-reader relay
H2	protocol-required tool/output assembly creates avoidable string-copy peaks	tool-assembly counters and JSC heap track large arguments	string concat vs chunks/join-once with identical AdapterEvent and wire output
H3	full-history continuation entries retain chain prefixes	responseState.totalBytes and largest entry rise by round	32 sessions x 10 recalls with parent-linked vs full-expanded state
H4	Cursor blob map retains large payloads despite entry cap	blob count stays bounded while blob bytes rise	skewed large-blob workload with byte metrics and pin accounting
H5	recall request parsing creates simultaneous raw/string/object copies	request-cost budget tracks synchronized recall spikes	barrier-synchronized large tool-result POSTs
H6	retry/account selection creates a herd	attempts and active load concentrate on one target	synchronized 503/429/reset fixture with full jitter and active-aware routing
H7	application state returns to baseline but RSS remains high	app counters/readers/leases return to idle; RSS plateaus	repeated identical waves after warm-up
H8	a real OpenCodex leak remains after bounded state	one or more app-owned counters/object populations rise each wave	heap-retainer comparison after H1-H6 are bounded

Target architecture

request
  -> logical session lane (one active turn per session)
  -> global active-turn permit + weighted request-memory permit
  -> affinity-aware, active-load-aware account selection
  -> bootstrap-only retry / circuit policy
  -> provider adapter
       -> provider-specific parallel-call assembly where required
       -> sequential canonical AdapterEvent contract
  -> protocol bridge with bounded current-item/full-response accounting
  -> one qualified upstream reader
       -> bounded inline inspector
       -> required payload rewrite
       -> downstream response
  -> idempotent terminal cleanup releases every lease/pin/reader

The one-reader invariant means one read of the upstream body, not zero transformation. Inspection and payload rewriting must be stages on the same semantic data path.

Proposed PR series

PR 1 — Acceptance harness and observe-only metrics

Add the real recall-burst harness before changing behavior:

• 32 sessions, 10 rounds each;
• 1..8 parallel function/MCP/custom/tool-search calls per round;
• provider-interleaved OpenAI Chat deltas while preserving the canonical sequential output contract;
• random tool latency 10..500 ms;
• barrier-synchronized recalls;
• slow consumers;
• 25% random cancellation;
• pre-first-byte resets, 503 outage, account 429;
• application-owned byte/readers/turn metrics and RSS samples.

No production behavior change in this PR.

PR 2 — Bounded, single-parse inspection

Implement the existing Phase 2 plan:

• 1 MiB unterminated frame cap with correct SSE resynchronization;
• one parse per payload;
• 256 reconstructed output items plus a reconstructed-byte cap;
• immediate map cleanup after synchronous continuation copy;
• no payload contents in overflow warnings.

PR 3 — Tool/output byte accounting without changing the canonical contract

• instrument adapter pending-tool bytes, bridge active-tool bytes, final output bytes, and outbound per-index bytes;
• replace hot-path args += fragment with chunks + join-once where A/B tests show lower peak/CPU;
• add explicit per-call/per-turn/process budgets and truthful oversize failure;
• make custom-tool JSON-string unwrapping incremental;
• clear adapter call buffers as each atomic sequence is consumed;
• optionally add an atomic tool_call_complete event only if it materially removes duplicate reassembly and passes the full protocol matrix.

PR 4 — Runtime qualification and protocol-safe one-reader relay

• add reproducible fixtures for Bun fetch receive backpressure and async-pull abort safety;
• record exact validated Bun.revision capabilities in the release manifest;
• keep legacy tee as an explicit compatibility fallback;
• compose inspection and required payload rewrites into one reader path;
• preserve synthetic failed-tail semantics;
• default post-cancel drain to zero, or a very small opt-in byte/time budget;
• add a process-wide stream byte budget.

PR 5 — Turn coordinator and session lanes

• replace controller-only tracking with metadata-rich TurnScope;
• one active turn per logical session, independent sessions parallel;
• global active and request-memory weighted permits;
• release only at stream terminal/cancel/error/shutdown;
• expose active/queued/by-phase/by-provider/by-account metrics.

PR 6 — Continuation, Cursor blob, and request-body byte budgets

• hard aggregate and per-entry bounds without the single-entry escape hatch;
• explicit oversize/tombstone behavior;
• parent-linked continuation deltas or content-addressed payloads;
• Cursor blob total bytes, per-blob limit, pins, expiry and eviction metrics;
• request cost admission before retaining large decoded bodies.

PR 7 — Bootstrap-only retry, circuit breaker, and active-aware account selection

• retry only before downstream commit;
• max two attempts / bounded elapsed time / full jitter;
• no retry after first downstream byte;
• provider+model transient circuit, account-scoped quota handling;
• affinity-safe active-load score.

Acceptance criteria

Correctness

• 32 sustained and 64 burst independent sessions complete without OpenCodex-imposed serialization below target.
• No call IDs, item IDs, indexes, namespaces, names, arguments, outputs, or session state cross-contaminate.
• Provider-interleaved fragments for at least 32 OpenAI Chat calls still emerge as the existing non-overlapping canonical sequences.
• Function, MCP namespace, custom/freeform, tool_search, image-bearing result, malformed args, no-arg call, and interrupted-call cases all preserve existing contracts.
• transport.postCommitRetryAttempts remains exactly zero.

Cleanup

After every wave, within the declared teardown window:

turns.active                    -> 0
turns.queued                    -> 0
stream.activeReaders            -> 0
stream.pendingBytes             -> idle baseline
stream.inspectorBytes           -> idle baseline
tool.pendingArgumentBytes       -> 0
tool.bridgeActiveBytes          -> 0
memory.requestCostInUse         -> idle baseline
memory.continuationBytes        -> configured bound
memory.cursorBlobPinnedBytes    -> idle baseline
budget waiters                  -> 0

Memory outcome

Success is not defined as RSS returning immediately to process-start values. Success requires:

1. all OpenCodex-owned byte/readers/lease counters return to the expected idle envelope;
2. repeated identical workload waves do not retain additional application state;
3. after warm-up, RSS does not maintain a continuously positive per-wave slope.

If application-owned state returns to baseline while RSS remains on a stable high plateau, classify the residual as runtime/allocator retention and attach the exact Bun revision and fixture result.

Explicit non-goals

• raw provider-byte passthrough for translated routes;
• reducing normal concurrency below 32 sessions;
• setting a default per-account concurrency of 1 or 2;
• treating Bun 1.3.14 as qualified for the new relay;
• using heapUsed alone to attribute Windows memory;
• replacing the existing sequential canonical tool-event contract without measured justification;
• a Go/Rust rewrite;
• one giant implementation PR.

Source pointers

OpenCodex pinned source and prior design records:

• https://github.com/lidge-jun/opencodex/blob/18545f87e92b856be180ba01e8b2c44d67163879/src/types.ts
• https://github.com/lidge-jun/opencodex/blob/18545f87e92b856be180ba01e8b2c44d67163879/src/responses/parser.ts
• https://github.com/lidge-jun/opencodex/blob/18545f87e92b856be180ba01e8b2c44d67163879/src/bridge.ts
• https://github.com/lidge-jun/opencodex/blob/18545f87e92b856be180ba01e8b2c44d67163879/src/adapters/openai-chat.ts
• https://github.com/lidge-jun/opencodex/blob/18545f87e92b856be180ba01e8b2c44d67163879/tests/openai-chat-parallel-stream.test.ts
• https://github.com/lidge-jun/opencodex/blob/18545f87e92b856be180ba01e8b2c44d67163879/devlog/_fin/260709_parallel_tool_calls/001_audit_round1.md
• https://github.com/lidge-jun/opencodex/blob/18545f87e92b856be180ba01e8b2c44d67163879/devlog/_fin/260709_parallel_tool_calls/010_phase1_stream_assembly.md
• https://github.com/lidge-jun/opencodex/blob/18545f87e92b856be180ba01e8b2c44d67163879/src/chat/outbound.ts
• https://github.com/lidge-jun/opencodex/blob/18545f87e92b856be180ba01e8b2c44d67163879/src/claude/outbound.ts
• https://github.com/lidge-jun/opencodex/blob/18545f87e92b856be180ba01e8b2c44d67163879/src/server/responses/core.ts
• https://github.com/lidge-jun/opencodex/blob/18545f87e92b856be180ba01e8b2c44d67163879/src/server/relay.ts
• https://github.com/lidge-jun/opencodex/blob/18545f87e92b856be180ba01e8b2c44d67163879/src/server/relay-eager.ts
• https://github.com/lidge-jun/opencodex/blob/18545f87e92b856be180ba01e8b2c44d67163879/src/server/lifecycle.ts
• https://github.com/lidge-jun/opencodex/blob/18545f87e92b856be180ba01e8b2c44d67163879/src/responses/state.ts
• https://github.com/lidge-jun/opencodex/blob/18545f87e92b856be180ba01e8b2c44d67163879/src/adapters/cursor/native-exec.ts
• https://github.com/lidge-jun/opencodex/blob/18545f87e92b856be180ba01e8b2c44d67163879/devlog/_plan/260731_macos_rss_retention/020_inspection_bounds.md
• https://github.com/lidge-jun/opencodex/blob/18545f87e92b856be180ba01e8b2c44d67163879/devlog/_plan/260731_macos_rss_retention/030_macos_single_reader.md

Runtime and comparative evidence:

• fetch().body piped through TransformStream does not propagate backpressure oven-sh/bun#28035
• http: couple fetch() receive backpressure to JS body consumption (h1/h2/h3) oven-sh/bun#29831
• Native segfault in RequestContext.onAbort when returning async ReadableStream + client aborts oven-sh/bun#32111
• Fix crash when a client aborts a streaming response under backpressure oven-sh/bun#32120
• process.memoryUsage: source heapTotal and heapUsed from the same heap oven-sh/bun#33368
• Unbounded memory growth: default InMemoryEventStore is never evicted and cannot be disabled punkpeye/mcp-proxy#72
• https://github.com/Portkey-AI/gateway/blob/669825cbe89ee51569918b8f78a9db486fd69dd4/src/handlers/streamHandler.ts
• https://github.com/musistudio/claude-code-router/blob/4a152d959c016b476220339e856c9f4f94624c42/packages/core/src/gateway/request/pipeline.ts

Definition of done

Close only after the protocol matrix and 32-session recall harness pass on every supported OS/runtime combination, the application-owned state is bounded and observable, and the bundled Bun revision is explicitly qualified for the selected default relay mode.

[github-actions]

_{Automated translation bookkeeping — detected language: unknown.}

[Ingwannu]

Maintainer status after reviewing the current implementation attempt (#829): the architecture goal is valid, but that PR is not a mergeable slice of this plan yet. It spans 79 files rather than starting with the observe-only acceptance harness, and its current Windows run exposes two concrete blockers: the new state-store fixtures exceed their declared 5-second budgets (about 17.7s and 34.2s), and ACL memo lifecycle state leaks across the combined suite (expected one harden operation, observed five).

I requested changes on #829. Before implementation continues, please isolate the harness environment/ACL memo state, make the Windows timing contract deterministic, and split the work along the PR series defined here so the metrics/harness can land independently of storage behavior. Keeping this issue open; no architecture acceptance is implied by the current PR.

[Ingwannu]

Confirmed Windows-specific process-lifetime retention in the ACL memo sets: atomic config/response-state writes, migration backups, management-token creation, and tray replacement mint unique hardened temp paths that were never released.

I opened focused PR #840. It keys timeout memoization by the stable destination, releases only temp entries whose rename/unlink/ENOENT proves the file is absent, and deliberately retains memo state for residual hardened temps. Focused Windows/config/management/tray tests pass (167/167).

I am keeping the other confirmed memory bounds in separate PRs so each can be reviewed and rolled back independently.

[Ingwannu]

Focused PR #841 fixes the confirmed single-entry escape hatch in the Responses continuation store.

The old byte pruner stopped at states.size > 1, so one expanded/image-heavy entry could remain above the nominal 64 MiB cap. #841 measures the complete retained map entry in UTF-8 bytes before mutation, rejects an individually over-cap candidate without evicting unrelated chains, removes stale same-ID state, and applies the same rule on snapshot load. Focused tests pass 43/43.

[Ingwannu]

Focused PR #843 fixes the confirmed Antigravity replay retention: raw canonical tool arguments were retained as Map keys and each session had no call/byte bound.

#843 hashes model/session/call identities, limits signatures to 64 KiB, sessions to 256 calls / 2 MiB, and the cache to 32 MiB, with nested LRU and bounded TTL sweeping. The existing long sequential replay behavior remains covered; focused Google tests pass 56/56.

[Ingwannu]

Focused PR #844 fixes the confirmed Cursor Connect active-turn buffer peak. The decoder accepted a uint32 payload declaration and the transport repeatedly concatenated pending bytes with no inbound cap; partial EOF was also accepted as success after any prior frame.

#844 rejects >32 MiB declarations from the five-byte header, incrementally completes one bounded pending frame, preserves outbound uint32 compatibility, and fails incomplete EOF. Focused transport/framing tests pass 45/45.