[BAD CODE™] In-memory reference staging still materializes all missing chunks before publish

[flyingrobots]

Affected revision

main

Invariant at risk

Large input reads/stores must remain bounded and recoverable; staging should not require unbounded memory as a prerequisite for correctness.

Observed behavior

The staging adapter stores every missing chunk in-process before publication.

• src/reference/chunk_staging.rs:10-12,29-47 keeps BTreeMap<ChunkId, Box<[u8]>>.
• src/reference/chunk_staging.rs:69-79 copies incoming bytes into owned buffers before they can be committed.
• src/reference/staged_blob.rs:11-17 documents that materialized bytes may grow with logical blob length.

This is explicit in-code behavior, not just implementation detail. It blocks practical zero-copy or bounded-memory ingestion for very large blobs and increases failure impact when capacity is high.

Expected behavior

Staging should have a bounded, deterministic streaming path that does not retain full logical blob content in a single adapter object unless explicitly unavoidable.

Deterministic reproduction

Create a ReferenceStore::stage flow over a synthetic source with a deterministic large input and low available host memory while tracking ReferenceStore memory footprint; observe growth proportional to missing-chunk set size.

Recovery and integrity impact

Core integrity logic remains sound, but operational safety is reduced by increased heap exposure during intake; a crash during staging can discard large pending buffers before publication, and large producers are blocked by memory pressure rather than policy limits.

Evidence and acceptance criteria

• Define and measure a bounded-memory streaming staging contract (or explicit rationale for unavoidable materialization).
• Add regression coverage for memory floor/ceiling in staging acceptance tests.
• Preserve StagedBlob semantics while adding a streaming publication path where possible.

[coderabbitai]

🔗 Related PRs

#26 - Implement M1 Golden File Worldline identity [merged]
#48 - Specify canonical flat chunk layouts [merged]
#50 - Implement bounded streaming CAS [merged]
#51 - Implement exact byte-range reads [merged]
#54 - Define the durable segment store protocol [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!