[COOL IDEA™] Add bounded streaming write-through path for read→write and copy pipelines

[flyingrobots]

Problem

ReferenceStore read APIs currently materialize selected chunks through &mut impl Write with callers owning copy loops and chunk-by-chunk bookkeeping. For cross-repo handoff and future Echo/CAS integration, we need a bounded, backpressure-aware stream pipeline that can move bytes from verified source chunks to verified destination readers/writers without per-call allocation churn.

Invariant impact

Should preserve deterministic chunk verification, exact byte accounting, and write-boundary proof obligations while avoiding temporary global buffers or unbounded caller-level staging.

Proposed boundary

Keep core should remain chunk-authenticated and policy-checked.
Adapters should own the transfer adapter:

• a source adapter that emits verified range segments,
• a sink adapter that applies exactly-at-once write protocol,
• a transfer adapter coordinating bounded chunk windows and receipts.

Evidence (current gap)

• ReferenceStore read entrypoints in src/reference/range_read.rs and src/reference/reconstruction.rs only write to generic Write output.
• In-memory staging for references is explicit materialization (BTreeMap<ChunkId, Box<[u8]>>) in src/reference/staged_blob.rs and src/reference/chunk_staging.rs, which limits practical streaming for very large/continuous producers.

Alternatives considered

1. Keep current manual copy loops in each caller (no API contract change).
2. Introduce optional ReadWritePlan/ChunkPipeline APIs that return explicit transport receipts and support cancellation.
3. Add temporary adapters in each caller only (deferred contract, inconsistent behavior).

Evidence and acceptance criteria

• Add API that supports zero-copy chunk handoff within bounds of immutable chunk bytes.
• Add benchmark evidence showing lower CPU and allocation compared to current copy loops for large files.
• Preserve full-binary identity, range correctness, and corruption refusal semantics.
• Add at least one integration-style test showing read-to-write and copy-to-write use-case.

Failure modes

• Source process death or sink failure at mid-transfer.
• Short writes and interrupted output streams.
• Profile policy mismatch and partial transfer retries.
• Bounded memory violations under multi-threaded pipelines.

[coderabbitai]

🔗 Related PRs

#26 - Implement M1 Golden File Worldline identity [merged]
#31 - Implement canonical ChunkId and deterministic streaming CDC [merged]
#50 - Implement bounded streaming CAS [merged]
#51 - Implement exact byte-range reads [merged]
#54 - Define the durable segment store protocol [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!