Delta sync: large files move as content-defined chunks

[ssowonny]

TL;DR

• Editing one byte of a 20 MiB file now syncs ~2 MB instead of ~21 MB, both directions — big files move as content-defined chunks.
• Old clients keep working untouched: the hub reassembles whole blobs on demand, proven by e2e tests that build the real pre-change binary.
• The per-file sync ceiling rises 32 → 100 MiB, so files that size actually reach every device now.
• Deploy hubs before clients: an old hub refuses chunk keys (push safely retries), and an old client misreports 32–100 MiB files as "blob corrupt" until it upgrades.
• Known gap: chunk transfer is serialized per file, so a cold 100 MiB push pays ~250 sequential round trips — parallelizing that loop is the best follow-up.

What this is

Files over 4 MiB push as chunks/<sha256> pieces plus a manifests/<sha256> chunk list keyed by the whole file's hash — so Op.Blob alone locates it and the journal format is byte-identical (journal.Less and Replay untouched; a test diffs a chunked op's JSON fields against a small file's). Chunk boundaries come from a rolling hash (restic/chunker, fixed polynomial, 256K/1M/4M), so an insertion at the front of a 20 MiB file moves ~527 KB, not the file. Local volume stores keep files whole — chunking exists only on the wire and in the hub's store.

Measured (byte-counting tests in internal/syncer/delta_test.go, real-HTTP variants in internal/webapp/delta_e2e_test.go):

Scenario	Before	After
1-byte edit, 20 MiB file (push)	20,972,098 B	2,050,572 B
Same, real HTTP push / pull	~21 MB each way	1,954,726 / 1,954,696 B
Front insertion (the CDC discriminator)	~21 MB	526,944 B
Identical 12 MiB content at a second path	~12 MB	1,373 B (manifest only)

Backward compatibility

RemoteSource.OpenBlob is the single hub-side content read (viewer, history, shares, downloads, /store/*); when blobs/<sha> is absent it reassembles from the manifest — spool, hash-verify, backfill, serve — so a client that has never heard of a manifest keeps working with no change. The e2e suite builds the actual pre-change binary from the pinned merge-base commit (git archive, no worktree state) and drives it against a live hub: chunked-storage reads, old-writes/new-reads round trips, concurrent old↔new conflicts with conflict-copy preservation, and delete propagation.

Sequencing (also in the CHANGELOG): hub first, then clients. An old hub 400s chunk keys — a new client's large-file push degrades to offline-retry, no data loss. An old client caps reads at its 32 MiB bound, so 32–100 MiB files report "blob corrupt on remote" (the hub is healthy) until that client upgrades.

Hostile-member hardening

Four rounds of adversarial review (beardrive-cto agent; scoreboard and round log in .claude/delta-sync-goal.md) drove the design to its final shape:

• The push skip-proof is one Exists per chunk. Three cheaper proxies — local basis possession, manifest existence, stored manifest content — each proved false or forgeable (a member who can read a file can publish its true chunk hashes without uploading a byte). The dead proxies are recorded in the code comment so the next round-trip optimization doesn't resurrect one.
• Manifests are write-once and must name only chunks the store holds — enforced at the hub's PUT door, making "a manifest exists ⟹ its chunks exist" a real invariant for every consumer. A refused manifest falls back to a whole-blob push, so nothing can wedge a device's push leg (and chunker parameters are no longer load-bearing).
• Reassembly is bounded (256 MiB on the declared sum, per-chunk declared==actual), closing the amplified-manifest spool attack.
• bdrive import refuses archives whose journals reference content they don't hold — the silent-loss shape an old export produces against a new hub; --allow-incomplete overrides with per-path warnings.

Also in this PR

• maxPullBytes 32 → 100 MiB (hostile-hub sec-test ceilings now derive from the constant).
• False "blob corrupt on remote" no longer reported when the whole-blob fallback lands the file.
• Docs: README layout section, web/docs reference pages (cli, project-files), CHANGELOG Unreleased entry with the deploy-order note.
• Follow-ups recorded in the goal file's Leads: parallel chunk transfer (M1), quota accounting on reassembly backfill (M2, managed-layer), reassembly-spool concurrency bound (M3). Transport compression is BEA-144.

Architecture changes

architecture/cli-sync.md — Session gained pushChunked / fetchChunked / chunkSpans; a new Manifest type (chunks.go) carries the chunk list, with a new Session → Manifest relationship for the over-4-MiB paths.

✅ added · ❌ removed (strikethrough) · unmarked = unchanged

flowchart TB
    Session["<div style='text-align:left'><b>Session</b><br/>+Cycle(ctx) Result<br/>+Restore(ctx, path, sha)<br/><span style='background:#22c55e55;padding:0 4px;border-radius:3px'>✅ -pushChunked(ctx, blob) bytes</span><br/><span style='background:#22c55e55;padding:0 4px;border-radius:3px'>✅ -fetchChunked(ctx, op, basis)</span><br/><span style='background:#22c55e55;padding:0 4px;border-radius:3px'>✅ -chunkSpans(blob) []span</span></div>"]
    Manifest["<div style='text-align:left'><b>Manifest</b><br/>+V int<br/>+Size int64<br/>+Chunks []chunkRef h, n</div>"]
    Backend["<b>remote.Backend</b><br/>Put · Get · List · Exists"]
    Note["chunks.go — files &gt; 4 MiB push as<br/>chunks/&lt;sha256&gt; + manifests/&lt;file-sha&gt;<br/>keyed by Op.Blob: journal format untouched.<br/>Skip-proof = one Exists per chunk;<br/>manifest refusal → whole-blob fallback"]
    Session -- "pull and push" --> Backend
    Session -- "✅ <span style='background:#22c55e55;padding:0 5px;border-radius:3px'>chunked push/pull, files &gt; 4 MiB</span>" --> Manifest
    Manifest -.- Note
    classDef added fill:#22c55e22,stroke:#22c55e,stroke-width:2px
    classDef noteBox fill:#88888822,stroke:#888888,stroke-dasharray:2 2
    class Manifest added
    class Note noteBox
    linkStyle 1 stroke:#22c55e,stroke-width:2px

Loading

architecture/webapp-server.md — RemoteSource gained reassemble (the chunked-storage fallback behind OpenBlob), and the /store/* door's key space grew the two new classes with their ingest gates.

flowchart TB
    RemoteSource["<div style='text-align:left'><b>RemoteSource</b><br/>+OpenBlob(ctx, sha)<br/><span style='background:#22c55e55;padding:0 4px;border-radius:3px'>✅ -reassemble(ctx, sha) chunked fallback</span><br/>-verify(ctx, sha) re-hash until sealed</div>"]
    journalDoor["<div style='text-align:left'><b>journalDoor</b> /api/p/id/store/*<br/>ownJournal · journalOps<br/>opsNameTheirAuthor</div>"]
    Keys["✅ new key classes<br/>chunks/&lt;sha256&gt; — content-addressed,<br/>presigned like blobs, PUT must hash to key<br/>manifests/&lt;sha256&gt; — never presigned,<br/>WRITE-ONCE, every named chunk must exist"]
    Reassembly["✅ reassemble: manifest → concat chunks →<br/>hash-verify → backfill blobs/&lt;sha&gt; → serve.<br/>Old clients never learn anything changed.<br/>Bounded: maxReassembleBytes 256 MiB"]
    RemoteSource -.- Reassembly
    journalDoor -.- Keys
    classDef added fill:#22c55e22,stroke:#22c55e,stroke-width:2px
    classDef noteBox fill:#88888822,stroke:#888888,stroke-dasharray:2 2
    class Keys added
    class Reassembly added
    linkStyle 0 stroke:#22c55e,stroke-width:2px
    linkStyle 1 stroke:#22c55e,stroke-width:2px

Loading

🤖 Generated with Claude Code

https://claude.ai/code/session_01R6nqxi5a9qcENmJvrgBJF7