feat(sync): gzip the sync wire, without touching what a hash means (BEA-144)

[ssowonny]

TL;DR

• Nothing on the sync wire was compressed. Text now crosses it far smaller — the e2e measures 13.4× push / 16.0× pull on its corpus; the bar the test enforces is ≥2.5×.
• The pull half is free for devices that already exist: net/http has always sent Accept-Encoding: gzip and inflated the answer itself, so deploying the hub alone wins it, no client update.
• Push had to be negotiated (sign() now answers accept_encoding), because a gzip body posted to an old hub would be stored under the sha256 of its plaintext.
• A compressed PUT is inflated above spool and bounded at 256 MiB — otherwise a 1 MB body becomes an arbitrary hub-side disk write before any quota check can run.
• Rebased onto delta sync (Delta sync: large files move as content-defined chunks #161) — the work this was sequenced behind. chunks/ and manifests/ ride the same compressed wire, and the old-binary claim is now proven against the real pre-compression binary rather than argued.

flowchart TB
    Wire["<b>one gzip switch on /store/*</b>"]
    Pull["<div style='text-align:left'><b>PULL</b> hub -&gt; device<br/><br/>hub gzips the response<br/>net/http ALREADY sends Accept-Encoding: gzip<br/>and inflates transparently<br/><br/><b>no client change - old binaries win on deploy</b></div>"]
    Push["<div style='text-align:left'><b>PUSH</b> device -&gt; hub<br/><br/>sign() must advertise accept_encoding<br/>hub inflates BEFORE spool -&gt; hash -&gt; quota<br/>inflate is bounded (256 MiB)<br/><br/><b>old hub advertises nothing -&gt; client sends raw</b></div>"]
    Wire --> Pull
    Wire --> Push
    classDef free fill:#22c55e22,stroke:#22c55e,stroke-width:2px
    classDef work fill:#f59e0b22,stroke:#f59e0b,stroke-width:2px
    class Pull free
    class Push work

Loading

Everything is a transport concern. Hashes, the storage layout and the journal format are all still over the uncompressed bytes — the hub stores plaintext, bills plaintext, and inflates before it hashes.

What can't break

Invariant	Why it still holds
Content addressing	The hub inflates above spool, so the sha it checks is the plaintext's. A gzip body decoding to the wrong bytes still gets content does not hash to its key.
One writer per journal	ownJournal, journalOps, opsNameTheirAuthor and journalKeepsItsOps all run below the inflate, on the plaintext, untouched. A gzipped truncating journal push still 409s.
Blobs before the journal	Ordering is in syncer.push; nothing here moves it.
Journal format	No Op field, no journal.Less, no Replay.
Presigned direct-to-storage	putDirect stays raw — there is no hub in that path to inflate, so compressing it would corrupt content addressing at rest.

What I'm accepting

• Chunked request framing on a compressed push. The compressed length isn't knowable up front, so ContentLength is cleared and the PUT goes out chunked. spool already handles that (it measures the body rather than believing a header), but a self-hosted deployment with an intermediary proxy now sees chunked bodies where it saw sized ones.
• maxInflatedPut = 256 MiB is a precedent, not a measurement — same caveat maxImportBlob carries. It applies only when Content-Encoding is present, so no honest raw push that works today can start failing.
• httpBackend.do must never set Accept-Encoding. If it ever does, Go stops inflating, the hub keeps answering Content-Encoding: gzip, and every blob fails its sha check while looking like a corrupt hub. There's a comment at do saying so.

The old binary, for real

buildOldBinary is pinned at 33ca0ca — the commit before delta sync, which is also the commit before this. So it is a genuine pre-compression client, and TestCompressionE2E_OldBinaryPullsCompressedAndPushesRaw runs it against this hub:

• its pull is compressed with no client change — 19,964 bytes received across its entire session (sign-in, listings, journal and 40 blobs) for a 148 KB corpus, 7.4×. With compression off the same session receives 170,930 bytes, i.e. 0.87×.
• its push stays raw, asserted on the headers the hub actually received rather than on a byte count, so a daemon tick can't change the answer.

TestCompressionE2E_ChunksAndManifestsOverGzip covers the two key classes chunking added. The manifest is the one that matters: it is never presigned, so it always takes the relay path that compresses, and its write-once compare and chunks-exist gate read the spooled plaintext below the inflate.

One thing worth knowing about how this is measured: each watched client gets its own proxy, rather than a mark taken on a shared counter between phases. bdrive init starts a daemon, so a mark placed after it measures only what the daemon hadn't already done — the first draft of this test reported a 282× pull ratio and was proving nothing.

Deviations from the reviewed plan

1. countingHub is duplicated, not shared. The plan (correctly) rejected countingBackend for the ratio test — it wraps a remote.Backend above httpBackend and counts plaintext by construction. internal/syncer/compress_e2e_test.go has its own hub-level counter because it measures a real Session.Cycle(), one package over from webapp's; sharing it would mean exporting a test helper across packages.
2. The old-hub half is a unit test, not a CLI e2e. An old hub is exactly a sign() answer with no accept_encoding, which TestPushCompressesOnlyWhenTheHubAdvertisesIt drives in four variants (absent, empty, a codec we don't speak, gzip). Building a whole binary to assert one absent JSON field would be slower and prove less. The old-client half is the real thing — see below.

What was run

• Rebased onto origin/main (delta sync, Delta sync: large files move as content-defined chunks #161). Two conflicts, both additive-adjacent: the store.go import block, and the inflate bound landing beside the new chunk/manifest key checks. The bound goes first — it must refuse before anything reads the body.
• go build ./... && go vet ./... && go test ./... — green, including delta sync's own e2e rows now running over the compressed wire.
• npm run e2e (Playwright, internal/webapp/frontend) — 170 passed, 1 skipped.
• Sanity-checked that every new test actually fails without the feature: with the probe forced to false, the syncer ratio test reports 0.95×, the old-binary e2e drops to 0.87×, and the chunked push sends nothing compressed.
• No frontend change, so no npm run build and no screenshots — the browser never touches /store/*.

Architecture changes

architecture/webapp-server.md: one new type, wireCodec (internal/remote/compress.go) — the probe both legs share, plus the putPlan.AcceptEncoding field that carries the negotiation — and two new dependency edges onto it, from Server (the hub's GET/list/PUT handlers) and from Backend (httpBackend's relayed push). Nothing was removed. The journalDoor and countingWriter notes gained sentences about inflate-above-spool and counter-outside-gzip; those are behavioral, not structural.

✅ added · ❌ removed (strikethrough) · unmarked = unchanged

flowchart TB
    Server["<div style='text-align:left'><b>Server</b><br/>/api/p/&lt;id&gt;/store/*</div>"]
    RemoteSource["<div style='text-align:left'><b>RemoteSource</b><br/>+Backend remote.Backend</div>"]
    Backend["<div style='text-align:left'><b>Backend</b> &lt;&lt;interface&gt;&gt;<br/>+Put +Get +List +Exists +Close<br/>impls: local, s3, gcs, httpBackend</div>"]
    PutSigner["<div style='text-align:left'><b>PutSigner</b> &lt;&lt;interface&gt;&gt;<br/>+SignPut(ctx, key, size, ttl)</div>"]
    journalDoor["<div style='text-align:left'><b>journalDoor</b> &lt;&lt;store.go&gt;&gt;<br/>ownJournal(key)<br/>journalOps(key, spooled)<br/>opsNameTheirAuthor(ops)<br/>journalKeepsItsOps(ctx, be, key, ops)</div>"]
    countingWriter["<div style='text-align:left'><b>countingWriter</b> &lt;&lt;quota.go&gt;&gt;<br/>+Write(p) n<br/>+n int64</div>"]
    wireCodec["<div style='text-align:left'><b>wireCodec</b> &lt;&lt;internal/remote, compress.go&gt;&gt;<br/>+Compressible(r) rejoined, worth, err<br/>+AcceptsGzip(req) bool<br/>putPlan.AcceptEncoding []string</div>"]
    Legs["Pull needs no negotiation:<br/>net/http sends Accept-Encoding: gzip<br/>and inflates transparently.<br/>Push compresses only when sign()<br/>advertised accept_encoding.<br/>putDirect stays raw - no hub in the path."]
    RemoteSource -- "Prefixed(Root, projectID)" --> Backend
    Backend -. "optional capability" .-> PutSigner
    Server -- "/store/* is the only way a device writes" --> journalDoor
    Server -- "every bytes-out route that bills" --> countingWriter
    Server -- "<span style='background:#22c55e55;padding:0 5px;border-radius:3px'>✅ gzip on /store GET+list, inflate above spool on PUT</span>" --> wireCodec
    Backend -- "<span style='background:#22c55e55;padding:0 5px;border-radius:3px'>✅ httpBackend gzips a relayed PUT when sign() allows</span>" --> wireCodec
    wireCodec -.- Legs
    classDef added fill:#22c55e22,stroke:#22c55e,stroke-width:2px
    classDef noteBox fill:#88888822,stroke:#888888,stroke-dasharray:2 2
    class wireCodec added
    class Legs noteBox
    linkStyle 4 stroke:#22c55e,stroke-width:2px
    linkStyle 5 stroke:#22c55e,stroke-width:2px

Loading

Closes BEA-144.

Build session

cd $(git worktree list | grep bea-144 | awk '{print $1}') && claude --resume 9b13a152-adfa-4ebd-a8ef-75ea7b92e7e0

(only works on this machine)