Inspectors fill the same form on different tablets with no connectivity, then reconnect. Every edit survives, every merge is deterministic, and the disagreements a clock has no business settling reach a human instead. Across 200 randomised offline-edit schedules this design loses nothing — while the three sync designs it is argued against, replaying the identical op log, lose 1,255–1,340 pieces of inspection content and silently settle 279–436 professional disagreements.
Offline sync is a problem everyone thinks they have solved with updated_at and a
MAX(), and the reason that keeps shipping is that its failures are invisible. The
crew's tablet says the note is there. The report does not have it. Nobody knows when it
went missing, or which of the two inspectors was overruled by a clock that happened to be
nine minutes fast.
So this project is built around making that class of failure visible and measurable:
-
Three oracles, not one "did it work". Data loss on an accumulating field, a reverted correction, and a swallowed disagreement are three different failures with three different causes. They partition the space; none subsumes another. Every one is keyed on op identity, never on value equality — an inspection form's vocabularies are small (severity is 1–5, status is one of three strings), so two inspectors writing the same value is routine, and a value-keyed check both invents failures and hides real ones.
-
Three control arms that are not strawmen. The strongest one — per-field ops, causal ordering, last-writer-wins on every field — is what a careful engineer builds, and it commits zero causal violations. It still loses 1,338 pieces of content, because causal ordering gets you a consistent answer, not the right one about a form whose fields mean different things. If any arm ever stops failing, CI fails on that rather than quietly reporting a tie.
-
Every guarantee is mutation-tested. 22 mutations each remove exactly one thing the design claims. All 22 are caught, each by the test named as its guard — because "something went red" is not the same as "the guarantee is covered".
-
The real client is driven in a real browser. With 124 Python tests green and the mutation check at 21/21, five genuine bugs were still sitting in the PWA — including one where a background sync wiped a half-typed note. A system about never losing a field observation was losing them in the text box. See What driving a real browser found.
Built and measured. Every number below comes from a committed script, reproducible with the commands in Reproduce. No network, no secrets, no services — SQLite and a browser.
| Evidence | Result |
|---|---|
| Test suite (unit + property, real SQLite, real HTTP) | 127 passed |
| Browser suite — the real PWA in Chromium, network genuinely down | 7 passed (found 5 real bugs) |
| Property suite: 8 properties × randomised schedules | ops created 5,942 → durable 5,942, unsynced 0 |
| Schedules that failed to converge | 0 / 200 |
This design (oplog) — content lost / corrections reverted / disagreements swallowed |
0 / 0 / 0 |
| Control: whole-record last-writer-wins | 1,255 / 125 / 436 |
| Control: per-field, wall-clock ordered | 1,340 / 86 / 308 |
| Control: per-field, causal, LWW everywhere (the honest baseline) | 1,338 / 0 / 279 |
| Comparison still discriminating (each control still fails its oracle) | holds |
| Interrupted upload: every boundary × 2 failure modes | 13 / 13 resume byte-exact, 0 chunks wasted |
| Negative control — a client trusting its own progress | refused, as designed |
| 20-device reconnection storm | converged 1.6 s (bound 60 s), 1,500 / 1,500 ops durable, 0 unsynced |
| Storm vs. serial control | 1.4× — concurrency is real, not a serialised queue |
| Schema migration: app v1 + v2 + v3 → one backend | 13 ops in log, 2 deferred (recoverable), 0 lost |
| Negative control — the same fold with migration off | produces severity 3 instead of 5 |
| PWA ↔ Python parity (op ids, canonical JSON, HLC encoding) | 5 assertions, byte-identical digests |
| Mutation check: remove one guarantee at a time | 22 / 22 caught, each by its named guard |
Each control arm fails a different oracle, which is what makes the comparison informative rather than a scoreboard. All four arms fold the same op log — the only variable is the merge design.
| content lost | corrections reverted | disagreements swallowed | schedules with ≥1 failure | |
|---|---|---|---|---|
oplog (this design) |
0 | 0 | 0 | 0 / 200 |
record_lww |
1,255 | 125 | 436 | 200 / 200 |
field_wallclock |
1,340 | 86 | 308 | 200 / 200 |
field_hlc_lww_everywhere |
1,338 | 0 | 279 | 200 / 200 |
What each one is, and why it is a real design rather than a punching bag:
record_lww— sync whole records, newest wins. What "add anupdated_atcolumn" gets you. Two inspectors editing different fields of one record destroy each other's work, because a snapshot overwrites fields its author never touched.field_wallclock— per-field merge ordered by the device's own clock. Fixes the granularity bug and keeps a real one. Concretely, at seed 4 it reverts a GPS fix from ±37.8 m back to ±57.3 m, because the worse reading came from a device whose clock ran ahead.field_hlc_lww_everywhere— per-field, causally ordered, LWW everywhere. The one that matters. Zero causal violations: no correction is ever reverted. It loses 1,338 pieces of content anyway — 446 notes overwritten instead of appended, 483 photo ids and 409 defect tags dropped — and settles 279 pass/fail disagreements by microseconds.
Where the loss lands is the argument for per-field policy:
| arm | notes | photo_ids | defect_tags |
|---|---|---|---|
record_lww |
423 | 445 | 387 |
field_hlc_lww_everywhere |
446 | 483 | 409 |
oplog |
0 | 0 | 0 |
Full table, per-field breakdown and the discriminating-check output: results/arm_comparison.md.
uv run fieldsync demo runs this. Okafor's tablet clock is nine minutes fast — the
detail that decides the outcome under any wall-clock design.
08:10 Okafor (offline): spalling on pier 2, severity 4, FAIL, ±45 m GPS fix
08:11 Reyes (offline): corrosion, severity 2, PASS, ±6 m GPS fix, gauge 12.4
11:40 Reyes reaches the road and syncs
12:05 Okafor syncs, and now sees Reyes's entries
12:10 Okafor, having seen both grades, revises severity 4 → 3
| field | merged | what the policy did |
|---|---|---|
severity |
3 | Okafor's revision was made while looking at both grades, so it stands. A plain max-over-all-ops would have ratcheted to 4 forever, with no way to correct a mistyped grade. |
status |
held for review | pass vs. fail is a professional disagreement. Under any LWW design Okafor's FAIL wins purely because his clock is fast, and nobody is ever told Reyes said PASS. |
notes |
3 entries | every observation kept, in causal order |
defect_tags |
corrosion, spalling |
neither inspector's finding displaced the other's |
gps |
±6.0 m | Reyes's better fix wins over Okafor's later ±45 m one — best accuracy, not latest reading |
All three replicas agree afterwards. Then a supervisor resolves status → fail, and the
conflict closes.
tablet (PWA, IndexedDB) tablet (PWA) tablet (CLI/py)
┌───────────────────┐ ┌───────────────────┐ ┌───────────────────┐
│ local oplog │ │ local oplog │ │ local oplog │
│ outbox ──┐ │ │ outbox │ │ outbox │
│ same merge code │ │ │ │ │
└──────────┼────────┘ └─────────┬─────────┘ └─────────┬─────────┘
│ one op per FIELD edit, id = sha256(content) │
└──────────────────────┬─────────────────────────────-┘
▼ POST /sync (idempotent)
┌──────────────────────────────────────────────────────────────────────────┐
│ server │
│ │
│ 1. INSERT OR IGNORE into the oplog, COMMIT ← durable BEFORE │
│ 2. migrate → validate → project interpreted │
│ anything uninterpretable is DEFERRED, never refused │
│ │
│ state = fold(oplog) per field, per declared policy │
│ lww · max_wins · append · or_set · g_set · best_accuracy · │
│ manual_review → emits a Conflict, never picks │
│ │
│ media: chunk bitmap is the server's own — a resume asks, never assumes │
└──────────────────────────────────────────────────────────────────────────┘
The five decisions that carry the design:
One op per field, never per record. A record-granular sync overwrites fields its author never touched. Per-field ops make that class of loss structurally impossible — there is no op that says "the record is now this".
An op's id is the hash of its content. Retransmission is therefore free of
consequence: the same edit through five flaky reconnects is the same id five times, and
the primary key collapses it. No dedupe table, no reasoning about retries. The PWA
computes the same hash, verified byte-for-byte in tests/test_pwa_parity.py.
Durable before interpreted. ingest commits, then migrates and validates. Every
later failure — a future schema, an unknown field, an invalid value — is a deferral,
not a rejection, and reproject() recovers it after a deploy with no action from devices
that may still be offline. The tempting implementation validates first and returns 400,
throwing away data the device has already cleared from its outbox on the strength of that
response.
Ops carry declared causality (deps), not just timestamps. An op names the values it
supersedes — what the editor was looking at. That is the only sound way to tell "someone
considered this and changed it" from "two people wrote without seeing each other", and
max_wins, or_set and manual_review all need exactly that distinction. It cannot be
inferred from timestamps: in a total order a concurrent op can carry a smaller timestamp,
and mistaking that for "was observed" silently discards it.
Merge is a pure fold over the complete, unordered op set. So arrival order cannot
change the answer — asserted directly by shuffling the log and re-folding
(test_merge_is_order_independent).
Even a human's decision is an op. Resolving a manual_review conflict writes an op that
supersedes every candidate, rather than recording the choice in a side table. So it syncs to
every device like any other edit, is attributed to the person who made it, and reopens by the
ordinary frontier rule when a value they never saw arrives.
Which field gets which policy, and why: docs/MERGE_POLICIES.md.
A field photo is megabytes over a link that drops. The study kills the link at every chunk boundary and checks the assembled file's SHA-256 — not its length.
Two modes, because they are genuinely different problems:
before_send— the link died between chunks. The pending chunk never landed.lost_ack— the chunk did land and its response was lost. The client cannot tell these apart.
All 13 cases resume to a byte-identical file, with 0 chunks of wasted transfer — in
both modes. Zero waste in lost_ack is the result worth pausing on: the client has no way
to know whether its write landed, but it does not need to, because it resumes from the
server's missing list. A client-owned progress record cannot reach zero there — it must
either resend the ambiguous chunk, or assume it landed and leave a hole.
That hole is the negative control:
| negative control | outcome |
|---|---|
a client resuming from its own last_sent + 1 (skipping the in-flight chunk) |
server refused to finalise — missing 1 chunk(s): [3], no good file produced |
| a chunk corrupted in flight | rejected at the door, chunk 0 digest mismatch, left unrecorded so the resume refetches it |
The first one matters most. It is the design most people write first, and it fails in the least visible way available: the file ends up exactly the right length with a hole in it, so every length check passes. If that case could also produce a good file, the other 13 rows would prove nothing. Details: results/uploads.md.
Twenty devices, 15 form-fills each (1,500 ops) accumulated with no connectivity, then all reconnecting inside the same moment against one uvicorn server over real HTTP — 19 private records plus one contended record that all twenty write concurrently.
| measure | result |
|---|---|
| time to full convergence | 1.6 s (bound 60 s in CI) |
| slowest single device | 0.85 s |
| ops durable / created | 1,500 / 1,500 |
| ops still unsynced | 0 |
| devices diverged from server | 0 |
| serial control — same work, one device at a time | 2.28 s → 1.4× speedup |
Two things make this number mean something. The serial control supplies a denominator: without it, "converged in 1.6 s" could be genuine concurrency or just a fast disk. And the timing is only credited alongside three assertions checked after the run — every op durable, every outbox empty, every device's projection byte-identical to the server's. A storm that finishes quickly by dropping ops is the fastest possible result.
Timings are wall-clock on Darwin arm64, Python 3.12.13. CI asserts the bound, never the
figure. results/storm.md.
Crews update when they have signal, so v1, v2 and v3 tablets sync into one v3 backend simultaneously. Three silent failures, each checked:
| failure it would cause | result |
|---|---|
a field rename (remarks→notes) drops the old entry |
the 1998 remark survives |
a legacy severity=3 — that scale's maximum — reads as "moderate" on the new 1–5 scale |
merges to 5 |
| a migrated GPS fix with unknown accuracy beats a measured ±4 m fix | the ±4 m fix wins |
The middle one is the cautionary case: nothing errors, no type is violated, the record still validates — a critical defect has simply been downgraded. Only asserting the value catches it, which is why the study runs a negative control that folds the same log with migration off. It produces severity 3 instead of 5. If it produced the same answer, the assertion would be a tautology.
Forward compatibility both ways: a v4 op from a crew that updated early is stored and deferred, not rejected. A v1 tablet shows a partial record for fields and scales it cannot interpret — migrations run forward only, and there is no defensible way to render a v3 grade of 5 on a scale whose maximum is 3 — while still holding and forwarding every one of those ops. results/migration.md.
22 mutations, each removing exactly one guarantee. All 21 caught, each by the test named as its guard.
| guarantee removed | caught by |
|---|---|
HLC.observe stops folding in the remote timestamp |
test_observe_makes_the_next_stamp_beat_the_observed_one |
| the op id stops covering the value | test_two_edits_differing_only_in_value_are_distinct_ops |
| the server rejects an uninterpretable op instead of deferring it | test_an_op_is_durable_before_it_is_interpreted |
| the v2→v3 severity migration becomes the identity | test_severity_rescale_is_not_the_identity |
max_wins becomes a ratchet |
test_max_wins_honours_a_sequential_downgrade |
| OR-Set removal keys on values, cancelling a concurrent re-add | test_or_set_concurrent_readd_survives_a_removal |
manual_review resolves silently by clock |
test_concurrent_verdicts_are_held_not_resolved |
unknown GPS accuracy sorts as perfect (acc or 0) |
test_unknown_gps_accuracy_is_treated_as_worst_case |
| the device clears its whole outbox on any response | test_ack_only_clears_the_ids_the_server_named |
| an old client prunes ops it cannot read | test_an_old_client_still_forwards_ops_it_cannot_read |
| an upload resumes from the client's own index | test_an_upload_survives_an_interruption_over_http |
| a resolution sorts before the verdict it settles | test_a_resolution_outranks_a_candidate_from_a_fast_clock |
| …and 10 more | results/mutation_check.md |
Two ways this check could lie to itself, both closed: a mutation whose search string
stops matching would leave the file unchanged and read exactly like a coverage gap (so
every mutation asserts it changed the file, and a test in the ordinary suite fails the
moment a target drifts), and a same-length edit could let Python reuse a stale .pyc after
restore (so __pycache__ is purged every run and PYTHONDONTWRITEBYTECODE is set). The
suite must also be green before any mutation — from a red baseline, "everything failed"
reads as "every mutant caught".
The Python suite was at 124 passing tests, the mutation check at 21/21, and every gate green — while five genuine defects sat in the browser path. None of them were visible from Python, and four of the five are silent in exactly the way this project is about.
| # | the bug | why no API test caught it |
|---|---|---|
| 1 | A resolution never reached the devices. It lived in a server-side table, so every tablet kept showing "needs a human" forever. | The server's own projection looked correct. Only a device's projection was wrong, and only the browser had one. |
| 2 | The resolver never received its own resolution. since() excludes the caller's own ops — sound for edits, wrong for an op the server authors on the reviewer's behalf. Every other tablet cleared; theirs did not. |
Needs two independent clients, one of which resolves. |
| 3 | The app rendered a blank page on a genuine offline reload. The shell was cached, /schema was not, and app.js cannot build one control without the field list. |
The app's own "go offline" toggle does not catch it — the schema is already in memory by then. Only a real reload with the network down. |
| 4 | A record URL was unreachable offline. The service worker cached / but a navigation to /?record=BR-1042 is a different cache key, so the reload failed with ERR_FAILED. |
Requires the browser's HTTP cache semantics. ignoreSearch: true fixes it. |
| 5 | A background sync wiped a half-typed note. render() rebuilds the form with replaceChildren, so another device's ops arriving mid-sentence destroyed uncommitted text. |
Data loss caused by a UI decision rather than the sync protocol — invisible to any test of the protocol. |
Every one is now pinned by a test in tests/test_pwa_browser.py, and each of those tests
is verified to fail when its fix is reverted. Bug 5 is the one worth dwelling on: a system
whose whole purpose is never losing a field observation was losing them in the text box,
and no amount of oplog correctness would have shown it.
Bug 1's fix is also the better design. A resolution is now an op that supersedes every candidate rather than a side table, so it syncs like any other edit, is attributed to the person who made it, sits in the same append-only log as the readings it settled — and reopening falls out of the ordinary frontier rule instead of needing a special case. When a fourth inspector's concurrent verdict arrives, the conflict reopens between their verdict and the reviewer's decision, which is the correct pair.
uv run fieldsync serve serves an installable, dependency-free capture app at
http://127.0.0.1:8412/. No framework, no build step: a field app that needs a toolchain
to reason about is one nobody can debug on a tailgate.
Press go offline and keep working. Edits queue in IndexedDB (not localStorage —
which is synchronous, ~5 MB, and gets cleared under storage pressure on mobile Safari,
i.e. "the crew's afternoon disappeared when the OS needed room"). Press go online and
watch the log: pushed 7, accepted 7, received 3. Sync twice and the second says
dup 7 — nothing duplicated.
The app reimplements the HLC format and the op-id hash in JavaScript, because a browser
cannot import the Python module. That is the riskiest thing in the repo, so
tests/test_pwa_parity.py runs the shipped pwa/app.js under Node and compares digests
with Python's across 7 adversarial cases (non-ASCII, unsorted object keys, dots in the
device id, floats, empty strings, unsorted deps). Drift there would make every browser
retry a new op — one note appearing three times in a report — and would break every
deps reference so that sequential edits looked concurrent. Neither is visible in a test
of either half alone. The check is verified to actually fail: narrowing the HLC pad width
or dropping key sorting both break it.
The service worker caches the app shell but never /sync, /pull or /record. There
is no acceptable stale answer to "what is the current state of this record", so there is no
fallback — offline, the app works from its own local log, which is the entire design.
uv sync --extra dev
uv run fieldsync demo # the two-inspector walkthrough
uv run fieldsync fields # every field's policy and its rationale
uv run fieldsync serve # the PWA + sync backend on :8412
uv run pytest tests properties # 127 tests + 7 browser (needs the browser extra)
uv run fieldsync arms --schedules 200 # the four-arm comparison
uv run fieldsync upload-study # 13 interruption points
uv run fieldsync migration-study # v1 + v2 + v3 → one backend
uv run fieldsync storm --devices 20 # the reconnection storm
uv run python scripts/mutation_check.py # slow: runs the suite 22 times
uv run fieldsync report # regenerate every file in results/
uv run fieldsync check --seed 42 # replay one schedule, all oraclesTo run exactly what CI runs — parsed from the workflow YAML, not retyped, so "it passes locally" is a statement about the real workflow:
python scripts/ci_local.py --list
python scripts/ci_local.pysrc/fieldsync/
hlc.py hybrid logical clocks — the causal ordering key
schema.py the form, each field's merge policy + rationale, the migration chain
ops.py one content-addressed op per field edit
policies.py the seven merge functions; pure folds over one field's ops
client.py a device: local oplog, outbox, projection — the same merge code
store.py the server's durable oplog, projection, deferral, reprojection
media.py chunked resumable upload; the server owns the progress bitmap
transport.py the sync loop, and a link that can be told to fail on a schedule
server.py the HTTP surface (FastAPI) and the PWA
arms.py the three control designs this one is argued against
simulate.py randomised offline-edit schedules + the three oracles
study.py the arm comparison and its discriminating-check
pwa/ the capture app: index.html, app.js, sw.js, manifest
properties/ 8 property tests over Hypothesis-generated schedules
tests/ 134 tests, incl. PWA↔Python parity, a Chromium suite, drift guards
bench/ storm, interrupted uploads, schema migration
scripts/ mutation_check.py, ci_local.py
results/ every committed number, regenerable with `fieldsync report`
docs/ MERGE_POLICIES.md — which field merges how, and why
RUNBOOK.md operating it: conflicts, deferred ops, stuck uploads, rollouts
- Concurrent writes to an
lwwfield still tie-break on the physical clock. HLC guarantees a correction beats what it corrects; it cannot order two writes that never saw each other. That is whystatusandmeter_readingaremanual_reviewrather thanlww, and it is a policy decision, not a fix. - The oplog never compacts. State is a fold over the log, and the log is the record. A long-lived record grows without bound. Compaction is the only operation in this system that can destroy data, so it would need its own design review rather than a default.
- Writes serialise behind one lock. Correct for SQLite, and the storm shows it is not the bottleneck at 20 devices. At a scale where it became one, the sequence allocation is the piece to move — not the lock to remove.
- Migrations run forward only. An old tablet shows a partial record for newer fields and scales. Rendering a v3 grade on a v1 scale would mean guessing, so it does not.
manual_reviewconflicts need a human, and a record can sit unresolved indefinitely. There is no escalation or timeout; adding one would mean choosing a default, which is the thing this design refuses to do.