feat(hub): count file changes and headless users server-side

[ssowonny]

TL;DR

• DAU and "how many files changed" were both undercounting: an agent syncing all day never loads a page, so the frontend's tracker never saw it.
• One files_changed event now fires from every hub write door — sync, upload, remove, restore — with puts/deletes counts.
• Counts only ops the hub hasn't stored before. A device re-PUTs its whole journal every cycle, so the naive version would re-report its entire history every 10 seconds.
• Same distinct_id as the browser tracker (account email), so one person on a laptop and a browser is one user, not two.
• No new dependency and no new config: a hub with no analytics key still contacts nobody.

Why

frontend/src/analytics.ts covers everything a person clicks, but it only runs in a browser. A device syncing through /store/* never loads a page, and its own instrumentation table (api/http.ts) deliberately excludes /store/* as "device replication, not something a person did". That is correct for product events and wrong for these two metrics — whatever share of the product runs headless was simply missing from both.

What lands

New internal/webapp/analytics.go:

• capture(email, event, props) — one JSON POST to Endpoint() + "/i/v0/e/", on its own goroutine, 10s client timeout, first failure logged then silent. Returns immediately when Analytics.Key is empty.
• captureChange(r, source, puts, deletes) — the single funnel every write path calls, so the change count is one event rather than a per-route set that silently misses whichever route someone forgets to add.
• countOps(ops, storedMax) — new ops only, split on journal.KindDelete.

Call sites: store.go (sync), upload.go (relay + direct commit), remove.go (deletes), restore.go (a restore writes a put op like any other edit).

Properties are puts, deletes, source (sync | browser), project. No path, no file name.

The counting

Journals are PUT whole every cycle — journalKeepsItsOps exists precisely because the body must be a superset of what's stored. So len(ops) per PUT is the device's entire history, every 10 seconds, forever.

journalKeepsItsOps was already fetching and parsing the stored journal for the append-only check and discarding the sequence numbers. It returns storedMax now, so the new-op count costs no extra read. Blob PUTs are deliberately not change events: content-addressed storage skips a blob it already holds, so blob writes undercount edits while ops are exact.

Why no SDK

posthog-go would land in go.mod and ship inside every self-hoster's binary — the exact thing the frontend avoids by CDN-loading posthog-js only when a key is configured. AnalyticsConfig.Key is already a public write-only project token served to the browser, so this needs no new credential and no new config seam.

Reading the metrics

• File changes — Trends → files_changed → property sum of puts, second series summing deletes. Break down by source for agent vs. browser.
• DAU — Trends → Unique users, two series: $pageview and files_changed. Same email as distinct_id on both.

Tests

analytics_test.go, two cases aimed at how this rots silently:

• Push two ops → puts: 2. Re-push the same journal plus one delete → puts: 0, deletes: 1, not 2 again. Re-push unchanged → no event at all. Also asserts distinct_id, since a mismatch there splits one person into two users and nothing else would notice.
• Key empty, host set → zero requests.

Verified: go build ./..., go vet ./..., full internal/webapp suite (225s), full internal/webapp suite under -race -timeout 40m (1429s, clean), plus journal/syncer/store. Note the suite exceeds go test's default 10m deadline under -race.

Architecture changes

architecture/webapp-server.md: new productAnalytics class (the server-side emitter, analytics.go) composed into Server, reading AnalyticsConfig and fed by journalDoor. journalDoor.journalKeepsItsOps gains a second return, storedMax.

✅ added · ❌ removed (strikethrough) · unmarked = unchanged

flowchart TB
    Server["Server"]
    DeviceRegistry["DeviceRegistry"]
    journalDoor["<div style='text-align:left'><b>journalDoor</b><br/>&lt;&lt;Server, /api/p/id/store/*&gt;&gt;<br/>ownJournal(key) whose journal is this<br/>journalOps(key, spooled) parse + validate<br/>opsNameTheirAuthor(ops) whose name<br/><span style='background:#22c55e55;padding:0 4px;border-radius:3px'>✅ journalKeepsItsOps(ctx, be, key, ops) ok + storedMax</span></div>"]
    AnalyticsConfig["<div style='text-align:left'><b>AnalyticsConfig</b><br/>+Key string<br/>+Host string<br/>+Endpoint() string</div>"]
    productAnalytics["<div style='text-align:left'><b>productAnalytics</b><br/>&lt;&lt;Server, analytics.go&gt;&gt;<br/>capture(email, event, props) one POST, own goroutine<br/>captureChange(r, source, puts, deletes) files_changed<br/>countOps(ops, storedMax) new ops only</div>"]
    Server -- "AnalyticsConfig" --> AnalyticsConfig
    Server -- "<span style='background:#22c55e55;padding:0 5px;border-radius:3px'>✅ every write door emits files_changed</span>" --> productAnalytics
    productAnalytics -. "<span style='background:#22c55e55;padding:0 5px;border-radius:3px'>✅ Key gates it, Endpoint() addresses it</span>" .-> AnalyticsConfig
    journalDoor -. "<span style='background:#22c55e55;padding:0 5px;border-radius:3px'>✅ storedMax tells this cycle's ops from the whole history</span>" .-> productAnalytics
    Server -- "/store/* is the only way a device writes" --> journalDoor
    journalDoor -. "OwnerOf gates the journal key" .-> DeviceRegistry
    classDef added fill:#22c55e22,stroke:#22c55e,stroke-width:2px
    classDef removed fill:#ef444422,stroke:#ef4444,stroke-width:2px,stroke-dasharray:4 3
    classDef noteBox fill:#88888822,stroke:#888888,stroke-dasharray:2 2
    class productAnalytics added
    linkStyle 1 stroke:#22c55e,stroke-width:2px
    linkStyle 2 stroke:#22c55e,stroke-width:2px
    linkStyle 3 stroke:#22c55e,stroke-width:2px

Loading

Known gaps

• No batching or retry — one POST per sync cycle that carried new ops. A daemon only pushes when it has ops, so an actively-editing agent is ~6 events/min and an idle one is zero. Revisit if volume bites.
• A stored journal the hub cannot parse resets storedMax to 0, over-counting that one push. Rare, and the alternative is reading the object twice.

🤖 Generated with Claude Code