feat(analytics): recover gateway and first-party diagnostic retention

README.md

@@ -94,7 +94,12 @@ registered event name, its exact allowlisted property set, the event's declared
 privacy level, and sufficient explicit consent. Unknown events, extra
 properties, raw text, and mismatched consent or privacy classifications are
 rejected before storage. The versioned registry and its focused tests live in
-`workers/events/src/eventContract.ts`.
+`workers/events/src/eventContract.ts`. Revision 2 is generated from
+`scient-desktop/packages/scient-analytics/src/wireContract.ts`, with a shared
+90-case conformance fixture for all 45 registered events. Do not edit that copy
+independently; the desktop analytics document owns regeneration instructions.
+New events add an optional bounded `contractRevision`; legacy revision-1
+payloads remain supported. Deploy this validator before releasing new producers.
 
 The public desktop endpoint is disabled unless the Cloudflare runtime variable
 `DESKTOP_INGESTION_ENABLED` is exactly `true`. It also requires a random
@@ -103,20 +108,44 @@ limit per opaque installation ID. It does not use or store an IP address as a
 rate-limit key. Keep the variable absent or false during preparation and use it
 as the immediate ingestion kill switch during a selected-user rollout.
 
+Desktop forwarding to PostHog has its own gate,
+`DESKTOP_POSTHOG_EXPORT_ENABLED`, also false by default. Turning off ingress
+does not drain queued data; turning off export prevents queued desktop copies
+from being forwarded. Neither gate affects existing website forwarding.
+
 `POST /v1/installations/delete` authenticates an installation, deletes its D1
 events, consent, identity links, and identity record, and queues the matching
 PostHog person and historical-event deletion by opaque distinct ID. A request from an installation that has never uploaded is
 acknowledged idempotently so the desktop can still clear local data and rotate
-its anonymous identity. The scheduled Worker retries a failed PostHog submission
-up to ten times and records a blocked queue item for operator review instead of
-silently claiming success. Do not describe remote deletion as complete while
-the queued PostHog state remains pending or blocked.
+its anonymous identity. A minimal opaque-ID/authentication-hash tombstone blocks
+late uploads from recreating deleted history. Migration 0006 preserves legacy
+erasure tombstones conservatively; missing legacy credentials cannot be guessed.
+No behavioral payload is retained in the tombstone.
+
+Erasure completes without PostHog only if no export was ever attempted. Otherwise
+the scheduled Worker submits/polls PostHog's person/event deletion, with bounded
+failure retries and an operator-visible blocked state. Provider verification,
+not submission, completes the gateway request. Export/deletion share a lease;
+deleted identities are tombstoned and never deliberately reused. Completion
+records PostHog's verified operation, not a synchronous transaction covering
+every ambiguous capture or a guaranteed physical-deletion deadline.
+Do not claim remote deletion is complete while its state is pending or blocked.
 
 The scheduled Worker prunes canonical raw events older than 180 days in
-bounded batches. D1 remains the source of truth for retention and delivery;
-dashboard filters are not retention controls.
-
-Website visitors, desktop installations, sessions, and future Scient accounts use separate opaque identifiers. The service-authenticated `POST /v1/identity/link` endpoint can connect a visitor or installation to an account after Scient's account service has authenticated that user. Browser and desktop clients cannot call this endpoint directly or claim an account identifier. Linking updates first-party historical events without changing the user's analytics choice; the corresponding anonymous-to-account PostHog identity event is forwarded only for product-or-higher consent.
+bounded batches of 5,000; diagnostics have a 30-day limit. An unfinished backlog
+does not count as a successful retention pass. Desktop occurrence age is also
+enforced during ingestion, export and pruning, so offline delivery does not
+reset its retention clock. D1 remains the source of truth
+for retention and delivery; dashboard filters are not physical retention controls.
+Desktop Diagnostic-class events remain only in Scient's central D1 ledger and
+are never exported to PostHog. `bun run analytics:report` exposes their bounded
+30-day aggregate breakdown alongside maintenance health; no access to a user's
+computer is required. Essential/Product-class events can be exported even when
+the user's consent level is Diagnostic. PostHog retention for those copies is
+provider-managed; its query-access window is not a physical-deletion deadline.
+Do not advertise 13-month or 30-day PostHog deletion guarantees.
+
+Website visitors, desktop installations, sessions, and future Scient accounts use separate opaque identifiers. The service-authenticated `POST /v1/identity/link` endpoint can connect a website visitor to an account after Scient's account service has authenticated that user. Desktop linking is rejected until its per-installation erasure model is qualified. Browser and desktop clients cannot claim account identity. Website linking preserves consent; the PostHog identity event is forwarded only for Product-or-higher consent.
 
 Generate binding types and validate the Worker with:
 
@@ -125,6 +154,24 @@ bun run events:types
 bun run events:typecheck
 ```
 
+The normal test suite uses synthetic records and real local SQLite migrations.
+An additional cross-repository proof is opt-in: build the exact desktop
+candidate, then run:
+
+```sh
+SCIENT_ANALYTICS_DESKTOP_ROOT=/absolute/desktop bun run test workers/events/src/desktopPipeline.test.ts
+```
+
+This test connects the built desktop worker to a loopback gateway and mocked
+PostHog exporter, checks forbidden-data removal and runtime-source metadata,
+then exercises consent, deletion, and late-replay rejection. It does not touch
+production and is intentionally skipped when no explicit desktop path is set.
+Record the desktop revision/build as well as the website revision; ordinary
+website CI alone does not qualify this cross-repository path.
+The desktop's `docs/internals/product-analytics.md` also documents a non-GUI
+Electron-runtime invocation. Use it to qualify the native SQLite/runtime
+boundary; passing under ordinary Node alone does not prove desktop packaging.
+
 Deploy the Worker only from an approved production change:
 
 ```sh
@@ -133,19 +180,36 @@ bun run events:deploy
 
 `POSTHOG_PROJECT_TOKEN`, `POSTHOG_PERSONAL_API_KEY`, and `IDENTITY_LINK_TOKEN`
 are Cloudflare Worker secrets and must never be committed. The personal key is
-used only for queued deletion and needs the narrow `person:write` scope;
+used only for queued deletion and needs the reviewed person read/write scopes
+for lookup, submission, and verification (qualify the exact provider permissions);
 `POSTHOG_PROJECT_ID` selects the project. If the project token is absent,
 ingestion continues and events remain queued in D1 for later delivery. If the
 deletion key or project ID is absent, accepted erasures remain queued in D1. If
 the identity-link token is absent, account linking returns `503` while ordinary
 ingestion continues.
 
-Before any production activation, apply the migration, deploy the reviewed
-Worker, verify that `/health` reports forwarding, deletion, rate limiting, and
-storage ready, run the D1/PostHog reconciliation command, and only
-then set `DESKTOP_INGESTION_ENABLED=true` for the approved cohort. Reversing
-that variable to false stops new desktop ingestion without changing website
-measurement.
+Before activating an owner-approved rollout:
+
+1. Apply the approved migrations and deploy the reviewed Worker with **both
+   desktop gates false**. Website Pages deployment is not Worker deployment.
+2. Verify `/health` against the exact deployed revision: required schema and a
+   recent successful retention pass are checked, but configured secrets are not
+   proof of valid permissions.
+3. Confirm the approved first-party-only diagnostic routing and truthful
+   PostHog-managed retention wording. Verify asynchronous provider erasure with
+   synthetic identifiers; an arbitrary delay or repeat-delete loop is not proof.
+4. Exercise authorized synthetic end-to-end delivery,
+   rejection, erasure, retry, retention and aggregate reconciliation. Never use
+   live researchers' records for a test or expose credentials in logs.
+5. Complete human privacy-copy/consent/cohort review. Enable the approved
+   ingress/export gates only after qualification. Packaged desktop availability
+   does not override a user's Off choice; a desktop release is still needed.
+
+Export uses stable capture UUIDs and bounded retries, with a database lease
+renewed before each outbound call. This prevents concurrent local exporters;
+it is not a provider-side transactional fence. Persisted desktop properties are
+revalidated so malformed/legacy rows cannot bypass today's privacy contract.
+`posthog_state='sent'` means capture acknowledged, not erasure settled.
 
 The identity-link token is service-to-service authority. Rotate it if it is exposed, and never embed it in website or desktop bundles:
 
@@ -158,7 +222,6 @@ After an account service has authenticated a user and obtained their opaque acco
 ```sh
 SCIENT_IDENTITY_LINK_TOKEN=... bun run identity:link \
   --account account:<uuid> \
-  --identity installation:<uuid> \
   --identity visitor:<uuid>
 ```
 
@@ -193,14 +256,39 @@ create or update ready dashboards:
 bun run analytics:dashboards --apply-ready
 ```
 
-The script never deletes dashboards or insights. D1 delivery state remains the
+Operator API requests are project-origin restricted, time/body bounded, and do
+not blindly retry ambiguous creates. Pagination is bounded. The script never
+deletes dashboards or insights. D1 delivery state remains the
 operational source of truth and should be reconciled with PostHog using
-`bun run analytics:report` before relying on a dashboard. The exact per-event
+`bun run analytics:report` before relying on a dashboard. That aggregate-only
+report includes pending/blocked deletion, exhausted or quarantined delivery,
+and missing/stale maintenance; it is not a claim of end-to-end healthy delivery.
+Maintenance rows have a status and timestamp, not an event count. The exact per-event
 D1-sent and PostHog counts can be checked without exposing the personal API key:
 
 ```sh
 bun run analytics:reconcile
 ```
 
-That command exits non-zero when the two systems disagree; pending D1 events
-are reported separately rather than counted as delivered.
+Reconciliation defaults to desktop events in the last seven days, excluding
+the newest hour, and compares the same occurrence window and deduplicated event
+IDs in both stores, excluding first-party-only diagnostics. Override it with `ANALYTICS_RECONCILE_SOURCE`,
+`ANALYTICS_RECONCILE_FROM`, and `ANALYTICS_RECONCILE_TO` (maximum 30 days).
+The one-hour delay is a reporting convention, not an erasure guarantee.
+Mismatch, pending events, outstanding deletions, and no data exit non-zero;
+an empty dashboard is not a verified pipeline.
+
+Prepared metrics count consenting installation profiles, not all people. Product
+success rates use a consistent Product/Diagnostic population and exclude terminal
+stops. Activation uses the gateway's first-observed Product cohort anchor,
+excludes unknown legacy anchors, requires ordered steps, and reports immature
+cohorts separately. Retention needs complete follow-up windows. Billing allowances
+are not hardcoded as current facts. Exact HogQL execution and installed-dashboard
+behavior still require authorized provider-side qualification.
+
+Scientific `source-import` outcomes have item-attempt grain, not batch grain.
+Saved source-store results count as completions even when later batch cleanup
+fails; duplicate/possible-match skips are reported separately as
+`scient.operation.skipped` and do not qualify for meaningful-use metrics.
+Retries are new attempts, not a second completion of an already saved source.
+These events do not claim batch-conversion or human-review coverage.

migrations/0006_analytics_readiness.sql

@@ -0,0 +1,69 @@
+-- A minimal authenticated tombstone prevents late uploads recreating deleted data.
+-- It deliberately has no FK to the identity record, which is erased.
+CREATE TABLE analytics_deleted_installations (
+  installation_id TEXT PRIMARY KEY,
+  deletion_token_hash TEXT NOT NULL,
+  request_id TEXT NOT NULL UNIQUE,
+  requested_at TEXT NOT NULL
+);
+
+-- Earlier releases erased authentication history. Keep those IDs blocked rather
+-- than treating a subsequent upload as a new installation. The sentinel cannot
+-- match any SHA-256 token; these legacy receipts need operator reconciliation.
+INSERT INTO analytics_deleted_installations
+  (installation_id, deletion_token_hash, request_id, requested_at)
+SELECT installation_id, 'legacy-authentication-unavailable', request_id, requested_at
+FROM (
+  SELECT *, row_number() OVER (PARTITION BY installation_id ORDER BY requested_at, request_id) AS ordinal
+  FROM analytics_deletion_requests
+) WHERE ordinal = 1;
+
+CREATE TRIGGER analytics_no_deleted_identity
+BEFORE INSERT ON analytics_identities
+WHEN EXISTS (SELECT 1 FROM analytics_deleted_installations WHERE installation_id = NEW.identity_id)
+BEGIN
+  SELECT RAISE(ABORT, 'deleted-installation');
+END;
+
+CREATE TRIGGER analytics_no_deleted_event
+BEFORE INSERT ON analytics_events
+WHEN EXISTS (SELECT 1 FROM analytics_deleted_installations WHERE installation_id = NEW.distinct_id)
+BEGIN
+  SELECT RAISE(ABORT, 'deleted-installation');
+END;
+
+ALTER TABLE analytics_identities ADD COLUMN posthog_attempted INTEGER NOT NULL DEFAULT 0;
+-- Existing identities have unknown export history, possibly older than retention.
+UPDATE analytics_identities SET posthog_attempted = 1 WHERE identity_type = 'desktop_installation';
+
+ALTER TABLE analytics_deletion_requests ADD COLUMN posthog_person_uuid TEXT;
+ALTER TABLE analytics_deletion_requests ADD COLUMN posthog_submitted_at TEXT;
+-- Provider verification has a cutoff; it is not proof all captures have settled.
+ALTER TABLE analytics_deletion_requests ADD COLUMN posthog_verified_at TEXT;
+ALTER TABLE analytics_deletion_requests ADD COLUMN next_attempt_at TEXT;
+-- Old acknowledgements proved submission, not verified event erasure.
+UPDATE analytics_deletion_requests
+SET posthog_state = 'blocked', posthog_last_error_class = 'legacy-unverified-deletion', completed_at = NULL
+WHERE posthog_state IN ('completed', 'pending');
+
+-- Only post-migration installations have complete observation history. Never
+-- manufacture an activation cohort from a rolling window of old events.
+ALTER TABLE analytics_identities ADD COLUMN product_first_seen_at TEXT;
+ALTER TABLE analytics_identities ADD COLUMN cohort_eligible INTEGER NOT NULL DEFAULT 1;
+UPDATE analytics_identities SET cohort_eligible = 0;
+
+CREATE TABLE analytics_maintenance_leases (
+  name TEXT PRIMARY KEY,
+  owner TEXT NOT NULL,
+  expires_at INTEGER NOT NULL
+);
+
+CREATE TABLE analytics_maintenance_status (
+  name TEXT PRIMARY KEY,
+  completed_at TEXT NOT NULL,
+  outcome TEXT NOT NULL CHECK (outcome IN ('ok', 'failed'))
+);
+
+CREATE INDEX analytics_events_retention ON analytics_events (received_at, event_id);
+ALTER TABLE analytics_events ADD COLUMN posthog_next_attempt_at TEXT;
+ALTER TABLE analytics_identity_links ADD COLUMN posthog_next_attempt_at TEXT;

scripts/analytics-reconciliation.mjs

@@ -0,0 +1,62 @@
+/** Same source, occurrence window and event-ID grain on both sides of the gateway. */
+export function reconciliationQueries({ source = "desktop", from, to }) {
+  if (!["desktop", "website"].includes(source)) throw new Error("Invalid source");
+  const start = new Date(from);
+  const end = new Date(to);
+  if (
+    !Number.isFinite(+start) ||
+    !Number.isFinite(+end) ||
+    +end <= +start ||
+    +end - +start > 30 * 86400000
+  ) {
+    throw new Error("Use an increasing window no longer than 30 days");
+  }
+  const since = start.toISOString();
+  const until = end.toISOString();
+  return {
+    source,
+    from: since,
+    to: until,
+    d1: `SELECT event_name, posthog_state, COUNT(*) AS event_count
+      FROM analytics_events WHERE source = '${source}'
+        AND (source <> 'desktop' OR privacy_level <> 'diagnostic')
+        AND julianday(occurred_at) >= julianday('${since}') AND julianday(occurred_at) < julianday('${until}')
+      GROUP BY event_name, posthog_state ORDER BY event_name, posthog_state`,
+    posthog: `SELECT event, uniqExact(properties.event_id) FROM events
+      WHERE properties.source = '${source}' AND properties.event_id IS NOT NULL
+        AND (properties.source != 'desktop' OR coalesce(properties.privacy_level, '') != 'diagnostic')
+        AND timestamp >= parseDateTimeBestEffort('${since}') AND timestamp < parseDateTimeBestEffort('${until}')
+      GROUP BY event ORDER BY event`,
+    backlog: `SELECT posthog_state, count(*) AS request_count FROM analytics_deletion_requests
+      WHERE posthog_state <> 'completed' GROUP BY posthog_state`,
+  };
+}
+
+export function comparePipeline(d1Rows, posthogRows, deletionBacklog = 0) {
+  const counts = new Map();
+  const rowFor = (name) => {
+    if (!counts.has(name)) counts.set(name, { name, sent: 0, pending: 0, posthog: 0 });
+    return counts.get(name);
+  };
+  const count = (value) => {
+    const result = Number(value);
+    if (!Number.isSafeInteger(result) || result < 0)
+      throw new Error("Invalid reconciliation count");
+    return result;
+  };
+  for (const row of d1Rows) {
+    if (!["sent", "pending"].includes(row.posthog_state)) throw new Error("Unknown delivery state");
+    rowFor(String(row.event_name))[row.posthog_state] += count(row.event_count);
+  }
+  for (const [name, value] of posthogRows) rowFor(String(name)).posthog += count(value);
+  const rows = [...counts.values()].sort((a, b) => a.name.localeCompare(b.name));
+  const status =
+    count(deletionBacklog) > 0 || rows.some((row) => row.pending > 0)
+      ? "unsettled"
+      : rows.length === 0
+        ? "no-data"
+        : rows.some((row) => row.sent !== row.posthog)
+          ? "mismatch"
+          : "matched";
+  return { status, rows, deletionBacklog };
+}

scripts/analytics-reconciliation.test.mjs

@@ -0,0 +1,34 @@
+import { describe, expect, it } from "vitest";
+import { comparePipeline, reconciliationQueries } from "./analytics-reconciliation.mjs";
+
+describe("analytics reconciliation", () => {
+  it("uses a shared bounded source/time window and deduplicated capture IDs", () => {
+    const query = reconciliationQueries({ from: "2026-08-20", to: "2026-08-27" });
+    for (const sql of [query.d1, query.posthog]) {
+      expect(sql).toContain("desktop");
+      expect(sql).toContain("2026-08-20T00:00:00.000Z");
+      expect(sql).toContain("2026-08-27T00:00:00.000Z");
+    }
+    expect(query.posthog).toContain("uniqExact(properties.event_id)");
+    expect(query.d1).toContain("privacy_level <> 'diagnostic'");
+    expect(query.posthog).toContain("coalesce(properties.privacy_level, '') != 'diagnostic'");
+    expect(() =>
+      reconciliationQueries({ source: "desktop' OR 1=1", from: "2026-08-20", to: "2026-08-27" }),
+    ).toThrow();
+    expect(() => reconciliationQueries({ from: "2026-01-01", to: "2026-08-27" })).toThrow();
+  });
+  it("does not call empty, pending or deletion-affected populations healthy", () => {
+    const rows = [{ event_name: "app.health", posthog_state: "sent", event_count: 2 }];
+    expect(comparePipeline([], []).status).toBe("no-data");
+    expect(comparePipeline([], [], 1).status).toBe("unsettled");
+    expect(comparePipeline(rows, [["app.health", 2]]).status).toBe("matched");
+    expect(comparePipeline(rows, [["app.health", 1]]).status).toBe("mismatch");
+    expect(comparePipeline(rows, [["app.health", 2]], 1).status).toBe("unsettled");
+    expect(
+      comparePipeline(
+        [...rows, { event_name: "app.health", posthog_state: "pending", event_count: 1 }],
+        [["app.health", 2]],
+      ).status,
+    ).toBe("unsettled");
+  });
+});