From 0b98fdbed68420e2721f916bb4310293c84045fe Mon Sep 17 00:00:00 2001 From: Yaacov Date: Sun, 6 Sep 2026 19:35:04 +0300 Subject: [PATCH 1/3] feat(analytics): recover gateway readiness and retain diagnostics first-party --- README.md | 130 +- migrations/0006_analytics_readiness.sql | 69 + scripts/analytics-reconciliation.mjs | 62 + scripts/analytics-reconciliation.test.mjs | 34 + scripts/analytics-report.mjs | 92 +- scripts/analytics-report.test.mjs | 79 ++ scripts/manage-posthog-dashboards.mjs | 51 +- scripts/posthog-api.mjs | 62 + scripts/posthog-api.test.mjs | 53 + scripts/posthog-dashboard-manifest.mjs | 243 ++-- scripts/posthog-dashboard-manifest.test.mjs | 301 +++++ scripts/reconcile-analytics-pipeline.mjs | 99 +- src/pages/privacy.astro | 9 +- workers/events/fixtures/contract-v2.json | 1272 +++++++++++++++++++ workers/events/src/conformance.test.ts | 50 + workers/events/src/desktopPipeline.test.ts | 147 +++ workers/events/src/eventContract.test.ts | 22 +- workers/events/src/eventContract.ts | 225 +++- workers/events/src/exportLease.ts | 40 + workers/events/src/index.test.ts | 160 ++- workers/events/src/index.ts | 583 ++++++--- workers/events/src/readiness.test.ts | 532 ++++++++ workers/events/src/sqlite.testSupport.ts | 49 + workers/events/src/transport.ts | 73 ++ workers/events/worker-configuration.d.ts | 12 +- workers/events/wrangler.jsonc | 4 + 26 files changed, 3994 insertions(+), 459 deletions(-) create mode 100644 migrations/0006_analytics_readiness.sql create mode 100644 scripts/analytics-reconciliation.mjs create mode 100644 scripts/analytics-reconciliation.test.mjs create mode 100644 scripts/analytics-report.test.mjs create mode 100644 scripts/posthog-api.mjs create mode 100644 scripts/posthog-api.test.mjs create mode 100644 scripts/posthog-dashboard-manifest.test.mjs create mode 100644 workers/events/fixtures/contract-v2.json create mode 100644 workers/events/src/conformance.test.ts create mode 100644 workers/events/src/desktopPipeline.test.ts create mode 100644 workers/events/src/exportLease.ts create mode 100644 workers/events/src/readiness.test.ts create mode 100644 workers/events/src/sqlite.testSupport.ts create mode 100644 workers/events/src/transport.ts diff --git a/README.md b/README.md index 3ae260d..a9bbb1e 100644 --- a/README.md +++ b/README.md @@ -94,7 +94,12 @@ registered event name, its exact allowlisted property set, the event's declared privacy level, and sufficient explicit consent. Unknown events, extra properties, raw text, and mismatched consent or privacy classifications are rejected before storage. The versioned registry and its focused tests live in -`workers/events/src/eventContract.ts`. +`workers/events/src/eventContract.ts`. Revision 2 is generated from +`scient-desktop/packages/scient-analytics/src/wireContract.ts`, with a shared +90-case conformance fixture for all 45 registered events. Do not edit that copy +independently; the desktop analytics document owns regeneration instructions. +New events add an optional bounded `contractRevision`; legacy revision-1 +payloads remain supported. Deploy this validator before releasing new producers. The public desktop endpoint is disabled unless the Cloudflare runtime variable `DESKTOP_INGESTION_ENABLED` is exactly `true`. It also requires a random @@ -103,20 +108,44 @@ limit per opaque installation ID. It does not use or store an IP address as a rate-limit key. Keep the variable absent or false during preparation and use it as the immediate ingestion kill switch during a selected-user rollout. +Desktop forwarding to PostHog has its own gate, +`DESKTOP_POSTHOG_EXPORT_ENABLED`, also false by default. Turning off ingress +does not drain queued data; turning off export prevents queued desktop copies +from being forwarded. Neither gate affects existing website forwarding. + `POST /v1/installations/delete` authenticates an installation, deletes its D1 events, consent, identity links, and identity record, and queues the matching PostHog person and historical-event deletion by opaque distinct ID. A request from an installation that has never uploaded is acknowledged idempotently so the desktop can still clear local data and rotate -its anonymous identity. The scheduled Worker retries a failed PostHog submission -up to ten times and records a blocked queue item for operator review instead of -silently claiming success. Do not describe remote deletion as complete while -the queued PostHog state remains pending or blocked. +its anonymous identity. A minimal opaque-ID/authentication-hash tombstone blocks +late uploads from recreating deleted history. Migration 0006 preserves legacy +erasure tombstones conservatively; missing legacy credentials cannot be guessed. +No behavioral payload is retained in the tombstone. + +Erasure completes without PostHog only if no export was ever attempted. Otherwise +the scheduled Worker submits/polls PostHog's person/event deletion, with bounded +failure retries and an operator-visible blocked state. Provider verification, +not submission, completes the gateway request. Export/deletion share a lease; +deleted identities are tombstoned and never deliberately reused. Completion +records PostHog's verified operation, not a synchronous transaction covering +every ambiguous capture or a guaranteed physical-deletion deadline. +Do not claim remote deletion is complete while its state is pending or blocked. The scheduled Worker prunes canonical raw events older than 180 days in -bounded batches. D1 remains the source of truth for retention and delivery; -dashboard filters are not retention controls. - -Website visitors, desktop installations, sessions, and future Scient accounts use separate opaque identifiers. The service-authenticated `POST /v1/identity/link` endpoint can connect a visitor or installation to an account after Scient's account service has authenticated that user. Browser and desktop clients cannot call this endpoint directly or claim an account identifier. Linking updates first-party historical events without changing the user's analytics choice; the corresponding anonymous-to-account PostHog identity event is forwarded only for product-or-higher consent. +bounded batches of 5,000; diagnostics have a 30-day limit. An unfinished backlog +does not count as a successful retention pass. Desktop occurrence age is also +enforced during ingestion, export and pruning, so offline delivery does not +reset its retention clock. D1 remains the source of truth +for retention and delivery; dashboard filters are not physical retention controls. +Desktop Diagnostic-class events remain only in Scient's central D1 ledger and +are never exported to PostHog. `bun run analytics:report` exposes their bounded +30-day aggregate breakdown alongside maintenance health; no access to a user's +computer is required. Essential/Product-class events can be exported even when +the user's consent level is Diagnostic. PostHog retention for those copies is +provider-managed; its query-access window is not a physical-deletion deadline. +Do not advertise 13-month or 30-day PostHog deletion guarantees. + +Website visitors, desktop installations, sessions, and future Scient accounts use separate opaque identifiers. The service-authenticated `POST /v1/identity/link` endpoint can connect a website visitor to an account after Scient's account service has authenticated that user. Desktop linking is rejected until its per-installation erasure model is qualified. Browser and desktop clients cannot claim account identity. Website linking preserves consent; the PostHog identity event is forwarded only for Product-or-higher consent. Generate binding types and validate the Worker with: @@ -125,6 +154,24 @@ bun run events:types bun run events:typecheck ``` +The normal test suite uses synthetic records and real local SQLite migrations. +An additional cross-repository proof is opt-in: build the exact desktop +candidate, then run: + +```sh +SCIENT_ANALYTICS_DESKTOP_ROOT=/absolute/desktop bun run test workers/events/src/desktopPipeline.test.ts +``` + +This test connects the built desktop worker to a loopback gateway and mocked +PostHog exporter, checks forbidden-data removal and runtime-source metadata, +then exercises consent, deletion, and late-replay rejection. It does not touch +production and is intentionally skipped when no explicit desktop path is set. +Record the desktop revision/build as well as the website revision; ordinary +website CI alone does not qualify this cross-repository path. +The desktop's `docs/internals/product-analytics.md` also documents a non-GUI +Electron-runtime invocation. Use it to qualify the native SQLite/runtime +boundary; passing under ordinary Node alone does not prove desktop packaging. + Deploy the Worker only from an approved production change: ```sh @@ -133,19 +180,36 @@ bun run events:deploy `POSTHOG_PROJECT_TOKEN`, `POSTHOG_PERSONAL_API_KEY`, and `IDENTITY_LINK_TOKEN` are Cloudflare Worker secrets and must never be committed. The personal key is -used only for queued deletion and needs the narrow `person:write` scope; +used only for queued deletion and needs the reviewed person read/write scopes +for lookup, submission, and verification (qualify the exact provider permissions); `POSTHOG_PROJECT_ID` selects the project. If the project token is absent, ingestion continues and events remain queued in D1 for later delivery. If the deletion key or project ID is absent, accepted erasures remain queued in D1. If the identity-link token is absent, account linking returns `503` while ordinary ingestion continues. -Before any production activation, apply the migration, deploy the reviewed -Worker, verify that `/health` reports forwarding, deletion, rate limiting, and -storage ready, run the D1/PostHog reconciliation command, and only -then set `DESKTOP_INGESTION_ENABLED=true` for the approved cohort. Reversing -that variable to false stops new desktop ingestion without changing website -measurement. +Before activating an owner-approved rollout: + +1. Apply the approved migrations and deploy the reviewed Worker with **both + desktop gates false**. Website Pages deployment is not Worker deployment. +2. Verify `/health` against the exact deployed revision: required schema and a + recent successful retention pass are checked, but configured secrets are not + proof of valid permissions. +3. Confirm the approved first-party-only diagnostic routing and truthful + PostHog-managed retention wording. Verify asynchronous provider erasure with + synthetic identifiers; an arbitrary delay or repeat-delete loop is not proof. +4. Exercise authorized synthetic end-to-end delivery, + rejection, erasure, retry, retention and aggregate reconciliation. Never use + live researchers' records for a test or expose credentials in logs. +5. Complete human privacy-copy/consent/cohort review. Enable the approved + ingress/export gates only after qualification. Packaged desktop availability + does not override a user's Off choice; a desktop release is still needed. + +Export uses stable capture UUIDs and bounded retries, with a database lease +renewed before each outbound call. This prevents concurrent local exporters; +it is not a provider-side transactional fence. Persisted desktop properties are +revalidated so malformed/legacy rows cannot bypass today's privacy contract. +`posthog_state='sent'` means capture acknowledged, not erasure settled. The identity-link token is service-to-service authority. Rotate it if it is exposed, and never embed it in website or desktop bundles: @@ -158,7 +222,6 @@ After an account service has authenticated a user and obtained their opaque acco ```sh SCIENT_IDENTITY_LINK_TOKEN=... bun run identity:link \ --account account: \ - --identity installation: \ --identity visitor: ``` @@ -193,14 +256,39 @@ create or update ready dashboards: bun run analytics:dashboards --apply-ready ``` -The script never deletes dashboards or insights. D1 delivery state remains the +Operator API requests are project-origin restricted, time/body bounded, and do +not blindly retry ambiguous creates. Pagination is bounded. The script never +deletes dashboards or insights. D1 delivery state remains the operational source of truth and should be reconciled with PostHog using -`bun run analytics:report` before relying on a dashboard. The exact per-event +`bun run analytics:report` before relying on a dashboard. That aggregate-only +report includes pending/blocked deletion, exhausted or quarantined delivery, +and missing/stale maintenance; it is not a claim of end-to-end healthy delivery. +Maintenance rows have a status and timestamp, not an event count. The exact per-event D1-sent and PostHog counts can be checked without exposing the personal API key: ```sh bun run analytics:reconcile ``` -That command exits non-zero when the two systems disagree; pending D1 events -are reported separately rather than counted as delivered. +Reconciliation defaults to desktop events in the last seven days, excluding +the newest hour, and compares the same occurrence window and deduplicated event +IDs in both stores, excluding first-party-only diagnostics. Override it with `ANALYTICS_RECONCILE_SOURCE`, +`ANALYTICS_RECONCILE_FROM`, and `ANALYTICS_RECONCILE_TO` (maximum 30 days). +The one-hour delay is a reporting convention, not an erasure guarantee. +Mismatch, pending events, outstanding deletions, and no data exit non-zero; +an empty dashboard is not a verified pipeline. + +Prepared metrics count consenting installation profiles, not all people. Product +success rates use a consistent Product/Diagnostic population and exclude terminal +stops. Activation uses the gateway's first-observed Product cohort anchor, +excludes unknown legacy anchors, requires ordered steps, and reports immature +cohorts separately. Retention needs complete follow-up windows. Billing allowances +are not hardcoded as current facts. Exact HogQL execution and installed-dashboard +behavior still require authorized provider-side qualification. + +Scientific `source-import` outcomes have item-attempt grain, not batch grain. +Saved source-store results count as completions even when later batch cleanup +fails; duplicate/possible-match skips are reported separately as +`scient.operation.skipped` and do not qualify for meaningful-use metrics. +Retries are new attempts, not a second completion of an already saved source. +These events do not claim batch-conversion or human-review coverage. diff --git a/migrations/0006_analytics_readiness.sql b/migrations/0006_analytics_readiness.sql new file mode 100644 index 0000000..eacf8fb --- /dev/null +++ b/migrations/0006_analytics_readiness.sql @@ -0,0 +1,69 @@ +-- A minimal authenticated tombstone prevents late uploads recreating deleted data. +-- It deliberately has no FK to the identity record, which is erased. +CREATE TABLE analytics_deleted_installations ( + installation_id TEXT PRIMARY KEY, + deletion_token_hash TEXT NOT NULL, + request_id TEXT NOT NULL UNIQUE, + requested_at TEXT NOT NULL +); + +-- Earlier releases erased authentication history. Keep those IDs blocked rather +-- than treating a subsequent upload as a new installation. The sentinel cannot +-- match any SHA-256 token; these legacy receipts need operator reconciliation. +INSERT INTO analytics_deleted_installations + (installation_id, deletion_token_hash, request_id, requested_at) +SELECT installation_id, 'legacy-authentication-unavailable', request_id, requested_at +FROM ( + SELECT *, row_number() OVER (PARTITION BY installation_id ORDER BY requested_at, request_id) AS ordinal + FROM analytics_deletion_requests +) WHERE ordinal = 1; + +CREATE TRIGGER analytics_no_deleted_identity +BEFORE INSERT ON analytics_identities +WHEN EXISTS (SELECT 1 FROM analytics_deleted_installations WHERE installation_id = NEW.identity_id) +BEGIN + SELECT RAISE(ABORT, 'deleted-installation'); +END; + +CREATE TRIGGER analytics_no_deleted_event +BEFORE INSERT ON analytics_events +WHEN EXISTS (SELECT 1 FROM analytics_deleted_installations WHERE installation_id = NEW.distinct_id) +BEGIN + SELECT RAISE(ABORT, 'deleted-installation'); +END; + +ALTER TABLE analytics_identities ADD COLUMN posthog_attempted INTEGER NOT NULL DEFAULT 0; +-- Existing identities have unknown export history, possibly older than retention. +UPDATE analytics_identities SET posthog_attempted = 1 WHERE identity_type = 'desktop_installation'; + +ALTER TABLE analytics_deletion_requests ADD COLUMN posthog_person_uuid TEXT; +ALTER TABLE analytics_deletion_requests ADD COLUMN posthog_submitted_at TEXT; +-- Provider verification has a cutoff; it is not proof all captures have settled. +ALTER TABLE analytics_deletion_requests ADD COLUMN posthog_verified_at TEXT; +ALTER TABLE analytics_deletion_requests ADD COLUMN next_attempt_at TEXT; +-- Old acknowledgements proved submission, not verified event erasure. +UPDATE analytics_deletion_requests +SET posthog_state = 'blocked', posthog_last_error_class = 'legacy-unverified-deletion', completed_at = NULL +WHERE posthog_state IN ('completed', 'pending'); + +-- Only post-migration installations have complete observation history. Never +-- manufacture an activation cohort from a rolling window of old events. +ALTER TABLE analytics_identities ADD COLUMN product_first_seen_at TEXT; +ALTER TABLE analytics_identities ADD COLUMN cohort_eligible INTEGER NOT NULL DEFAULT 1; +UPDATE analytics_identities SET cohort_eligible = 0; + +CREATE TABLE analytics_maintenance_leases ( + name TEXT PRIMARY KEY, + owner TEXT NOT NULL, + expires_at INTEGER NOT NULL +); + +CREATE TABLE analytics_maintenance_status ( + name TEXT PRIMARY KEY, + completed_at TEXT NOT NULL, + outcome TEXT NOT NULL CHECK (outcome IN ('ok', 'failed')) +); + +CREATE INDEX analytics_events_retention ON analytics_events (received_at, event_id); +ALTER TABLE analytics_events ADD COLUMN posthog_next_attempt_at TEXT; +ALTER TABLE analytics_identity_links ADD COLUMN posthog_next_attempt_at TEXT; diff --git a/scripts/analytics-reconciliation.mjs b/scripts/analytics-reconciliation.mjs new file mode 100644 index 0000000..7e84efa --- /dev/null +++ b/scripts/analytics-reconciliation.mjs @@ -0,0 +1,62 @@ +/** Same source, occurrence window and event-ID grain on both sides of the gateway. */ +export function reconciliationQueries({ source = "desktop", from, to }) { + if (!["desktop", "website"].includes(source)) throw new Error("Invalid source"); + const start = new Date(from); + const end = new Date(to); + if ( + !Number.isFinite(+start) || + !Number.isFinite(+end) || + +end <= +start || + +end - +start > 30 * 86400000 + ) { + throw new Error("Use an increasing window no longer than 30 days"); + } + const since = start.toISOString(); + const until = end.toISOString(); + return { + source, + from: since, + to: until, + d1: `SELECT event_name, posthog_state, COUNT(*) AS event_count + FROM analytics_events WHERE source = '${source}' + AND (source <> 'desktop' OR privacy_level <> 'diagnostic') + AND julianday(occurred_at) >= julianday('${since}') AND julianday(occurred_at) < julianday('${until}') + GROUP BY event_name, posthog_state ORDER BY event_name, posthog_state`, + posthog: `SELECT event, uniqExact(properties.event_id) FROM events + WHERE properties.source = '${source}' AND properties.event_id IS NOT NULL + AND (properties.source != 'desktop' OR coalesce(properties.privacy_level, '') != 'diagnostic') + AND timestamp >= parseDateTimeBestEffort('${since}') AND timestamp < parseDateTimeBestEffort('${until}') + GROUP BY event ORDER BY event`, + backlog: `SELECT posthog_state, count(*) AS request_count FROM analytics_deletion_requests + WHERE posthog_state <> 'completed' GROUP BY posthog_state`, + }; +} + +export function comparePipeline(d1Rows, posthogRows, deletionBacklog = 0) { + const counts = new Map(); + const rowFor = (name) => { + if (!counts.has(name)) counts.set(name, { name, sent: 0, pending: 0, posthog: 0 }); + return counts.get(name); + }; + const count = (value) => { + const result = Number(value); + if (!Number.isSafeInteger(result) || result < 0) + throw new Error("Invalid reconciliation count"); + return result; + }; + for (const row of d1Rows) { + if (!["sent", "pending"].includes(row.posthog_state)) throw new Error("Unknown delivery state"); + rowFor(String(row.event_name))[row.posthog_state] += count(row.event_count); + } + for (const [name, value] of posthogRows) rowFor(String(name)).posthog += count(value); + const rows = [...counts.values()].sort((a, b) => a.name.localeCompare(b.name)); + const status = + count(deletionBacklog) > 0 || rows.some((row) => row.pending > 0) + ? "unsettled" + : rows.length === 0 + ? "no-data" + : rows.some((row) => row.sent !== row.posthog) + ? "mismatch" + : "matched"; + return { status, rows, deletionBacklog }; +} diff --git a/scripts/analytics-reconciliation.test.mjs b/scripts/analytics-reconciliation.test.mjs new file mode 100644 index 0000000..38f3ad1 --- /dev/null +++ b/scripts/analytics-reconciliation.test.mjs @@ -0,0 +1,34 @@ +import { describe, expect, it } from "vitest"; +import { comparePipeline, reconciliationQueries } from "./analytics-reconciliation.mjs"; + +describe("analytics reconciliation", () => { + it("uses a shared bounded source/time window and deduplicated capture IDs", () => { + const query = reconciliationQueries({ from: "2026-08-20", to: "2026-08-27" }); + for (const sql of [query.d1, query.posthog]) { + expect(sql).toContain("desktop"); + expect(sql).toContain("2026-08-20T00:00:00.000Z"); + expect(sql).toContain("2026-08-27T00:00:00.000Z"); + } + expect(query.posthog).toContain("uniqExact(properties.event_id)"); + expect(query.d1).toContain("privacy_level <> 'diagnostic'"); + expect(query.posthog).toContain("coalesce(properties.privacy_level, '') != 'diagnostic'"); + expect(() => + reconciliationQueries({ source: "desktop' OR 1=1", from: "2026-08-20", to: "2026-08-27" }), + ).toThrow(); + expect(() => reconciliationQueries({ from: "2026-01-01", to: "2026-08-27" })).toThrow(); + }); + it("does not call empty, pending or deletion-affected populations healthy", () => { + const rows = [{ event_name: "app.health", posthog_state: "sent", event_count: 2 }]; + expect(comparePipeline([], []).status).toBe("no-data"); + expect(comparePipeline([], [], 1).status).toBe("unsettled"); + expect(comparePipeline(rows, [["app.health", 2]]).status).toBe("matched"); + expect(comparePipeline(rows, [["app.health", 1]]).status).toBe("mismatch"); + expect(comparePipeline(rows, [["app.health", 2]], 1).status).toBe("unsettled"); + expect( + comparePipeline( + [...rows, { event_name: "app.health", posthog_state: "pending", event_count: 1 }], + [["app.health", 2]], + ).status, + ).toBe("unsettled"); + }); +}); diff --git a/scripts/analytics-report.mjs b/scripts/analytics-report.mjs index 86932b1..15efae1 100644 --- a/scripts/analytics-report.mjs +++ b/scripts/analytics-report.mjs @@ -1,13 +1,18 @@ import { spawnSync } from "node:child_process"; +import { resolve } from "node:path"; +import { fileURLToPath } from "node:url"; -const query = ` +export const analyticsReportQuery = ` + WITH expected_maintenance(name) AS ( + VALUES ('retention'), ('deletion'), ('identity-export'), ('event-export') + ) SELECT '30_day_identity' AS report_section, identity_type AS item, COUNT(*) AS event_count, MAX(last_seen_at) AS latest_event FROM analytics_identities - WHERE last_seen_at >= datetime('now', '-30 days') + WHERE julianday(last_seen_at) >= julianday('now', '-30 days') GROUP BY identity_type UNION ALL @@ -19,7 +24,7 @@ const query = ` MAX(occurred_at) AS latest_event FROM analytics_events WHERE session_id IS NOT NULL - AND occurred_at >= datetime('now', '-30 days') + AND julianday(occurred_at) >= julianday('now', '-30 days') GROUP BY source UNION ALL @@ -30,7 +35,7 @@ const query = ` COUNT(*) AS event_count, MAX(recorded_at) AS latest_event FROM analytics_consents - WHERE recorded_at >= datetime('now', '-30 days') + WHERE julianday(recorded_at) >= julianday('now', '-30 days') GROUP BY source, consent_level UNION ALL @@ -46,7 +51,7 @@ const query = ` UNION ALL SELECT - 'all_time_event' AS report_section, + 'retained_event' AS report_section, source || ':' || event_name AS item, COUNT(*) AS event_count, MAX(occurred_at) AS latest_event @@ -62,7 +67,7 @@ const query = ` MAX(occurred_at) AS latest_event FROM analytics_events WHERE event_name = 'download_clicked' - AND occurred_at >= datetime('now', '-30 days') + AND julianday(occurred_at) >= julianday('now', '-30 days') GROUP BY json_extract(properties_json, '$.asset_key') UNION ALL @@ -74,7 +79,7 @@ const query = ` MAX(occurred_at) AS latest_event FROM site_events WHERE event_name = 'download_clicked' - AND occurred_at >= datetime('now', '-30 days') + AND julianday(occurred_at) >= julianday('now', '-30 days') GROUP BY asset_key UNION ALL @@ -87,7 +92,7 @@ const query = ` MAX(occurred_at) AS latest_event FROM analytics_events WHERE event_name = 'outbound_link_clicked' - AND occurred_at >= datetime('now', '-30 days') + AND julianday(occurred_at) >= julianday('now', '-30 days') GROUP BY json_extract(properties_json, '$.destination_host'), json_extract(properties_json, '$.destination_path') @@ -101,7 +106,7 @@ const query = ` MAX(occurred_at) AS latest_event FROM site_events WHERE event_name = 'outbound_link_clicked' - AND occurred_at >= datetime('now', '-30 days') + AND julianday(occurred_at) >= julianday('now', '-30 days') GROUP BY destination_host, destination_path UNION ALL @@ -114,7 +119,7 @@ const query = ` MAX(occurred_at) AS latest_event FROM analytics_events WHERE event_name = 'download_failed' - AND occurred_at >= datetime('now', '-30 days') + AND julianday(occurred_at) >= julianday('now', '-30 days') GROUP BY json_extract(properties_json, '$.failure_stage'), json_extract(properties_json, '$.failure_reason') @@ -128,31 +133,72 @@ const query = ` MAX(occurred_at) AS latest_event FROM site_events WHERE event_name = 'download_failed' - AND occurred_at >= datetime('now', '-30 days') + AND julianday(occurred_at) >= julianday('now', '-30 days') GROUP BY failure_stage, failure_reason UNION ALL SELECT 'posthog_delivery' AS report_section, - posthog_state AS item, + CASE WHEN source = 'desktop' AND privacy_level = 'diagnostic' THEN 'first-party-only' + ELSE posthog_state END AS item, COUNT(*) AS event_count, MAX(received_at) AS latest_event FROM analytics_events + GROUP BY item + + UNION ALL + + SELECT '30_day_desktop_diagnostics', + event_name || ':' || COALESCE(json_extract(properties_json, '$.deliveryClass'), 'unknown'), + COUNT(*), MAX(occurred_at) + FROM analytics_events + WHERE source = 'desktop' AND privacy_level = 'diagnostic' + AND julianday(occurred_at) >= julianday('now', '-30 days') + GROUP BY event_name, json_extract(properties_json, '$.deliveryClass') + + UNION ALL + + SELECT + 'delivery_attention' AS report_section, + source || ':' || CASE WHEN posthog_last_error = 'contract-rejected' THEN 'contract-rejected' + ELSE 'retry-exhausted' END AS item, + COUNT(*) AS event_count, + MAX(received_at) AS latest_event + FROM analytics_events + WHERE posthog_state = 'pending' AND posthog_attempts >= 20 + GROUP BY item + + UNION ALL + + SELECT 'deletion_state', posthog_state, COUNT(*), MAX(requested_at) + FROM analytics_deletion_requests GROUP BY posthog_state + UNION ALL + + SELECT 'maintenance', expected.name || ':' || + CASE WHEN actual.name IS NULL THEN 'never-run' + WHEN julianday(actual.completed_at) < julianday('now', '-20 minutes') THEN 'stale-' || actual.outcome + ELSE actual.outcome END, + NULL, actual.completed_at + FROM expected_maintenance AS expected + LEFT JOIN analytics_maintenance_status AS actual ON actual.name = expected.name + ORDER BY report_section, event_count DESC, item `; -const result = spawnSync( - "wrangler", - ["d1", "execute", "scientfactory-downloads", "--remote", "--command", query], - { stdio: "inherit" }, -); - -if (result.error) { - console.error(result.error.message); - process.exitCode = 1; -} else { - process.exitCode = result.status ?? 1; +// Importing the query for local fixture tests must never contact production. +if (process.argv[1] && resolve(process.argv[1]) === fileURLToPath(import.meta.url)) { + const result = spawnSync( + "wrangler", + ["d1", "execute", "scientfactory-downloads", "--remote", "--command", analyticsReportQuery], + { stdio: "inherit", timeout: 60_000 }, + ); + if (result.error) { + console.error("Analytics aggregate report failed or timed out"); + process.exitCode = 1; + } else { + process.exitCode = result.status ?? 1; + } } diff --git a/scripts/analytics-report.test.mjs b/scripts/analytics-report.test.mjs new file mode 100644 index 0000000..a19d792 --- /dev/null +++ b/scripts/analytics-report.test.mjs @@ -0,0 +1,79 @@ +import { expect, it } from "vitest"; +import { analyticsReportQuery } from "./analytics-report.mjs"; +import { testDatabase } from "../workers/events/src/sqlite.testSupport.ts"; + +const fixedQuery = analyticsReportQuery.replaceAll("'now'", "'2026-08-31T12:00:00.000Z'"); + +it("reports centrally retained diagnostics separately from PostHog delivery", () => { + const store = testDatabase(); + try { + store.sqlite.exec(`INSERT INTO analytics_events + (event_id, event_name, source, privacy_level, occurred_at, distinct_id, properties_json) + VALUES ('diagnostic', 'app.diagnostics', 'desktop', 'diagnostic', '2026-08-30', 'private-id', '{"deliveryClass":"retrying"}');`); + const rows = store.sqlite.prepare(fixedQuery).all(); + expect(rows.find((row) => row.report_section === "posthog_delivery")).toMatchObject({ + item: "first-party-only", + event_count: 1, + }); + expect(rows.find((row) => row.report_section === "30_day_desktop_diagnostics")).toMatchObject({ + event_count: 1, + }); + expect(JSON.stringify(rows)).not.toContain("private-id"); + } finally { + store.close(); + } +}); + +it("keeps the aggregate report's 30-day window correct across SQLite and ISO timestamps", () => { + const store = testDatabase(); + try { + const insert = store.sqlite.prepare(`INSERT INTO analytics_events + (event_id, event_name, source, privacy_level, occurred_at, distinct_id, session_id, properties_json) + VALUES (?, 'app.session.started', 'desktop', 'essential', ?, 'private-installation', ?, '{}')`); + insert.run("old", "2026-08-01T09:00:00.000Z", "old-session"); + insert.run("recent", "2026-08-01 15:00:00", "recent-session"); + const rows = store.sqlite.prepare(fixedQuery).all(); + expect(rows.find((row) => row.report_section === "30_day_session")).toMatchObject({ + item: "desktop", + event_count: 1, + }); + expect(rows.find((row) => row.report_section === "retained_event")).toMatchObject({ + event_count: 2, + }); + expect(JSON.stringify(rows)).not.toContain("private-installation"); + expect(JSON.stringify(rows)).not.toContain("old-session"); + } finally { + store.close(); + } +}); + +it("shows blocked erasure, quarantine and missing or stale maintenance without raw errors", () => { + const store = testDatabase(); + try { + store.sqlite.exec(`INSERT INTO analytics_events + (event_id, event_name, source, privacy_level, occurred_at, distinct_id, properties_json, posthog_attempts, posthog_last_error) + VALUES ('retry', 'app.health', 'desktop', 'essential', '2026-08-30', 'private-id', '{}', 20, 'private-error-text'); + INSERT INTO analytics_deletion_requests (request_id, installation_id, requested_at, posthog_state) + VALUES ('erase', 'private-id', '2026-08-30', 'blocked'); + INSERT INTO analytics_maintenance_status VALUES ('retention', '2026-08-31 11:00:00', 'ok');`); + const rows = store.sqlite.prepare(fixedQuery).all(); + expect(rows.find((row) => row.report_section === "delivery_attention")).toMatchObject({ + item: "desktop:retry-exhausted", + event_count: 1, + }); + expect(rows.find((row) => row.report_section === "deletion_state")).toMatchObject({ + item: "blocked", + event_count: 1, + }); + const maintenance = rows.filter((row) => row.report_section === "maintenance"); + expect(maintenance.map((row) => row.item).sort()).toEqual([ + "deletion:never-run", + "event-export:never-run", + "identity-export:never-run", + "retention:stale-ok", + ]); + expect(JSON.stringify(rows)).not.toMatch(/private-id|private-error-text/); + } finally { + store.close(); + } +}); diff --git a/scripts/manage-posthog-dashboards.mjs b/scripts/manage-posthog-dashboards.mjs index 3df1dd9..25b5d30 100644 --- a/scripts/manage-posthog-dashboards.mjs +++ b/scripts/manage-posthog-dashboards.mjs @@ -3,9 +3,9 @@ import { execFileSync } from "node:child_process"; import { dashboards } from "./posthog-dashboard-manifest.mjs"; +import { createPosthogApi } from "./posthog-api.mjs"; const PROJECT_ID = "228610"; -const API_ORIGIN = "https://eu.posthog.com"; const KEYCHAIN_SERVICE = "scient-posthog-personal-api-key"; const apply = process.argv.includes("--apply-ready"); const validateQueries = process.argv.includes("--validate-queries"); @@ -31,32 +31,7 @@ if (!apiKey) { process.exit(1); } -async function api(path, init = {}) { - const url = path.startsWith("https://") - ? path - : `${API_ORIGIN}/api/projects/${PROJECT_ID}/${path}`; - for (let attempt = 0; attempt < 3; attempt += 1) { - const response = await fetch(url, { - ...init, - headers: { - Authorization: `Bearer ${apiKey}`, - ...(init.body ? { "Content-Type": "application/json" } : {}), - ...init.headers, - }, - }); - if (response.ok) return response.json(); - const message = await response.text(); - const retryable = response.status === 429 || response.status >= 500; - if (retryable && attempt < 2) { - await new Promise((resolve) => setTimeout(resolve, 250 * 2 ** attempt)); - continue; - } - throw new Error( - `PostHog ${init.method ?? "GET"} ${path} failed (${response.status}): ${message}`, - ); - } - throw new Error(`PostHog ${init.method ?? "GET"} ${path} exhausted retries`); -} +const api = createPosthogApi({ apiKey, projectId: PROJECT_ID }); async function observedEvents() { const response = await api("query/", { @@ -64,7 +39,8 @@ async function observedEvents() { body: JSON.stringify({ query: { kind: "HogQLQuery", - query: "SELECT event, count() FROM events GROUP BY event ORDER BY event", + query: + "SELECT event, count() FROM events WHERE timestamp >= now() - INTERVAL 30 DAY GROUP BY event ORDER BY event", }, }), }); @@ -74,8 +50,12 @@ async function observedEvents() { async function allPages(path) { const results = []; let next = `${path}${path.includes("?") ? "&" : "?"}limit=100`; + const seen = new Set(); while (next) { + if (seen.has(next) || seen.size >= 100) throw new Error("PostHog pagination limit exceeded"); + seen.add(next); const page = await api(next); + if (!Array.isArray(page.results)) throw new Error("Invalid PostHog pagination response"); results.push(...page.results); next = page.next ?? ""; } @@ -91,7 +71,7 @@ async function ensureDashboard(definition, existingDashboards, existingInsights) body: JSON.stringify({ name: definition.name, description: definition.description, - tags: ["scient-managed", "analytics-contract-v1"], + tags: ["scient-managed", "analytics-contract-v2"], }), }); console.log(`created dashboard: ${definition.name}`); @@ -101,16 +81,25 @@ async function ensureDashboard(definition, existingDashboards, existingInsights) body: JSON.stringify({ name: definition.name, description: definition.description, - tags: ["scient-managed", "analytics-contract-v1"], + tags: ["scient-managed", "analytics-contract-v2"], }), }); console.log(`updated dashboard: ${definition.name}`); } for (const insightDefinition of definition.insights ?? []) { + const missing = (insightDefinition.requiredEvents ?? []).filter((name) => !observed.has(name)); + if (missing.length > 0) { + console.log(`skipped insight: ${insightDefinition.name} (unobserved required events)`); + continue; + } + const acceptedInsightNames = new Set([ + insightDefinition.name, + ...(insightDefinition.aliases ?? []), + ]); const existing = existingInsights.find( (candidate) => - candidate.name === insightDefinition.name && candidate.tags?.includes("scient-managed"), + acceptedInsightNames.has(candidate.name) && candidate.tags?.includes("scient-managed"), ); const currentDashboards = existing?.dashboards ?? []; const payload = { diff --git a/scripts/posthog-api.mjs b/scripts/posthog-api.mjs new file mode 100644 index 0000000..964f692 --- /dev/null +++ b/scripts/posthog-api.mjs @@ -0,0 +1,62 @@ +const ORIGIN = "https://eu.posthog.com"; +const MAX_RESPONSE_BYTES = 2 * 1024 * 1024; + +/** Operator-only API transport. Never runs in the desktop app or on page visits. */ +export function createPosthogApi({ apiKey, projectId, fetchImpl = fetch }) { + if (!/^\d+$/.test(projectId)) throw new Error("Invalid PostHog project"); + const root = `${ORIGIN}/api/projects/${projectId}/`; + return async (path, init = {}) => { + const url = new URL(path, root); + if ( + url.origin !== ORIGIN || + !url.pathname.startsWith(new URL(root).pathname) || + url.username || + url.password + ) { + throw new Error("PostHog API URL is outside the configured project"); + } + // No blind retries: a timed-out create may already have succeeded. + const response = await fetchImpl(url.href, { + ...init, + redirect: "error", + signal: AbortSignal.timeout(30_000), + headers: { + ...(init.body ? { "Content-Type": "application/json" } : {}), + Authorization: `Bearer ${apiKey}`, + }, + }); + if (!response.ok) { + await response.body?.cancel(); + throw new Error(`PostHog API request failed (${response.status})`); + } + const reader = response.body?.getReader(); + if (!reader) throw new Error("PostHog API returned no response"); + let size = 0; + const chunks = []; + try { + while (true) { + const result = await reader.read(); + if (result.done) break; + size += result.value.byteLength; + if (size > MAX_RESPONSE_BYTES) throw new Error("PostHog API response exceeded limit"); + chunks.push(result.value); + } + } catch (error) { + await reader.cancel().catch(() => undefined); + throw error; + } finally { + reader.releaseLock(); + } + const bytes = new Uint8Array(size); + let offset = 0; + for (const chunk of chunks) { + bytes.set(chunk, offset); + offset += chunk.byteLength; + } + try { + return JSON.parse(new TextDecoder().decode(bytes)); + } catch { + throw new Error("PostHog API returned invalid JSON"); + } + }; +} diff --git a/scripts/posthog-api.test.mjs b/scripts/posthog-api.test.mjs new file mode 100644 index 0000000..9601737 --- /dev/null +++ b/scripts/posthog-api.test.mjs @@ -0,0 +1,53 @@ +import { describe, expect, it, vi } from "vitest"; +import { createPosthogApi } from "./posthog-api.mjs"; + +describe("operator PostHog API", () => { + it("restricts pagination and credentials to the configured project", async () => { + const fetchImpl = vi.fn().mockResolvedValue(Response.json({ results: [] })); + const api = createPosthogApi({ apiKey: "synthetic", projectId: "123", fetchImpl }); + for (const path of [ + "https://elsewhere.invalid/api/projects/123/", + "https://eu.posthog.com/api/projects/456/", + "//elsewhere.invalid/", + "../456/", + ]) { + await expect(api(path)).rejects.toThrow("outside the configured project"); + } + expect(fetchImpl).not.toHaveBeenCalled(); + expect(await api("insights/?limit=100")).toEqual({ results: [] }); + expect(fetchImpl).toHaveBeenCalledWith( + "https://eu.posthog.com/api/projects/123/insights/?limit=100", + expect.objectContaining({ + redirect: "error", + signal: expect.any(AbortSignal), + headers: { Authorization: "Bearer synthetic" }, + }), + ); + }); + it("does not leak error bodies or retry ambiguous creates", async () => { + const fetchImpl = vi + .fn() + .mockResolvedValue(new Response("private upstream detail", { status: 503 })); + const api = createPosthogApi({ apiKey: "synthetic", projectId: "123", fetchImpl }); + await expect(api("dashboards/", { method: "POST", body: "{}" })).rejects.toThrow( + "PostHog API request failed (503)", + ); + expect(fetchImpl).toHaveBeenCalledOnce(); + }); + it("bounds streamed responses and cancels oversized bodies", async () => { + const cancel = vi.fn(); + const body = new ReadableStream({ + start(controller) { + controller.enqueue(new Uint8Array(2 * 1024 * 1024 + 1)); + }, + cancel, + }); + const api = createPosthogApi({ + apiKey: "synthetic", + projectId: "123", + fetchImpl: async () => new Response(body), + }); + await expect(api("query/", { method: "POST" })).rejects.toThrow("exceeded limit"); + expect(cancel).toHaveBeenCalledOnce(); + }); +}); diff --git a/scripts/posthog-dashboard-manifest.mjs b/scripts/posthog-dashboard-manifest.mjs index 886bb61..8af3881 100644 --- a/scripts/posthog-dashboard-manifest.mjs +++ b/scripts/posthog-dashboard-manifest.mjs @@ -32,12 +32,47 @@ const trends = ({ series, display = "ActionsLineGraph", dateFrom = "-30d", inter const hogql = (query) => ({ kind: "HogQLQuery", query }); -const preparedInsight = (name, description, query) => ({ +const preparedInsight = (name, description, query, aliases = []) => ({ name, description, query: hogql(query), + aliases, }); +// Product denominators never mix Product successes with Essential-only failures. +// These are participating installations/profiles, not accounts or all Scient users. +export const PRODUCT_POPULATION = + "properties.source = 'desktop' AND properties.consent_level IN ('product', 'diagnostic')"; +export const SCIENTIFIC_OUTCOME = + "event = 'scient.operation.completed' AND properties.operationKind IN ('pdf-export', 'source-import', 'compute-run', 'compute-artifact', 'latex-build', 'document-export')"; +const JOURNEYS = `cohort_events AS ( + SELECT distinct_id, event, timestamp, properties.productFirstSeenAt AS cohort_start + FROM events + WHERE ${PRODUCT_POPULATION} AND properties.productFirstSeenAt IS NOT NULL + AND timestamp >= now() - INTERVAL 180 DAY +), projects AS ( + SELECT distinct_id, + min(parseDateTimeBestEffort(cohort_start)) AS first_seen, + minIf(timestamp, event = 'project.opened') AS project_opened + FROM cohort_events + GROUP BY distinct_id + HAVING first_seen >= now() - INTERVAL 180 DAY +), providers AS ( + SELECT projects.distinct_id, projects.first_seen, projects.project_opened, + minIf(cohort_events.timestamp, cohort_events.event = 'provider.session.started' + AND cohort_events.timestamp >= projects.project_opened) AS provider_started + FROM projects INNER JOIN cohort_events ON projects.distinct_id = cohort_events.distinct_id + GROUP BY projects.distinct_id, projects.first_seen, projects.project_opened +), journeys AS ( + SELECT providers.distinct_id, providers.first_seen, providers.project_opened, providers.provider_started, + minIf(cohort_events.timestamp, cohort_events.event = 'provider.turn.completed' + AND cohort_events.timestamp >= providers.provider_started) AS turn_completed + FROM providers INNER JOIN cohort_events ON providers.distinct_id = cohort_events.distinct_id + GROUP BY providers.distinct_id, providers.first_seen, providers.project_opened, providers.provider_started +)`; +const ACTIVATED = + "project_opened >= first_seen AND provider_started >= project_opened AND turn_completed >= provider_started AND turn_completed <= first_seen + INTERVAL 7 DAY"; + export const dashboards = [ { key: "pipeline", @@ -71,11 +106,12 @@ export const dashboards = [ }), }, { - name: "Active website visitors", + name: "Observed website identities", + aliases: ["Active website visitors"], description: - "Unique consented or event-scoped website identities. Interpret with the consent model documented in the repository.", + "Consent-dependent identity count, not a visitor count: without persistent consent each event may have its own identity.", query: trends({ - series: [event("page_viewed", "Active website visitors", "dau")], + series: [event("page_viewed", "Observed website identities", "dau")], interval: "day", }), }, @@ -88,10 +124,50 @@ export const dashboards = [ }), }, preparedInsight( - "Monthly event volume and free-tier budget", - "Rolling 30-day event volume. Compare this total with the configured PostHog billing limit before enabling a wider cohort.", - "SELECT count() AS events_last_30_days, round(count() / 1000000 * 100, 1) AS percent_of_one_million FROM events WHERE timestamp >= now() - INTERVAL 30 DAY", + "Monthly event volume", + "Rolling 30-day event volume. Billing limits are configured separately; this is not a cost estimate or calendar-month invoice count.", + "SELECT count() AS events_last_30_days FROM events WHERE timestamp >= now() - INTERVAL 30 DAY", + ["Monthly event volume and free-tier budget"], ), + { + ...preparedInsight( + "Observed provider lifecycle outcomes", + "Observed starts and terminal outcomes, not clicks or a current-state fleet. Product/Diagnostic population only; failures at Essential consent appear in the failure view.", + `SELECT properties.appVersion AS app_version, properties.provider AS provider, + properties.action AS action, properties.runtimeSource AS runtime_source, event, + uniqExact(properties.event_id) AS observations +FROM events WHERE ${PRODUCT_POPULATION} AND timestamp >= now() - INTERVAL 30 DAY + AND event IN ('provider.lifecycle.started', 'provider.lifecycle.completed', 'provider.lifecycle.failed', 'provider.lifecycle.cancelled') +GROUP BY app_version, provider, action, runtime_source, event ORDER BY observations DESC`, + ), + requiredEvents: ["provider.lifecycle.started"], + }, + { + ...preparedInsight( + "Observed provider readiness transitions", + "Reported changes only. Missing or offline installations are not assumed ready, and no observations is not a healthy zero.", + `SELECT properties.provider AS provider, properties.from AS previous_state, + properties.to AS next_state, uniqExact(properties.event_id) AS transitions +FROM events WHERE ${PRODUCT_POPULATION} AND event = 'provider.readiness.changed' + AND timestamp >= now() - INTERVAL 30 DAY +GROUP BY provider, previous_state, next_state ORDER BY transitions DESC`, + ), + requiredEvents: ["provider.readiness.changed"], + }, + { + ...preparedInsight( + "Observed app health outcomes", + "Server startup and renderer termination observations by release and consent. Desktop-update and migration coverage is not implied; counts are not a success rate.", + `SELECT properties.appVersion AS app_version, properties.component AS component, + properties.operation AS operation, properties.outcome AS outcome, properties.consent_level AS consent, + uniqExact(properties.event_id) AS observations +FROM events WHERE properties.source = 'desktop' AND event = 'app.health' + AND properties.outcome IN ('completed', 'failed', 'abnormal') AND timestamp >= now() - INTERVAL 30 DAY +GROUP BY app_version, component, operation, outcome, consent ORDER BY observations DESC`, + ["Observed server health outcomes"], + ), + requiredEvents: ["app.health"], + }, ], }, { @@ -110,81 +186,63 @@ export const dashboards = [ insights: [ preparedInsight( "Weekly Meaningful Active Installations", - "An installation qualifies after three completed turns across two sessions, or one completed scientific operation, in a calendar week.", + "Twelve complete calendar weeks. An installation qualifies after three completed turns across two sessions, or one completed scientific operation; the current partial week is excluded.", `SELECT week, countIf(turns >= 3 AND sessions >= 2 OR scientific_operations >= 1) AS meaningful_installations FROM ( SELECT toStartOfWeek(timestamp) AS week, distinct_id, - countIf(event = 'provider.turn.completed') AS turns, + uniqExactIf(properties.event_id, event = 'provider.turn.completed') AS turns, uniqIf(properties.$session_id, event = 'provider.turn.completed') AS sessions, - countIf(event = 'scient.operation.completed') AS scientific_operations + uniqExactIf(properties.event_id, ${SCIENTIFIC_OUTCOME}) AS scientific_operations FROM events - WHERE timestamp >= now() - INTERVAL 12 WEEK + WHERE ${PRODUCT_POPULATION} AND timestamp >= toStartOfWeek(now()) - INTERVAL 12 WEEK + AND timestamp < toStartOfWeek(now()) GROUP BY week, distinct_id ) GROUP BY week ORDER BY week`, ), preparedInsight( "Successful assistant-turn rate", - "Completed provider turns divided by all terminal provider-turn outcomes.", + "Product-consenting completed turns divided by completed plus failed turns. Stops/cancellations are reported separately; Essential-only failures are excluded from this denominator.", `SELECT toStartOfDay(timestamp) AS day, - round(100 * countIf(event = 'provider.turn.completed') / nullIf(countIf(event IN ('provider.turn.completed', 'provider.turn.failed')), 0), 1) AS success_percent + uniqExactIf(properties.event_id, event = 'provider.turn.completed') AS completed, + uniqExactIf(properties.event_id, event IN ('provider.turn.completed', 'provider.turn.failed')) AS terminal, + round(100 * completed / nullIf(terminal, 0), 1) AS success_percent FROM events -WHERE timestamp >= now() - INTERVAL 30 DAY +WHERE ${PRODUCT_POPULATION} AND timestamp >= now() - INTERVAL 30 DAY + AND (event <> 'provider.turn.failed' OR properties.failureClass NOT IN ('cancelled', 'interrupted')) GROUP BY day ORDER BY day`, ), preparedInsight( "Activated installations", - "Installations that opened a project, started a provider session, and completed a provider turn within seven days of first use.", - `WITH journeys AS ( - SELECT distinct_id, - minIf(timestamp, event = 'app.session.started') AS first_seen, - minIf(timestamp, event = 'project.opened') AS project_opened, - minIf(timestamp, event = 'provider.session.started') AS provider_started, - minIf(timestamp, event = 'provider.turn.completed') AS turn_completed - FROM events - WHERE timestamp >= now() - INTERVAL 90 DAY - AND event IN ('app.session.started', 'project.opened', 'provider.session.started', 'provider.turn.completed') - GROUP BY distinct_id -) -SELECT countIf( - project_opened >= first_seen AND project_opened <= first_seen + INTERVAL 7 DAY - AND provider_started >= first_seen AND provider_started <= first_seen + INTERVAL 7 DAY - AND turn_completed >= first_seen AND turn_completed <= first_seen + INTERVAL 7 DAY -) AS activated_installations + "Ordered project → provider session → completed turn within seven days of first observed Product participation. Only complete seven-day windows and known cohort origins count; not install-to-activation conversion.", + `WITH ${JOURNEYS} +SELECT countIf(first_seen <= now() - INTERVAL 7 DAY) AS eligible_installations, + countIf(first_seen > now() - INTERVAL 7 DAY) AS immature_installations, + countIf(first_seen <= now() - INTERVAL 7 DAY AND ${ACTIVATED}) AS activated_installations FROM journeys`, ), preparedInsight( "Week-one and week-four retained activation", - "Activated cohorts that later qualify for meaningful weekly use in week one or week four.", - `WITH journeys AS ( - SELECT distinct_id, - minIf(timestamp, event = 'app.session.started') AS first_seen, - minIf(timestamp, event = 'project.opened') AS project_opened, - minIf(timestamp, event = 'provider.session.started') AS provider_started, - minIf(timestamp, event = 'provider.turn.completed') AS turn_completed - FROM events - WHERE timestamp >= now() - INTERVAL 180 DAY - GROUP BY distinct_id -), activated AS ( + "Activated Product cohorts qualifying for meaningful later-week use. Separate mature denominators exclude incomplete week-one/week-four windows; absent cohort history is unknown, not new.", + `WITH ${JOURNEYS}, activated AS ( SELECT distinct_id, toStartOfWeek(greatest(project_opened, greatest(provider_started, turn_completed))) AS activation_week FROM journeys - WHERE project_opened >= first_seen AND project_opened <= first_seen + INTERVAL 7 DAY - AND provider_started >= first_seen AND provider_started <= first_seen + INTERVAL 7 DAY - AND turn_completed >= first_seen AND turn_completed <= first_seen + INTERVAL 7 DAY + WHERE ${ACTIVATED} ), meaningful AS ( SELECT distinct_id, toStartOfWeek(timestamp) AS week, - countIf(event = 'provider.turn.completed') AS turns, + uniqExactIf(properties.event_id, event = 'provider.turn.completed') AS turns, uniqIf(properties.$session_id, event = 'provider.turn.completed') AS sessions, - countIf(event = 'scient.operation.completed') AS scientific_operations + uniqExactIf(properties.event_id, ${SCIENTIFIC_OUTCOME}) AS scientific_operations FROM events - WHERE timestamp >= now() - INTERVAL 180 DAY + WHERE ${PRODUCT_POPULATION} AND timestamp >= now() - INTERVAL 180 DAY GROUP BY distinct_id, week HAVING turns >= 3 AND sessions >= 2 OR scientific_operations >= 1 ) SELECT activation_week, - uniqExact(activated.distinct_id) AS activated, - uniqExactIf(activated.distinct_id, meaningful.week = activation_week + INTERVAL 1 WEEK) AS retained_week_one, - uniqExactIf(activated.distinct_id, meaningful.week = activation_week + INTERVAL 4 WEEK) AS retained_week_four + uniqExactIf(activated.distinct_id, activation_week + INTERVAL 2 WEEK <= toStartOfWeek(now())) AS eligible_week_one, + uniqExactIf(activated.distinct_id, activation_week + INTERVAL 5 WEEK <= toStartOfWeek(now())) AS eligible_week_four, + uniqExactIf(activated.distinct_id, activation_week + INTERVAL 2 WEEK <= toStartOfWeek(now()) AND meaningful.week = activation_week + INTERVAL 1 WEEK) AS retained_week_one, + uniqExactIf(activated.distinct_id, activation_week + INTERVAL 5 WEEK <= toStartOfWeek(now()) AND meaningful.week = activation_week + INTERVAL 4 WEEK) AS retained_week_four FROM activated LEFT JOIN meaningful ON activated.distinct_id = meaningful.distinct_id GROUP BY activation_week ORDER BY activation_week`, @@ -203,7 +261,7 @@ GROUP BY activation_week ORDER BY activation_week`, "provider.turn.completed", ], description: - "First-session and seven-day activation from app start through a completed provider turn.", + "First-observed Product participation and seven-day activation through an ordered project/provider/answer journey. Not install conversion.", plannedInsights: [ "App start → project → provider session → successful turn funnel", "Median time-to-activation bucket", @@ -212,20 +270,21 @@ GROUP BY activation_week ORDER BY activation_week`, insights: [ preparedInsight( "Activation stage reach", - "Unique installations reaching each durable activation stage in the selected period.", + "Product-participating installations observed at each stage. These independent reach counts are not an ordered funnel or conversion rate.", `SELECT event, uniqExact(distinct_id) AS installations FROM events -WHERE timestamp >= now() - INTERVAL 30 DAY +WHERE ${PRODUCT_POPULATION} AND timestamp >= now() - INTERVAL 30 DAY AND event IN ('app.session.started', 'project.opened', 'provider.session.started', 'provider.turn.completed') GROUP BY event ORDER BY installations DESC`, ), preparedInsight( - "First-answer activation by build channel", - "Activated installation counts grouped by the bounded build channel recorded on session start.", + "Answer-producing installations by build channel", + "Product-participating installations with a completed turn, grouped by its build channel. This is not first-use activation.", `SELECT properties.buildChannel AS build_channel, uniqExact(distinct_id) AS installations FROM events -WHERE event = 'provider.turn.completed' AND timestamp >= now() - INTERVAL 30 DAY +WHERE ${PRODUCT_POPULATION} AND event = 'provider.turn.completed' AND timestamp >= now() - INTERVAL 30 DAY GROUP BY build_channel ORDER BY installations DESC`, + ["First-answer activation by build channel"], ), ], }, @@ -247,16 +306,18 @@ GROUP BY build_channel ORDER BY installations DESC`, "Completed turns per active installation", "Distribution of completed provider turns per pseudonymous installation over the last 30 days.", `SELECT turns, count() AS installations FROM ( - SELECT distinct_id, countIf(event = 'provider.turn.completed') AS turns - FROM events WHERE timestamp >= now() - INTERVAL 30 DAY GROUP BY distinct_id + SELECT distinct_id, uniqExactIf(properties.event_id, event = 'provider.turn.completed') AS turns + FROM events WHERE ${PRODUCT_POPULATION} AND timestamp >= now() - INTERVAL 30 DAY GROUP BY distinct_id ) GROUP BY turns ORDER BY turns`, ), preparedInsight( "Returning active installations by week", - "Installations active in both the current and immediately preceding week.", + "Product-participating installations with completed turns in consecutive complete calendar weeks. Not the stricter meaningful-use retention KPI.", `WITH weekly AS ( SELECT distinct_id, toStartOfWeek(timestamp) AS week - FROM events WHERE event = 'provider.turn.completed' GROUP BY distinct_id, week + FROM events WHERE ${PRODUCT_POPULATION} AND event = 'provider.turn.completed' + AND timestamp >= toStartOfWeek(now()) - INTERVAL 13 WEEK AND timestamp < toStartOfWeek(now()) + GROUP BY distinct_id, week ) SELECT current.week, uniqExact(current.distinct_id) AS returning_installations FROM weekly AS current @@ -285,27 +346,27 @@ GROUP BY current.week ORDER BY current.week`, insights: [ preparedInsight( "Provider terminal outcomes", - "Completed and failed provider turns by bounded provider kind.", - `SELECT properties.provider AS provider, event, count() AS turns + "Completed, failed and stopped turns for the same Product-consenting population, by provider. Stopped turns are not failures.", + `SELECT properties.provider AS provider, event, uniqExact(properties.event_id) AS turns FROM events -WHERE timestamp >= now() - INTERVAL 30 DAY - AND event IN ('provider.turn.completed', 'provider.turn.failed') +WHERE ${PRODUCT_POPULATION} AND timestamp >= now() - INTERVAL 30 DAY + AND event IN ('provider.turn.completed', 'provider.turn.failed', 'provider.turn.stopped') GROUP BY provider, event ORDER BY provider, event`, ), preparedInsight( "Model selection", - "Completed and attempted turns by maintained public model key; private custom model names collapse to other.", - `SELECT properties.modelKey AS model_key, count() AS turns + "Attempted turns by maintained public model key; private custom model names collapse to other.", + `SELECT properties.modelKey AS model_key, uniqExact(properties.event_id) AS turns FROM events -WHERE event = 'provider.turn.sent' AND timestamp >= now() - INTERVAL 30 DAY +WHERE ${PRODUCT_POPULATION} AND event = 'provider.turn.sent' AND timestamp >= now() - INTERVAL 30 DAY GROUP BY model_key ORDER BY turns DESC`, ), preparedInsight( "Provider failure classes", "Bounded provider failure classes without raw messages or stack traces.", - `SELECT properties.provider AS provider, properties.failureClass AS failure_class, count() AS failures + `SELECT properties.provider AS provider, properties.failureClass AS failure_class, uniqExact(properties.event_id) AS failures FROM events -WHERE event = 'provider.turn.failed' AND timestamp >= now() - INTERVAL 30 DAY +WHERE properties.source = 'desktop' AND event = 'provider.turn.failed' AND timestamp >= now() - INTERVAL 30 DAY GROUP BY provider, failure_class ORDER BY failures DESC`, ), ], @@ -332,26 +393,26 @@ GROUP BY provider, failure_class ORDER BY failures DESC`, preparedInsight( "Feature completion by installation", "Unique installations completing bounded Scient feature outcomes.", - `SELECT event, uniqExact(distinct_id) AS installations, count() AS completions + `SELECT event, uniqExact(distinct_id) AS installations, uniqExact(properties.event_id) AS completions FROM events -WHERE timestamp >= now() - INTERVAL 30 DAY +WHERE ${PRODUCT_POPULATION} AND timestamp >= now() - INTERVAL 30 DAY AND event IN ('project.initialization.completed', 'thread.fork.completed', 'thread.revert.completed', 'voice.transcription.completed') GROUP BY event ORDER BY installations DESC`, ), preparedInsight( "Selected surfaces opened", "Once-per-session style surface signals; this is deliberately not clickstream tracking.", - `SELECT properties.surface AS surface, uniqExact(distinct_id) AS installations, count() AS opens + `SELECT properties.surface AS surface, uniqExact(distinct_id) AS installations, uniqExact(properties.event_id) AS opens FROM events -WHERE event = 'surface.opened' AND timestamp >= now() - INTERVAL 30 DAY +WHERE ${PRODUCT_POPULATION} AND event = 'surface.opened' AND timestamp >= now() - INTERVAL 30 DAY GROUP BY surface ORDER BY installations DESC`, ), preparedInsight( "Measured settings choices", "Bounded direction, theme, and notification choices only.", - `SELECT properties.setting AS setting, properties.value AS value, count() AS changes + `SELECT properties.setting AS setting, properties.value AS value, uniqExact(properties.event_id) AS changes FROM events -WHERE event = 'setting.changed' AND timestamp >= now() - INTERVAL 30 DAY +WHERE ${PRODUCT_POPULATION} AND event = 'setting.changed' AND timestamp >= now() - INTERVAL 30 DAY GROUP BY setting, value ORDER BY setting, changes DESC`, ), ], @@ -379,19 +440,26 @@ GROUP BY setting, value ORDER BY setting, changes DESC`, preparedInsight( "Failures by class and release", "Bounded failures grouped by event, class, and application version.", - `SELECT properties.appVersion AS app_version, event, properties.failureClass AS failure_class, count() AS failures + `SELECT properties.appVersion AS app_version, event, properties.failureClass AS failure_class, uniqExact(properties.event_id) AS failures FROM events -WHERE timestamp >= now() - INTERVAL 30 DAY - AND event IN ('provider.turn.failed', 'project.add.failed', 'project.initialization.failed', 'thread.fork.failed', 'thread.revert.failed', 'voice.transcription.failed') +WHERE properties.source = 'desktop' AND timestamp >= now() - INTERVAL 30 DAY + AND (event IN ('provider.turn.failed', 'project.add.failed', 'project.initialization.failed', 'thread.fork.failed', 'thread.revert.failed', 'voice.transcription.failed', 'provider.lifecycle.failed', 'scient.operation.failed') + OR (event = 'app.health' AND properties.outcome IN ('failed', 'abnormal'))) GROUP BY app_version, event, failure_class ORDER BY failures DESC`, ), preparedInsight( "Duration bucket distribution", - "Coarse latency buckets for completed and failed product operations.", - `SELECT event, properties.durationBucket AS duration_bucket, count() AS outcomes + "Coarse terminal-outcome latency histograms, not exact percentiles. Starts are excluded; missing duration remains unknown.", + `SELECT event, properties.component AS component, properties.operation AS operation, + properties.operationKind AS operation_kind, properties.outcome AS health_outcome, + properties.durationBucket AS duration_bucket, uniqExact(properties.event_id) AS outcomes FROM events -WHERE timestamp >= now() - INTERVAL 30 DAY AND properties.durationBucket IS NOT NULL -GROUP BY event, duration_bucket ORDER BY event, outcomes DESC`, +WHERE ${PRODUCT_POPULATION} AND timestamp >= now() - INTERVAL 30 DAY + AND (event IN ('provider.turn.completed', 'provider.turn.failed', 'provider.turn.stopped', + 'provider.lifecycle.completed', 'provider.lifecycle.failed', 'provider.lifecycle.cancelled', + 'scient.operation.completed', 'scient.operation.failed', 'scient.operation.cancelled', 'scient.operation.skipped', 'voice.transcription.completed') + OR (event = 'app.health' AND properties.outcome IN ('completed', 'failed', 'abnormal'))) +GROUP BY event, component, operation, operation_kind, health_outcome, duration_bucket ORDER BY event, outcomes DESC`, ), ], }, @@ -405,7 +473,7 @@ GROUP BY event, duration_bucket ORDER BY event, outcomes DESC`, "scient.operation.failed", ], description: - "Registered scientific operations and reviewed outcomes once those operations exist.", + "Prepared for actual compute-run, latex-build, agent pdf-export and per-item source-import outcomes. Import skips are separate from saved sources; technical completion does not prove scientific review.", plannedInsights: [ "Completed scientific operations", "Reviewed outcome rate", @@ -415,10 +483,11 @@ GROUP BY event, duration_bucket ORDER BY event, outcomes DESC`, insights: [ preparedInsight( "Scientific operation outcomes", - "Registered scientific operation completions and bounded failures after those operations ship.", - `SELECT properties.operationKind AS operation_kind, event, count() AS outcomes + "Technical completions, failures, cancellations and no-op skips for qualified producers in one consent population. Source imports count individual attempts, not batches; no reviewed-outcome rate is inferred.", + `SELECT properties.operationKind AS operation_kind, event, uniqExact(properties.event_id) AS outcomes FROM events -WHERE event IN ('scient.operation.completed', 'scient.operation.failed') +WHERE ${PRODUCT_POPULATION} AND timestamp >= now() - INTERVAL 30 DAY + AND event IN ('scient.operation.completed', 'scient.operation.failed', 'scient.operation.cancelled', 'scient.operation.skipped') GROUP BY operation_kind, event ORDER BY outcomes DESC`, ), ], diff --git a/scripts/posthog-dashboard-manifest.test.mjs b/scripts/posthog-dashboard-manifest.test.mjs new file mode 100644 index 0000000..9d6271e --- /dev/null +++ b/scripts/posthog-dashboard-manifest.test.mjs @@ -0,0 +1,301 @@ +import { DatabaseSync } from "node:sqlite"; +import { expect, it } from "vitest"; +import { + dashboards, + PRODUCT_POPULATION, + SCIENTIFIC_OUTCOME, +} from "./posthog-dashboard-manifest.mjs"; + +const insights = dashboards.flatMap((dashboard) => dashboard.insights ?? []); +const query = (name) => insights.find((insight) => insight.name === name).query.query; + +it("keeps reliability ratios within one consent population and stops separate", () => { + expect(query("Successful assistant-turn rate")).toContain(PRODUCT_POPULATION); + expect(query("Successful assistant-turn rate")).toContain("nullIf(terminal, 0)"); + expect(query("Successful assistant-turn rate")).not.toContain("provider.turn.stopped"); + expect(query("Provider terminal outcomes")).toContain("provider.turn.stopped"); + expect(query("Provider terminal outcomes")).toContain(PRODUCT_POPULATION); +}); + +it("uses durable known Product cohort origins and complete later-week denominators", () => { + const activation = query("Activated installations"); + expect(activation).toContain("properties.productFirstSeenAt IS NOT NULL"); + expect(activation).not.toContain("minIf(timestamp, event = 'app.session.started')"); + expect(activation).toContain("eligible_installations"); + expect(activation).toContain("immature_installations"); + expect(activation).toContain("provider_started >= project_opened"); + const retention = query("Week-one and week-four retained activation"); + expect(retention).toContain("eligible_week_one"); + expect(retention).toContain("eligible_week_four"); + expect(retention).toContain("INTERVAL 2 WEEK <= toStartOfWeek(now())"); + expect(retention).toContain("INTERVAL 5 WEEK <= toStartOfWeek(now())"); +}); + +it("does not count opening a browser/file as a scientific outcome", () => { + expect(query("Weekly Meaningful Active Installations")).toContain(SCIENTIFIC_OUTCOME); + expect(SCIENTIFIC_OUTCOME).not.toContain("file-preview"); + expect(SCIENTIFIC_OUTCOME).not.toContain("browser"); + expect( + insights.some((insight) => insight.name === "First-answer activation by build channel"), + ).toBe(false); +}); + +// Execute the actual query text against synthetic records. This small adapter +// covers only the date/property/aggregate functions used below; it does not claim +// to qualify PostHog's parser, timezone configuration, permissions or live data. +const DAY = 86_400; +const WEEK = 7 * DAY; +const NOW = Date.parse("2026-08-31T12:00:00Z") / 1000; +const startOfWeek = (timestamp) => { + const date = new Date(timestamp * 1000); + return ( + Date.UTC(date.getUTCFullYear(), date.getUTCMonth(), date.getUTCDate() - date.getUTCDay()) / 1000 + ); +}; +const runQuery = (name, rows) => { + const database = new DatabaseSync(":memory:"); + try { + database.exec( + "CREATE TABLE events (distinct_id TEXT, event TEXT, timestamp INTEGER, properties TEXT)", + ); + const insert = database.prepare("INSERT INTO events VALUES (?, ?, ?, ?)"); + rows.forEach((row, index) => + insert.run( + row.id ?? "desktop-one", + row.event, + row.timestamp ?? NOW - DAY, + JSON.stringify({ + source: "desktop", + consent_level: "product", + event_id: `event-${index}`, + ...row.properties, + }), + ), + ); + database.function("now", () => NOW); + database.function("parseDateTimeBestEffort", (value) => Date.parse(value) / 1000); + database.function("toStartOfWeek", startOfWeek); + database.function("greatest", (left, right) => Math.max(left, right)); + database.aggregate("countIf", { + start: 0, + step: (count, condition) => count + (condition ? 1 : 0), + }); + database.aggregate("minIf", { + start: () => null, + step: (minimum, value, condition) => + condition ? (minimum === null ? value : Math.min(minimum, value)) : minimum, + result: (minimum) => minimum ?? 0, + }); + database.aggregate("uniqExact", { + start: () => new Set(), + step: (values, value) => { + if (value !== null) values.add(value); + return values; + }, + result: (values) => values.size, + }); + for (const aggregate of ["uniqExactIf", "uniqIf"]) { + database.aggregate(aggregate, { + start: () => new Set(), + step: (values, value, condition) => { + if (condition && value !== null) values.add(value); + return values; + }, + result: (values) => values.size, + }); + } + const sql = query(name) + .replace( + /properties\.([A-Za-z_$][\w$]*)/g, + (_, key) => `json_extract(properties, '$."${key}"')`, + ) + .replace(/INTERVAL (\d+) (DAY|WEEK)/g, (_, amount, unit) => + String(Number(amount) * (unit === "DAY" ? DAY : WEEK)), + ); + return database + .prepare(sql) + .all() + .map((row) => ({ ...row })); + } finally { + database.close(); + } +}; + +it("finds a valid ordered activation even after earlier out-of-order attempts", () => { + const firstSeen = NOW - 14 * DAY; + const properties = { productFirstSeenAt: new Date(firstSeen * 1000).toISOString() }; + const records = [ + ["provider.session.started", 0], + ["provider.turn.completed", 60], + ["project.opened", 100], + ["provider.session.started", 120], + ["provider.turn.completed", 150], + ].map(([event, offset]) => ({ event, timestamp: firstSeen + offset, properties })); + const unordered = records.slice(0, 3).map((row) => ({ ...row, id: "not-activated" })); + expect(runQuery("Activated installations", [...records, ...unordered])).toEqual([ + { eligible_installations: 2, immature_installations: 0, activated_installations: 1 }, + ]); +}); + +it("does not count website or Essential-only identities in desktop engagement", () => { + const website = Array.from({ length: 100 }, (_, index) => ({ + id: `website-${index}`, + event: "page_viewed", + properties: { source: "website" }, + })); + expect( + runQuery("Completed turns per active installation", [ + ...website, + { + id: "essential-only", + event: "provider.turn.failed", + properties: { consent_level: "essential" }, + }, + { event: "provider.turn.completed" }, + ]), + ).toEqual([{ turns: 1, installations: 1 }]); +}); + +it("includes the entire oldest calendar week and excludes the current partial week", () => { + const oldest = startOfWeek(NOW) - 12 * WEEK; + const turns = [0, DAY, 2 * DAY].map((offset, index) => ({ + event: "provider.turn.completed", + timestamp: oldest + offset, + properties: { $session_id: `session-${index % 2}`, event_id: `turn-${index}` }, + })); + expect( + runQuery("Weekly Meaningful Active Installations", [ + ...turns, + turns[0], // Duplicate delivery must not add a completed turn. + { + event: "scient.operation.completed", + timestamp: startOfWeek(NOW) + 1, + properties: { operationKind: "pdf-export" }, + }, + { + event: "scient.operation.completed", + timestamp: oldest - 1, + properties: { operationKind: "compute-run" }, + }, + ]), + ).toEqual([{ week: oldest, meaningful_installations: 1 }]); +}); + +it("counts terminal duration observations without counting their start event", () => { + const properties = { component: "server", operation: "startup", durationBucket: "lt_1s" }; + const result = runQuery("Duration bucket distribution", [ + { event: "app.health", properties: { ...properties, outcome: "started" } }, + { event: "app.health", properties: { ...properties, outcome: "completed" } }, + { + event: "scient.operation.started", + properties: { operationKind: "latex-build", durationBucket: "unknown" }, + }, + ]); + expect(result).toHaveLength(1); + expect(result[0]).toMatchObject({ + event: "app.health", + health_outcome: "completed", + outcomes: 1, + }); +}); + +it("reports import skips separately without counting them as meaningful work", () => { + const week = startOfWeek(NOW) - WEEK; + const records = [ + { + id: "skipped-only", + event: "scient.operation.skipped", + timestamp: week + DAY, + properties: { operationKind: "source-import", durationBucket: "under-1s" }, + }, + { + id: "imported", + event: "scient.operation.completed", + timestamp: week + DAY, + properties: { operationKind: "source-import", durationBucket: "under-1s" }, + }, + { + id: "essential-only", + event: "scient.operation.failed", + timestamp: week + DAY, + properties: { operationKind: "source-import", consent_level: "essential" }, + }, + ]; + expect(runQuery("Weekly Meaningful Active Installations", records)).toEqual([ + { week, meaningful_installations: 1 }, + ]); + expect(runQuery("Scientific operation outcomes", records)).toEqual( + expect.arrayContaining([ + { operation_kind: "source-import", event: "scient.operation.skipped", outcomes: 1 }, + { operation_kind: "source-import", event: "scient.operation.completed", outcomes: 1 }, + ]), + ); + expect(runQuery("Scientific operation outcomes", records)).toHaveLength(2); + expect( + runQuery("Duration bucket distribution", records) + .map((row) => row.event) + .toSorted(), + ).toEqual(["scient.operation.completed", "scient.operation.skipped"]); +}); + +it("uses separate mature week-one and week-four retention denominators", () => { + const matureWeek = startOfWeek(NOW) - 8 * WEEK; + const recentWeek = startOfWeek(NOW) - 2 * WEEK; + const activation = (id, week) => + ["project.opened", "provider.session.started", "provider.turn.completed"].map( + (event, index) => ({ + id, + event, + timestamp: week + DAY + index, + properties: { + productFirstSeenAt: new Date((week + DAY) * 1000).toISOString(), + $session_id: "activation", + }, + }), + ); + const returns = [0, DAY, 2 * DAY].map((offset, index) => ({ + id: "mature", + event: "provider.turn.completed", + timestamp: matureWeek + WEEK + offset, + properties: { + productFirstSeenAt: new Date((matureWeek + DAY) * 1000).toISOString(), + $session_id: `return-${index % 2}`, + }, + })); + expect( + runQuery("Week-one and week-four retained activation", [ + ...activation("mature", matureWeek), + ...activation("recent", recentWeek), + ...returns, + { + id: "mature", + event: "scient.operation.completed", + timestamp: matureWeek + 4 * WEEK + DAY, + properties: { operationKind: "latex-build" }, + }, + ]), + ).toEqual([ + { + activation_week: matureWeek, + eligible_week_one: 1, + eligible_week_four: 1, + retained_week_one: 1, + retained_week_four: 1, + }, + { + activation_week: recentWeek, + eligible_week_one: 1, + eligible_week_four: 0, + retained_week_one: 0, + retained_week_four: 0, + }, + ]); +}); + +it("keeps managed insight aliases unique so corrected names update rather than duplicate", () => { + const names = insights.flatMap((insight) => [insight.name, ...(insight.aliases ?? [])]); + expect(new Set(names).size).toBe(names.length); + expect( + insights.find((insight) => insight.name === "Observed website identities").aliases, + ).toContain("Active website visitors"); +}); diff --git a/scripts/reconcile-analytics-pipeline.mjs b/scripts/reconcile-analytics-pipeline.mjs index ae490c6..bf52812 100644 --- a/scripts/reconcile-analytics-pipeline.mjs +++ b/scripts/reconcile-analytics-pipeline.mjs @@ -1,6 +1,8 @@ #!/usr/bin/env node import { execFileSync, spawnSync } from "node:child_process"; +import { reconciliationQueries, comparePipeline } from "./analytics-reconciliation.mjs"; +import { createPosthogApi } from "./posthog-api.mjs"; const PROJECT_ID = "228610"; const KEYCHAIN_SERVICE = "scient-posthog-personal-api-key"; @@ -23,28 +25,40 @@ function personalApiKey() { } } -const d1Query = ` - SELECT event_name, posthog_state, COUNT(*) AS event_count - FROM analytics_events - GROUP BY event_name, posthog_state - ORDER BY event_name, posthog_state -`; -const d1 = spawnSync( - "wrangler", - ["d1", "execute", "scientfactory-downloads", "--remote", "--json", "--command", d1Query], - { encoding: "utf8" }, -); -if (d1.error) fail(`Unable to query D1: ${d1.error.message}`); -if (d1.status !== 0) fail(`D1 query failed: ${d1.stderr.trim()}`); +// Exclude the newest hour, where accepted captures may still be processing. +// A settled window is a reporting convention, not proof of completed deletion. +const until = process.env.ANALYTICS_RECONCILE_TO ?? new Date(Date.now() - 3600000).toISOString(); +const queries = reconciliationQueries({ + source: process.env.ANALYTICS_RECONCILE_SOURCE ?? "desktop", + from: + process.env.ANALYTICS_RECONCILE_FROM ?? + new Date(Date.parse(until) - 7 * 86400000).toISOString(), + to: until, +}); +function queryD1(query) { + const d1 = spawnSync( + "wrangler", + ["d1", "execute", "scientfactory-downloads", "--remote", "--json", "--command", query], + { encoding: "utf8", timeout: 60_000, maxBuffer: 2 * 1024 * 1024 }, + ); + if (d1.error) fail("Unable to run the bounded D1 query"); + if (d1.status !== 0) fail("D1 query failed; verify operator access and database availability"); -let d1Body; -try { - d1Body = JSON.parse(d1.stdout); -} catch { - fail("D1 returned an unreadable reconciliation response"); + let d1Body; + try { + d1Body = JSON.parse(d1.stdout); + } catch { + fail("D1 returned an unreadable reconciliation response"); + } + const d1Rows = d1Body?.[0]?.results; + if (!Array.isArray(d1Rows)) fail("D1 reconciliation response has no result rows"); + return d1Rows; } -const d1Rows = d1Body?.[0]?.results; -if (!Array.isArray(d1Rows)) fail("D1 reconciliation response has no result rows"); +const d1Rows = queryD1(queries.d1); +const deletionBacklog = queryD1(queries.backlog).reduce( + (sum, row) => sum + Number(row.request_count), + 0, +); const apiKey = personalApiKey(); if (!apiKey) { @@ -52,48 +66,23 @@ if (!apiKey) { `PostHog personal API key unavailable. Set POSTHOG_PERSONAL_API_KEY or add macOS Keychain service '${KEYCHAIN_SERVICE}'.`, ); } -const response = await fetch(`https://eu.posthog.com/api/projects/${PROJECT_ID}/query/`, { +const api = createPosthogApi({ apiKey, projectId: PROJECT_ID }); +const posthogBody = await api("query/", { method: "POST", - headers: { - Authorization: `Bearer ${apiKey}`, - "Content-Type": "application/json", - }, body: JSON.stringify({ query: { kind: "HogQLQuery", - query: "SELECT event, count() FROM events GROUP BY event ORDER BY event", + query: queries.posthog, }, }), }); -if (!response.ok) fail(`PostHog reconciliation query failed (${response.status})`); -const posthogBody = await response.json(); if (!Array.isArray(posthogBody.results)) fail("PostHog returned no reconciliation rows"); -const d1Sent = new Map(); -const d1Pending = new Map(); -for (const row of d1Rows) { - const target = row.posthog_state === "sent" ? d1Sent : d1Pending; - target.set(row.event_name, (target.get(row.event_name) ?? 0) + Number(row.event_count)); -} -const posthog = new Map( - posthogBody.results.map(([eventName, eventCount]) => [String(eventName), Number(eventCount)]), +const report = comparePipeline(d1Rows, posthogBody.results, deletionBacklog); +console.log(`${queries.source}: [${queries.from}, ${queries.to}) — event-ID counts`); +console.table(report.rows); +console.log(`Result: ${report.status}. Outstanding deletions: ${report.deletionBacklog}.`); +console.log( + "Identity-link events are intentionally excluded. No data does not mean the pipeline is verified.", ); -const eventNames = [...new Set([...d1Sent.keys(), ...d1Pending.keys(), ...posthog.keys()])].sort(); -let mismatches = 0; - -console.log("event | d1 sent | d1 pending | posthog | status"); -for (const eventName of eventNames) { - const sent = d1Sent.get(eventName) ?? 0; - const pending = d1Pending.get(eventName) ?? 0; - const delivered = posthog.get(eventName) ?? 0; - const status = sent === delivered ? "MATCH" : "MISMATCH"; - if (status === "MISMATCH") mismatches += 1; - console.log(`${eventName} | ${sent} | ${pending} | ${delivered} | ${status}`); -} - -if (mismatches > 0) { - console.error(`\n${mismatches} event count mismatch(es) require investigation.`); - process.exitCode = 2; -} else { - console.log("\nD1 sent counts and PostHog counts match exactly."); -} +if (report.status !== "matched") process.exitCode = 2; diff --git a/src/pages/privacy.astro b/src/pages/privacy.astro index 014de34..c15005b 100644 --- a/src/pages/privacy.astro +++ b/src/pages/privacy.astro @@ -9,7 +9,7 @@ import Layout from "../layouts/Layout.astro"; description="A plain-language explanation of how the ScientFactory website handles visitor information." > @@ -66,13 +66,16 @@ import Layout from "../layouts/Layout.astro";

Desktop measurement is controlled separately inside Scient.

- Scient runs as a workspace layer on your computer. Desktop analytics remain off unless a Scient build deliberately enables the feature. In an enabled build, the Privacy and analytics setting lets you choose Off, Essential reliability, Product improvement, or Diagnostics. Desktop events use a random installation identifier and never derive identity from a connected AI-provider account. + Scient runs as a workspace layer on your computer. Builds with analytics available let you choose Off, Essential reliability, Product improvement, or Diagnostics in Privacy and analytics. The default is Off; making the feature available does not change your saved choice. Desktop events use a random installation identifier and never derive identity from a connected AI-provider account.

Normal product analytics exclude prompts, assistant responses, research documents, filenames, file paths, URLs, source text, generated scientific content, credentials, provider account identities, and raw error messages. Scient does not use desktop autocapture or session replay. When you use a connected AI provider, that provider receives the prompts and supporting context needed for the session under its own terms and privacy policy. This website notice does not replace those terms.

- An enabled build lets you request deletion for the current anonymous installation. Scient clears its local analytics state only after the first-party deletion gateway accepts the request, queues removal of the matching PostHog profile and historical events, and then replaces the anonymous installation identifier. + Shared desktop events go to ScientFactory's central Cloudflare storage. Diagnostic-only events stay there, with scheduled removal after 30 days; other raw events are scheduled for removal after 180 days. Product and reliability events may also be processed in our EU-hosted PostHog project under PostHog-managed retention. We do not promise those downstream copies are physically deleted within the same periods. Local unsent events are filtered by age before delivery, and cleanup runs while the app is running. +

+

+ An enabled build lets you request deletion for the current installation. Scient clears its local analytics state after the first-party gateway accepts the request and replaces the random analytics identifier. Downstream removal of the matching PostHog profile and events is processed separately; acceptance does not mean every copy is already gone. A minimal deletion receipt and identifier-verification record remain to prevent delayed retries from recreating deleted history. They contain no usage events or research content.

diff --git a/workers/events/fixtures/contract-v2.json b/workers/events/fixtures/contract-v2.json new file mode 100644 index 0000000..324da61 --- /dev/null +++ b/workers/events/fixtures/contract-v2.json @@ -0,0 +1,1272 @@ +{ + "schemaVersion": 1, + "contractRevision": "2", + "cases": [ + { + "case": "app.session.started:fallbacks", + "name": "app.session.started", + "privacyLevel": "essential", + "consentLevel": "essential", + "properties": { + "appVersion": "0.6.8", + "buildChannel": "stable", + "platform": "other", + "architecture": "other", + "contractRevision": "2" + } + }, + { + "case": "app.session.started:representative", + "name": "app.session.started", + "privacyLevel": "essential", + "consentLevel": "essential", + "properties": { + "appVersion": "0.6.8", + "buildChannel": "stable", + "platform": "macos", + "architecture": "arm64", + "contractRevision": "2" + } + }, + { + "case": "app.session.ended:fallbacks", + "name": "app.session.ended", + "privacyLevel": "essential", + "consentLevel": "essential", + "properties": { + "durationBucket": "unknown", + "shutdownClass": "unknown", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "app.session.ended:representative", + "name": "app.session.ended", + "privacyLevel": "essential", + "consentLevel": "essential", + "properties": { + "durationBucket": "5-15s", + "shutdownClass": "graceful", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "app.health:fallbacks", + "name": "app.health", + "privacyLevel": "essential", + "consentLevel": "essential", + "properties": { + "component": "unknown", + "operation": "unknown", + "outcome": "unknown", + "failureClass": "unknown", + "durationBucket": "unknown", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "app.health:representative", + "name": "app.health", + "privacyLevel": "essential", + "consentLevel": "essential", + "properties": { + "component": "server", + "operation": "startup", + "outcome": "completed", + "failureClass": "permission", + "durationBucket": "5-15s", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "app.diagnostics:fallbacks", + "name": "app.diagnostics", + "privacyLevel": "diagnostic", + "consentLevel": "diagnostic", + "properties": { + "queuedCountBucket": "unknown", + "droppedCountBucket": "unknown", + "retryCountBucket": "unknown", + "deliveryClass": "unknown", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "app.diagnostics:representative", + "name": "app.diagnostics", + "privacyLevel": "diagnostic", + "consentLevel": "diagnostic", + "properties": { + "queuedCountBucket": "4-10", + "droppedCountBucket": "2-3", + "retryCountBucket": "1", + "deliveryClass": "network", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "server.boot.heartbeat:fallbacks", + "name": "server.boot.heartbeat", + "privacyLevel": "essential", + "consentLevel": "essential", + "properties": { + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "server.boot.heartbeat:representative", + "name": "server.boot.heartbeat", + "privacyLevel": "essential", + "consentLevel": "essential", + "properties": { + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "provider.session.started:fallbacks", + "name": "provider.session.started", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "provider": "other", + "runtimeMode": "other", + "hasResumeCursor": false, + "hasCwd": false, + "hasModel": false, + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "provider.session.started:representative", + "name": "provider.session.started", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "provider": "antigravity", + "runtimeMode": "full-access", + "hasResumeCursor": true, + "hasCwd": true, + "hasModel": true, + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "provider.session.recovered:fallbacks", + "name": "provider.session.recovered", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "provider": "other", + "strategy": "resume-thread", + "hasResumeCursor": false, + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "provider.session.recovered:representative", + "name": "provider.session.recovered", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "provider": "antigravity", + "strategy": "adopt-existing", + "hasResumeCursor": true, + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "provider.session.stopped:fallbacks", + "name": "provider.session.stopped", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "provider": "other", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "provider.session.stopped:representative", + "name": "provider.session.stopped", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "provider": "antigravity", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "provider.sessions.stopped_all:fallbacks", + "name": "provider.sessions.stopped_all", + "privacyLevel": "essential", + "consentLevel": "essential", + "properties": { + "sessionCountBucket": "unknown", + "shutdownClass": "unknown", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "provider.sessions.stopped_all:representative", + "name": "provider.sessions.stopped_all", + "privacyLevel": "essential", + "consentLevel": "essential", + "properties": { + "sessionCountBucket": "4-10", + "shutdownClass": "unknown", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "provider.runtime_mode.changed:fallbacks", + "name": "provider.runtime_mode.changed", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "provider": "other", + "from": "other", + "to": "other", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "provider.runtime_mode.changed:representative", + "name": "provider.runtime_mode.changed", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "provider": "antigravity", + "from": "other", + "to": "other", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "provider.turn.sent:fallbacks", + "name": "provider.turn.sent", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "provider": "other", + "modelFamily": "unknown", + "modelKey": "unknown", + "interactionMode": "unknown", + "runtimeMode": "other", + "attachmentCountBucket": "unknown", + "hasInput": false, + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "provider.turn.sent:representative", + "name": "provider.turn.sent", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "provider": "antigravity", + "modelFamily": "openai", + "modelKey": "gpt-5.6-sol", + "interactionMode": "plan", + "runtimeMode": "full-access", + "attachmentCountBucket": "1", + "hasInput": true, + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "provider.turn.completed:fallbacks", + "name": "provider.turn.completed", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "provider": "other", + "modelKey": "unknown", + "durationBucket": "unknown", + "usedTools": false, + "hasAttachment": false, + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "provider.turn.completed:representative", + "name": "provider.turn.completed", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "provider": "antigravity", + "modelKey": "gpt-5.6-sol", + "durationBucket": "5-15s", + "usedTools": true, + "hasAttachment": true, + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "provider.turn.failed:fallbacks", + "name": "provider.turn.failed", + "privacyLevel": "essential", + "consentLevel": "essential", + "properties": { + "provider": "other", + "modelKey": "unknown", + "failureClass": "unknown", + "durationBucket": "unknown", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "provider.turn.failed:representative", + "name": "provider.turn.failed", + "privacyLevel": "essential", + "consentLevel": "essential", + "properties": { + "provider": "antigravity", + "modelKey": "gpt-5.6-sol", + "failureClass": "unknown", + "durationBucket": "5-15s", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "provider.turn.stopped:fallbacks", + "name": "provider.turn.stopped", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "provider": "other", + "modelKey": "unknown", + "durationBucket": "unknown", + "stopClass": "unknown", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "provider.turn.stopped:representative", + "name": "provider.turn.stopped", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "provider": "antigravity", + "modelKey": "gpt-5.6-sol", + "durationBucket": "5-15s", + "stopClass": "unknown", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "provider.turn.interrupted:fallbacks", + "name": "provider.turn.interrupted", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "provider": "other", + "initiator": "user", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "provider.turn.interrupted:representative", + "name": "provider.turn.interrupted", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "provider": "antigravity", + "initiator": "user", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "provider.request.responded:fallbacks", + "name": "provider.request.responded", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "provider": "other", + "requestKind": "approval", + "decision": "unknown", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "provider.request.responded:representative", + "name": "provider.request.responded", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "provider": "antigravity", + "requestKind": "approval", + "decision": "approved", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "provider.conversation.rolled_back:fallbacks", + "name": "provider.conversation.rolled_back", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "provider": "other", + "turnCountBucket": "unknown", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "provider.conversation.rolled_back:representative", + "name": "provider.conversation.rolled_back", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "provider": "antigravity", + "turnCountBucket": "2-3", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "provider.discovered:fallbacks", + "name": "provider.discovered", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "provider": "other", + "runtimeSource": "unknown", + "state": "unknown", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "provider.discovered:representative", + "name": "provider.discovered", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "provider": "antigravity", + "runtimeSource": "scient_managed", + "state": "ready", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "provider.readiness.changed:fallbacks", + "name": "provider.readiness.changed", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "provider": "other", + "from": "unknown", + "to": "unknown", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "provider.readiness.changed:representative", + "name": "provider.readiness.changed", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "provider": "antigravity", + "from": "unknown", + "to": "unknown", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "provider.runtime.source.changed:fallbacks", + "name": "provider.runtime.source.changed", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "provider": "other", + "from": "unknown", + "to": "unknown", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "provider.runtime.source.changed:representative", + "name": "provider.runtime.source.changed", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "provider": "antigravity", + "from": "unknown", + "to": "unknown", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "provider.lifecycle.started:fallbacks", + "name": "provider.lifecycle.started", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "provider": "other", + "action": "unknown", + "runtimeSource": "unknown", + "stage": "unknown", + "failureClass": "unknown", + "durationBucket": "unknown", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "provider.lifecycle.started:representative", + "name": "provider.lifecycle.started", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "provider": "antigravity", + "action": "repair", + "runtimeSource": "scient_managed", + "stage": "downloading", + "failureClass": "permission", + "durationBucket": "5-15s", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "provider.lifecycle.completed:fallbacks", + "name": "provider.lifecycle.completed", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "provider": "other", + "action": "unknown", + "runtimeSource": "unknown", + "stage": "unknown", + "failureClass": "unknown", + "durationBucket": "unknown", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "provider.lifecycle.completed:representative", + "name": "provider.lifecycle.completed", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "provider": "antigravity", + "action": "repair", + "runtimeSource": "scient_managed", + "stage": "downloading", + "failureClass": "permission", + "durationBucket": "5-15s", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "provider.lifecycle.failed:fallbacks", + "name": "provider.lifecycle.failed", + "privacyLevel": "essential", + "consentLevel": "essential", + "properties": { + "provider": "other", + "action": "unknown", + "runtimeSource": "unknown", + "stage": "unknown", + "failureClass": "unknown", + "durationBucket": "unknown", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "provider.lifecycle.failed:representative", + "name": "provider.lifecycle.failed", + "privacyLevel": "essential", + "consentLevel": "essential", + "properties": { + "provider": "antigravity", + "action": "repair", + "runtimeSource": "scient_managed", + "stage": "downloading", + "failureClass": "permission", + "durationBucket": "5-15s", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "provider.lifecycle.cancelled:fallbacks", + "name": "provider.lifecycle.cancelled", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "provider": "other", + "action": "unknown", + "runtimeSource": "unknown", + "stage": "unknown", + "failureClass": "unknown", + "durationBucket": "unknown", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "provider.lifecycle.cancelled:representative", + "name": "provider.lifecycle.cancelled", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "provider": "antigravity", + "action": "repair", + "runtimeSource": "scient_managed", + "stage": "downloading", + "failureClass": "permission", + "durationBucket": "5-15s", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "project.added:fallbacks", + "name": "project.added", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "method": "unknown", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "project.added:representative", + "name": "project.added", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "method": "picker", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "project.add.failed:fallbacks", + "name": "project.add.failed", + "privacyLevel": "essential", + "consentLevel": "essential", + "properties": { + "stage": "unknown", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "project.add.failed:representative", + "name": "project.add.failed", + "privacyLevel": "essential", + "consentLevel": "essential", + "properties": { + "stage": "unknown", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "project.opened:fallbacks", + "name": "project.opened", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "projectState": "unknown", + "initializationState": "unknown", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "project.opened:representative", + "name": "project.opened", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "projectState": "existing", + "initializationState": "initialized", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "project.initialization.completed:fallbacks", + "name": "project.initialization.completed", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "outcome": "unknown", + "filesCreatedBucket": "unknown", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "project.initialization.completed:representative", + "name": "project.initialization.completed", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "outcome": "unknown", + "filesCreatedBucket": "2-3", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "project.initialization.failed:fallbacks", + "name": "project.initialization.failed", + "privacyLevel": "essential", + "consentLevel": "essential", + "properties": { + "failureClass": "unknown", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "project.initialization.failed:representative", + "name": "project.initialization.failed", + "privacyLevel": "essential", + "consentLevel": "essential", + "properties": { + "failureClass": "permission", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "thread.created:fallbacks", + "name": "thread.created", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "creationSource": "unknown", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "thread.created:representative", + "name": "thread.created", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "creationSource": "new", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "thread.fork.completed:fallbacks", + "name": "thread.fork.completed", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "workspaceMode": "unknown", + "boundaryClass": "unknown", + "refork": false, + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "thread.fork.completed:representative", + "name": "thread.fork.completed", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "workspaceMode": "local", + "boundaryClass": "checkpoint", + "refork": true, + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "thread.fork.failed:fallbacks", + "name": "thread.fork.failed", + "privacyLevel": "essential", + "consentLevel": "essential", + "properties": { + "workspaceMode": "unknown", + "failureClass": "unknown", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "thread.fork.failed:representative", + "name": "thread.fork.failed", + "privacyLevel": "essential", + "consentLevel": "essential", + "properties": { + "workspaceMode": "local", + "failureClass": "unknown", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "thread.revert.completed:fallbacks", + "name": "thread.revert.completed", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "boundaryClass": "checkpoint", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "thread.revert.completed:representative", + "name": "thread.revert.completed", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "boundaryClass": "checkpoint", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "thread.revert.failed:fallbacks", + "name": "thread.revert.failed", + "privacyLevel": "essential", + "consentLevel": "essential", + "properties": { + "failureClass": "unknown", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "thread.revert.failed:representative", + "name": "thread.revert.failed", + "privacyLevel": "essential", + "consentLevel": "essential", + "properties": { + "failureClass": "unknown", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "voice.transcription.started:fallbacks", + "name": "voice.transcription.started", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "engineClass": "other", + "languageMode": "unknown", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "voice.transcription.started:representative", + "name": "voice.transcription.started", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "engineClass": "local-whisper", + "languageMode": "automatic", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "voice.transcription.completed:fallbacks", + "name": "voice.transcription.completed", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "engineClass": "other", + "durationBucket": "unknown", + "audioDurationBucket": "unknown", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "voice.transcription.completed:representative", + "name": "voice.transcription.completed", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "engineClass": "local-whisper", + "durationBucket": "5-15s", + "audioDurationBucket": "5-15s", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "voice.transcription.failed:fallbacks", + "name": "voice.transcription.failed", + "privacyLevel": "essential", + "consentLevel": "essential", + "properties": { + "engineClass": "other", + "failureClass": "unknown", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "voice.transcription.failed:representative", + "name": "voice.transcription.failed", + "privacyLevel": "essential", + "consentLevel": "essential", + "properties": { + "engineClass": "local-whisper", + "failureClass": "permission", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "voice.transcription.cancelled:fallbacks", + "name": "voice.transcription.cancelled", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "stage": "unknown", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "voice.transcription.cancelled:representative", + "name": "voice.transcription.cancelled", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "stage": "unknown", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "surface.opened:fallbacks", + "name": "surface.opened", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "surface": "unknown", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "surface.opened:representative", + "name": "surface.opened", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "surface": "preview", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "setting.changed:fallbacks", + "name": "setting.changed", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "setting": "unknown", + "value": "unknown", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "setting.changed:representative", + "name": "setting.changed", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "setting": "direction", + "value": "rtl", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "scient.operation.started:fallbacks", + "name": "scient.operation.started", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "operationKind": "other", + "trigger": "other", + "durationBucket": "unknown", + "failureClass": "unknown", + "reviewRequired": false, + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "scient.operation.started:representative", + "name": "scient.operation.started", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "operationKind": "latex-build", + "trigger": "agent", + "durationBucket": "5-15s", + "failureClass": "permission", + "reviewRequired": true, + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "scient.operation.completed:fallbacks", + "name": "scient.operation.completed", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "operationKind": "other", + "trigger": "other", + "durationBucket": "unknown", + "failureClass": "unknown", + "reviewRequired": false, + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "scient.operation.completed:representative", + "name": "scient.operation.completed", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "operationKind": "latex-build", + "trigger": "agent", + "durationBucket": "5-15s", + "failureClass": "permission", + "reviewRequired": true, + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "scient.operation.failed:fallbacks", + "name": "scient.operation.failed", + "privacyLevel": "essential", + "consentLevel": "essential", + "properties": { + "operationKind": "other", + "trigger": "other", + "durationBucket": "unknown", + "failureClass": "unknown", + "reviewRequired": false, + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "scient.operation.failed:representative", + "name": "scient.operation.failed", + "privacyLevel": "essential", + "consentLevel": "essential", + "properties": { + "operationKind": "latex-build", + "trigger": "agent", + "durationBucket": "5-15s", + "failureClass": "permission", + "reviewRequired": true, + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "scient.operation.cancelled:fallbacks", + "name": "scient.operation.cancelled", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "operationKind": "other", + "trigger": "other", + "durationBucket": "unknown", + "failureClass": "unknown", + "reviewRequired": false, + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "scient.operation.cancelled:representative", + "name": "scient.operation.cancelled", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "operationKind": "latex-build", + "trigger": "agent", + "durationBucket": "5-15s", + "failureClass": "permission", + "reviewRequired": true, + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "scient.operation.skipped:fallbacks", + "name": "scient.operation.skipped", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "operationKind": "other", + "trigger": "other", + "durationBucket": "unknown", + "failureClass": "unknown", + "reviewRequired": false, + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + }, + { + "case": "scient.operation.skipped:representative", + "name": "scient.operation.skipped", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "operationKind": "latex-build", + "trigger": "agent", + "durationBucket": "5-15s", + "failureClass": "permission", + "reviewRequired": true, + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "2" + } + } + ] +} diff --git a/workers/events/src/conformance.test.ts b/workers/events/src/conformance.test.ts new file mode 100644 index 0000000..087184b --- /dev/null +++ b/workers/events/src/conformance.test.ts @@ -0,0 +1,50 @@ +import { describe, expect, it } from "vitest"; +import fixture from "../fixtures/contract-v2.json"; +import { eventContractViolation, EVENT_DEFINITIONS, type PrivacyLevel } from "./eventContract"; + +describe("desktop contract v2 conformance", () => { + it("accepts every generated desktop event including unknown-value fallbacks", () => { + expect(fixture.contractRevision).toBe("2"); + expect(new Set(fixture.cases.map((entry) => entry.name))).toEqual( + new Set(Object.keys(EVENT_DEFINITIONS)), + ); + for (const entry of fixture.cases) { + expect( + eventContractViolation({ + ...entry, + privacyLevel: entry.privacyLevel as PrivacyLevel, + consentLevel: entry.consentLevel as PrivacyLevel, + }), + entry.case, + ).toBeNull(); + } + }); + + it("rejects unknown properties, malformed values, and insufficient consent for every event", () => { + for (const entry of fixture.cases) { + const event = { + ...entry, + privacyLevel: entry.privacyLevel as PrivacyLevel, + consentLevel: entry.consentLevel as PrivacyLevel, + }; + expect( + eventContractViolation({ + ...event, + properties: { ...event.properties, path: "/private/fixture" }, + }), + ).not.toBeNull(); + for (const key of Object.keys(event.properties)) { + expect( + eventContractViolation({ + ...event, + properties: { ...event.properties, [key]: { raw: "private" } }, + }), + `${entry.case}:${key}`, + ).not.toBeNull(); + } + if (event.privacyLevel !== "essential") { + expect(eventContractViolation({ ...event, consentLevel: "essential" })).not.toBeNull(); + } + } + }); +}); diff --git a/workers/events/src/desktopPipeline.test.ts b/workers/events/src/desktopPipeline.test.ts new file mode 100644 index 0000000..b3be72e --- /dev/null +++ b/workers/events/src/desktopPipeline.test.ts @@ -0,0 +1,147 @@ +/// +import { mkdtempSync, rmSync } from "node:fs"; +import { createServer } from "node:http"; +import { tmpdir } from "node:os"; +import { join, resolve } from "node:path"; +import { pathToFileURL } from "node:url"; +import { expect, it, vi } from "vitest"; +import worker, { flushPendingEvents } from "./index"; +import { testDatabase } from "./sqlite.testSupport"; + +// Opt-in cross-repository qualification, not an implicit sibling-checkout dependency. +// Build the exact desktop candidate first; all traffic below stays on loopback/mock. +const desktopRoot = process.env.SCIENT_ANALYTICS_DESKTOP_ROOT; +it.skipIf(!desktopRoot)( + "qualifies the built desktop worker through the gateway and real ledger", + async () => { + const root = resolve(desktopRoot!); + const { createAnalyticsRuntime } = await import( + /* @vite-ignore */ pathToFileURL(join(root, "packages/scient-analytics/src/runtime.ts")).href + ); + const fixture = mkdtempSync(join(tmpdir(), "scient-pipeline-proof-")); + const store = testDatabase(); + const tasks: Promise[] = []; + const uploads: { token: string; body: string }[] = []; + const env = { + ANALYTICS_DB: store.database, + DESKTOP_INGESTION_ENABLED: "true", + ANALYTICS_INGESTION_RATE_LIMITER: { limit: async () => ({ success: true }) }, + }; + const context = { + waitUntil: (task: Promise) => tasks.push(task), + } as unknown as ExecutionContext; + const server = createServer((request, response) => { + void (async () => { + const chunks: Buffer[] = []; + for await (const chunk of request) chunks.push(Buffer.from(chunk)); + const body = Buffer.concat(chunks).toString("utf8"); + const token = String(request.headers["x-scient-installation-token"] ?? ""); + if (request.url === "/v1/events") uploads.push({ token, body }); + const result = await worker.fetch!( + new Request(`http://127.0.0.1${request.url}`, { + method: "POST", + headers: { "Content-Type": "application/json", "X-Scient-Installation-Token": token }, + body, + }) as Request, + env, + context, + ); + response.writeHead(result.status, { "Content-Type": "application/json" }); + response.end(await result.text()); + })().catch(() => { + response.writeHead(500); + response.end(); + }); + }); + await new Promise((done) => server.listen(0, "127.0.0.1", done)); + const address = server.address(); + if (!address || typeof address === "string") throw new Error("No fixture port"); + const endpoint = `http://127.0.0.1:${address.port}/v1/events`; + const runtime = createAnalyticsRuntime({ + enabled: true, + consent: "product", + outboxPath: join(fixture, "outbox.sqlite"), + endpoint, + workerUrl: pathToFileURL(join(root, "apps/server/dist/analytics-worker.mjs")), + appVersion: "0.6.8", + buildChannel: "development", + }); + const originalFetch = globalThis.fetch; + try { + runtime.record("provider.turn.sent", { + provider: "codex", + model: "PRIVATE-MODEL", + prompt: "PRIVATE-CONTENT", + }); + runtime.record("scient.operation.completed", { + operationKind: "pdf-export", + durationMs: 1234, + }); + runtime.record("provider.lifecycle.completed", { + provider: "codex", + action: "install", + source: "scient_managed", + }); + runtime.record("scient.operation.skipped", { + operationKind: "source-import", + trigger: "user", + title: "PRIVATE-SOURCE", + }); + expect(await runtime.flush()).toBe(4); + expect(await runtime.pendingCount()).toBe(0); + expect(store.sqlite.prepare("SELECT count(*) AS n FROM analytics_events").get()!.n).toBe(4); + expect(uploads[0]!.body).not.toContain("PRIVATE"); + const exportRequest = vi.fn().mockResolvedValue(new Response(null, { status: 200 })); + vi.stubGlobal("fetch", exportRequest); + expect( + await flushPendingEvents({ + ...env, + DESKTOP_POSTHOG_EXPORT_ENABLED: "true", + POSTHOG_PROJECT_TOKEN: "synthetic", + }), + ).toBe(4); + expect(exportRequest).toHaveBeenCalledOnce(); + expect(String(exportRequest.mock.calls[0]![1].body)).not.toContain("PRIVATE"); + const exported = JSON.parse(String(exportRequest.mock.calls[0]![1].body)); + expect( + exported.batch.find( + (event: { event: string }) => event.event === "provider.lifecycle.completed", + ).properties, + ).toMatchObject({ source: "desktop", runtimeSource: "scient_managed" }); + expect( + exported.batch.find( + (event: { event: string }) => event.event === "scient.operation.skipped", + ).properties, + ).toMatchObject({ operationKind: "source-import", trigger: "user" }); + await runtime.setConsent("off"); + expect(runtime.record("project.opened")).toBe(false); + expect(await runtime.flush()).toBe(0); + expect(await runtime.deleteData()).toBe(true); + expect(store.sqlite.prepare("SELECT count(*) AS n FROM analytics_events").get()!.n).toBe(0); + expect( + store.sqlite.prepare("SELECT posthog_state FROM analytics_deletion_requests").get()! + .posthog_state, + ).toBe("pending"); + // An acknowledged deletion must resist delayed uploads with the old credential. + const replay = await originalFetch(endpoint, { + method: "POST", + headers: { + "Content-Type": "application/json", + "X-Scient-Installation-Token": uploads[0]!.token, + }, + body: uploads[0]!.body, + }); + expect(replay.status).toBe(403); + await replay.body?.cancel(); + } finally { + await runtime.close(); + server.closeAllConnections(); + await new Promise((done) => server.close(() => done())); + await Promise.allSettled(tasks); + store.close(); + rmSync(fixture, { recursive: true, force: true }); + vi.unstubAllGlobals(); + } + }, + 15_000, +); diff --git a/workers/events/src/eventContract.test.ts b/workers/events/src/eventContract.test.ts index 4447771..d499c18 100644 --- a/workers/events/src/eventContract.test.ts +++ b/workers/events/src/eventContract.test.ts @@ -3,8 +3,26 @@ import { describe, expect, it } from "vitest"; import { EVENT_DEFINITIONS, eventContractViolation } from "./eventContract"; describe("desktop event contract", () => { - it("keeps the initial registry deliberately bounded", () => { - expect(Object.keys(EVENT_DEFINITIONS)).toHaveLength(33); + it("keeps the revision-two registry deliberately bounded", () => { + expect(Object.keys(EVENT_DEFINITIONS)).toHaveLength(45); + }); + + it("accepts the no-op outcome only with sufficient consent and bounded properties", () => { + const event = { + name: "scient.operation.skipped", + privacyLevel: "product", + consentLevel: "product", + properties: { + appVersion: "0.6.8", + buildChannel: "development", + operationKind: "source-import", + }, + } as const; + expect(eventContractViolation(event)).toBeNull(); + expect(eventContractViolation({ ...event, consentLevel: "essential" })).not.toBeNull(); + expect( + eventContractViolation({ ...event, properties: { ...event.properties, title: "PRIVATE" } }), + ).not.toBeNull(); }); it("allows higher consent for a lower-level event", () => { diff --git a/workers/events/src/eventContract.ts b/workers/events/src/eventContract.ts index 8a77341..16e5aee 100644 --- a/workers/events/src/eventContract.ts +++ b/workers/events/src/eventContract.ts @@ -1,3 +1,5 @@ +// Generated from scient-desktop/packages/scient-analytics/src/wireContract.ts. Do not edit here. +// Scient desktop wire contract. The website gateway consumes a generated copy. export const PRIVACY_LEVELS = ["essential", "product", "diagnostic", "contribution"] as const; export type PrivacyLevel = (typeof PRIVACY_LEVELS)[number]; @@ -14,7 +16,17 @@ interface EventDefinition { const provider = { kind: "enum", - values: ["codex", "claudeAgent", "cursor", "grok", "opencode", "other"], + values: [ + "codex", + "claudeAgent", + "antigravity", + "droid", + "cursor", + "grok", + "opencode", + "pi", + "other", + ], } as const satisfies PropertyRule; const runtimeMode = { kind: "enum", @@ -34,7 +46,8 @@ const buildChannel = { } as const satisfies PropertyRule; const appVersion = { kind: "pattern", - pattern: /^[0-9A-Za-z][0-9A-Za-z.+-]{0,63}$/, + pattern: + /^(?:unknown|\d{1,4}\.\d{1,4}\.\d{1,4}(?:-(?:beta|nightly|rc|dev)(?:\.\d{1,14}){0,3})?)$/, } as const satisfies PropertyRule; const modelKey = { kind: "enum", @@ -62,7 +75,153 @@ const modelKey = { ], } as const satisfies PropertyRule; +const runtimeSource = { + kind: "enum", + values: ["custom", "system", "scient_managed", "missing", "unknown"], +} as const satisfies PropertyRule; +const providerState = { + kind: "enum", + values: ["ready", "warning", "error", "disabled", "unknown"], +} as const satisfies PropertyRule; +const failureClass = { + kind: "enum", + values: [ + "configuration", + "authentication", + "connection", + "permission", + "provider", + "timeout", + "filesystem", + "checkpoint", + "validation", + "unavailable", + "incompatible-version", + "missing-dependency", + "resource-exhaustion", + "process-crash", + "internal", + "unknown", + ], +} as const satisfies PropertyRule; +const lifecycleProperties = { + provider, + action: { + kind: "enum", + values: [ + "install", + "update", + "repair", + "remove", + "sign-in", + "sign-out", + "source-switch", + "unknown", + ], + }, + runtimeSource, + stage: { + kind: "enum", + values: [ + "preparing", + "downloading", + "verifying", + "installing", + "testing", + "activating", + "removing", + "starting", + "waiting_for_browser", + "waiting_for_device_code", + "queued", + "running", + "unknown", + ], + }, + failureClass, + durationBucket, +} as const satisfies Readonly>; +const operationProperties = { + operationKind: { + kind: "enum", + values: [ + "file-preview", + "pdf-open", + "pdf-search", + "pdf-export", + "source-import", + "browser", + "chart-render", + "math-render", + "diagram-render", + "compute-session", + "compute-run", + "compute-artifact", + "latex-build", + "document-export", + "source-control", + "built-in-skill", + "worktree-provision", + "thread-fork", + "thread-revert", + "turn-retry", + "turn-steer", + "queued-follow-up", + "provider-handoff", + "other", + ], + }, + trigger: { kind: "enum", values: ["user", "agent", "automation", "other"], optional: true }, + durationBucket: { ...durationBucket, optional: true }, + failureClass: { ...failureClass, optional: true }, + reviewRequired: { kind: "boolean", optional: true }, +} as const satisfies Readonly>; + export const EVENT_DEFINITIONS = { + "app.health": { + privacyLevel: "essential", + properties: { + component: { + kind: "enum", + values: ["desktop", "server", "renderer", "browser", "analytics", "unknown"], + }, + operation: { + kind: "enum", + values: ["startup", "restart", "shutdown", "termination", "migration", "update", "unknown"], + }, + outcome: { kind: "enum", values: ["started", "completed", "failed", "abnormal", "unknown"] }, + failureClass, + durationBucket, + }, + }, + "app.diagnostics": { + privacyLevel: "diagnostic", + properties: { + queuedCountBucket: countBucket, + droppedCountBucket: countBucket, + retryCountBucket: countBucket, + deliveryClass: { + kind: "enum", + values: ["idle", "delivered", "network", "timeout", "rejected", "unavailable", "unknown"], + }, + }, + }, + "provider.discovered": { + privacyLevel: "product", + properties: { provider, runtimeSource, state: providerState }, + }, + "provider.readiness.changed": { + privacyLevel: "product", + properties: { provider, from: providerState, to: providerState }, + }, + "provider.runtime.source.changed": { + privacyLevel: "product", + properties: { provider, from: runtimeSource, to: runtimeSource }, + }, + "provider.lifecycle.started": { privacyLevel: "product", properties: lifecycleProperties }, + "provider.lifecycle.completed": { privacyLevel: "product", properties: lifecycleProperties }, + "provider.lifecycle.failed": { privacyLevel: "essential", properties: lifecycleProperties }, + "provider.lifecycle.cancelled": { privacyLevel: "product", properties: lifecycleProperties }, "app.session.started": { privacyLevel: "essential", properties: { @@ -207,6 +366,15 @@ export const EVENT_DEFINITIONS = { durationBucket, }, }, + "provider.turn.stopped": { + privacyLevel: "product", + properties: { + provider, + modelKey, + durationBucket, + stopClass: { kind: "enum", values: ["aborted", "cancelled", "interrupted", "unknown"] }, + }, + }, "provider.turn.interrupted": { privacyLevel: "product", properties: { provider, initiator: { kind: "enum", values: ["user", "system", "unknown"] } }, @@ -233,15 +401,15 @@ export const EVENT_DEFINITIONS = { "thread.fork.completed": { privacyLevel: "product", properties: { - workspaceMode: { kind: "enum", values: ["local", "new-worktree"] }, - boundaryClass: { kind: "enum", values: ["conversation", "checkpoint"] }, + workspaceMode: { kind: "enum", values: ["local", "new-worktree", "unknown"] }, + boundaryClass: { kind: "enum", values: ["conversation", "checkpoint", "unknown"] }, refork: { kind: "boolean" }, }, }, "thread.fork.failed": { privacyLevel: "essential", properties: { - workspaceMode: { kind: "enum", values: ["local", "new-worktree"] }, + workspaceMode: { kind: "enum", values: ["local", "new-worktree", "unknown"] }, failureClass: { kind: "enum", values: [ @@ -301,14 +469,23 @@ export const EVENT_DEFINITIONS = { properties: { surface: { kind: "enum", - values: ["files", "preview", "browser", "terminal", "usage", "settings", "whats-new"], + values: [ + "files", + "preview", + "browser", + "terminal", + "usage", + "settings", + "whats-new", + "unknown", + ], }, }, }, "setting.changed": { privacyLevel: "product", properties: { - setting: { kind: "enum", values: ["direction", "theme", "notifications"] }, + setting: { kind: "enum", values: ["direction", "theme", "notifications", "unknown"] }, value: { kind: "enum", values: [ @@ -327,41 +504,18 @@ export const EVENT_DEFINITIONS = { }, "scient.operation.started": { privacyLevel: "product", - properties: { - operationKind: { kind: "enum", values: ["other"] }, - trigger: { kind: "enum", values: ["user", "agent", "automation", "other"] }, - }, + properties: operationProperties, }, "scient.operation.completed": { privacyLevel: "product", - properties: { - operationKind: { kind: "enum", values: ["other"] }, - durationBucket, - reviewRequired: { kind: "boolean" }, - }, + properties: operationProperties, }, "scient.operation.failed": { privacyLevel: "essential", - properties: { - operationKind: { kind: "enum", values: ["other"] }, - failureClass: { - kind: "enum", - values: [ - "configuration", - "connection", - "permission", - "provider", - "timeout", - "filesystem", - "checkpoint", - "validation", - "unavailable", - "internal", - "unknown", - ], - }, - }, + properties: operationProperties, }, + "scient.operation.cancelled": { privacyLevel: "product", properties: operationProperties }, + "scient.operation.skipped": { privacyLevel: "product", properties: operationProperties }, } as const satisfies Readonly>; export type RegisteredEventName = keyof typeof EVENT_DEFINITIONS; @@ -407,6 +561,7 @@ export function eventContractViolation(input: { const rules: Readonly> = { appVersion, buildChannel, + contractRevision: { kind: "enum", values: ["1", "2"], optional: true }, ...definition.properties, }; for (const key of Object.keys(input.properties)) { diff --git a/workers/events/src/exportLease.ts b/workers/events/src/exportLease.ts new file mode 100644 index 0000000..6e1a722 --- /dev/null +++ b/workers/events/src/exportLease.ts @@ -0,0 +1,40 @@ +/** Serialize PostHog exports and erasures across Worker invocations, not just one isolate. */ +export async function withExportLease( + database: D1Database, + operation: (beforeRequest: () => Promise) => Promise, +): Promise { + const owner = crypto.randomUUID(); + const now = Date.now(); + const claim = await database + .prepare( + `INSERT INTO analytics_maintenance_leases (name, owner, expires_at) + VALUES ('posthog', ?, ?) + ON CONFLICT(name) DO UPDATE SET owner = excluded.owner, expires_at = excluded.expires_at + WHERE analytics_maintenance_leases.expires_at < ?`, + ) + .bind(owner, now + 60_000, now) + .run(); + if (claim.meta.changes !== 1) return 0; + try { + return await operation(async () => { + // D1 work can outlive the original lease. Renew using the database clock + // immediately before every external request, but never revive a stale owner. + const renewed = await database + .prepare(`UPDATE analytics_maintenance_leases + SET expires_at = CAST((julianday('now') - 2440587.5) * 86400000 AS INTEGER) + 60000 + WHERE name = 'posthog' AND owner = ? + AND expires_at > CAST((julianday('now') - 2440587.5) * 86400000 AS INTEGER) + RETURNING expires_at`) + .bind(owner) + .first<{ expires_at: number }>(); + if (!renewed || renewed.expires_at - Date.now() < 10_000) { + throw new Error("export-lease-lost"); + } + }); + } finally { + await database + .prepare("DELETE FROM analytics_maintenance_leases WHERE name = 'posthog' AND owner = ?") + .bind(owner) + .run(); + } +} diff --git a/workers/events/src/index.test.ts b/workers/events/src/index.test.ts index 2572d12..e52e97a 100644 --- a/workers/events/src/index.test.ts +++ b/workers/events/src/index.test.ts @@ -7,6 +7,14 @@ import worker, { validateIdentityLinkPayload, validateIngestionPayload, } from "./index"; +import { testDatabase } from "./sqlite.testSupport"; + +const realDatabases: ReturnType[] = []; +function realDatabase() { + const database = testDatabase(); + realDatabases.push(database); + return database; +} const INSTALLATION_TOKEN = "a".repeat(64); @@ -75,6 +83,7 @@ function incoming(request: Request): Request { + for (const database of realDatabases.splice(0)) database.close(); vi.unstubAllGlobals(); vi.restoreAllMocks(); }); @@ -216,7 +225,7 @@ describe("event gateway routes", () => { }); it("acknowledges deletion when the installation has never uploaded data", async () => { - const database = createDatabase(); + const database = realDatabase(); const response = await worker.fetch!( incoming( new Request("https://events.scientfactory.com/v1/installations/delete", { @@ -236,12 +245,15 @@ describe("event gateway routes", () => { ); expect(response.status).toBe(202); - expect(await response.json()).toEqual({ + expect(await response.json()).toMatchObject({ accepted: true, - local_state: "not_found", - posthog_state: "not_required", + local_state: "deleted", + posthog_state: "completed", + request_id: expect.any(String), }); - expect(database.batch).not.toHaveBeenCalled(); + expect( + database.sqlite.prepare("SELECT count(*) AS n FROM analytics_deleted_installations").get()!.n, + ).toBe(1); }); it("rate limits one validated installation without storing its IP address", async () => { @@ -295,6 +307,7 @@ describe("event gateway routes", () => { expect.any(String), expect.any(String), expect.any(String), + expect.any(String), ); expect(database.bind).toHaveBeenCalledWith( "8e0ee7d5-2c4b-48b6-8209-08f1e536f665", @@ -345,7 +358,7 @@ describe("event gateway routes", () => { }); it("reports whether optional PostHog forwarding is configured", async () => { - const database = createDatabase(); + const database = realDatabase(); const response = await worker.fetch!( incoming(new Request("https://events.scientfactory.com/health")), gatewayEnv(database.database), @@ -355,6 +368,9 @@ describe("event gateway routes", () => { expect(response.status).toBe(200); expect(await response.json()).toMatchObject({ status: "ready", + storage: "ready", + retention: "pending_verification", + activation_prerequisites_configured: false, posthog_forwarding: "pending_configuration", identity_linking: "pending_configuration", }); @@ -385,11 +401,11 @@ describe("event gateway routes", () => { expect(database.batch).not.toHaveBeenCalled(); }); - it("links installation history to an authenticated account id", async () => { + it("links consented visitor history to an authenticated account id", async () => { const database = createDatabase(); const waitUntil = vi.fn(); const accountId = "account:16ace444-e7c3-4b26-893f-98713188ae52"; - const installationId = "installation:8e0ee7d5-2c4b-48b6-8209-08f1e536f665"; + const installationId = "visitor:8e0ee7d5-2c4b-48b6-8209-08f1e536f665"; const response = await worker.fetch!( incoming( new Request("https://events.scientfactory.com/v1/identity/link", { @@ -435,7 +451,7 @@ describe("event gateway routes", () => { body: JSON.stringify({ schema_version: 1, account_id: "account:22222222-2222-4222-8222-222222222222", - identity_ids: ["installation:8e0ee7d5-2c4b-48b6-8209-08f1e536f665"], + identity_ids: ["visitor:8e0ee7d5-2c4b-48b6-8209-08f1e536f665"], }), }), ), @@ -482,24 +498,21 @@ describe("PostHog forwarding", () => { identity_type: "desktop_installation", session_id: "session:8e0ee7d5-2c4b-48b6-8209-08f1e536f665", consent_level: "product", - properties_json: JSON.stringify({ - appVersion: "0.0.32", - buildChannel: "development", - provider: "codex", - }), + properties_json: JSON.stringify(validPayload().events[0]!.properties), }; - const run = vi.fn().mockResolvedValue({ success: true }); - const all = vi.fn().mockResolvedValue({ results: [row] }); - const bind = vi.fn(() => ({ run, all })); - const prepare = vi.fn((_query: string) => ({ bind })); - const batch = vi.fn().mockResolvedValue([]); - const database = { prepare, batch } as unknown as D1Database; + const { database, sqlite } = realDatabase(); + sqlite + .prepare(`INSERT INTO analytics_events + (event_id, event_name, source, privacy_level, occurred_at, distinct_id, canonical_id, identity_type, session_id, consent_level, properties_json) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`) + .run(...Object.values(row)); const fetchMock = vi.fn().mockResolvedValue(new Response(null, { status: 200 })); vi.stubGlobal("fetch", fetchMock); const forwarded = await flushPendingEvents({ ANALYTICS_DB: database, POSTHOG_PROJECT_TOKEN: "phc_scientfactory_test", + DESKTOP_POSTHOG_EXPORT_ENABLED: "true", }); expect(forwarded).toBe(1); @@ -524,77 +537,112 @@ describe("PostHog forwarding", () => { $session_id: row.session_id, $process_person_profile: true, }); - expect(batch).toHaveBeenCalledOnce(); + expect(sqlite.prepare("SELECT posthog_state FROM analytics_events").get()!.posthog_state).toBe( + "sent", + ); }); - it("submits queued installation erasure through PostHog's distinct-id API", async () => { + it("resolves the person before submitting an erasure and keeps it pending verification", async () => { const row = { request_id: "delete-1", posthog_distinct_id: "installation:16ace444-e7c3-4b26-893f-98713188ae52", posthog_attempts: 0, }; - const run = vi.fn().mockResolvedValue({ success: true }); - const all = vi.fn().mockResolvedValue({ results: [row] }); - const bind = vi.fn(() => ({ run, all })); - const prepare = vi.fn((_query: string) => ({ bind })); - const fetchMock = vi.fn().mockResolvedValue( - new Response( - JSON.stringify({ - persons_found: 1, - persons_deleted: 1, - events_queued_for_deletion: true, - recordings_queued_for_deletion: false, - deletion_errors: [], - }), - { status: 202, headers: { "Content-Type": "application/json" } }, - ), - ); + const { database, sqlite } = realDatabase(); + sqlite + .prepare(`INSERT INTO analytics_deletion_requests + (request_id, installation_id, posthog_distinct_id, requested_at) VALUES (?, ?, ?, ?)`) + .run( + row.request_id, + row.posthog_distinct_id, + row.posthog_distinct_id, + new Date().toISOString(), + ); + const personUuid = "11111111-1111-4111-8111-111111111111"; + const fetchMock = vi + .fn() + .mockResolvedValueOnce( + Response.json({ results: [{ uuid: personUuid, distinct_ids: [row.posthog_distinct_id] }] }), + ) + .mockResolvedValueOnce( + new Response( + JSON.stringify({ + persons_found: 1, + persons_deleted: 1, + events_queued_for_deletion: true, + recordings_queued_for_deletion: false, + deletion_errors: [], + }), + { status: 202, headers: { "Content-Type": "application/json" } }, + ), + ); vi.stubGlobal("fetch", fetchMock); const submitted = await flushPendingDeletions({ - ANALYTICS_DB: { prepare, batch: vi.fn() } as unknown as D1Database, + ANALYTICS_DB: database, POSTHOG_PERSONAL_API_KEY: "phx_person_write_test", POSTHOG_PROJECT_ID: "228610", }); - expect(submitted).toBe(1); - expect(fetchMock).toHaveBeenCalledOnce(); - const [url, request] = fetchMock.mock.calls[0] ?? []; + expect(submitted).toBe(0); + expect(fetchMock).toHaveBeenCalledTimes(2); + const [url, request] = fetchMock.mock.calls[1] ?? []; expect(url).toBe("https://eu.posthog.com/api/projects/228610/persons/bulk_delete/"); expect(new Headers(request?.headers).get("Authorization")).toBe("Bearer phx_person_write_test"); expect(JSON.parse(String(request?.body))).toEqual({ - distinct_ids: [row.posthog_distinct_id], + ids: [personUuid], delete_events: true, delete_recordings: false, keep_person: false, }); - expect(run).toHaveBeenCalledOnce(); + expect( + sqlite + .prepare("SELECT posthog_state, posthog_submitted_at FROM analytics_deletion_requests") + .get(), + ).toMatchObject({ posthog_state: "pending", posthog_submitted_at: expect.any(String) }); }); it("forwards anonymous-to-account identity events from the first-party link queue", async () => { const row = { link_id: "link-1", - source_identity_id: "installation:16ace444-e7c3-4b26-893f-98713188ae52", + source_identity_id: "visitor:16ace444-e7c3-4b26-893f-98713188ae52", canonical_id: "account:8e0ee7d5-2c4b-48b6-8209-08f1e536f665", linked_at: new Date().toISOString(), }; - const run = vi.fn().mockResolvedValue({ success: true }); - const all = vi.fn().mockResolvedValue({ results: [row] }); - const bind = vi.fn(() => ({ run, all })); - const prepare = vi.fn((_query: string) => ({ bind })); - const batch = vi.fn().mockResolvedValue([]); + const { database, sqlite } = realDatabase(); + const addIdentity = sqlite.prepare(`INSERT INTO analytics_identities + (identity_id, identity_type, canonical_id, consent_level, first_seen_at, last_seen_at) + VALUES (?, ?, ?, 'essential', ?, ?)`); + addIdentity.run(row.canonical_id, "account", row.canonical_id, row.linked_at, row.linked_at); + addIdentity.run( + row.source_identity_id, + "web_visitor", + row.canonical_id, + row.linked_at, + row.linked_at, + ); + sqlite + .prepare(`INSERT INTO analytics_identity_links (link_id, source_identity_id, canonical_id, linked_at) + VALUES (?, ?, ?, ?)`) + .run(...Object.values(row)); const fetchMock = vi.fn().mockResolvedValue(new Response(null, { status: 200 })); vi.stubGlobal("fetch", fetchMock); - const forwarded = await flushPendingIdentityLinks({ - ANALYTICS_DB: { prepare, batch } as unknown as D1Database, + const env = { + ANALYTICS_DB: database, POSTHOG_PROJECT_TOKEN: "phc_scientfactory_test", - }); + }; + expect(await flushPendingIdentityLinks(env)).toBe(0); + expect(fetchMock).not.toHaveBeenCalled(); + sqlite + .prepare("UPDATE analytics_identities SET consent_level = 'product' WHERE identity_id = ?") + .run(row.source_identity_id); + const forwarded = await flushPendingIdentityLinks(env); expect(forwarded).toBe(1); - expect(prepare.mock.calls[0]?.[0]).toContain( - "identities.consent_level IN ('product', 'diagnostic', 'contribution')", - ); + expect( + sqlite.prepare("SELECT posthog_state FROM analytics_identity_links").get()!.posthog_state, + ).toBe("sent"); const [, request] = fetchMock.mock.calls[0] ?? []; const payload = JSON.parse(String(request?.body)) as { batch: ReadonlyArray<{ diff --git a/workers/events/src/index.ts b/workers/events/src/index.ts index e27a0d1..403ee3f 100644 --- a/workers/events/src/index.ts +++ b/workers/events/src/index.ts @@ -1,4 +1,6 @@ import { eventContractViolation, PRIVACY_LEVELS, type PrivacyLevel } from "./eventContract"; +import { posthogEventUuid, posthogRequest, readBoundedJson, TransportFailure } from "./transport"; +import { withExportLease } from "./exportLease"; const ALLOWED_WEB_ORIGINS = new Set(["https://scientfactory.com", "https://www.scientfactory.com"]); const IDENTIFIER_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$/; @@ -12,19 +14,26 @@ const MAX_PROPERTIES_BYTES = 16 * 1024; const POSTHOG_BATCH_SIZE = 100; const POSTHOG_HOST = "https://eu.i.posthog.com"; const POSTHOG_API_HOST = "https://eu.posthog.com"; -const POSTHOG_DELETION_BATCH_SIZE = 10; +const POSTHOG_DELETION_BATCH_SIZE = 1; const POSTHOG_DELETION_MAX_ATTEMPTS = 10; const INSTALLATION_TOKEN_HEADER = "X-Scient-Installation-Token"; const INSTALLATION_TOKEN_PATTERN = /^[0-9a-f]{64}$/i; const RAW_EVENT_RETENTION_DAYS = 180; +const DIAGNOSTIC_EVENT_RETENTION_DAYS = 30; const RETENTION_BATCH_SIZE = 5_000; -type AnalyticsEnv = Omit & { +type AnalyticsEnv = Omit< + AnalyticsWorkerBindings, + | "ANALYTICS_INGESTION_RATE_LIMITER" + | "DESKTOP_INGESTION_ENABLED" + | "DESKTOP_POSTHOG_EXPORT_ENABLED" +> & { readonly POSTHOG_PROJECT_TOKEN?: string; readonly POSTHOG_PERSONAL_API_KEY?: string; readonly POSTHOG_PROJECT_ID?: string; readonly IDENTITY_LINK_TOKEN?: string; readonly DESKTOP_INGESTION_ENABLED?: string; + readonly DESKTOP_POSTHOG_EXPORT_ENABLED?: string; readonly ANALYTICS_INGESTION_RATE_LIMITER?: RateLimit; }; @@ -52,6 +61,7 @@ interface PendingEventRow { readonly session_id: string | null; readonly consent_level: string; readonly properties_json: string; + readonly product_first_seen_at?: string | null; } interface PendingIdentityLinkRow { @@ -65,6 +75,9 @@ interface PendingDeletionRow { readonly request_id: string; readonly posthog_distinct_id: string; readonly posthog_attempts: number; + readonly requested_at: string; + readonly posthog_person_uuid: string | null; + readonly posthog_submitted_at: string | null; } class RequestValidationError extends Error {} @@ -119,7 +132,9 @@ function parseEvent(value: unknown): AcceptedEvent { if (occurredAtDate.valueOf() > now + 24 * 60 * 60 * 1000) { throw new RequestValidationError("occurred_at is too far in the future"); } - if (occurredAtDate.valueOf() < now - 180 * 24 * 60 * 60 * 1000) { + const retentionDays = + privacyLevel === "diagnostic" ? DIAGNOSTIC_EVENT_RETENTION_DAYS : RAW_EVENT_RETENTION_DAYS; + if (occurredAtDate.valueOf() < now - retentionDays * 24 * 60 * 60 * 1000) { throw new RequestValidationError("occurred_at is too old"); } @@ -187,18 +202,14 @@ async function sha256(value: string): Promise { } async function readJsonBody(request: Request): Promise { - const contentLength = Number(request.headers.get("Content-Length") ?? "0"); - if (Number.isFinite(contentLength) && contentLength > MAX_REQUEST_BYTES) { - throw new RequestValidationError("Request body is too large"); - } - const body = await request.text(); - if (new TextEncoder().encode(body).byteLength > MAX_REQUEST_BYTES) { - throw new RequestValidationError("Request body is too large"); - } try { - return JSON.parse(body) as unknown; - } catch { - throw new RequestValidationError("Request body must be valid JSON"); + return await readBoundedJson(request, MAX_REQUEST_BYTES); + } catch (error) { + throw new RequestValidationError( + error instanceof TransportFailure && error.kind === "body-too-large" + ? "Request body is too large" + : "Request body must be valid JSON", + ); } } @@ -208,9 +219,22 @@ async function persistEvents( deletionTokenHash: string, ): Promise { const installationId = events[0]?.distinctId; + const deleted = await database + .prepare("SELECT 1 AS deleted FROM analytics_deleted_installations WHERE installation_id = ?") + .bind(installationId) + .first(); + if (deleted) throw new InstallationAuthenticationError("Installation authentication failed"); const latestEvent = events.reduce((latest, event) => event.occurredAt > latest.occurredAt ? event : latest, ); + const firstEvent = events.reduce((first, event) => + event.occurredAt < first.occurredAt ? event : first, + ); + const firstProductAt = + events + .filter((event) => event.consentLevel === "product" || event.consentLevel === "diagnostic") + .map((event) => event.occurredAt) + .sort()[0] ?? null; const existing = await database .prepare("SELECT deletion_token_hash FROM analytics_identities WHERE identity_id = ?") .bind(installationId) @@ -227,11 +251,19 @@ async function persistEvents( consent_level, first_seen_at, last_seen_at, - deletion_token_hash - ) VALUES (?, 'desktop_installation', ?, ?, ?, ?, ?) + deletion_token_hash, + product_first_seen_at + ) VALUES (?, 'desktop_installation', ?, ?, ?, ?, ?, ?) ON CONFLICT(identity_id) DO UPDATE SET - consent_level = excluded.consent_level, - last_seen_at = excluded.last_seen_at, + consent_level = CASE WHEN excluded.last_seen_at >= analytics_identities.last_seen_at + THEN excluded.consent_level ELSE analytics_identities.consent_level END, + first_seen_at = min(analytics_identities.first_seen_at, excluded.first_seen_at), + last_seen_at = max(analytics_identities.last_seen_at, excluded.last_seen_at), + product_first_seen_at = CASE WHEN analytics_identities.cohort_eligible = 1 + THEN CASE WHEN analytics_identities.product_first_seen_at IS NULL THEN excluded.product_first_seen_at + WHEN excluded.product_first_seen_at IS NULL THEN analytics_identities.product_first_seen_at + ELSE min(analytics_identities.product_first_seen_at, excluded.product_first_seen_at) END + ELSE NULL END, deletion_token_hash = COALESCE(analytics_identities.deletion_token_hash, excluded.deletion_token_hash) WHERE analytics_identities.deletion_token_hash IS NULL OR analytics_identities.deletion_token_hash = excluded.deletion_token_hash @@ -265,9 +297,10 @@ async function persistEvents( installationId, installationId, latestEvent.consentLevel, - latestEvent.occurredAt, + firstEvent.occurredAt, latestEvent.occurredAt, deletionTokenHash, + firstProductAt, ), ...events.map((event) => database @@ -300,7 +333,7 @@ async function persistEvents( } } -function posthogEvent(row: PendingEventRow): Record { +async function posthogEvent(row: PendingEventRow): Promise> { let properties: Record = {}; try { const parsed = JSON.parse(row.properties_json) as unknown; @@ -309,6 +342,7 @@ function posthogEvent(row: PendingEventRow): Record { // The gateway writes valid JSON; retaining an empty object makes a malformed legacy row retryable. } return { + uuid: await posthogEventUuid(row.event_id), event: row.event_name, distinct_id: row.canonical_id, timestamp: row.occurred_at, @@ -320,6 +354,7 @@ function posthogEvent(row: PendingEventRow): Record { privacy_level: row.privacy_level, consent_level: row.consent_level, identity_type: row.identity_type, + ...(row.product_first_seen_at ? { productFirstSeenAt: row.product_first_seen_at } : {}), ...(row.session_id ? { $session_id: row.session_id } : {}), // A minimal pseudonymous person record is necessary for PostHog's // supported distinct-id event deletion API. No person properties are set. @@ -340,7 +375,8 @@ async function markPosthogFailure( .prepare( `UPDATE analytics_events SET posthog_attempts = posthog_attempts + 1, - posthog_last_error = ? + posthog_last_error = ?, + posthog_next_attempt_at = datetime('now', '+' || min(1800, 30 * (1 << min(posthog_attempts, 6))) || ' seconds') WHERE event_id = ? AND posthog_state = 'pending'`, ) .bind(error.slice(0, 500), row.event_id), @@ -350,7 +386,15 @@ async function markPosthogFailure( export async function flushPendingEvents(env: AnalyticsEnv): Promise { if (!env.POSTHOG_PROJECT_TOKEN) return 0; + return withExportLease(env.ANALYTICS_DB, (beforeRequest) => + exportPendingEvents(env, beforeRequest), + ); +} +async function exportPendingEvents( + env: AnalyticsEnv, + beforeRequest: () => Promise, +): Promise { const result = await env.ANALYTICS_DB.prepare( `SELECT event_id, @@ -363,38 +407,99 @@ export async function flushPendingEvents(env: AnalyticsEnv): Promise { identity_type, session_id, consent_level, - properties_json + properties_json, + (SELECT product_first_seen_at FROM analytics_identities WHERE identity_id = analytics_events.distinct_id) AS product_first_seen_at FROM analytics_events WHERE posthog_state = 'pending' + AND (source <> 'desktop' OR privacy_level <> 'diagnostic') + AND (source <> 'desktop' OR ? = 1) + AND (source <> 'desktop' OR ( + julianday(occurred_at) >= julianday('now', CASE WHEN privacy_level = 'diagnostic' THEN '-${DIAGNOSTIC_EVENT_RETENTION_DAYS} days' ELSE '-${RAW_EVENT_RETENTION_DAYS} days' END) + AND julianday(received_at) >= julianday('now', CASE WHEN privacy_level = 'diagnostic' THEN '-${DIAGNOSTIC_EVENT_RETENTION_DAYS} days' ELSE '-${RAW_EVENT_RETENTION_DAYS} days' END) + AND julianday(occurred_at) <= julianday('now', '+1 day') + )) + AND posthog_attempts < 20 + AND (posthog_next_attempt_at IS NULL OR julianday(posthog_next_attempt_at) <= julianday('now')) + AND NOT EXISTS (SELECT 1 FROM analytics_deleted_installations WHERE installation_id = analytics_events.distinct_id) ORDER BY received_at, event_id LIMIT ?`, ) - .bind(POSTHOG_BATCH_SIZE) + .bind(env.DESKTOP_POSTHOG_EXPORT_ENABLED === "true" ? 1 : 0, POSTHOG_BATCH_SIZE) .all(); - const rows = result.results; + let rows = result.results; + if (rows.length === 0) return 0; + + // Revalidate persisted desktop rows too: a legacy/corrupt row must not bypass + // today's privacy contract, nor poison every later event in its batch. + const rejected: string[] = []; + rows = rows.filter((row) => { + if (row.source !== "desktop") return true; + try { + const properties: unknown = JSON.parse(row.properties_json); + if ( + isRecord(properties) && + PRIVACY_LEVELS.includes(row.privacy_level as PrivacyLevel) && + PRIVACY_LEVELS.includes(row.consent_level as PrivacyLevel) && + eventContractViolation({ + name: row.event_name, + privacyLevel: row.privacy_level as PrivacyLevel, + consentLevel: row.consent_level as PrivacyLevel, + properties, + }) === null + ) + return true; + } catch { + /* Quarantine by a fixed class, never by raw properties/error text. */ + } + rejected.push(row.event_id); + return false; + }); + if (rejected.length > 0) + await env.ANALYTICS_DB.batch( + rejected.map((id) => + env.ANALYTICS_DB.prepare( + "UPDATE analytics_events SET posthog_attempts = 20, posthog_last_error = 'contract-rejected' WHERE event_id = ? AND posthog_state = 'pending'", + ).bind(id), + ), + ); + if (rows.length === 0) return 0; + + // Persist before sending: a concurrent erasure must know about an uncertain export. + await env.ANALYTICS_DB.batch( + rows.map((row) => + env.ANALYTICS_DB.prepare( + "UPDATE analytics_identities SET posthog_attempted = 1 WHERE identity_id = ?", + ).bind(row.distinct_id), + ), + ); + const surviving = await env.ANALYTICS_DB.prepare( + `SELECT event_id FROM analytics_events WHERE event_id IN (${rows.map(() => "?").join(",")})`, + ) + .bind(...rows.map((row) => row.event_id)) + .all<{ event_id: string }>(); + const survivingIds = new Set(surviving.results.map((row) => row.event_id)); + rows = rows.filter((row) => survivingIds.has(row.event_id)); if (rows.length === 0) return 0; let response: Response; try { - response = await fetch(`${POSTHOG_HOST}/batch`, { + const batch = await Promise.all(rows.map(posthogEvent)); + await beforeRequest(); + response = await posthogRequest(`${POSTHOG_HOST}/batch`, { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ api_key: env.POSTHOG_PROJECT_TOKEN, - batch: rows.map(posthogEvent), + batch, }), }); } catch (error) { - const message = error instanceof Error ? error.message : String(error); + const message = error instanceof TransportFailure ? error.kind : "internal"; await markPosthogFailure(env.ANALYTICS_DB, rows, message); throw error; } - if (!response.ok) { - const message = `PostHog returned ${response.status}`; - await markPosthogFailure(env.ANALYTICS_DB, rows, message); - throw new Error(message); - } + await response.body?.cancel(); await env.ANALYTICS_DB.batch( rows.map((row) => @@ -411,8 +516,11 @@ export async function flushPendingEvents(env: AnalyticsEnv): Promise { return rows.length; } -function identityIdentifyEvent(row: PendingIdentityLinkRow): Record { +async function identityIdentifyEvent( + row: PendingIdentityLinkRow, +): Promise> { return { + uuid: await posthogEventUuid(row.link_id), event: "$identify", distinct_id: row.canonical_id, timestamp: row.linked_at, @@ -428,12 +536,24 @@ function identityIdentifyEvent(row: PendingIdentityLinkRow): Record { if (!env.POSTHOG_PROJECT_TOKEN) return 0; + return withExportLease(env.ANALYTICS_DB, (beforeRequest) => + exportPendingIdentityLinks(env, beforeRequest), + ); +} + +async function exportPendingIdentityLinks( + env: AnalyticsEnv, + beforeRequest: () => Promise, +): Promise { const result = await env.ANALYTICS_DB.prepare( `SELECT links.link_id, links.source_identity_id, links.canonical_id, links.linked_at FROM analytics_identity_links AS links JOIN analytics_identities AS identities ON identities.identity_id = links.source_identity_id WHERE links.posthog_state = 'pending' + AND links.posthog_attempts < 20 + AND (links.posthog_next_attempt_at IS NULL OR julianday(links.posthog_next_attempt_at) <= julianday('now')) + AND identities.identity_type = 'web_visitor' AND identities.consent_level IN ('product', 'diagnostic', 'contribution') ORDER BY links.linked_at, links.link_id LIMIT ?`, @@ -445,22 +565,25 @@ export async function flushPendingIdentityLinks(env: AnalyticsEnv): Promise env.ANALYTICS_DB.prepare( `UPDATE analytics_identity_links SET posthog_attempts = posthog_attempts + 1, - posthog_last_error = ? + posthog_last_error = ?, + posthog_next_attempt_at = datetime('now', '+' || min(1800, 30 * (1 << min(posthog_attempts, 6))) || ' seconds') WHERE link_id = ? AND posthog_state = 'pending'`, ).bind(message.slice(0, 500), row.link_id), ), @@ -468,20 +591,7 @@ export async function flushPendingIdentityLinks(env: AnalyticsEnv): Promise - env.ANALYTICS_DB.prepare( - `UPDATE analytics_identity_links - SET posthog_attempts = posthog_attempts + 1, - posthog_last_error = ? - WHERE link_id = ? AND posthog_state = 'pending'`, - ).bind(message, row.link_id), - ), - ); - throw new Error(message); - } + await response.body?.cancel(); await env.ANALYTICS_DB.batch( rows.map((row) => @@ -508,6 +618,7 @@ async function markPosthogDeletionFailure( `UPDATE analytics_deletion_requests SET posthog_attempts = posthog_attempts + 1, posthog_last_error_class = ?, + next_attempt_at = datetime('now', '+30 minutes'), posthog_state = CASE WHEN posthog_attempts + 1 >= ? THEN 'blocked' ELSE 'pending' @@ -518,77 +629,161 @@ async function markPosthogDeletionFailure( .run(); } -/** Submit accepted installation erasures through PostHog's supported deletion API. */ +/** A capture acknowledgement is not an erasure acknowledgement. Poll verified status. */ export async function flushPendingDeletions(env: AnalyticsEnv): Promise { if (!env.POSTHOG_PERSONAL_API_KEY || !env.POSTHOG_PROJECT_ID) return 0; + return withExportLease(env.ANALYTICS_DB, (beforeRequest) => + processPendingDeletion(env, beforeRequest), + ); +} +async function processPendingDeletion( + env: AnalyticsEnv, + beforeRequest: () => Promise, +): Promise { const result = await env.ANALYTICS_DB.prepare( - `SELECT request_id, posthog_distinct_id, posthog_attempts + `SELECT request_id, posthog_distinct_id, posthog_attempts, requested_at, + posthog_person_uuid, posthog_submitted_at FROM analytics_deletion_requests WHERE posthog_state = 'pending' + AND (next_attempt_at IS NULL OR julianday(next_attempt_at) <= julianday('now')) ORDER BY requested_at, request_id LIMIT ?`, ) .bind(POSTHOG_DELETION_BATCH_SIZE) .all(); - let submitted = 0; + const base = `${POSTHOG_API_HOST}/api/projects/${encodeURIComponent(env.POSTHOG_PROJECT_ID!)}`; + const headers = { + Authorization: `Bearer ${env.POSTHOG_PERSONAL_API_KEY}`, + "Content-Type": "application/json", + }; + const api = async (path: string, init: RequestInit = {}) => { + await beforeRequest(); + return readBoundedJson(await posthogRequest(`${base}${path}`, { ...init, headers }), 64 * 1024); + }; + let failed = false; + let completed = 0; for (const row of result.results) { try { - const response = await fetch( - `${POSTHOG_API_HOST}/api/projects/${encodeURIComponent(env.POSTHOG_PROJECT_ID)}/persons/bulk_delete/`, - { - method: "POST", - headers: { - Authorization: `Bearer ${env.POSTHOG_PERSONAL_API_KEY}`, - "Content-Type": "application/json", - }, - body: JSON.stringify({ - distinct_ids: [row.posthog_distinct_id], - delete_events: true, - delete_recordings: false, - keep_person: false, - }), - }, + if (!INSTALLATION_ID_PATTERN.test(row.posthog_distinct_id)) { + await env.ANALYTICS_DB.prepare( + "UPDATE analytics_deletion_requests SET posthog_state = 'blocked', posthog_last_error_class = 'linked-identity-review' WHERE request_id = ?", + ) + .bind(row.request_id) + .run(); + failed = true; + continue; + } + if (row.posthog_person_uuid) { + const body = await api( + `/persons/deletion_status/?person_uuid=${encodeURIComponent(row.posthog_person_uuid)}&limit=10`, + ); + if (!isRecord(body) || !Array.isArray(body.results)) + throw new TransportFailure("invalid-json"); + const status = body.results.find( + (item: unknown) => + isRecord(item) && + item.person_uuid === row.posthog_person_uuid && + typeof item.created_at === "string" && + Date.parse(item.created_at) >= Date.parse(row.requested_at), + ) as Record | undefined; + if (status) { + const verified = + status.status === "completed" && + typeof status.delete_verified_at === "string" && + Date.parse(status.delete_verified_at) >= Date.parse(String(status.created_at)); + await env.ANALYTICS_DB.prepare( + `UPDATE analytics_deletion_requests SET posthog_state = ?, + posthog_verified_at = ?, posthog_last_error_class = ?, + completed_at = ?, + next_attempt_at = datetime('now', '+30 minutes') WHERE request_id = ?`, + ) + .bind( + verified ? "completed" : "pending", + verified ? status.delete_verified_at : null, + null, + verified ? status.delete_verified_at : null, + row.request_id, + ) + .run(); + // Completion is PostHog's verified asynchronous erasure, not its + // submission acknowledgement. The tombstone rejects future uploads; + // the shared export lease prevents a later exporter reusing this ID. + if (verified) completed += 1; + continue; + } + } + + // Resolve and save the person UUID before a possibly ambiguous submission. + // Never erase a person that also represents another installation/account. + const people = await api( + `/persons/?distinct_id=${encodeURIComponent(row.posthog_distinct_id)}&limit=2`, ); - if (!response.ok) { - await markPosthogDeletionFailure(env.ANALYTICS_DB, row, `http-${response.status}`); + if (!isRecord(people) || !Array.isArray(people.results)) + throw new TransportFailure("invalid-json"); + const person: unknown = people.results[0]; + if ( + people.results.length !== 1 || + !isRecord(person) || + typeof person.uuid !== "string" || + !/^[0-9a-f-]{36}$/i.test(person.uuid) || + !Array.isArray(person.distinct_ids) || + person.distinct_ids.length !== 1 || + person.distinct_ids[0] !== row.posthog_distinct_id + ) { + await markPosthogDeletionFailure( + env.ANALYTICS_DB, + row, + people.results.length === 0 ? "export-not-yet-visible" : "linked-identity-review", + ); + failed = true; continue; } - const body = (await response.json().catch(() => null)) as { - readonly persons_found?: unknown; - readonly events_queued_for_deletion?: unknown; - readonly deletion_errors?: unknown; - } | null; + await env.ANALYTICS_DB.prepare( + "UPDATE analytics_deletion_requests SET posthog_person_uuid = ? WHERE request_id = ?", + ) + .bind(person.uuid, row.request_id) + .run(); + const body = await api("/persons/bulk_delete/", { + method: "POST", + body: JSON.stringify({ + ids: [person.uuid], + delete_events: true, + delete_recordings: false, + keep_person: false, + }), + }); if ( - body?.persons_found !== 1 || + !isRecord(body) || + body.persons_found !== 1 || body.events_queued_for_deletion !== true || !Array.isArray(body.deletion_errors) || body.deletion_errors.length !== 0 ) { - await markPosthogDeletionFailure(env.ANALYTICS_DB, row, "invalid-acknowledgement"); - continue; + throw new TransportFailure("invalid-json"); } await env.ANALYTICS_DB.prepare( `UPDATE analytics_deletion_requests - SET posthog_state = 'completed', + SET posthog_submitted_at = CURRENT_TIMESTAMP, posthog_attempts = posthog_attempts + 1, posthog_last_error_class = NULL, - completed_at = CURRENT_TIMESTAMP + next_attempt_at = datetime('now', '+30 minutes') WHERE request_id = ? AND posthog_state = 'pending'`, ) .bind(row.request_id) .run(); - submitted += 1; } catch (error) { await markPosthogDeletionFailure( env.ANALYTICS_DB, row, - error instanceof Error ? error.name || "network" : "network", + error instanceof TransportFailure ? error.kind : "internal", ); + failed = true; } } - return submitted; + if (failed) throw new Error("posthog-deletion-incomplete"); + return completed; } function identityType(identityId: string): "web_visitor" | "desktop_installation" { @@ -708,13 +903,18 @@ async function handleIdentityLink( } try { const link = validateIdentityLinkPayload(await readJsonBody(request)); + // Installation-only erasure must not delete a merged person's other installations. + // Keep this unlaunched capability closed until account-scoped erasure is designed. + if (link.identityIds.some((id) => INSTALLATION_ID_PATTERN.test(id))) { + return jsonResponse({ error: "Desktop account linking is not available" }, 409); + } await persistIdentityLinks(env.ANALYTICS_DB, link.accountId, link.identityIds); context.waitUntil( flushPendingIdentityLinks(env).catch((error: unknown) => { console.error( JSON.stringify({ message: "PostHog identity linking failed", - error: error instanceof Error ? error.message : String(error), + error_class: error instanceof TransportFailure ? error.kind : "internal", }), ); }), @@ -727,7 +927,7 @@ async function handleIdentityLink( console.error( JSON.stringify({ message: "Identity linking failed", - error: error instanceof Error ? error.message : String(error), + error_class: error instanceof TransportFailure ? error.kind : "internal", }), ); return jsonResponse({ error: "Identity linking failed" }, 500); @@ -739,19 +939,47 @@ export async function pruneExpiredAnalyticsEvents( now = new Date(), ): Promise { const cutoff = new Date(now.valueOf() - RAW_EVENT_RETENTION_DAYS * 24 * 60 * 60 * 1_000); + const diagnosticCutoff = new Date(now.valueOf() - DIAGNOSTIC_EVENT_RETENTION_DAYS * 86400000); const result = await database .prepare( `DELETE FROM analytics_events WHERE event_id IN ( SELECT event_id FROM analytics_events - WHERE received_at < ? + WHERE julianday(received_at) < julianday(?) + OR (privacy_level = 'diagnostic' AND julianday(received_at) < julianday(?)) + OR (source = 'desktop' AND (julianday(occurred_at) < julianday(?) + OR (privacy_level = 'diagnostic' AND julianday(occurred_at) < julianday(?)))) ORDER BY received_at, event_id LIMIT ? )`, ) - .bind(cutoff.toISOString(), RETENTION_BATCH_SIZE) + .bind( + cutoff.toISOString(), + diagnosticCutoff.toISOString(), + cutoff.toISOString(), + diagnosticCutoff.toISOString(), + RETENTION_BATCH_SIZE, + ) .run(); + if (result.meta.changes === RETENTION_BATCH_SIZE) { + const remaining = await database + .prepare(`SELECT 1 AS expired FROM analytics_events + WHERE julianday(received_at) < julianday(?) + OR (privacy_level = 'diagnostic' AND julianday(received_at) < julianday(?)) + OR (source = 'desktop' AND (julianday(occurred_at) < julianday(?) + OR (privacy_level = 'diagnostic' AND julianday(occurred_at) < julianday(?)))) LIMIT 1`) + .bind( + cutoff.toISOString(), + diagnosticCutoff.toISOString(), + cutoff.toISOString(), + diagnosticCutoff.toISOString(), + ) + .first(); + // Stay bounded; the next scheduled batch continues. Do not report a clean + // retention pass while records older than the policy still remain. + if (remaining) throw new Error("retention-backlog"); + } return result.meta.changes; } @@ -770,57 +998,68 @@ async function handleInstallationDeletion(request: Request, env: AnalyticsEnv): const tokenHash = await sha256(requireInstallationToken(request)); const installationId = validateDeletionPayload(await readJsonBody(request)); const identity = await env.ANALYTICS_DB.prepare( - "SELECT deletion_token_hash, canonical_id FROM analytics_identities WHERE identity_id = ?", + "SELECT deletion_token_hash FROM analytics_identities WHERE identity_id = ?", ) .bind(installationId) .first<{ readonly deletion_token_hash: string | null; - readonly canonical_id: string; }>(); - // A local installation can ask to delete before its first accepted upload. - // There is no remote data to authenticate or erase in that case, so an - // idempotent acknowledgement lets the client safely rotate its local id. - if (!identity) { - return jsonResponse( - { - accepted: true, - local_state: "not_found", - posthog_state: "not_required", - }, - 202, - ); - } - if (!identity.deletion_token_hash || identity.deletion_token_hash !== tokenHash) { + if (identity && (!identity.deletion_token_hash || identity.deletion_token_hash !== tokenHash)) { throw new InstallationAuthenticationError("Installation authentication failed"); } const requestId = crypto.randomUUID(); const requestedAt = new Date().toISOString(); await env.ANALYTICS_DB.batch([ + env.ANALYTICS_DB.prepare( + `INSERT INTO analytics_deleted_installations (installation_id, deletion_token_hash, request_id, requested_at) + SELECT ?, ?, ?, ? WHERE NOT EXISTS ( + SELECT 1 FROM analytics_identities WHERE identity_id = ? + AND (deletion_token_hash IS NULL OR deletion_token_hash <> ?) + ) ON CONFLICT(installation_id) DO NOTHING`, + ).bind(installationId, tokenHash, requestId, requestedAt, installationId, tokenHash), env.ANALYTICS_DB.prepare( `INSERT INTO analytics_deletion_requests ( - request_id, installation_id, posthog_distinct_id, requested_at, posthog_state - ) VALUES (?, ?, ?, ?, 'pending')`, - ).bind(requestId, installationId, identity.canonical_id, requestedAt), + request_id, installation_id, posthog_distinct_id, requested_at, posthog_state, + completed_at, next_attempt_at, posthog_last_error_class + ) SELECT tomb.request_id, tomb.installation_id, tomb.installation_id, tomb.requested_at, + CASE WHEN identities.canonical_id <> tomb.installation_id THEN 'blocked' + WHEN COALESCE(identities.posthog_attempted, 0) = 0 THEN 'completed' ELSE 'pending' END, + CASE WHEN identities.canonical_id <> tomb.installation_id THEN NULL + WHEN COALESCE(identities.posthog_attempted, 0) = 0 THEN tomb.requested_at ELSE NULL END, + datetime('now', '+5 minutes'), + CASE WHEN identities.canonical_id <> tomb.installation_id THEN 'linked-identity-review' ELSE NULL END + FROM analytics_deleted_installations AS tomb + LEFT JOIN analytics_identities AS identities ON identities.identity_id = tomb.installation_id + WHERE tomb.installation_id = ? AND tomb.deletion_token_hash = ? + ON CONFLICT(request_id) DO NOTHING`, + ).bind(installationId, tokenHash), env.ANALYTICS_DB.prepare( - "DELETE FROM analytics_identity_links WHERE source_identity_id = ?", - ).bind(installationId), - env.ANALYTICS_DB.prepare("DELETE FROM analytics_consents WHERE identity_id = ?").bind( - installationId, - ), - env.ANALYTICS_DB.prepare("DELETE FROM analytics_events WHERE distinct_id = ?").bind( - installationId, - ), - env.ANALYTICS_DB.prepare("DELETE FROM analytics_identities WHERE identity_id = ?").bind( - installationId, + `DELETE FROM analytics_identity_links WHERE source_identity_id = ? AND EXISTS + (SELECT 1 FROM analytics_deleted_installations WHERE installation_id = ? AND deletion_token_hash = ?)`, + ).bind(installationId, installationId, tokenHash), + ...(["analytics_consents", "analytics_events", "analytics_identities"] as const).map( + (table) => + env.ANALYTICS_DB.prepare( + `DELETE FROM ${table} WHERE ${table === "analytics_events" ? "distinct_id" : "identity_id"} = ? AND EXISTS + (SELECT 1 FROM analytics_deleted_installations WHERE installation_id = ? AND deletion_token_hash = ?)`, + ).bind(installationId, installationId, tokenHash), ), ]); + const receipt = await env.ANALYTICS_DB.prepare( + `SELECT requests.request_id, requests.posthog_state FROM analytics_deleted_installations AS tomb + JOIN analytics_deletion_requests AS requests ON requests.request_id = tomb.request_id + WHERE tomb.installation_id = ? AND tomb.deletion_token_hash = ?`, + ) + .bind(installationId, tokenHash) + .first<{ request_id: string; posthog_state: string }>(); + if (!receipt) throw new InstallationAuthenticationError("Installation authentication failed"); return jsonResponse( { accepted: true, - request_id: requestId, + request_id: receipt.request_id, local_state: "deleted", - posthog_state: "pending", + posthog_state: receipt.posthog_state, }, 202, ); @@ -872,7 +1111,7 @@ async function handleIngestion( console.error( JSON.stringify({ message: "PostHog forwarding failed", - error: error instanceof Error ? error.message : String(error), + error_class: error instanceof TransportFailure ? error.kind : "internal", }), ); }), @@ -888,7 +1127,7 @@ async function handleIngestion( console.error( JSON.stringify({ message: "Analytics ingestion failed", - error: error instanceof Error ? error.message : String(error), + error_class: error instanceof TransportFailure ? error.kind : "internal", }), ); return jsonResponse({ error: "Analytics ingestion failed" }, 500, origin); @@ -899,20 +1138,55 @@ const worker: ExportedHandler = { async fetch(request, env, context) { const url = new URL(request.url); if (request.method === "GET" && url.pathname === "/health") { - return jsonResponse({ - status: "ready", - storage: "configured", - desktop_ingestion: env.DESKTOP_INGESTION_ENABLED === "true" ? "enabled" : "disabled", - rate_limiting: env.ANALYTICS_INGESTION_RATE_LIMITER - ? "configured" - : "pending_configuration", - posthog_forwarding: env.POSTHOG_PROJECT_TOKEN ? "configured" : "pending_configuration", - posthog_deletion: - env.POSTHOG_PERSONAL_API_KEY && env.POSTHOG_PROJECT_ID + let storageReady = false; + let retentionReady = false; + try { + // Compile against the required schema without scanning or exposing user rows. + await env.ANALYTICS_DB.prepare(`SELECT events.posthog_next_attempt_at, identities.posthog_attempted, + deletions.posthog_person_uuid, tomb.deletion_token_hash, leases.expires_at, maintenance.outcome + FROM analytics_events AS events, analytics_identities AS identities, + analytics_deletion_requests AS deletions, analytics_deleted_installations AS tomb, + analytics_maintenance_leases AS leases, analytics_maintenance_status AS maintenance + LIMIT 0`).all(); + storageReady = true; + const retention = await env.ANALYTICS_DB.prepare( + "SELECT completed_at, outcome FROM analytics_maintenance_status WHERE name = 'retention'", + ).first<{ completed_at: string; outcome: string }>(); + retentionReady = + retention?.outcome === "ok" && + Date.now() - Date.parse(retention.completed_at) < 20 * 60 * 1000; + } catch { + // Health is safe and useful even with an unavailable DB or unapplied migration. + } + return jsonResponse( + { + status: storageReady ? "ready" : "degraded", + contract_revision: "2", + storage: storageReady ? "ready" : "unavailable_or_unmigrated", + retention: retentionReady ? "recent_success" : "pending_verification", + activation_prerequisites_configured: Boolean( + storageReady && + retentionReady && + env.ANALYTICS_INGESTION_RATE_LIMITER && + env.POSTHOG_PROJECT_TOKEN && + env.POSTHOG_PERSONAL_API_KEY && + env.POSTHOG_PROJECT_ID, + ), + desktop_ingestion: env.DESKTOP_INGESTION_ENABLED === "true" ? "enabled" : "disabled", + rate_limiting: env.ANALYTICS_INGESTION_RATE_LIMITER ? "configured" : "pending_configuration", - identity_linking: env.IDENTITY_LINK_TOKEN ? "configured" : "pending_configuration", - }); + posthog_forwarding: env.POSTHOG_PROJECT_TOKEN ? "configured" : "pending_configuration", + desktop_posthog_export: + env.DESKTOP_POSTHOG_EXPORT_ENABLED === "true" ? "enabled" : "disabled", + posthog_deletion: + env.POSTHOG_PERSONAL_API_KEY && env.POSTHOG_PROJECT_ID + ? "configured" + : "pending_configuration", + identity_linking: env.IDENTITY_LINK_TOKEN ? "configured" : "pending_configuration", + }, + storageReady ? 200 : 503, + ); } if (request.method === "OPTIONS" && url.pathname === "/v1/events") { const origin = request.headers.get("Origin"); @@ -944,18 +1218,37 @@ const worker: ExportedHandler = { async scheduled(_controller, env, context) { context.waitUntil( - pruneExpiredAnalyticsEvents(env.ANALYTICS_DB) - .then(() => flushPendingDeletions(env)) - .then(() => flushPendingIdentityLinks(env)) - .then(() => flushPendingEvents(env)) - .catch((error: unknown) => { - console.error( - JSON.stringify({ - message: "Scheduled PostHog forwarding failed", - error: error instanceof Error ? error.message : String(error), - }), - ); - }), + (async () => { + for (const [name, operation] of [ + ["retention", () => pruneExpiredAnalyticsEvents(env.ANALYTICS_DB)], + ["deletion", () => flushPendingDeletions(env)], + ["identity-export", () => flushPendingIdentityLinks(env)], + ["event-export", () => flushPendingEvents(env)], + ] as const) { + let outcome = "ok"; + try { + await operation(); + } catch { + outcome = "failed"; + console.error( + JSON.stringify({ message: "Analytics maintenance failed", operation: name }), + ); + } + try { + await env.ANALYTICS_DB.prepare(`INSERT INTO analytics_maintenance_status (name, completed_at, outcome) + VALUES (?, ?, ?) ON CONFLICT(name) DO UPDATE SET completed_at = excluded.completed_at, outcome = excluded.outcome`) + .bind(name, new Date().toISOString(), outcome) + .run(); + } catch { + console.error( + JSON.stringify({ + message: "Analytics maintenance status unavailable", + operation: name, + }), + ); + } + } + })(), ); }, }; diff --git a/workers/events/src/readiness.test.ts b/workers/events/src/readiness.test.ts new file mode 100644 index 0000000..af2c42c --- /dev/null +++ b/workers/events/src/readiness.test.ts @@ -0,0 +1,532 @@ +import { afterEach, describe, expect, it, vi } from "vitest"; +import worker, { + flushPendingDeletions, + flushPendingEvents, + pruneExpiredAnalyticsEvents, +} from "./index"; +import { testDatabase } from "./sqlite.testSupport"; +import { posthogEventUuid, readBoundedJson } from "./transport"; +import { withExportLease } from "./exportLease"; + +const installation = "installation:10000000-0000-4000-8000-000000000001"; +const token = "a".repeat(64); +const databases: ReturnType[] = []; +function fixture(beforeReadiness?: Parameters[0]) { + const store = testDatabase(beforeReadiness); + databases.push(store); + const tasks: Promise[] = []; + const env = { + ANALYTICS_DB: store.database, + DESKTOP_INGESTION_ENABLED: "true", + ANALYTICS_INGESTION_RATE_LIMITER: { limit: async () => ({ success: true }) }, + }; + const context = { + waitUntil: (task: Promise) => { + tasks.push(task); + }, + } as ExecutionContext; + const request = (path: string, data: unknown, secret = token) => + worker.fetch!( + new Request(`https://example.invalid${path}`, { + method: "POST", + headers: { "Content-Type": "application/json", "X-Scient-Installation-Token": secret }, + body: JSON.stringify(data), + }) as Request, + env, + context, + ); + const event = (id = crypto.randomUUID()) => ({ + schema_version: 1, + source: "desktop", + events: [ + { + id, + name: "app.session.started", + distinct_id: installation, + session_id: "session:10000000-0000-4000-8000-000000000002", + occurred_at: new Date().toISOString(), + privacy_level: "essential", + consent_level: "essential", + properties: { + appVersion: "0.6.8", + buildChannel: "stable", + platform: "macos", + architecture: "arm64", + }, + }, + ], + }); + return { + ...store, + env, + request, + event, + tasks, + context, + erase: () => + request("/v1/installations/delete", { schema_version: 1, installation_id: installation }), + }; +} +afterEach(() => { + vi.unstubAllGlobals(); + vi.restoreAllMocks(); + for (const db of databases.splice(0)) db.close(); +}); + +describe("inactive gateway readiness with real SQL", () => { + it("keeps diagnostic events queryable in D1 without exporting them to PostHog", async () => { + const f = fixture(); + await f.request("/v1/events", f.event()); + f.sqlite.exec(`UPDATE analytics_events SET event_name = 'app.diagnostics', + privacy_level = 'diagnostic', consent_level = 'diagnostic', properties_json = + '{"queuedCountBucket":"0","droppedCountBucket":"0","retryCountBucket":"0","deliveryClass":"idle"}'`); + const fetcher = vi.fn(); + vi.stubGlobal("fetch", fetcher); + expect( + await flushPendingEvents({ + ...f.env, + POSTHOG_PROJECT_TOKEN: "synthetic", + DESKTOP_POSTHOG_EXPORT_ENABLED: "true", + }), + ).toBe(0); + expect(fetcher).not.toHaveBeenCalled(); + expect(f.sqlite.prepare("SELECT count(*) AS n FROM analytics_events").get()!.n).toBe(1); + }); + it("keeps desktop export off independently of existing website forwarding", async () => { + const f = fixture(); + await f.request("/v1/events", f.event()); + const fetcher = vi.fn().mockResolvedValue(new Response(null, { status: 200 })); + vi.stubGlobal("fetch", fetcher); + const env = { ...f.env, POSTHOG_PROJECT_TOKEN: "synthetic" }; + expect(await flushPendingEvents(env)).toBe(0); + expect(fetcher).not.toHaveBeenCalled(); + f.sqlite.exec("UPDATE analytics_events SET source = 'website', event_name = 'page_viewed'"); + expect(await flushPendingEvents(env)).toBe(1); + expect(fetcher).toHaveBeenCalledOnce(); + }); + + it("quarantines invalid persisted desktop properties without blocking valid rows", async () => { + const f = fixture(); + const invalid = f.event(); + await f.request("/v1/events", invalid); + await f.request("/v1/events", f.event()); + f.sqlite + .prepare("UPDATE analytics_events SET properties_json = ? WHERE event_id = ?") + .run(JSON.stringify({ prompt: "private research" }), invalid.events[0]!.id); + const fetcher = vi.fn().mockResolvedValue(new Response(null, { status: 200 })); + vi.stubGlobal("fetch", fetcher); + expect( + await flushPendingEvents({ + ...f.env, + POSTHOG_PROJECT_TOKEN: "synthetic", + DESKTOP_POSTHOG_EXPORT_ENABLED: "true", + }), + ).toBe(1); + expect(String(fetcher.mock.calls[0]?.[1]?.body)).not.toContain("private research"); + expect( + f.sqlite + .prepare( + "SELECT posthog_attempts, posthog_last_error FROM analytics_events WHERE event_id = ?", + ) + .get(invalid.events[0]!.id), + ).toMatchObject({ posthog_attempts: 20, posthog_last_error: "contract-rejected" }); + }); + + it("keeps retention bounded and reports an outstanding backlog as unqualified", async () => { + const f = fixture(); + f.sqlite + .exec(`WITH RECURSIVE seq(n) AS (SELECT 1 UNION ALL SELECT n + 1 FROM seq WHERE n < 5001) + INSERT INTO analytics_events (event_id, event_name, source, privacy_level, occurred_at, received_at, distinct_id, properties_json) + SELECT 'expired-' || n, 'test', 'desktop', 'essential', '2020-01-01', '2020-01-01', 'test', '{}' FROM seq`); + await expect(pruneExpiredAnalyticsEvents(f.database)).rejects.toThrow("retention-backlog"); + expect(f.sqlite.prepare("SELECT count(*) AS n FROM analytics_events").get()!.n).toBe(1); + expect(await pruneExpiredAnalyticsEvents(f.database)).toBe(1); + }); + + it("does not reset diagnostic retention when old offline events arrive or await export", async () => { + const f = fixture(); + const old = new Date(Date.now() - 31 * 86400000).toISOString(); + const diagnostic = { + ...f.event(), + events: [ + { + ...f.event().events[0]!, + name: "app.diagnostics", + occurred_at: old, + privacy_level: "diagnostic", + consent_level: "diagnostic", + properties: { + appVersion: "0.6.8", + buildChannel: "stable", + contractRevision: "2", + queuedCountBucket: "0", + retryCountBucket: "0", + droppedCountBucket: "unknown", + deliveryClass: "idle", + }, + }, + ], + }; + expect( + ( + await f.request("/v1/events", { + ...diagnostic, + events: [{ ...diagnostic.events[0]!, occurred_at: new Date().toISOString() }], + }) + ).status, + ).toBe(202); + expect((await f.request("/v1/events", diagnostic)).status).toBe(400); + f.sqlite + .prepare("UPDATE analytics_events SET privacy_level = 'diagnostic', occurred_at = ?") + .run(old); + const fetcher = vi.fn().mockResolvedValue(new Response(null, { status: 200 })); + vi.stubGlobal("fetch", fetcher); + expect( + await flushPendingEvents({ + ...f.env, + POSTHOG_PROJECT_TOKEN: "synthetic", + DESKTOP_POSTHOG_EXPORT_ENABLED: "true", + }), + ).toBe(0); + expect(fetcher).not.toHaveBeenCalled(); + expect(await pruneExpiredAnalyticsEvents(f.database)).toBe(1); + }); + + it("does not mark a blocked linked-identity erasure as completed", async () => { + const f = fixture(); + await f.request("/v1/events", f.event()); + f.sqlite.exec("UPDATE analytics_identities SET canonical_id = 'account:synthetic'"); + expect((await f.erase()).status).toBe(202); + expect( + f.sqlite.prepare("SELECT posthog_state, completed_at FROM analytics_deletion_requests").get(), + ).toMatchObject({ posthog_state: "blocked", completed_at: null }); + }); + + it("records deletion maintenance failure without preventing the other maintenance passes", async () => { + const f = fixture(); + await f.request("/v1/events", f.event()); + f.sqlite.exec("UPDATE analytics_identities SET posthog_attempted = 1"); + await f.erase(); + f.sqlite.exec("UPDATE analytics_deletion_requests SET next_attempt_at = NULL"); + vi.spyOn(console, "error").mockImplementation(() => undefined); + vi.stubGlobal("fetch", vi.fn().mockRejectedValue(new Error("private remote detail"))); + await worker.scheduled!( + {} as ScheduledController, + { ...f.env, POSTHOG_PROJECT_ID: "1", POSTHOG_PERSONAL_API_KEY: "synthetic" }, + f.context, + ); + await Promise.all(f.tasks); + expect( + f.sqlite + .prepare("SELECT outcome FROM analytics_maintenance_status WHERE name = 'deletion'") + .get(), + ).toEqual({ outcome: "failed" }); + expect( + f.sqlite + .prepare("SELECT outcome FROM analytics_maintenance_status WHERE name = 'retention'") + .get(), + ).toEqual({ outcome: "ok" }); + expect( + f.sqlite.prepare("SELECT count(*) AS n FROM analytics_maintenance_status").get()!.n, + ).toBe(4); + expect(console.error).not.toHaveBeenCalledWith( + expect.stringContaining("private remote detail"), + ); + }); + + it("preserves a legacy erasure across migration without inventing authentication or completion", async () => { + const f = fixture((sqlite) => { + sqlite + .prepare(`INSERT INTO analytics_deletion_requests + (request_id, installation_id, requested_at, posthog_state, posthog_distinct_id) + VALUES ('old-request', ?, '2026-08-01T00:00:00Z', 'pending', ?)`) + .run(installation, installation); + }); + expect((await f.request("/v1/events", f.event())).status).toBe(403); + expect((await f.erase()).status).toBe(403); + expect( + f.sqlite + .prepare( + "SELECT request_id, posthog_state, posthog_last_error_class FROM analytics_deletion_requests", + ) + .all(), + ).toEqual([ + { + request_id: "old-request", + posthog_state: "blocked", + posthog_last_error_class: "legacy-unverified-deletion", + }, + ]); + expect(f.sqlite.prepare("SELECT count(*) AS n FROM analytics_events").get()!.n).toBe(0); + }); + + it("does not let out-of-order observations roll consent or last-seen time backward", async () => { + const f = fixture(); + const latest = f.event(); + const old = f.event(); + old.events[0]!.consent_level = "product"; + old.events[0]!.occurred_at = new Date(Date.now() - 3600000).toISOString(); + await f.request("/v1/events", latest); + await f.request("/v1/events", old); + expect( + f.sqlite + .prepare( + "SELECT consent_level, first_seen_at, last_seen_at, product_first_seen_at FROM analytics_identities", + ) + .get(), + ).toMatchObject({ + consent_level: "essential", + first_seen_at: old.events[0]!.occurred_at, + last_seen_at: latest.events[0]!.occurred_at, + product_first_seen_at: old.events[0]!.occurred_at, + }); + }); + + it("renews a live lease before transport and aborts an expired owner", async () => { + const f = fixture(); + let requests = 0; + await expect( + withExportLease(f.database, async (beforeRequest) => { + await beforeRequest(); + f.sqlite.exec("UPDATE analytics_maintenance_leases SET expires_at = 0"); + expect( + await withExportLease(f.database, async (newOwnerRequest) => { + await newOwnerRequest(); + requests += 1; + return 1; + }), + ).toBe(1); + await beforeRequest(); + requests += 1; + return 1; + }), + ).rejects.toThrow("export-lease-lost"); + expect(requests).toBe(1); + }); + + it("rejects desktop account linking even with service credentials", async () => { + const f = fixture(); + const response = await worker.fetch!( + new Request("https://example.invalid/v1/identity/link", { + method: "POST", + headers: { "Content-Type": "application/json", Authorization: "Bearer synthetic" }, + body: JSON.stringify({ + schema_version: 1, + account_id: "account:10000000-0000-4000-8000-000000000003", + identity_ids: [installation], + }), + }) as Request, + { ...f.env, IDENTITY_LINK_TOKEN: "synthetic" }, + f.context, + ); + expect(response.status).toBe(409); + expect(f.sqlite.prepare("SELECT count(*) AS n FROM analytics_identity_links").get()!.n).toBe(0); + }); + it("deduplicates accepted uploads and blocks resurrection after authenticated erasure", async () => { + const f = fixture(); + const payload = f.event(); + expect((await f.request("/v1/events", payload)).status).toBe(202); + expect((await f.request("/v1/events", payload)).status).toBe(202); + expect(f.sqlite.prepare("SELECT count(*) AS n FROM analytics_events").get()!.n).toBe(1); + const first = await (await f.erase()).json(); + expect(first).toMatchObject({ accepted: true, posthog_state: "completed" }); + expect(await (await f.erase()).json()).toEqual(first); + expect((await f.request("/v1/events", f.event())).status).toBe(403); + expect(f.sqlite.prepare("SELECT count(*) AS n FROM analytics_events").get()!.n).toBe(0); + expect(f.sqlite.prepare("SELECT count(*) AS n FROM analytics_identities").get()!.n).toBe(0); + expect( + ( + await f.request( + "/v1/installations/delete", + { schema_version: 1, installation_id: installation }, + "b".repeat(64), + ) + ).status, + ).toBe(403); + }); + + it("tombstones a never-uploaded installation too; SQL guards cover stale writers", async () => { + const f = fixture(); + expect((await f.erase()).status).toBe(202); + expect(() => + f.sqlite + .prepare(`INSERT INTO analytics_identities + (identity_id, identity_type, canonical_id, consent_level, first_seen_at, last_seen_at) + VALUES (?, 'desktop_installation', ?, 'essential', '', '')`) + .run(installation, installation), + ).toThrow("deleted-installation"); + expect((await f.request("/v1/events", f.event())).status).toBe(403); + }); + + it("keeps an in-flight export and deletion truthful and serializes exporters", async () => { + const f = fixture(); + await f.request("/v1/events", f.event()); + let finish!: (response: Response) => void; + let started!: () => void; + const signal = new Promise((resolve) => { + started = resolve; + }); + vi.stubGlobal( + "fetch", + vi.fn(() => { + started(); + return new Promise((resolve) => { + finish = resolve; + }); + }), + ); + const env = { + ...f.env, + POSTHOG_PROJECT_TOKEN: "synthetic", + DESKTOP_POSTHOG_EXPORT_ENABLED: "true", + POSTHOG_PROJECT_ID: "1", + POSTHOG_PERSONAL_API_KEY: "synthetic", + }; + const sending = flushPendingEvents(env); + await signal; + expect(await flushPendingEvents(env)).toBe(0); + const receipt = await (await f.erase()).json(); + expect(receipt).toMatchObject({ posthog_state: "pending" }); + expect(await (await f.erase()).json()).toEqual(receipt); + expect(await flushPendingDeletions(env)).toBe(0); + finish(new Response("{}", { status: 200 })); + await sending; + expect(f.sqlite.prepare("SELECT count(*) AS n FROM analytics_events").get()!.n).toBe(0); + expect( + f.sqlite.prepare("SELECT posthog_state FROM analytics_deletion_requests").get()! + .posthog_state, + ).toBe("pending"); + }); + + it("uses a stable UUID and bounded transport for retries, without raw error persistence", async () => { + const f = fixture(); + await f.request("/v1/events", f.event()); + const fetcher = vi + .fn() + .mockRejectedValueOnce(new Error("private credential/path")) + .mockResolvedValue(new Response("{}")); + vi.stubGlobal("fetch", fetcher); + const env = { + ...f.env, + POSTHOG_PROJECT_TOKEN: "synthetic", + DESKTOP_POSTHOG_EXPORT_ENABLED: "true", + }; + await expect(flushPendingEvents(env)).rejects.toThrow("network"); + expect( + f.sqlite.prepare("SELECT posthog_last_error FROM analytics_events").get()!.posthog_last_error, + ).toBe("network"); + f.sqlite.exec("UPDATE analytics_events SET posthog_next_attempt_at = NULL"); + expect(await flushPendingEvents(env)).toBe(1); + const first = JSON.parse(fetcher.mock.calls[0]![1].body).batch[0]; + const second = JSON.parse(fetcher.mock.calls[1]![1].body).batch[0]; + expect(first).toEqual(second); + expect(first.uuid).toMatch( + /^[a-f0-9]{8}-[a-f0-9]{4}-8[a-f0-9]{3}-[89ab][a-f0-9]{3}-[a-f0-9]{12}$/, + ); + expect(fetcher.mock.calls[0]![1].redirect).toBe("error"); + expect(fetcher.mock.calls[0]![1].signal).toBeInstanceOf(AbortSignal); + }); + + it("completes only after provider-verified erasure and continues rejecting the old identity", async () => { + const f = fixture(); + await f.request("/v1/events", f.event()); + f.sqlite.exec("UPDATE analytics_identities SET posthog_attempted = 1"); + await f.erase(); + f.sqlite.exec("UPDATE analytics_deletion_requests SET next_attempt_at = NULL"); + const uuid = "20000000-0000-4000-8000-000000000001"; + const fetcher = vi + .fn() + .mockResolvedValueOnce(Response.json({ results: [{ uuid, distinct_ids: [installation] }] })) + .mockResolvedValueOnce( + Response.json({ persons_found: 1, events_queued_for_deletion: true, deletion_errors: [] }), + ) + .mockResolvedValueOnce( + Response.json({ + results: [ + { + person_uuid: uuid, + created_at: new Date(Date.now() + 1000).toISOString(), + status: "pending", + delete_verified_at: null, + }, + ], + }), + ) + .mockResolvedValueOnce( + Response.json({ + results: [ + { + person_uuid: uuid, + created_at: new Date(Date.now() + 1000).toISOString(), + status: "completed", + delete_verified_at: new Date(Date.now() + 2000).toISOString(), + }, + ], + }), + ); + vi.stubGlobal("fetch", fetcher); + const env = { ...f.env, POSTHOG_PROJECT_ID: "1", POSTHOG_PERSONAL_API_KEY: "synthetic" }; + expect(await flushPendingDeletions(env)).toBe(0); + expect( + f.sqlite + .prepare("SELECT posthog_state, posthog_submitted_at FROM analytics_deletion_requests") + .get(), + ).toMatchObject({ posthog_state: "pending", posthog_submitted_at: expect.any(String) }); + f.sqlite.exec("UPDATE analytics_deletion_requests SET next_attempt_at = NULL"); + expect(await flushPendingDeletions(env)).toBe(0); + f.sqlite.exec("UPDATE analytics_deletion_requests SET next_attempt_at = NULL"); + expect(await flushPendingDeletions(env)).toBe(1); + expect(await (await f.erase()).json()).toMatchObject({ posthog_state: "completed" }); + expect((await f.request("/v1/events", f.event())).status).toBe(403); + expect(await flushPendingDeletions(env)).toBe(0); + expect( + f.sqlite + .prepare( + "SELECT posthog_verified_at, posthog_last_error_class, completed_at FROM analytics_deletion_requests", + ) + .get(), + ).toMatchObject({ + posthog_verified_at: expect.any(String), + posthog_last_error_class: null, + completed_at: expect.any(String), + }); + }); + + it("uses actual instants for retention, with a shorter diagnostic lifetime", async () => { + const f = fixture(); + const now = new Date("2026-08-31T12:00:00Z"); + const cutoff = new Date(now.valueOf() - 180 * 86400000).toISOString().slice(0, 10); + const insert = f.sqlite.prepare(`INSERT INTO analytics_events + (event_id, event_name, source, privacy_level, occurred_at, received_at, distinct_id, properties_json) + VALUES (?, 'test', 'desktop', ?, ?, ?, 'retention-test', '{}')`); + insert.run("keep", "essential", now.toISOString(), `${cutoff} 15:00:00`); + insert.run("expire", "essential", now.toISOString(), `${cutoff} 09:00:00`); + insert.run("diagnostic-old", "diagnostic", now.toISOString(), "2026-07-30 12:00:00"); + insert.run("diagnostic-new", "diagnostic", now.toISOString(), "2026-08-30 12:00:00"); + expect(await pruneExpiredAnalyticsEvents(f.database, now)).toBe(2); + expect( + f.sqlite + .prepare("SELECT event_id FROM analytics_events ORDER BY event_id") + .all() + .map((row) => row.event_id), + ).toEqual(["diagnostic-new", "keep"]); + }); +}); + +it("bounds unknown-length bodies while streaming and preserves stable legacy IDs", async () => { + let cancelled = false; + const body = new ReadableStream({ + pull(controller) { + controller.enqueue(new Uint8Array(1024)); + }, + cancel() { + cancelled = true; + }, + }); + await expect(readBoundedJson(new Response(body), 100)).rejects.toThrow("body-too-large"); + expect(cancelled).toBe(true); + expect(await posthogEventUuid("legacy:event-1")).toBe(await posthogEventUuid("legacy:event-1")); + expect(await posthogEventUuid("legacy:event-1")).not.toBe( + await posthogEventUuid("legacy:event-2"), + ); +}); diff --git a/workers/events/src/sqlite.testSupport.ts b/workers/events/src/sqlite.testSupport.ts new file mode 100644 index 0000000..fcc3019 --- /dev/null +++ b/workers/events/src/sqlite.testSupport.ts @@ -0,0 +1,49 @@ +/// +import { readFileSync, readdirSync } from "node:fs"; +import { DatabaseSync, type SQLInputValue } from "node:sqlite"; + +/** Exercises real migration/transaction SQL; not a substitute for D1 deployment proof. */ +export function testDatabase(beforeReadiness?: (sqlite: DatabaseSync) => void) { + const sqlite = new DatabaseSync(":memory:"); + const migrations = new URL("../../../migrations/", import.meta.url); + for (const file of readdirSync(migrations) + .filter((name) => name.endsWith(".sql")) + .sort()) { + if (file === "0006_analytics_readiness.sql") beforeReadiness?.(sqlite); + sqlite.exec(readFileSync(new URL(file, migrations), "utf8")); + } + const prepare = (sql: string, bindings: SQLInputValue[] = []) => { + const result = () => { + const before = Number(sqlite.prepare("SELECT total_changes() AS n").get()!.n); + const results = sqlite.prepare(sql).all(...bindings); + const changes = Number(sqlite.prepare("SELECT total_changes() AS n").get()!.n) - before; + return { success: true, results, meta: { changes } }; + }; + return { + bind: (...values: SQLInputValue[]) => prepare(sql, values), + first: async (column?: string) => { + const row = sqlite.prepare(sql).get(...bindings); + return row ? (column ? row[column] : row) : null; + }, + all: async () => result(), + run: async () => result(), + execute: result, + }; + }; + const adapter = { + prepare, + batch: async (statements: ReturnType[]) => { + sqlite.exec("BEGIN"); + try { + const results = statements.map((statement) => statement.execute()); + sqlite.exec("COMMIT"); + return results; + } catch (error) { + sqlite.exec("ROLLBACK"); + throw error; + } + }, + }; + // This test adapter implements only the D1 methods used by the gateway. + return { database: adapter as unknown as D1Database, sqlite, close: () => sqlite.close() }; +} diff --git a/workers/events/src/transport.ts b/workers/events/src/transport.ts new file mode 100644 index 0000000..1a88a3b --- /dev/null +++ b/workers/events/src/transport.ts @@ -0,0 +1,73 @@ +/** Bounded transport shared by ingestion and the optional analysis exporter. */ +export class TransportFailure extends Error { + constructor(readonly kind: "body-too-large" | "invalid-json" | "network" | "timeout" | "http") { + super(kind); + } +} + +export async function readBoundedJson( + message: Request | Response, + maximumBytes: number, +): Promise { + const declaredLength = Number(message.headers.get("Content-Length") ?? 0); + if (declaredLength > maximumBytes) throw new TransportFailure("body-too-large"); + const reader = message.body?.getReader(); + if (!reader) throw new TransportFailure("invalid-json"); + const chunks: Uint8Array[] = []; + let length = 0; + try { + while (true) { + const { done, value } = await reader.read(); + if (done) break; + length += value.byteLength; + if (length > maximumBytes) throw new TransportFailure("body-too-large"); + chunks.push(value); + } + const bytes = new Uint8Array(length); + let offset = 0; + for (const chunk of chunks) { + bytes.set(chunk, offset); + offset += chunk.byteLength; + } + try { + return JSON.parse(new TextDecoder().decode(bytes)) as unknown; + } catch { + throw new TransportFailure("invalid-json"); + } + } finally { + await reader.cancel().catch(() => undefined); + reader.releaseLock(); + } +} + +/** Never follow redirects with an analysis project token or deletion credential. */ +export async function posthogRequest(url: string, init: RequestInit): Promise { + try { + const response = await fetch(url, { + ...init, + redirect: "error", + signal: AbortSignal.timeout(5_000), + }); + if (!response.ok) { + await response.body?.cancel(); + throw new TransportFailure("http"); + } + return response; + } catch (error) { + if (error instanceof TransportFailure) throw error; + throw new TransportFailure( + error instanceof Error && error.name === "TimeoutError" ? "timeout" : "network", + ); + } +} + +/** Stable UUIDv8 for event IDs that can also come from legacy website writers. */ +export async function posthogEventUuid(id: string): Promise { + const hash = new Uint8Array( + await crypto.subtle.digest("SHA-256", new TextEncoder().encode(`scient:analytics:event:${id}`)), + ); + hash[6] = (hash[6]! & 0x0f) | 0x80; + hash[8] = (hash[8]! & 0x3f) | 0x80; + const hex = [...hash.subarray(0, 16)].map((byte) => byte.toString(16).padStart(2, "0")).join(""); + return `${hex.slice(0, 8)}-${hex.slice(8, 12)}-${hex.slice(12, 16)}-${hex.slice(16, 20)}-${hex.slice(20)}`; +} diff --git a/workers/events/worker-configuration.d.ts b/workers/events/worker-configuration.d.ts index 60d70a0..a332b56 100644 --- a/workers/events/worker-configuration.d.ts +++ b/workers/events/worker-configuration.d.ts @@ -1,8 +1,10 @@ /* eslint-disable */ -// Generated by Wrangler by running `wrangler types --config=workers/events/wrangler.jsonc --include-runtime=false --env-interface=AnalyticsWorkerBindings workers/events/worker-configuration.d.ts` (hash: 0ccff9c33ad214e4252157ed675b1c25) +// Generated by Wrangler by running `wrangler types --config=workers/events/wrangler.jsonc --include-runtime=false --env-interface=AnalyticsWorkerBindings workers/events/worker-configuration.d.ts` (hash: ec6c211557aaf856bf40f03bf0d7023a) interface __BaseEnv_AnalyticsWorkerBindings { ANALYTICS_DB: D1Database; ANALYTICS_INGESTION_RATE_LIMITER: RateLimit; + DESKTOP_INGESTION_ENABLED: "false"; + DESKTOP_POSTHOG_EXPORT_ENABLED: "false"; } declare namespace Cloudflare { interface GlobalProps { @@ -11,3 +13,11 @@ declare namespace Cloudflare { interface Env extends __BaseEnv_AnalyticsWorkerBindings {} } interface AnalyticsWorkerBindings extends __BaseEnv_AnalyticsWorkerBindings {} +type StringifyValues> = { + [Binding in keyof EnvType]: EnvType[Binding] extends string ? EnvType[Binding] : string; +}; +declare namespace NodeJS { + interface ProcessEnv extends StringifyValues< + Pick + > {} +} diff --git a/workers/events/wrangler.jsonc b/workers/events/wrangler.jsonc index 7217793..5dde4e4 100644 --- a/workers/events/wrangler.jsonc +++ b/workers/events/wrangler.jsonc @@ -4,6 +4,10 @@ "main": "src/index.ts", "compatibility_date": "2026-07-20", "compatibility_flags": ["nodejs_compat"], + "vars": { + "DESKTOP_INGESTION_ENABLED": "false", + "DESKTOP_POSTHOG_EXPORT_ENABLED": "false", + }, "routes": [ { "pattern": "events.scientfactory.com", From 33c037f89ebde080698ffd9a8958990ca27d5774 Mon Sep 17 00:00:00 2001 From: Yaacov Date: Sun, 6 Sep 2026 21:15:19 +0300 Subject: [PATCH 2/3] docs(privacy): match simplified desktop sharing controls --- src/pages/privacy.astro | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/pages/privacy.astro b/src/pages/privacy.astro index c15005b..fb6a3f7 100644 --- a/src/pages/privacy.astro +++ b/src/pages/privacy.astro @@ -66,7 +66,7 @@ import Layout from "../layouts/Layout.astro";

Desktop measurement is controlled separately inside Scient.

- Scient runs as a workspace layer on your computer. Builds with analytics available let you choose Off, Essential reliability, Product improvement, or Diagnostics in Privacy and analytics. The default is Off; making the feature available does not change your saved choice. Desktop events use a random installation identifier and never derive identity from a connected AI-provider account. + Scient runs as a workspace layer on your computer. In release builds with analytics available, usage and reliability sharing is on by default unless a different preference is saved. In Settings → General → Privacy and analytics, you can turn “Share usage and reliability” off or on and read “What’s shared?”. Existing Off and narrower sharing preferences are preserved; explicitly turning sharing off and back on enables feature usage, reliability and analytics-delivery counters. Desktop events use a random installation identifier and never derive identity from a connected AI-provider account.

Normal product analytics exclude prompts, assistant responses, research documents, filenames, file paths, URLs, source text, generated scientific content, credentials, provider account identities, and raw error messages. Scient does not use desktop autocapture or session replay. When you use a connected AI provider, that provider receives the prompts and supporting context needed for the session under its own terms and privacy policy. This website notice does not replace those terms. From 81b12ebdc6d72334882d085cafc2613d8e9455d1 Mon Sep 17 00:00:00 2001 From: Yaacov Date: Sun, 6 Sep 2026 21:31:33 +0300 Subject: [PATCH 3/3] docs(privacy): shorten desktop analytics explanation --- src/pages/privacy.astro | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/src/pages/privacy.astro b/src/pages/privacy.astro index fb6a3f7..81e052f 100644 --- a/src/pages/privacy.astro +++ b/src/pages/privacy.astro @@ -64,18 +64,18 @@ import Layout from "../layouts/Layout.astro";

Scient desktop

-

Desktop measurement is controlled separately inside Scient.

+

You control analytics in Scient.

- Scient runs as a workspace layer on your computer. In release builds with analytics available, usage and reliability sharing is on by default unless a different preference is saved. In Settings → General → Privacy and analytics, you can turn “Share usage and reliability” off or on and read “What’s shared?”. Existing Off and narrower sharing preferences are preserved; explicitly turning sharing off and back on enables feature usage, reliability and analytics-delivery counters. Desktop events use a random installation identifier and never derive identity from a connected AI-provider account. + Scient shares feature usage, failures and basic performance information to help improve the app. Sharing is on by default in supported releases; saved preferences are preserved. Turn it off or read “What’s shared?” in Settings → General → Privacy and analytics. Turning it back on includes usage, reliability and delivery counters.

- Normal product analytics exclude prompts, assistant responses, research documents, filenames, file paths, URLs, source text, generated scientific content, credentials, provider account identities, and raw error messages. Scient does not use desktop autocapture or session replay. When you use a connected AI provider, that provider receives the prompts and supporting context needed for the session under its own terms and privacy policy. This website notice does not replace those terms. + Analytics never collects conversations, file contents or names, paths, URLs, research results, credentials, provider account identities, or raw errors. There is no automatic click capture or session recording. Events use random installation identifiers. Your AI provider separately receives the content needed for your requests under its own privacy policy.

- Shared desktop events go to ScientFactory's central Cloudflare storage. Diagnostic-only events stay there, with scheduled removal after 30 days; other raw events are scheduled for removal after 180 days. Product and reliability events may also be processed in our EU-hosted PostHog project under PostHog-managed retention. We do not promise those downstream copies are physically deleted within the same periods. Local unsent events are filtered by age before delivery, and cleanup runs while the app is running. + We store events centrally in Cloudflare, with scheduled removal after 180 days, or 30 days for delivery diagnostics. Usage and reliability copies may also go to our EU-hosted PostHog project under its retention rules; those copies do not share the same deletion deadlines.

- An enabled build lets you request deletion for the current installation. Scient clears its local analytics state after the first-party gateway accepts the request and replaces the random analytics identifier. Downstream removal of the matching PostHog profile and events is processed separately; acceptance does not mean every copy is already gone. A minimal deletion receipt and identifier-verification record remain to prevent delayed retries from recreating deleted history. They contain no usage events or research content. + “Delete data” requests removal for this installation and resets its identifier after acceptance. PostHog deletion is processed separately, not instantly. We retain a minimal verification record, without usage or research content, to prevent delayed uploads from restoring deleted data.