Measure k: count the page's own script calls on both sides of net offload (browser v1.22.0, plugin v0.65.0, console v0.13.0)

[harper-joseph]

Closes #153. Follow-up to #152, which stated that the offload figure counted documents only and that the requests a page's own scripts make — when a crawler that executes JavaScript runs the page it was handed — were missing from both sides of the ledger. This measures that term and counts it.

Three packages, one PR, three release tags (the console's catalog guard scans the plugin's emit sites, so the plugin half cannot merge alone). Each stage tolerates the others being absent.

What k is and where it lands

k(page) = same-origin requests a page load makes beyond the document whose response no shared cache would serve — the XHR/API calls that reach the origin whoever runs the page.

page-view by a crawler that executes scripts	origin requests
without prerender	1 + k
cache-served, snapshot without scripts	0 — k saved
cache-served, snapshot with scripts	k incurred
proxied origin page	1 + k incurred
our own render	1 + k (the renderer runs the page too)

Browser v1.22.0 — measures it

The renderer's response hook already fires per response and already reads cache headers for its own resource cache. A pure classifier (subrequests.ts) judges every same-origin response by RFC 9111 shared-cache rules: uncacheable (non-GET, an uncacheable status, Set-Cookie, no-store/private/no-cache, zero max-age/s-maxage, Vary: *, expired Expires) is k; cacheable is explicit positive freshness; unspecified had no freshness headers at all and depends on the CDN's default TTL — reported, counted on neither side. blocked counts same-origin requests the fleet's block list aborted (a crawler would make them; class unknown), with static media left out since a CDN caches those regardless. Posted on every result as subrequests: { sameOrigin, cacheable, uncacheable, unspecified, blocked } plus scriptsStripped (the fleet's postProcess.stripScripts), which is what turns k from a cost into a saving at serve time. Header checks only; nothing awaited; ~5 ints on the wire.

Plugin prerender-v0.65.0 — stores and applies it

• PrerenderedPage gains uncacheableSubrequests and scriptsStripped (nullable — a pre-1.22.0 render stores null, never 0; the serve path reads null as unknown).
• render gains series subrequests (one VALUE per class per result): what the fleet's own renders cost the origin beyond the documents.
• New metric hydration_calls (side × botName × source, value = k), emitted on the serve path for crawlers the registry flags rendersJs: saved (script-stripped snapshot cache-served), incurred (snapshot kept scripts, or any origin serve), unknown (no k — a miss, or a page not yet re-rendered; value 0, count the rows). A row that doesn't say whether scripts were stripped reads as kept — the side that cannot over-credit.
• analytics.bots[].rendersJs flags the documented renderers only: Googlebot, Google InspectionTool, Bingbot, Applebot, YandexBot. Every AI crawler stays unflagged. It's a claim about the crawler, not an observation, and the schema text says so.
• Cost: two nullable fields per page write; one in-memory counter bump per rendering-crawler serve; ~30 combo rows/period/node.

Console prerender-console-v0.13.0 — counts it

originLoad() becomes

baseline = crawler requests + Σsaved + Σincurred
actual   = proxied + renders + Σk_renders + probes + sitemaps + Σincurred

The Traffic tile reads "script calls counted"; the panel shows Script calls: N saved · M incurred with the documents-only figure beside the full one for comparison, two new bars (render script calls, crawler script calls), and prints the unknown count as the size of the blind spot — while it outnumbers the known serves the panel warns that the figure is still mostly documents-only (it decays over one render cycle after the fleet upgrade). unspecified and blocked are printed as the bounds on k. Against an older plugin every panel keeps v0.12.0's reading.

Verification

• browser: npm test → 172 pass (tsc build + suite); classifier rules pinned in test/subrequests.test.ts, wire shape in test/jobResult.test.ts.
• plugin: node --test → 1009 pass; hydration_calls sides pinned per source × stripped × k-known in test/botServe.test.js; tally storage/emission and the null-not-zero rule in test/renderQueueRedirect.test.js; test/metrics.test.js enforces catalog + emitter + METRICS.md.
• console: node --test → 262 pass; measured-mode arithmetic, the mostly-unknown warning, and the older-plugin fallback in test/trafficView.test.js.
• npm run lint && npm run format:check clean. No PR CI in this repo.

Rollout order (after merge + three releases)

1. render-service (kohls branch) → browser 1.22.0, so k starts landing on pages as they re-render.
2. kohls-pr components/prerender → 0.65.0 (hydration_calls reads unknown until pages carry k).
3. kohls-pr components/console → 0.13.0.

Any order is safe; this one shortens the unknown window.

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request introduces tracking and counting of same-origin subrequests (XHR/fetch calls) made by pages when executed by JavaScript-running crawlers. It updates the browser renderer to classify and tally these requests based on RFC 9111 shared-cache rules, stores the results in the page cache, and integrates these metrics into the plugin and management console to provide a more accurate net offload calculation. Feedback on the changes suggests improving the robustness of Cache-Control directive parsing to allow optional whitespace around the equals sign per RFC 9111, and refactoring configuration caching in userAgent.js to pass cached values as explicit parameters for better testability.

[gemini-code-assist]

When caching configuration values or computed sets at the module level, pass the cached value as an explicit parameter to functions rather than closing over the module-level variable to keep functions pure and testable. Use nullish coalescing assignment (??=) to handle cases where the module is imported before the configuration is first applied, and add tests to verify that live configuration changes correctly update the cache.

rendersJsSet ??= new Set(config.analytics.bots || []);

References

1. When caching configuration values or computed sets at the module level (e.g., using config application hooks), pass the cached value as an explicit parameter to functions rather than closing over the module-level variable to keep functions pure and testable. Use nullish coalescing assignment (??=) to handle cases where the module is imported before the configuration is first applied, and add tests to verify that live configuration changes correctly update the cache.

[harper-joseph]

Declining the ??= shape, taking the test. botRendersJs deliberately mirrors botMayDiscover and botCountsAsDemand in the same file: an identity-keyed memo on config.analytics.bots, because applyOptions replaces that array on every change. A ??= would build the set once and then ignore a registry edit made from the console for the life of the process — the failure the identity check exists to prevent. (The suggested new Set(config.analytics.bots) would also be a set of entry objects, not of names.) The valid half of the finding — prove it follows a live change — is now pinned in test/userAgent.test.js in both directions, plus the junk-flag and derived-name cases (325f72e).

[gemini-code-assist]

RFC 9111 allows optional whitespace around the = character in Cache-Control directives (e.g., max-age = 60 or max-age= 60). The current regex (?:^|[,\\s])${name}=(\\d+) does not allow spaces around =, which can cause valid cache-control headers to be incorrectly classified as 'unspecified'. Updating the regex to (?:^|[,\\s])${name}\\s*=\\s*(\\d+) makes the parsing more robust and compliant with HTTP specifications.

Suggested change

	const directiveSeconds = (cc: string, name: string): number | null => {
	const m = cc.match(new RegExp(`(?:^|[,\\s])${name}=(\\d+)`));
	return m ? parseInt(m[1], 10) : null;
	};
	const directiveSeconds = (cc: string, name: string): number | null => {
	const m = cc.match(new RegExp('(?:^|[,\\s])' + name + '\\s*=\\s*(\\d+)'));
	return m ? parseInt(m[1], 10) : null;
	};

[harper-joseph]

Taken in 325f72e — whitespace around = and a quoted delta-seconds both parse now, with a test. Strictness here would have pushed a response a CDN happily caches into unspecified.

[harper-joseph]

Addressed both inline findings in 325f72e: lenient Cache-Control delta-seconds parsing (taken), and the userAgent memo (kept the identity-keyed pattern its siblings use — a ??= would ignore live registry edits — with the requested live-change tests added). Suites: browser 173, plugin 1014, console 262, all green; lint/format clean.