Change probe is blind to a value that changes and changes back: compare the origin against the rendered page, not against the previous probe

[harper-joseph]

The gap

The change probe compares the origin's answer to its own previous answer. That bounds drift — a value changing after a render — but it is structurally blind to a value that changes and changes back between two probes. If a render lands inside that window, the cached page keeps the transient state and the probe never notices, because the signature it reads still equals the one it stored.

Measured on a production commerce deployment (n=143 sampled cached product pages, 112 of them inside the serve window):

• price mismatches among served pages: 0/112 (0.0%) — the probe is doing its job on drift.
• availability mismatches among served pages: 3/112 (2.7%), every one of them the same shape: the cached page says OutOfStock, the origin says available, and the price matches exactly.

One case traced end to end: the probe stored [39.99,35.99,35.99,true] (available) at 11:04:37Z; the page rendered at 11:10:33Z — six minutes later — capturing OutOfStock; the origin now returns exactly the stored signature again. The probe's next pass will compare equal and trigger nothing, so the page serves a wrong availability until its render interval expires (48h in that deployment).

Four alternative explanations were tested and eliminated:

hypothesis	test	result
endpoint is flaky	5 consecutive calls	byte-identical signatures
answer varies per store	6 store identifiers incl. a nonsense one	identical for every probed field
render captured a different variant	compared SKUs	cached page carries the same SKU as the origin's default
systematic render defect	control pages	controls render the correct availability

At ~180k product-page serves/hour on that deployment, 2.7% is ~5k serves/hour carrying a stale availability. A shorter sweepInterval shrinks the window a round trip must fit inside but cannot close the class, and the exposure scales with render interval — so it gets worse exactly when you stretch intervals on the strength of the probe.

Proposal: compare the origin against what the PAGE says, not against the last probe

Make the stored signature mean "what the cached page currently claims" rather than "what the origin said last time". Then the existing comparison — origin-now vs stored — becomes origin-now vs page, which is the question that actually matters for structured-data correctness, and any disagreement triggers regardless of history.

Mechanism, reusing what already exists:

1. On render completion (RenderQueue.processJobResult), when a probe rule matches the URL and the new option is enabled, extract the rendered page's own structured-data values with the document-mode extractor already in changeProbeSpec (extractJsonLdOffers), normalize, and write that as the URL's ProbeState signature.
2. The probe pass is then unchanged. It fetches the origin, compares to the stored signature, and triggers on difference — which now means "the page disagrees with reality".

This needs no schema change and no new storage: ProbeState already holds exactly one signature per URL, and the render result is processed on the owner node (claims are owner-scoped), which is the same node the owner-scoped sweep uses.

The one genuinely new piece: field mapping

The request-mode signature is an ordered extract array (e.g. [regularPrice, salePrice, lowestApplicablePrice, isAvailableForShip]); the page's structured data yields (price, availability) per offer. They are not directly comparable, and the correspondence is site-specific — so the rule must declare it:

pageCheck:
  enabled: true
  priceFrom: 2        # index into `extract` holding the price the page renders
  availableFrom: 3    # index into `extract` holding the availability verdict

Comparison: mismatch when the origin's price is absent from the page's offer prices, or when the origin's availability disagrees with whether any offer on the page is in stock (a page whose every SKU is out of stock while the origin says available is a mismatch — that is one of the three measured cases).

Cost

One structured-data extraction per render on pages matching a pageCheck rule — a bounded regex scan plus a JSON parse, order milliseconds against a render that already stores a ~1MB blob. Nothing added to the serve path, and nothing added to the probe path.

Design questions worth settling before implementation

1. Device types. A URL renders per device and each render could capture different content, but ProbeState is one row per URL. Last-writer-wins, or only trust one device, or store per device?
2. Failure semantics. If the page yields no structured data (extraction returns null), it must leave the stored signature untouched rather than write a null — same rule the probe already applies to failed probes, so a markup change cannot mass-trigger.
3. Does this replace or supplement drift detection? Making the signature mean "what the page says" is strictly better for correctness, but it loses the ability to report a pure change rate from probe-to-probe deltas. If that metric matters, it needs its own counter.
4. Interaction with seeding. A first observation currently seeds without triggering. Under this design a freshly rendered page always has a signature, so seeding should become rarer — worth confirming it does not mask a real disagreement on the first pass after a render.

Happy to implement once the shape is agreed — particularly the mapping surface in (the pageCheck block) and the device-type answer in (1).

[harper-joseph]

Measured answers to the design questions, plus one fork the measurement exposed that changes the cost of the proposal.

(1) Device types — ANSWERED: structured data is device-independent

Compared the desktop and mobile cached pages of the same URL for 40 URLs that had both, on full (sku, price, availability) offer sets:

• 39/40 byte-identical.
• The single exception is explained by time, not device: the two renders were 41 hours apart (desktop 2026-08-23T18:12, mobile 2026-08-25T11:42) and the price moved between them. The stale side had already expired (2026-08-24T18:12) and was therefore not servable.

So last-writer-wins is correct and safe: both devices produce the same claim given the same moment, and a device page that lags far enough to hold a different claim is by definition past its expiry and not being served. One row per URL in ProbeState is the right shape; no per-device storage needed.

(2) Failure semantics — leave the signature untouched

If extraction yields nothing, write nothing — mirroring the existing rule that a failed probe changes no state. That keeps a markup change from mass-triggering, which is the same reasoning already documented for all-null extraction.

(4) Seeding — no masking

A freshly rendered page always writes a signature, so the pass after a render compares instead of seeding. That is the desired behaviour and cannot hide a disagreement.

(3) Replace vs supplement — this is the real fork, and it costs a column

I understated this in the issue. The two signatures are not the same shape, and that matters more than I credited:

• the drift signature is the full ordered extract array (e.g. [regularPrice, salePrice, lowestApplicablePrice, isAvailableForShip]);
• the page yields (price, availability) only.

So a page-derived signature can only ever be compared on the projected (price, available) pair. Making the stored signature mean "what the page says" therefore silently narrows drift detection: changes to regularPrice / salePrice — the strikethrough pair a shopper sees — would stop triggering, because they are not represented in the page-shaped signature. That is a real regression in exchange for closing round-trip blindness, and it is not a trade worth making silently.

Two honest options:

A. Supplement (recommended). Keep signature as the drift baseline and add a nullable pageSignature column to ProbeState, written at render completion. The pass then makes two comparisons: api-now vs signature (drift, unchanged) and project(api-now) vs pageSignature (page disagreement, new). Costs one nullable column on a node-local, non-replicated table — the cheapest schema change available here — and loses nothing.

B. Replace. No schema change, but drift detection narrows to (price, available) as described. Only acceptable if the deployment genuinely does not care about regular/sale price changes, which this one does — an earlier audit found 19 price mismatches driven by exactly those markdowns.

I would build A. It needs a nullable-column addition to ProbeState, which is why I am confirming before writing it rather than after.

[harper-joseph]

Implemented as option A in #139 (v0.58.0, default off).

All four design questions are answered in the PR — the load-bearing one empirically: desktop vs mobile structured data is 39/40 byte-identical across 40 URLs, and the single exception was two renders 41 hours apart with the stale side already expired, so one row per URL is correct and last-writer-wins is safe.

859/859 tests, 9 new, including the exact traced production case (origin matches its own baseline, page disagrees, trigger fires, claim cleared so it does not re-detect forever).