Skip to content

fix: reconcile big segment data with polling endpoint when stream is quiet - #790

Draft
kinyoklion wants to merge 1 commit into
v9from
rlamb/sdk-2840/bigsegment-stream-reconcile
Draft

fix: reconcile big segment data with polling endpoint when stream is quiet#790
kinyoklion wants to merge 1 commit into
v9from
rlamb/sdk-2840/bigsegment-stream-reconcile

Conversation

@kinyoklion

@kinyoklion kinyoklion commented Jul 31, 2026

Copy link
Copy Markdown
Member

Ticket: SDK-2840

Root cause of the big_segments integration flake

TestEndToEnd/big_segments/{Redis,DynamoDB}/another_big_segment_is_created_after_synchronizer_has_started is the most frequent remaining CI flake (12+ occurrences May–July). Log forensics on two recent failures (run 30562656155, v9, 07-30; run 29779343829, v8, 07-20) show the same signature:

  • The failing environment's initial revisions poll caught only a partial include list (the remaining include landed in a later revision), or the second segment's patches were pending.
  • The /big-segments SSE stream connected successfully and then delivered nothing for the entire wait — 60 seconds in the v8 run, so test(integrationtests): increase SDK key expiry margin and big-segment sync timeout #711's timeout bump demonstrably does not fix this.
  • The synchronizer never polled again: after the initial poll → stream → poll sandwich, the 30s timer in consumeStream only refreshed synchronizedOn. A missed stream event is never redelivered, so the revision stayed unapplied indefinitely.

The likely trigger in CI is that the test environments are created seconds before the stream connects, and events published while the upstream subscription is still being registered are dropped. But the underlying exposure is general: any silently-dropped stream event leaves Relay's big segment data stale until the next unrelated event for that environment arrives (heartbeats keep the connection alive, so the 5-minute read timeout doesn't rescue it).

Changes

  • consumeStream's periodic timer (30s of stream quiet) now runs a reconciliation poll against the cursor-based revisions endpoint — applying any missed revisions and notifying update listeners — before refreshing synchronizedOn. This bounds the staleness from a missed stream event to ~30s. When nothing was missed, the poll is a single cheap request returning an empty array (the request carries the cursor).
  • Ports test(integrationtests): increase SDK key expiry margin and big-segment sync timeout #711's evaluation-wait bump (20s → 60s) to the v9 integration test — it was only ever applied to v8. 60s covers a full reconciliation cycle with margin.
  • The supervisor no longer logs the will retry warning when it is shutting down cleanly — that warning appears at the tail of every failed-test log dump and reads as a stream failure when it's actually teardown (it misdirected this investigation for a while).
  • New unit test TestSyncReconcilesWhenStreamIsQuiet: stream connects but stays silent, a missed revision exists only on the poll endpoint, and the reconciliation poll applies it without a stream reconnect.

Trade-off for reviewers

On a quiet stream this adds one revisions poll per environment per 30 seconds (previously zero once the stream was up). For large auto-config deployments that is a real increase in requests to the big-segments revisions endpoint, though each request is cursor-based and cheap when there are no changes. If that's a concern, alternatives are a longer reconcile interval or backing off after consecutive empty reconciles — happy to adjust.

Verification

  • go test -race -count=5 ./internal/bigsegments/ green; the new and neighboring sync tests pass 50 consecutive -race runs under full CPU load.
  • go vet -tags integrationtests ./integrationtests/ clean.

…quiet

The big segment synchronizer syncs via poll, then connects the SSE
stream and bridges the gap with a final poll. After that it relied
entirely on stream events: the periodic 30-second timer only refreshed
the store's synchronizedOn timestamp. A stream event that is missed
(for example, one published while the upstream subscription is still
being established) is never redelivered, so the corresponding revision
stayed unapplied until some later unrelated event arrived.

The periodic timer now reconciles against the cursor-based revisions
polling endpoint, applying any revisions that were not delivered as
stream events, so a missed event is picked up in bounded time.

Also raises the big segments integration test evaluation wait from 20s
to 60s to cover a full reconciliation cycle (the same change was made
on the v8 branch but never ported), and stops logging the 'will retry'
warning when the synchronizer is shutting down cleanly.

SDK-2840
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant