fix: stop server-side stream replay producer when the SDK client disconnects - #774
Merged
Conversation
…onnects The server-side /sdk/stream (FDv2) and /all (FDv1) replay handers use the eventsource server's Repository mechanism: serverSideEnvStreamRepository.Replay returns a channel and a goroutine sends the replay events on it. That channel is unbuffered, so under backpressure (a slow or stalled SDK client) the producer goroutine parks on `out <- event` while the connection handler is busy writing to the socket. If the client then disconnects, the eventsource handler stops reading the channel and the producer is stranded on that send until the process exits -- a goroutine (and its payload) leaked per disconnected-mid-replay client. This adopts the new eventsource RepositoryWithContext extension: serverSideEnvStreamRepository now implements ReplayWithContext, and the send loop selects on the subscriber's request context so it returns immediately when the client disconnects. Replay is retained (delegating to the shared logic with a background context) to satisfy the Repository interface; the eventsource server prefers ReplayWithContext when present. Depends on the eventsource change that adds RepositoryWithContext and the handler-side batch drain (launchdarkly/eventsource#63). go.mod is bumped to a pseudo-version of that PR's branch commit; it must be re-pointed to the tagged eventsource release once that PR merges. Adds a unit test asserting the producer stops (channel closes) promptly when the context is cancelled without a reader. Verified end to end against mockld with a non-reading client: with the fix the replay producer goroutine exits within ~0.6s of a client FIN, whereas the stock build (eventsource v1.11.0) leaves it blocked on the channel send indefinitely (still present 3s after disconnect).
Add a compile-time assertion that serverSideEnvStreamRepository implements eventsource.RepositoryWithContext (so a future refactor can't silently drop back to the context-less Replay path), and skip building the replay payload entirely when the subscriber's context is already cancelled before the producer starts. Bump eventsource to the updated PR-branch pseudo-version that also drains the undelivered-batch path.
kinyoklion
commented
Jul 29, 2026
Replaces the pseudo-version pin of the RepositoryWithContext PR branch; v1.11.2 contains that change along with the once-per-batch replay flush.
Matches the select in the send loop below; ctx.Err() and a closed Done channel signal the same condition, so this is equivalent.
kinyoklion
marked this pull request as ready for review
July 29, 2026 16:46
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 3a9ff12. Configure here.
aaron-zeisler
approved these changes
Jul 29, 2026
aaron-zeisler
left a comment
Contributor
There was a problem hiding this comment.
👍 The code in the production file looks good.
My Claude also reported the test that Bugbot pointed out: that specific test passes even if the code changes are reverted. Below is the recommended test from my agent:
t.Run("ReplayWithContext stops producing when the subscriber's context is cancelled", func(t *testing.T) {
snapshotReturned := make(chan struct{}, 1)
underlyingQuery := queryThatIncrementsFlagVersionOnEachCall()
store := newMockStoreQueries()
store.setupSnapshotFn(func() (map[ldstoretypes.DataKind][]ldstoretypes.KeyedItemDescriptor, subsystems.Selector, error) {
data, selector, err := underlyingQuery()
snapshotReturned <- struct{}{}
return data, selector, err
})
repo := &serverSideEnvStreamRepository{store: store, logger: slog.Default()}
ctx, cancel := context.WithCancel(context.Background())
eventCh := repo.ReplayWithContext(ctx, "", "")
// Producer has built its payload and is parked on the unbuffered, unread send.
<-snapshotReturned
time.Sleep(50 * time.Millisecond)
cancel()
// Let the producer observe cancellation and return *before* any reader engages, so the
// send-loop select has only ctx.Done() ready (deterministic — no out<-event / ctx race).
time.Sleep(50 * time.Millisecond)
// The fix delivers ZERO events and closes. Reverting the send-loop ctx.Done() case makes the
// producer deliver its event first, which AssertChannelClosed rejects.
require.True(t, helpers.AssertChannelClosed(t, eventCh, time.Second),
"replay channel should close with no event delivered after cancellation")
})
Contributor
|
It looks like the CI steps are failing because:
|
…an event The previous assertion loop received from the channel, which itself would rescue a producer parked on the unbuffered send: with the single-event v1 replay, a producer that ignored cancellation would complete its send to the loop and close the channel, passing the test. Now the producer is given time to observe cancellation while no receiver exists -- its select then has only the ctx.Done case ready -- and the first receive must report closed with no event delivered. Verified red against a producer with the context handling stripped, green on the real code.
Resolves GHSA-259r-337f-4rfw flagged by Docker Scout; the advisory affects >=1.16.0 <1.18.7.
keelerm84
approved these changes
Jul 29, 2026
Distinct messages for the two exit points: disconnect observed before the payload is built, and disconnect observed mid-delivery.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Summary
The server-side stream replay path (
serverSideEnvStreamRepository.Replay, backing/sdk/streamfor FDv2 and/allfor FDv1) leaks a goroutine per SDK client that disconnects mid-replay.Replayreturns an unbuffered channel and spawns a producer goroutine that sends the replay events on it. Under backpressure — a slow or stalled SDK client — the producer parks onout <- eventwhile theeventsourceconnection handler is busy writing to the socket. If the client then disconnects, the handler stops reading the channel, and becauseReplayhas no cancellation hook, the producer is stranded on that send until the process exits. The goroutine and the replay payload it holds leak.Change
Adopt the new optional
eventsource.RepositoryWithContextextension:serverSideEnvStreamRepositorynow implementsReplayWithContext(ctx, channel, id). Theeventsourceserver calls it with the subscribing request's context, which is cancelled on disconnect. The send loopselects onctx.Done(), so the producer returns immediately instead of blocking on a send nobody will receive.Replayis retained (it delegates to the same logic with a background context) to satisfy theeventsource.Repositoryinterface; the server prefersReplayWithContextwhen a repository implements it.The shared
replayhelper keeps the existingIsInitializedshort-circuit and singleflight behavior unchanged.Dependency
This depends on launchdarkly/eventsource#63, which adds
RepositoryWithContext(plus a handler-side background drain that unblocks anyRepositoryproducer, even those that don't adopt the context). That change is now released:go.modpoints at the tagged eventsource v1.11.2, which contains #63 along with the once-per-batch replay flush (launchdarkly/eventsource#64) and the CI fixes (launchdarkly/eventsource#66). The earlier pseudo-version pin of the PR branch is gone.Testing
ReplayWithContextstops producing (channel closes) promptly when the subscriber's context is cancelled without a reader — before context propagation this producer would block forever.go test -race ./internal/streams/...and./relay/...pass, re-run against the released v1.11.2.mockld(a ~3MB, 6500-flag dataset) with a non-reading client that stalls the socket:Note
Medium Risk
Touches hot-path SSE replay for all server-side SDK connections; behavior change is limited to cleanup on disconnect, but large replay payloads and timing-sensitive send loops warrant careful review.
Overview
Fixes a goroutine leak on server-side SDK streams (
/sdk/stream,/all) when a client disconnects while replay is still sending on an unbuffered channel—the producer could block forever onout <- eventwith no reader.serverSideEnvStreamRepositorynow implementseventsource.RepositoryWithContext: sharedreplaylogic uses the subscribe request context (cancelled on disconnect), bails before building the snapshot if already cancelled, andselects onctx.Done()when sending events. LegacyReplaydelegates to the same helper withcontext.Background().Dependencies:
github.com/launchdarkly/eventsourcev1.11.0 → v1.11.2 (addsRepositoryWithContext);klauspost/compresspatch bump in lockfile.Adds a unit test that cancels context mid-replay with no consumer and asserts the channel closes without delivering events.
Reviewed by Cursor Bugbot for commit 9997de9. Bugbot is set up for automated code reviews on this repo. Configure here.