feat: add a progress-aware write deadline for initialization deliveries - #786
Open
kinyoklion wants to merge 8 commits into
Open
feat: add a progress-aware write deadline for initialization deliveries#786kinyoklion wants to merge 8 commits into
kinyoklion wants to merge 8 commits into
Conversation
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 1b95108. Configure here.
Base automatically changed from
rlamb/relay-init-concurrency-limiter
to
feat/concurrency-init-limits
August 5, 2026 16:34
Add internal/initwrite: a ResponseWriter wrapper that arms a write deadline sized to the payload it is about to send. The deadline combines a throughput floor with an absolute cap, so a client that stalls or reads slower than the floor is cut promptly while a healthy client sending a large payload is not. It writes strings without copying the payload per connection and exposes the underlying connection via Unwrap so http.NewResponseController can reach it. To let that (and any other) ResponseController operation reach the real connection, implement Unwrap on the logging and metrics ResponseWriter wrappers, which otherwise hide it. No runtime effect on its own: nothing wraps a connection with initwrite yet.
Make the write-deadline writer robust on every delivery-end path, not just the normal end-of-batch flush: - Done is now never nil. Outside a delivery it is an already-closed channel, so a producer that waits on it to release its budget slot is released at once instead of pinning the slot for the life of the connection. This also removes the need for a caller to capture the channel before closing its batch. - Add Finish: an idempotent backstop that clears the deadline and closes Done on any exit path. Without it, a delivery that ended without a final flush (an error or a cancelled context) left an armed deadline behind, which on HTTP/2 is a self-firing timer that resets an otherwise idle stream. - Begin closes any still-open Done before starting a new delivery, so a second Begin cannot orphan a waiter. - arm records a deadline as current only when SetWriteDeadline succeeds, so a transient failure does not leave the writer believing it armed a deadline it never set and then stop retrying. Rewrite the tests around an injected clock so the per-chunk deadline math is asserted exactly (floor, slack, cap, per-chunk re-arming) and the full gated lifecycle is covered (no deadline before Begin, armed during, cleared once at end, Done fail-safe, Finish backstop, string/byte parity). Coverage of the package rises from ~39% to ~94%. Also make the package doc self-contained: it no longer references a config file or default value that lives in the wiring change rather than here.
The previous Finish cleared the write deadline on any exit path. Called mid-send -- which is where a cancelled delivery lands, since the completion signal never fires on cancel -- clearing stripped the only bound on the remaining bytes, so a stalled client could hold the connection and its budget slot indefinitely. Clearing is correct only for a delivery that actually finished. Replace it with two operations that match the two ways a gated delivery ends: - A clean finish still clears the deadline at the end-of-delivery flush, so the now-idle persistent stream is left alone. - Abort ends a delivery abnormally by moving the deadline to now, forcing any in-flight or subsequent write to fail at once rather than running unbounded. It is safe on any exit path, including mid-send. - WaitAndFinish holds until the delivery is flushed (slot held across the real send) or the context is cancelled, aborting in the latter case. This is the one correct ordering, so the producer no longer has to assemble it. Also make teardown race-safe: a generation counter bumped in Begin lets a flush tear down only the delivery it observed, never a newer one that started meanwhile, and the deadline clear now runs under the lock like arm does. Remove Flush's pre-flush re-arm, which could stamp the delivery's start before its first payload byte and shrink a healthy client's budget. Guard the chunk loop against a (0, nil) writer. Rework the tests around the injected clock to assert the exact per-chunk budget with literal values, arm-before-write ordering, the error path, the gated string path (the production SSE path), post-delivery inertness across heartbeats, Abort vs clean-finish semantics, WaitAndFinish on both branches, and that a stale flush cannot clear a newer delivery's deadline.
A gated stream's producer goroutine can outlive the HTTP handler: the SSE server keeps draining it after the client leaves. A write deadline, though, is a capability scoped to the handler's lifetime -- once the handler returns the connection may be recycled or, on HTTP/2, gone -- so setting it from the producer after the handler returned dereferences freed state inside net/http (a process crash on HTTP/2) or poisons the next request on a recycled keep-alive connection. The previous Abort force-fired the deadline, and WaitAndFinish called it on context cancellation -- which is exactly when the handler has returned. Remove that: the producer now never touches the connection. Only the handler goroutine sets or clears the deadline, via the per-write arm and the end-of-delivery flush, both within its own lifetime. On cancellation WaitAndFinish just releases the waiter so the slot frees; the per-write deadline already armed still bounds anything in flight, and connection teardown clears it. Also clear the deadline before closing Done on the clean path, so a producer released by Done is guaranteed to observe the cleared deadline. Tests: prove the cancel path performs no SetWriteDeadline (counter and a panicking connection), and add an end-to-end HTTP/2 test where the producer outlives the handler and must not crash. Restore the multi-chunk string parity test and mirror the byte path's arm-order and error-path coverage onto the string path; pin the same-value re-arm ratchet; guard against a double flush; give the delivered-wait test a timeout so a regression fails instead of hangs.
Removing the force-fire left the producer-error-with-a-healthy-client case without a terminator, and the package doc narrowed the contract to "two ways" a delivery ends. If the producer errors after arming a deadline and skips End, the end-of-batch flush never tears down: the deadline is never cleared and the handler eventually cuts even a healthy stream at maxHold, while the slot stays pinned. State the real contract: a gated delivery ends in three ways, and End must be called on every exit after Begin -- a clean finish and an abandoned delivery alike -- before the batch channel is closed, so the handler's flush clears the deadline on the goroutine that owns the connection. Releasing the slot alone is not enough, because only the handler can clear the deadline. Also: - Flush now performs the teardown even when the underlying writer cannot flush, so a non-flushing chain cannot strand the deadline or pin the slot. - Note on WaitAndFinish that a context cancelled while the handler is still live returns early without ending the delivery; use End for that. Tests: pin that a flush without End does not tear down and that End+flush tears down even a partial delivery; that a cancel after a clean delivery does not double-close; the (0, nil) short-write guard; teardown on a non-Flusher chain; and make the HTTP/2 post-handler-return test deterministic so a reintroduced connection touch crashes reliably instead of flaking green.
…ve request A poll (Wrap) delivery arms a per-write deadline and relies on net/http to reset the connection's write deadline when the handler returns, since this server sets no WriteTimeout. Automated review flagged this as unsafe, on the theory that net/http only resets the deadline when WriteTimeout is set. That is not so on the Go versions this builds with: the reset is unconditional. Add a regression test that arms a short deadline in one request and reuses the keep-alive connection for a second one after it would have expired -- the second request succeeds -- so a future change to that behavior would fail CI rather than silently break a later request.
Round of test and doc hardening; one structural change: Flush's teardown now runs from a defer registered before the flush, so a flush that is unsupported, fails, or panics can no longer strand the deadline or pin the slot, and the (active, ending) sample is explicitly taken before flushing so a delivery that begins mid-flush belongs to the next flush. Tests, each verified to fail against the defect it pins: - The delivered-then-cancelled double-close check now loops (a single run took the panicking branch only ~half the time), and two waiters parked on one abandoned delivery reach the close deterministically. - The flush-side double-close guard: client leaves between End and the end-of-batch flush, which then must not re-close done (a panic here would land on the handler goroutine) yet must still clear the deadline. - Sample-before-flush ordering: a delivery begun during the flush must not be torn down pre-basis. - Re-Begin resets: ending (else the first flush after re-Begin kills the new delivery), the cap anchor, and the deadline ratchet. - The absolute cap is anchored to the delivery start, not per-write (a per-write anchor is no backstop at all). - minExtension suppression values, arm-after-end inertness, the string-path (0, nil) guard, and timeouts so a removed guard fails instead of hanging. - A concurrent lifecycle stress run so -race pins the lock discipline. Docs, each correcting a measured behavior: a missed End kills an HTTP/2 stream within seconds of its last write (not at the cap), disabling the cap makes that kill sooner rather than never, an error exit that wrote nothing still needs End, and defers must register the batch-channel close before End so End runs first. Softened "exactly once" to idempotent; the flush dispatches to the first Flusher in the chain.
The keep-alive regression test asserted only that a second request succeeded; if the premise it pins ever broke, the transport would silently retry the idempotent GET on a fresh connection and the test would stay green. It now also asserts the second request REUSED the connection. Correct the cap-disabling doc for HTTP/1.1, where no deadline self-fires: a missed End with the cap disabled leaves the budget slot pinned permanently rather than killing the stream, and the HTTP/1.1 kill (with the cap on) lands at the first write after the cap expires. Note that a heartbeat interval shorter than the slack postpones the HTTP/2 self-fire, which can hide the bug under fast test heartbeats. Pin the properties of the deferred-teardown change itself, each test verified to fail with the defect reintroduced: - a panicking flush still clears the deadline and releases the waiter during unwinding, and the panic propagates; - the clearing SetWriteDeadline runs strictly after the final flush, which must still execute under the armed deadline; - a stale waiter cannot release a newer delivery's slot (WaitAndFinish's channel capture is split out so the test can drive the cancellation branch with a stale capture); - Flush's state sample happens under the lock, pinned by a race test that actually splits the producer and handler across goroutines. Also release the waiter from a defer inside the teardown, so even a panicking deadline clear cannot strand it.
kinyoklion
force-pushed
the
rlamb/relay-init-concurrency-initwrite
branch
from
August 5, 2026 16:37
77697c7 to
dc68f44
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the write-deadline primitive that the init-concurrency wiring will use to reclaim a slot from a client that cannot keep up, without disturbing a healthy client mid-delivery.
What changes
internal/initwritepackage. AResponseWriterwrapper that, for each initialization delivery, arms a write deadline sized to the payload: a throughput floor (so a stalled or too-slow client is cut promptly) combined with an absolute cap (so a client stuck right at the floor on a very large payload is still bounded). It writes strings directly to avoid copying the payload once per connection, and implementsUnwrapsohttp.NewResponseControllercan reach the underlying connection to set the deadline.Unwrapon the logging and metricsResponseWriterwrappers. These wrap the connection for their own purposes and otherwise hide it, which would stop aResponseControllerfrom reaching the real connection. Each now returns its inner writer.No behavior change
Nothing wraps a connection with
initwriteyet, so this is inert at runtime. It builds, vets, and passes-raceon./internal/initwrite/,./internal/logging/, and./internal/middleware/.Second of the PRs splitting #782. Builds on #785 (the config + limiter refinement); the wiring PR will follow and depend on both.
Note
Low Risk
New isolated package with extensive tests; middleware changes are small Unwrap additions with no runtime wiring yet.
Overview
Introduces
internal/initwrite, aResponseWriterwrapper for initialization poll and SSE replay traffic. It arms per-chunk write deadlines from a 64 KiB/s throughput floor plus slack, optionally clamped bymaxHold, and supportsWrap(always armed) vsWrapGated(Begin/End/WaitAndFinishso SSE heartbeats stay deadline-free after the basis flush). The handler ownsSetWriteDeadline; producers coordinate only via channels.loggingHTTPResponseWriterandstatusRecordernow implementUnwrap()sohttp.NewResponseControllercan set deadlines through the middleware stack when this is wired in.Nothing uses
initwritein handlers yet, so relay behavior is unchanged until a follow-up PR.Reviewed by Cursor Bugbot for commit dc68f44. Bugbot is set up for automated code reviews on this repo. Configure here.