refactor(httpx): one bounded drain idiom for every response body - #560
Merged
Conversation
The repo carried three spellings of what happens to a response body after the
status is checked: a bare Close (25 sites), an unbounded io.Copy then Close (2),
and one bounded copy added last week. Only the last is right, and the unbounded
one is a hazard — a runaway upstream holds the caller's entire timeout open
inside what reads like cleanup.
httpx.Drain consumes up to 32 KB of whatever is unread; callers pair it with the
Close it deliberately does NOT perform:
defer func() { httpx.Drain(resp.Body); _ = resp.Body.Close() }()
That shape is uglier than a DrainClose(resp) helper, which is what was written
first. It is used anyway because hiding the Close inside a helper blinds the
bodyclose linter, which then reports "response body must be closed" at every
converted site. bodyclose catches a leak that is invisible in tests and expensive
in production; this drain is worth much less than that, so the drain gives way.
Worth recording how nearly that went unnoticed: golangci-lint's max-same-issues
defaults to 3, so 25 newly-broken sites surfaced as 3 — and a DIFFERENT 3 each
run, which reads exactly like the stale-cache phantom findings this repo has hit
before. The tell was that converting one file moved the reported set.
On the value of the drain itself, stated smaller than it first was. net/http has
often already buffered a small response by the time Close runs, so the connection
is pooled either way; past the cap the body never reaches EOF, so it is dropped
either way. In between is a band where draining rescues a connection. Where that
band falls moves with header size and the read-buffer boundary.
Which is why there are no connection-reuse TESTS here. Three were written: a
size-to-dials table (flipped between runs), and even "draining is never worse
than a bare Close" (failed at 64 KB on a rerun). Dial counts are not a
deterministic function of body size, so any assertion on them is a flaky test
wearing a proof — and a flaky test pinning a claim is worse than none, because it
gets muted and the claim is then both unguarded and believed. The bound and
nil-tolerance are deterministic and are asserted; the reuse rationale lives in
the doc comment, sized honestly. If it ever needs pinning, it wants a benchmark.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The repo carried three spellings of what happens to a response body once the status has been checked:
defer func() { _ = resp.Body.Close() }()defer func() { _, _ = io.Copy(io.Discard, resp.Body); _ = resp.Body.Close() }()httpx.Drainconsumes up to 32 KB of whatever is unread. Callers pair it with theCloseit deliberately does not perform:The unbounded pair is the part that actually mattered: a runaway upstream holds the caller's entire timeout open inside what reads like cleanup.
Linked issue
n/a — found reviewing #559.
Why the ugly shape, and not
DrainClose(resp)DrainClose(resp)reads better and was written first. It is not used, because hiding theCloseinside a helper blinds thebodycloselinter — it then reports "response body must be closed" at every converted site.bodyclosecatches a leak that is invisible in tests and expensive in production; this drain is worth far less than that, so the drain gives way.Worth recording how nearly that went unnoticed.
golangci-lint'smax-same-issuesdefaults to 3, so 25 newly-broken sites surfaced as 3 findings — a different 3 each run. That reads exactly like the stale-cache phantom findings this repo has hit before, and the instinct is to blame the cache. The tell was that converting a single file moved the reported set.On the value of the drain, stated smaller than it started
This began as a finding on #559 phrased as "every POST redoes DNS + TCP + TLS". Measured, that is not true.
net/httphas often already buffered a small response by the timeCloseruns, so the connection is pooled either way; past the cap the body never reaches EOF, so it is dropped either way. In between is a band where draining rescues a connection — and where that band falls moves with header size and the read-buffer boundary.The claim was corrected in #559 before it merged.
There are no connection-reuse tests, deliberately
Three were written, and all three were unreliable:
Close" — failed at 64 KB on a rerun;Dial counts are not a deterministic function of body size, so an assertion on them is a flaky test wearing a proof. A flaky test pinning a claim is worse than no test: it gets muted, and the claim ends up both unguarded and believed. So the rationale lives in
Drain's doc comment, sized honestly, and what is deterministic — the bound, and nil-tolerance — is what gets asserted. If the reuse effect is ever worth pinning, it wants a benchmark.The test file says this in place of the tests, so the next person does not read the gap as an oversight and "fix" it.
How was it verified?
bodyclosereporting 0 is the load-bearing gate result here: it is what says the shared helper did not cost the leak check.Checklist
go build ./...passesgo vet ./...passesgo test ./...passes (-raceacross./internal/...)gofmt -l .prints nothinggolangci-lint run ./...reports 0 issues!needed