Single reused-connection retry loses against correlated silent connection death

[fredrikekre]

Ran into strange hangs after ~9 hours on the package storage servers after upgrading to
HTTP.jl version 2. Have been debugging this for a week or so and this is part I of what we
found. Issue description below from Mr. Robot.

---

Summary

A client that sends periodic bursts of concurrent requests to Amazon S3 starts
seeing HTTP.ParseError("unexpected EOF while reading HTTP/1 data") surface
from HTTP.request after ~10 minutes of runtime, deterministically, and then
steadily (1–2 failures per burst, forever). The proximate cause is server-side
(S3 silently discards connections under this traffic shape — no FIN, no RST),
but HTTP.jl turns an undetectable-by-design situation into user-visible errors
because the transport retries a reused-connection failure exactly once:

• _roundtrip_incoming! gates the retry on attempt == 1
  (src/http_transport.jl, the catch block), and
• pooled connections die in correlated batches (dialed together in the
  same burst → discarded together), so the one retry draws the next pooled
  connection, which is dead for the same reason, and the error propagates.

HTTP.jl 1.x under the identical workload did not surface these errors. Found while operating
the Julia package storage servers after upgrading to HTTP.jl 2.x.

Versions

HTTP.jl 2.6.5, Julia 1.12.7, Linux x86_64 (EC2 us-east-1, no proxy/NAT in
path). Reproduced single-threaded and with -t4.

Reproduction (deterministic; needs only internet access)

Bursts of concurrent HEADs/GETs (some to missing keys) against public S3
objects, every 10 s. Onset was at cycle 52 (~9.5 min) in 2/2 independent runs,
then 1–2 errors per cycle indefinitely:

using HTTP

const B1 = "https://julialang-storage-us-east-1.s3.amazonaws.com"
const B2 = "https://julialang-storage-ap-northeast-2.s3.amazonaws.com"
const GOOD = "$B1/registries"
const GOOD2 = "$B2/registries"

using Random
head(url) = HTTP.request("HEAD", url; status_exception=false).status

for cycle in 1:120
    misses = ["$b/registry/23338594-aafe-5451-b93e-139f81909106/$(randstring(40)).tar.gz"
              for b in (B1, B2) for _ in 1:4]
    reqs = vcat([GOOD, GOOD2], misses)
    tasks = [Threads.@spawn try; head(u); catch e; sprint(showerror, e); end for u in reqs]
    gets = [Threads.@spawn try
                io = IOBuffer()
                HTTP.request("GET", GOOD; response_stream=io, status_exception=false).status
            catch e
                sprint(showerror, e)
            end for _ in 1:2]
    errs = [r for r in fetch.(vcat(tasks, gets)) if r isa String]
    isempty(errs) || println("cycle $cycle: $(length(errs)) ERRORS: ", first(errs))
    sleep(10)
end

Negative controls (all clean):

• the same mix at a 1 s period: 840+ cycles, zero errors — the ~10 s idle gaps
  between bursts are required
• sequential requests: S3's graceful closes (Connection: close after 100
  requests on a connection, and at ~235 s connection age) are handled fine,
  sequentially AND concurrently

Wire evidence

tcpdump (SYN/FIN/RST) on the erroring process: the failing connections show a
client FIN answered by an S3 RST — S3 had no state for the flow anymore.
Under this traffic shape S3 eventually discards connections silently
(no FIN/RST on the wire), which is invisible to any client. The pooled
connection still looks ESTAB; the next request written to it is answered with
RST; the response read gets EOF. Because the whole burst's connections were
dialed together, they are discarded together — so the single retry almost
always lands on a second dead connection. (pcap available on request.)

Suggested direction

The silent drop itself is not detectable client-side; the robust fix is in the
retry policy, not the pool:

1. Retry reused-connection failures until a fresh dial has been attempted
   (i.e. don't burn the only retry on another pooled connection), or
2. loop the attempt == 1 retry while was_reused holds (bounded by pool
   size), or
3. classify ParseError EOF-on-reused-connection as retryable in the outer
   retry layer (today it escapes even with default retry=true).

[quinnj]

Ah, I found what's going on here; subtle bug in our new retry bucket mechanism; when an outer retry is triggered, it pulls from teh bucket, but doesn't return on success. So you essentially burn down your retry credits until the bucket's empty. Fix coming up.

[quinnj]

Thanks for the detailed report @fredrikekre!