Fix retry-budget exhaustion and correlated reused-connection death

[quinnj]

Fixes #1353.

Root cause

The reported pattern — clean for ~50 burst cycles (~10 min), then 1–2 ParseError("unexpected EOF while reading HTTP/1 data") escapes per burst forever, despite default retry=true — is primarily a retry-budget accounting bug, with the issue's single-shot transport retry as the trigger:

1. Both pooled connections per host (max_idle_per_host = 2 default) die together between bursts (S3 discards them while parked). The unlucky request draws dead conn splitpath loses last character #1 and its single transport retry (attempt == 1 gate) draws dead conn Basic Documentation #2.
2. The outer client retry layer then rescues it with a fresh dial — but arming that retry reserved 10 of the per-host RetryBucket's 500 tokens, and the release call charged the full 10 even when the retried attempt succeeded with a 2xx. Nothing ever refilled the bucket.
3. 500 ÷ 10 = 50 recoveries and the budget is gone — the reporter's deterministic onset at cycle 52 is just the fuse burning down; S3's behavior never changed at the 10-minute mark.
4. Once empty, RetryDeniedError was swallowed silently and the raw ParseError rethrown, indistinguishable from "not retryable".

Reproduced deterministically against a local TLS server (park 2 conns → kill both → 1 request, every 10 s): the bucket drains 10/cycle through 50 successful recoveries, then the exact reported error escapes every cycle, first escape at cycle 51.

Changes

Retry-budget accounting. A retry reservation is settled by the effective retry decision for the response it produced: refunded in full when the built-in policy (or a custom retry_if) no longer wants a retry, partial cost (5) kept when the response is still classified as a failure. On the final attempt the built-in classification applies without invoking retry_if, and a retry_if-forced retry conservatively keeps cost on a non-2xx/3xx outcome. Each successful non-retried request restores one unit of previously consumed budget (AWS-style no-retry increment), tracked through a copy-on-write depleted-partition snapshot so healthy traffic never takes the bucket lock. Reservations and response connections are released even when a trace or retry_if callback throws, and the request deadline is rechecked after the backoff sleep.

Transport recovery through a poisoned pool. The reused-connection retry loops while failures keep landing on reused connections: up to max_idle_per_host reused acquisitions, then the final attempt forces a fresh dial — an idle pool hit or concurrent waiter handoff on that attempt is closed and its counted slot transferred to the dial, so max_conns_per_host stays exact. Slot release is now exactly-once via a slot_released flag (a connection closed early by a cancel callback no longer strands its slot), and a live connection drawn in the same pool pass that evicts stale peers is no longer dropped. PUT and DELETE get the same stale-connection recovery as the other idempotent methods (replayable-body gate unchanged). This makes correlated-death recovery free — no outer attempt, no bucket traffic, no backoff — and protects retry=false callers and direct roundtrip! users.

TLS error classification and typing. Dead reused TLS connections are classified by cause (e.g. an RST as a wrapped SystemError) and by Reseau's truncated-stream signature (a single documented constant; the coupling is pinned by end-to-end truncated-record tests). Handshake-phase failures are typed TLSHandshakeError at the dial sites (HTTP/1 and HTTP/2, including TLS.client setup); TLS failures on established connections surface as the new TLSTransportError across all public boundaries — request, HTTP.open, streaming body reads, roundtrip!, h2_roundtrip!, and the HTTP/2 read-loop/window-update paths. isrecoverable unwraps both public wrappers plus HTTP/2 ProtocolError connection wrappers. Non-Exception throws from user callbacks pass through unwrapped.

Observability. New public RetrySkippedEvent trace event fires when the policy wanted a retry but none was armed — reason = :retry_bucket (budget denied) or :deadline (deadline preempted the backoff). Previously a denied retry was indistinguishable from a non-retryable failure, which is what originally made this bug look like "retry=true doesn't retry".

Validation

Issue-shaped end-to-end harness (local TLS server, both pooled conns killed every cycle, 60 cycles):

	master	this PR
recovery per cycle	outer retry, −10 tokens	transport retry, outer_retries = 0
bucket after 60 cycles	0 (empty at cycle 50)	500
escapes	ParseError every cycle from 51 on	none

Under close_notify, bare-FIN, and RST kill modes, recovery now happens entirely at the transport layer with zero bucket consumption. The poisoned-pool scenario is a regression test (red on master with the exact reported error), alongside coverage for terminal accounting, retry_if accounting and suppression, trace/policy-callback failure cleanup (including pool-slot assertions), concurrent handoff under force_fresh, one-shot bodies, truncated TLS records (HTTP/1 + HTTP/2), stale PUT/DELETE recovery, and RetrySkippedEvent emission on both response and request paths.

Behavior notes

• Mid-request TLS failures now surface as TLSTransportError instead of TLSHandshakeError; callers matching HTTPError or using isrecoverable are unaffected (CHANGELOG entry included).
• retry_if is not invoked for the terminal response once retries are exhausted (matching the previous gate order) nor for unreplayable bodies.
• Operators on released 2.6.x can mitigate today with retry_bucket=false (or a large-capacity RetryBucket).

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 95.21127% with 17 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.84%. Comparing base (bc251f0) to head (0726d93).

Files with missing lines	Patch %	Lines
src/http_transport.jl	95.91%	6 Missing ⚠️
src/http_core.jl	82.60%	4 Missing ⚠️
src/http_client.jl	96.20%	3 Missing ⚠️
src/http2_client.jl	90.90%	2 Missing ⚠️
src/http_client_retry.jl	97.36%	1 Missing ⚠️
src/http_stream.jl	85.71%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1354      +/-   ##
==========================================
+ Coverage   88.56%   88.84%   +0.28%     
==========================================
  Files          31       31              
  Lines       11974    12132     +158     
==========================================
+ Hits        10605    10779     +174     
+ Misses       1369     1353      -16     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.