Require transport evidence before a copy-progress watchdog fire

[kriszyp]

Stops the copy-progress watchdog (#453) from killing healthy base copies whose peer goes briefly transport-dark — the 2026-08-29 scheduled Integration Tests failure on main (run 33239813839), where copyProgressBackpressureFalseFire.test.mjs failed with a fire whose own truth payload read connected: true … liveness: 2s ago.

Diagnosis (from the run's server-log artifacts). The #695 pause-guard did NOT regress: no fire occurred during the paused window. The subscriber resumed from commit-backlog back-pressure at 07:03:53.808, re-arming the watchdog on its 1500ms test budget — while the source's event loop was dark from 07:03:51.445 to 07:03:55.810 (4.365s: zero log lines, zero bytes, not even WS pongs, and the reconnect's TLS handshake was only accepted the moment it woke — a starved 2-core CI runner running concurrent test files). The truth payload was consistent, not contradictory: liveness staleness (2s) exceeded the watchdog budget (1.5s), so nothing at all had arrived in the window. The defect is jurisdictional: an app-level watchdog acted on transport-level silence with an app-level budget. Transport-dark connections belong to the byte-silence machinery (receive watchdog at copyTimeout during copy, keepalive idle terminate), which would have let this stall ride and the copy resume on the same connection.

Fix. createReceiveWatchdog accepts an optional transport-activity sampler (received socket bytes); the silent-window check re-arms unless the sampled counter moved and the strike survives a confirmation. A primary-silent window fires only when peer bytes moved during it AND keep moving through a transport-scale confirmation (2× pingInterval, floor 1s) — the #453 wedge signature is ping-alive-but-frame-dead, and a live peer proves itself with a fresh pong within a ping cycle. The confirmation exists because one window of movement is not proof: bytes buffered behind a receive pause or a suspended peer's kernel send buffer land moments after a re-arm (observed on a real run of the new suite as a partial-frame tail draining right after a back-pressure resume), and that residue always empties within milliseconds while it cannot recur a ping cycle later. Sizing the confirmation from the ping cadence — not a second blobTimeout — keeps the configured value the recovery bound (detection stays 1–2× blobTimeout + ≤2× pingInterval; invariant recorded in DESIGN.md item 15(c)). Windows dark on both counters re-arm and stand down. An unobservable or throwing sampler is "no evidence" — stand down and report once — never licence to fire. Only the copy-progress watchdog supplies the sampler; the byte, pause-stall, and finalize watchdogs keep their exact pre-gate path (selected at construction). Look hardest at the dark-branch state machine around the pending strike (transportEvidencePending) — it is the piece with the most edge cases (baseline acquisition, residue decay, strike cleared by progress), each pinned by a unit case.

Budget context. With production defaults (copyTimeout 300s < blobTimeout 900s) a continuously dark transport was already reaped by the byte watchdog long before the copy-progress budget, so this gate mostly matters where blobTimeout is configured below copyTimeout (tests, tightened deployments) and as the stated invariant; findStalledReceivingNodeUrls remains the 15-minute defense in depth behind both. Wedge detection with defaults moves from ≤2× blobTimeout to ≤2× blobTimeout + 60s (two default ping cycles).

harper-pro#697 should be reopened or this PR linked from it: the false-fire class it describes had a third mechanism beyond the two #695 closed (pause re-arm, superseded-instance fires) — transport-dark windows judged on the app-level budget.

For the human reviewer

• Framing-Verdict: chosen-approach-sound (planning review, round 1; the two-window confirmation was that round's "single-window evidence vs. byte-active-at-expiry are different contracts" finding, made concrete by a real failure of the new suite 2).
• The confirmation window began as a full second blobTimeout (round 2 of this branch); the round-2 reviewers correctly flagged that as silently redefining the configured timeout, so it is now 2× pingInterval. copyProgressWedgeRecovery pins the documented bound and that no byte-level fallback acts on the ping-alive wedge.
• Fire-line wording changed (for <N>ms (and through a <M>ms confirmation) while peer bytes kept arriving). Nothing greps the old wording in-repo; external log dashboards keyed on the old text would need updating.
• ws._socket?.bytesRead in the sampler adds one more instance of the file's pre-existing _socket-not-on-WebSocket type complaint (build is || true on type errors, emit unaffected). Matches the existing idiom at the byte watchdog's getBytesRead.

Verification

• Fails on base, exactly: the new copyProgressSourceStallFalseFire.test.mjs run against unmodified dist reproduces the nightly's line verbatim (… for 1500ms while connected — … truth={connected: true, state: 2, liveness: 2s ago}) in both suites; with the fix both pass (suite 1: no fire, no reconnect, converges on the original connection; suite 2: byte-level machinery reconnects past copyTimeout, copy-progress line never appears).
• The repro is deterministic: SIGSTOP the source process group mid-copy (parked via the one-shot HARPER_TEST_COPY_COMMIT_DELAY_ONCE_DB hold), with log-derived vacuity guards asserting an armed, unpaused, ≥ budget dark window actually preceded SIGCONT, and unconditional SIGCONT in finally/after.
• Existing watchdog regressions all green with the fix: copyProgressWedgeRecovery (the true a customer cluster preprod 5.1.7: replicated deploys wedge after rolling upgrade — stalled system blob send (#450/harper#1443) blocks COPY_COMPLETE; fixes absent from 5.1.7 (needs 5.1.8) #453 wedge still fires and recovers — now also asserting the fire lands within the documented bound, anchored at COPY_START, with no byte-level fallback acting), copyProgressBackpressureFalseFire, watchdogSupersededInstanceFire, copyFinalizeWedgeRecovery.
• Unit: 844 passing (npm run test:unit), including the new createReceiveWatchdog gate cases — dark-window deferral, evidence confirmation (default and confirmIntervalMs-bounded), residual-burst decay, strike cleared by progress, unobservable/throwing sampler and reporter containment, baseline acquisition with degraded-observability reporting, and a sampler-call-count bound across 1000 throttled resets.
• npm run lint:required and prettier clean. Full integration matrix left to CI (locally: the six watchdog-family suites above).

Refs #697

---

Written by Claude Fable (dispatch dev-agent).

🤖 Generated with Claude Code

_{Review-Coverage: authored=claude; ran=codex; adjudicated=domain; blocked=gemini(permission-denied); declined=cursor-grok,cursor-composer; rounds=3 @ b2714a1}

_{Human-Review-Need: 4 (decisions: confirmation-window-sizing, watchdog-layer-choice, evidence-signal-choice, fail-open-on-no-evidence, pause-guard-redundancy, sigstop-integration-shape) @ b2714a1}

[gemini-code-assist]

Code Review

This pull request introduces a transport-evidence gate to the copy-progress watchdog to prevent false fires when a replication source goes dark (e.g., due to an event-loop stall or SIGSTOP) below the copyTimeout threshold. It updates the watchdog implementation to verify that peer bytes are actively arriving before triggering a reconnect, documents this behavior in replication/DESIGN.md, and adds comprehensive integration and unit tests. The review feedback suggests wrapping individual node teardown steps in try-catch blocks within the test cleanup hooks to ensure robust cleanup and better diagnostics.

[github-actions]

Release cherry-pick v5.2: tests ✅ passed

Cherry-picked PR #782 onto v5.2 at branch cherry-pick/v5.2/pr-782.
Integration tests: ✅ passed — https://github.com/HarperFast/harper-pro/actions/runs/33256348575

On merge of this PR the cherry-pick branch will be fast-forwarded into v5.2.

[claude]

Reviewed; no blockers found.