dev-up: make port release deadline-based and distinguish a live foreign owner from a lingering socket

[Chris0Jeky]

Root cause

Tonight's hosted-Windows red PowerShell: Vite fallback leaves the foreign frontend-port owner alive
(scripts/ci/dev-up.test.mjs:1716, asserting inside stopSuccessfulStack at
scripts/ci/dev-up.test.mjs:955-961) was not about the foreign listener. The failure message names
state.Frontend.Port — the launcher's own Vite fallback port.

Path:

1. Stop-LoadedStack (scripts/dev-up.ps1:423-434) reaps both recorded trees cleanly.
2. It then calls Wait-PortRelease -Port $script:State.Frontend.Port (scripts/dev-up.ps1:367-374),
   which allowed only 50 iterations x 100 ms = 5 s and required Test-PortBindable
   (scripts/dev-up.ps1:357-365) to get an exclusive bind on both IPv4 and IPv6 loopback.
3. On a loaded hosted runner the kernel can hold the listening socket past 5 s after taskkill /T /F,
   so Wait-PortRelease returned $false -> "Frontend port N is still occupied. No foreign listener was killed; PID state is retained." -> $clean = $false -> throw "Stack cleanup was incomplete"
   (scripts/dev-up.ps1:699), exit 1.

A correctly dead stack was reported incomplete. Same family as the fixed 3 s marker budget in #2157:
a hard-coded bound measured against hosted wall-clock variance.

Occurrences: run 33835062683 (#2506) and run 33833389166 (#2497).

Fix

Both launchers (scripts/dev-up.ps1, scripts/dev-up.sh) had the same {1..50} / 50 x 100 ms bound.

1. Deadline, not iteration count. Wait-PortRelease / wait_for_port_release poll every 250 ms
   against an elapsed-time deadline from TASKDECK_DEV_UP_PORT_RELEASE_TIMEOUT_MS (default 30000).
   An unparseable value warns and falls back to the default.
2. Live foreign owner vs lingering socket. Only after the deadline is the port's LISTEN owner
   inventoried (Get-NetTCPConnection -State Listen -LocalPort, falling back to netstat -ano; lsof
   then ss on Bash), filtered to PIDs that are still live:
   • a live listener -> the existing warning, retained PID state, exit 1 (the message says whether the
     owner is unrelated or a recorded PID that outlived its reap, comparing against the reaped tree PIDs);
   • no live listener -> a lingering kernel socket behind our own confirmed-dead tree; a diagnostic
     line is printed and the port counts as released;
   • an inventory that cannot be read at all -> fails closed, same as a live owner.
3. Fail-closed semantics preserved. Nothing weakens the foreign-process guarantees: no listener is
   ever killed, and any live owner still retains state and exits 1.
4. No workflow files are touched.

Refs #1898
Refs #2378
Refs #2157

Verification

All local, Windows, Node 24, in an isolated worktree at head c4509b0ab.

New regression tests (scripts/ci/dev-up.test.mjs), both platforms, both stopping a synthetic
schema-v1 state whose recorded PID is already reaped so only the port-release stage is under test:

• a frontend port released after the old fixed budget still stops cleanly — an out-of-process holder
  occupies the recorded frontend port and releases it 12 s in; the launcher must exit 0 and remove state.
• a live listener still holding the frontend port fails closed — a live listener keeps the port; the
  launcher must exit non-zero, keep the PID state, and leave the listener alive.

Negative control (the point of the first test): with scripts/dev-up.ps1 / scripts/dev-up.sh
stashed back to origin/main and only the new test applied —

✖ PowerShell: a frontend port released after the old fixed budget still stops cleanly (6543ms)
✔ Bash:       a frontend port released after the old fixed budget still stops cleanly (12436ms)

and with the fix applied, both pass. (The Bash launcher's old loop spent ~12 s of wall clock on 50
node-subprocess probes, so its effective budget was already long; its change is consistency and the
same owner classification, and it is still covered by the test.)

Targeted runs — node --test --test-concurrency=1 --test-timeout=30000 --test-name-pattern=... scripts/ci/dev-up.test.mjs,
for the two new tests plus the originally-red Vite fallback leaves the foreign frontend-port owner alive:

run	result
1 (pre-rebase)	6/6 pass
2 (pre-rebase)	5/6 — Bash: Vite fallback... hit the 30 s --test-timeout at 30005 ms
3 (pre-rebase)	6/6 pass (Bash: Vite fallback... 20771 ms)
4 (post-rebase, PowerShell only)	3/3 pass

Full suite, split by platform to fit the local shell timeout:

• --test-name-pattern="^PowerShell": tests 44, pass 43, fail 1, duration 128989 ms. The single
  failure was an exited preflight broker does not fail process identity capture, and it failed on
  taskkill — FATAL: ERROR: The process with PID 47068 (child process of PID 27932) could not be terminated — i.e. the [Bug][Priority II] Close remaining dev-up fail-closed availability races #1898 taskkill race, before any port check runs. Re-run by name three times:
  pass, pass, pass.
• --test-name-pattern="^Bash": several reds, all 20-50 s durations against the 30 s --test-timeout.
  Compared against a clean origin/main worktree (C:/Users/jekyt/source/td-wt/base1898) running
  the same cases: tests 17, pass 6, fail 5 — clean main is equally red (and red on cases this branch
  passed, e.g. invalid Vite outcome duplicate, duplicate-property). These are the known Frontend Unit (windows-latest) times out on slow runners: dev-up.test.mjs PowerShell spawns hit ETIMEDOUT and the job hits its timeout #2378
  Git-Bash-on-Windows slowness class on this box, not this diff; the hosted lane runs Bash on Linux.

Hosted proof is this PR's own Frontend Unit jobs.

Not verified

• No hosted-Windows run of this branch yet — the local box cannot reproduce the loaded-runner socket
  lingering that motivated the change, so the original red is proven by causal reading of the code
  path plus the deadline negative control, not by a local reproduction of the hosted failure.
• The netstat -ano fallback branch of Get-LivePortListenerOwner was not exercised:
  Get-NetTCPConnection is present on this box, so only the primary path ran.
• The ss branch of live_port_listener_owners was not exercised; Git Bash on Windows has neither
  lsof nor ss, so the Bash fail-closed test took the "could not be inventoried" path locally. The
  lsof path will run for the first time on hosted Linux.
• Full-suite green in one process was not obtained locally on either platform (see above); the
  PowerShell failure and the Bash failures both reproduce on unmodified origin/main.

Risks

• The default stop-path wait grows from 5 s to a 30 s ceiling only when the port does not release.
  A healthy stop returns on the first successful bind and is unchanged; a genuinely stuck port now
  costs up to 30 s before the same fail-closed error.
• Treating "unbindable but with no live listener" as released is a deliberate relaxation. It is
  bounded to: after the deadline, after both recorded trees were proved gone, and only when a listener
  inventory was actually readable. A port with no listening process cannot serve anything, and a later
  launch that cannot bind it still fails loudly at Test-PortBindable.
• Get-NetTCPConnection can be slow on a busy machine; it is called at most twice per stop and only
  after the deadline has already expired.

---

Control-plane change: awaiting maintainer review per the ADR-0066 amendment (2026-09-03). This PR is
parked ready-for-review and must not be merged by an agent.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

[Chris0Jeky]

Review (agent half of the ADR-0066 gate; Codex credits exhausted, SC-9): one fresh-context reviewer confirmed no kill/signal path was added or widened (all new process interactions are read-only), enumerated every classification branch as fail-closed on Windows, and found Test-PortBindable unchanged. FIX-FIRST on one HIGH: in scripts/dev-up.sh an unprivileged lsof/ss cannot attribute another user's listener, so a root-owned foreign listener (docker-proxy, a systemd unit) reads as 'no live listener', the port is reported released, the PID file is removed and --stop exits 0 — a false clean report. Plus MEDIUM: the recorded/unrelated survivor label is bare-PID, contradicting the identity discipline of the kill path. Fix round in progress; after it, this PR stays parked for the maintainer's review per SC-10.

[Chris0Jeky]

Fix round pushed as d8d53002a (one commit).

HIGH (accepted, and it applied to Windows too). The bug was reading an empty owner list as
"nothing is listening". Socket existence and PID attribution are now separate concerns:
existence comes from a source that covers every account (ss -ltn, netstat -an on BSD/macOS, and
the kernel TCP table via Get-NetTCPConnection / netstat -ano), while attribution is best-effort
and only decorates the diagnostic. Release is accepted only when the port becomes bindable, or when
a readable inventory positively shows no listening socket; an unattributable listener and an
unreadable inventory both fail closed with the PID state retained. The same defect existed on the
PowerShell side — Get-Process -Id can fail for a socket owned by another account or by SYSTEM, and
that dropped owner would have made a live listener look ownerless — so Get-LivePortListenerOwner
became Get-PortListenerInventory with an explicit Listening flag.

New Bash test a listening socket with no attributable owner still fails closed reproduces exactly
the reported shape with fake-bin shims: an ss that prints a LISTEN row for the target port and
never a pid= field, plus an lsof that refuses. Negative control — with only scripts/dev-up.sh
reverted to the previous commit, the test fails; with the fix it passes.

MEDIUM (accepted). The recorded / unrelated label classified survivors by bare PID, which
contradicts the PID+name+token identity the kill path requires. Dropped; the warning now reports PID
and process name only.

LOW (accepted). TASKDECK_DEV_UP_PORT_RELEASE_TIMEOUT_MS is now bounded to int32 in the Bash
launcher, matching the PowerShell [int]::TryParse, with the same fallback warning. Both launchers
report elapsed time in ms (Bash gained a now_ms helper, so the deadline is millisecond-based
rather than second-rounded).

LOW (declined, tracked here). The release test's 12 s wall-clock hold stays. runLauncher uses
spawnSync, which blocks the test's event loop for the whole launcher run, so the test cannot
deliver a signal or touch a file while the launcher is waiting; the holder has to schedule its own
release. A signalling mechanism would need an extra watcher process and would trade one timing
assumption for a more complex one.

Verification (local, Windows, Node 24, head d8d53002a): bash -n scripts/dev-up.sh OK;
PowerShell AST parse of scripts/dev-up.ps1 OK. The two original new tests, the new
unattributable-owner test, and the originally-red Vite fallback leaves the foreign frontend-port owner alive, both platforms, run twice by name: 7/7 pass and 7/7 pass
(node --test --test-concurrency=1 --test-timeout=30000 --test-name-pattern=...).

Not re-verified in this round: the full per-platform suite (unchanged from the previous round's
report, where the residual reds reproduce on clean origin/main); the netstat fallback branch on
either platform; and the real lsof attribution path, which is stubbed out in the new test and
still first runs on hosted Linux.

Still a control-plane change awaiting maintainer review per the ADR-0066 amendment (2026-09-03).

[Chris0Jeky]

Round 2 (scoped to the fix diff): the fresh-context reviewer confirmed the HIGH and MEDIUM are closed on both launchers — existence is probed separately from attribution (ss without -p / netstat fallback; explicit Listening flag on Windows), release is accepted only on a successful bind or a readable inventory positively showing no listener, unattributable and unreadable both fail closed, the new Bash test's PATH shim genuinely shadows real ss/lsof on hosted Linux and discriminates against the previous script, and no kill path was added. Verdict SHIP (agent half of the gate). One new MEDIUM confined to the netstat fallback (English LISTEN token vs localized Windows) tracked in the issue linked above. This PR now stays parked for the maintainer's review per ADR-0066 / SC-10; hosted Frontend Unit runs at d8d5300 are the R4 proof.

[Chris0Jeky]

Coordinator note (2026-09-04 overnight run wrap-up): this PR is queued for your review under OUTSTANDING_TASKS SC-10, but GitHub now reports it CONFLICTING / DIRTY — two further dev-up commits landed on main after it opened (#2548 8c0779f and #2556 6c10850, both touching scripts/ci/dev-up.test.mjs). Its last hosted run is therefore against a stale base. It needs a merge from main and a fresh hosted re-prove before a review of it means anything; the agent-side gate (round 1 FIX-FIRST on the unprivileged-lsof/ss privilege hole, fixed, round 2 SHIP) still stands for the content. Its stacked follow-up #2531 (#2526, localized netstat states) should be re-targeted to main only after this lands.

[Chris0Jeky]

Conflict resolved — head is now 18d214ba2

SC-10 flagged this PR as queued but not mergeable. It is now MERGEABLE against main and re-proving hosted.

The conflict was one region, and mechanical. Merge base 1d768caaa. This branch inserts 182 lines of new port-release tests immediately above the high-volume stdout and stderr cannot deadlock marker acceptance declaration; main's 8c0779fca modified that exact declaration line, adding timeout: 60_000. Git cannot order an insertion against a change to its anchor line. scripts/dev-up.ps1 and scripts/dev-up.sh did not conflict at all.

Resolution keeps both sides — every inserted line verbatim, with main's declaration (timeout included) moved below the block; this branch's own copy of that declaration, which lacked the timeout, is dropped.

Merged rather than rebased, deliberately. git merge-base --is-ancestor origin/issue-1898/port-release-deadline origin/issue-2526/netstat-state-fallback returns true — #2531's head has this branch's old head d8d53002a as an ancestor, so a force-push here would have stranded the stacked child. SC-10's text suggests a rebase; that guidance is unsafe while #2531 is open, so I took the merge.

Verified after resolution: the diff against main contains zero removed lines (pure insertion, so nothing of main's was lost); node --check scripts/ci/dev-up.test.mjs passes; and both of main's fixes survive in the tree — closeAllConnections (3 hits) and timeout: 60_000 (1 hit).

This PR is not superseded

Worth stating plainly, since two dev-up commits landing on main invites the assumption. git diff --stat 1d768caaa origin/main over the three dev-up paths shows scripts/ci/dev-up.test.mjs changed and nothing else — neither landed commit touches a launcher. origin/main:scripts/dev-up.ps1 still has Wait-PortRelease looping for ($attempt = 0; $attempt -lt 50; ...) and origin/main:scripts/dev-up.sh:375 still has for _ in {1..50}; do ... sleep 0.1; done. The fixed 5-second budget this PR exists to remove is 100% intact on main. The two landed commits solve different problems: 6c10850e6 made the test fixture's stub server close accepted connections on TERM; 8c0779fca gave the high-volume drain test a longer per-test timeout on Windows. Both are test-harness hygiene; neither changes port-release semantics.

Still required before merge

1. The maintainer's own review. This is T2 control-plane (scripts/ci/**), so ADR-0066's 2026-09-03 amendment requires it in addition to the fresh-context review already on this thread. It stays parked under OUTSTANDING_TASKS.md SC-10.
2. A fresh hosted green at 18d214ba2. The previous head was itself red — Frontend Unit (windows-latest) on run 33840188131 was cancelled at exactly 25m00s, stalled in Run source launcher regression suite, with steps 5-12 never starting. That belongs to the Frontend Unit (windows-latest) times out on slow runners: dev-up.test.mjs PowerShell spawns hit ETIMEDOUT and the job hits its timeout #2378/[CI][Windows] Stabilize dev-up reset-seed first-use deadline #2161 Windows cohort, not to this diff, but it means the pre-merge evidence has to come from this new head, not the old one.
3. The earlier review rounds (05:15Z, 05:25Z) were posted against a base that has since moved. Per global law a base change counts as a head change; the fix here is a merge resolution with no logic change, so no fresh adversarial round is owed, but the CI evidence must be re-taken.

Stacked child

#2531 still targets this branch and now has its first fresh-context review posted (verdict FIX-FIRST — the netstat fallback stopped discriminating on state and reports TIME_WAIT/ESTABLISHED as listening). Order, per global law 4: this PR merges first; only after it has actually merged, retarget #2531 with gh pr edit 2531 --base main and confirm via the API. Do not --delete-branch issue-1898/port-release-deadline while #2531 targets it — that cascade-closes #2531 unreopenably. Repo delete_branch_on_merge is currently false, so a plain gh pr merge 2522 --merge is safe.

One disclosure for your review pass: #2531 also silently amends this PR's unlanded port_release_timeout_ms and now_ms functions, so the stack's combined effect differs from this base read in isolation. Details on #2531.

[Chris0Jeky]

The new red at 18d214ba2 is #2378, not the conflict resolution

Recording the attribution with its mechanism, because this PR does change scripts/ci/dev-up.test.mjs, so "known flake" is not something to assert without evidence here.

The failure (run 33922229492, job 101182893988, step 4 Run source launcher regression suite):

test at scripts\ci\dev-up.test.mjs:1157:3
✖ PowerShell: pipeline cancellation runs transactional cleanup from finally (20592.8533ms)
  AssertionError [ERR_ASSERTION]: ifError got unwanted exception:
    spawnSync C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe ETIMEDOUT
      at runPowerShellCancellation (scripts/ci/dev-up.test.mjs:829:14)
ℹ tests 90 / ℹ pass 89 / ℹ fail 1 / ℹ cancelled 0

Why it is not this PR's resolution. The conflict I resolved is a single region at merged-blob lines 1809-1994 — 183 added lines, all insertion. The failing test is at line 1157, roughly 650 lines above it, in code this branch does not touch. git diff origin/main -- scripts/ci/dev-up.test.mjs | grep '^-[^-]' is still empty, so nothing of main's was removed.

Why it is #2378 specifically, rather than a generic appeal to flakiness. #2378's first recorded occurrence is this same test by name, with this same error, at this same magnitude: "PR #2369 head 16194cc, run 33580047231: scripts/ci/dev-up.test.mjs 'PowerShell: pipeline cancellation runs transactional cleanup from finally' failed with spawnSync powershell.exe ETIMEDOUT (20.4 s)". Here it is 20592.85 ms. The mechanism is spawnSync's own timeout in runPowerShellCancellation (:829) elapsing on a slow runner — the suite launches real PowerShell processes serially and its wall time is dominated by runner speed. Not a job-level timeout: the suite ran 398 s inside a 7 m 29 s job, well under the 20-minute budget.

What the resolution positively proves. The three tests this branch inserts all passed on the Bash leg in this run, including a listening socket with no attributable owner still fails closed (5307 ms), and main's high-volume stdout and stderr cannot deadlock marker acceptance passed at 12723 ms with the timeout: 60_000 that main's 8c0779fca added and my resolution preserved. So both sides of the conflict are working.

Disposition. This PR is T2 control-plane and parked under SC-10 for the maintainer regardless, so this red blocks nothing today. One re-run is warranted to give the maintainer a clean head to merge from. Per the convergence rule I will not exceed three genuinely different attempts at this check — and if the same test fails twice, that stops being a runner-speed story and becomes a regression to investigate, not to re-run.

Adding this as a third recorded occurrence on #2378.

[Chris0Jeky]

Second red at 18d214ba2 — also not this PR, and a different cause from the first

Both reds in run 33922229492 are now diagnosed. Neither is the conflict resolution.

API Integration (windows-latest) (job 101182893700) — one C# test of 2 834 failed:

Taskdeck.Api.Tests.Resilience.WorkerResilienceTests
  .ProposalHousekeepingWorker_WhenDbThrows_LogsErrorAndContinuesPolling [FAIL]
  Expected callCount to be greater than 0, but found 0.
Failed! - Failed: 1, Passed: 2829, Skipped: 4, Total: 2834, Duration: 20 m 33 s

WorkerResilienceTests.cs:75-103 starts the worker, await Task.Delay(300), cancels, then asserts the worker polled at least once. The 300 ms fixed delay is the synchronisation — nothing waits for the condition. On a runner where the same suite took 20 m 33 s, the worker had not reached its first iteration inside that window. Scheduling artifact, not a resilience defect. Filed as #2572 with the suggested fix (bounded wait on the condition instead of a fixed delay).

This PR changes only scripts/ci/dev-up.test.mjs, scripts/dev-up.ps1 and scripts/dev-up.sh — no C#, no worker. git diff origin/main...HEAD | grep -ci 'ProposalHousekeeping\|WorkerResilience' returns 0.

Where that leaves the two reds. They are separate causes, and it is worth not collapsing them:

Check	Failure	Tracked as
Frontend Unit (windows-latest)	spawnSync powershell.exe ETIMEDOUT, dev-up.test.mjs:1157	#2378 (third occurrence)
API Integration (windows-latest)	fixed-Task.Delay polling assertion, C#	#2572 (new)

Both are Windows-runner-contention shapes on a leg where this PR's own content passed. The three tests this branch inserts passed, and main's timeout: 60_000 fix — preserved by the resolution — passed at 12 723 ms.

Disposition unchanged. This PR is T2 control-plane and parked under SC-10 for the maintainer, so neither red blocks anything tonight. When it is taken up: run gh pr update-branch 2522 (main has advanced four merges since this head — 8c511205d, 22ec0d333, 93d021d01, df1559fd8) and take the evidence from that fresh run rather than from this one. I have not re-run the current head, deliberately: a re-run would reuse the stale merge ref, and re-proving against a base that has since moved is the thing that actually needs doing.

Stop rule still stands: if PowerShell: pipeline cancellation runs transactional cleanup from finally fails again on a fresh head, that is a regression to investigate rather than re-run.