Skip to content

Fix the dev-db socket test wedge: bind conflicts hung server.start() - #1853

Merged
RhysSullivan merged 1 commit into
mainfrom
deflake-devdb-socket
Aug 29, 2026
Merged

Fix the dev-db socket test wedge: bind conflicts hung server.start()#1853
RhysSullivan merged 1 commit into
mainfrom
deflake-devdb-socket

Conversation

@RhysSullivan

Copy link
Copy Markdown
Collaborator

Symptom

The dev-db PGlite socket tests were the loudest CI flake. Different tests in
apps/cloud/src/db/dev-db-socket-concurrency.node.test.ts died as bare vitest
timeouts (30s/60s) across main and PR branches — five wedges in the last two
days alone — and test-globalsetup-exit.node.test.ts flaked alongside them.
The wedges never carried diagnostics, and hundreds of local replays never
reproduced one.

Mechanism

The tests bound fixed ports 45993-45998, and the globalsetup meta-test used
45435/45436. All of these sit inside the default Linux ephemeral port range
(32768-60999). On a busy CI runner another socket can hold one of them at bind
time: an outbound connection from a sibling suite in the same turbo shard, or
a leaked e2e server (the e2e harness's own port code documents this exact
contention for its 42000-45999 block).

When that happened, the bind failure vanished: pglite-socket's start() sets
active = true before listen(), and its error handler only rejected the
start promise while active was false. EADDRINUSE therefore dispatched an
error event nobody listened to, and await server.start() never settled.
The test then died at its vitest timeout with zero output — before reaching
the wedge diagnostics added in #1767. The meta-test variant: the nested vitest
run hung the same way until spawnSync's timeout killed it (signal !== null).

macOS assigns ephemeral ports from 49152, so the collision was impossible
locally on a Mac — matching the "CI-only, any test, no diagnostics" shape of
every occurrence.

Fix

  • Patch (patches/@electric-sql%2Fpglite-socket@0.1.4.patch): start() now
    always rejects on a server error. A settled promise ignores later rejects,
    so post-listen errors still surface only through the error event.
  • The concurrency tests bind port 0 and read the OS-assigned port from the
    server's listening event.
  • The globalsetup fixture accepts port 0 and the meta-test uses it; the main
    suite keeps 5434, which is outside the contested range.
  • New regression test: a bind conflict rejects start() instead of hanging.
  • The Attach a diagnosis to the dev-db socket wedge timeouts #1767 wedge diagnostics stay in place for any future wedge that really
    is in the query queue.

No changeset: the patch and tests only affect the repo-local dev/test
topology. pglite-socket consumers here are the root package, the private
cloud app, and runtime-dynamic-worker (changeset-ignored).

Proof

  • Red: with a squatter on 45996 + 45998 and the old code, one run reproduced
    the exact CI signatures deterministically — interleave test "timed out in
    60000ms", idle-backstop test "timed out in 30000ms", all other tests green.
  • Red: the new regression test on the old patch hangs to its 30s timeout.
  • Green: same squatters plus the fix — all tests pass; 10 consecutive green
    runs of both files; one green run under 2x-CPU-count busy-loop load with all
    eight legacy ports squatted; full apps/cloud suite green (50 files, 359
    tests).
  • Controls: 10 plain + 12 Node-24-under-load runs on macOS and 30 runs on an
    idle Linux box all passed on the old code, confirming the wedge needs port
    contention, not load.

@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
executor-marketing a3ed588 Commit Preview URL

Branch Preview URL
Aug 29 2026, 09:16 AM

@github-actions

github-actions Bot commented Aug 29, 2026

Copy link
Copy Markdown
Contributor

Cloudflare preview

Torn down — the PR is closed.

@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
✅ Deployment successful!
View logs
executor-cloud a3ed588 Aug 29 2026, 09:17 AM

@pkg-pr-new

pkg-pr-new Bot commented Aug 29, 2026

Copy link
Copy Markdown

Open in StackBlitz

@executor-js/cli

npm i https://pkg.pr.new/@executor-js/cli@1853

@executor-js/config

npm i https://pkg.pr.new/@executor-js/config@1853

@executor-js/execution

npm i https://pkg.pr.new/@executor-js/execution@1853

@executor-js/sdk

npm i https://pkg.pr.new/@executor-js/sdk@1853

@executor-js/codemode-core

npm i https://pkg.pr.new/@executor-js/codemode-core@1853

@executor-js/runtime-quickjs

npm i https://pkg.pr.new/@executor-js/runtime-quickjs@1853

@executor-js/plugin-file-secrets

npm i https://pkg.pr.new/@executor-js/plugin-file-secrets@1853

@executor-js/plugin-graphql

npm i https://pkg.pr.new/@executor-js/plugin-graphql@1853

@executor-js/plugin-keychain

npm i https://pkg.pr.new/@executor-js/plugin-keychain@1853

@executor-js/plugin-mcp

npm i https://pkg.pr.new/@executor-js/plugin-mcp@1853

@executor-js/plugin-onepassword

npm i https://pkg.pr.new/@executor-js/plugin-onepassword@1853

@executor-js/plugin-openapi

npm i https://pkg.pr.new/@executor-js/plugin-openapi@1853

executor

npm i https://pkg.pr.new/executor@1853

commit: a3ed588

@RhysSullivan
RhysSullivan marked this pull request as ready for review August 29, 2026 18:33
@RhysSullivan
RhysSullivan merged commit 3ba28ca into main Aug 29, 2026
44 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant