Skip to content

fix(storage): await Worker exit before isolate reclaim on Windows - #813

Merged
Wibias merged 14 commits into
lidge-jun:devfrom
Wibias:fix/ci-windows-no-isolate
Jul 31, 2026
Merged

fix(storage): await Worker exit before isolate reclaim on Windows#813
Wibias merged 14 commits into
lidge-jun:devfrom
Wibias:fix/ci-windows-no-isolate

Conversation

@Wibias

@Wibias Wibias commented Jul 31, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Keep bun test --isolate on every OS (do not drop isolation on Windows).
  • Deterministically terminate and await storage Bun Workers (close listener at spawn, spawn gate, Windows post-close join, async reset/abort).
  • Split Worker-spawning policy API cases into one-test-per-file suites; add repeated setup/teardown regression coverage.
  • Do not use bun test --shard with --isolate on Bun 1.3.14 — both shards ran the full suite and doubled Windows CI load until GUI tests failed.

Test plan

  • Local: reproduce panic with bun test --isolate across storage suites
  • Local: worker lifecycle + teardown isolate tests green
  • Local: stress of split policy + worker suites without panic
  • CI: Windows/Linux/macOS green with single isolate Test step
  • Confirm required CI keeps bun test --isolate tests (no Windows non-isolate workaround)

Bun 1.3.14 can panic in the isolated-worker path on Windows; keep isolation on Linux/macOS until upstream is fixed.
@github-actions github-actions Bot added the bug Something isn't working label Jul 31, 2026
@coderabbitai

coderabbitai Bot commented Jul 31, 2026

Copy link
Copy Markdown

Review Change Stack

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: e2e49c68-85bc-45ff-9e6b-19c70db9e8ed

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

The CI workflow now uses separate test commands for Windows and non-Windows runners. Windows runs bun test tests; Linux and macOS run bun test --isolate tests. The job timeout is 20 minutes.

Changes

CI test execution

Layer / File(s) Summary
Platform-specific test commands
.github/workflows/ci.yml:84-92
The workflow runs bun test tests on Windows. Linux and macOS continue to run bun test --isolate tests.

Estimated code review effort: 2 (Simple) | ~10 minutes

Suggested reviewers: lidge-j, ingwannu

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately identifies the Windows isolate issue, but it describes an implementation change that is not present in the CI workflow.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/ci.yml:
- Line 89: Update both CI test steps in the workflow to use the configured Bun
test root: replace the explicit `tests` path with `bun test` and apply the
corresponding `bun test --isolate` command to the isolated step, preserving the
existing step structure.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 9a33246f-ead4-4b7c-9b5a-67e496d575b5

📥 Commits

Reviewing files that changed from the base of the PR and between 33caf33 and b57c062.

📒 Files selected for processing (1)
  • .github/workflows/ci.yml

Comment thread .github/workflows/ci.yml Outdated
@Wibias
Wibias marked this pull request as draft July 31, 2026 11:24
@Wibias

Wibias commented Jul 31, 2026

Copy link
Copy Markdown
Collaborator Author

[shipping-github] Verdict: approve-comment

PR: #813 — fix(storage): await Worker exit before isolate reclaim on Windows
Head: 797dec1a on dev (mergeStateStatus: CLEAN)
Linked: none

Usefulness

Real Windows CI failure: Bun bun test --isolate panicking with workers_spawned(N) workers_terminated(N-1) when storage Bun.Worker threads were still exiting at the isolate boundary. Tip dev was also red with the same panic class after unrelated merges; this PR keeps --isolate on every OS and makes teardown joinable. Dropping isolate (earlier Sol YAML) hung Windows (~20m cancel) and is correctly abandoned.

Bugs / correctness

  • Method: bug-review.md — Bugbot: yes; complementary: done (silent_failures / resource_leaks / edge_cases)
  • Findings fixed this session:
    • Mutation slot released before worker join on async abort / watchdog timeout (src/storage/policy-job.ts, restore sibling)
    • Windows post-close settle skipped when terminate timed out (src/storage/worker-lifecycle.ts)
    • Queued spawnGate callbacks could spawn after teardown (cancelQueuedStorageWorkerSpawns)
    • Shutdown aborted only the first job on join failure (drainAndShutdown uses Promise.allSettled)
    • Tip-introduced silent inspection drain hang: unref'd timer never woke Bun reader.read on Windows (src/server/relay.ts) — reproduced locally, fixed
    • Tip-introduced required:true ACL harden aborting OpenAI-tier backup on CI temp volumes (src/config.ts soft-fail + EBUSY retry in tests/server-auth.test.ts)
  • Residual: sync abortStorageCleanupPolicyJob() still cannot join (documented; shutdown/tests use async paths)

Security

  • Scope reviewed: authz (management helpers retained), business_logic/races (mutation slot), removed_controls (managementFetch / authMode: "forward" still in tests/helpers/storage-policy-api.ts), AI/MCP keyword hits are storage "policy" false positives
  • Findings: none confirmed · Decision: Pass · Risk: Low
  • Fixed this session: none security-class; ACL soft-fail is scoped to migration backup only (credential writes stay required: true)

Spec / standards

  • Spec source: keep --isolate everywhere; fix Worker/handle teardown; no retries / no job-timeout raise; shard only if isolate race remained (Bun --shard under --isolate double-ran suite — dropped)
  • Gaps: none vs that bar; CI tip still runs bun test --isolate tests on all OSes

Reviews

  • Owners/maintainers: none open
  • Bots: CodeRabbit positional-tests nit resolved/declined earlier; Codex P2 spawn-gate + independent abort — fixed and threads resolved; 0 unresolved threads after thin settle

Base / CI

  • Behind/conflicts: clean vs dev (ahead 14, behind 0)
  • Required checks: green on 797dec1a (ubuntu / macos / windows / npm-global / enforce-target / react-doctor)
  • Local tip: focused storage worker suites + tests/consume-for-inspection-cancel.test.ts green after relay fix

Gate

none (draft cleared; mergeable CLEAN)

Bottom line

Approve for merge when ready: isolate stays, worker teardown is deterministic, tip CI regressions pulled in by sync are patched, and required CI is green on tip.

Keep bun test --isolate on every OS. Join storage Worker threads (and
server shutdown) before realm teardown, split Worker-spawning policy API
cases into one-test-per-file suites, and shard CI so Bun's residual join
race cannot turn Windows into a roulette wheel.
@Wibias Wibias changed the title fix(ci): skip bun --isolate on Windows runners fix(storage): await Worker exit before isolate reclaim on Windows Jul 31, 2026
Wibias added 6 commits July 31, 2026 21:24
Also drop the startServer stop-wrapper that hung aborted pool-retry
auth tests while awaiting storage Worker drain on every server.stop.
Bun 1.3.14 ran the full suite for both --shard=1/2 and --shard=2/2, so
Windows CI executed the isolate tests twice and then failed GUI tests.
Keep a single bun test --isolate tests step; Worker teardown remains.
Abort/timeout paths were releasing the shared CODEX_HOME slot (or skipping the Windows post-close settle) before the thread finished, which could overlap concurrent mutators and recreate the isolate reclaim panic on shutdown.
Eight rapid spawn/reset cycles on Linux CI segfaulted Bun 1.3.14 after a balanced worker count; keep the heavy loop on win32 where the isolate panic was observed.
@Wibias
Wibias marked this pull request as ready for review July 31, 2026 21:11

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 13bbe72df6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/storage/worker-lifecycle.ts
Comment thread src/server/lifecycle.ts Outdated
Wibias added 5 commits July 31, 2026 23:22
Drain no longer skips restore when policy join fails, and spawnGate callbacks waiting outside liveWorkers cannot create a Worker after shutdown/reset.
Tip startServer was aborting server-auth tests when icacls failed on CI temp volumes; keep chmod and retry EBUSY cleanup of the shared harness dir.
An unref'd setTimeout never woke Bun's pending reader.read on Windows, so the silent time-bound inspection drain hung CI until the 20m job cancel.
@Wibias

Wibias commented Jul 31, 2026

Copy link
Copy Markdown
Collaborator Author

[shipping-github] Merge ready

PR: #813 — fix(storage): await Worker exit before isolate reclaim on Windows
Head: 797dec1adev (mergeStateStatus: CLEAN)
Linked issues: none

Reviews

  • Humans (owners first): none open
  • Bots: 0 unresolved useful threads; CodeRabbit bun test root nit declined (keep positional tests); Codex spawn-gate + shutdown abort independence fixed in 93fee2f2 / tip sync
  • Own bug + security + spec/standards: Bugbot + complementary lenses + security-scope matrix on tip; blockers fixed (mutation-slot join, Windows settle-on-timeout, spawn cancel epoch, drainAndShutdown allSettled, relay drain timer ref, OpenAI-tier backup ACL soft-fail)

Tip freshness

  • Updated from dev: yes (797dec1a includes tip stream/inspection + auth/config merges)
  • Compiles/tests against tip: bun test tests/consume-for-inspection-cancel.test.ts + storage worker teardown suites pass; tsc --noEmit clean earlier in session
  • Conflicts: none

Checks

  • Local/CLI: focused Bun suites above green
  • Required CI: green on 797dec1a (windows-latest ~13m49s; ubuntu/mac/npm-global/enforce-target/react-doctor)
  • Policy: CODEOWNERS enforcement / stale approvals / merge-queue: n/a / clear

Residual

  • Sync abortStorageCleanupPolicyJob() remains fire-and-forget by design; production drain uses async abort
  • Tip dev had been failing Windows on worker panic / inspection hang — this PR is the recovery path

Ready to merge.

@Wibias

Wibias commented Jul 31, 2026

Copy link
Copy Markdown
Collaborator Author

Merging this.

Why it helps: Windows bun test --isolate was panicking when storage Bun.Worker threads were still exiting at the file realm boundary (workers_spawned(N) workers_terminated(N-1)). Teardown now joins those workers (and related mutation/spawn gates) before reclaim, so cross-platform CI can keep isolate on every OS instead of weakening Windows. Tip sync also fixed a silent post-cancel inspection drain hang that was cancelling the Windows job at 20m.

Ship it.

@Wibias
Wibias merged commit 7b6f4f4 into lidge-jun:dev Jul 31, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant