feat: N=1 cooperative fiber scheduler replacing per-thread host threads

[smmathews]

feat: N=1 cooperative fiber scheduler replacing per-thread host threads

What this replaces and why

Upstream runs each guest EE thread on its own host std::thread (g_hostThreads in Helpers/State.h, spawned from Thread.cpp/ps2_runtime.cpp). That model allows two guest threads to execute truly concurrently, which the real EE (a single core) never does — games' kernel-object usage (semaphores as pure ownership tokens, priority-based cooperative scheduling, RotateThreadReadyQueue round-robin) assumes exactly one running thread. Under the std::thread model this shows up as (DQ8 examples observed while debugging): audio-thread starvation, missed wakeups between WaitSema/SignalSema pairs, and lock-order deadlocks between the run token and guest mutexes (earlier attempt: #134).

This PR replaces that model with an N=1 cooperative fiber scheduler:

• Exactly one host OS thread (the guest executor) runs all guest fibers; one fiber runs at a time, highest priority first, FIFO within a priority (EE semantics; RotateThreadReadyQueue rotates the equal-priority group).
• Guest threads are fibers (ps2_fiber.{h,cpp}) with per-fiber R5900Context state; blocking syscalls park the fiber via an arm/publish/park protocol that cannot lose wakeups (see below).
• Host workers (IRQ/vsync/alarm/RPC) never run guest code directly; they either wake fibers through gated entry points or borrow the guest token (AsyncGuestScope) so handler code is serialized against fibers.
• Preemption points: recompiled code samples yield_point() every 128 back-edges (terminate/suspend/priority checks); blocking syscalls yield cooperatively.

Wakeup protocol (the core correctness argument)

A blocking syscall does: publish to the object wait-list under the object mutex → arm_park() (marks Blocked under the scheduler mutex) → release object mutex → block_current(). A waker that fires in the publish/arm window finds the fiber still running and records wake_pending instead of enqueueing (wake_locked); block_current() consumes it and returns WokenInWindow without parking; all wait sites re-check their predicate in a Mesa-style loop. Wakeups gate on suspendCount (a suspended waiter stays THS_WAITSUSPEND until ResumeThread). Stale wakeups after tid reuse are rejected by {generation, tid} tokens (enqueue_external_wakeup_validated). Host threads that call blocking syscalls (no fiber) use bounded-backoff Mesa loops on the same predicates.

Backends

Backend	Platform	Selection	Stack safety	Status
ucontext	Linux/macOS (default)	default	mmap + PROT_NONE guard page	tested: full suite N runs, ASan+LSan clean, deterministic
Win32 Fibers	Windows/MSVC	_WIN32	OS-native guard pages, FIBER_FLAG_FLOAT_SWITCH	compiles clean (mingw-w64 gcc+clang); validated by this PR's windows-msvc CI run
SceFiber	PS Vita	PLATFORM_VITA	sceKernelAllocMemBlock (no guard page — Vita has no per-subrange protection API; documented)	compiles clean against real VitaSDK (_sceFiberInitializeImpl, <psp2/fiber.h>); untested on hardware
pthread	Linux (testing only)	-DPS2X_FIBER_PTHREAD=ON	mmap guard + pthread_attr_setstack	exists so TSan can see the scheduler (TSan can't instrument swapcontext); full suite green

-DPS2X_SANITIZE=thread|address (top-level) wires sanitizers through every target; TSan requires the pthread backend (enforced at configure time).

Testing

• ~6k lines of scheduler tests: wakeup-window protocol (arm/park/wake races driven deterministically via atomics handshakes, not sleeps), suspend/resume gating, tid reuse + stale wakeup rejection (including a join ABA regression test), terminate-during-park-window, priority scheduling/rotation/join floors, shutdown/teardown ordering, borrowed-worker paths, EventFlag modes, alarm/vsync integration.
• Full suite passes on the ucontext and pthread backends (multiple consecutive runs, deterministic).
• ThreadSanitizer (pthread backend): clean except two documented intentional reports (the guest-visible vsync flag/tick words are deliberately plain memory — memory-mapped-register emulation, as on hardware) and one pre-existing upstream report in updateGsCsrFieldForVSync (untouched by this PR; fixed separately in fix(runtime): make GS CSR atomic to fix vsync-worker/guest data race #145).
• AddressSanitizer + LeakSanitizer: zero reports.
• Bugs found and fixed while hardening: lost ReleaseWaitThread during Mesa retries, alarm-worker use-after-free vs CancelAlarm, non-fiber WaitEventFlag/WaitForNextVSyncTick returning without waiting, INTC/DMAC enable-mask races, a terminate/suspend wake dropped in the publish→park window, and a tid-recycling ABA in join_fiber.

Guest-visible semantics notes (for review against ps2sdk)

• Semaphore syscalls keep fix: semaphore syscalls return sid on success instead of KE_OK #136's return-value contract (sid on success).
• GetThreadId from the primary host thread returns 1 (upstream reserved tid 1 for main; the runtime claims it at construction).
• Suspended waiters: a SetEventFlag/SignalSema arriving while suspendCount > 0 leaves the thread THS_WAITSUSPEND; it completes the wait after ResumeThread. (Divergence from literal EE kernel timing, where the wake transitions WAITSUSPEND→SUSPEND immediately; the end state after resume is identical. Called out in the adapted kernel test.)
• Exit handlers may block: fibers have an Exiting state so a guest exit handler can make blocking syscalls without the scheduler freeing the live trampoline frame.

Vita status (honest labeling)

The SceFiber backend is compile-verified against the real VitaSDK toolchain (reproducible container harness; zero warnings) but has not run on hardware — I don't have a Vita to test with. The previous "vita is broken" state is resolved at the compile level: <psp2/fiber.h>, _sceFiberInitializeImpl (vita-headers exposes no sceFiberInitialize wrapper), and sceKernelAllocMemBlock stacks (VitaSDK has no <sys/mman.h>). If someone with hardware can boot-test, I'll support fixes.

[smmathews]

leaving this in draft while I give it a more thorough review, but overall I think it should be a more robust solution than #134

[smmathews]

still leaving in draft, I don't like how vita is broken and I can't test it. working on it.

[smmathews]

Updated and out of draft. Since the last revision:

• Merged upstream main through Feature/mpeg decoder #120 (conflict-free now; resolutions summarized in the merge commit — fix: semaphore syscalls return sid on success instead of KE_OK #136's sid-on-success semantics preserved throughout, including in the older scheduler tests).
• Windows: new Win32 Fibers backend; the windows-msvc CI leg builds and passes the full suite.
• Vita: the SceFiber backend now compiles against a real VitaSDK (<psp2/fiber.h>, _sceFiberInitializeImpl, sceKernelAllocMemBlock stacks). Untested on hardware — honestly labeled in the description.
• Test suite: 351/351, deterministic across repeated runs; TSan/ASan/LSan status documented in the description. Several real races found and fixed along the way (details in the description and commit messages).

Happy to re-merge over #140 if that lands first.

[smmathews]

Re-merged after #140/#141 landed — branch is again a single commit (93be65c) on current main, conflict-free, all CI green.

Notes on the resolution:

• feat: add SSE2 support and fix SSE4.1 paths for PADDSW and PSUBSW instructions #141 (SSE2/PADDSW/PSUBSW) has zero overlap with this branch.
• Refactor runtime for move speed and better code style #140's runtime-side changes merged cleanly with the scheduler. The new GuestBranchKind / MissingFunctionPolicy / dispatchGuestBranch machinery and the dense recompiled-function table are all kept as-is; the scheduler tests register through the new reset_ps2_test_function_table() BeforeEach hook.
• Refactor runtime for move speed and better code style #140 rewrote the guest-execution mutex machinery (enterGuestExecution et al.), which this branch deletes outright — under the N=1 scheduler only one fiber ever executes guest code, so the scope guards remain no-ops and the mutex block stays removed. Nothing else in the merged tree references it.
• Verified the two cooperative-yield surfaces survived the emitter split: recompiled back-edges still emit shouldPreemptGuestExecution() (control_flow_emitter.cpp), and the kernel-syscall ps2sched::maybe_yield() sites are untouched.

Local validation: 354/354 (incl. the new dispatchGuestBranch tests) on ucontext and pthread backends, TSan and ASan/LSan clean apart from the documented intentional MMIO-polling reports.

One observation from the merge, independent of this PR: #140 made lookupFunction exact-entry-only, so a mid-function PC now resolves to the missing-function stub instead of its owning function. That's inert for this branch (kernel PC loops step entry→entry and are hasFunction-guarded; interrupted fibers resume via context switch, not re-lookup), but flagging it in case other callers relied on mid-function resolution.

[smmathews]

I have this running well with my dq8 recomp project, and I'm feeling confident in the direction, but at net 12k lines of code, I don't trust my LLMs judgement on it. Gonna spend some time better understanding it myself.

[TH3BACKLOG]

Downstream findings from integrating this scheduler (SDBZ Recomp fork)

Tracking a black-screen boot blocker in our fork (SDBZ Recomp) and traced it far enough that it may be relevant to this PR's fiber scheduler, though the link is not yet proven — flagging now so it's visible while we finish verifying.

Confirmed (static + log evidence):

• GameInit (fn_421B70) returns early with $v0 still holding a prior callee's (MemSysInit) result, rather than reaching GameState_Init — confirmed via bisect-stub logging ([BOOT421B70] EXIT dword_63FDF8_after=0x0 v0=0x8028), and the returned value is non-deterministic across runs (0x8028 vs 0x8018 in different captures), which points to leftover-register state from a bailed-out call rather than a designed error path.
• This is consistent with a generated dispatch-guard or cooperative-yield early return (if (ctx->pc != <expected>) return; / the shouldPreemptGuestExecution() back-edge check) firing inside GameInit's call chain and never being resumed — dispatchLoop()'s lookupFunction() alias-owner fallback was verified to route any resumption back through the same registered override/bisect stub, so an absent second log line is a valid (not just suggestive) signal that resumption never happened.

Not yet confirmed — this is the part that needs your eyes or more testing on our end:

• We have not set a live breakpoint at the actual yield/back-edge site (~0x110f68) to directly observe whether dispatchLoop ever re-enters it, and we have not inspected g_host_token_waiters at runtime to see if it's stuck non-zero. The "lost-wakeup in the N=1 scheduler" explanation is currently an inference from log absence + static reading of ps2_scheduler.cpp's yield_point()/AsyncGuestScope, not a proven mechanism.
• We integrated this PR's scheduler into our main tree and separately logged 3 test-suite regressions (2 VIF1/DMAC-dispatch-on-store logic failures, 2 TerminateThread/WaitSema failures) plus a reproducible STATUS_STACK_OVERFLOW crash right after a semaphore-legacy-layout test — these land in the same Interrupt.cpp/Sync.cpp areas this PR touches, so it's plausible (not confirmed) that our GameInit stall and that crash share a root cause in the fiber resumption path.

Next step on our side: breakpoint at the yield back-edge + live read of the guest-token waiter state to confirm/refute non-resumption directly, rather than relying on log-absence inference. Will follow up here with results — posting now in case the AsyncGuestScope/yield-point resumption path rings a bell for anything already known to be incomplete in this PR.

---

Update: pinned the exact back-edge, and the ~0x110f68 guess above was right on the money.

Re-ran with our existing per-callsite bisect instrumentation (logs ctx->pc immediately after each inner call returns, compared against the expected fallthrough) and got a clean chain:

[BISECT18D680]  #1 pc_after=0x422584 expect=0x422584          (MemSysInit's own call — OK)
[BISECT18D6D8]  #1 pc_after=0x422590 expect=0x422590          (OK)
[BISECT18D680]  #2 pc_after=0x110e84 expect=0x422584          (nested call from inside CMemory_Init — stub's hardcoded "expect" doesn't apply here, not a real mismatch)
[BISECT110E60]  #1 pc_after=0x110f68 expect=0x4225b4 v0=0x8018   <-- real signal
[BOOT422570]    #1 v0=0x00008018                               (MemSysInit propagates the stale v0)
[BOOT421B70]    #1 EXIT v0=0x00008018 pc=0x00110f68 dword_63FDF8_after=0x0

0x110f68 isn't a return address at all — it's loc_110F68, the back-edge label of a partition-flag copy loop inside CMemory_Init (fn_110e60, called from MemSysInit). Pulled the generated source directly:

// fn_110E60_0x110e60.cpp, ~line 681
if (branch_taken_0x111138) {
    ctx->pc = 0x110F68u;
    if (runtime->shouldPreemptGuestExecution()) {
        return;
    }
    goto label_110f68;
}

So this is confirmed to be the exact mechanism, not just an inference from log absence: shouldPreemptGuestExecution() returns true mid-loop, the function returns with ctx->pc parked on its own back-edge, and that stale exit unwinds all the way up through MemSysInit → GameInit with the loop's last $v0 still attached — GameState_Init never gets called, dword_63FDF8 never gets its pointer, game stays on the black-screen/loading-placeholder render path.

Still open on our end: haven't yet read g_host_token_waiters live at this exact freeze point to see whether anything is actually still queued to resume it, or whether this is a case where nothing is scheduled to resume a fiber that yielded from inside CMemory_Init rather than at a designed suspension point. Given this is a different call site than the smstt/smrdy WaitSema stall we'd flagged before, it does strengthen the read that these are two symptoms of the same gap (a preempted-mid-function guest call with no generic resume path) rather than something specific to WaitSema. Will keep updating here as we narrow it down further.

[smmathews]

join_fiber priority floor should be target->priority, not target->priority + 1

In ps2xRuntime/src/lib/ps2_scheduler.cpp, join_fiber() lowers the joining fiber to a priority floor while it waits for the target to finish:

const int floor = t->priority + 1;

The intent — make the target run before the joiner's next poll — is correct, but the + 1 (one level below the target, since a higher priority number is a lower scheduling priority) turns the floor into a starvation deadlock.

Failure mode. Once the target fiber finishes and leaves the run queue, the joiner is parked one full level below the target's original priority. If any other runnable fiber sits at that original priority — e.g. a worker that re-wakes every frame — the scheduler keeps servicing that fiber and never returns to the joiner. The joiner therefore never observes the target's Finished state, never restores its saved priority, and TerminateThread hangs forever.

Concretely, in DQ8 movie teardown: TerminateThread of the video-pump thread (guest priority 40) never returns, because a movie idle-spinner thread at the same priority 40 — woken once per frame — permanently starves the prio-41 joiner, even though the pump itself unwinds correctly via the terminateRequested check in yield_point.

Fix — floor at the target's priority, not one below:

const int floor = t->priority;  // equal, not +1

The run queue is stable FIFO within a priority level, so a joiner that yields re-enqueues behind the still-queued target at the same level. The target is therefore still guaranteed to run before the joiner's next poll — the only property the floor needs to guarantee — but no equal-priority sibling can indefinitely starve the joiner after the target exits. Only this one line changes; the save/restore bookkeeping around it is unaffected.

Suggested regression test. Three fibers — A (prio 10) loops forever waking itself and never blocks, B (prio 10), and C (prio 20) which calls TerminateThread(B). Assert that C's TerminateThread returns (B unwinds) within a bounded number of scheduler cycles while A remains runnable throughout. With the + 1 floor this times out; with the equal floor it completes.

[smmathews]

Downstream findings from integrating this scheduler (SDBZ Recomp fork)

Tracking a black-screen boot blocker in our fork (SDBZ Recomp) and traced it far enough that it may be relevant to this PR's fiber scheduler, though the link is not yet proven — flagging now so it's visible while we finish verifying.

Confirmed (static + log evidence):

* `GameInit` (`fn_421B70`) returns early with `$v0` still holding a prior callee's (`MemSysInit`) result, rather than reaching `GameState_Init` — confirmed via bisect-stub logging (`[BOOT421B70] EXIT dword_63FDF8_after=0x0 v0=0x8028`), and the returned value is non-deterministic across runs (`0x8028` vs `0x8018` in different captures), which points to leftover-register state from a bailed-out call rather than a designed error path.

* This is consistent with a generated dispatch-guard or cooperative-yield early return (`if (ctx->pc != <expected>) return;` / the `shouldPreemptGuestExecution()` back-edge check) firing inside `GameInit`'s call chain and never being resumed — `dispatchLoop()`'s `lookupFunction()` alias-owner fallback was verified to route any resumption back through the same registered override/bisect stub, so an absent second log line is a valid (not just suggestive) signal that resumption never happened.

Not yet confirmed — this is the part that needs your eyes or more testing on our end:

* We have **not** set a live breakpoint at the actual yield/back-edge site (`~0x110f68`) to directly observe whether `dispatchLoop` ever re-enters it, and we have **not** inspected `g_host_token_waiters` at runtime to see if it's stuck non-zero. The "lost-wakeup in the N=1 scheduler" explanation is currently an inference from log absence + static reading of `ps2_scheduler.cpp`'s `yield_point()`/`AsyncGuestScope`, not a proven mechanism.

* We integrated this PR's scheduler into our main tree and separately logged 3 test-suite regressions (2 VIF1/DMAC-dispatch-on-store logic failures, 2 TerminateThread/WaitSema failures) plus a reproducible `STATUS_STACK_OVERFLOW` crash right after a semaphore-legacy-layout test — these land in the same `Interrupt.cpp`/`Sync.cpp` areas this PR touches, so it's plausible (not confirmed) that our `GameInit` stall and that crash share a root cause in the fiber resumption path.

Next step on our side: breakpoint at the yield back-edge + live read of the guest-token waiter state to confirm/refute non-resumption directly, rather than relying on log-absence inference. Will follow up here with results — posting now in case the AsyncGuestScope/yield-point resumption path rings a bell for anything already known to be incomplete in this PR.

Update: pinned the exact back-edge, and the ~0x110f68 guess above was right on the money.

Re-ran with our existing per-callsite bisect instrumentation (logs ctx->pc immediately after each inner call returns, compared against the expected fallthrough) and got a clean chain:

[BISECT18D680]  #1 pc_after=0x422584 expect=0x422584          (MemSysInit's own call — OK)
[BISECT18D6D8]  #1 pc_after=0x422590 expect=0x422590          (OK)
[BISECT18D680]  #2 pc_after=0x110e84 expect=0x422584          (nested call from inside CMemory_Init — stub's hardcoded "expect" doesn't apply here, not a real mismatch)
[BISECT110E60]  #1 pc_after=0x110f68 expect=0x4225b4 v0=0x8018   <-- real signal
[BOOT422570]    #1 v0=0x00008018                               (MemSysInit propagates the stale v0)
[BOOT421B70]    #1 EXIT v0=0x00008018 pc=0x00110f68 dword_63FDF8_after=0x0

0x110f68 isn't a return address at all — it's loc_110F68, the back-edge label of a partition-flag copy loop inside CMemory_Init (fn_110e60, called from MemSysInit). Pulled the generated source directly:

// fn_110E60_0x110e60.cpp, ~line 681
if (branch_taken_0x111138) {
    ctx->pc = 0x110F68u;
    if (runtime->shouldPreemptGuestExecution()) {
        return;
    }
    goto label_110f68;
}

So this is confirmed to be the exact mechanism, not just an inference from log absence: shouldPreemptGuestExecution() returns true mid-loop, the function returns with ctx->pc parked on its own back-edge, and that stale exit unwinds all the way up through MemSysInit → GameInit with the loop's last $v0 still attached — GameState_Init never gets called, dword_63FDF8 never gets its pointer, game stays on the black-screen/loading-placeholder render path.

Still open on our end: haven't yet read g_host_token_waiters live at this exact freeze point to see whether anything is actually still queued to resume it, or whether this is a case where nothing is scheduled to resume a fiber that yielded from inside CMemory_Init rather than at a designed suspension point. Given this is a different call site than the smstt/smrdy WaitSema stall we'd flagged before, it does strengthen the read that these are two symptoms of the same gap (a preempted-mid-function guest call with no generic resume path) rather than something specific to WaitSema. Will keep updating here as we narrow it down further.

Thanks for the detailed trace. I don't think the mechanism you've flagged is a flaw in the preemption protocol, the bare return at the back-edge is only half of it. Every recompiled JAL/JALR call site (upstream #73, 8584c06, predates this PR) emits a caller-side sentinel: it captures __entryPc, and after the call does if (ctx->pc != fallthroughPc) return;. So when CMemory_Init preempts with ctx->pc parked at 0x110f68, MemSysInit's sentinel sees the mismatch and returns too, your own trace confirms this (both MemSysInit and GameInit exit with pc=0x110f68). The unwind reaching dispatchLoop is by design; no register state is lost since everything's in ctx->r[]/RDRAM.

The reason it doesn't resume is downstream: dispatchLoop resolves ctx->pc=0x110f68 via lookupFunction and re-enters fn_110e60 at its case 0x110f68: label, which only works if 0x110f68 is registered in the address table. I hit this exact symptom booting DQ8 on this scheduler: preemption correctly unwound to an internal label of a manually-spliced function, but only the entry point was registered, not the internal resume labels, so lookupFunction missed, the recover-pc fallback fired, and boot corrupted. Fix was registration completeness, not a protocol change. I'd check whether 0x110f68 (and the other internal labels of fn_110e60, plus any manually-spliced/packed-entry functions in your tree) are in your lookupFunction table at the resume point, that's likely your smstt/smrdy WaitSema stall too, same root cause, different call site, as you suspected.