fix(cluster): crash-rejoin re-declare barrier (Shape A) + cold-GRD fail-closed#58
Merged
Merged
Conversation
added 3 commits
July 15, 2026 18:16
Transplants the bounce-read/write P0 probes into a committed 2-node TAP encoding the fix contract (Rule 8.A): a crash-rejoined node's home-block read must be coherent or fail-closed (never a silent 0), its write must be coherent or fail-closed (never a silent divergent/lost commit). RED on main 6ef83c2: L4 (silent 0 read) + L5 (silent split write, node1=1 node0=64) fail; L3 diagnostic measures the membership split (both nodes see the rejoiner as member with online_join=off). No backend change — RED harness only. Fix (crash-rejoin re-declare barrier + cold-GRD watermark fail-closed) is gated on a boot-semantics decision reported to the gatekeeper. Spec: spec-5.16 (extended to online_join=off crash-rejoin).
…e A checkpoint)
Closes the fail-OPEN crash-rejoin hole for the SLOW-rejoin case and the
boot-to-decision race; the FAST-rejoin-within-deadband residual needs a
formation-level signal (escalated, NOT claimed done).
Mechanism (fail-closed, off path only; online_join=on untouched):
- New per-incarnation shmem flag ClusterGrdShared.offpath_boot_decided
(init 0). The phase gate (cluster_gcs_block.c) fences a self-home
block RECOVERING while the flag is 0, so from process start every
backend fail-closes self-home reads AND writes (both reach the gate
via cluster_pcm_lock_acquire_buffer) — zero cold-serve window.
- cluster_reconfig_offpath_rejoin_tick(): the online_join=off counterpart
of the joiner self-tick. already_running -> arm self-fence({self})
then demote self_join_admitted to 0 (8.A: never raised to 1 before the
fence is armed); bootstrap-at-initial or single node -> lift the barrier.
Measured (scratchpad bounce-read-p0/rejoin-state-measured.log):
- SLOW rejoin (survivor advanced epoch first): tick detects
already_running, arms the fence, self demotes to JOINING. WORKS.
- FAST rejoin (node restarts within the survivor dead-deadband, survivor
still at epoch INITIAL): already_running is false and
bootstrap_quorum_at_initial cannot distinguish a co-boot from a
long-running-but-not-yet-reconfigured peer -> mis-lifts -> t/404 L4/L5
still RED. This gap is pre-existing and affects online_join=on too
(existing tests only do slow rejoins). The durable prior-self-epoch
signal is erased by qvotec_poll_once (writes self slot before reading,
cluster_qvotec.c:578) — robust closure is a formation-level change.
Spec: spec-5.16 (extended to online_join=off crash-rejoin).
…ail-closed Closes the P0 that a crash-rejoined node with cluster.online_join=off cold-serves its home blocks from a wiped GRD: a silent stale READ and a silently-diverging WRITE that, after flush ordering, durably LOSES a peer's committed rows (diagnosed 2026-07-15, both reproduced minimally on main including the committed-write loss). Fix 1 — the off-path counterpart of the online joiner tick. A per- incarnation shmem flag ClusterGrdShared.offpath_boot_decided (init 0) makes the phase gate fence self-home blocks RECOVERING from process start (before any LMON tick), so reads AND writes fail-closed with zero cold- serve window. cluster_reconfig_offpath_rejoin_tick() then classifies the incarnation: - crash-rejoin -> arm the join fence + demote self_join_admitted to 0 (writes 53R60) and LEAVE offpath_boot_decided 0, so the boot barrier persists as the read fence for the incarnation. The epoch-keyed join fence is a no-op on a fast rejoin (this node is still at CLUSTER_EPOCH_INITIAL, so the monotonic-max fence epoch cannot rise); the epoch-independent boot barrier carries the read fence. - clean cold-bootstrap / single node -> set the flag, serve normally. The crash-rejoin signal is the prior-incarnation voting-disk self-slot's ALIVE bit (守门裁决 07-15: ALIVE, not epoch — a fast rejoin leaves BOTH sides at INITIAL, so an epoch test is blind; a clean shutdown clears ALIVE via qvotec_clear_self_alive_on_clean_shutdown while a crash leaves it set). qvotec's startup ghost-detect latches it into QvotecShmem.prior_unclean_death (offset 72; the convert-queue lane owns 64..71 — StaticAssert-pinned, this lane rebases after queue merges). Fix 2 (defense in depth) — cluster_gcs_cold_grd_watermark_verdict(): an InvalidScn master watermark under an active self-fence FAILs CLOSED instead of SKIPping (the wiped watermark could mask a stale block), except for a genuine extension block (never cross-node written). Pure, header-only, unit-tested truth table; wired into fallback_verify_refresh. Approved semantic consequence: a full-outage crash co-boot fences every node (53R60) — no silent auto-formation; the cluster waits for online admission (spec-5.22 cold-formation follow-up) or a clean restart. A CLEAN full shutdown clears ALIVE and co-boots normally. Tests: t/404 (crash-rejoin read/write/boot-race/full-outage/clean-restart legs, all GREEN), fold unit truth table, t/249 grd_recovery roster 31->32 (offpath_crash_rejoin_fenced dump key). Gates: unit clean 178/178, t/017+401+402+404 PASS, cluster_regress 13/13, PG 219/219, format/headers/scn-cmp/ges-mode/clog all 0. (t/249's cooperative-rebind timeout legs L6/L3/L13 flake under mac load — confirmed pre-existing on a binary without these changes; the roster ripple passes.) Known-scoped-out (守门裁决): a CLEAN fast-restart of a cluster member (without clean-leave) shares the hazard but is out of this fix's scope (the ALIVE bit is cleared on clean shutdown); clean departures should run clean-leave flush/handoff. Spec: spec-5.16 (extended to online_join=off crash-rejoin); Rule 8.A.
sqlrush
force-pushed
the
fix/s3-crash-rejoin-redeclare-barrier
branch
from
July 15, 2026 13:15
de3a4e1 to
32edefe
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Test plan
Spec: spec-5.16 (extended), spec-5.22 follow-up registered