Add sequence diagrams for monitoring, FSM, and fault-tolerance docs#1160
Merged
Conversation
…ance docs architecture.rst: the node_active() polling and health-check protocols between a node's keeper and the monitor, and when a node counts as unhealthy. failover-state-machine.rst: the "birth of a cluster" walkthrough (two freshly registered nodes converging to a primary/secondary pair) that was previously prose-only. fault-tolerance.rst (previously had no diagrams at all): - primary self-fencing to demote_timeout after losing both the monitor and the secondary (network_partition_timeout) - asymmetric partition: the monitor's 90s safety wait before promoting after assigning the secondary stop_replication - falling back to asynchronous replication when the secondary goes unhealthy, and resynchronizing back to synchronous The two existing sequence diagrams (seq-normal-failover, seq-stuck-failover) stay on architecture-multi-standby.rst, where they illustrate number_sync_standbys and its failure-mode consequences.
Remove the "Citus Data team at Microsoft generously maintains" framing, which is no longer accurate. State plainly that the project is now maintained on volunteers' free time with no financial backing, and link to oss.theartofpostgresql.com for professional support or sponsoring.
…t state Add two explicit broken-connection arrows in the "network partition" section: the attempted node_active() call that gets no response, and the WAL stream to the secondary going dark. Previously this was only a text note asserting both were unreachable; now the diagram shows why the primary concludes it might be isolated.
Distinct from the primary's own self-fence diagram: this one is from the monitor's side, which doesn't know why the primary stopped responding, only that it did. Shows the health-check retries failing, the secondary being promoted through prepare_promotion (with the 30s WAL-delta timeout noted), and the old primary's goal being fenced to demoted regardless of whether it's actually reachable -- cross-referencing the self-fence diagram for what happens if it's still alive.
Every "goal: X" arrow was an abstraction hiding how the protocol actually works: the monitor never pushes a goal state, a node only ever learns its goal as the return value of its own node_active(reported=...) call. Replace every one of those abstracted pushes with the real exchange -- and where a goal change is decided while processing a *different* node's call (e.g. B converging triggers A's promotion), show it landing on that node's own next poll, not as an immediate cross-node push. Also make explicit which node can actually reach the monitor at each point: - seq-monitor-cant-reach-primary: primary's node_active attempts fail alongside the health-check failures already shown; secondary keeps polling and getting responses throughout, visibly unaffected, until the monitor uses its next poll to hand it the failover goal. - seq-primary-self-fence: same contrast -- secondary's node_active keeps succeeding while primary's own attempt goes unanswered. - seq-asymmetric-partition: only the primary-monitor link is shown broken; secondary's polling continues normally, which is how it receives stop_replication. - seq-secondary-unhealthy: secondary's failed attempt is shown explicitly, followed by primary's own next poll revealing wait_primary, and secondary's eventual recovery shown as real reported/goal exchanges rather than a bare "report:"/"goal:" shorthand. - seq-birth-of-a-cluster: register_node() kept distinct from node_active() polling; every FSM transition now shows the actual reported/goal pair, including the two places (A's promotion, B's catchingup assignment) where the goal is decided while processing the other node's call and only delivered on that node's own next poll. seq-monitoring-protocol.svg (architecture.rst) is unchanged -- it's already the generic node_active/goal_state protocol reference with no concrete scenario to trace state through.
…connects Answers what happens to the secondary once the primary demotes itself: in this full-isolation scenario the monitor's own health checks to the primary are failing too, on a shorter timer (~10s vs. the primary's 20s), so the monitor's own failover promotes the secondary before the primary even finishes self-fencing -- cross-referencing the "Monitor can't connect to Primary" diagram for the promotion mechanics rather than repeating them. Then shows the old primary reconnecting: it reports demote_timeout and gets back goal=demoted directly, because that goal was already recorded during the monitor's STONITH action -- a real demote_timeout -> demoted edge, so no guard logic is even needed for this path. Ties the self-fence and monitor-failover scenarios together into one coherent, consistent outcome instead of leaving them as two disconnected stories.
…iagram The compressed promotion sequence jumped straight from goal=prepare_promotion to reported=stop_replication, skipping the candidate's own confirmation of reaching prepare_promotion (and the goal=stop_replication response to it). Added that missing exchange back, matching the sibling diagram, and replaced the vague "same sequence as..." note with a short explanation of what stop_replication actually gates on. Also extends the story past "old primary stays fenced": once the new primary has confirmed wait_primary, demoted -> catchingup is a valid FSM edge (group_state_machine.c:909), so the diagram now shows the old primary reconnecting further, rewinding/rebasing, and rejoining as a secondary -- full role swap, not just a fenced dead end. The primary's lifeline switches back from "dead" to normal once it successfully reconnects.
dimitri
added a commit
that referenced
this pull request
Jul 26, 2026
…narios The Unhealthy Nodes section had prose but no diagram for two of its four bullets. Round these out to match the other two (Primary can't connect to Monitor, Monitor can't connect to Secondary), already illustrated in #1160. - Primary node is monitored unhealthy: new diagram with explicit node_active traffic, illustrating the actual dual-signal unhealthy determination (health check + node_active/pgIsRunning) -- plus a new clarifying paragraph on that combination, since neither signal alone is sufficient and the prose didn't previously say so. - Monitor node has failed: new diagram (adds an Application actor) showing primary/secondary roles and read-write access are entirely unaffected while the monitor is down; only automated failover isn't available. Also adds a paragraph on pg_auto_failover's single-failure design scope: losing the monitor alone isn't a problem, but a second node failing in that window is two out of three down at once, which has no automated repair. Split out of the timeline-fork-detection branch, which had bundled these in among unrelated feature commits -- this is pure documentation of existing, pre-existing monitor/keeper behavior, unrelated to that feature.
dimitri
added a commit
that referenced
this pull request
Jul 26, 2026
…narios (#1162) The Unhealthy Nodes section had prose but no diagram for two of its four bullets. Round these out to match the other two (Primary can't connect to Monitor, Monitor can't connect to Secondary), already illustrated in #1160. - Primary node is monitored unhealthy: new diagram with explicit node_active traffic, illustrating the actual dual-signal unhealthy determination (health check + node_active/pgIsRunning) -- plus a new clarifying paragraph on that combination, since neither signal alone is sufficient and the prose didn't previously say so. - Monitor node has failed: new diagram (adds an Application actor) showing primary/secondary roles and read-write access are entirely unaffected while the monitor is down; only automated failover isn't available. Also adds a paragraph on pg_auto_failover's single-failure design scope: losing the monitor alone isn't a problem, but a second node failing in that window is two out of three down at once, which has no automated repair. Split out of the timeline-fork-detection branch, which had bundled these in among unrelated feature commits -- this is pure documentation of existing, pre-existing monitor/keeper behavior, unrelated to that feature.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Diagrams
Six new tikz sequence diagrams, dispatched across three pages:
architecture.rst: thenode_active()polling and health-check protocols between a node's keeper and the monitor.failover-state-machine.rst: the "birth of a cluster" walkthrough (two freshly registered nodes converging to a primary/secondary pair), previously prose-only.fault-tolerance.rst(previously had no diagrams at all):demote_timeoutafter losing contact with both the monitor and the secondary — including the monitor's own parallel failover (shorter health-check timeout), and the old primary's full reintegration back to secondary once it reconnectsEvery exchange is drawn as the real protocol shape: a node calls
node_active(reported=X)and the monitor's response carriesgoal=Y. The monitor never pushes a goal state unprompted — where one node's report causes another node's goal to change (e.g. a candidate converging promotes the old primary's replacement), that new goal is only shown landing on the affected node's own next poll. Broken connections are drawn explicitly (arrow cut short, ×, "no response") rather than asserted in a text note, so it's visible at each step which node can actually reach the monitor.The two pre-existing sequence diagrams (
seq-normal-failover,seq-stuck-failover) are untouched and stay onarchitecture-multi-standby.rst, where they illustratenumber_sync_standbysand its failure-mode consequences.Project history note
Removes the "Citus Data team at Microsoft generously maintains" framing from the documentation landing page, which is no longer accurate. States plainly that the project is now maintained on volunteers' free time, and links to oss.theartofpostgresql.com for support/sponsoring.