Follow-up to #101 / PR #105. PR #105 closed the empty-voter fallback (a node with no established unique-voter set no longer seals over the full membership). A related empty-map hole remains open.
The hole
The synchronous seal_or_queue_unique self-seals when unique_majority() == 1. unique_voter_group() currently returns {self} whenever the voter set is empty and unique_fanout_group().len() <= 1 — and an empty/unsynced local partition map also satisfies that (all_nodes() is empty ⇒ fan-out is {self}). So a node right after a restart, before update_partition_map installs the raft map (the event loop calls become_primary before installing the map), treats itself as single-node and self-seals every partition it is named primary of, skipping merge_for_seal — then, once it adopts the voter set, it can serve reserves against a store that is missing reservations it did not receive while down.
Scope: narrow, restart-only oversell. Committed claims are backstopped by the record-driven reconciler; the residual is the uncommitted window. Reproduced by a unit test (empty_map_node_does_not_self_seal in the reverted work): on today's code the node self-seals (unique_sealed == Some(epoch)); it should fail closed (None).
Why the straightforward fix isn't enough
The obvious guard — in the empty-voter branch of unique_voter_group(), self-serve only when all_nodes() == {self} (a genuine single-node cluster), else fail closed — closes the hole and passes the full unit + integration suite and the V7 model. But a real-cluster smoke caught a regression the unit tests and an analytical quorum both missed: removing the fast (buggy) self-seal made a fresh 3-node cluster take ~60–80s to fully accept unique creates (creates fail 503 not-yet-durable, recovering gradually per partition). Root cause: a partition can only seal via the proper path once its primary has a non-empty voter set, and on a fresh cluster the leader founds voters = all_nodes() — which is just {leader} before the rebalance distributes partitions — then paces promotion up to the full set over ~20s, on top of formation epoch churn.
That trade (a narrow restart-only oversell for a ~60–80s degraded-serving window on every bootstrap) is why #4 was reverted from PR #105 rather than shipped.
Proposed fix
- Fail-closed guard: distinguish a genuine single-node cluster (
all_nodes() == {self}) from an unsynced empty map (all_nodes() == []) in unique_voter_group()'s empty-voter branch; the latter fails closed.
- Bootstrap founding speed-up: so the guard doesn't slow bootstrap, safely found
voters = full membership at once on a fresh cluster (safe — no reservations exist yet — matching the V5/V6 "founding takes the whole membership at once" rule) instead of founding {leader} and pacing up. Consider deferring the founding until membership/partition assignment has settled, or founding from the raft membership rather than all_nodes().
Acceptance
- Empty-map node fails closed (unit test, before/after).
- Real-cluster 3-node bootstrap accepts unique creates within a few seconds (no ~60–80s window).
- TLA: extend
specs/ClusterUniqueReconfigV7.tla (or a V8) to cover the founding/bootstrap transition, not just the abstract "empty ⇒ seal disabled".
- Real-cluster smoke required (this is exactly the class of regression unit tests + analytical review missed).
Documented in docs/design/cluster-unique-hardening.md under "Open follow-up — empty-MAP sync self-seal".
Follow-up to #101 / PR #105. PR #105 closed the empty-voter fallback (a node with no established unique-voter set no longer seals over the full membership). A related empty-map hole remains open.
The hole
The synchronous
seal_or_queue_uniqueself-seals whenunique_majority() == 1.unique_voter_group()currently returns{self}whenever the voter set is empty andunique_fanout_group().len() <= 1— and an empty/unsynced local partition map also satisfies that (all_nodes()is empty ⇒ fan-out is{self}). So a node right after a restart, beforeupdate_partition_mapinstalls the raft map (the event loop callsbecome_primarybefore installing the map), treats itself as single-node and self-seals every partition it is named primary of, skippingmerge_for_seal— then, once it adopts the voter set, it can serve reserves against a store that is missing reservations it did not receive while down.Scope: narrow, restart-only oversell. Committed claims are backstopped by the record-driven reconciler; the residual is the uncommitted window. Reproduced by a unit test (
empty_map_node_does_not_self_sealin the reverted work): on today's code the node self-seals (unique_sealed == Some(epoch)); it should fail closed (None).Why the straightforward fix isn't enough
The obvious guard — in the empty-voter branch of
unique_voter_group(), self-serve only whenall_nodes() == {self}(a genuine single-node cluster), else fail closed — closes the hole and passes the full unit + integration suite and the V7 model. But a real-cluster smoke caught a regression the unit tests and an analytical quorum both missed: removing the fast (buggy) self-seal made a fresh 3-node cluster take ~60–80s to fully accept unique creates (creates fail503 not-yet-durable, recovering gradually per partition). Root cause: a partition can only seal via the proper path once its primary has a non-empty voter set, and on a fresh cluster the leader foundsvoters = all_nodes()— which is just{leader}before the rebalance distributes partitions — then paces promotion up to the full set over ~20s, on top of formation epoch churn.That trade (a narrow restart-only oversell for a ~60–80s degraded-serving window on every bootstrap) is why #4 was reverted from PR #105 rather than shipped.
Proposed fix
all_nodes() == {self}) from an unsynced empty map (all_nodes() == []) inunique_voter_group()'s empty-voter branch; the latter fails closed.voters = full membershipat once on a fresh cluster (safe — no reservations exist yet — matching the V5/V6 "founding takes the whole membership at once" rule) instead of founding{leader}and pacing up. Consider deferring the founding until membership/partition assignment has settled, or founding from the raft membership rather thanall_nodes().Acceptance
specs/ClusterUniqueReconfigV7.tla(or a V8) to cover the founding/bootstrap transition, not just the abstract "empty ⇒ seal disabled".Documented in
docs/design/cluster-unique-hardening.mdunder "Open follow-up — empty-MAP sync self-seal".