Skip to content

cluster set_null: executing-node schema re-check for transient schema-sync-skew window #107

Description

@fabracht

Follow-up to the #94 comprehensive fix (PR forthcoming). The steady-state (converged) cluster path is closed: prepare_fk_side_effects fails the delete closed when a set_null (direct or cross-owned) would null a schema-typed field, and the constraint API rejects such definitions. This tracks a narrow transient residual an adversarial review surfaced.

The window

The runtime guard (reject_typed_set_null in crates/mqdb-cluster/src/cluster/node_controller/db_ops.rs) runs on the coordinating node and fails open when that node's schema is not yet synced (schema_get returns None). The executing primary for a RemoteSetNull handles it as a plain JsonDbOp::Update in handle_json_update_local (and execute_local_cascade_set_null) and does not re-check the schema. So during a node-join / snapshot-sync window — a coordinator that already holds the FK constraint but has not yet imported the broadcast schema — a RemoteSetNull can reach a remote primary that does hold the typed schema and write the null with no second check.

Mitigation already in place

DB_SCHEMA and DB_CONSTRAINT are broadcast in the same set (replication_ops.rs), and snapshots assert schema uniformity across nodes. A node lacking the schema very likely also lacks the FK constraint/reverse index, in which case it produces no set_null effect at all. So the window is narrow and not provably reachable — but schema and constraint are separate stores with no proven ordered/atomic arrival, and the referencing child records are partitioned data a primary can hold independently, so it is not provably empty.

Options

  1. Remote-leg defense-in-depth (partial): invoke a schema type-check on the executing node inside handle_json_update_local / execute_local_cascade_set_null when __mqdb_fk_expected is present, refusing to write null into a typed field. Closes the remote sub-case. Interacts with cascade-ack/retry semantics (a refused RemoteSetNull leaves the cascade un-acked → dangling reference rather than corruption) — needs care so it does not create a stuck-retry state. The local-leg (coordinator is the primary) shares the coordinator's schema view, so a same-node re-check does not help it.
  2. Full bootstrap-ordering fix: do not process FK-cascade deletes that produce set_null effects until the node's schema catalog is confirmed synced. This is the airtight closure and the same class as cluster unique: empty-map sync self-seal + bootstrap founding speed-up #106; it should carry a TLA model + real-cluster smoke.

Acceptance

  • A model (extend the cluster unique/fk specs or a new one) demonstrating the skew window and the chosen fix's safety.
  • Real-cluster smoke: a joining node coordinating an FK set_null delete before schema import does not corrupt a typed field, and does not wedge the cascade.
  • No regression to steady-state cluster delete throughput.

This is a restart/join-window residual only; steady-state clusters are unaffected.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions