Skip to content

fix(mysql): harden Router bootstrap and topology operations - #5

Merged
viasnake merged 3 commits into
masterfrom
codex/mysql-runtime-safety
Aug 30, 2026
Merged

fix(mysql): harden Router bootstrap and topology operations#5
viasnake merged 3 commits into
masterfrom
codex/mysql-runtime-safety

Conversation

@viasnake

@viasnake viasnake commented Aug 29, 2026

Copy link
Copy Markdown
Member

Problem

  • New Router bootstrap depended on the operator-facing runtime primary DNS alias instead of stable ReplicaSet member identities.
  • Router clients used the full ReplicaSet administrator for bootstrap.
  • Topology operations only probed the backup/restore lock and released it before MySQL Shell ran, leaving a TOCTOU race with backup and restore.
  • The physical-backup systemd check treated unknown unit state as idle, and the standard application privilege profile still included schema DDL.
  • Suspending backup timers during topology operations could leave them stopped after controller loss.

Changes

  • New Router bootstrap discovers an available ReplicaSet member from the stable mysql-shared01 and mysql-shared02 identities derived from svc_mysql. A TLS MySQL session probe selects the first available member; Router metadata then resolves the current primary.
  • Add the TLS-required mysql_router_bootstrap account with the MySQL Router 8.4 minimum bootstrap grants, restrict it to mysql_router_clients, separate its secret, restrict the ReplicaSet administrator to database-node sources, and remove obsolete administrator accounts for Router client sources.
  • Replace the probe-only guard with a bounded systemd lock holder that owns /run/lock/mysql-physical-backup.lock for the complete topology operation. Planned switchover holds it on both database nodes; emergency failover holds it on the authorized target.
  • Keep backup timers enabled and active during topology operations. The scheduled service passes --skip-if-lock-busy and returns changed=false with reason=shared lock busy; explicit backup and restore-test operations continue to fail on lock contention.
  • Keep the lock through ReplicaSet mutation, role-DNS update, and Router validation, and release it from always cleanup. Raise RuntimeMaxSec from 900 to 3600 seconds and reject lower runtime values before lock acquisition.
  • Accept the backup service only when LoadState=loaded and ActiveState=inactive; active, transitional, failed, unknown, missing, and query-failure states are denied.
  • Split tenant privileges into application, migration, and read_only. Runtime application accounts no longer receive DDL; migration accounts exist only when tenants declare them.
  • Document stable Router discovery, account boundaries, role-DNS scope, shared-lock behavior, tenant account separation, and the physical-backup-dependent off-host recovery-point limit.
  • Add regression coverage for candidate selection, dedicated bootstrap grants, both directions of lock exclusion, scheduled skip versus explicit-operation failure, controller-loss-safe timer behavior, fail-closed systemd states, cleanup structure, and tenant privilege separation.

Impact

  • Existing bootstrapped Router metadata and keyrings are unchanged. New Router bootstrap no longer depends on role DNS.
  • Deployment now requires mysql_router_bootstrap_password. Normal MySQL platform convergence must install the dedicated account and topology lock unit before running topology operation playbooks.
  • Backup and restore cannot start while a topology operation owns the shared lock, and topology mutation cannot start while backup or restore owns it. Backup timer state is never changed by topology playbooks.
  • Closed binlog archival remains tied to successful physical-backup jobs; this is not continuous PITR shipping or an RPO=0 design.
  • No production ReplicaSet, Router, DNS, backup, restore, account, timer, switchover, or failover mutation was performed. In a read-only host check, mysql-shared01 was reachable but the not-yet-deployed backup service, timer, and lock-holder units reported LoadState=not-found, so the new preflight rejected the operation before MySQL Shell. mysql-shared02 remains unreachable with No route to host.
  • mise run check passes 71 tests, seven playbook syntax checks, and production-profile lint with zero failures or warnings. git diff --check and ansible-inventory --graph also pass.

@viasnake
viasnake merged commit 8ebb3b1 into master Aug 30, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant