feat(mysql): store physical backups in Backblaze B2

[viasnake]

Problem

Why

The MySQL platform needs off-host physical backup storage with remote restore validation. The final platform uses a private Backblaze B2 bucket while retaining prepared XtraBackup data, role-aware scheduling, shared locking, and restore validation.

Changes

Architecture

The backup path is SECONDARY -> local XtraBackup staging -> xtrabackup --prepare -> rclone over HTTPS -> Backblaze B2. Restore validation always downloads from B2 into isolated local scratch before starting a network-disabled MySQL instance.

Why rclone

rclone provides a native B2 backend, Application Key authentication, checksum-aware upload and download, and direct transfer of prepared XtraBackup data to object storage.

B2 credential model

Inventory contains only the bucket, mysql-shared prefix, and rclone remote name. The Application Key ID and Application Key are required secret inputs rendered to /etc/rclone/mysql-backup.conf as root:root mode 0600 under no_log. The host role validates but does not create the bucket or key.

Backup flow

The current healthy SECONDARY creates and prepares a local physical backup, uploads it with rclone copy, verifies it with rclone check, and writes complete.json last. Only then does the job archive closed binlogs, write the run manifest, update status, and remove local staging.

Binlog flow

Closed binlogs use mysql-shared/binlog/<node>/<server-uuid>/. Existing objects with the same SHA-1 and size are skipped; the same name with different content fails. Each run writes its GTID and checksum manifest under manifests/<backup-run-id>.json.

Restore flow

The restore test selects the latest completed UTC run or an explicit mysql_restore_backup_id, validates marker identity, downloads with rclone, checks remote object names and checksums, validates prepared metadata, then runs the isolated MySQL checks and cleanup.

Impact

MySQL convergence requires an existing private B2 bucket plus a dedicated bucket- and prefix-restricted Application Key. The committed bucket value remains empty until the operator supplies the real name.

Security

Credentials are absent from command lines and logs. rclone v1.75.0 is installed from pinned official archives with architecture-specific SHA256 checksums. Backup code never uses remote synchronization or deletion commands. Remote identities, completion metadata, local staging, downloaded object paths, and restore process ownership fail closed. A B2 outage fails backup monitoring without coupling B2 to MySQL, ReplicaSet, or Router availability.

Validation

• PYTHONDONTWRITEBYTECODE=1 mise run check: 88 tests passed; seven playbook syntax checks passed; ansible-lint reported zero failures and zero warnings.
• git diff --check
• .venv/bin/ansible-inventory --graph
• Official rclone v1.75.0 local-backend probe: empty-prefix preflight, upload/check/marker, latest and explicit selection, remote download/integrity validation, binlog idempotence, and mismatched-binlog rejection passed.

Runtime acceptance still blocked

Implemented and statically validated. No production apply, live B2 upload, live B2 download, production backup, or production restore was performed. Runtime acceptance remains pending until an operator creates the private bucket and dedicated Application Key, supplies the bucket and secrets, and runs backup and restore validation on the MySQL nodes. Retention, Lifecycle Rules, and Object Lock remain intentionally deferred.