Support managed blue/green AKS Flex Node agent binary upgrades

[bcho]

Summary

Add managed, rollback-safe upgrades for the running aks-flex-node binary through the Machina AgentUpgrade MachineOperation.

AKS Flex Node currently supports NodeReboot and AgentReset, but explicitly rejects AgentUpgrade:

machinav1alpha3.OperationAgentUpgrade: handlers.unsupportedOperation,

Unbounded already implements most of the required mechanics for its standalone agent:

• blue/green binary slots;
• current and last-good symlinks;
• release archive download and extraction;
• candidate version verification;
• atomic switching;
• pending/success/failure signals;
• daemon restart and systemd rollback recovery.

Relevant upstream packages and implementation:

• github.com/Azure/unbounded/pkg/agent/agentbinary
• github.com/Azure/unbounded/pkg/agent/goalstates (AgentUpgradePaths)
• cmd/agent/internal/daemon/agentupgrade.go
• cmd/agent/internal/daemon/controller_machineoperation.go

The exported library mechanics can be reused, but the standalone daemon orchestration is under cmd/agent/internal and is not directly importable by AKS Flex Node.

Motivation

Today, selecting an agent version is supported during initial bootstrap.sh, but updating an existing node requires manual binary replacement or a broader node lifecycle operation. Manual replacement has no managed status, automatic rollback, or synchronization with the copy of aks-flex-node inside the active nspawn worker.

A first-class upgrade operation is needed to:

• roll out agent fixes independently of Kubernetes repaves;
• report operation progress and failure through MachineOperation;
• roll back automatically if the new daemon does not start;
• avoid leaving nodes with partially switched binaries;
• define how the nspawn copy used for kubelet exec credentials is updated.

Proposed behavior

Handle OperationAgentUpgrade in the Flex daemon:

1. Validate operation parameters, including an exact HTTPS archive URL and expected SHA-256.
2. Download the architecture-specific AKS Flex Node archive without logging sensitive URL query strings.
3. Install the candidate into the inactive blue/green slot.
4. Verify the candidate with aks-flex-node version.
5. Atomically update last-good and current links.
6. Persist a pending upgrade signal.
7. Restart aks-flex-node-agent.service.
8. Have the new daemon mark the MachineOperation complete and clear the signal.
9. If the new daemon repeatedly fails, restore last-good, restart the service, and report the operation as failed.

The implementation should migrate existing direct-file installations at /usr/local/bin/aks-flex-node into the blue/green layout idempotently.

Upstream reuse/generalization

Consider contributing reusable APIs to Unbounded rather than copying its orchestration:

• generalize InstallAndSwitchFromTarGz to accept the expected archive member name instead of hardcoding unbounded-agent;
• export upgrade signal/recovery primitives that are independent of the standalone Unbounded daemon;
• allow callers to provide binary paths and the systemd service name.

AKS Flex Node release archives currently contain:

aks-flex-node-linux-amd64
aks-flex-node-linux-arm64

Nspawn binary synchronization

AKS Flex Node also copies its binary into the nspawn rootfs, where kubelet may invoke it for MSI/service-principal exec credentials. The design must explicitly choose and test one of:

• atomically update the active nspawn copy as part of AgentUpgrade and roll it back with the host binary;
• expose a read-only host-managed current binary through an nspawn bind mount;
• document and enforce that the nspawn copy updates only on repave, including compatibility guarantees.

Upgrading only the outer daemon without defining this behavior can leave host and nspawn versions skewed.

Security requirements

• HTTPS-only download URLs.
• SHA-256 verification before switching.
• URL redaction because URLs may contain SAS tokens.
• Safe archive path and expected-member validation.
• Bounded download/extraction size.
• Root-owned executable installation with atomic file/link updates.
• Preserve a verified last-good binary until a later upgrade succeeds.
• Never return or log credentials/tokens.

Acceptance criteria

• AgentUpgrade is no longer mapped to unsupportedOperation.
• Existing installations migrate idempotently to blue/green binary slots.
• A valid MachineOperation upgrades the running Flex daemon and reports success.
• Invalid URL, digest, archive, architecture, and candidate binaries report failure without switching.
• Failure of the new daemon automatically restores the last-good binary and reports failure.
• Repeating or reconciling the same operation is safe.
• Sensitive download URLs are redacted from logs and status.
• The active nspawn binary synchronization policy is implemented and tested.
• The Kubernetes Node remains Ready during a successful host-daemon-only upgrade, aside from any explicitly documented nspawn synchronization step.
• Unit tests cover migration, blue/green switching, rollback, status transitions, and idempotency.
• E2E covers successful upgrade, forced rollback, retry after rollback, kubelet exec authentication, and repave after upgrade.