Support managed blue/green Flex Node agent upgrades

cmd/aks-flex-node/main.go

@@ -47,7 +47,7 @@ func newRootCommand() *cobra.Command {
 	rootCmd.AddCommand(start.NewCommand())
 	rootCmd.AddCommand(bootstrapdata.NewCommand())
 	rootCmd.AddCommand(preflight.NewCommand())
-	rootCmd.AddCommand(daemon.NewCommand())
+	rootCmd.AddCommand(daemon.NewCommands()...)
 	rootCmd.AddCommand(nspawnlifecycle.NewCommand())
 	rootCmd.AddCommand(reset.NewCommand())
 	rootCmd.AddCommand(version.NewCommand())

docs/usages/operations.md

@@ -47,6 +47,42 @@ systemctl is-active aks-flex-node-agent
 journalctl -u aks-flex-node-agent -f
 ```
 
+## Managed Agent Upgrade
+
+When the Unbounded `MachineOperation` API is installed, submit an `AgentUpgrade` with an HTTP or HTTPS release archive and, when available, the SHA-256 of the compressed archive:
+
+```yaml
+apiVersion: unbounded-cloud.io/v1alpha3
+kind: MachineOperation
+metadata:
+  name: upgrade-agent-worker-01
+spec:
+  machineRef: worker-01
+  operationKind: AgentUpgrade
+  parameters:
+    downloadURL: https://example.com/aks-flex-node-linux-amd64.tar.gz
+    sha256: 0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
+```
+
+The archive must contain exactly the architecture-specific release member used by AKS Flex Node (`aks-flex-node-linux-amd64` or `aks-flex-node-linux-arm64`). The `sha256` parameter is optional; when supplied, the daemon verifies the compressed archive digest. Prefer HTTPS and a digest for production downloads. Plain HTTP is intended for explicitly trusted networks such as a VM-local loopback server; omit the digest only when both the archive source and transport path are trusted. The daemon always verifies the candidate `version` command before switching its blue/green binary links. It also atomically updates the binary in the active nspawn rootfs so kubelet exec authentication uses the same version.
+
+The restarted daemon marks the operation `Complete`. If the candidate cannot remain running, systemd restores the last-known-good host and nspawn binaries and marks the operation `Failed`. URL query strings, which may contain SAS credentials, are omitted from logs and operation status.
+
+MachineOperations are cluster-scoped. The daemon group requires cluster-wide read access to MachineOperations and Nodes, plus MachineOperation status update access, so restrict who can create operations and treat parameter values as sensitive API data. Prefer short-lived, read-only download credentials.
+
+```bash
+kubectl get machineoperation upgrade-agent-worker-01 -w
+```
+
+A host provisioning system that has already authenticated and staged a candidate can activate it directly without creating an Unbounded `MachineOperation`:
+
+```bash
+sudo /var/tmp/aks-flex-node-candidate agent-upgrade --preflight
+sudo /var/tmp/aks-flex-node-candidate agent-upgrade
+```
+
+The candidate must be staged separately from the installed binary. Direct activation and `MachineOperation` activation share one host lock and refuse to overlap with a pending operation signal. Both paths verify the candidate, switch the same blue/green layout, and restore last-good on activation failure. If `aks-flex-node-agent.service` is active, direct activation restarts it, verifies the running executable, and synchronizes the active nspawn exec-credential binary. If the service is already inactive during reset/rejoin provisioning, activation preserves that stopped state; the subsequent bootstrap starts the service and worker.
+
 ## Nspawn Worker
 
 Inspect the local nspawn-backed worker:

docs/usages/operator-first-boot.md

@@ -249,7 +249,7 @@ label.
 ## 3. Install temporary AKS Flex daemon RBAC
 
 > [!IMPORTANT]
-> **Temporary preview requirement:** When the Machina MachineOperation CRD is
+> **Temporary preview requirement:** When the Unbounded MachineOperation CRD is
 > installed, AKS Flex Node discovers it and enables its MachineOperation
 > reconciler. A future AKS RP release will install and manage the required
 > ClusterRole and ClusterRoleBinding automatically as part of FlexNodes pool
@@ -287,7 +287,17 @@ rules:
   resources:
   - machineoperations/status
   verbs:
+  - get
+  - patch
   - update
+- apiGroups:
+  - ""
+  resources:
+  - nodes
+  verbs:
+  - get
+  - list
+  - watch
 ---
 apiVersion: rbac.authorization.k8s.io/v1
 kind: ClusterRoleBinding

hack/controller-deployment/rbac.yaml

@@ -103,3 +103,33 @@ subjects:
   - apiGroup: rbac.authorization.k8s.io
     kind: Group
     name: aks-flex-node-daemons
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: aks-flex-node-daemon-machineoperations
+rules:
+  - apiGroups: ["unbounded-cloud.io"]
+    resources: ["machineoperations"]
+    verbs: ["get", "list", "watch"]
+  - apiGroups: ["unbounded-cloud.io"]
+    resources: ["machineoperations/status"]
+    verbs: ["get", "patch", "update"]
+  # The shared MachineOperation selector implementation evaluates labels on the
+  # local Node when an operation uses machineSelector.
+  - apiGroups: [""]
+    resources: ["nodes"]
+    verbs: ["get", "list", "watch"]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: aks-flex-node-daemon-machineoperations
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: aks-flex-node-daemon-machineoperations
+subjects:
+  - apiGroup: rbac.authorization.k8s.io
+    kind: Group
+    name: aks-flex-node-daemons

hack/e2e/README.md

@@ -54,16 +54,17 @@ The default `all` command runs:
 7. Unjoin all Flex Nodes and verify they are absent, including reset cleanup of host network artifacts.
 8. Rejoin all Flex Nodes and validate again.
 9. Validate the installed nspawn lifecycle helper and generated systemd hooks, then reconcile a running node through the helper.
-10. Run controller-machine-driven repave validation.
-11. Collect logs and clean up Azure resources.
+10. Validate managed agent upgrade, forced rollback, retry, nspawn synchronization, and kubelet authentication.
+11. Run controller-machine-driven repave validation after the agent upgrade.
+12. Collect logs and clean up Azure resources.
 
 ## Commands
 
 `run.sh` accepts a command as its first positional argument. When omitted, it defaults to `all`.
 
 | Command | Description |
 |---------|-------------|
-| `all` | Full flow: build, infra, join, validate, unjoin, validate absent, rejoin, validate, repave, logs, cleanup. |
+| `all` | Full flow: build, infra, join, validate, unjoin, validate absent, rejoin, validate, lifecycle, agent upgrade, repave, logs, cleanup. |
 | `infra` | Deploy AKS cluster, four VMs, Unbounded-Net CNI, the local registry, and the in-cluster controller. |
 | `join` | Join all Flex Node VMs. |
 | `join-msi` | Join only the managed-identity node. |
@@ -79,6 +80,7 @@ The default `all` command runs:
 | `validate-absent` | Verify Flex Node objects are absent after unjoin. |
 | `smoke` | Run smoke workloads only. |
 | `nspawn-lifecycle` | Validate lifecycle helper installation and generated hooks on all nodes, then regenerate config and restart the token node through lifecycle reconciliation. |
+| `agent-upgrade` | Validate managed agent upgrade, forced rollback, retry, direct host activation, and nspawn synchronization. |
 | `upgrade-drift` | Validate controller-machine-driven repave to the alternate nspawn side. |
 | `logs` | Collect logs from VMs. |
 | `cleanup` | Collect logs and delete Azure resources. |
@@ -123,6 +125,7 @@ Additional environment variables:
 | `E2E_SSH_WAIT_TIMEOUT` | `300` | Timeout in seconds while waiting for SSH. |
 | `E2E_NODE_JOIN_TIMEOUT` | `300` | Timeout in seconds while waiting for node bootstrap. |
 | `E2E_POD_READY_TIMEOUT` | `120` | Timeout in seconds while waiting for smoke pods. |
+| `E2E_AGENT_UPGRADE_TIMEOUT` | `300` | Timeout in seconds while waiting for an AgentUpgrade result. |
 | `E2E_DRIFT_UPGRADE_TIMEOUT` | `900` | Timeout in seconds while waiting for repave. |
 | `AZURE_SUBSCRIPTION_ID` | auto-detected | Azure subscription. |
 | `AZURE_TENANT_ID` | auto-detected | Azure tenant. |
@@ -143,7 +146,26 @@ its Azure resource name remains lowercase. This verifies that an omitted
 `agent.nodeName` is derived from the normalized hostname and still joins the
 cluster under the lowercase VM name.
 
-Each join path uploads the locally built binary, renders a config file, installs the binary through `scripts/install.sh` with `AKS_FLEX_NODE_LOCAL_BINARY`, and starts the node through a transient systemd unit. The installed agent service is then validated with systemd checks.
+Each join path uploads the locally built binary and renders a config file. Fresh hosts install it through `scripts/install.sh` with `AKS_FLEX_NODE_LOCAL_BINARY`; rejoin hosts with an existing managed layout invoke the uploaded candidate's `agent-upgrade` command before bootstrap. The node starts through a transient systemd unit, and the installed agent service is then validated with systemd checks.
+
+## Agent Upgrade Validation
+
+The `agent-upgrade` command uses the bootstrap-token VM to exercise the complete managed binary lifecycle:
+
+1. Serve architecture-specific release archives over VM-local loopback HTTP to validate HTTP transport support.
+2. Submit an `AgentUpgrade` with an archive SHA-256 and a query credential.
+3. Verify successful daemon restart, operation completion, binary replacement, and host/nspawn binary equality.
+4. Restart kubelet to exercise the synchronized nspawn exec-credential binary and require the Node to remain Ready.
+5. Upgrade to a candidate that passes `version` but fails daemon startup, then verify automatic rollback and a failed operation.
+6. Confirm status does not expose the sensitive URL query and retry successfully into the inactive slot without the optional archive digest.
+7. Stage a distinct candidate and validate direct host activation preflight, inactive-slot switch, service health, shared layout, and active-nspawn synchronization without creating a `MachineOperation` signal.
+8. Restart kubelet through the directly activated nspawn credential binary, require Lease renewal and Node readiness, then run a workload before the subsequent repave test.
+
+Run it against an already joined environment:
+
+```bash
+./hack/e2e/run.sh agent-upgrade
+```
 
 ## Nspawn Lifecycle Validation
 
@@ -234,6 +256,7 @@ hack/e2e/
     node-join-offline.sh  Offline artifacts join/unjoin.
     node-join-kubeadm.sh  Kubeadm-style bootstrap-token join/unjoin.
     nspawn-lifecycle.sh   Lifecycle helper and managed restart validation.
+    agent-upgrade.sh      Managed agent upgrade and rollback validation.
     upgrade-drift.sh      Controller machine goal repave validation.
     validate.sh           Node readiness and smoke tests.
     cleanup.sh            Log collection and Azure resource cleanup.