agent: reconcile nspawn lifecycle configuration and NVIDIA state

[hbc (bcho)]

Summary

Combines and supersedes the implementation work from #412 and #589 behind one coherent nspawn lifecycle:

• refresh host device, mount, and GPU configuration before every nspawn start
• rewire NVIDIA driver-root and CDI state after every GPU machine start
• safely restart kubelet and containerd around NVIDIA rewiring
• migrate machines created by older agents without recreating or restarting the active rootfs
• provide a host command that triggers both lifecycle phases as a mini repave

Fixes #588.

Lifecycle commands

The rollback-stable helper exposes these hidden internal operations:

/usr/local/lib/unbounded-agent/nspawn-lifecycle-helper nspawn-lifecycle pre-start <machine>
/usr/local/lib/unbounded-agent/nspawn-lifecycle-helper nspawn-lifecycle post-start <machine>
/usr/local/lib/unbounded-agent/nspawn-lifecycle-helper nspawn-lifecycle reconcile <machine>

reconcile restarts the managed systemd-nspawn@<machine>.service. The service refreshes host mounts before startup and performs NVIDIA rewiring afterward.

Current-host discovery

There is no additional lifecycle goal-state or readiness file. Each lifecycle run discovers devices and NVIDIA state from the current host:

• no NVIDIA devices skips post-start rewiring
• complete NVIDIA discovery renders mounts and rebuilds the driver root and CDI state
• devices with incomplete driver state fail the start so systemd retries
• NVIDIA hardware appearing or disappearing is reflected on the next lifecycle run

On GPU hosts, the reusable post-start flow performs:

stop kubelet
-> stop containerd
-> rebuild NVIDIA links, driver root, and CDI
-> start containerd
-> import staged images
-> start kubelet

This flow is exported as nodestart.ReconcileNVIDIA, so AKS Flex and other pkg/agent consumers can reuse the same service-safe rewiring sequence.

Failure handling

• pre-start failures retry in the required systemd oneshot and keep nspawn from starting
• post-start failures fail ExecStartPost, causing the nspawn unit to restart
• services are stopped before the ephemeral driver root is replaced
• the lifecycle helper remains available if the daemon binary symlink is rolled back

Upgrade migration

When the upgraded daemon starts, it idempotently installs or updates only:

• the rollback-stable lifecycle helper
• host nspawn config and service drop-in
• host pre/post hooks
• host systemd daemon reload

It does not recreate the rootfs, rewrite working containerd or kubelet service configuration, or restart currently running in-machine services.

Validation

Passed locally:

• focused gofumpt
• focused golangci-lint
• focused race tests for agent command, daemon, goalstates, nodestart, rootfs, and reset packages
• go build ./...
• git diff --check

Relationship to existing PRs

This draft is intended to replace the separate merge paths for #412 and #589. It preserves device refresh and NVIDIA rewiring while using current-host discovery and no extra lifecycle state file. Neither existing PR is merged by this change.

[Copilot]

Pull request overview

Introduces a unified nspawn lifecycle that refreshes host configuration before startup and reconciles NVIDIA state afterward.

Changes:

• Adds pre-start regeneration and persisted NVIDIA lifecycle state.
• Gates containerd and kubelet on NVIDIA readiness.
• Migrates existing machines and adds lifecycle validation tests.

Reviewed changes

Copilot reviewed 46 out of 46 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
pkg/agent/phases/rootfs/testdata/service-override-kube2.conf.golden	Updates kube2 lifecycle snapshot.
pkg/agent/phases/rootfs/testdata/service-override-kube1.conf.golden	Updates kube1 lifecycle snapshot.
pkg/agent/phases/rootfs/testdata/render/nvidia-gb300-rack-full.service-override.conf.golden	Captures NVIDIA lifecycle hooks.
pkg/agent/phases/rootfs/testdata/render/nvidia-all-helpers.service-override.conf.golden	Captures helper-enabled lifecycle hooks.
pkg/agent/phases/rootfs/testdata/render/cpu-only.service-override.conf.golden	Captures CPU pre-start behavior.
pkg/agent/phases/rootfs/nspawn.go	Renders lifecycle units and state.
pkg/agent/phases/rootfs/nspawn_render_test.go	Tests lifecycle rendering.
pkg/agent/phases/rootfs/assets/service-override.conf	Adds pre/post-start dependencies.
pkg/agent/phases/rootfs/assets/config-regeneration.service	Defines retrying pre-start regeneration.
pkg/agent/phases/reset/nspawn.go	Removes lifecycle artifacts.
pkg/agent/phases/nodestart/start.go	Delegates NVIDIA setup to lifecycle hook.
pkg/agent/phases/nodestart/start_test.go	Verifies revised startup sequence.
pkg/agent/phases/nodestart/nvidia.go	Adds locking, validation, and readiness.
pkg/agent/phases/nodestart/nvidia_test.go	Tests provisioned capability handling.
pkg/agent/phases/nodestart/nspawn.go	Exports machine readiness polling.
pkg/agent/phases/nodestart/lifecycle_units.go	Migrates in-machine readiness units.
pkg/agent/phases/nodestart/lifecycle_units_test.go	Tests legacy unit migration.
pkg/agent/phases/nodestart/kubelet.go	Supplies NVIDIA gating state.
pkg/agent/phases/nodestart/kubelet_test.go	Tests kubelet gating.
pkg/agent/phases/nodestart/cri.go	Manages containerd readiness units.
pkg/agent/phases/nodestart/cri_test.go	Tests containerd gating.
pkg/agent/phases/nodestart/assets/unbounded-nvidia-ready.service	Defines NVIDIA readiness gate.
pkg/agent/phases/nodestart/assets/kubelet.service	Gates kubelet on NVIDIA readiness.
pkg/agent/phases/nodestart/assets/containerd.service	Gates containerd on NVIDIA readiness.
pkg/agent/goalstates/rootfs.go	Adds lifecycle and capability fields.
pkg/agent/goalstates/resolve.go	Preserves provisioned NVIDIA capability.
pkg/agent/goalstates/resolve_test.go	Tests capability resolution and migration.
pkg/agent/goalstates/nvidia.go	Makes runtime enablement capability-driven.
pkg/agent/goalstates/nspawn_lifecycle.go	Defines persisted lifecycle state.
pkg/agent/goalstates/nspawn_lifecycle_test.go	Tests lifecycle state validation.
pkg/agent/goalstates/nodestart.go	Records provisioned GPU capability.
pkg/agent/goalstates/containerd.go	Resolves containerd from capability.
pkg/agent/goalstates/constants.go	Adds lifecycle paths and unit names.
hack/agent/e2e-kind/node-configs/README.md	Documents device-refresh scenarios.
hack/agent/e2e-kind/node-configs/additional-host-mounts.json	Enables reboot refresh validation.
hack/agent/e2e-kind/e2e.py	Adds host-reboot device-refresh E2E coverage.
docs/content/reference/agent/nspawn.md	Documents the lifecycle behavior.
cmd/agent/internal/daemon/nodeoperator.go	Migrates lifecycle artifacts on startup.
cmd/agent/internal/daemon/migration_test.go	Tests migration retry behavior.
cmd/agent/internal/daemon/daemon.go	Runs startup lifecycle migration.
cmd/agent/internal/daemon/controller_test.go	Extends the fake node operator.
cmd/agent/internal/cmd/reconcile_nvidia.go	Implements post-start reconciliation.
cmd/agent/internal/cmd/reconcile_nvidia_test.go	Tests post-start state handoff.
cmd/agent/internal/cmd/nspawn_config.go	Implements pre-start regeneration.
cmd/agent/internal/cmd/nspawn_config_test.go	Tests regeneration and state persistence.
cmd/agent/internal/cmd/cmd.go	Registers lifecycle commands.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 48 out of 48 changed files in this pull request and generated no new comments.

Suppressed comments (1)

pkg/agent/phases/nodestart/nvidia.go:65

• The SetupNVIDIA documentation still says that a disabled runtime or missing libraries is a no-op, but this branch now deliberately returns an error for a GPU-provisioned machine with incomplete state. Update the comment so callers understand that only CPU-provisioned machines skip setup and incomplete GPU state blocks startup.

[Copilot]

Pull request overview

Copilot reviewed 48 out of 48 changed files in this pull request and generated no new comments.

Suppressed comments (2)

pkg/agent/phases/nodestart/nvidia.go:65

• The function comment still says disabled runtime or missing libraries are a no-op, but this branch now returns an error for any provisioned NVIDIA machine with incomplete runtime/host state. Update the documented contract so callers do not treat this new hard failure as optional behavior.
  pkg/agent/phases/nodestart/nvidia.go:70
• This lock is the safety mechanism preventing concurrent lifecycle callers from rebuilding a live driver root, but the new tests do not exercise it. Add coverage proving same-machine callers serialize, different machines do not block each other, and a waiting caller exits on context cancellation; regressions here can corrupt NVIDIA state after services are released.