Reconcile NVIDIA state on every nspawn machine start - #589
Reconcile NVIDIA state on every nspawn machine start#589hbc (bcho) with Copilot wants to merge 5 commits into
Conversation
Co-authored-by: bcho <1975118+bcho@users.noreply.github.com>
|
Self-review update pushed in 06d4432 after merging current main safely. Key fixes: (1) NVIDIA boot reconciliation now uses a narrow host-only resolver instead of full machine resolution, avoiding LocalDNS/network dependencies during early boot; (2) managed starts and the nspawn ExecStartPost path are serialized with a host lock and use the lifecycle ready marker to avoid concurrently rebuilding the driver root after containerd is released; (3) a missing agent executable now fails ExecStartPost so the existing nspawn Restart=on-failure policy retries instead of leaving containerd/kubelet blocked forever; and (4) command coverage now includes persisted-config success, missing initial config, checksum failure, unavailable NVIDIA state, and CPU rendering.\n\nOverlap review: #309 remains compatible because disabled/non-GPU goal state does not render the hook or readiness dependencies; its config-aware NVIDIA resolution should be carried into ResolveNVIDIASetup when that PR lands. #412 remains complementary and should run before nspawn startup to refresh changed host device/library bind mounts; this PR reconciles in-machine runtime state after startup and does not replace #412.\n\nValidation: full build, lint, generated-file checks, focused tests, all non-race Go tests, and agent e2e jobs passed. The CI Test job hit the existing unrelated TestConfigureLocalDNS text-file-busy race after all changed packages passed; I reran the failed job. |
NVIDIA driver roots and CDI specifications could remain stale after host reboots or direct nspawn restarts, allowing containerd and kubelet to start before GPU runtime state was rebuilt.
Lifecycle reconciliation
ExecStartPosthook for NVIDIA machines.SetupNVIDIAtask.Service readiness
unbounded-nvidia-ready.service.Failure handling