fix: fall back to the default runtime when the toolkit config file is missing

[polyglotm]

What happens today

The generated runtime wrapper has one guard: it checks whether the NVIDIA driver modules are
loaded, and falls back to the default low-level runtime if they are not. It never checks
whether its own configuration file is present.

When the NVIDIA runtime is containerd's default runtime, every container on the node is
routed through that wrapper, not only GPU containers. If the driver is loaded but
.config/nvidia-container-runtime/config.toml is absent, the module guard passes,
nvidia-container-runtime.real runs, and it fails with couldn't open required configuration file. Sandbox creation then fails for every pod on the node — including the
nvidia-container-toolkit DaemonSet that would rewrite the file.

The node reports Ready with clean conditions and zero allocatable GPUs. Kubernetes cannot
repair it, because every tool it has is a container. Recovery requires host access.

We hit this in production on 2026-08-23. A reboot interrupted a toolkit reinstall after the
executables were written and before config.toml was. Restoring that single 1122-byte file
released the node completely: the toolkit pod went Pending to Running within 46 seconds,
allocatable GPUs returned, and the node recovered with no other change. A sibling node with
the same toolkit build and its config file intact was healthy throughout.

What this changes

One additional guard, in the same shape and with the same fallback target as the existing
module check. A missing configuration file becomes a reason to invoke the default runtime
rather than a reason to fail.

The node degrades to "GPU containers start without GPU access" instead of "nothing starts at
all" — and because the toolkit DaemonSet can now start, it repairs the node itself.

collectExecutables already computes this path and already passes it to the wrapper as
NVIDIA_CTK_CONFIG_FILE_PATH, so no new plumbing is required.

Why the guard is gated on CheckModules as well

nvidia-container-runtime-hook also receives a config path but is not a runtime, so it
must not be given a low-level runtime fallback. Gating on .ConfigFilePath alone would inject
exec runc "$@" into the hook's wrapper. The condition is therefore
{{- if and .CheckModules .ConfigFilePath }}, reusing the flag that already means "this is a
runtime".

Tests

Three cases were added to the existing TestWrapperRender table:

case	CheckModules	ConfigFilePath	expected
config check is added when a config path is set	true	set	both guards
config check is not added without the module check	false	set	neither guard
config check is not added without a config path	true	empty	module guard only

Each was confirmed to fail when the implementation is wrong, not merely to pass when it is
right:

• with the guard written as {{- if .ConfigFilePath }}, exactly one case fails, and it is
  config check is not added without the module check;
• with the guard removed entirely, exactly one case fails, and it is
  config check is added when a config path is set.

TestToolkitInstaller fixtures were updated for the three runtime wrappers. The
nvidia-container-runtime-hook and nvidia-container-cli fixtures are deliberately
unchanged, which is an independent check that the guard scopes correctly.

gofmt, go vet, go build ./... and go test ./cmd/nvidia-ctk-installer/... are clean.

The trade-off, stated up front

This fails open. During the window a GPU container may start believing it has a GPU and not
have one. That is a real cost, and it is a policy decision that belongs to you. Two things
bound it: the window ends when the DaemonSet rewrites the file, and the state is externally
visible as a Ready node with zero allocatable GPUs.

For non-GPU containers the change is strictly an improvement — today they are taken down by a
GPU-specific configuration file with which they have no relationship.

If you would rather fail closed, or fix the install sequence so that the wrapper cannot
become active before its configuration exists, we would be glad to take that instead. The
reproduction stands either way, and the outcome we care about is that a reboot cannot strand a
node with no automated recovery.

Versions

Reproduced in production on v1.17.8. Confirmed still present on main and in v1.20.0 by source
inspection: the generator moved from tools/container/toolkit/runtime.go to
cmd/nvidia-ctk-installer/toolkit/installer/executables.go and became a template, but no
configuration check was added. Upgrading does not resolve this.

Related

• invoke the actual default low-level runtime in the nvidia-ctk wrapper script #1365 introduced DefaultRuntimeExecutablePath, which this change reuses as its fallback
  target.
• containerd getting restarted on gpu node reboots gpu-operator#594 describes the install-time containerd restart that we believe can
  interrupt the install. A maintainer noted in 2023 that unnecessary restarts were worth
  avoiding; that issue was closed for inactivity in 2026 without resolution. We are not
  claiming this proves our sequence, only that the behaviour is known.

Open questions

1. Could the install-time containerd restart interrupt the installer between writing the
   executables and writing config.toml?
2. Is there an existing ordering or atomic-install guarantee intended to prevent a generated
   wrapper from becoming active before its required configuration file is present?
3. Should the fallback be unconditional for generated runtime wrappers, or controlled by a
   setting?

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[cdesiniotis]

When the NVIDIA runtime is containerd's default runtime, every container on the node is
routed through that wrapper, not only GPU containers. If the driver is loaded but
.config/nvidia-container-runtime/config.toml is absent, the module guard passes,
nvidia-container-runtime.real runs, and it fails with couldn't open required configuration file. Sandbox creation then fails for every pod on the node — including the
nvidia-container-toolkit DaemonSet that would rewrite the file.

The nvidia-container-runtime should fallback to using a default config TOML if the config file does not exist on the system. The below is the strack trace:

config.GetConfig() call in top-level Run():

nvidia-container-toolkit/internal/runtime/runtime.go

Lines 46 to 49 in 3a21b97

	cfg, err := config.GetConfig()
	if err != nil {
	return fmt.Errorf("error loading config: %v", err)
	}

Constructing a new config (note, that the WithRequired option is NOT set):

nvidia-container-toolkit/api/config/v1/config.go

Lines 93 to 95 in 3a21b97

	cfg, err := New(
	WithConfigFile(GetConfigFilePath()),
	)

Relevant code in loadConfigToml() that demonstrates the fallback:

nvidia-container-toolkit/api/config/v1/toml.go

Lines 88 to 98 in 3a21b97

	_, err := os.Stat(filename)
	if os.IsNotExist(err) && o.required {
	return nil, os.ErrNotExist
	}
	tomlFile, err := os.Open(filename)
	if os.IsNotExist(err) {
	return defaultToml()
	} else if err != nil {
	return nil, fmt.Errorf("failed to load specified config file: %w", err)
	}

@polyglotm can you provide more information on the exact scenario you hit? Steps on how to reproduce and exact logs from the incident, if possible, would help diagnose this better. I am not convinced the code in this PR is the direction we should go in.