node-agent crashes (CrashLoopBackOff) when fanotify is unsupported - no fallback to Pod Informer

[khuswant18]

Description

While investigating the node-agent container discovery startup path, I found that node-agent currently uses WithContainerFanotifyEbpf() directly without checking whether fanotify is supported by the environment.

I traced what happens when fanotify initialization fails, and the error propagates through ContainerCollection.Initialize() all the way to main(), where the node-agent process exits. In Kubernetes, this can result in the Pod being restarted and entering CrashLoopBackOff.

The current flow is:

node-agent starts
    ↓
WithContainerFanotifyEbpf()
    ↓
fanotify initialization fails
    ↓
ContainerCollection.Initialize() returns error
    ↓
ContainerWatcher.StartContainerCollection() returns error
    ↓
node-agent exits
    ↓
Kubernetes restarts the Pod
    ↓
CrashLoopBackOff

The resulting error can look like:

starting container collection: initializing container collection: starting container fanotify: fanotify: init error, operation not permitted

What I found

While tracing this, I also checked the Inspektor Gadget implementation used by node-agent.

Inspektor Gadget already provides containerhook.Supported() to check whether the fanotify-based container hook can be initialized. Its hookMode: auto logic uses this to choose between fanotify/eBPF and WithPodInformer() when fanotify is unavailable.

The relevant upstream IG logic is in pkg/operators/kubemanager/hooks-utils.go (hookMode2ccOpts function).

However, node-agent constructs the ContainerCollection directly and currently calls WithContainerFanotifyEbpf() without this capability check. I also couldn't find any current use of WithPodInformer() or WithFallbackPodInformer() in the node-agent codebase.

The relevant call is currently here:

pkg/containerwatcher/v2/container_watcher_collection.go:93

Proposed solution

I think node-agent could follow the same capability-detection approach already present in Inspektor Gadget:

              Is fanotify supported?
                    /        \
                  yes         no
                   |           |
                   ↓           ↓
             fanotify/eBPF  Pod Informer

Conceptually:

func (cw *ContainerWatcher) containerDiscoveryOption() containercollection.ContainerCollectionOption {
	return func(cc *containercollection.ContainerCollection) error {
		if containerhook.Supported() {
			return containercollection.WithContainerFanotifyEbpf()(cc)
		}

		logger.L().Warning(
			"fanotify+eBPF not supported, falling back to Pod Informer",
		)

		return containercollection.WithPodInformer(cw.cfg.NodeName)(cc)
	}
}

This would:

• preserve the current fanotify/eBPF behavior when supported,
• fall back to Kubernetes Pod Informer when fanotify is unavailable,
• avoid adding a new configuration flag,
• avoid retrying ContainerCollection.Initialize(),
• keep fanotify as the preferred mechanism on supported Linux environments.

Why WithPodInformer() instead of WithFallbackPodInformer()?

From my investigation:

• WithPodInformer(nodeName) provides container discovery through Kubernetes and can act as the primary discovery mechanism.
• WithFallbackPodInformer(nodeName) is intended to catch containers that were missed by an already-running primary hook.

If fanotify cannot be initialized, there is no functioning primary fanotify hook to supplement, so WithPodInformer() appears to be the appropriate fallback.

Actual failure path

The current error propagation looks like:

main.go:494
    mainHandler.Start(ctx)
        ↓
container_watcher.go:314
    cw.StartContainerCollection(ctx)
        ↓
container_watcher_collection.go:93
    containercollection.WithContainerFanotifyEbpf()
        ↓
options.go
    containerhook.NewContainerNotifier()
        ↓
tracer.go
    n.install()
        ↓
tracer.go
    initFanotify()
        ↓
fanotify.go
    unix.FanotifyInit()
        ↓
    EPERM / ENOSYS
        ↓
    "fanotify: init error"
        ↓
ContainerCollection.Initialize()
        ↓
ContainerWatcher.StartContainerCollection()
        ↓
main.go
    os.Exit(utils.ExitCodeError)
        ↓
Kubernetes restarts the Pod
        ↓
CrashLoopBackOff

Verification

I was able to confirm the fanotify failure path by tracing the current node-agent and Inspektor Gadget source code.

I also tested the available Kind-on-Docker-Desktop environment, but fanotify initialization succeeds there because Kind runs inside a Linux environment that provides the required kernel functionality.

So I have not been able to reproduce the actual CrashLoopBackOff on a truly fanotify-unsupported environment yet. The startup failure path itself is confirmed from the source code.

Expected behavior

When fanotify is available:

node-agent
    ↓
fanotify/eBPF
    ↓
container discovery

When fanotify is unavailable:

node-agent
    ↓
Pod Informer
    ↓
container discovery

In both cases, the node-agent should be able to complete initialization instead of exiting solely because fanotify is unavailable.

Additional context

• cw.cfg.NodeName is already populated from NODE_NAME and is the value used to identify the local node, so it can be passed to WithPodInformer().
• The Inspektor Gadget implementation already contains the containerhook.Supported() + Pod Informer selection pattern.
• No new configuration flag should be required.
• Fanotify/eBPF should remain the preferred mechanism wherever it is supported.
• The fallback should happen during container discovery selection rather than by attempting to initialize the collection twice.

Would maintainers be open to using the existing containerhook.Supported() capability check and falling back to WithPodInformer(cw.cfg.NodeName) when fanotify isn't available?