Recursive catalog watcher traverses Resource payloads before first reconcile

[schickling]

st2 up can spend its entire startup traversing Resource payloads before its first reconciliation pass. If watcher installation then fails, the intended timer fallback can also continuously reconcile instead of honoring its interval.

Reproduction

https://github.com/schickling-repros/2026-08-st2-notify-recursive-resource-tree

cargo run --release

The synthetic catalog has four declaration-space directories and a 512-directory Resource payload. On Linux with notify 8.2.0, recursive registration allocates a watch for the payload directories as well:

recursive watches: 519
declaration-only shallow watches: 4

In the motivating deployment, the inotify descriptor grew past 193,000 watches while registration followed a Resource worktree into an external dependency tree. The first reconcile did not begin for roughly 20 minutes. When recursive installation finally returned/dropped, the watch count fell to four and timer polling reconciled the pending state.

Root cause

Unbounded installation

watch_catalog_declarations filters callback paths, but it first calls watcher.watch(root, RecursiveMode::Recursive). The Linux backend eagerly walks the full tree and allocates inotify watches before any callback filtering occurs. Catalog Resource directories may contain arbitrarily large worktrees and dependency trees, including links outside the declaration namespace.

Disconnected fallback bypasses the interval

If best-effort watcher installation fails, its channel sender is dropped. The receiver then returns RecvTimeoutError::Disconnected immediately. The old wait loop treats that as the end of the wait and starts another full reconcile, so the nominal timer fallback can become a continuous reconcile loop.

The installation error was also discarded behind Option, making both the loss of immediate wakeups and the cadence failure hard to diagnose.

Expected

• Watcher installation scales with declaration-space directories, not Resource payloads.
• Resource/worktree links cannot escape the catalog watch boundary.
• Declaration creation, update, rename, and removal still wake reconciliation.
• Installation failures are visible.
• When no watcher is available, reconciliation waits for the configured timer interval and does not spin.
• Stop requests remain responsive while waiting.

Actual

• Recursive installation traverses Resource payloads before the first reconcile.
• Callback filtering does not reduce startup traversal or inotify watch count.
• Resource worktree links can carry traversal outside the declaration namespace.
• Installation failures and timer-only fallback are silent.
• A disconnected watcher channel bypasses the configured timer interval and can continuously reconcile.

Versions

• st2: current main and 1a8f4009b4c9a85e37cdb035463959494ede8628
• notify: 8.2.0
• OS: Linux with inotify

[schickling]

The live dev3 failure reveals a second watcher-fallback bug that #316 does not currently cover.

At installed st2 b8a3b24, recursive watcher setup failed and the watcher was dropped. The process has one thread, six file descriptors, and no inotify descriptor, but it runs about 19 full catalog reconciles per 31 seconds despite the default 30-second timer. An 8-second bounded sample observed seven reconcile summaries, 196 Git-exclude warnings, 86.3 MB of characters read, and 2.07 CPU-seconds.

Mechanism: watcher setup returning None drops the final channel sender. up_loop receives RecvTimeoutError::Disconnected during each 250 ms wait slice and treats it like a change by breaking the wait, immediately starting another full pass. Therefore the claimed timer-only fallback is a tight reconcile loop.

#316 bounds watcher registration and preserves the setup diagnostic, but its failure path still returns None with no retained sender, and the loop still treats Disconnected as an immediate wake. Its fallback test checks only that setup returns None; it does not exercise supervisor cadence.

Suggested smallest addition to #316:

• Keep one sender alive for the supervisor lifetime, or explicitly wait the remaining interval after channel disconnection.
• Add an end-to-end control where watcher setup fails and assert that reconcile count remains timer-bounded while the diagnostic is emitted once.

The eight live flapping seats are parked after Tokenlens-related launch failures, but they only make each pass noisy. They do not cause the rapid cadence; the same disconnected channel spins with an otherwise converged catalog.