fix(run): bound catalog watches to declaration space

[schickling-assistant]

Problem

The supervisor registers one recursive notify watch over the entire catalog root before its first reconcile. On Linux, notify eagerly walks the full tree and allocates one inotify watch per directory before any callback filtering runs, and follows symlinks by default. Catalog Resource payload trees (a production deployment measured 37 GB / 1.74 M entries / 151 k directories) therefore dominated startup: ~13 minutes of traversal before the first reconcile, a multi-GiB memory peak, and installation failure once kernel limits were hit.

When installation failed, the dropped channel sender made RecvTimeoutError::Disconnected look like an immediate wake in the wait loop (run.rs), turning the nominal 30 s timer into a tight full-catalog reconcile loop: ~48 passes/min at ~70 % of one core, with per-pass git subprocess churn. The failure itself was discarded behind an Option, so timer-only fallback was silent.

Goal

Watcher installation scales with declaration space, not Resource payloads; declaration mutations still wake reconciliation immediately; failures are visible; the timer fallback honors its interval; stop stays responsive.

Decisions

• Shallow per-directory watches over the declaration namespace instead of one recursive registration: recursion is gated by agent_spec::is_catalog_path, so resources//archive/inbox/.git/.st2 prune discovery, and RecursiveMode::NonRecursive never traverses payloads. A refresh() after each pass picks up newly created directories without backend traversal.
• Directory topology mutations wake even though no agent.kdl exists yet — the created dir may receive one next, and refresh needs to run.
• Deadline-based wait: disconnection sleeps the remaining slice instead of waking; change/timer/stop are distinct outcomes.
• Injectable watcher factory on the supervisor loop so tests can drive the real loop under installation failure (prior art fix(run): bound catalog watches to declaration space #316 tested only the helper).
• Ported the sound core of the closed fix(run): bound catalog watches to declaration space #316 branch; this PR adds what it lacked (see Verification).

Verification

• New e2e tests drive up_loop_until: injected watcher failure stays on timer cadence (~4 passes in 350 ms at a 100 ms interval; the old code produced hundreds), a live watcher wakes on declaration mutation long before the timer, and a disconnected channel waits out the interval.
• Watch-boundary unit test: symlinked external worktree is not entered; payload depth adds zero watches.
• Live reproduction against a synthetic oversized catalog (1539 payload directories plus a symlinked external tree), fixed binary:
  • inotify watches installed: 11 (declaration dirs only; recursive registration would allocate 1500+)
  • first reconcile: immediate
  • idle CPU over a 4 s window: 3 ticks of 400 (~0.75 %)
  • SIGINT stops cleanly mid-wait

Complexity

CatalogDeclarationWatcher is a small BTreeSet-diff over declaration dirs; it replaces rather than adds machinery. No new dependencies.

Concerns

• Watch count now scales with agent count (one watch per declaration-space directory); bounded by fleet size, not payload size.
• refresh() walks declaration space each pass; pruned at payload boundaries, so cost tracks agents, not data.

Follow-ups

• Watcher setup failure turns timer fallback into a tight reconcile loop #328 covers the cadence regression this closes.
• Per-pass advisory churn (git-exclude spawn + warning repetition) is addressed separately in this stack.
• Pre-existing unrelated red test on main: agent_publish::incomplete_apply_marker_blocks_declarations_but_not_the_state_plane fails identically at base commit (resident supervisor exits because publish_owner_binding treats the incomplete-apply marker as fatal); filed as Resident supervisor exits on incomplete-apply catalog instead of degrading: publish_owner_binding treats the marker as fatal #337.

References

Closes #314. Supersedes the unmerged #316 approach with the missing supervisor-level proof. Related: #328.

Posted on behalf of @schickling

field	value
agent_identity	unknown
agent_persona	generalist
agent_supervisor	unavailable
agent_tool	OMP
agent_tool_version	18.0.3
agent_runtime	OMP 18.0.3
tooling_profile	dotfiles@f33cd9c-dirty

[schickling-assistant]

@codex Please review this PR.

Posted on behalf of @schickling

field	value
agent_identity	unknown
agent_persona	generalist
agent_supervisor	unavailable
agent_tool	OMP
agent_tool_version	18.0.3
agent_runtime	OMP 18.0.3
tooling_profile	dotfiles@f33cd9c-dirty

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: dd0fcc7bee

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".