🐛 fix: raise RLIMIT_NOFILE at serve startup — fd exhaustion silently wedges accept()

[tony-nexartis]

The bug

codesearch serve's file-descriptor demand scales with registered repo count — each warm repo holds an LMDB env + tantivy FTS segment handles + file-watcher handles (≈ 15–20 fds measured on macOS). Under process supervisors the default soft RLIMIT_NOFILE is often 256 (macOS launchd agents, some systemd/docker configs). Once repo warmup saturates that limit:

• tantivy logs Too many open files (errno 24) warnings, and
• accept(2) fails with EMFILE; axum 0.7's accept loop sleeps and retries silently — the daemon looks alive to its supervisor (KeepAlive sees a healthy pid) while every new connection is refused or reset. No ERROR log, no exit. A silent wedge.

How we hit it

Production workspace with 60 registered repos (~1000 fds needed) running serve as a macOS LaunchAgent: serve answered MCP handshakes for ~15s after every start (until warmup consumed the fd budget), then reset every connection while the process stayed 'healthy' — deterministic across restarts. Diagnosis artifacts: errno-24 lines in serve.log, sample showing all threads idle-parked with the listener effectively dead, process fd table saturated at exactly 255/256 under a ulimit -n 256 repro.

The fix

At run_serve startup, before any store open or listener bind (unix only, no-op elsewhere):

1. Raise the soft RLIMIT_NOFILE to the hard limit — standard daemon practice (nginx/envoy/postgres all do it). On macOS the target is clamped to kern.maxfilesperproc so setrlimit cannot fail with EINVAL. Failures are non-fatal and logged.
2. Log the raise at INFO: Raised RLIMIT_NOFILE soft limit 256 → 61440.
3. If the effective limit still looks too small for the registered repo count (repos × 20 + 256 headroom), emit a loud actionable WARN naming the supervisor knobs (launchd SoftResourceLimits.NumberOfFiles, systemd LimitNOFILE, ulimit -n) — so the failure mode becomes an explicit operator decision instead of a silent wedge.

Adds libc as a direct cfg(unix) dependency (already in the tree transitively).

Verification

• At-scale repro: ulimit -n 256 + 60 registered repos — unpatched serve saturates at 255/256 fds (EMFILE in logs; wedges under launchd); patched serve logs the raise, runs at ~300 fds, and answers MCP handshakes indefinitely under the same conditions.
• cargo clippy --lib --bins -- -D warnings clean.
• cargo test --lib --bins green (579 + 575 pass, 13 ignored — same as develop).

Possible follow-up (not in this PR)

Even with the raise, a truly exhausted process still wedges silently because axum swallows accept errors. A custom accept loop (or a periodic fd-headroom check) that logs at ERROR when accept hits EMFILE/ENFILE would make the residual case loud too. Happy to take a stab if you want it.

[flupkede]

✅ Approve — well-motivated, production-tested, standard daemon practice.

The silent-wedge failure mode (axum 0.7 accept loop retrying EMFILE while the process looks healthy to its supervisor) is exactly the class of bug that ships to production undetected. Raising soft → hard RLIMIT_NOFILE at startup is what nginx/envoy/postgres all do; the macOS kern.maxfilesperproc clamp is the correct pre-emptive fix for setrlimit EINVAL.

Correctness verified:

• Call-site placement is right: after auto_discover_from_cwd (repo count is final), before ServeState::new (consumes config) and before TcpListener::bind.
• #[cfg(unix)] gating consistent at function def + call site.
• unsafe block is sound (locally-owned rlimit struct, SAFETY comment present).
• #[cfg(target_os = "macos")] / #[cfg(not(...))] paired for sysctl clamp.
• Non-fatal error handling (log + continue with inherited limit) — correct choice, never breaks startup.
• lim.rlim_cur = before rollback on setrlimit failure.
• info!/warn! already imported at src/serve/mod.rs:34, no new imports needed.
• libc correctly added as [target.'cfg(unix)'.dependencies] (not unconditional).

Minor nits (non-blocking, feel free to ignore):

1. CString::new("kern.maxfilesperproc").unwrap() — .expect("kernel sysctl name has no NUL") is slightly more idiomatic for a known-safe constant.
2. FDS_PER_REPO_ESTIMATE = 20 is macOS-measured; on Linux the per-repo fd cost may differ, so the warning could fire conservatively at high repo counts. Not wrong, just informational.
3. No unit test — acceptable for a platform-specific syscall with a documented manual repro in the PR body.

The follow-up idea (custom accept loop / periodic fd-headroom check that logs at ERROR on residual EMFILE) is a good candidate for a separate PR — agree it's out of scope here.

Will rebase + merge. Thanks for the thorough write-up and repro.

[flupkede]

✅ Approve — well-motivated, production-tested, standard daemon practice.

The silent-wedge failure mode (axum 0.7 accept loop retrying EMFILE while the process looks healthy to its supervisor) is exactly the class of bug that ships to production undetected. Raising soft → hard RLIMIT_NOFILE at startup is what nginx/envoy/postgres all do; the macOS kern.maxfilesperproc clamp is the correct pre-emptive fix for setrlimit EINVAL.

Correctness verified:

• Call-site placement is right: after auto_discover_from_cwd (repo count is final), before ServeState::new (consumes config) and before TcpListener::bind.
• #[cfg(unix)] gating consistent at function def + call site.
• unsafe block is sound (locally-owned rlimit struct, SAFETY comment present).
• #[cfg(target_os = "macos")] / #[cfg(not(...))] paired for sysctl clamp.
• Non-fatal error handling (log + continue with inherited limit) — correct choice, never breaks startup.
• lim.rlim_cur = before rollback on setrlimit failure.
• info!/warn! already imported at src/serve/mod.rs:34, no new imports needed.
• libc correctly added as [target.'cfg(unix)'.dependencies] (not unconditional).

Minor nits (non-blocking, feel free to ignore):

1. CString::new("kern.maxfilesperproc").unwrap() — .expect("kernel sysctl name has no NUL") is slightly more idiomatic for a known-safe constant.
2. FDS_PER_REPO_ESTIMATE = 20 is macOS-measured; on Linux the per-repo fd cost may differ, so the warning could fire conservatively at high repo counts. Not wrong, just informational.
3. No unit test — acceptable for a platform-specific syscall with a documented manual repro in the PR body.

The follow-up idea (custom accept loop / periodic fd-headroom check that logs at ERROR on residual EMFILE) is a good candidate for a separate PR — agree it's out of scope here.

Will rebase + merge. Thanks for the thorough write-up and repro.