Hot reload the models config block when the config file changes

[Devin-Holland]

Closes #2344.

The models: block of harperdb-config.yaml now hot-reloads when the file changes: an orchestrator (or a self-hosted operator) can rotate a credential, re-point a baseUrl, or add/remove an entry by rewriting the file, and every worker reprojects its registry with no restart. For Fabric that removes the container recreate — and the GTM drain — that a models change costs today (central-manager#753).

Boot publishes each entry as it is built, in config order — module factories keep their order-dependent composition (a cached wrapper still resolves its earlier base), exactly as before this PR. A reload stages changed entries off to the side through the same factories (constructBackend diverts the factory's install — including its helper registrations — via an AsyncLocalStorage capture scope), then one synchronous publish tick swaps them in conditionally, removes entries absent from the block (helpers cascade; a helper whose name a config entry claimed is suppressed, and restored the moment that entry is removed), and rebuilds fallback routing — so a request observes either the previous projection or the new one, never a partial rebuild. Reload runs factories for built-ins only: module-backed entries are restart-managed in both directions (add, change, and remove are refused with a warning), because a module factory may compose across entries and staged construction cannot honor that ordering — review decision with Kris. Applies are serialized; boot and reload coalesce latest-wins in separate lanes (a boot supersedes reloads observed before it — including a pending debounce snapshot, whose invalidation-on-boot guard is mutation-verified through an onSnapshotObserved test seam — while a reload arriving after still refines under reload rules), and a reload snapshot is validated against the same Joi modelsSchema boot enforces (hoisted out of configValidator, allowUnknown mirrored) — a snapshot boot would reject keeps the prior projection wholesale. One isolate-shared RootConfigWatcher drives it (getSharedRootConfigWatcher()), consumed by both logging and models — no second native watcher per worker.

Removal semantics are deliberate and partial-write-hardened. models: {} (or a shrunken block) removes built-in-backed entries (module-backed ones are restart-managed, above); models: null (a bare models: key) is rejected exactly as boot's validator rejects it, so a reload can never accept a file the next restart would refuse; a snapshot with no models key is a no-op, because a non-atomic in-place rewrite can be observed as a valid YAML prefix that hasn't reached the block yet, and adopting that would tear down every backend. The watcher additionally applies only the last event in a 150ms quiet window. The orchestrator contract is still an atomic tmp+rename write.

For the human reviewer

1. The compatibility gate is the design's hinge. If HARPER_SET_CONFIG / HARPER_CONFIG / HARPER_DEFAULT_CONFIG also names models, hot reload stays off — the watcher sees the raw file, env layers compose over it at boot, so the file alone is not authoritative for the block. This is deliberately also the migration story: a host-manager still injecting models through HARPER_SET_CONFIG keeps today's restart semantics untouched; one that writes the file instead (the follow-up HM change) gets live reload. Mixed fleets need no version negotiation.

2. Boot and reload deliberately differ in one respect. Boot overwrites whatever occupies a slot (the documented bootstrapModels contract); a reload swaps only against the instance the config projection itself installed, so a programmatic registerBackend that shadows a config entry keeps its documented precedence and a reload can never clobber it. Removal is equally guarded (removeIfCurrent).

3. Change detection is per-entry raw-JSON equality. An unrelated config edit (logging level, ports) re-runs no factories; a key-reorder inside an entry would spuriously rebuild it — harmless for built-ins (pure constructors), and module factories are only re-invoked when their own entry text changes.

4. Carried, not fixed — two ledger items for your judgment. (a) Helper (extras) ownership is publish-order and history dependent — inherent to conditional-swap semantics; the concrete failure modes found in review are fixed and tested, the abstract property stands. (b) bootstrapModels no longer rejects at boot on a wholesale failure (it logs at error and continues) — this matches the file's documented per-entry design, and the only reachable systemic throw is a non-YAML poisoned block, but it is a behavior change from main.

5. Review history, for the record. Seven implementation review rounds (codex/gemini/harper-domain via the pre-push CLI, plus two standalone Cursor-Grok legs after the CLI's per-branch Cursor cap was spent on the abandoned design). They produced 5 majors, ~20 minors, and one Grok blocker (partial-write adoption), all fixed with mutation-verified tests or declined with a stated contract rationale.

6. Planning history, for the record. Eight planning-review rounds on earlier designs (secrets-store credential rotation) never cleared the framing gate; the owner then clarified the actual goal — replace HARPER_SET_CONFIG injection with file writes that hot-reload — and that design was abandoned. What survived from it are the registry primitives (constructBackend / replaceIfCurrent / removeIfCurrent), which are exactly the staged-apply mechanics this needs.

7. Docs: user-facing behavior (the models block hot-reloads; env-layer pinning). Companion documentation PR to follow together with the host-manager change, since the operator guidance (write via tmp+rename, move models out of HARPER_SET_CONFIG) is about the orchestrator side.

Verification

Live, on a running instance — harper dev 5.2.5 from this branch, threads_count=2, isolated scratch HOME/rootPath; a component with the documented shape (import { models } from 'harper', a Resource.get() calling models.embed) and a stub provider recording every Authorization header Harper sends:

• Add the models: block to the live config file → /Embed/ went 500 ModelBackendNotFoundError → 200 in ~1s, stub saw the configured key
• Rotate apiKey (plain write) → requests carry the new bearer in ~1s
• Rotate again via tmp+rename (how an orchestrator should write) → ~1s
• 12 consecutive requests post-rotation: 12/12 on the new credential — both worker threads reprojected
• Remove the block → 500 ModelBackendNotFoundError in ~1s
• One pid throughout — the process never restarted

That live run predates the final review rounds; behaviors added after it (reload validation, models: null semantics, whole-key-absence no-op, the 150ms settle window, helper lifecycle) are covered by the unit suite below, and the file-watch path itself is exercised end-to-end by a real-chokidar test. Happy to re-run the live smoke on request before merge.

Unit — unitTests/resources/models/modelsConfigReload.test.js (28 tests): rewritten credential reaches a real embed() through resolveEmbedding (fetch-stubbed); unchanged entries keep their instance; removed entries stop serving while a present-but-malformed entry retains its last applied backend; application overrides survive reload and removal; a failed rebuild keeps the previous backend and its fallback routing; fallback routing updates with the block; rapid applies coalesce latest-wins (asserted with no settling re-apply, so a dropped queued update fails it); a poisoned block neither rejects into the watcher's fire-and-forget call nor blocks the next apply; the env-pin gate stays off under HARPER_SET_CONFIG; and a real-file chokidar end-to-end (temp config, getConfigFilePath patched without sinon, the shared waitFor helper). unitTests/resources/models/backendRegistry.test.js covers the capture scope (concurrent constructions stay separate; an unrelated registration mid-capture installs normally; capture survives a throwing factory) and both conditional primitives.

Mutation checks (against dist, which the tests actually load) — unconditional swap, disabled removal pass, disabled change-diff, disabled env gate, malformed-treated-as-absent, drain-catch removal, and dropped-queued-update each fail their targeted tests; baseline restored green.

Boot parity — the full models suite (431 tests, including the pre-existing bootstrap.test.js boot contract: overwrite semantics, fallback clear-on-rebootstrap, module specifier resolution) passes unmodified.

Suites that could not run locally: test:unit:components / test:unit:config fail at load in this checkout on pre-existing ESM-only dependency issues (chai, @harperfast/skills) — verified pre-existing by running untouched files; package-lock.json is unmodified. CI runs them.

Complexity: moderate

_{Review-Coverage: authored=claude; ran=codex; blocked=gemini(auth); declined=cursor-grok,cursor-composer,domain; rounds=20 @ 40052f3}

_{Human-Review-Need: 3 @ 40052f3}

[gemini-code-assist]

Code Review

This pull request introduces hot reloading for the models configuration block, enabling credential rotation and endpoint updates without requiring a process restart. It adds atomic conditional replacement and removal of backends using AsyncLocalStorage to capture registrations during construction, alongside a file watcher to apply updates. A review comment correctly identifies a high-severity issue where the isBoot flag can be lost during coalesced applies if a reload is queued immediately after a boot, and provides a robust suggestion to make the flag sticky within the coalescing window.

[kriszyp]

This looks good, and I approve of moving forward with this. However, I will note that this PR inherently creates an interesting situation: Currently it is targeting 5.3, and I think that is the right target for a PR of this size that is adding functionality. But that means you will still need to employ (thread) restarts on 5.2 servers to adopt backend config changes.
_{🤖 Reviewed with Codex}

[Devin-Holland]

Re the 5.2 note in Kris's review: agreed, and the host-manager follow-up is scoped exactly that way — HM only switches from HARPER_SET_CONFIG injection to file writes for instances whose core version advertises this capability (5.3+); 5.2 instances keep today's compose-recreate path. Mixed fleets need no negotiation because the env-layer gate in this PR keeps hot reload off wherever HM still injects models through HARPER_SET_CONFIG.

— Claude (devain)

[kriszyp]

Looks like one fix, but not a blocker. Sounds like we are in agreement on target.
_{🤖 Reviewed with Codex}

[claude]

Suggestion (non-blocking): this subscribes with a raw .on('ready', applyFromFile) rather than gating on the ready promise, the way harper_logger.ts's updateLogSettings() does (await rootConfig.ready before attaching 'change', at utility/logging/harper_logger.ts:171-175).

RootConfigWatcher#config (config/RootConfigWatcher.ts) flips from undefined exactly once per instance; only that first handleChange() fires 'ready', every later one fires 'change'. modelsConfigWatcher here is usually the shared singleton, and logging is typically the first consumer to construct it. If the shared instance's one-time 'ready' already fired (e.g. logger init resolving await rootConfig.ready before component loading reaches this line) before this .on('ready', ...) attaches, the event is gone for good — EventEmitter doesn't replay past emissions to a late listener — so this code never sees it.

Impact is narrow, not a correctness break: bootstrapModels(config) already applies the real initial snapshot independently of the watcher, and any write after the missed one still emits 'change' (which this code does catch), so it self-heals on the next config write. The only loss is a rewrite landing in the gap between Harper's boot-time config read and the shared watcher's first scan completing, with no further writes before the next restart.

Might be worth mirroring the ready-promise gating, or synchronously checking modelsConfigWatcher.config right after obtaining the shared instance and calling applyFromFile immediately if it's already populated, so a second consumer can't miss the one-time signal.