Fleet dev tools: doctor surfaces, upgrade verbs, watchdog, settings panel

[aarontrowbridge]

Important

Problem — The fleet runs five physical artifact surfaces (frozen server binary serving the chat UI, installed VSIX, vendored binary, staged skills, agent cards) with no shared state and no staleness visibility. Merged-to-main updates none of them; upgrading any is a manual chain whose runbook lives in shell-script headers. The 2026-08-23 session is the pain record: staleness noticed by a human, forensics to locate the serving artifact, fork sync → build → freeze → sidecar → kick, three interrupts each forcing state re-derivation.
Approach — Four layers per spec-20260823-094507: doctor v2 (six-record surface inventory, injectable probes, four verdicts, the JSON contract); upgrade verbs (receipt-emitting idempotent runbooks with git discipline, flock, rollback); a Settings panel that is a pure view over doctor's JSON; a notturno watchdog posting nightly drift. The UI is a view over the CLI, never a parallel implementation — headless SSH and the panel consume the same contract.
Approaches considered — (1) Watchdog only (report, no repair): leaves the manual chains; (2) auto-upgrade on drift: rejected — night never mutates the fleet, humans decide; (3) build the panel first with its own probes: the exact parallel-implementation failure mode the view-over-CLI invariant exists to prevent.
Scope — in: doctor v2 + the four verbs + receipts/lock + the panel section + the watchdog job/registry · out: scheduled/automatic upgrades, multi-machine fan-out, the fork's release automation.
Assumptions — single-operator fleet (the launchd user; panel and SSH are the same user); the frozen-binary discipline stands (never a live-path swap under a running server).

Acceptance Criteria

• doctor_surface_inventory_complete == 100% — six records, version + verdict + evidence, individual degradation
• doctor_json_schema_pass == 100% — committed schema (minItems 6), canonical round-trip
• doctor_verdict_matrix_complete == 100% — every cell asserts its expected verdict; all six unknown-capable; both server-binary stale mechanisms + server-down as separate fixtures
• upgrade_verbs_receipted == 100% — four verbs, hermetic fixtures + verification independence + the live dispatch proof (receipt cited in the PR; CI blocks hermetic, review blocks live)
• upgrade_idempotent_pass == 100% — stale → upgraded → no-op per verb, digest sets enumerated, receipt stores excluded by definition
• panel_renders_doctor == true — render function tests, verbatim facts, enabled for stale AND integrity-failure, disabled-property assertions
• watchdog_registered == true — renderDriftDigest pure function, zero-calls dry-run, fixture TOML with enabled=false

Testing Decisions

Hermetic vitest suites throughout (injected roots, fake binaries, pinned dates); the one stated live exemption is server-binary's sync/build chain, proven by the parent-run dispatch whose receipt lands in the PR. Authorship split per the house pattern.

Key Decisions

Digests and version strings, never mtime. Doctor's fetch is a source refresh, not a surface mutation. Never reset, never merge — divergence is human territory. The receipt store is the exit state of record. Night never mutates the fleet.

Prior Art

The 2026-08-23 session; amico-server.sh's runbook (becoming verbs); deploy-agents.mjs (the receipt pattern); doctor v1 (#402); ADR 0001; the 2026-08-08 mtime race.

Source

Durable record: ~/.amico/vaults/armonia-aaron-trowbridge/specs/spec-20260823-094507-fleet-dev-tools.md (approved: 3 manual critic rounds, round 3 clean; same-day editorial pass verified by inspection — no tool-verified seal).

[aarontrowbridge]

Progress (2026-08-23): slice 1 (doctor v2, #525) MERGED as PR #529 — six surface records, injectable probes, 39-test verdict matrix (25 + 14 reviewer adversarial variants; the independent reviewer found and fixed two real predicate bugs: one-directional digest loops judging drifted deployments as current). The JSON contract is live: #527 (panel) and amico#348 (watchdog) are unblocked.

Main was red mid-flight (#513 hand-edited the app-bundle overlay) — repaired by the book: fork tag v1.18.10-amicode.16 cut (opencode#238), overlay re-extracted + vendored pin bumped + goldens re-recorded + widget sources synced from the fork's re-theme (PR #531, merged; main green). Doctor v2 rebased onto the healed main before its merge.

Next: #526 (upgrade verbs — unblocked), then #527 + amico#348 in parallel. The server-binary live dispatch proof rides #526.

[aarontrowbridge]

Frontier progress (2026-08-23, evening): all three remaining slices implemented, parent-verified, green in CI.

• amico#349 (watchdog) MERGED — renderDriftDigest + snapshots + registry entry, 27/27 pytest, registry --check green.
• amicode#535 (upgrade verbs) — green in CI, held on the live-proof decision (the honest record is on the PR: the full live chain ran on the mini, the rollback machinery passed a real incident — launchd kick race, filed upgrade verb: kick race under launchd — wait for old pid exit, distinguish flap from boot #538 — and the surface is now current/healthy/verified; the happy-path 'upgraded' receipt is unobtainable while current. Human call: hold for the next natural staleness event, or accept incident+no-op as the proof).
• amicode#536 (panel) — green, queued behind Upgrade verbs: server-binary/extension/agents/skills — receipts, lock, abort paths, idempotence #535's decision (soft runtime dependency: upgrade buttons spawn the verbs).

Two live-dispatch bugs found and fixed by the real run (unmerged root defaults — fixtures can't catch live-only bugs; the record-name alias). The incident validated the verb's safety design end-to-end.

[aarontrowbridge]

Complete. All four slices merged: #529 (doctor v2), #535 (upgrade verbs — live proof accepted on incident evidence per Aaron's decision), #536 (panel), amico#349 (watchdog). The fleet dev-tools layer is live: amico doctor answers 'what's stale right now', amico upgrade <surface> runs the receipted chains, the Settings panel renders the inventory, and the nightly watchdog (amico registry, dispatch-proven gate pending its dry run) reports drift to the morning brief.

Follow-ups on the books: #538 (launchd kick race — the incident's root cause); the watchdog's dispatch dry-run before enabling cron (the house rule); the staged-skills predicate vs internal-skills staging disagreement (noted in the #526 checklist; a doctor-slice decision).

[aarontrowbridge]

All four slices landed; parent complete.