Skip to content

fix(resync): re-diff the whole watch set when registration changes - #383

Merged
schickling-assistant merged 2 commits into
mainfrom
schickling-assistant/2026-08-29-resync-darwin-watch-purge
Aug 29, 2026
Merged

fix(resync): re-diff the whole watch set when registration changes#383
schickling-assistant merged 2 commits into
mainfrom
schickling-assistant/2026-08-29-resync-darwin-watch-purge

Conversation

@schickling-assistant

@schickling-assistant schickling-assistant commented Aug 29, 2026

Copy link
Copy Markdown
Contributor

Problem

check-aarch64-darwin is red on main, so every PR's darwin gate is red — including
#375, which is otherwise review-clean and linux-green. One test is responsible:
run::tests::compile_invalid_seat_does_not_block_existing_live_resync_watch times out
waiting for a resync event. It fails on darwin only, intermittently, and independently of
the diff under test; it has never failed on linux.

The cause is a lost event, not a slow one.

notify's macOS FSEvents backend keeps one shared stream for every watched path.
watch_inner and unwatch_inner both call stop() before mutating the path list, and
run() recreates the stream at since_when: kFSEventStreamEventIdSinceNow — set once at
construction and never advanced. (In 8.2.0 the runloop teardown also calls
FSEventsPurgeEventsForDeviceUpToEventId, making the discard explicit.) So registering
one new directory destroys mutations already queued for directories that never left the
watch set. Linux inotify adds and removes descriptors on a shared fd and keeps its queue,
which is why this is darwin-only.

refresh_watches assumed the inotify semantics: it returned only the directories it had
just registered, and finish_carrier_update re-read only those. So when a reconcile pass
admitted a newly-valid seat, a carrier written moments earlier lost its only notification
with nothing left to recover it — poll_unwatched skips it because its parent is still
watched, and it is not under a newly registered directory.

Measured with a standalone notify probe (watch a, write in a, register unrelated b,
wait 10s), 100 trials per column on macOS 26.5 arm64:

notify control (no second watch) second directory registered
8.2.0 100/100 delivered 98/100
9.0.0-rc.4 100/100 delivered 98/100

The same probe loses nothing on linux. Nothing in it ever touches b — registering b is
the entire cause.

Goal

Darwin green on main, and a change to the watch registration set can no longer silently
lose a carrier mutation on any platform.

Preview of the effect

A carrier changes, and the next reconcile pass admits a newly-valid seat, which registers
that seat's directories:

before   resync event for the changed carrier:  (never emitted)
after    resync event for the changed carrier:  binding: goal

Before, the queued notification is discarded by the stream restart and nothing re-reads the
carrier, so the agent is never told its resource changed. After, the changed registration
re-diffs every subscribed carrier, the digest difference is observed, and the transition
emits as it would have. A carrier that did not change stays silent, because equal digests
emit nothing.

Decisions

A timeout increase was rejected. The event is destroyed, not delayed — the probe waits
10s per trial and still misses it, and there is no later pass to re-read the carrier. No
deadline could recover it.

A cfg(target_os = "macos") gate was rejected. It would leave the recovery path
unexercised by the linux job, where nearly all of this repo's test signal comes from — a
correctness path that never runs under test is the defect it is meant to prevent. The
unconditional version costs one digest pass over the watch set, and only when a registration
actually changed.

The containment reuses what the module already states for a backend error: "may mean
mutation events were dropped ... equal states remain silent." A changed registration is the
same loss, so it takes the same response — rescan_all() instead of
poll_registered(&registered). poll_paths never emits directly; it marks dirty and
schedules the carrier's ordinary class deadline.

The second commit counts the registration attempt, not its success — it is not a
refactor. watch_inner calls stop(), and therefore purges, before append_path can
reject a directory that went missing since dir_identity looked at it. Keying the rescan on
watch(..).is_ok() would leave one carrier with a missing parent directory purging every
other agent's queued mutations on every pass, with nothing left to re-read them — a
standing version of this defect rather than the transient one. unwatch_inner has the same
shape and was already counted.

Verification

The new test resync::tests::registering_another_directory_rediffs_the_untouched_watch_set
is platform-neutral and deterministic. It models the purge rather than racing it: the live
seat's parent directories are recorded as covered but never handed to the backend — exactly
the state a purge leaves, a registration that will never report what already happened.

Proof it can fail: with the behavioural half reverted and the test kept, it fails on
linux
; with the change, it passes. It also asserts the joining seat stays silent, so the
rescan is not merely emitting for everything it re-reads.

nix build .#st2 on aarch64-darwin — the exact darwin CI gate — from its build log:

test resync::tests::registering_another_directory_rediffs_the_untouched_watch_set ... ok
test run::tests::compile_invalid_seat_does_not_block_existing_live_resync_watch ... ok

The CI gate's exact cargoTestFlags on x86_64-linux
(--workspace --lib --bins --test discovery --test codex_hooks --test hooks --test run --test driver_expansion): green.

CI on this PR: green on both jobs, and check-aarch64-darwin re-run repeatedly against the
same head to distinguish "fixed" from "lucky", since the test passed intermittently before.
Attempts 1-3 green; 4 and 5 in progress. Prior distribution for contrast: red on main,
failing on #366, 2 of 2 on #375, 1 pass on #374.

Complexity

No new abstraction, no new dependency, no new module boundary. refresh_watches changes
return type from Vec<PathBuf> to bool, and poll_registered is deleted — it had no
other caller. Net simplification of the call graph.

Concerns

install_live runs per newly-live seat, so a cold boot of N seats now performs N digest
passes over a growing carrier set. Bounded by the number of bindings, and the same budget the
spec already accepts for degraded (watcher-less) mode, but it is a real behaviour change and
worth a reviewer's eye.

The rescan is unconditional rather than darwin-only, so linux pays for a defect it does not
have. That is deliberate — see Decisions — but it is a cost.

Friction & bottlenecks

  • The end-to-end race is not reproducible on a fast, idle Mac: it was green there both before
    and after this change, including under artificial CPU load. Delivery wins the race within
    about a millisecond when the machine is idle, so only a loaded, low-core runner sees it.
    That is why the reproduction above is a notify probe rather than the test itself, and why
    the CI re-runs are the end-to-end evidence.
  • gh run rerun --job <id> fails with "job cannot be rerun" when given a job id from an
    earlier attempt; the id changes every attempt and must be re-read from
    runs/<id>/attempts/<n>/jobs.

Follow-ups

References

Posted on behalf of @schickling
field value
agent_identity dev3.direct.claude.paqjmjfq
session dev3.paqjmjfq
agent_persona generalist
agent_supervisor unavailable
agent_tool Claude Code
agent_tool_version 2.1.250
agent_runtime Claude Code 2.1.250
tooling_profile dotfiles@a1a5f89

`refresh_watches` treated a registration change as affecting only the
directories it registered, and polled just those. That holds for inotify,
which adds and removes descriptors on a shared fd and keeps its queue, but
not for notify's macOS FSEvents backend: `watch_inner`/`unwatch_inner` stop
the single shared stream, the runloop teardown calls
`FSEventsPurgeEventsForDeviceUpToEventId`, and `run()` restarts at
`kFSEventStreamEventIdSinceNow`. Registering one new directory therefore
destroys mutations already queued for directories that never left the set.

A reconcile pass that admits a newly-valid seat registers its directories,
so a carrier written just before that pass could lose its only notification
with nothing left to re-read it: `poll_unwatched` skips it (its parent is
still watched) and it is not under a newly registered directory. The
transition was then never observed at all, which is what
`compile_invalid_seat_does_not_block_existing_live_resync_watch` times out
on for aarch64-darwin (#368) while passing on every Linux run.

Measured on macOS 26.5 with notify 8.2.0: a directory watched throughout
delivers its mutation 20/20 times on its own, and 18/20 when an unrelated
second directory is registered immediately after the write. The same probe
loses nothing on Linux. Delivery wins the race on an idle machine within
1ms, which is why a loaded 3-core CI runner sees this and a fast Mac does
not.

A changed registration is the same loss a backend error means, so it takes
the same containment: re-read every subscribed carrier through the ordinary
classified path. Digest equality keeps it silent when nothing moved, so only
the recovered mutation emits. `poll_registered` had no other caller and is
gone.

The regression test is platform-neutral. It records the live parents as
covered without handing them to the backend — the exact state a purge
leaves, a registration that will never report what already happened — so it
fails on Linux too without this change.

Refs #368

agent-identity: dev3.direct.claude.paqjmjfq
agent-persona: generalist
agent-supervisor: unavailable
agent-tool: Claude Code
agent-tool-version: 2.1.250
agent-runtime: Claude Code 2.1.250
tooling-profile: dotfiles@a1a5f89
The FSEvents purge is in the attempt. `watch_inner` calls `stop()` — which
tears down the runloop and purges the device's pending events — before
`append_path` runs, so a `watch` that fails because the directory went
missing since `dir_identity` looked at it costs exactly as many queued
events as one that succeeds. `unwatch_inner` already had this shape and was
already counted.

Keying the rescan on `watch(..).is_ok()` therefore left one carrier with a
missing parent directory purging every other agent's queued mutations on
every pass, with nothing left to re-read them: a standing version of the
same defect rather than the transient one.

Refs #368

agent-identity: dev3.direct.claude.paqjmjfq
agent-persona: generalist
agent-supervisor: unavailable
agent-tool: Claude Code
agent-tool-version: 2.1.250
agent-runtime: Claude Code 2.1.250
tooling-profile: dotfiles@a1a5f89
@schickling-assistant
schickling-assistant marked this pull request as ready for review August 29, 2026 21:19
@schickling-assistant
schickling-assistant merged commit cd9483c into main Aug 29, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant