fix(ding): say why a delivery was deferred instead of breaking silently - #375
Conversation
|
The Failing test: The same test, with the same assertion message, also fails on #366 — a different branch and an (The line numbers differ only because the two branches differ.) Flagging rather than fixing: the darwin resync-watch failure looks like it wants its own Posted on behalf of @schickling
|
|
Tracked already: #368 — "main is red on check-aarch64-darwin: So this is not merely pre-existing on other branches, it is red on Posted on behalf of @schickling
|
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ffd8aa8ec5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…verage gap A harness that locates its composer and still returns Ambiguous — an active turn, a modal, an unrecognised footer — was folded into NoMaintainedComposer, which says no harness could locate the pane and that it will not clear until one can. Both halves were false for a covered pane that may settle on its own. Per review on #375. agent-identity: dev3.direct.claude.paqjmjfq agent-persona: generalist agent-supervisor: unavailable agent-tool: Claude Code agent-tool-version: 2.1.250 agent-runtime: Claude Code 2.1.250 tooling-profile: dotfiles@a1a5f89
A deferral is the one outcome that delivers nothing and leaves nothing behind, and it had no log line at all: any pane the classifier could not positively type as empty-and-safe got zero delivery and zero diagnostic. PokeOutcome::Deferred now carries which cause produced it, flush_pending reports it to the watch loop, and the loop logs the edge only, so a pane stuck in one verdict costs one line rather than one per retry. agent-identity: dev3.direct.claude.paqjmjfq agent-persona: generalist agent-supervisor: unavailable agent-tool: Claude Code agent-tool-version: 2.1.250 agent-runtime: Claude Code 2.1.250 tooling-profile: dotfiles@a1a5f89
…verage gap A harness that locates its composer and still returns Ambiguous — an active turn, a modal, an unrecognised footer — was folded into NoMaintainedComposer, which says no harness could locate the pane and that it will not clear until one can. Both halves were false for a covered pane that may settle on its own. Per review on #375. agent-identity: dev3.direct.claude.paqjmjfq agent-persona: generalist agent-supervisor: unavailable agent-tool: Claude Code agent-tool-version: 2.1.250 agent-runtime: Claude Code 2.1.250 tooling-profile: dotfiles@a1a5f89
c13b666 to
5f08eee
Compare
Problem
flush_pendinghad exactly this:No log line, no state change, nothing written anywhere.
classify_composerreturnsDeferredforChanged | Ambiguous, so any pane the classifier cannot positively type as empty-and-safe got zero delivery and zero diagnostic: no text on the pane, no line in the log, nothing in the inbox to show a delivery had been attempted and abandoned.A deferral is the one outcome that both delivers nothing and leaves nothing behind. It was also the one outcome that said nothing, which makes it the only failure mode that cannot be noticed. In one observed deployment that silence covered an eleven-day delivery failure. Two supporting numbers from the same observation window: 913 delivery failures were recorded across the ding logs of a fleet of agent sessions (768
timed out after 2.0s, 145staged retry observation failed) — those at least produced a line; the deferrals produced none. Separately, a session whose pane no maintained harness recognised received a message and produced no pane text and no log line at all over three minutes.This is not a timeout problem, and this PR deliberately does not touch
PTY_COMMAND_TIMEOUT. Two sessions with 110+ timeouts each had empty inboxes, so timeouts alone do not produce backlog. The defect is the silence, not the duration.Goal
A delivery that performs no input says so, and says why — cheaply enough that it is safe to leave on, and specifically enough that an operator can tell a wait apart from a coverage gap.
Preview of the effect
Before, a session whose pane could not be classified produced no output at all — the loop broke and retried every 15 s, forever, silently. After, the first deferral prints one line naming the recipient and the cause:
The three read differently on purpose. The first two are waits — someone is typing, or a turn is in flight — and clear on their own. The third is a coverage gap: nothing about that pane is understood, and it will never clear without a harness change. That is the line worth paging on.
Volume: one line per verdict, not one per retry. A pane stuck in one state costs a single line no matter how long it stays stuck; a changed verdict, or the first deferral after delivery resumed, prints again. No pane text is ever logged.
Decisions
PokeOutcome::Deferredcarries the reason. "Deferred" with no reason is precisely the defect, so the outcome type is where the reason belongs rather than a side channel.Four reasons, because they want different responses.
ComposerChanged { harness }is a human drafting in a harness we understand.ComposerUnproven { harness }is a located composer that proved nothing about this screen — an active turn, a modal, an unrecognised footer. Both are waits.NoMaintainedComposeris a pane no maintained harness could locate at all, which never clears on its own.NoInputPerformedcovers the trait default. Folding the first three together would throw away the wait-versus-gap distinction that makes the line worth reading.No pane text is logged, ever.
Changedmeans a human's unsent draft may be on that pane. The diagnostic is the verdict plus the harness name, which is enough to separate the causes without putting someone's text in a log.The harness name comes from a new
classify_located_composer. The router already picked a harness and then discarded it; that name is the whole difference between a wait and a gap.Harnessgains aname(), andclassify_composerdelegates to the new function, so every existing caller is untouched.flush_pendingreports outward rather than logging inline. It has no recipient and no cross-tick state; the watch loop has both.FlushReportis a plain struct rather than anOptionon purpose, so the existing statement-position callers compile unchanged.Log the edge, not the level. Delivery retries at
DELIVERY_RETRY_BACKOFF(15 s), so a permanently unrecognised pane would otherwise emit roughly 5,760 identical lines a day.DeferralJournalreports only on change. Observation stays fail-open and off the delivery path: no I/O, no allocation, one comparison.Temporarily based on another branch — see Concerns.
Verification
CI: run 33272028700 —
check-aarch64-darwinpass 5m27s,check-x86_64-linuxrunning at time of writing (it passed at 57m03s on the identical tree in the previous run).Three new tests, each checked by inverting it and confirming it fails:
a_deferral_names_the_cause_that_produced_it— a human codex draft yieldsComposerChanged { harness: "codex" }; a located codex composer under a non-idle footer yieldsComposerUnproven { harness: "codex" }; an unrecognised renderer yieldsNoMaintainedComposer. Inverting the middle row fails withleft: ComposerUnproven { harness: "codex" }, which is the mislabel this PR removes.flush_reports_a_deferral_outward— a deferring poker setsreport.deferred; a delivering one leaves itNone.only_a_changed_deferral_verdict_is_worth_reporting— the second consecutive identical deferral reports nothing; a changed verdict, and the first deferral after delivery resumed, both do. Removing the dedupe fails this.Complexity
One enum with four variants, one
Displayimpl, a one-field journal struct with a singleobservemethod, and one plain report struct.PokeOutcome::Deferredgains a payload, touching six call sites.Harnessgains aname(), implemented once per harness as a string literal.The journal is the only piece that could be argued away, and the argument against removing it is quantitative: without it a stuck pane emits ~5,760 lines a day, which would get the logging turned off, which returns the system to the silence this PR exists to remove.
Concerns
Ambiguousfor a case that is really a wait, the operator sees "coverage gap" for something that would clear on its own. The distinction is now representable; keeping it accurate is ongoing work on the harness adapters.NoInputPerformedis reachable from thePokertrait default, so a poker that legitimately performs no input will log on its first deferral. Cheap, and arguably correct, but it is noise for a caller that never intended to deliver.was_stagedarm is still not reported. That arm relinquishes ownership and leaves the notice pending for a fresh attempt, so it is a state transition rather than a silent no-op. Scoping this PR to the unreported break at the heart of the defect kept the change reviewable; that arm is worth a second look.Why this PR is temporarily based on another branch
Base:
schickling-assistant/2026-08-29-resync-darwin-watch-purge(#383), notmain. Temporary, and unrelated to this change.check-aarch64-darwinwas failing here on #368 —run::tests::compile_invalid_seat_does_not_block_existing_live_resync_watch, which fails onmainand on unrelated PRs and touches nothing this PR changes. It reproduced twice on this branch (most recently job 99124668684, same assertion atsrc/run.rs:4484), each time with 538 passed / 1 failed and nothing fromding::among the failures. Linux was green throughout.#383 fixes #368: a lost-event bug in
notify's FSEvents backend, where everywatch/unwatchrestarts the shared stream atkFSEventStreamEventIdSinceNowand discards events already queued for directories that never left the watch set. Linux inotify keeps its queue, which is why only darwin saw it.Stacked via
gh stack link(stack #387): base ref only — no rebase, no force-push, and #383's branch was not touched. The two changes are disjoint (src/ding/*here;src/resync.rsand the resync spec there) andgit merge-treereports a clean merge. Verified that CI genuinely carries the fix rather than trusting the base ref: bothe2d4fc98(#383's head) and this branch's head are ancestors ofrefs/pull/375/merge.This PR returns to
mainas its base once #383 merges. It carries no dependency on #383 beyond the CI baseline.Friction & bottlenecks
pull_requestwith actionedited, and a bareon: pull_requestworkflow runs only foropened/synchronize/reopened. Verified rather than assumed: after the base change, the branch still listed only the pre-restack runs. Without a trigger the PR would have sat on a stale red result indefinitely. Cost one empty commit (chore: re-run CI against the new stacked base) to produce asynchronize; it can be squashed when the base returns tomain. Worth knowing for anyone else stacking to escape a red baseline.cargo test(all targets) intermittently failscodex_app_server::tests::runtime_owner_lock_is_nonblocking_and_released_on_closeon lock contention. Reproduces on a branch touching neither module, passes when the lib suite runs alone. Unrelated to this change; logged, not tracked.Follow-ups
was_stageddeferral arm noted under Concerns.NoMaintainedComposerat all — every one of those is a harness the adapters do not cover. This PR makes them countable for the first time, which is the prerequisite.References
Refs #368 — the darwin failure this PR was blocked behind.
Depends on #383 (stack #387) for the CI baseline only; base returns to
mainon merge.Posted on behalf of @schickling
agent_identitysessionagent_personaagent_supervisoragent_toolagent_tool_versionagent_runtimetooling_profile