A failed Codex turn is reported as two conflicting turns and stops accepting messages until the next idle

[schickling]

Problem

observe_turn_completed has no arm for Held{SystemError}, so a turn that ends in an error is
recorded as Held{ConflictingTurn} — two turns believed live at once. That is false, and it is
delivery-gating.

ConflictingTurn is then preserved by every other transition. observe_thread_status("active")
preserves it (src/codex_app_server.rs:773-781) and observe_turn_completed preserves it
(src/codex_app_server.rs:841-844). Only an idle status clears it.

maybe_request returns None for every Held{..} (src/codex_app_server.rs:415-421), so the agent
accepts no native delivery for as long as the wrong state stands.

Fleet evidence

We are a downstream consumer running one catalog with 627 seats (measured 2026-08-16, #261).

A usage limit is an account-level condition, so it hits every agent on that account at the same
moment. One recoverable error therefore converts a set of agents into agents that silently stop
accepting messages, and the state they report — conflicting turns — gives the operator no way to
tell why. The agents look busy. Nothing is running.

Stated as a mechanism, not as an incident count. We have not measured how many seats have hit
this. What is measured is the state machine: two independent captures, both ending in the same wrong
state, from an error payload that is ordinary.

Cause

src/codex_app_server.rs:826 (observe_turn_completed) enumerates arms for Idle,
Active{matching}, Held{Review|Compaction}, AwaitingStatus | Held{ActiveWithoutTurn}, and
Held{ConflictingTurn}. Held{SystemError} and Held{NotLoaded} match none of them and fall to the
trailing _ => Held{reason: ConflictingTurn, turn_id: None} (:845-848).

observe_thread_status mints Held{SystemError} on the "systemError" status word
(src/codex_app_server.rs:791-794), and a terminal error emits thread/status/changed → systemError
immediately before turn/completed. This is the ordinary path for a failed turn, not a corner case.

Separately, and worth fixing in the same pass: observe_thread_status's trailing _ arm
(:795-798) folds every unrecognized status word into SystemError. A status word Codex adds
tomorrow surfaces to an operator as a system error that did not happen.

Measured result

Two captured codex app-server streams from codex-cli 0.147.0, replayed through the actual
CodexControlState::observe in a cargo test harness. Both runs, independently captured, ended
identically:

t=  5655.3ms  thread/status/changed -> Held { reason: SystemError,     turn_id: None }
t=  5655.7ms  turn/completed        -> Held { reason: ConflictingTurn, turn_id: None }

The triggering error was ordinary:

{"error":{"message":"You've hit your usage limit…","codexErrorInfo":"usageLimitExceeded"},
 "willRetry":false}

and turn/completed carried "status": "failed" with the same error attached.

Version scope. The captures are codex-cli 0.147.0.
SUPPORTED_CODEX_CLI_VERSIONS = ["codex-cli 0.145.0", "codex-cli 0.146.0"]
(src/codex_app_server.rs:38, enforced at :2101), so st2 rejects the version these frames came
from. The defect is in st2's own match arms and does not depend on the wire version; the capture does.
Confirming the same systemError → turn/completed ordering on 0.145 or 0.146 would remove that
caveat.

Required behavior

A turn that ends in a system error resolves to a state that reflects the error. It does not report
conflicting turns. Whether that state gates delivery is a deliberate decision, not a fall-through.

Acceptance

• Held{SystemError} + turn/completed resolves to a state that names the error, not ConflictingTurn.
• Held{NotLoaded} + turn/completed likewise.
• Delivery eligibility after a failed turn is stated in the Codex spec, whichever way it is decided.
• observe_thread_status's unrecognized-status-word arm is distinguishable from a real systemError.
• Regression test replaying the captured systemError → turn/completed sequence.
• Existing conflicting-turn tests stay green.

---

[myobie]

I checked the upstream source for both Codex versions that st2 supports.

• codex-cli 0.145.0 peels to commit 25af12f7e61572b0bc18ddb1008be543b91519b0.
• codex-cli 0.146.0 is commit e363b08c9175ac1cbe5893615dd2cb9ddf95043b.

Both commits have the same relevant behavior. In codex-rs/app-server/src/bespoke_event_handling.rs, EventMsg::Error calls note_system_error. EventMsg::TurnComplete calls note_turn_completed.

In codex-rs/app-server/src/thread_status.rs, note_system_error sets has_system_error = true. note_turn_completed calls clear_active_state, which does not clear has_system_error. A later note_turn_started clears the error.

The upstream test is system_error_sets_idle_flag_until_next_turn. It checks that a new turn clears SystemError. This behavior is present in both supported commits.

The version caveat does not apply to the mechanism. A failed turn can leave the supported Codex versions in SystemError until the next turn starts. st2 currently blocks that next native turn after it converts the state to ConflictingTurn.

We have not observed this incident on the fleet. That absence does not remove the deterministic wedge.

[myobie]

Inconclusive synthetic result.

This test proves that maybe_request returns None at the turn/completed boundary while the state is Held { SystemError }.
It does not prove a real delivery wedge by itself.
A later provider idle notification would clear the hold and resume delivery.

I am checking the upstream Codex notification path to determine whether that later notification occurs.