Codex version gate is fatal where the OpenCode one degrades, and goes stale silently (0.150.1 refused)

[schickling-assistant]

Problem

SUPPORTED_CODEX_CLI_VERSIONS (src/codex_app_server.rs:46) admits codex-cli 0.145.0, 0.146.0, 0.147.0. The currently packaged Codex is 0.150.1. ensure_supported_version (:2593) enforces the list with anyhow::ensure!, so the mismatch is a hard failure: st2 driver codex exits 1 before it spawns anything.

Measured 2026-08-29 against st2 0.1.0+4c4e7f8, in a throwaway catalog:

$ st2 --catalog <scratch> driver codex \
    --identity <id> --runtime-id <rid> \
    -- codex --model gpt-5.6-sol -c model_reasoning_effort=low "canary"
Error: unsupported Codex app-server protocol version 'codex-cli 0.150.1'
       (expected one of: codex-cli 0.145.0, codex-cli 0.146.0, codex-cli 0.147.0)
exit 1

Nothing at all is written on this path — no status file, no harness-state record, no st2/codex state directory. The wrapper dies before it can publish presence or observed state.

This issue is not a request to relax the admission policy. The constant's doc comment is right that semver proximity is not compatibility evidence. It is about the failure mode of the gate and about the fact that it goes stale silently.

The gate is fatal, where the sibling gate degrades

opencode_session.rs:93 handles the same class of problem the opposite way:

st2 opencode-session: version <x> is unverified (supported: <y>); native delivery disabled

That seat keeps running. It loses a capability — native delivery — and falls back. The Codex gate instead costs the entire seat.

The asymmetry matters because of what sits on the other side of it. A seat wrapped in st2 driver codex is, by the mutual-exclusion rule, a seat that no longer carries a generic Ding sidecar: the driver owns delivery. So the fatal gate converts "this harness version is unverified" into "this agent does not exist", with its previous delivery owner already removed.

Suggested direction, in preference order:

1. Degrade like opencode_session. Refuse native delivery on an unverified version, log it loudly, publish a driver diagnostic saying so, and let the seat run. The seat is then merely un-upgraded rather than dead. This also makes the gate safe to adopt incrementally, which the current shape is not.
2. If the controlled app-server path genuinely cannot run at all on an unverified version, fail at admission rather than at launch — reject the spec when the catalog is validated/materialized, so the breakage surfaces to whoever publishes the seat instead of to the supervisor at 3am.

It goes stale silently, and it already has

• st2 refuses the locally installed codex-cli, and the version pin is now behind by one release #267 ("st2 refuses the locally installed codex-cli, and the version pin is now behind by one release") was filed and closed-completed on 2026-08-18, admitting 0.147.0.
• The downstream packaging left 0.147 on 2026-08-21, three days later, and is now on 0.150.1.
• So the gate has been out of date for eight days, and this is the second time the same defect has been filed.

The underlying shape is a hard gate on a fast-moving external version with nothing that notices when it drifts. Every Codex release silently re-breaks the driver, and — because nothing in a fleet currently routes seats through it — the breakage is invisible until someone tries to adopt it. #267 already observed that codex update invites an operator to take the harness out of service; that has now happened again without anyone running codex update deliberately.

Two things that would stop the recurrence independently of which fix above is chosen:

• A test that fails when the packaged/available Codex is outside the admitted set, rather than only unit tests over the constant itself. Today the constant can only be proven stale by running the driver.
• A published diagnostic on the unsupported path. driverDiagnostic reports status: "absent" for a seat killed this way, which is indistinguishable from "no driver declared". A seat refused for version reasons should say so in its diagnostic.

Consequence when a fleet adopts the driver

Concretely, on the fleet where this was measured, the plan was to route N=22 managed Codex seats through st2 driver codex — which also removes their Ding sidecar, since the two are mutually exclusive. Under a restart policy of attempts 3 / interval 60s / delay 0s / mode "fail" (a normal policy for a managed seat), a canary reproduced the end state:

restarted: <seat>          (x3, one per reconcile pass, ~31s apart, each exiting 1 in ~17ms)
ERROR st2::run: st2: GAVE UP on '<seat>' — crash-looping past its restart{} policy
  (mode=fail); leaving it parked and its last session for inspection.
  ... Fix the cause, then `st2 unpark <seat>` — no supervisor restart needed.

So the adoption outcome is: every converted seat crash-loops ~3x over ~2 minutes, then parks, and each one needs a manual st2 unpark after the cause is fixed. st2 doctor reports the seat correctly (task 'agent' alive — session dead/missing, presence missing, observed harness state absent), but the reconcile pass that gave up still reported result="pass" with report.errors=0; only st2.crash_loops moved 0 → 1. A fleet watching reconcile results would not see this.

A canary caught it, so nothing live was converted. The point of filing is that the next adopter has no reason to expect a fatal gate here, given the sibling driver degrades.

Why this is worth unblocking rather than routing around

Reading the driver past the gate, it looks like the most complete of the session drivers, and specifically well-built against the defect classes that make observability claims untrustworthy:

• Observed harness state is heartbeated, not written once. It is published per control-state transition and refreshed on the periodic wakeup — but only while harness evidence holds. On evidence loss it deliberately stops heartbeating so the record ages out to unknown rather than freezing at a stale value. That is the right behaviour and it is rarer than it should be.
• Delivery is idempotent and FIFO. Typed turn/start / turn/steer over the control stream with a stable clientUserMessageId, driven by an inbox watcher with a poll fallback, rather than by synthesized keystrokes.
• Restart is fail-closed. The thread id is re-attached from the persisted binding, and delivery state is reconciled against resumed thread history so a restart cannot replay an already-delivered message. Observed state is deliberately cold on a new incarnation instead of claiming continuity across an unobserved interval.

None of that is reachable while the version gate is fatal.

Adjacent, found while reading: expand_codex emits no render

driver::expand_codex emits only an argv node, unlike expand_claude which also emits a render block. A seat declared with the typed codex { … } form therefore materializes with no rendered workspace context at all — no agent instructions file, no hook configuration — even though examples/native/agent-codex.kdl shows a Codex seat needing exactly those, and hooks::required_by_codex expects the hook set to be installed.

Callers that bring their own render block are unaffected, which is probably why this has not bitten yet. Anyone adopting the typed-block form directly will hit it. Happy to split this into its own issue if you would rather keep this one focused.

Environment

• st2 0.1.0+4c4e7f8; also confirmed against main at cd9483c, which still lists the same three admitted versions.
• codex-cli 0.150.1.
• Linux.

Posted on behalf of @schickling

field	value
agent_identity	dev3.direct.claude.paqjmjfq
session	dev3.paqjmjfq
agent_persona	generalist
agent_supervisor	unavailable
agent_tool	Claude Code
agent_tool_version	2.1.250
agent_runtime	Claude Code 2.1.250
tooling_profile	dotfiles@a1a5f89