[#441] feat: Codex workflow orchestration — tier-1 in-harness fan-out for pair-loop - #469
[#441] feat: Codex workflow orchestration — tier-1 in-harness fan-out for pair-loop#469rucka wants to merge 10 commits into
Conversation
…degrade - New skill-convention: availability is established by probing the session, never inferred from a product name or a version; the vendor surface is one data structure; ceilings compose (min) and the skill adds none - Cross-referenced from graceful-degradation (a fifth scenario: the missing thing is an execution mechanism) and resolution-cascade (which resolves a declared VALUE, not a vendor mechanism) - Dataset mirror + README index row + llms.txt entry - Task: T-1 — KB skill-convention, harness realization cascade and capability probing Refs: #441
…rness realization - Surface map as DATA: both Codex multi-agent toolsets (handles, namespace, gating + bounding config keys, verified-against note). A vendor rename is an edit to this array and to nothing else - Probe > bind > announce > degrade: availability comes only from the tools the session exposes; a miss degrades to the external driver, else to the one-card path, never to an in-context multi-card loop - min(dependency, policy, harness) with the binding limit named; a cap of 0 is a no-dispatch iteration, a malformed ceiling stops the run - Context/role packets per phase: one card, role text in the request (no profile dependency), the return schema attached - Reviewer blindness as a pre-spawn rejection naming the offending entry, not a sentence in a prompt - Result contract = the workflow's own per-phase schemas; absent/unparseable/schema-invalid returns are a FAILED phase - Terminal outcomes (completed / failed-validation / timed-out / cancelled / died / not-started), partial batches collected, unknown fails closed - Audit appended AND read back — an unauditable run stops; resume rebuilds per-card phase state from the audit and never re-opens an existing PR - Ships as a generated KB asset (both copies byte-identical); the transpile moves to a shared build-kb-asset so two assets cannot be built by two compilers - 73 tests against mocked primitives + a parity guard asserting the phase schemas equal the workflow's - Tasks: T-2, T-4, T-5, T-6, T-7, T-8, T-9, T-10 Refs: #441
… in-harness branch - Step 1 resolves the realization by PROBING the session through the fan-out asset, prints which realization won and which primitive it bound to, and degrades in the declared order on a miss - New Step 1b: the Codex in-harness fan-out — resume from the audit, compose under min(dependency, policy, harness), dispatch each phase into a fresh subagent from an explicit packet, bounded waits, declared terminal outcomes, partial batches collected, audit-or-HALT - Claude branch untouched: the workflow is still delegated to verbatim - Boundaries gain three: never assert an unprobed realization, never iterate several cards in one context, never invent a second handoff format - Output format carries Realization + Parallelism lines; degradation cases added - version 0.1.0 -> 0.2.0 (new algorithm step + routing); description no longer says every non-Claude harness degrades - Task: T-3 — Realization resolution in pair-loop Refs: #441
…guards - Containment: the vendor's handles and config keys appear in the surface map and nowhere else — asserted over both skill copies and over the module's logic half; a renamed handle degrades fail-closed instead of leaking into a step - Zero merit logic: the Codex section carries no tier, no severity floor, no merge claim of its own, and the module declares no policy constant - Regression by absence: the Claude branch still delegates the whole run to the workflow, the cascade never binds Codex for a session without its tools, no `.codex/` distribution target exists, and the registry set is unchanged - Skill: the wait bound is referred to by role, not by the vendor's config key (containment) - Tasks: T-11, T-12 Refs: #441
…execution-layer ADL - ADR-021 §7 (new): a realization is PROBED, never inferred — product name, version string, documented-but-unobserved feature and a settable config key are all inadmissible; unknown reads as absent; the bound realization is announced and audited; degradation never drops the one-card-per-context invariant - The three tiers, their order and every other clause are unchanged — this AMENDS the existing record rather than writing a second three-tier framing - Trade-offs gain two: tier 1 now has two realizations to keep in step, and a model-driven orchestrator is less deterministic than a JS one - ADL 2026-07-11-agent-execution-layer: the claim that assistants such as Codex have no subagent primitives is struck and corrected with the probe evidence; the portability boundary it was really making (the .claude artifacts, not the capability) survives - architecture.md gains the Unattended Fan-Out current-state section - Task: T-13 — Decision record extending ADR-017 §4 Refs: #441
… terms registered - integrations/codex.mdx gains "Unattended Delivery (fan-out)": what must be on, the announcement a run prints, and exactly where it stops (probe miss degrades, unwritable audit halts, merge never automatic) - batch-engine.mdx: the pair-loop row no longer claims every non-Claude harness degrades; the Claude-Code-specific section now says what it is really about — the FILES, not fan-out - unattended-delivery.mdx: the stop-when-nothing-eligible note covers both in-harness realizations - collaborative-workflow.context.md registers harness realization, capability probe, context packet, role packet, result contract, terminal phase outcome - Task: T-14 — Documentation and ubiquitous-language registration Refs: #441
Verdict
PR: #469 · Author: rucka · Reviewer: independent review agent · Date: 2026-08-28 · Story: US-441 · Type: feature Classification matrix — per dimension
Tier = max(assessed) = risk:red, confirmed, not raised. Cost = yellow, confirmed (bounded LLM-call signal, every ceiling declared). AssessmentsSecurity — Input validationVerdict: red — the packet allow-list is the security control of this diff and it is bypassable by two ordinary path spellings. Details
Security — Output handlingVerdict: green — outputs are Security — AuthenticationVerdict: not applicable — no auth surface touched. Security — AuthorizationVerdict: yellow — the only authorization-shaped control in the diff is the reviewer allow-list, and it is the subject of Major 1 and Major 2. Merge authority correctly stays with Security — Introduced vulnerabilitiesVerdict: yellow — 0 conventional OWASP findings; 2 introduced control-bypasses in the story's own declared invariant. Details
CostVerdict: Architecture (Coupling)Verdict: yellow — the anti-corruption layer is real, not claimed. Bug fix — Red test before fixVerdict: not applicable — not a bug fix. The PR states the test-after ordering explicitly rather than hiding it; DetailsFindings by severityCritical (must fix before merge)
Major (must fix before merge)
Minor (must fix before merge — same bar as Major, just lower impact)
Questions (informational, never blocking)
Positive feedback
Functionality & requirements (AC coverage)
Testing & quality gates
Adoption compliance
|
…oped halts - blindness: `working_path` override is an input (default `.pair/working`), and an attachment escaping the project is rejected — both spellings reached the reviewer - collect: a caller schema TIGHTENS the phase contract, never replaces it; enums enforced; an unknown phase is `failed-validation`, not a TypeError - converge: new command — findings owe the fix, every fix is re-reviewed, capped at 3 rounds then escalate; resume no longer owes a phantom `fix` - resume: a halt is scoped to its run and retired by a later success of the phase - bind: the probed namespace applies only to an entry declaring `namespaceKey` - docs: v1 already contributes a ceiling; convention states unverifiable ⇒ not mapped - ADR-021 §8: what "one lane" fixes for both realizations Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…request shape fails closed Claude Code's `Workflow` was missing from the surface map, so the `bind` every session is told to run answered a Claude session `degraded-one-card`: a false announcement printed and audited on a run that fans out through the workflow, and a degradation branch whose stated condition held at the same time as the in-harness one. Handles are now a union discriminated by dispatch shape (spawn-wait / delegated-run), both realizations are entries in one map, and the skill branches on the returned `dispatch` rather than on a product name. Three silent request-shape losses closed the same way — loudly: `workingPath`/`worktreeRoot`/ `findings` are read at the request root as well as inside `packet`, unknown keys are rejected at every level, a `fix` packet with no findings is refused, `bind` returns the wait bound (value or config keys) the caller must apply, and the audit takes a `kind:"run"` record so the announcement needs no invented card id. Refs #441
… last two silent drops
- audit: a `review`/`completed` record without `action` in {converged,fix,escalate}
is refused at write; on read only `converged` closes the cycle, so an omitted
stamp re-enters the card at `review` instead of reaching the merge gate unfixed.
- surface map: v1 declares NO wait-timeout key (it has none — verified on
codex-cli 0.150.1); every spawn-wait entry declares a fallback bound, `bind`
always returns a number plus `waitTimeoutSource`, announced and audited.
- CLI: unknown keys rejected inside `ceilings` and `probe` too, as at packet/card.
- ADR-021 §8 gains the fail-closed-on-model-omission property + the fallback trade-off.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UQJzGMhRqBRRboxMrRqFPP
This comment has been minimized.
This comment has been minimized.
Close audit, blindness, ceiling and command-contract gaps.\n\nRefs: #441
rucka
left a comment
There was a problem hiding this comment.
Verdict
risk:red · cost:green — APPROVED
Open findings: 0. All Major/Minor findings from prior rounds are resolved and verified on the current head
cf967d999c3f1d90a32560013d22ce916214456b.
PR: #469 · Story: #441 · Author/reviewer: rucka · Date: 2026-08-30
risk:red still requires a non-author human approval on this exact head before merge; this self-review cannot satisfy that authorization requirement.
Classification
| Dimension | Assessment | Result |
|---|---|---|
| Criticality | delivery orchestration and audit trail | yellow |
| Change risk | fan-out lifecycle, resume safety, generated distributed asset | red |
| Business impact | repository-wide autonomous-workflow control | yellow |
| Security | command policy remains deny-first; no auth/trust-boundary change | green |
| Coupling | existing knowledge-hub tool plus intentionally synchronized asset/mirror | balanced |
| Cost | no billable/external-resource surface | green |
Existing labels remain correct: risk:red, cost:green.
Findings
Critical
None.
Major
None.
Minor — resolved
| Finding | Resolution | Verification |
|---|---|---|
| Audit history could be associated with an unstamped run | Non-empty run is now required for audit records and named resume |
TDD RED → GREEN |
| Reviewer prompt could see only attachments | Deny-first blindness now also rejects card title and notes | TDD RED → GREEN |
| Binding omitted the selected harness concurrency key | Binding returns concurrencyKey; loop guidance halts fan-out when the ceiling cannot be resolved |
TDD RED → GREEN |
| ADR property count contradicted its list | Corrected three/four mismatch | Documentation review |
Missing cap produced a raw TypeError |
Ceiling validation returns a normalized domain error | TDD RED → GREEN |
Questions / residual scope
The story's manual Codex dogfood DoD remains unexecuted because it would act on a real eligible card. It is explicitly recorded as external side-effect scope, not misclassified as a Major/Minor.
Review evidence
| Area | Result |
|---|---|
| Functionality / AC | Covered, including audit identity, resume safety, reviewer blindness, concurrency resolution, and error normalization |
| Test-first remediation | Six focused red tests failed before source changes and pass after them |
| Target tests | @pair/knowledge-hub: 51 files, 5,064 tests passed |
| Generated asset | codex:asset regenerated; source and packaged asset/mirror verified together |
| Quality gate | pnpm quality-gate passed on exact head |
| CI | build, preview, secret scan, smoke: passed |
| Security | Green — deny-first command policy preserved; no new authorization or vulnerability finding |
| Adoption | Level 1 conformant; ADR aligns with implementation; no new dependency or migration |
| Debt | None introduced |
| Documentation | Loop skill, packaged mirror, Codex integration docs, and ADR updated consistently |
| Performance / deploy | No runtime hot path, migration, or deployment impact |
Why prior reviews kept finding issues
Earlier fixes closed only the directly reported representation: attachment text but not card title/notes; configuration map but not returned binding; audit record prose but not resume identity; source but not every distributed representation. The new tests cover these sibling boundaries and the shipped asset, so the review evaluates the whole contract rather than a single path.
|
Synthesis of the whole review↔fix cycle, replacing the round-4 escalation flush (now minimized, per the convention that comment itself stated). Round-by-round detail stays in the untracked working log Convergence — round 4 (commit
|
| round | commit | verdict | outcome |
|---|---|---|---|
| 1 | ca8600e7 |
fix | 5 Major + 5 Minor — blindness workingPath input, attachment .. rejection, tightening-only schema override, run-scoped halts, converge + fix-round cap |
| 2 | c5eb965d |
fix | 2 Major + 3 Minor — dispatch-shape realization map (no product-name special case), root/packet/card key handling incl. findings, probed wait bound, AuditRecord.kind |
| 3 | 3ba02891 |
fix | 2 Major + 1 Minor — the review action stamp, v1's declared wait-bound fallback + waitTimeoutSource, unknown probe/ceilings keys |
| 4 | cf967d99 |
converged | the 5 findings above; independent re-review on this head returns 0 open Critical/Major/Minor |
Verification on the current head cf967d99
| Gate | Result |
|---|---|
pnpm quality-gate |
PASS (exit 0) — re-run at publish time |
CI: build, smoke, secret-scan, preview |
PASS |
pair-review status on head |
success — APPROVED — 0 open Major/Minor findings |
| Codex suites | 167 passing (codex-fanout.test.ts 126, codex-fanout-asset.test.ts 23, codex-realization.test.ts 18) |
@pair/knowledge-hub |
5,064 passing |
The PR description has been refreshed to describe this head (the round-4 semantics above, and the corrected counts) rather than the round-3 one.
PR state — pr-state:to-be-reviewed is correct, not stale
Computed with the shipped, provider-agnostic evaluator rather than asserted:
resolve_tier "risk:red,pr-state:to-be-reviewed" → red
resolve_pr_state pass approved red 0 → to-be-reviewed
(pr-state: tier 'red' requires explicit human approval)
merge_allowed to-be-reviewed → NO
Green gates + an APPROVED review at 🔴 with no explicit human approval on the current head synthesizes to to-be-reviewed — the fifth row of the synthesis table in pr-states.md. The label therefore stays as it is: the review is converged, the human approval is what remains. This repo is single-maintainer and Review enforcement is disabled, so pair-review here is advisory and the 🔴 approval rule is a convention, not a branch-protection block — which makes this line the operative one rather than a formality.
Nothing has been merged. The merge is a human act.
7b55900 to
adb9627
Compare
PR Information
PR Title: [#441] feat: Codex workflow orchestration — tier-1 in-harness fan-out for
pair-loopStory/Epic: #441 · Epic #212 — Supervised automation
Type: Feature
Priority: High
Assignee: rucka
Labels:
risk:red(cost: yellow)Summary
What Changed
pair-loopgains a second tier-1 fan-out realization. It probes its session for a fan-out primitive, binds the first realization the surface map confirms, announces it, and — where the binding dispatches by spawn/wait — drives the same implement → PR → review ↔ fix lane through the harness's own subagent tools instead of degrading to one card per invocation.Everything in that lane that is a rule rather than a judgement lives in one tested module,
packages/knowledge-hub/src/tools/codex-fanout.ts, shipped as the generated KB asset.pair/knowledge/assets/codex-fanout.cjsand invoked as a 7-command CLI (bind,cap,packet,collect,converge,audit,resume). Only the harness tool calls stay with the model. Same pattern as the coverage ratchet (ADL2026-07-13-gate-tooling-code-in-tested-modules+ ADR-023) — not a new one, and not a ported workflow: Codex has no workflow runtime, its orchestrator is the model.The orchestrator being a model is the design constraint the whole module answers to. Every request it composes and every audit line it writes is something a killed or careless run can get wrong, so each one fails closed: an unknown key is rejected rather than dropped, a missing required field is refused at write time rather than defaulted, and every ambiguous read resolves to "more work owed", never to "done".
Why This Change
Before this branch, a Codex session ran the delivery process one card per invocation while a Claude Code session fanned out — the same policy file, two different throughputs, and the difference documented as a permanent property of the harness. It is not: the probe found two multi-agent toolsets exposed. ADR-021 gains §7 and the ADL that asserted otherwise is corrected in place.
Story Context
User Story: As the delivery process, I want the unattended loop to run context-safely in whichever harness the session is in, so that the same cards reach the same gate with the same audit regardless of the product driving them.
Acceptance Criteria: AC1–AC14 (story body). AC11 and AC13 are guarantees of absence and are asserted mechanically, not claimed — see Reviewer Guide.
Changes Made
Implementation Details
Workflowand both Codex toolsets are entries inHARNESS_SURFACE_MAP, differing only in dispatch shape:delegated-run(one call hands the whole run to a runtime that fans out itself — no cap, no wait bound for this orchestrator) vsspawn-wait(this orchestrator starts and awaits each subagent, so it owns both).bindreturns the shape; the skill branches on it, never on a product name. A vendor rename is an edit to that array and nothing else — asserted by a containment test.ProbeObservationhas fields for exposed tool names, a reported namespace, a concurrency ceiling and a wait timeout — and no field for a product name or a version, so the shape is the rule. An unrecognised probe reads as absent and the cascade degrades: external driver, else one card + continue-token. Never an in-context multi-card loop.bindresolves the bound in three steps and always terminates in a number for aspawn-waitbinding: the session's reported value, else a config key the entry names, else the entry's declared fallback — withwaitTimeoutSourcenaming which, and the announcement carrying it into the audit. The generation a bare Codex session binds has no wait-timeout config key at all, so "read the harness's configured maximum" resolves to nothing there; leaving the caller withnullmet AC10 on paper and not on the common path. Verified againstcodex-cli 0.150.1per entry, inverifiedAgainst.packetbuilds one card's packet — role text in the request (no harness profile required), the card, its worktree, the skill, the return schema, and the findings the role must act on. A review packet carrying anything under the project's resolved working area is rejected before any spawn — in an attachment or in the card's owntitle/notes, because the reviewer receives the card verbatim and a pointer in prose is as harmful as one in a path. Afixpacket with no findings is rejected too.collectvalidates every return against the built-in phase contract, plus the project's generated review contract when it is on disk and fresh. The override tightens only. A conformance guard slices the workflow's ownSTEP_SCHEMA/PR_SCHEMA/LOOSE_REVIEW_SCHEMA/FIX_SCHEMAout of.claude/workflows/pair-implement-batch.jsand asserts deep equality, so a divergence fails on whichever side moves first.convergepartitions a review's findings, converges on zero actionable ones (no fixer is spawned on an approved PR), dispatches one fix round followed by a re-review, and escalates at the cap.OWED_PHASESexcludesfix, so a resumed card re-enters atreview. The decision is carried on the review's audit record asaction, and both halves are mechanical:auditrefuses a completedreviewrecord whoseactionis notconverged/fix/escalate, and the resume reconstruction closes the cycle onconvergedalone.capreturnsmin(dependency, policy, harness)and which of the three bound it, and it owns no policy knob — asserted. When the probe reports no harness ceiling,bindhands back theconcurrencyKeythe entry declares so the caller can read it rather than invent a number; a key it cannot use halts the fan-out instead of guessing. A ceiling nobody observed is never silently a ceiling of 1.capcalled without aceilingsobject answers with a command-level error naming the fields it needs, never a rawTypeError.auditappends and reads back; an unwritable audit HALTs the run. Every record — card or run — must carry a non-emptyrun, refused at write time and again on a namedresume, because a halt is scoped to the invocation that recorded it and an unstamped line cannot be placed: inferring the boundary from a later caller is how one run's escalation is read as another's history.resumereconstructs per-card state from the audit alone and retires a halt on a later success of the same phase. Records arekind:'card'(the default, naming a card) orkind:'run'(the realization announcement — no card id to invent).packet,card,probeandceilings. A dropped key reinstates exactly the default it was passed to override — a droppedharnessCeilingdispatches the policy's parallelism into a harness that allows less, a droppedworkingPathhands the author's checkpoint to the independent reviewer.Files Changed
packages/knowledge-hub/src/tools/codex-fanout.ts+.test.ts(126),build-codex-asset.ts,build-kb-asset.ts(extracted generic asset builder),src/conformance/codex-fanout-asset.test.ts(23),src/conformance/codex-realization.test.ts(18),.pair/knowledge/assets/codex-fanout.cjs(+ dataset mirror),.../skill-conventions/harness-realization.md(+ mirror).claude/skills/pair-loop/SKILL.md(+ dataset source, 0.1.0 → 0.2.0),build-ratchet-asset.ts(now delegates; output byte-identical, own guard),adr-021-fan-out-three-realizations.md(§7, §8, tier row, 5 trade-offs),decision-log/2026-07-11-agent-execution-layer.md(factual correction),tech/architecture.md(## Unattended Fan-Out),collaborative-workflow.context.md(6 terms),skill-conventions/README.md+graceful-degradation.md+resolution-cascade.md+llms.txt(registration),docs/integrations/codex.mdx,docs/reference/batch-engine.mdx,docs/tutorials/unattended-delivery.mdxTesting
Test Coverage
codex-fanout.test.ts— 126 cases against mocked harness primitives and an in-memory filesystem double. CI needs no Codex binary and no network: the far side of this integration is observably in motion, and a suite that depended on the vendor would go red for reasons that are not this repo's.codex-realization.test.ts(18) — containment, zero-merit-logic, regression-by-absence.codex-fanout-asset.test.ts(23) — asset drift, CLI smoke against the shipped artifact, result-contract parity with the workflow. The three suites are 167 cases at this head; every review finding across four rounds carries a case, and the exploit reproductions run against the shipped asset rather than the source.Test Results
Pre-merge tiering: disabled⇒ the full suite runs regardless of tier, which is a superset of the 🔴 set this story's classification requires.Testing Strategy
action; an audit line with norunstamp; a working-area pointer smuggled intocard.title/card.notes;capinvoked with noceilingsat all; a misspelled key in any of the five request objects.Quality Assurance
Review Areas
assertBlind+assertBlindCardText+blindDenyPrefixes) — it is the AC5 guarantee, and it takes the project's resolvedworking_pathas an input. A parent-relative attachment is rejected outright rather than prefix-matched, and both prose channels of the card are scanned, not attachments alone.collect's override semantics — the party composing that JSON is a model, so the override tightens and never replaces.runstamp on every audit record — the invocation boundary is the only thing that keeps one run's halt out of the next run's reconstruction, so it is required on write and on a named resume rather than inferred from the caller.converge+ the review audit record — the loop bound is the loop's own; it decides nothing about what to work on. Worth checking as a pair:convergereturningfixand the audit line recording it are two separate acts by the same non-deterministic party, and the second is now refused when it omits the decision.Documentation
2026-07-11-agent-execution-layerADL's "Codex does not have subagent primitives" clause is struck and corrected with the probe evidence.skill-conventions/harness-realization.md— probe → bind → announce → degrade; the surface map as data; the map holds every realization; dispatch shapes; the bind returns a bound the caller can always apply (three-row resolution, plus: bounding keys belong to the entry that owns them, andVerified againstcovers them too); ceilings compose.collaborative-workflow.context.md.Risk Assessment
verifiedAgainstnote per entry; a rename is a one-line edit, and a probe miss fail-closes to a lower tiertimed-out, the card halts for that run only, and the next run re-drives it. The alternative — an unbounded wait — hangs the unattended run outrightReviewer Guide
Review Focus Areas
git diff --stat origin/main...HEADshows no file under.claude/workflows/or.claude/agents/, andcodex-realization.test.tsasserts a Claude session bindsclaude-code-workflow/ tier 1 /delegated-runand that the workflow's argument list ({ policyText, root, overrides, predicateOverride, startIteration, tagProjectionFamily }) is untouched..codex/target exists; asserted againstapps/pair-cli/config.json's real registry set, which is also asserted unchanged.bindon a barespawn_agent/wait_agentprobe must return a positivewaitTimeoutMs.Testing the Changes
Dependencies & Related Work
pair run, the tier-2 external driver) — it wrote ADR-021's three tiers and already reserved the tier-1 slot this story fills. This branch amends that ADR; it does not write a second, divergent three-tier record.implement-batch, the per-card pipeline this skill composes and never re-implements), ADR-017 §3 (context isolation, the invariant every tier preserves).2026-08-20-a-changeset-is-added-when-a-release-is-scheduled.This description reflects the current head commit, not a round-by-round history.