Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion agents/test-quality-auditor.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,20 @@
name: test-quality-auditor
description: Read-only verifier that audits one task's diff and tests for quality. Invoked between self-review and done so the session that wrote the code does not grade its own tests (self-grading guard). Returns a fixed VERDICT and REASONS.
tools: Read, Grep, Glob, Bash
model: inherit
model: fable
---

You are an independent test-quality auditor for loop-orchestrator. You DO NOT
modify code or tests — you are read-only. Your only job is to judge whether the
tests genuinely verify the change.

> This agent's model is **pinned** rather than `inherit`. It is the self-grading
> guard, so it must not follow the worker down: when a worker is pinned to a
> cheaper tier (`DEV_LOOP_WORKER_MODEL`, see `skills/orchestrate/scripts/`), an
> inheriting auditor would grade that worker at the worker's own tier — writer
> and grader sharing blind spots is the exact failure this agent exists to
> prevent. Raise the pin, never lower it.

Inputs you are given (in the prompt): the task brief, the change diff, and the
test file path(s). If any are missing, ask for them rather than guessing.

Expand Down
5 changes: 3 additions & 2 deletions skills/loop-implement/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,9 @@ tasks in the plan's order, loading exactly the wiki pages each task names, and
citing them — so every change traces back to a verified wiki page.

It works the same whether it runs standalone (you produce the plan here in step 2)
or as an orchestrated worker (the orchestrator hands you a task brief and you plan
+ implement it here). Either path, one loop.
or as an orchestrated worker (the orchestrator hands you a task brief *and* the
plan it already produced, and you adopt and implement it here). Either path, one
loop.

## When to use
- Use: logic changes, new features, bug fixes, behavior-changing refactors.
Expand Down
22 changes: 22 additions & 0 deletions skills/orchestrate/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -336,6 +336,28 @@ reasoning-effort flags) that `worker-start` cannot express.
tools (the plan step is fixed to `wiki-plan`, not a configurable role), and for
a UI-facing task fill `<design_spec>` with the `design` role's pulled spec
(Phase 2) — then

**2a. Plan it yourself, here, before launching.** Invoke the bundled `wiki-plan`
skill for this task and write the result to `plans/<task>.md`. Planning runs in
THIS coordinator session on purpose: a worker can be pinned to a cheaper tier
(`DEV_LOOP_WORKER_MODEL`), and a plan is where an unmade decision becomes the
implementer's guess — so the plan must come from the strongest model in the run,
not from whatever tier is executing. Every design decision must be made and
grounded in a `wiki/` page (record the decision->page map); leave nothing "as
appropriate". The worker then ADOPTS this plan (session-prompt §1 / O1) instead
of authoring one, and still signals `plan_ready` — so the phase sequence, the
`plan_ready` watch, and the ready-set scheduler are all unchanged.

Because planning happens here, **the planning model is whatever model this
coordinator session is running**. There is no separate setting to turn: to plan
on a stronger tier than you implement on, start the coordinator on that tier
(`claude --model <planning model>`) and leave `DEV_LOOP_WORKER_MODEL` pointed at
the cheaper implementer tier.

A worker that reports the plan is contradictory or under-decided is telling you
the planning pass was wrong: fix `plans/<task>.md` here and re-send §1. Do not
let the worker re-plan — that silently moves planning back onto the worker tier,
which is the thing this step exists to prevent. Then
`LO_STATUS_DIR=<abs status dir> LO_TASK_ID=<task> scripts/launch-session.sh
lo-<n> <worktree> bypassPermissions "<plan prompt>"`
(plan prompt = templates/session-prompt.md §1 — the tmux set — with the
Expand Down
13 changes: 13 additions & 0 deletions skills/orchestrate/scripts/launch-session.sh
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,8 @@
# LO_DRY_RUN print the resolved session name and exit
# LO_TMUX tmux binary (default: the one on PATH)
# LO_CLAUDE claude binary (default: the search below)
# DEV_LOOP_WORKER_MODEL model the WORKER runs (e.g. claude-sonnet-5). Unset =
# omit --model, so the worker inherits the configured model.
# LO_READY_TIMEOUT / LO_READY_INTERVAL REPL-ready budget (default 60 / 2);
# attempts = floor(timeout/interval), minimum 1
# LO_READY_EXTRA / LO_TRUST_EXTRA extra screen-match substrings
Expand Down Expand Up @@ -73,6 +75,16 @@ case "$perm" in
*) echo "launch-session: invalid permission mode '$perm'" >&2; exit 2 ;;
esac

# Same guard for the worker model: ids/aliases are alphanumerics plus . _ - and
# the [1m] context suffix (e.g. opus[1m]). Unset = omit --model entirely, so the
# worker inherits the user's configured model.
model="${DEV_LOOP_WORKER_MODEL:-}"
if [ -n "$model" ]; then
case "$model" in
*[!A-Za-z0-9._\[\]-]*) echo "launch-session: invalid model '$model'" >&2; exit 2 ;;
esac
fi

# Resolve-only mode: print the effective session name and exit before touching
# tmux/claude. Lets the orchestrator (and tests) learn the exact name.
if [ -n "${LO_DRY_RUN:-}" ]; then echo "session=$session"; exit 0; fi
Expand Down Expand Up @@ -125,6 +137,7 @@ if [ -n "$esc" ]; then
launchcmd="$launchcmd && export GROUNDWORK_ESCALATION_DIR='$esc' && export GROUNDWORK_TASK_ID='$session'"
fi
launchcmd="$launchcmd && \"$CLAUDE\" --permission-mode $perm"
[ -n "$model" ] && launchcmd="$launchcmd --model '$model'"
"$TMUX_BIN" send-keys -t "$session" "$launchcmd" Enter

# Pass trust screen + permission warning, wait for REPL ready (~60s).
Expand Down
16 changes: 15 additions & 1 deletion skills/orchestrate/scripts/orca-spawn.sh
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,9 @@
# ORCA_BIN orca executable (default: orca)
# GROUNDWORK_ESCALATION_DIR exported into the worker so an `ask` escalates
# GROUNDWORK_TASK_ID worker task label
# DEV_LOOP_WORKER_MODEL model the WORKER runs (e.g. claude-sonnet-5).
# Unset = omit --model, so the worker inherits the
# user's configured model (unchanged behavior).
# LO_READY_TIMEOUT seconds to wait for TUI readiness (default 60)
# ORCA_SPAWN_DRYRUN=1 print the orca commands instead of running them
# ORCA_SPAWN_CREATE_JSON canned `terminal create --json` (tests)
Expand All @@ -32,6 +35,15 @@ case "$perm" in
*) echo "orca-spawn: invalid permission mode '$perm'" >&2; exit 2 ;;
esac

# Same for the model: ids/aliases are alphanumerics plus . _ - and the [1m]
# context suffix. Reject anything else rather than sanitizing it.
model="${DEV_LOOP_WORKER_MODEL:-}"
if [ -n "$model" ]; then
case "$model" in
*[!A-Za-z0-9._\[\]-]*) echo "orca-spawn: invalid model '$model'" >&2; exit 2 ;;
esac
fi

rt="${LO_READY_TIMEOUT:-60}"; [ "$rt" -ge 1 ] 2>/dev/null || rt=60
timeout_ms=$(( rt * 1000 ))

Expand All @@ -44,7 +56,9 @@ env_prefix=""
if [ -n "${GROUNDWORK_ESCALATION_DIR:-}" ]; then
env_prefix="export GROUNDWORK_ESCALATION_DIR='$(esc_sq "$GROUNDWORK_ESCALATION_DIR")' && export GROUNDWORK_TASK_ID='$(esc_sq "${GROUNDWORK_TASK_ID:-}")' && "
fi
worker_cmd="${env_prefix}claude --permission-mode ${perm}"
model_arg=""
[ -n "$model" ] && model_arg=" --model '$(esc_sq "$model")'"
worker_cmd="${env_prefix}claude --permission-mode ${perm}${model_arg}"

print_cmd() { printf 'orca'; for a in "$@"; do printf ' [%s]' "$a"; done; printf '\n'; }
orca_run() { # $1 = fatal flag (1 = return non-zero on failure); rest = orca args
Expand Down
32 changes: 28 additions & 4 deletions skills/orchestrate/scripts/orca-worker-start.sh
Original file line number Diff line number Diff line change
Expand Up @@ -32,17 +32,26 @@
#
# When GROUNDWORK_ESCALATION_DIR is NOT set, worker mode falls back to Orca's
# composed agent-first `worker-start --agent`, which also accepts new-child /
# new-top-level and adds no fallback shell.
# new-top-level and adds no fallback shell. `--model` is inert on that path for
# the same reason `--permission-mode` is: Orca builds the agent command itself,
# so there is nothing of ours to append to. Set the escalation dir (which
# orchestrate always does) to get a model-pinned worker.
#
# usage:
# orca-worker-start.sh --task <task_id> --worktree <selector> --agent <agent>
# [--name <name>] [--perm <permission-mode>]
# [--name <name>] [--perm <permission-mode>] [--model <model>]
# orca-worker-start.sh --task <task_id> --terminal <handle>
#
# env (also test hooks):
# GROUNDWORK_ESCALATION_DIR exported into the worker so a guardrails `ask`
# escalates instead of blocking (activates worker mode)
# GROUNDWORK_TASK_ID worker task label
# DEV_LOOP_WORKER_MODEL default for --model — the model the WORKER runs
# (e.g. claude-sonnet-5). Unset = omit the flag, so
# the worker inherits the user's configured model.
# Lets the implementer run a cheaper tier than the
# coordinator; the auditor is pinned separately in
# agents/test-quality-auditor.md.
# LO_READY_TIMEOUT seconds to wait for TUI readiness (default 60)
# ORCA_BIN orca executable (default: orca)
# ORCA_WORKER_START_DRYRUN print the orca commands instead of running them
Expand All @@ -63,11 +72,12 @@ ORCA="${ORCA_BIN:-orca}"
JQ=$(command -v jq) || { echo "orca-worker-start: jq not found" >&2; exit 127; }

usage() {
echo "usage: orca-worker-start.sh --task <task_id> (--worktree <selector> --agent <agent> [--name <name>] [--perm <mode>] | --terminal <handle>)" >&2
echo "usage: orca-worker-start.sh --task <task_id> (--worktree <selector> --agent <agent> [--name <name>] [--perm <mode>] [--model <model>] | --terminal <handle>)" >&2
exit 1
}

task=""; wt=""; agent=""; name=""; term=""; perm="bypassPermissions"
model="${DEV_LOOP_WORKER_MODEL:-}"
while [ $# -gt 0 ]; do
case "$1" in
--task) task="${2:-}"; shift 2 || usage ;;
Expand All @@ -76,6 +86,7 @@ while [ $# -gt 0 ]; do
--name) name="${2:-}"; shift 2 || usage ;;
--terminal) term="${2:-}"; shift 2 || usage ;;
--perm) perm="${2:-}"; shift 2 || usage ;;
--model) model="${2:-}"; shift 2 || usage ;;
*) echo "orca-worker-start: unknown argument '$1'" >&2; usage ;;
esac
done
Expand All @@ -98,6 +109,17 @@ if [ -n "$agent" ]; then
*) echo "orca-worker-start: unsupported agent '$agent'" >&2; exit 2 ;;
esac
fi

# The model is interpolated into the same command line. Model ids and aliases are
# alphanumerics plus . _ - and the [1m] context suffix (e.g. opus[1m]); anything
# else — spaces, quotes, $, ;, backticks — is a command-injection vector, so
# reject the whole value rather than trying to sanitize it.
if [ -n "$model" ]; then
case "$model" in
*[!A-Za-z0-9._\[\]-]*)
echo "orca-worker-start: invalid model '$model'" >&2; exit 2 ;;
esac
fi
case "$perm" in
bypassPermissions|acceptEdits|plan|default) : ;;
*) echo "orca-worker-start: invalid permission mode '$perm'" >&2; exit 2 ;;
Expand Down Expand Up @@ -213,7 +235,9 @@ if [ "$worker_mode" = 1 ] && [ -z "$reused" ]; then
# Single-quote the values with embedded quotes escaped (`'\''`) so a path with
# any metacharacter — including a quote — cannot break out of the command.
esc_sq() { printf '%s' "$1" | sed "s/'/'\\\\''/g"; }
worker_cmd="export GROUNDWORK_ESCALATION_DIR='$(esc_sq "$esc_dir")' && export GROUNDWORK_TASK_ID='$(esc_sq "${GROUNDWORK_TASK_ID:-}")' && claude --permission-mode ${perm}"
model_arg=""
[ -n "$model" ] && model_arg=" --model '$(esc_sq "$model")'"
worker_cmd="export GROUNDWORK_ESCALATION_DIR='$(esc_sq "$esc_dir")' && export GROUNDWORK_TASK_ID='$(esc_sq "${GROUNDWORK_TASK_ID:-}")' && claude --permission-mode ${perm}${model_arg}"

set -- terminal create --worktree "$wt" --command "$worker_cmd" --json
[ -n "$name" ] && set -- "$@" --title "$name"
Expand Down
9 changes: 7 additions & 2 deletions skills/orchestrate/templates/brief.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,9 +58,14 @@ specific tags below as authority.
<!-- scale effort to complexity; bound the retries -->
<effort_level>complexity={simple|medium|complex}; loop-implement max 3 retries; stop exploring once DoD is met</effort_level>

<!-- output contract: where the plan goes + how to signal completion -->
<!-- the plan is an INPUT, not an output: the coordinator ran `wiki-plan` on the
planning model and wrote it before this session launched. Adopt it; report a
gap rather than re-planning (re-planning would move the decisions onto the
worker's tier). -->
<plan>.orchestration/plans/{TASK}.md — written by the coordinator; adopt, verify against this brief, do not re-author</plan>

<!-- output contract: how to signal completion -->
<output_contract>
plan -> .orchestration/plans/{TASK}.md
signal -> STATUS_DIR={STATUS_DIR} sh {SKILL}/scripts/status-update.sh {TASK} <phase> worktree=$PWD
</output_contract>

Expand Down
18 changes: 12 additions & 6 deletions skills/orchestrate/templates/session-prompt.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ block — to every §1–§4 prompt, flattened into the single sent line.

## (1) Plan — injected at session launch

You are the session for {TASK}. Treat .orchestration/briefs/{TASK}.md `<task_brief>` as authority — especially `<scope_boundaries>`, `<dependencies>`, and `<definition_of_done>`. Use the loop-implement skill but STOP after planning: run its step 2 with the bundled `wiki-plan` skill (make every design decision grounded in a `wiki/` page — record the decision->page map; leave nothing "as appropriate"), write the resulting implementation plan to .orchestration/plans/{TASK}.md, then run `STATUS_DIR={STATUS_DIR} sh {SKILL}/scripts/status-update.sh {TASK} plan_ready worktree=$PWD` and wait for an approval message. Do NOT write implementation code yet.
You are the session for {TASK}. Treat .orchestration/briefs/{TASK}.md `<task_brief>` as authority — especially `<scope_boundaries>`, `<dependencies>`, and `<definition_of_done>`. Use the loop-implement skill but STOP after planning. The coordinator has ALREADY run `wiki-plan` and written the plan to .orchestration/plans/{TASK}.md, so take loop-implement step 2's "a plan already exists" path: ADOPT that plan, do not re-plan it. Check it against the brief — every decision actually made (nothing left "as appropriate"), each with its decision->page map entry, and no contradiction with `<scope_boundaries>`, `<dependencies>`, or `<definition_of_done>`. If it fails any of those, do NOT quietly rewrite it: report the specific gap as a failure and stop, so the coordinator re-plans on the planning model. Otherwise run `STATUS_DIR={STATUS_DIR} sh {SKILL}/scripts/status-update.sh {TASK} plan_ready worktree=$PWD` and wait for an approval message. Do NOT write implementation code yet.

## (2) Implement — injected after plan approval

Expand Down Expand Up @@ -104,13 +104,19 @@ prompt here says "wait" — the worker reports and ends its turn.

You are the worker session for {TASK}. Treat .orchestration/briefs/{TASK}.md
`<task_brief>` as authority — especially `<scope_boundaries>`, `<dependencies>`, and
`<definition_of_done>`. Use the loop-implement skill but STOP after planning: run its
step 2 with the bundled `wiki-plan` skill (make every design decision grounded in a
`wiki/` page — record the decision->page map; leave nothing "as appropriate"), write the
resulting implementation plan to .orchestration/plans/{TASK}.md, then run
`<definition_of_done>`. Use the loop-implement skill but STOP after planning. The
coordinator has ALREADY run `wiki-plan` and written the plan to
.orchestration/plans/{TASK}.md, so take loop-implement step 2's "a plan already exists"
path: ADOPT that plan, do not re-plan it. Read it against the brief and check it is
executable — every design decision actually made (nothing left "as appropriate"), each
one carrying its decision->page map entry, and no contradiction with
`<scope_boundaries>`, `<dependencies>`, or `<definition_of_done>`. If it fails any of
those, do NOT quietly rewrite it: report the specific gap as a failure and stop, so the
coordinator re-plans on the planning model. Otherwise run
`STATUS_DIR={STATUS_DIR} sh {SKILL}/scripts/status-update.sh {TASK} plan_ready worktree=$PWD`
and report exactly once:
`orca orchestration send --type worker_done --subject "plan_ready: {TASK}" --body "<what the plan decides, what remains>" --task-id {ORCA_TASK_ID} --dispatch-id {ORCA_DISPATCH_ID} --outcome succeeded --files-modified ".orchestration/plans/{TASK}.md" --json`
`orca orchestration send --type worker_done --subject "plan_ready: {TASK}" --body "<what the adopted plan decides, what remains>" --task-id {ORCA_TASK_ID} --dispatch-id {ORCA_DISPATCH_ID} --outcome succeeded --json`
(no `--files-modified`: you adopted the coordinator's plan and wrote nothing.)
(a failure is `--outcome failed`, never failure encoded only in prose).
Then END YOUR TURN. Do NOT write implementation code yet.

Expand Down
39 changes: 39 additions & 0 deletions tests/launch-session.bats
Original file line number Diff line number Diff line change
Expand Up @@ -349,3 +349,42 @@ pane_not_ready() { # matches none of the ready/trust patterns
[ "$(jq -r .session "$st/t3.json")" = "lo-1" ]
[ "$(jq -r .worktree "$st/t3.json")" = "$BATS_TEST_TMPDIR/wt" ]
}

# --- DEV_LOOP_WORKER_MODEL --------------------------------------------------
# The worker may run a cheaper tier than the coordinator. Unset MUST stay
# byte-identical to the previous behavior (no --model at all), or every existing
# deployment silently changes model on upgrade.
# The unset case uses `env -u`: DEV_LOOP_WORKER_MODEL is a real user setting,
# so a developer with it exported would otherwise see this test pass vacuously
# (or fail) depending on their shell rather than on the code.

@test "MODEL: an invalid model is rejected before anything launches (injection guard)" {
run env LO_DRY_RUN=1 DEV_LOOP_WORKER_MODEL='bad; rm -rf ~' \
bash "$LS" lo-1 "${BATS_TEST_TMPDIR}/wt" bypassPermissions "prompt"
[ "$status" -eq 2 ]
[[ "$output" == *"invalid model"* ]]
}

@test "MODEL: unset adds no --model flag (boundary — unchanged behavior)" {
sd="$BATS_TEST_TMPDIR/sd"; mkdir -p "$sd"
# capture #1 is the readiness check, #2 the submission confirm
pane_ready_submitted > "$sd/pane-1"
pane_ready_submitted > "$sd/pane-2"
run env -u DEV_LOOP_WORKER_MODEL STUB_DIR="$sd" LO_TMUX="$(mk_tmux_stub)" LO_CLAUDE=/bin/echo \
LO_READY_TIMEOUT=2 LO_READY_INTERVAL=1 LO_SUBMIT_TIMEOUT=4 LO_SUBMIT_INTERVAL=1 \
sh "$LS" lo-1 "$BATS_TEST_TMPDIR" bypassPermissions "ZZPROMPTHEAD p"
[ "$status" -eq 0 ]
! grep -q -- '--model' "$sd/keys"
}

@test "MODEL: a set model reaches the launched claude command" {
sd="$BATS_TEST_TMPDIR/sd"; mkdir -p "$sd"
pane_ready_submitted > "$sd/pane-1"
pane_ready_submitted > "$sd/pane-2"
run env STUB_DIR="$sd" LO_TMUX="$(mk_tmux_stub)" LO_CLAUDE=/bin/echo \
DEV_LOOP_WORKER_MODEL=claude-sonnet-5 \
LO_READY_TIMEOUT=2 LO_READY_INTERVAL=1 LO_SUBMIT_TIMEOUT=4 LO_SUBMIT_INTERVAL=1 \
sh "$LS" lo-1 "$BATS_TEST_TMPDIR" bypassPermissions "ZZPROMPTHEAD p"
[ "$status" -eq 0 ]
grep -q -- "--model 'claude-sonnet-5'" "$sd/keys"
}
28 changes: 28 additions & 0 deletions tests/orca-spawn.bats
Original file line number Diff line number Diff line change
Expand Up @@ -69,3 +69,31 @@ setup() {
# the POSIX single-quote escape '\'' must appear — proves the value was escaped
[[ "$output" == *"/p'\\''q"* ]]
}

# --- worker model pin (DEV_LOOP_WORKER_MODEL) ---------------------------------
# orca-spawn takes positional args only, so the model arrives by env — the same
# variable orca-worker-start.sh and launch-session.sh read.
# The unset case uses `env -u`: DEV_LOOP_WORKER_MODEL is a real user setting,
# so a developer with it exported would otherwise see this test pass vacuously
# (or fail) depending on their shell rather than on the code.

@test "model: DEV_LOOP_WORKER_MODEL reaches the claude command" {
run env ORCA_SPAWN_DRYRUN=1 DEV_LOOP_WORKER_MODEL=claude-sonnet-5 \
bash "$OS" "r::/wt" bypassPermissions "p"
[ "$status" -eq 0 ]
[[ "$output" == *"claude --permission-mode bypassPermissions --model 'claude-sonnet-5'"* ]]
}

@test "model: unset adds no --model flag (boundary — unchanged behavior)" {
run env -u DEV_LOOP_WORKER_MODEL ORCA_SPAWN_DRYRUN=1 bash "$OS" "r::/wt" bypassPermissions "p"
[ "$status" -eq 0 ]
[[ "$output" != *"--model"* ]]
}

@test "model: a shell-metacharacter model is rejected, nothing is created" {
run env ORCA_SPAWN_DRYRUN=1 DEV_LOOP_WORKER_MODEL='x; rm -rf ~' \
bash "$OS" "r::/wt" bypassPermissions "p"
[ "$status" -eq 2 ]
[[ "$output" == *"invalid model"* ]]
[[ "$output" != *"[terminal] [create]"* ]]
}
Loading
Loading