Fix swarm transport walls: out-of-band prompt, telemetry, lens split - #54
Open
gering wants to merge 5 commits into
Open
Fix swarm transport walls: out-of-band prompt, telemetry, lens split#54gering wants to merge 5 commits into
gering wants to merge 5 commits into
Conversation
The prompt travelled as one argv word, so exec's MAX_ARG_STRLEN (128 KiB on Linux) was the binding limit and forced a 120 KiB cap. Above it the skill set EXTERNALS_OVERSIZE and dropped EVERY external voice — the same damage as a backend timeout, from a limit that was never inherent to the CLIs. Neither CLI needs the prompt on argv: codex reads it from stdin (`-- -`), grok takes `--prompt-file`. The adapter now normalizes every input form to one file and hands over the PATH, so the diff never enters a shell variable either. - Cap now bounds model context, not exec: SWARM_MAX_PROMPT_BYTES (default 512 KiB). Adapter and the skill's oversize guard read the same env knob with the same default, so an override reaches the externals instead of being short-circuited by a skip that never heard about it. - Temp prompts are chmod 600 before content lands and removed by the EXIT trap on every path; a caller-owned --prompt-file is never mutated, so concurrent per-cluster voices sharing one prompt file cannot corrupt each other. - grok's --prompt-file is preflighted (stubbable, so the argv tests stay hermetic) with an upgrade error — never a silent fallback to --single, which would reinstate the wall as a mystery failure on big diffs. - test_sandbox_deny.py pins the transport itself (regressions are silent: only large diffs would start failing); test_lens_sync.py pins the two cap defaults together and keeps the 4 KiB --lens-instr headroom covered. Verified end-to-end at 164 KiB through both backends. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JHKreruna9RfZHcEq6YPub
The timeouts were long attributed to prompt size. Measured on one 42 KB diff, one variable at a time, that is wrong: grok high breakage 374s 4 findings grok low breakage 161s 4 findings grok high consistency 28s 6 findings codex high breakage 104s 2 findings A 164 KiB control prompt returned in 20s (grok) / 8.6s (codex). The lens CLUSTER dominates (13x), effort is secondary (2.3x, and bought zero extra findings here), and backends differ 3.6x — grok is simply the slow voice on the cluster whose briefs demand exploration (cross-file-trace reads neighboring files). Size is ruled out. Chunking the diff would therefore target the one variable measurement excludes; splitting by LENS targets the one that dominates. Recorded in the knowledge entry so the next round starts from data instead of the old assumption. - agents.sh: `run --telemetry <file> --unit <name>` appends one JSON line per call (duration, effective effort/model, prompt bytes, backend rc, timed_out, and the wall the call actually ran under). Written from the EXIT trap, so a timeout is recorded too; the backend's own rc is captured before it is translated into the adapter exit code, or 124 and 1 would be indistinguishable. - telemetry-report.py renders it under the balance block and flags any SURVIVING call at >=60% of its wall — the case backendErrors structurally cannot show, since a voice finishing at 550s and one at 20s are both "ok". - Opt-in end to end: no --telemetry means the previous behaviour, byte for byte. Diagnostics never fail a review — a missing, truncated or malformed file degrades to less output, never a non-zero exit. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JHKreruna9RfZHcEq6YPub
Proposed as a speed fix; the measurement corrected that, and the correction is recorded rather than quietly dropped: old: one 3-lens breakage call 374s -> 4 findings, THREE of them cross-file-trace new: breakage (2 lenses) 313s -> 4 findings the combined call missed entirely new: reach (cross-file-trace) 126s -> 4 findings, ~the combined call's set The real defect was LENS CROWD-OUT: one lens consumed the call while correctness/removed-behavior barely reported. Split, the diff-local lenses found four issues the combined call never surfaced. It is NOT a throughput win — the longest call drops only 374->313s and total work rises to 439s, so no single lens split clears the 600s wall. Effort remains the largest untried runtime lever (374->161s for identical findings). Two further effects, neither reachable by lowering effort: a timeout now costs one lens instead of three (the family-critical case — grok is the only third-family voice), and `reach` holds no MANDATORY lens, so the gate may prune the whole call on a diff with no cross-file surface. Fixes found by the new breakage voice reviewing this very branch: - The skill's EXTERNALS_OVERSIZE guard read SWARM_MAX_PROMPT_BYTES without the adapter's validation. A malformed value expands to 0, the threshold goes negative, and EVERY external voice is dropped SILENTLY — while the adapter refuses the same value loudly. Now rejected symmetrically (SWARM_CFG_ERR), pinned by test_lens_sync.py in both directions. - The grok --prompt-file capability probe ran `grok --help` unbounded, outside with_timeout: a wedged CLI would hang the review before any review work. Now capped by SWARM_PROBE_TIMEOUT with -k, degrading to "assume supported" where no timeout binary exists rather than hanging or refusing. - A header comment still promised a --single fallback the preflight replaced with a hard error. Also measured and REJECTED: `grok --max-turns` (10 -> 10s but ZERO findings; 20 -> 279s, 4). The truncated run exits rc=0 with empty findings, so the pipeline reads a silenced voice as "reviewed cleanly, found nothing" — worse than a timeout, which at least reaches backendErrors. Documented in the knowledge entry so it is not retried blind. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JHKreruna9RfZHcEq6YPub
Two leftovers from the timeout task, both about making an unavoidable failure legible rather than preventing it. Timeout passthrough. The adapter's cap and the Bash window the transport agent runs under both defaulted to 600s, so which fired first was undefined — and when the outer one won, the run lost rc=124, the "timed out after Ns" message and the telemetry timeout flag, leaving a bare killed command. That lost diagnosis is why raising SWARM_TIMEOUT looked counterproductive. It now travels skill → workflow, which pins the adapter cap a margin below the Bash window so it always wins the race, and says so when a requested value exceeds what one Bash call can hold. SWARM_TIMEOUT=0 is passed through, with a log line that the outer window still kills at 600s and will report generically. This does NOT raise the ceiling — only async transport can (tasks/async-poll-external-voices.md). Family coverage. Consensus is defined as ">=2 agreeing families", so losing one silently changes what every CONSENSUS and every solo MEANS: a finding that would have been corroborated is routed through the adversarial verifier instead. The counts look identical to a healthy run — that silent degradation is the reason this task existed. balance now carries familiesExpected/Present/Lost and consensusReachable, and the report prints a warning directly under Bilanz:, including the case where fewer than two families survived and NO finding can reach consensus. Presence is per family, not per call: one dead cluster beside a live one is not a lost family (that stays a backendErrors entry). Both pinned by test_lens_sync.py, including that the Bash window is derived rather than a second hard-coded literal — the tie is the bug. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JHKreruna9RfZHcEq6YPub
grok had been absent from EVERY review since CLI 1.0.3 — silently. That release
changed the `grok models` bullet marker so only the DEFAULT keeps `*`:
* grok-4.6 (default)
- grok-4.5 <- the pinned model, invisible to a `*`-only matcher
So readiness reported "this CLI does not offer grok-4.5" for a CLI that offers
it, and the sole third model family dropped out of the ensemble. No timeout, no
error — just two families where the report claimed three. Same damage as the
600s wall this branch is about, reached by a different route.
Rather than re-pin to 4.6 and wait for the next break, the model is now
DISCOVERED. Ported from ~/dotfiles' cc-harness-agents (same provider), with one
gate substituted: that helper withholds an upgrade until a model's context
window is known; the adapter withholds it until --json-schema ENFORCEMENT is
known, because a model that merely accepts the flag returns
structuredOutput:null and fails after a full review is paid for.
- GROK_CANONICAL_RE accepts only bare version ids, major >= 4 — rejecting dated
snapshots, reasoning/non-reasoning splits, multi-agent, build, composer and
image/video variants. Major >= 4 keeps a catalog regressing to grok-3* from
pulling the ensemble backwards.
- Ordering is component-wise: grok-4.20 beats grok-4.6. As a decimal fraction it
would lose, but the provider means the 20th minor release and already ships
4.20-derived ids.
- GROK_SCHEMA_VERIFIED is the hard gate AND the upgrade ritual: a newer
canonical model is named on stderr, never selected. Verified on CLI 1.0.3 that
grok-4.5 and grok-4.6 both return an envelope whose .structuredOutput carries
the schema's findings; reviews now run on grok-4.6.
- Readiness and the run_grok preflight moved from "is THIS id listed/requested"
to "is a schema-verified model on offer/requested" — the exact-id form is what
let the marker change drop the backend. An explicit --model still bypasses
discovery, never the schema gate.
test_grok_models.py pins both listing formats against the SHIPPED awk program
(extracted, never re-typed) plus the ordering, the filter against the live
14-id catalog, the verified gate, and that the fallback pin is itself verified.
Two self-inflicted test bugs found and fixed while writing it: an extraction
regex that silently matched nothing (every assertion then passing over empty
output), and a verdict block left mid-file so the discovery half's failures
were recorded but never read.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JHKreruna9RfZHcEq6YPub
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Swarm reviews reproducibly lost an entire model family — three runs in a row, always
grok × breakageat exactly 600 s. This PR removes the adjacent failure mode, measures away several wrong theories, and makes the remaining one legible. It does not raise the 600 s ceiling; that needs an async transport and is split out intotasks/async-poll-external-voices.md.exec'sMAX_ARG_STRLENis no longer the binding limit. Above the old cap the skill dropped every external voice — the same damage as a timeout, from a limit that was never inherent to the CLIs.rc=124survives; per-call telemetry exposes voices approaching the wall before they cross it; a lost model family is called out where it changes meaning — in the consensus line.Measurements (these drove the decisions)
Same 42 KB diff, one variable at a time:
Control: a 164 KiB prompt returned in 20 s (grok) / 8.6 s (codex) — 4× the bytes, 1/20th the time.
grok --max-turns: rejected. N=10 → 10 s but zero findings; N=20 → 279 s. The truncated run exitsrc=0with empty findings, so the pipeline reads a silenced voice as "reviewed cleanly, found nothing" — worse than a timeout, which at least reachesbackendErrors.add-swarm-review-profiles.Changes
Transport (0.8.0) — codex reads the prompt from stdin (
-- -), grok via--prompt-file; the adapter passes the file path through instead of reading it into a shell variable. Cap is now model context (SWARM_MAX_PROMPT_BYTES, 512 KiB), read by adapter and skill from the same knob so an override reaches both. Verified end-to-end at 164 KiB on both backends.Telemetry (0.8.1) —
agents.sh run --telemetry <file> --unit <name>records duration, effective effort/model, prompt bytes, backend rc,timed_outand the wall the call ran under, written from the EXIT trap so timeouts are captured.telemetry-report.pyflags any surviving call at ≥60 % of its wall — the casebackendErrorsstructurally cannot show.Lens split (0.9.0) —
cross-file-tracemoves into its ownreachcluster. The measured reason is lens crowd-out: the combined 3-lens call returned 3 of 4 findings from one lens; split, the other two found 4 issues it had missed entirely. Not a speed fix — the longest call drops only 374 → 313 s.Diagnosis (0.9.1) — both timeouts derive from one value, with the adapter cap pinned below the Bash window so it wins the race and
rc=124survives (the old 600/600 tie is why raisingSWARM_TIMEOUTlooked counterproductive). The balance block now names a lost model family and states that consensus weakened — or became unreachable.Fixes found by this branch reviewing itself (the new
breakagevoice, first run): the skill's oversize guard readSWARM_MAX_PROMPT_BYTESwithout the adapter's validation, so a malformed value made the threshold negative and dropped every external voice silently; the grok capability probe rangrok --helpunbounded outsidewith_timeout; a header comment promised a--singlefallback the preflight had replaced with a hard error.Readiness
test_telemetry_report.py, plus transport + coupling guards in existing suites)check-structure.pygreen (version sync, frontmatter, shell syntax)0.9.1, synced inplugin.json+marketplace.jsonswarm-backend-adapter.md+swarm-review-pipeline.mdupdated with the measurement matrix and the rejected alternativesmainTest plan
/swarm:reviewon a normal diff — externals run,Voices:timing section appears under the balance block/swarm:reviewon a >120 KiB diff — externals now participate instead of being skipped wholesaleSWARM_MAX_PROMPT_BYTES=abc /swarm:review— refused loudly by both skill and adapter, not a silent Claude-only reviewSWARM_TIMEOUT=5) —rc=124, telemetrytimed_out, and the reduced-consensus warning all appear🤖 Generated with Claude Code
https://claude.ai/code/session_01JHKreruna9RfZHcEq6YPub