Observed in the agent-graphs gen2 run (skills/agent-graphs/gen2.json, upstreamGate): one candidate cell failed on a router HTTP 503 (twice), which made candidate coverage incomplete → eligibleForPromotion=false → the loop held the baseline without ever scoring the candidate on holdout.
Fail-closed on incomplete evidence is the right default. But a transient infra failure is distinguishable from a scoring failure, and today the only remedies are a full re-run of every arm or a caller-side supplementary measurement outside the loop (which is what gen2 did, disclosed). Suggestion: an opt-in bounded cell-retry (same seed, n attempts, failure receipts retained) before coverage is declared incomplete — the failure-receipt already exists per cell, so the loop has what it needs to tell 503 from a judge error.
Observed in the agent-graphs gen2 run (skills/agent-graphs/gen2.json,
upstreamGate): one candidate cell failed on a router HTTP 503 (twice), which made candidate coverage incomplete →eligibleForPromotion=false→ the loop held the baseline without ever scoring the candidate on holdout.Fail-closed on incomplete evidence is the right default. But a transient infra failure is distinguishable from a scoring failure, and today the only remedies are a full re-run of every arm or a caller-side supplementary measurement outside the loop (which is what gen2 did, disclosed). Suggestion: an opt-in bounded cell-retry (same seed, n attempts, failure receipts retained) before coverage is declared incomplete — the failure-receipt already exists per cell, so the loop has what it needs to tell 503 from a judge error.