fix(orchestration): isolate per-task config-resolution failures from the suite

[bai-uipath]

What

Make one incompatible task YAML skip instead of aborting the whole coder-eval run, by extending resolve_all_tasks' per-task error isolation from the load phase to config resolution (layers 1–5).

A resolution failure now demotes just that task file to skipped (surfaced in the run summary and run.json) — but if every task that reaches resolution fails, the cause is a global invocation error (a bad --type / -D value, repeats over the cap) rather than a per-task incompatibility, so it is re-raised and the CLI aborts cleanly. Early-stop arming errors always propagate.

Why

resolve_all_tasks already isolated task YAMLs that fail to load (#219), but layers 1–5 override resolution stayed unguarded. So a single task whose own YAML is incompatible with the run — the concrete case being Claude-only agent fields (sdk_options) surviving a --type codex override, which CodexAgentConfig forbids (extra="forbid") — aborted the entire invocation with no run.json written.

That is what collapsed the scheduled codex weekly: a handful of Maestro tasks carry sdk_options, and codex is the first harness to run them under a non-Claude --type. The gap is ~4 months old (it dates to the original resolution loop), not new — the load-isolation PR simply never covered this layer.

The "re-raise only when every attempted task fails" rule keeps the existing fail-fast behaviour for genuinely global misconfigurations (invalid --type, bad -D values, repeats cap, early-stop arming) while letting a suite shrug off the odd incompatible task and run the rest.

[github-actions]

Claude finished @bai-uipath's task in 2m 42s —— View job

---

Code Review in Progress

Todo List:

• Read .github/code_review.md for review guidelines
• Read CLAUDE.md for project conventions
• Run git diff origin/main...HEAD to see the full diff
• Read full context of changed files
• Analyze changes for correctness and design
• Check cross-file consistency
• Perform "what's missing" analysis
• Format and post review

[uipreliga]

Review: coder_eval — pr:25 (3 files) axis:1,2,3,4,5,6,7,8

Scope: pr:25 (3 files) axis:1,2,3,4,5,6,7,8 · branch bai/isolate-task-config-resolution · 143b9df · 2026-07-16T20:17Z · workflow variant

Change class: complex — introduces new control flow in the suite resolution loop (per-task config-resolution isolation via buffer-and-commit plus an all-fail re-raise heuristic); correctness requires reasoning about which failures are per-task vs. global

coder_eval is in excellent health (9.7/10, clean on security and error-handling) with the only real risk isolated to one PR-introduced regression in the experiment resolver: config-incompatible tasks that used to abort the run loudly are now silently demoted to skipped, so the bottom line is that the harness's exit-code contract, not its code quality, is what needs fixing before the next nightly.

Summary

Axis	Score	🔴	🟠	🟡	🔵	Top Issue
1. Code Quality & Style	9.4 / 10	0	0	1	1	resolve_all_tasks compounded to CC E(35); one ~220-line function now conflates load-isolation, per-file buffer-and-commit resolution, an all-fail re-raise heuristic, tag filtering, dedup, and sorting
2. Type Safety	9.5 / 10	0	0	1	0	All-fail re-raise propagates a non-ValueError (FileNotFoundError/OSError/yaml.YAMLError) that the caller's except ValueError misses, producing a raw traceback
3. Test Health	9.9 / 10	0	0	0	1	All-fail global-abort branch is covered only by a degenerate single-file case, not a purpose-built multi-file test
4. Security	10 / 10	0	0	0	0	—
5. Architecture & Design	9.9 / 10	0	0	0	1	Duplicated exception-catch tuple and reason-string between the two isolation blocks risks silent drift
6. Error Handling & Resilience	10 / 10	0	0	0	0	—
7. API Surface & Maintainability	9.9 / 10	0	0	0	1	Skipped-tasks warning omits the new config-resolution-failure skip category the PR introduces
8. Evaluation Harness Quality	9 / 10	0	1	0	0	Config-incompatible tasks now demoted to skipped do not fail the run exit code — a previously-loud abort becomes a silent pass-rate denominator drop (nightly stays green while a task vanishes from the suite)

Overall Score: 9.7 / 10 · Weakest Axis: Evaluation Harness Quality at 9 / 10
Totals: 🔴 0 · 🟠 1 · 🟡 2 · 🔵 4 across 8 axes.

Blockers

1. [Axis 8] Config-incompatible tasks now demoted to skipped do not fail the run exit code — a previously-loud abort becomes a silent pass-rate denominator drop (nightly stays green while a task vanishes from the suite) (src/coder_eval/orchestration/experiment.py:724) — experiment.py:721-724 newly routes per-task config-resolution failures into skipped:

for err_file, exc in resolution_errors:
    reason = f"{type(exc).__name__}: {exc}"[:500]
    logger.warning("Skipping task file %s — config resolution failed: %s", err_file, reason)
    skipped.append(SkippedTask(path=str(err_file), reason=reason))

Pre-PR (origin/main experiment.py:626+) the resolution loop had NO try/except, so a config-incompatible task (e.g. Claude-only sdk_options surviving --type codex) raised a Pydantic ValidationError that propagated out of resolve_all_tasks and was converted to a hard typer.BadParameter abort (exit non-zero) in run_command.py. Now that same task is demoted to skipped and the run continues. Crucially, skipped tasks do NOT gate the run: run_command.py:473-474 exits non-zero only on summary.tasks_failed > 0 or summary.tasks_error > 0 or failed_suite_gates > 0 — skipped is absent. Because RunSummary excludes skipped tasks from tasks_run (results.py:782-787 invariant is over succeeded+failed+error only), a task that becomes config-incompatible silently leaves the suite's pass-rate BOTH numerator and denominator with exit code 0. Failure scenario: a mixed suite where one task drifts config-incompatible now goes GREEN in the nightly with that task silently missing from coverage, where it previously went RED. Mitigations exist (a yellow console warning at run_command.py:604-607 and a skipped_tasks entry in run.json) but neither affects the exit code the nightly (in the separate coder-eval-uipath / eval-runner repo) gates on. Recommendation: make the coverage drop loud — either fail the exit code when config-resolution skips occur (distinct from intentional skip: true opt-outs), or add a distinct SkippedTask reason category so the cross-repo nightly can gate on resolution-failure skips vs. intentional opt-outs; at minimum document that the nightly must inspect skipped_tasks or coverage will silently erode.

Non-blocking, but please consider before merge

1. [Axis 1] resolve_all_tasks compounded to CC E(35); one ~220-line function now conflates load-isolation, per-file buffer-and-commit resolution, an all-fail re-raise heuristic, tag filtering, dedup, and sorting (src/coder_eval/orchestration/experiment.py:536) — resolve_all_tasks spans lines 536-757 and radon rates it E(35). This PR raised the complexity by wrapping the layer-1-5 resolution in a per-file try/except (lines 649-708) plus the post-loop heuristic if resolution_errors and len(resolution_errors) == attempted: raise resolution_errors[0][1] (lines 719-720). The isolation itself is fine, but it's grafted onto a function that already does far too much. Extract the per-file resolution loop (the for expanded_task ... for variant ... body that builds file_resolved) into a helper such as _resolve_file(...) -> list[ResolvedTask] that either returns the file's fan-out or raises, so resolve_all_tasks reads as orchestration: load -> _resolve_file (collect errors) -> decide -> filter/dedup/sort. That drops the nesting depth and lets the buffer-and-commit atomicity live in one named place. experiment.py is not one of the anchor's named hot modules (orchestrator/checker/sandbox), so this is scored Medium rather than High, but E(35) is well past a readable threshold.
2. [Axis 2] All-fail re-raise propagates a non-ValueError (FileNotFoundError/OSError/yaml.YAMLError) that the caller's except ValueError misses, producing a raw traceback (src/coder_eval/orchestration/experiment.py:720) — resolution_errors: list[tuple[Path, Exception]] (experiment.py:593) is populated by the inner handler except (FileNotFoundError, OSError, ValueError, yaml.YAMLError) (experiment.py:707), so the tuple can legitimately hold a non-ValueError. In the all-failed branch raise resolution_errors[0][1] (experiment.py:720) re-raises that Exception. But the sole production caller catches only except ValueError as e: (run_command.py:601 → raise typer.BadParameter(str(e)) from e), and the docstring documents only ValueError (experiment.py:580-581). A reachable case: a single task file whose variant injects a missing system_prompt_file makes resolve_agent_system_prompt raise FileNotFoundError (an OSError, not a ValueError) during resolve_task_files inside the inner try; with attempted==1==len(resolution_errors) the FileNotFoundError is re-raised and escapes except ValueError, producing a raw traceback instead of the clean CLI error the PR's own comment promises ('surface them as a clean CLI error instead of a traceback'). Fix by either narrowing/wrapping the re-raise into a ValueError (e.g. raise ValueError(str(exc)) from exc) so the type matches the caller's contract and the documented Raises: ValueError, or by widening run_command's except ValueError to also catch OSError/yaml.YAMLError.

Nits

1. [Axis 1] Stale Raises: docstring on resolve_all_tasks — omits the re-raised resolution error (and EarlyStopConfigError) (src/coder_eval/orchestration/experiment.py:580) — The structured Raises: block (lines 580-581) reads only ValueError: If duplicate task IDs are found after resolution. After this PR the function also (a) re-raises the first collected resolution error via raise resolution_errors[0][1] at line 720 when every attempted task fails (this can be any of FileNotFoundError/OSError/ValueError/yaml.YAMLError), and (b) always propagates EarlyStopConfigError via except EarlyStopConfigError: raise at line 699. The prose docstring (lines 555-564) describes both behaviors, so the structured Raises: list is now the out-of-date half. Add entries for the re-raised all-fail resolution error and EarlyStopConfigError so the structured section matches the prose and the code.
2. [Axis 3] All-fail global-abort branch is covered only by a degenerate single-file case, not a purpose-built multi-file test (src/coder_eval/orchestration/experiment.py:720) — The new branch if resolution_errors and len(resolution_errors) == attempted:\n raise resolution_errors[0][1] (experiment.py:719-720) — abort when EVERY attempted task fails rather than produce an empty all-skipped run — is executed (line 720 is covered in the broad suite) but only incidentally by the pre-existing test_repeats_100_raises_before_any_sandbox (tests/test_experiment_runner.py:175), which passes a SINGLE task file, making len(resolution_errors)==attempted==1 a degenerate case where the len == attempted discrimination does no real work. Both new PR tests exercise only the demote-to-skipped half (a sibling resolves, so len < attempted). The motivating scenario in the branch comment — multiple task files ALL failing resolution under a bad --type/-D → global abort — is not directly tested, so the multi-file discrimination boundary between demote and abort has no dedicated guard. Add a test with two or more task files that all fail resolution (e.g. both carry Claude-only sdk_options under --type codex) asserting resolve_all_tasks raises rather than returning an empty resolved list.
3. [Axis 5] Duplicated exception-catch tuple and reason-string between the two isolation blocks risks silent drift (src/coder_eval/orchestration/experiment.py:706) — The new resolution-isolation block duplicates the load block's handler shape: the catch tuple except (FileNotFoundError, OSError, ValueError, yaml.YAMLError) as exc: appears at line 636 and again at 706, and reason = f"{type(exc).__name__}: {exc}"[:500] appears at 637 and 722. These must stay in sync; an edit to one tuple/cap and not the other would silently demote different failures to skipped vs crash. Extract a module-level exception constant and a small _skip_reason(exc) helper so both share one definition. Behavior is correct today - maintainability/drift nit only.
4. [Axis 7] Skipped-tasks warning omits the new config-resolution-failure skip category the PR introduces (src/coder_eval/cli/run_command.py:607) — The warning at run_command.py:606-607 enumerates skip causes as '(load errors or skip: true - see run.json skipped_tasks for reasons)', but this PR adds a third reason: per-task config-resolution incompatibilities (e.g. Claude-only sdk_options surviving --type codex) are now demoted to skipped. A user whose task was dropped for a config incompatibility sees a message naming neither their cause. Broaden the parenthetical (e.g. 'load errors, config incompatibilities, or skip: true') so the surfaced category matches what resolve_all_tasks now demotes.

What's Missing

Parallel paths:

• 🔵 🔵 Parallel paths — the new per-file config-resolution isolation block (experiment.py:~706) is a hand-copy of the load/expand isolation block (experiment.py:618): identical except (FileNotFoundError, OSError, ValueError, yaml.YAMLError) tuple and identical f"{type(exc).__name__}: {exc}"[:500] reason format. Both must move together; a future edit that widens/narrows one catch tuple (or changes the 500-char cap) but not the other silently diverges which failures demote-to-skipped vs crash. Verified there is NO second resolution path to update — plan_command does not call resolve_all_tasks and run_batch is the sole downstream consumer — so the only parallel-path risk is this in-function duplication. Extract a shared exception constant + _skip_reason(exc) helper. (trigger: src/coder_eval/orchestration/experiment.py) (restates: Axis 5: Duplicated exception-catch tuple and reason-string between the two isolation blocks)

Tests:

• 🔵 🔵 Tests — the new except EarlyStopConfigError: raise branch (experiment.py:~699), whose whole job is to keep a misarmed early-stop propagating EVEN WHEN sibling tasks resolve, is only exercised by a single-file test (test_early_stop.py:test_run_surface_rejects_bad_armed_task). With one file, len(resolution_errors)==attempted==1 so the post-loop all-fail heuristic would re-raise it anyway — deleting the dedicated except EarlyStopConfigError: raise line would NOT fail any test. Add a multi-file case where one sibling resolves and the armed task raises, asserting EarlyStopConfigError propagates rather than being demoted to skipped. (trigger: src/coder_eval/orchestration/experiment.py) (restates: Axis 3: All-fail global-abort branch is covered only by a degenerate single-file case)
• 🔵 🔵 Tests — the all-fail abort heuristic if len(resolution_errors)==attempted: raise (experiment.py:~719-720) has no purpose-built multi-file test. Both new tests in test_dataset_expansion.py exercise only the demote-to-skipped half (a sibling resolves, so len<attempted). Add a test with 2+ task files that ALL fail resolution asserting resolve_all_tasks raises rather than returning an empty resolved list. (trigger: src/coder_eval/orchestration/experiment.py) (restates: Axis 3: All-fail global-abort branch is covered only by a degenerate single-file case)
• 🟡 🟡 Tests — no test asserts the CLI surfaces a clean error when the all-fail re-raised exception is a non-ValueError. raise resolution_errors[0][1] (experiment.py:~720) can re-raise FileNotFoundError/OSError/yaml.YAMLError, but run_command.py:601 catches only except ValueError. Add a test that a single task whose config resolution raises FileNotFoundError (e.g. a variant injecting a missing system_prompt_file) reaches run_command and yields a typer.BadParameter (clean CLI error) rather than a raw traceback — guarding the PR comment's own promise of 'a clean CLI error instead of a traceback.' (trigger: src/coder_eval/cli/run_command.py) (restates: Axis 2: All-fail re-raise propagates a non-ValueError the caller's except ValueError misses)

Downstream consumers:

• 🟠 🟠 Downstream consumers — the counting/classification change (config-incompatible tasks now routed to skipped instead of aborting) was not propagated to the consumers of that classification: the exit-code gate (run_command.py:474 tasks_failed/tasks_error/failed_suite_gates) omits skipped, and RunSummary's tasks_run denominator excludes skipped tasks. Net effect: a config-drifted task silently leaves BOTH numerator and denominator of the suite pass-rate with exit code 0. Either gate the exit code on resolution-failure skips (distinct from intentional skip: true) or document that consumers must inspect skipped_tasks. (trigger: src/coder_eval/orchestration/experiment.py) (restates: Axis 8: Config-incompatible tasks demoted to skipped do not fail the run exit code)
• 🔵 🔵 Downstream consumers — SkippedTask carries only a free-form reason string with no structured category field, yet this PR introduces a THIRD skip cause (config-resolution failure) whose reason format (f"{type(exc).__name__}: {exc}") is byte-identical to a load-failure skip (only intentional opt-outs are string-prefixed skip: true). Consumers (the run_command.py:606 warning, run.json readers, the cross-repo nightly, reports) cannot distinguish config-failure from load-failure without brittle string-parsing, and the SkippedTask docstring (models/results.py:792) still says the record is written 'in two cases.' Add a typed category enum so consumers can gate/render the three skip kinds distinctly. (trigger: src/coder_eval/orchestration/experiment.py) (restates: Axis 7: Skipped-tasks warning omits the new config-resolution-failure skip category)

Daily/nightly:

• 🟠 🟠 Daily/nightly — the PR does not state its blast radius on the production nightly. Pre-PR a config-incompatible task (e.g. Claude-only sdk_options surviving --type codex) raised a ValidationError → hard typer.BadParameter abort (exit non-zero); post-PR it demotes to skipped and the run continues green. The cross-repo eval-runner / coder-eval-uipath nightly gates on the exit code (tasks_failed), not run.json skipped_tasks, so a task that drifts config-incompatible now vanishes from coverage while the nightly stays GREEN where it previously went RED. The PR must state this and give the nightly a signal (exit-code gate or a distinct resolution-failure skip category) or coverage erodes silently. (trigger: src/coder_eval/orchestration/experiment.py) (restates: Axis 8: Config-incompatible tasks demoted to skipped do not fail the run exit code)

Harness & Lint Improvements

Static checks (lint / type):

• [ruff] Enable ruff's mccabe cyclomatic-complexity gate: add C901 to [tool.ruff.lint] select and set [tool.ruff.lint.mccabe] max-complexity to a baselined ceiling (the current tolerated max is F(48)/aggregate_results, so pick a policy ceiling and add a visible # noqa: C901 debt marker to the existing offenders — the exact pattern pyproject.toml already documents for PLR0915/PLR0912). Today only statements (max 80) and branches (max 25) are gated; cyclomatic complexity is not, which is precisely why resolve_all_tasks could climb D(30)->E(35) in this PR while staying under those caps and passing ruff check clean. Lighter alternative if a global C901 baseline is too costly: a CE022-style per-function statement/complexity cap targeting orchestration/experiment.py::resolve_all_tasks (CE022 already establishes this exact 'bound one named god-function against regrowth' precedent). Prevents: A1 medium: resolve_all_tasks compounded to CC E(35) (222-line god-function) with ruff passing clean because PLR0915/PLR0912 gate statements/branches, not McCabe complexity.
• [ce-lint] Add CE025 (next free number; wire into tests/lint/runner.py ALL_RULES) forbidding the duplicated skip-isolation shape in orchestration/experiment.py: the identical except (FileNotFoundError, OSError, ValueError, yaml.YAMLError) handler tuple and the inline reason = f"{type(exc).__name__}: {exc}"[:500] reason-string/truncation literal each appear twice (load block ~636/637 and resolution block ~706/722). The rule flags a second inline occurrence of that reason-formatting expression (and/or the repeated exception tuple) in the same module, requiring a shared _skip_reason(exc) helper + a module-level exception-tuple constant so the two blocks cannot silently drift (e.g. one tuple gains a type or the [:500] cap changes on only one side). Grep/AST-detectable duplication. Prevents: A5 low: duplicated exception-catch tuple and reason-string across the two isolation blocks risks silent drift between which failures demote-to-skipped vs crash.
• [ce-lint] Optional CE026: a docstring-contract rule that checks a function's Raises: section names every exception type appearing in a direct raise SomeError(...) in its body. This catches the general stale-Raises drift class. Caveat to record explicitly: it would NOT catch THIS instance, because the offending statement is raise resolution_errors[0][1] (re-raise of a stored variable whose runtime type is heterogeneous) — a variable re-raise has no static type, so the docstring-vs-body check is blind to it. That gap is the honest static-analysis boundary here: the ValueError-contract mismatch itself (finding A2) is not statically reachable and must fall back to the harness test below. Prevents: A1/A7 low: stale Raises: docstring on resolve_all_tasks omits the re-raised resolution error and EarlyStopConfigError (partial coverage only).

Harness improvements (not statically reachable):

• Add a purpose-built multi-file all-fail test that drives the full load -> resolve -> CLI boundary: two or more task files that ALL fail resolution (e.g. both carry Claude-only sdk_options under --type codex, or a missing system_prompt_file). Assert (a) resolve_all_tasks raises rather than returning an empty resolved list (exercises the len(resolution_errors) == attempted abort branch with real discrimination, not the degenerate single-file case), AND (b) invoked through run_command the failure surfaces as a clean typer.BadParameter, not a raw traceback — i.e. drive the FileNotFoundError/OSError path, since the current except ValueError in run_command.py:601 would let a non-ValueError escape. Why not static: Python has no checked exceptions and pyright does not model exception-type flow, so no static check can see that raise resolution_errors[0][1] yields a non-ValueError (FileNotFoundError/OSError/yaml.YAMLError) that slips past the caller's except ValueError into a raw traceback. Only exercising the load->resolve->CLI path at runtime reveals it. The abort branch also needs >=2 files to make the len == attempted boundary do real work — a semantic scenario, not a pattern. Prevents: A2/A6/A7 medium (all-fail re-raise propagates a type the caller's except ValueError misses) and A3 low (all-fail abort branch covered only by a degenerate single-file case).
• Make the coverage drop loud instead of silent: give config-resolution failures a distinct SkippedTask reason category (separate from intentional skip: true opt-outs) and gate the run exit code on it — a task dropped for a config incompatibility should exit non-zero (or at minimum be independently gate-able), not leave the run green. Add a run-level test: a mixed suite where one task drifts config-incompatible exits non-zero (or reports the distinct category) rather than passing with the task silently absent from both numerator and denominator of the suite pass-rate. Why not static: The exit-code gate is a semantic policy over aggregated RunSummary counts (tasks_failed > 0 or tasks_error > 0 or failed_suite_gates > 0, with skipped excluded from tasks_run). No grep/AST pattern can encode 'a resolution-failure skip category must contribute to the exit code' or 'coverage must not silently erode' — it requires a run-level assertion over the summary and the cross-repo (coder-eval-uipath / eval-runner) nightly's gating contract, which lives outside this repo. Prevents: A8 high: config-incompatible tasks now demoted to skipped do not fail the exit code, so a previously-loud abort becomes a silent pass-rate denominator drop (nightly stays green while a task vanishes from the suite).
• Wire the run_command skipped-tasks warning to enumerate skip causes from the SkippedTask reason-category source of truth rather than a hardcoded parenthetical ('load errors or skip: true'), and add a test asserting every reason category is named in the surfaced message. If the A8 improvement above introduces a distinct resolution-failure category, this keeps the user-facing warning from silently omitting it. Why not static: The defect is message-content drift (a hardcoded string omitting a real skip category) — a semantic/text property, not a detectable code pattern; it needs a test that asserts the rendered warning covers the category set, which requires exercising the reporting path. Prevents: A7 low: skipped-tasks warning omits the new config-resolution-failure skip category the PR introduces.

Top 5 Priority Actions

1. Restore loud failure for config-resolution skips (src/coder_eval/orchestration/experiment.py:721-724): either fail the exit code or emit a distinct SkippedTask reason so the exit-code gate (run_command.py:474-475, which ignores skipped) cannot go green while a task silently vanishes from the suite's pass-rate denominator — this is the only finding that flips a task's final_status (RED abort → silent skip) for identical agent output.
2. Wrap the all-fail re-raise at src/coder_eval/orchestration/experiment.py:720 into a ValueError (e.g. raise ValueError(str(exc)) from exc) so a non-ValueError like FileNotFoundError from a missing system_prompt_file matches the documented Raises: contract and the caller's except ValueError at run_command.py:601, yielding a clean typer.BadParameter instead of a raw traceback.
3. Add a purpose-built multi-file test (tests/test_experiment_runner.py) where two or more task files all fail resolution, asserting resolve_all_tasks aborts rather than returning an empty resolved list — the abort-vs-demote boundary is currently exercised only by a degenerate single-file case.
4. Extract the per-file resolution loop from resolve_all_tasks into a _resolve_file(...) -> list[ResolvedTask] helper (src/coder_eval/orchestration/experiment.py:536) to bring the 222-line E(35) function back to readable orchestration and give the buffer-and-commit atomicity one named home.
5. Deduplicate the two isolation blocks and refresh docs: factor the shared except (FileNotFoundError, OSError, ValueError, yaml.YAMLError) tuple and _skip_reason(exc) (experiment.py:636/706) into one definition, update the stale Raises: docstring (experiment.py:580) to list the re-raised error and EarlyStopConfigError, and broaden the skipped-tasks warning (run_command.py:607) to name config incompatibilities.

---

Stats: 0 🔴 · 1 🟠 · 2 🟡 · 4 🔵 across 8 axes reviewed.

[mohsen-uipath]

Approving. The core fix is sound and verified: reproduced the original abort on main and confirmed the PR isolates the incompatible task to skipped while siblings still resolve. Lint, typecheck, and the touched test file (54 passed) are clean.

Left one non-blocking inline note on the len(resolution_errors) == attempted heuristic (can't distinguish a genuine global misconfig from N identical per-task failures), and would like a follow-up test for the all-failed → re-raise branch, which the two new tests don't cover. Neither blocks merge.