Skip to content

Hypothesis state-machine tests for pipeline completion, and fail-fast telemetry (#73, #285) - #243

Merged
leynos merged 50 commits into
mainfrom
python-pipeline-wait-tests
Aug 25, 2026
Merged

Hypothesis state-machine tests for pipeline completion, and fail-fast telemetry (#73, #285)#243
leynos merged 50 commits into
mainfrom
python-pipeline-wait-tests

Conversation

@leynos

@leynos leynos commented Jul 28, 2026

Copy link
Copy Markdown
Owner

Summary

Pipeline completion ordering — first-failure latching, timing-slot population,
and the fail-fast termination decision — used to live inline in
process_completed_task. It is now a deterministic state transition, which
addresses #73.

Seam

PipelineWaitState.record_completion(completed_idx, exit_code, *, ended_at):

  • stamps the matching exit-code and completion-time slots;
  • latches the first non-zero exit in completion order; and
  • returns true only when that completion is the first failure from a non-final
    stage.

The async wait layer supplies the clock, checks whether a stage remains to
terminate, publishes the event, and then performs termination.

Tests

A Hypothesis RuleBasedStateMachine exercises randomized completion orders
across 1–8 stages, plus pinned examples for completion order, final and
single-stage failures, all-success runs, and exactly-once fail-fast.

Review follow-up

  • Route the fail-fast decision exclusively through the typed pipeline_fail_fast
    observe event; metrics, tracing, and structured logging consume the same
    domain-neutral output.
  • Prevent warning-level fail-fast logs from projecting raw argv or arbitrary
    tags, with regression coverage across log, trace, metric, and snapshot-ready
    payloads.
  • Replace timing-based test coordination with FIFO gates released only after
    the event is observed, and guarantee cancellation-test PID cleanup in a
    finally block.

Validation

Full gates green: make check-fmt, make lint (ruff, interrogate 100%, pylint
10.00/10), make typecheck, and make test (898 passed / 52 skipped; behavioural
suites 38 passed, 10 passed / 3 skipped, and 18 passed / 8 skipped; Rust
nextest 104/104).

Closes #73

This branch also carries the pipeline fail-fast telemetry — the
pipeline_fail_fast ExecPhase, the cuprum_pipeline_fail_fast_total counter, the
cuprum.pipeline_fail_fast span event on the failing stage's open span, the
LogLevels.fail_fast_level logging arm, and the adapter and tracing-protocol
changes that carry them. That work is required by
#285, which records its
acceptance criteria; it is not incidental scope.

Closes #285

🤖 Generated with Claude Code

Summary by Sourcery

Extract the pipeline completion decision logic into a pure state transition and
adapt the async handler to delegate to it while preserving behaviour.

Enhancements:

  • Introduce a deterministic record_completion method on PipelineWaitState to
    separate exit-code, timing, and fail-fast bookkeeping from async effects.

Tests:

  • Add Hypothesis state-machine tests and example-based tests validating
    completion ordering, first-failure semantics, and termination decisions.

References

@sourcery-ai

sourcery-ai Bot commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Reviewer's Guide

Extracts pipeline completion decision logic into a pure _PipelineWaitState.record_completion method and adds comprehensive Hypothesis state-machine tests plus example tests to validate completion ordering, failure latching, and fail-fast termination semantics; wires _process_completed_task to use the new method and updates the maturin snapshot for the new test file.

Sequence diagram for pipeline task completion and fail-fast termination

sequenceDiagram
participant Task
participant _process_completed_task
participant _PipelineWaitState
participant _terminate_pipeline_remaining_stages

Task->>_process_completed_task: result()
_process_completed_task->>_PipelineWaitState: record_completion(idx, exit_code, ended_at)
alt [record_completion returns True]
  _process_completed_task->>_terminate_pipeline_remaining_stages: _terminate_pipeline_remaining_stages(processes, wait_tasks, idx, cancel_grace)
end
Loading

File-Level Changes

Change Details Files
Extract pure completion-ordering transition from _process_completed_task into _PipelineWaitState.record_completion and adapt caller.
  • Introduce record_completion(completed_idx, exit_code, *, ended_at) -> bool on _PipelineWaitState to stamp exit codes and end timestamps, latch the first non-zero exit as failure_index, and report whether fail-fast termination should occur.
  • Update _process_completed_task to delegate state mutation and fail-fast decision to record_completion, limiting its responsibilities to reading the clock and performing async termination side effects.
  • Preserve existing behaviour by keeping termination condition semantics identical: only the first non-zero exit on a non-final stage triggers termination of remaining downstream stages.
cuprum/_pipeline_wait.py
Add Hypothesis state-machine tests and example tests for pipeline completion semantics.
  • Introduce _make_wait_state helper to construct a minimal _PipelineWaitState suitable for pure transition testing without event loop or subprocess involvement.
  • Add _PipelineCompletionMachine Hypothesis RuleBasedStateMachine to exercise randomized completion orders, model expected failure index and termination behaviour, and assert invariants about exit-code/timestamp placement and failure-index correctness.
  • Add TestRecordCompletionExamples class with targeted example tests covering boundary scenarios: first-completed failure vs lowest index, final-stage and single-stage failures not requesting termination, all-success runs leaving failure_index unset, and fail-fast firing exactly once.
cuprum/unittests/test_pipeline_wait.py
Update maturin build snapshot to include the new test file.
  • Regenerate test_maturin_build.ambr snapshot so the Rust/maturin wheel manifest accounts for the new test_pipeline_wait.py file.
cuprum/unittests/__snapshots__/test_maturin_build.ambr

Assessment against linked issues

Issue Objective Addressed Explanation
#73 Refactor pipeline completion handling into a pure transition method on _PipelineWaitState (taking state/completed_idx/exit_code and associated data) to isolate the completion-ordering logic from async side effects.
#73 Add Hypothesis state-machine tests that exercise _PipelineWaitState completion ordering, proving first-failure semantics, timing-slot population, and the fail-fast termination decision.
#73 Introduce CrossHair-based verification for the refactored pure transition logic after the refactor. The PR only adds the record_completion method and Hypothesis state-machine tests plus example tests; there is no CrossHair integration or specification/analysis code included.

Possibly linked issues


Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@coderabbitai

coderabbitai Bot commented Jul 28, 2026

Copy link
Copy Markdown

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

Summary

  • Extract completion handling into deterministic _PipelineWaitState.record_completion(...).
  • Latch the first non-zero exit code by completion order.
  • Record completion timestamps and resolve asyncio.wait batch ties by stage index.
  • Verify transitions with Hypothesis, CrossHair, example, asynchronous, and integration tests for Issues #73 and #285.
  • Publish sanitised pipeline_fail_fast events with typed stage metadata, correlation, metrics, tracing, structured logging, and adapter support.
  • Exclude raw arguments and caller-supplied tags from fail-fast warning records.
  • Return per-stage termination outcomes and count only verified terminations.
  • Make timeout and fail-fast teardown resilient to repeated cancellation.
  • Consolidate tracing protocols behind cuprum.adapters.tracing_protocols with compatibility re-exports.
  • Document the behaviour in docs/cuprum-design.md, the user and developer guides, the roadmap, and CHANGELOG.md.
  • Update wheel manifests and related test support.

Walkthrough

The pipeline wait path now records deterministic completion metadata, latches the first failure, publishes a sanitised pipeline_fail_fast event, and terminates remaining stages. Logging, metrics, tracing, lifecycle teardown, tests, and documentation support this behaviour.

Changes

Pipeline fail-fast observability

Layer / File(s) Summary
Deterministic completion and termination
cuprum/_pipeline_wait.py, cuprum/_pipeline_types.py, cuprum/_pipeline_wait_records.py, cuprum/_process_lifecycle.py, cuprum/_pipeline_collect.py
The wait path records stage completion data, selects failures by completion order, emits structured records, and terminates remaining stages.
Event adapter projections
cuprum/events.py, cuprum/adapters/*, cuprum/_testing.py
Fail-fast events carry typed stage metadata. Logging, metrics, and tracing adapters project the event with sanitised fields and correlation rules.
Validation and integration coverage
cuprum/unittests/*, tests/behaviour/*
Tests cover completion transitions, simultaneous completions, event ordering, tag shadowing, adapter output, tracing races, and cancellation-safe teardown.
Contract and operational documentation
docs/*, CHANGELOG.md
Documentation defines fail-fast scope, ordering, event fields, adapter behaviour, timeout handling, cancellation handling, and related operational contracts.

Sequence Diagram(s)

sequenceDiagram
  participant Stage as completed pipeline stage
  participant Wait as _process_completed_task
  participant State as _PipelineWaitState
  participant Event as ExecEvent observer
  participant Adapter as logging metrics tracing adapters
  participant Teardown as pipeline teardown
  Stage->>Wait: return exit code
  Wait->>State: record completion and latch failure
  State->>Event: publish pipeline_fail_fast
  Event->>Adapter: project structured telemetry
  Wait->>Teardown: terminate remaining stages
  Teardown->>Stage: complete escalation and reaping
Loading

Suggested labels: Issue, Roadmap

Poem

Stages finish in ordered flight,
One failure marks the night.
Events carry facts, clean and bright,
Logs and spans record the sight.
Teardown shields the final rite.


Caution

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

  • Ignore

❌ Failed checks (1 error, 3 warnings)

Check name Status Explanation Resolution
Unit Architecture ❌ Error FAIL — the new fail-fast event path hides a wall-clock dependency. The pull request adds _StageObservation.emit_fail_fast() in cuprum/_pipeline_types.py, and that method constructs ExecEvent wit… Inject a narrow wall-clock callable at the observation boundary, for example wall_clock: Callable[[], float], and require production and test constructors to provide it. Use that callable for both lifecycle and pipeline_fail_fast event …
Out of Scope Changes check ⚠️ Warning The PR includes changes that are not clearly required by #73 or #285, including general cancellation-safe teardown changes, timeout teardown updates, and the unrelated line-splitting test-helper refac… Remove these unrelated changes into a separate PR, or link explicit issue requirements that justify them. Keep only changes required for completion-state verification and pipeline fail-fast observability.
Domain Architecture ⚠️ Warning The PR introduces a core-to-logging representation leak. cuprum/_pipeline_types.py adds _PipelineWaitReporter.report_pipeline_wait(message, args, extra), and `_StageObservation.report_pipeline_wai… Remove the direct report_pipeline_wait(message, args, extra) path from _StageObservation and _pipeline_wait_records.py. Represent pipeline completion information with a typed, domain-neutral record or ExecEvent contract. Pass that c…
Performance And Resource Use ⚠️ Warning The logging adapter adds an avoidable per-event allocation on a hot path. Before the pull request, structured_logging_hook() built level_map once and the returned hook performed one lookup per `Ex… Remove the per-event dictionary construction. Build the phase-to-level mapping once in _StructuredLoggingHook.__init__() and perform a lookup in __call__(), or use a match/conditional dispatch that allocates no mapping. Preserve the `…
✅ Passed checks (16 passed)
Check name Status Explanation
Title check ✅ Passed The title accurately identifies the state-machine tests and fail-fast telemetry changes. It references both linked issues, #73 and #285.
Description check ✅ Passed The description clearly explains the completion transition, fail-fast telemetry, tests, validation, and linked issue scope.
Linked Issues check ✅ Passed Mark the linked objectives as met. The changes provide deterministic completion handling and verification coverage for #73, plus typed events, metrics, tracing, logging, sanitisation, documentation, a…
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 165 functions across 29 files. (4 skipped:…
Testing (Overall) ✅ Passed PASS: Verify the changed behaviour is covered by substantive tests. Hypothesis state-machine, CrossHair contracts, pinned examples, and async-boundary tests check completion-slot writes, completion-or…
User-Facing Documentation ✅ Passed Pass the user-facing documentation check. The PR changes docs/users-guide.md and documents the changed pipeline behaviour, including first-failure ordering, upstream and downstream termination, no-o…
Developer Documentation ✅ Passed Pass the developer documentation check. docs/developers-guide.md documents the new _PipelineWaitState command/query seam, completion ordering, _StageWaitContext, pipeline-wait reporting, fail-fa…
Module-Level Documentation ✅ Passed All changed and newly added Python modules have a module-level docstring. The docstrings state each module's purpose and utility. The relevant component relationships are also documented, including `_…
Testing (Unit And Behavioural) ✅ Passed Accept the testing coverage. The unit tests exercise the extracted transition with pinned examples, randomized Hypothesis state-machine checks, CrossHair contracts, batch ordering, timing slots, first…
Testing (Property / Proof) ✅ Passed Pass this check. The PR introduces a Hypothesis RuleBasedStateMachine that varies stage counts from 1–8, completion order, exit codes, and repeated pipeline runs. It checks slot population, first-fa…
Testing (Compile-Time / Ui) ✅ Passed Record PASS. The PR changes Python only; no Rust or TypeScript files changed, so no trybuild-equivalent test applies. The structured adapter projections use focused Syrupy snapshots, including `pipeli…
Observability ✅ Passed Mark Observability as PASS. The changed pipeline wait path logs first-failure, termination-start, and termination-outcome records with stage, exit, correlation, elapsed-time, and confirmed termination…
Security And Privacy ✅ Passed Pass the Security and Privacy check. The PR adds no credentials, secrets, permissions, or unsafe deserialisation. Allowlist enforcement remains before pipeline spawning, and subprocess calls remain pa…
Concurrency And State ✅ Passed PASS — the pull request makes ownership, ordering, locking, task lifetime, and cancellation behaviour explicit, and adds focused interleaving tests. _PipelineWaitState is owned by the single async w…
Architectural Complexity And Maintainability ✅ Passed Keep the change. The new abstractions map to concrete seams: _PipelineWaitState.record_completion and should_terminate_others isolate the required deterministic transition; _StageWaitContext gro…
Rust Compiler Lint Integrity ✅ Passed Pass this check. The complete PR range from merge-base e6775b8 to HEAD contains no changes to Rust source, Cargo manifests, Cargo.lock, or Rust lint configuration. The patch adds no Rust lint suppress…
Full details: Linked Issues check

Explanation

Mark the linked objectives as met. The changes provide deterministic completion handling and verification coverage for #73, plus typed events, metrics, tracing, logging, sanitisation, documentation, and end-to-end tests for #285.

Full details: Out of Scope Changes check

Explanation

The PR includes changes that are not clearly required by #73 or #285, including general cancellation-safe teardown changes, timeout teardown updates, and the unrelated line-splitting test-helper refactor.

Full details: Docstring Coverage

Explanation

Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 165 functions across 29 files. (4 skipped: 4 unsupported.)

Full details: Testing (Overall)

Explanation

PASS: Verify the changed behaviour is covered by substantive tests. Hypothesis state-machine, CrossHair contracts, pinned examples, and async-boundary tests check completion-slot writes, completion-order failure latching, timestamp handling, final/single-stage rules, batch ordering, and exactly-once termination. Direct event tests and real three-stage subprocess tests check fail-fast emission, ordering, correlation, sanitisation, tag shadowing, no-emission cases, and hook failures. Adapter tests check exact metric labels, tracing span lifecycle, warning level, configurable logging, and sensitive-data exclusion. Cancellation tests exercise SIGTERM-to-SIGKILL escalation, repeated cancellation, real subprocess reaping, and late-settling termination counts. The tests assert product state and published payloads, not only mock calls.

Full details: User-Facing Documentation

Explanation

Pass the user-facing documentation check. The PR changes docs/users-guide.md and documents the changed pipeline behaviour, including first-failure ordering, upstream and downstream termination, no-op final or single-stage failures, event ordering, typed stage fields, termination records, cancellation-safe teardown, timeout behaviour, metrics, tracing, and warning-level logging. It also gives migration guidance for the removed cuprum.context.ExecHook import and for exhaustive hooks handling the new pipeline_fail_fast phase. CHANGELOG.md records the new functionality and both breaking changes under [Unreleased]; docs/contents.md identifies the changelog as the project document for release notes and migration impact. The README keeps a high-level signpost to the users' guide, and no additional locale users' guide exists.

Full details: Developer Documentation

Explanation

Pass the developer documentation check. docs/developers-guide.md documents the new _PipelineWaitState command/query seam, completion ordering, _StageWaitContext, pipeline-wait reporting, fail-fast event handling, adapter projections, tracing protocol boundary, verification commands, and cancellation-safe teardown. docs/cuprum-design.md records the pipeline failure policy, event contract, completion ordering decision, and telemetry adapter design. Roadmap item 2.1.2 remains checked off and reflects the completed behaviour. The branch adds no new execplan or ADR, and no parallel documentation locale is present.

Full details: Module-Level Documentation

Explanation

All changed and newly added Python modules have a module-level docstring. The docstrings state each module's purpose and utility. The relevant component relationships are also documented, including _pipeline_wait with _process_lifecycle, _pipeline_streams, and _pipeline_wait_records; _pipeline_wait_records with _pipeline_wait; and tracing_protocols with tracing_adapter. No changed module lacks the required documentation.

Full details: Testing (Unit And Behavioural)

Explanation

Accept the testing coverage. The unit tests exercise the extracted transition with pinned examples, randomized Hypothesis state-machine checks, CrossHair contracts, batch ordering, timing slots, first-failure latching, final and single-stage failures, success paths, hook failures, late-settlement outcomes, and cancellation teardown. The adapter tests cover metrics, tracing, logging, sanitisation, correlation, and error handling. The behavioural BDD tests call the public pipeline API with real stages, and the new wiring tests run real subprocess pipelines through sh.observe() to verify fail-fast events and lifecycle ordering. The tests are included by the configured cuprum/unittests/test_*.py and behavioural test targets. No explicit testing failure condition is introduced.

Full details: Testing (Property / Proof)

Explanation

Pass this check. The PR introduces a Hypothesis RuleBasedStateMachine that varies stage counts from 1–8, completion order, exit codes, and repeated pipeline runs. It checks slot population, first-failure latching, final-stage boundaries, termination decisions, and query purity against an independent model. It also adds substantive bounded CrossHair contracts for slot writes, no-relatch behaviour, termination boundaries, and query purity. The contracts use bounded preconditions and require MessageType.CONFIRMED; they do not silently accept unconfirmed proofs. The normal make test target discovers both new test modules.

Full details: Testing (Compile-Time / Ui)

Explanation

Record PASS. The PR changes Python only; no Rust or TypeScript files changed, so no trybuild-equivalent test applies. The structured adapter projections use focused Syrupy snapshots, including pipeline_fail_fast, stage metadata, metric labels, and sanitised logging fields. The snapshots redact volatile values such as PIDs, durations, paths, wheel versions, platform tags, and SBOM names. Targeted assertions also verify the exact fail-fast log message, log level, event fields, and exclusion of sensitive arguments and tags. The snapshots therefore encode meaningful, stable behaviour without an explicit testing-check failure.

Full details: Unit Architecture

Explanation

FAIL — the new fail-fast event path hides a wall-clock dependency. The pull request adds _StageObservation.emit_fail_fast() in cuprum/_pipeline_types.py, and that method constructs ExecEvent with timestamp=time.time(). _StageObservation has no clock dependency, and its production constructors do not receive one. The new path therefore reads ambient wall-clock state inside an observation command. The tests inject _pipeline_wait.perf_counter, but they do not inject or control the event timestamp clock. This makes the new event path less visible and less deterministic under the Unit Architecture criteria. The existing emit() call does not remove causality: this pull request newly activates the same hidden dependency for pipeline_fail_fast events.

Resolution

Inject a narrow wall-clock callable at the observation boundary, for example wall_clock: Callable[[], float], and require production and test constructors to provide it. Use that callable for both lifecycle and pipeline_fail_fast event timestamps. Remove direct time.time() calls from _StageObservation. Pass a deterministic fake clock in unit tests and assert the emitted event timestamp. Keep the injected monotonic perf_counter separate for elapsed-duration calculations.

Full details: Domain Architecture

Explanation

The PR introduces a core-to-logging representation leak. cuprum/_pipeline_types.py adds _PipelineWaitReporter.report_pipeline_wait(message, args, extra), and _StageObservation.report_pipeline_wait() discovers and calls that method on every observe hook. cuprum/_pipeline_wait_records.py supplies logging-shaped messages, positional formatting arguments, and cuprum_* log fields. Only cuprum/adapters/logging_adapter.py implements the method by calling logger.warning(...). This bypasses the domain-neutral ExecEvent adapter boundary. The core imports no concrete adapter, but the new protocol still exposes adapter implementation concerns through a logging-specific API. The changed code therefore violates the requirement that core domain logic must not reach into adapter implementation concerns.

Resolution

Remove the direct report_pipeline_wait(message, args, extra) path from _StageObservation and _pipeline_wait_records.py. Represent pipeline completion information with a typed, domain-neutral record or ExecEvent contract. Pass that contract through the existing observe boundary. Make logging_adapter.py translate the contract into messages, levels, and logging extras. Keep metrics and tracing dependent only on their adapter-specific projections, and test the core without importing or structurally detecting logging adapters.

Full details: Observability

Explanation

Mark Observability as PASS. The changed pipeline wait path logs first-failure, termination-start, and termination-outcome records with stage, exit, correlation, elapsed-time, and confirmed termination fields. It emits a sanitized pipeline_fail_fast event before termination. The event omits argv, environment, working directory, and caller tags. MetricsHook adds cuprum_pipeline_fail_fast_total with only program and project labels. TracingHook records cuprum.pipeline_fail_fast on the failing stage's existing exec_id-keyed span. structured_logging_hook renders the decision at configurable fail_fast_level, WARNING by default. Tests and documentation cover ordering, no-emission cases, privacy, cardinality, correlation, and adapter handling.

Full details: Security And Privacy

Explanation

Pass the Security and Privacy check. The PR adds no credentials, secrets, permissions, or unsafe deserialisation. Allowlist enforcement remains before pipeline spawning, and subprocess calls remain parameterized. The new fail-fast event removes argv, cwd, env, tags, output, and error text. Logging applies the same sanitisation and emits only bounded decision fields plus the intended execution correlation ID. Metrics use only program and project labels. Tracing records only typed stage, exit, and duration fields on the matching span. Added test markers such as pipeline-secret and fail-fast-sensitive-value are clearly synthetic and unusable. The project name and execution ID exposure is intentional and required by the linked telemetry contract; project metadata was already present in stage tags.

Full details: Performance And Resource Use

Explanation

The logging adapter adds an avoidable per-event allocation on a hot path. Before the pull request, structured_logging_hook() built level_map once and the returned hook performed one lookup per ExecEvent. The new _StructuredLoggingHook.__call__() calls _level_for() for every event, and _level_for() constructs a six-entry dictionary on every call (cuprum/adapters/logging_adapter.py:99-108). This affects high-volume stdout and stderr streams, including events that logging filters out, and the pull request provides no benchmark or complexity evidence for the regression. The pipeline ordering sort and per-stage teardown collections are linear or bounded by the number of pipeline stages and are not the failure.

Resolution

Remove the per-event dictionary construction. Build the phase-to-level mapping once in _StructuredLoggingHook.__init__() and perform a lookup in __call__(), or use a match/conditional dispatch that allocates no mapping. Preserve the pipeline_fail_fast mapping and add a regression benchmark or profiling check that exercises a realistic high-volume event stream with logging disabled.

Full details: Concurrency And State

Explanation

PASS — the pull request makes ownership, ordering, locking, task lifetime, and cancellation behaviour explicit, and adds focused interleaving tests. _PipelineWaitState is owned by the single async wait loop. _StageWaitContext is immutable, and the wait state copies its timing data. asyncio.wait batches are processed in stage order. Completion transitions are tested with Hypothesis and CrossHair. Termination tasks have an owned shielded lifetime, repeated cancellation is handled and re-raised after teardown, and late-settling targets are tested. TracingHook protects its shared execution map with a lock and uses per-span state locks. Tests cover blocked span callbacks, exit races, duplicate tokens, and unrelated executions. No unowned global mutable state or untested ordering or cancellation path was introduced.

Full details: Architectural Complexity And Maintainability

Explanation

Keep the change. The new abstractions map to concrete seams: _PipelineWaitState.record_completion and should_terminate_others isolate the required deterministic transition; _StageWaitContext groups immutable, stage-indexed wait data; _pipeline_wait_records.py separates report payloads from async control flow; and _ActiveSpan plus per-span locks enforce the required tracing lifecycle invariant. The optional _PipelineWaitReporter has one immediate consumer in _StructuredLoggingHook and carries distinct first-failure and termination-outcome records, while the typed pipeline_fail_fast event remains the shared metrics, tracing, and event-logging path. The tracing protocol move has a clear public boundary and a compatibility re-export. No dependency manifest changes, registries, global mutable state, or production import cycle were introduced. Shared test helpers serve multiple new test modules, and the production modules remain within the repository's 400-line module limit.

Full details: Rust Compiler Lint Integrity

Explanation

Pass this check. The complete PR range from merge-base e6775b8 to HEAD contains no changes to Rust source, Cargo manifests, Cargo.lock, or Rust lint configuration. The patch adds no Rust lint suppressions or clone calls. Existing Rust expectations are outside the PR diff, so the custom Rust integrity failure conditions are not applicable.

✨ Finishing Touches 💡 1
🛠️ Fix failing CI checks 💡
  • Create stacked PR
  • Commit on current branch
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch python-pipeline-wait-tests

Warning

Your free Security trial is over. An organization admin can activate billing to continue.


Comment @coderabbitai help to get the list of available commands.

codescene-access[bot]

This comment was marked as outdated.

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've reviewed your changes and they look great!


Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

@coderabbitai coderabbitai Bot added the Issue label Jul 28, 2026
chatgpt-codex-connector[bot]

This comment was marked as resolved.

coderabbitai[bot]

This comment was marked as resolved.

codescene-access[bot]

This comment was marked as outdated.

codescene-access[bot]

This comment was marked as outdated.

codescene-access[bot]

This comment was marked as outdated.

codescene-access[bot]

This comment was marked as outdated.

codescene-access[bot]

This comment was marked as outdated.

@buzzybee-df12

Copy link
Copy Markdown
Collaborator

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jul 29, 2026

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@leynos

leynos commented Jul 29, 2026

Copy link
Copy Markdown
Owner Author

@coderabbitai Have the following failed checks now been resolved?

If further work is required, please provide an AI agent prompt for the remaining work to be done to address these failures.

Do not treat warnings as optional or aspirational. Where a change is out of scope for this PR, propose a GitHub issue unless one exists already. (Treat documentation and validation coverage as in scope).

#73 Introduce CrossHair-based verification for the refactored pure transition logic after the refactor. ❌ The PR only adds the record_completion method and Hypothesis state-machine tests plus example tests; there is no CrossHair integration or specification/analysis code included.

@coderabbitai

This comment was marked as resolved.

@leynos

This comment was marked as resolved.

@coderabbitai

This comment was marked as resolved.

@pandalump

Copy link
Copy Markdown
Collaborator

Completed issue #73's CrossHair criterion and both unresolved pre-merge warnings in 28ca2a7. No follow-up issue raised — this was in scope.

CrossHair verification

cuprum/unittests/test_pipeline_wait_crosshair.py adds four PEP 316 contracts over a bounded symbolic model:

Contract Invariant
records_completion the completion writes its exit code and timestamp to its own slot, and no other slot is touched
first_failure_latch the first non-zero completion in completion order latches failure_index, and a later non-zero completion never replaces it
should_terminate_others true exactly for a non-final first failure — the bounds start at one stage, so final-stage and single-stage failures are both covered
query_purity repeated queries agree and leave exit_codes, ended_at, and failure_index unchanged

The model is deliberately small: preconditions cap the pipeline at three stages, exit codes at -2..2, and timestamps at 0.0..4.0, and the state is constructed directly with only the fields the pure transition reads — no asyncio task, subprocess, or clock enters the symbolic space.

I confirmed the contracts actually verify rather than rubber-stamp. Two mutations of the production code both produce POST_FAIL instead of CONFIRMED:

  • dropping the final-stage exclusion from should_terminate_others;
  • letting a later failure re-latch failure_index.

The import-time probe follows test_line_splitting.py and degrades to a skip only for a missing dependency (ImportError) or a tracer that cannot handle the interpreter (TraceException); everything else is re-raised, so a supported interpreter runs the verification rather than warning past it.

One thing I changed beyond the brief: that probe logic already existed in test_line_splitting.py, and AGENTS.md requires sweeping for an existing equivalent before adding a helper. Rather than duplicate ~50 lines, the three helpers moved to a shared cuprum/unittests/_crosshair_support.py that both modules import. That also answers the "add harness tests if new probe logic is introduced" point — no new probe logic exists, and test_line_splitting.py's eleven harness tests cover the shared code unchanged (all 25 tests in that module still pass).

Observability

_process_completed_task now emits two structured records via logging.getLogger(__name__), distinguished by a stable cuprum_action and sharing cuprum_stage_index, cuprum_exit_code, and cuprum_duration_s:

  • pipeline_stage_first_failure — when a completion newly latches failure_index (captured by comparing against the pre-command value);
  • pipeline_fail_fast_termination — immediately before _terminate_pipeline_remaining_stages is awaited.

Neither fires for a successful exit, a later failure once latched, or a final-stage or single-stage failure. Logging stays in the async caller — moving it into record_completion or should_terminate_others would break the determinism the symbolic verification depends on.

Five log-capture tests pin those cases with a monkeypatched clock (so elapsed times are exact, not timing-dependent), and assert the termination-call count stays consistent with the records. Also mutation-checked: logging the first-failure record unconditionally, or dropping the termination record, each fail them.

Documentation

docs/developers-guide.md gains a _pipeline_wait subsection under Pipeline execution helper contracts covering the command, the query, _process_completed_task's runtime ownership, the log records and their fields, the bounded symbolic model, the supported-interpreter behaviour, and both commands:

uv run pytest -q cuprum/unittests/test_pipeline_wait_crosshair.py -m crosshair
uv run crosshair check cuprum/unittests/test_pipeline_wait_crosshair.py --analysis_kind=PEP316

docs/cuprum-design.md gains a matching note so the two stay consistent.

Validation

Command Outcome
uv run pytest cuprum/unittests/test_pipeline_wait.py 14 passed
uv run pytest cuprum/unittests/test_pipeline_wait_crosshair.py 4 passed (all CONFIRMED)
uv run crosshair check ... --analysis_kind=PEP316 exit 0, no counterexamples
uv run pytest cuprum/unittests/test_line_splitting.py 25 passed (shared probe unchanged)
make check-fmt pass
make lint pass (ruff, interrogate 100%, pylint 10.00/10, clippy)
make test pass — 773 passed / 47 skipped, Rust nextest 57/57
make markdownlint pass (0 errors)
make nixie pass

Two issues surfaced during validation and were fixed: pylint counts cls, so the from_state classmethod became a module-level _completion_log_fields (4 args); and the new _crosshair_support.py changed the wheel manifest, so the maturin snapshot was regenerated. The existing Hypothesis state-machine and async-boundary tests are unchanged — CrossHair complements them.

codescene-access[bot]

This comment was marked as outdated.

codescene-access[bot]

This comment was marked as outdated.

lodyai Bot pushed a commit that referenced this pull request Jul 29, 2026
Completes issue #75's CrossHair criterion. The Hypothesis module is
unchanged; this adds bounded symbolic verification alongside it.

Five PEP 316 contracts, executed through check_states and required to
return MessageType.CONFIRMED:

_resolve_timeout_payload — a carried _SubprocessTimeoutError returns its
own timeout, stdout, stderr, and exit time verbatim (the fallback in that
contract holds different values and a None configured timeout, so a
resolver consulting it would return a wrong field or raise); a bare
TimeoutError with a configured timeout returns the fallback's four fields
exactly; a bare TimeoutError without one raises _SubprocessInvariantError.

_stages_to_terminate — the selection is in range, unique, ordered,
excludes the failure index, contains only stages whose done flag is
False, and equals exactly the unfinished non-failed set; and cleanup is
idempotent, since marking the selected stages done makes a second pass
select nothing.

The domains are bounded so CrossHair exhausts them rather than returning
CANNOT_CONFIRM: at most three stages with failure_index constrained by
precondition, done flags encoded as one bounded integer bitmask rather
than a symbolic list of symbolic booleans, and three-value enumerations
for times and text (including None) since the reducers only copy those
values while keeping carried and fallback distinguishable.

Verified genuine rather than assumed: making the carried branch read the
fallback's stdout, and dropping the failed-stage exclusion from the
selection, each yield POST_FAIL instead of CONFIRMED.

The module matches PYTEST_TARGETS' cuprum/unittests/test_*.py glob, so
make test collects and runs it — the checks are automated, not just
documented. Availability uses the shared _crosshair_support.py probe
(byte-identical to the copy on the #243 branch, so the two merge
cleanly), which degrades to a skip only for a missing dependency or an
untraceable interpreter; anything else, including CANNOT_CONFIRM, fails.

Documents the verified invariants, both commands, and the deliberate
bounds in the developers guide, and regenerates the wheel manifest for
the two new files.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
codescene-access[bot]

This comment was marked as outdated.

@leynos

leynos commented Jul 29, 2026

Copy link
Copy Markdown
Owner Author

@coderabbitai Please suggest a fix for this issue and supply a prompt for an AI coding agent to enable it to apply the fix. Include the file and symbol names indicated in the issue at the head of your response. Ensure that this is validated against the current version of the codegraph.

If further refinement to address this finding would be deleterious, please supply a clear explanatory one to two paragraph markdown message I can paste into the CodeScene web ui's diagnostic suppression function so this diagnostic can be silenced.

Excess Number of Function Arguments

cuprum/unittests/test_pipeline_wait_crosshair.py: _latches_first_failure

What lead to degradation?

_latches_first_failure has 5 arguments, max arguments = 4

Why does this problem occur?

Functions with many arguments indicate either a) low cohesion where the function has too many responsibilities, or b) a missing abstraction that encapsulates those arguments.
The threshold for the Python language is 4 function arguments.

How to fix it?

Start by investigating the responsibilities of the function. Make sure it doesn't do too many things, in which case it should be split into smaller and more cohesive functions. Consider the refactoring INTRODUCE PARAMETER OBJECT to encapsulate arguments that refer to the same logical concept.

@leynos

leynos commented Jul 29, 2026

Copy link
Copy Markdown
Owner Author

@coderabbitai Please suggest a fix for this issue and supply a prompt for an AI coding agent to enable it to apply the fix. Include the file and symbol names indicated in the issue at the head of your response. Ensure that this is validated against the current version of the codegraph.

If further refinement to address this finding would be deleterious, please supply a clear explanatory one to two paragraph markdown message I can paste into the CodeScene web ui's diagnostic suppression function so this diagnostic can be silenced.

Excess Number of Function Arguments

cuprum/unittests/test_pipeline_wait_crosshair.py: _first_failure_latch_contract

What lead to degradation?

_first_failure_latch_contract has 5 arguments, max arguments = 4

Why does this problem occur?

Functions with many arguments indicate either a) low cohesion where the function has too many responsibilities, or b) a missing abstraction that encapsulates those arguments.
The threshold for the Python language is 4 function arguments.

How to fix it?

Start by investigating the responsibilities of the function. Make sure it doesn't do too many things, in which case it should be split into smaller and more cohesive functions. Consider the refactoring INTRODUCE PARAMETER OBJECT to encapsulate arguments that refer to the same logical concept.

@coderabbitai

This comment was marked as resolved.

@coderabbitai

This comment was marked as resolved.

@pandalump

Copy link
Copy Markdown
Collaborator

CodeScene finding cleared in 5457037 — file health 9.68 → 10.0.

One decision worth surfacing

The two specifications conflicted, so I checked the actual finding before choosing:

🚩 Issue: Excess Number of Function Arguments
   Functions: _latches_first_failure         at line 116 (Arguments = 5)
              _first_failure_latch_contract  at line 177 (Arguments = 5)

CodeScene flags both functions. The second specification asked to refactor only _latches_first_failure and explicitly keep _first_failure_latch_contract's five scalars — that would have left half the finding unresolved.

So I refactored both, but used the second specification's tuple shape: one (stage_index, exit_code) pair per completion, rather than the first's grouping by kind (completion_indexes, exit_codes). That is what the first specification's own rationale argues for — "two cohesive completion records in completion order, each consisting of a stage index and exit code" — and it keeps the two values describing a single event together instead of splitting them across parallel arguments.

Both functions now take three arguments:

_latches_first_failure:        ['stage_count', 'first_completion', 'second_completion']
_first_failure_latch_contract: ['stages', 'first_completion', 'second_completion']

The helper unpacks both tuples immediately; the state construction, the ordered record_completion calls with their 1.0 / 2.0 timestamps, and the latch assertions are untouched. No dataclass, NamedTuple, or runtime construction — plain typed tuples keep CrossHair's inputs primitive and bounded.

The symbolic domain is preserved exactly, with the preconditions now bounding each tuple element:

pre: 2 <= stages <= 3
pre: 0 <= first_completion[0] < stages
pre: 0 <= second_completion[0] < stages
pre: first_completion[0] != second_completion[0]
pre: -2 <= first_completion[1] <= 2
pre: -2 <= second_completion[1] <= 2
post: _latches_first_failure(stages, first_completion, second_completion)

The refactor did not weaken the verification

Symbolic tuple parameters could in principle degrade CrossHair's exploration, so I checked rather than trusting four green ticks. Two mutations of record_completion both still yield POST_FAIL instead of CONFIRMED:

  • letting a later failure re-latch failure_index;
  • latching a hard-coded index instead of completed_idx.

Commands and outcomes

Command Outcome
uv run pytest cuprum/unittests/test_pipeline_wait_crosshair.py -m crosshair 4 passed, all CONFIRMED
uv run pytest cuprum/unittests/test_pipeline_wait_crosshair.py 4 passed
uv run crosshair check … --analysis_kind=PEP316 exit 0, no counterexamples
uv run ruff check / ruff format clean, unchanged
cs review …test_pipeline_wait_crosshair.py 10.0 ✅, no findings
make check-fmt pass
make lint (ruff, interrogate 100%, pylint 10.00/10) pass
make typecheck pass
make test pass — Rust nextest 57/57, full Python suite

cuprum/_pipeline_wait.py, the CrossHair probe and fallback handling, documentation, snapshots, and the other three contracts are all untouched.

leynos added 12 commits August 25, 2026 12:49
Preserve the lost-wakeup fallback while passing immutable stage context
through the updated wait API. Align the legacy and synthetic test doubles,
stabilise timeout teardown startup, and regenerate the wheel manifest.
Preserve the rebased fail-fast and cancellation behaviour while applying
the formatter and NumPy return documentation required by the new main
branch lint policy.
Extract the independent logging, tracing, and metrics assertion groups so
the Hypothesis property remains a concise coordinator without weakening its
projection contracts.
Route pipeline-wait warnings through the structured logging adapter,
keep the sanitised event's trusted project label, and serialise same-stage
trace callbacks against exit.

Cover record ordering, process-exit polling isolation, and the fail-fast
tracing race while updating the user and design contracts.
Move the record-ordering contract out of the async-boundary suite so
each pipeline-wait test module stays below the repository's size limit.

Document the complete split suite and regenerate the wheel-manifest
snapshot for the new test module.
Count one verified outcome for every selected teardown target, excluding
stages that settle after selection before they can be terminated.

Cover the race directly, split batch completion coverage into its own
module, and clarify the completion-record and event channels.
Carry each record's action, message, and optional fields in one immutable
report value so the adapter boundary no longer accepts an excessive argument
surface.

Keep the published fields unchanged and assert that teardown outcome fields
remain exclusive to the termination-outcome record.
Match target-era pipeline tests to the current wait context and final-stage
fail-fast policy. Restore formatter-required spacing and remove the replayed
documentation blank line.
Keep legacy tracing imports as compatibility re-exports while making the
public module the sole protocol definition. Document the separate direct
pipeline-wait record contract and extract the verified-teardown race setup.
Preserve the direct pipeline-wait records while adding their execution
correlation token to the structured fail-fast log projection. Clarify
the associated telemetry and protocol boundaries, and improve failure
diagnostics without changing teardown semantics.
Make lifecycle and fail-fast event timestamps deterministic at the
observation boundary, and avoid rebuilding logging level maps for every
event. Keep the reporter and structured-event channels distinct in the
user documentation.
@leynos
leynos force-pushed the python-pipeline-wait-tests branch from aa614eb to 60ca50c Compare August 25, 2026 10:57
codescene-access[bot]

This comment was marked as outdated.

Restore target-branch timeout observability documentation that was lost
during the rebase, while retaining the pipeline fail-fast additions.

Keep `TracingHook` below the module-size limit without weakening its
documented attribute contract, and resolve the remaining Markdown and
spelling gate findings.
codescene-access[bot]

This comment was marked as outdated.

@leynos

leynos commented Aug 25, 2026

Copy link
Copy Markdown
Owner Author

@coderabbitai Please investigate the cause of the following issue using codegraph exploration and research, identify a fix and provide an AI coding agent prompt for the fix:

=================================== FAILURES ===================================
____________ test_warn_crosshair_unavailable_emits_runtime_warning _____________

    def test_warn_crosshair_unavailable_emits_runtime_warning() -> None:
        """Warning helper reports the CrossHair fallback reason."""
        reason = "CrossHair unavailable: TraceException: unsupported opcode"
    
        with pytest.warns(RuntimeWarning) as warning_info:
            _warn_crosshair_unavailable(reason)
    
        message = str(warning_info[0].message)
>       assert reason in message, "warning includes failure reason"
E       AssertionError: warning includes failure reason
E       assert 'CrossHair unavailable: TraceException: unsupported opcode' in "Implicitly cleaning up <HTTPError 429: 'transient'>"

cuprum/unittests/test_line_splitting.py:192: AssertionError
--------------------------- snapshot report summary ----------------------------
17 snapshots passed.
=========================== short test summary info ============================
FAILED cuprum/unittests/test_line_splitting.py::test_warn_crosshair_unavailable_emits_runtime_warning - AssertionError: warning includes failure reason
assert 'CrossHair unavailable: TraceException: unsupported opcode' in "Implicitly cleaning up <HTTPError 429: 'transient'>"
============ 1 failed, 1137 passed, 54 skipped in 88.50s (0:01:28) =============

https://github.com/leynos/cuprum/actions/runs/32840569935/job/97778914224?pr=243

@coderabbitai

This comment was marked as resolved.

@leynos

leynos commented Aug 25, 2026

Copy link
Copy Markdown
Owner Author

@coderabbitai Have the following failed checks now been resolved?

If further work is required, please provide an AI agent prompt for the remaining work to be done to address these failures.

Do not treat warnings as optional or aspirational. Where a change is out of scope for this PR, propose a GitHub issue unless one exists already. (Treat o11y, code safety, documentation and validation coverage as in scope).

❌ Failed checks (1 error, 2 warnings)

Check name Status Explanation Resolution
Unit Architecture ❌ Error FAIL — the new fail-fast event path hides a wall-clock dependency. The pull request adds _StageObservation.emit_fail_fast() in cuprum/_pipeline_types.py, and that method constructs ExecEvent wit… Inject a narrow wall-clock callable at the observation boundary, for example wall_clock: Callable[[], float], and require production and test constructors to provide it. Use that callable for both lifecycle and pipeline_fail_fast event …
Domain Architecture ⚠️ Warning The PR introduces a core-to-logging representation leak. cuprum/_pipeline_types.py adds _PipelineWaitReporter.report_pipeline_wait(message, args, extra), and `_StageObservation.report_pipeline_wai… Remove the direct report_pipeline_wait(message, args, extra) path from _StageObservation and _pipeline_wait_records.py. Represent pipeline completion information with a typed, domain-neutral record or ExecEvent contract. Pass that c…
Performance And Resource Use ⚠️ Warning The logging adapter adds an avoidable per-event allocation on a hot path. Before the pull request, structured_logging_hook() built level_map once and the returned hook performed one lookup per `Ex… Remove the per-event dictionary construction. Build the phase-to-level mapping once in _StructuredLoggingHook.__init__() and perform a lookup in __call__(), or use a match/conditional dispatch that allocates no mapping. Preserve the `…

@leynos

leynos commented Aug 25, 2026

Copy link
Copy Markdown
Owner Author

@coderabbitai Have the following now been resolved?

cuprum/unittests/test_pipeline_fail_fast_tag_shadowing.py (1)

115-115: 📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win
Use phase for event selection.
Replace the local list comprehension with phase(events, _FAIL_FAST_PHASE).

Keep phase filtering in the shared helper.
As per coding guidelines: “Use composition and reusable functions to avoid

duplication”.

Proposed change
-    (event,) = [item for item in events if item.phase == _FAIL_FAST_PHASE]
+    (event,) = phase(events, _FAIL_FAST_PHASE)
🤖 Detailed instructions

Use a wyvern agent team to verify each finding against the current code and only fix it if needed. Fix any documentation issues using a scribe agent team. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. Use a scrutineer agent for execution of tests and summarizing the results.

In @cuprum/unittests/test_pipeline_fail_fast_tag_shadowing.py at line 115, In
the event selection within test_pipeline_fail_fast_tag_shadowing, replace the
local list comprehension filtering on _FAIL_FAST_PHASE with the shared
phase(events, _FAIL_FAST_PHASE) helper, preserving the single-event unpacking
behavior.

Source: Coding guidelines

docs/cuprum-design.md (1)

1257-1261: 🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
Limit the reducer statement to fail-fast event publication.
Change “before it reports anything”. _process_completed_task logs

pipeline_stage_first_failure before it checks terminate_others. A final-stage,

single-stage, or already-settled non-final failure emits that direct record even

when no pipeline_fail_fast event is published.

Proposed wording
-`cuprum._pipeline_wait` consults the same reducer *before* it reports anything,
-so a completion that would select no stages is not announced as a fail-fast
-decision at all: no `pipeline_fail_fast` event is published.
+`cuprum._pipeline_wait` consults the same reducer before deciding whether to
+publish a fail-fast event. A completion that would select no stages is not
+announced as a `pipeline_fail_fast` decision, but its first-failure record is
+still published.
🤖 Detailed instructions

Use a wyvern agent team to verify each finding against the current code and only fix it if needed. Fix any documentation issues using a scribe agent team. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. Use a scrutineer agent for execution of tests and summarizing the results.

In @docs/cuprum-design.md around lines 1257 - 1261, Update the documentation
around cuprum._pipeline_wait to state that the reducer is consulted only before
publishing a pipeline_fail_fast event. Clarify that completions selecting no
stages do not publish that fail-fast event, while _process_completed_task still
publishes the pipeline_stage_first_failure record.

docs/users-guide.md (1)

959-960: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Describe cuprum_pipeline_fail_fast_total as a decision counter.
The wait path publishes pipeline_fail_fast before _terminate_and_report starts.

MetricsHook increments the counter from that event. The counter therefore

records a fail-fast decision, not a completed teardown.

Proposed wording
-- `cuprum_pipeline_fail_fast_total`: Counter incremented once per pipeline torn
-  down early because a non-final stage was the first to fail
+- `cuprum_pipeline_fail_fast_total`: Counter incremented once per fail-fast
+  decision when a non-final stage is the first to fail and another stage is
+  still running
🤖 Detailed instructions

Use a wyvern agent team to verify each finding against the current code and only fix it if needed. Fix any documentation issues using a scribe agent team. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. Use a scrutineer agent for execution of tests and summarizing the results.

In @docs/users-guide.md around lines 959 - 960, Update the
cuprum_pipeline_fail_fast_total documentation to describe it as incrementing
once per fail-fast decision, triggered when a non-final stage fails first while
another stage remains running, rather than as a completed pipeline teardown.

@leynos

leynos commented Aug 25, 2026

Copy link
Copy Markdown
Owner Author

@coderabbitai Have the following now been resolved?

docs/users-guide.md (2)

992-996: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Remove the second-person pronoun at Line 996.
"which you can then use to find the individual stage spans behind the spike" still contains "you". The path instructions forbid second-person pronouns in docs/ outside README.md. This was raised on the thread anchored at Lines 993-996 and remains open.

📝 Proposed wording
-`cuprum_exec_id`. Both fields carry the same existing execution token, which
-you can then use to find the individual stage spans behind the spike.
+`cuprum_exec_id`. Both fields carry the same existing execution token, which
+then identifies the individual stage spans behind the spike.

Triage: [type:docstyle]
As per path instructions: "Avoid 2nd person or 1st person pronouns ("I", "you", "we"), exceptions: README.md, BDD .feature files."

🤖 Detailed instructions

Use a wyvern agent team to verify each finding against the current code and only fix it if needed. Fix any documentation issues using a scribe agent team. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. Use a scrutineer agent for execution of tests and summarizing the results.

In @docs/users-guide.md around lines 992 - 996, Revise the sentence in the
documentation paragraph around ExecEvent.exec_id and cuprum_exec_id to remove
the second-person pronoun while preserving the meaning that the shared execution
token identifies the individual stage spans behind the spike.

Source: Path instructions
278-282: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
State the retained runtime fields.
This passage says that "only the program, trusted configured project, typed decision fields, and ExecEvent.exec_id remain". The event builder also sets pid and timestamp, so the word "only" is inaccurate. Describe the removal of the caller-controlled fields instead, and list the retained runtime fields. The same point was raised on the thread anchored at Lines 278-282.

📝 Proposed wording
The event carries the failing stage's index, pipeline width, exit code,
-duration, and execution token. It is sanitized: `argv` is empty, `cwd` and
-`env` are `None`, and `tags` is empty; only the program, trusted configured
-project, typed decision fields,
-and `ExecEvent.exec_id` remain.
+duration, and execution token. It is sanitized: the caller-controlled `argv`,
+`cwd`, `env`, and `tags` are removed, while the program, trusted configured
+project, typed decision fields, `pid`, `timestamp`, and `ExecEvent.exec_id`
+remain.

Run the following script to confirm which fields the builder still populates:

#!/bin/bash
set -eu

printf '%s\n' '--- fail-fast event construction ---'
rg -n -C 20 'pipeline_fail_fast' cuprum/_pipeline_types.py
🤖 Detailed instructions

Use a wyvern agent team to verify each finding against the current code and only fix it if needed. Fix any documentation issues using a scribe agent team. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. Use a scrutineer agent for execution of tests and summarizing the results.

In @docs/users-guide.md around lines 278 - 282, Update the event documentation
near the sanitized-event description to avoid saying “only” the listed fields
remain; describe the removal of caller-controlled fields instead, and explicitly
list all retained runtime fields, including pid and timestamp alongside the
program, trusted configured project, typed decision fields, and
ExecEvent.exec_id.

@coderabbitai

coderabbitai Bot commented Aug 25, 2026

Copy link
Copy Markdown

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '== Current revision =='
git rev-parse --short HEAD
git show --no-patch --format='%H%n%s' HEAD

printf '%s\n' '== Changed target files =='
git diff --name-status origin/main...HEAD | rg \
  'cuprum/(_pipeline_types|_pipeline_wait|_pipeline_wait_records)\.py|cuprum/adapters/logging_adapter\.py|test_.*(pipeline|adapter).*\.py|docs/' \
  || true

printf '%s\n' '== Pipeline types outline =='
ast-grep outline cuprum/_pipeline_types.py --items all \
  --match 'StageObservation|PipelineWaitReporter|StageWaitContext' --view expanded

printf '%s\n' '== Pipeline event time source, reporter port, and constructors =='
rg -n -C 8 \
  'time\.(time|monotonic)|datetime|def emit_fail_fast\b|def emit\b|class _PipelineWaitReporter\b|def report_pipeline_wait\b|class _StageObservation\b|_StageObservation\(' \
  cuprum/_pipeline_types.py cuprum

printf '%s\n' '== Pipeline-wait reporting record boundary =='
ast-grep outline cuprum/_pipeline_wait_records.py cuprum/_pipeline_wait.py --items all --view expanded
rg -n -C 7 \
  'report_pipeline_wait|_report_completion_event|_Completion(LogFields|Report)|pipeline_stage_first_failure|pipeline_fail_fast_(termination|terminated)|emit_fail_fast' \
  cuprum/_pipeline_wait.py cuprum/_pipeline_wait_records.py cuprum/_pipeline_types.py

printf '%s\n' '== Logging hook level dispatch =='
ast-grep outline cuprum/adapters/logging_adapter.py --items all --match 'StructuredLoggingHook|structured_logging_hook|LogLevels' --view expanded
rg -n -C 8 \
  'class _StructuredLoggingHook|def __call__|def structured_logging_hook|level_map|pipeline_fail_fast|LogLevels' \
  cuprum/adapters/logging_adapter.py

printf '%s\n' '== Focused tests and docs =='
rg -n -C 5 \
  'emit_fail_fast|wall_clock|time\.time|report_pipeline_wait|pipeline_stage_first_failure|pipeline_fail_fast_terminated|level_map|fail_fast_level' \
  cuprum/unittests docs

printf '%s\n' '== Test and lint commands =='
rg -n -C 3 'pytest|ruff|pylint|typecheck|pyright|mypy' pyproject.toml Makefile tox.ini 2>/dev/null || true

Length of output: 107795


🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '== Stage observation timestamp use =='
sed -n '76,178p' cuprum/_pipeline_types.py | cat -n

printf '%s\n' '== Production construction boundary =='
sed -n '88,126p' cuprum/_pipeline_internals.py | cat -n
sed -n '372,402p' cuprum/sh.py | cat -n

printf '%s\n' '== Deterministic observation support and fail-fast tests =='
sed -n '1,120p' cuprum/unittests/_pipeline_wait_support.py | cat -n
rg -n -C 10 \
  'timestamp|12\.5|pipeline_fail_fast|emit_fail_fast|wall_clock' \
  cuprum/unittests/test_pipeline_wait_fail_fast_event.py \
  cuprum/unittests/test_pipeline_fail_fast_wiring.py \
  cuprum/unittests/test_adapter_fail_fast.py \
  cuprum/unittests/test_adapter_projection.py

printf '%s\n' '== Direct-record logger and adapter path =='
sed -n '1,150p' cuprum/_pipeline_wait_records.py | cat -n
sed -n '68,114p' cuprum/adapters/logging_adapter.py | cat -n
rg -n -C 6 \
  'getLogger\("cuprum\._pipeline_wait"\)|report_pipeline_wait\(|pipeline_stage_first_failure|pipeline_fail_fast_termination|pipeline_fail_fast_terminated' \
  cuprum/unittests docs CHANGELOG.md

printf '%s\n' '== Per-event allocation regression search =='
rg -n -C 4 \
  'level_map\s*=\s*\{|_level_by_phase|structured_logging_hook\(' \
  cuprum/adapters/logging_adapter.py cuprum/unittests/test_logging_adapter_properties.py

printf '%s\n' '== Current check status, if GitHub reports it =='
gh pr checks 243 --repo leynos/cuprum || true

Length of output: 84241


@leynos The three listed checks have the following disposition at current head a3cc7b4.

Check Disposition Current evidence
Unit Architecture Resolved _StageObservation requires wall_clock: Callable[[], float]. Both emit(...) and emit_fail_fast(...) set ExecEvent.timestamp from self.wall_clock(). Production constructors inject time.time. Pipeline-wait test support injects a deterministic clock. test_pipeline_wait_fail_fast_event.py asserts event.timestamp == 12.5.
Performance And Resource Use Resolved _StructuredLoggingHook.__init__() creates _level_by_phase once. _StructuredLoggingHook.__call__() performs a lookup through _level_for(event.phase). It does not construct level_map for each event.
Domain Architecture Suppress The proposed removal would break the documented direct pipeline-wait record contract. The direct records and typed pipeline_fail_fast event have separate purposes.

Use this suppression text for the Domain Architecture warning:

Keep the direct pipeline-wait reporter path. It emits the documented
`pipeline_stage_first_failure`, `pipeline_fail_fast_termination`, and
`pipeline_fail_fast_terminated` standard-library records on the
`cuprum._pipeline_wait` logger at a fixed WARNING level. These records provide
ordered lifecycle diagnostics through the stable `cuprum_action` field.

Keep this path separate from the typed `pipeline_fail_fast` `ExecEvent`. The
event drives metrics, tracing, and opt-in structured logging at
`LogLevels.fail_fast_level`. A first-failure record is required even when no
fail-fast event is valid, including a final-stage failure and a completion batch
where no other stage remains running. Replacing the direct records with events
would change the documented logging, ordering, and privacy contracts.

Do not create a new GitHub issue. No remaining code change is valid for these three checks.

The current check status still shows a separate failure in Typecheck and test (Python 3.14). That failure is outside these three CodeScene findings. The earlier CI investigation identified the CrossHair warning-capture failure and the unclosed HTTPError resource leak as its cause. Complete and validate that separate fix before merge.


🧠 Learnings used
Learnt from: pandalump
Repo: leynos/cuprum PR: 243
File: cuprum/unittests/test_pipeline_fail_fast_wiring.py:90-114
Timestamp: 2026-08-06T23:06:36.341Z
Learning: In `cuprum/unittests/test_pipeline_fail_fast_wiring.py`, downstream stdin-reader stages must remain active after `_READ_STDIN` completes. Without `_SETTLE_DELAY_S`, `asyncio.wait(..., FIRST_COMPLETED)` can return all pipeline tasks in one settled batch; ascending stage-index processing then handles the failed stage after its siblings, `_has_stages_to_terminate` returns `False`, and no `pipeline_fail_fast` event is emitted. The module-scoped fixture documentation must retain this rationale.

Learnt from: pandalump
Repo: leynos/cuprum

Timestamp: 2026-08-06T13:50:35.318Z
Learning: In Cuprum pipeline fail-fast observability, `pipeline_stage_first_failure`, `pipeline_fail_fast_termination`, and `pipeline_fail_fast_terminated` are direct standard-library records from `cuprum/_pipeline_wait_records.py` on the `cuprum._pipeline_wait` logger at a fixed `WARNING` level. `LogLevels.fail_fast_level` applies only to the separate opt-in `structured_logging_hook` handling of the `pipeline_fail_fast` `ExecEvent`.

Learnt from: pandalump
Repo: leynos/cuprum PR: 243
File: cuprum/_pipeline_wait_records.py:36-39
Timestamp: 2026-08-05T10:14:47.042Z
Learning: In `cuprum/_pipeline_wait_records.py`, fail-fast pipeline records must use `logging.getLogger("cuprum._pipeline_wait")`, rather than `logging.getLogger(__name__)`. `cuprum._pipeline_wait` is a documented, user-facing logging contract for handlers and filters; `_pipeline_wait_records` is an implementation-detail module.

You are interacting with an AI system.

@leynos

leynos commented Aug 25, 2026

Copy link
Copy Markdown
Owner Author

@coderabbitai Have the following now been resolved?

docs/cuprum-design.md (1)

814-815: 🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
Add project to the documented ExecEvent schema.
Add project: str | None to this code example. The actual ExecEvent contract

defines project, and the fail-fast event retains this trusted value for metrics

projection. The current schema omits it, so it conflicts with the later

fail-fast contract.
As per path instructions: documentation in docs/ is the source of truth and

must stay aligned with behaviour.

🤖 Detailed instructions

Use a wyvern agent team to verify each finding against the current code and only fix it if needed. Fix any documentation issues using a scribe agent team. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. Use a scrutineer agent for execution of tests and summarizing the results.

In @docs/cuprum-design.md around lines 814 - 815, Add project: str | None to the
documented ExecEvent schema alongside stage_index and stage_count, keeping the
example aligned with the actual contract and preserving the fail-fast event’s
project value.

Source: Path instructions

docs/users-guide.md (1)

263-269: 🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
Separate the fixed pipeline-wait records from the structured event adapter.
Replace the statement that the structured logging adapter renders the three canonical wait records. The optional pipeline-wait reporter emits those records at fixed WARNING severity on cuprum._pipeline_wait. structured_logging_hook() separately renders the pipeline_fail_fast ExecEvent at LogLevels.fail_fast_level. Keep these channels distinct so users do not configure or query the wrong observability surface.

Proposed wording
-The structured logging adapter renders these canonical wait records at WARNING.
+The optional pipeline-wait reporter emits these canonical wait records at a
+fixed WARNING level on `cuprum._pipeline_wait`. `structured_logging_hook()`
+separately renders the `pipeline_fail_fast` `ExecEvent` at
+`LogLevels.fail_fast_level`.

Based on learnings, the fixed WARNING pipeline-wait records and the configurable LogLevels.fail_fast_level event path are separate channels.

🤖 Detailed instructions

Use a wyvern agent team to verify each finding against the current code and only fix it if needed. Fix any documentation issues using a scribe agent team. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. Use a scrutineer agent for execution of tests and summarizing the results.

In @docs/users-guide.md around lines 263 - 269, Update the documentation around
the canonical pipeline-wait records to distinguish the optional reporter from
the structured event adapter: state that the reporter emits them at fixed
WARNING severity on cuprum._pipeline_wait, while structured_logging_hook()
separately renders the pipeline_fail_fast ExecEvent at
LogLevels.fail_fast_level.

Source: Learnings

@coderabbitai

coderabbitai Bot commented Aug 25, 2026

Copy link
Copy Markdown

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '== Current revision =='
git rev-parse --short HEAD
git show --no-patch --format='%H%n%s' HEAD

printf '%s\n' '== Sanitized fail-fast event documentation =='
sed -n '260,292p' docs/users-guide.md | cat -n

printf '%s\n' '== Metrics/tracing token documentation =='
sed -n '980,1004p' docs/users-guide.md | cat -n

printf '%s\n' '== First/second-person pronouns in the relevant document ranges =='
rg -n -i -C 2 '\b(I|we|you|our|us)\b' docs/users-guide.md || true

printf '%s\n' '== Pipeline fail-fast event construction =='
ast-grep outline cuprum/_pipeline_types.py --items all \
  --match 'StageObservation|emit_fail_fast' --view expanded
rg -n -C 20 'def emit_fail_fast\b|ExecEvent\(|pipeline_fail_fast|timestamp=|pid=|argv=|cwd=|env=|tags=' \
  cuprum/_pipeline_types.py

Length of output: 10986


@leynos Both findings are resolved at current head a3cc7b4.

  • Second-person pronoun: The reported sentence no longer appears in docs/users-guide.md. The document search found no you occurrence near the specified Lines 992-996. The prohibited second-person wording is removed.

  • Retained runtime fields: The sanitized-event text now states that it removes caller-controlled argv, cwd, env, and tags. It lists retained program, pid, timestamp, trusted configured project, typed decision fields, and ExecEvent.exec_id.

The documentation matches cuprum._pipeline_types._StageObservation.emit_fail_fast(...). That builder sets empty or None caller-controlled fields and continues to populate pid and timestamp.

No further change or follow-up issue is required for these two findings.

You are interacting with an AI system.

@coderabbitai

coderabbitai Bot commented Aug 25, 2026

Copy link
Copy Markdown

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '== Current revision =='
git rev-parse --short HEAD
git show --no-patch --format='%H%n%s' HEAD

printf '%s\n' '== Documented ExecEvent schema =='
sed -n '790,835p' docs/cuprum-design.md | cat -n

printf '%s\n' '== Actual ExecEvent fields and fail-fast construction =='
ast-grep outline cuprum/events.py cuprum/_pipeline_types.py \
  --items all --match 'ExecEvent|emit_fail_fast' --view expanded
rg -n -C 10 \
  'class ExecEvent\b|project:|def emit_fail_fast\b|ExecEvent\(' \
  cuprum/events.py cuprum/_pipeline_types.py

printf '%s\n' '== Pipeline-wait documentation =='
sed -n '245,285p' docs/users-guide.md | cat -n

printf '%s\n' '== Direct record and structured event logging contracts =='
rg -n -C 7 \
  'report_pipeline_wait|pipeline_stage_first_failure|pipeline_fail_fast_termination|pipeline_fail_fast_terminated|getLogger\("cuprum\._pipeline_wait"\)|fail_fast_level|structured_logging_hook|pipeline_fail_fast' \
  cuprum/_pipeline_wait_records.py cuprum/_pipeline_types.py \
  cuprum/adapters/logging_adapter.py docs/users-guide.md

Length of output: 45604


@leynos Mark both findings resolved at current head a3cc7b4.

  1. docs/cuprum-design.mdExecEvent.project schema field

    The documented ExecEvent example now includes:

    project: str | None  # trusted configured project for metrics projection

    This matches cuprum.events.ExecEvent.project. It also matches the sanitized pipeline_fail_fast event contract, which retains the trusted configured project.

  2. docs/users-guide.md — separate reporting channels

    The guide now states that the optional pipeline-wait reporter emits:

    • pipeline_stage_first_failure
    • pipeline_fail_fast_termination
    • pipeline_fail_fast_terminated

    at fixed WARNING level on cuprum._pipeline_wait.

    The guide separately states that structured_logging_hook() renders the typed pipeline_fail_fast ExecEvent at LogLevels.fail_fast_level.

The wording keeps the fixed pipeline-wait records separate from the configurable structured-event adapter. The inspection used the existing fail-fast observability contract learning.


🧠 Learnings used
Learnt from: pandalump
Repo: leynos/cuprum

Timestamp: 2026-08-06T13:50:35.318Z
Learning: In Cuprum pipeline fail-fast observability, `pipeline_stage_first_failure`, `pipeline_fail_fast_termination`, and `pipeline_fail_fast_terminated` are direct standard-library records from `cuprum/_pipeline_wait_records.py` on the `cuprum._pipeline_wait` logger at a fixed `WARNING` level. `LogLevels.fail_fast_level` applies only to the separate opt-in `structured_logging_hook` handling of the `pipeline_fail_fast` `ExecEvent`.

You are interacting with an AI system.

Close handled `HTTPError` instances before retrying or propagating them,
preventing deferred resource-cleanup warnings in CI.

Make CrossHair fallback tests select their own warning so unrelated
warnings cannot change their assertions.
codescene-access[bot]

This comment was marked as outdated.

@codescene-access codescene-access Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No quality gates enabled for this code.

@leynos
leynos merged commit dbe01f1 into main Aug 25, 2026
21 checks passed
@leynos
leynos deleted the python-pipeline-wait-tests branch August 25, 2026 18:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

4 participants