Skip to content

feat(validation): mode=release advisory tier — gate on correctness, not the perf-flake tail#75

Closed
Jammy2211 wants to merge 1 commit into
mainfrom
feature/release-advisory-tier-slow-scripts
Closed

feat(validation): mode=release advisory tier — gate on correctness, not the perf-flake tail#75
Jammy2211 wants to merge 1 commit into
mainfrom
feature/release-advisory-tier-slow-scripts

Conversation

@Jammy2211

Copy link
Copy Markdown
Contributor

Summary

The mode=release advisory tier — gate release validation on correctness, not the perf-flake tail (closes #74, mechanism; workspace advisory.yaml seeds follow).

After the 2026-07-13 tail burn-down (#72) fixed every real bug, the integrate stage still couldn't reach a clean verdict: a population of genuinely-slow real-search scripts flakes around the timeout cap and the failing set shifts run to run, conflating correctness with perf-flakiness into one RED. This teaches Heart to treat a declared-slow timeout as YELLOW: the report carries an advisory_timeouts count (fed by Build's runner tier), and readiness raises validation_advisory_timeout (YELLOW) when nonzero — never RED. A timeout on any undeclared script still fails the stage (RED). This is what lets a correctness-clean release reach its shippable GREEN/YELLOW.

Paired with PyAutoBuild's runner support (the timeout_advisory status + config/build/advisory.yaml).

API Changes

Internal validation contract only — no science-library API. validation_report.json gains an advisory_timeouts count (schema_version unchanged, additive); readiness gains the validation_advisory_timeout reason key (weight 8, YELLOW). See full details below.

Test Plan

  • tests/test_validate.pyto_stage_report carries the advisory count (+ summary fallback); ingest folds it into validation_report; status stays pass (new).
  • tests/test_readiness.pyadvisory_timeouts > 0 on an otherwise-green report → YELLOW (not RED); 0 stays GREEN (new).
  • Full suite: 268 passed.
Full API Changes (for automation & release notes)

Added

  • validation_report.json / stage-report field advisory_timeouts: int — declared-slow real-search timeouts (advisory).
  • _Accumulator.advisory_timeouts; summed in add_stage / add_report; emitted by ingest.
  • to_stage_report carries advisory_timeouts (from the aggregate list, or the summary.timeout_advisory fallback).
  • readiness._WEIGHTS["validation_advisory_timeout"] = (8, 8); a YELLOW reason when the report's advisory_timeouts > 0.

Changed Behaviour

  • A validation report that passed but carries advisory timeouts is now YELLOW (was implicitly GREEN); a report that failed is unchanged (RED dominates).

Migration

  • None. Reports predating the field read as 0 (no advisory reason).

Validation checklist (--auto run — plan was not pre-approved)

  • Effective level: supervised (header: supervised, cap: feature → supervised)
  • Plan: on the issue (feat: mode=release advisory tier — gate on correctness, not the perf-flake tail #74), written at start, unmodified since
  • Gate: tests 268 passed · smoke n/a — heart tooling, no workspace scripts changed · review CLEAN · Heart RED, human-authorized corrective-PR exception for release validation FAILED (stage integrate) (the structural reason this change fixes)
  • Human: plan sound in hindsight?
  • Human: diff matches plan (no scope creep)?
  • Human: merge, amend, or reject — then log the outcome

Generated by the PyAutoLabs agent workflow.

Carry an `advisory_timeouts` count through the stage report and
validation_report; readiness raises a YELLOW reason (validation_advisory_timeout)
when nonzero instead of RED. This lets mode=release gate on correctness rather
than the perf-flake tail, so a correctness-clean run reaches its shippable
GREEN/YELLOW. Docs + workspace-validation.yml updated.

Mechanism for #74 (workspace advisory.yaml seeds follow).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@Jammy2211 Jammy2211 added the pending-release Merged changes awaiting the next release cycle label Jul 14, 2026
@Jammy2211

Copy link
Copy Markdown
Contributor Author

Closing in favour of the simpler no_run approach.

On reflection the advisory tier is more machinery than warranted here. Its core premise — that no_run SLOW-skips erode coverage forever and never converge — is substantially handled by the maintainer's periodic hygiene sweep of no_run. And for scripts that reliably exceed the cap, a skip and an advisory-timeout are equivalent anyway (a timed-out script asserts nothing that run).

Instead, the affected slow real-search scripts will be added to each workspace's no_run.yaml as SLOW and reviewed via hygiene over time — no new mechanism, no second config convention, smaller footprint. See PyAutoHeart#74.

@Jammy2211 Jammy2211 closed this Jul 14, 2026
@Jammy2211 Jammy2211 deleted the feature/release-advisory-tier-slow-scripts branch July 14, 2026 16:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pending-release Merged changes awaiting the next release cycle

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: mode=release advisory tier — gate on correctness, not the perf-flake tail

1 participant