feat(validation): mode=release advisory tier — gate on correctness, not the perf-flake tail#75
Closed
Jammy2211 wants to merge 1 commit into
Closed
feat(validation): mode=release advisory tier — gate on correctness, not the perf-flake tail#75Jammy2211 wants to merge 1 commit into
Jammy2211 wants to merge 1 commit into
Conversation
Carry an `advisory_timeouts` count through the stage report and validation_report; readiness raises a YELLOW reason (validation_advisory_timeout) when nonzero instead of RED. This lets mode=release gate on correctness rather than the perf-flake tail, so a correctness-clean run reaches its shippable GREEN/YELLOW. Docs + workspace-validation.yml updated. Mechanism for #74 (workspace advisory.yaml seeds follow). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
Author
|
Closing in favour of the simpler On reflection the advisory tier is more machinery than warranted here. Its core premise — that Instead, the affected slow real-search scripts will be added to each workspace's |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The
mode=releaseadvisory tier — gate release validation on correctness, not the perf-flake tail (closes #74, mechanism; workspaceadvisory.yamlseeds follow).After the 2026-07-13 tail burn-down (#72) fixed every real bug, the
integratestage still couldn't reach a clean verdict: a population of genuinely-slow real-search scripts flakes around the timeout cap and the failing set shifts run to run, conflating correctness with perf-flakiness into one RED. This teaches Heart to treat a declared-slow timeout as YELLOW: the report carries anadvisory_timeoutscount (fed by Build's runner tier), and readiness raisesvalidation_advisory_timeout(YELLOW) when nonzero — never RED. A timeout on any undeclared script still fails the stage (RED). This is what lets a correctness-clean release reach its shippable GREEN/YELLOW.Paired with PyAutoBuild's runner support (the
timeout_advisorystatus +config/build/advisory.yaml).API Changes
Internal validation contract only — no science-library API.
validation_report.jsongains anadvisory_timeoutscount (schema_version unchanged, additive); readiness gains thevalidation_advisory_timeoutreason key (weight 8, YELLOW). See full details below.Test Plan
tests/test_validate.py—to_stage_reportcarries the advisory count (+ summary fallback); ingest folds it intovalidation_report; status stayspass(new).tests/test_readiness.py—advisory_timeouts > 0on an otherwise-green report → YELLOW (not RED);0stays GREEN (new).268 passed.Full API Changes (for automation & release notes)
Added
validation_report.json/ stage-report fieldadvisory_timeouts: int— declared-slow real-search timeouts (advisory)._Accumulator.advisory_timeouts; summed inadd_stage/add_report; emitted byingest.to_stage_reportcarriesadvisory_timeouts(from the aggregate list, or thesummary.timeout_advisoryfallback).readiness._WEIGHTS["validation_advisory_timeout"] = (8, 8); a YELLOW reason when the report'sadvisory_timeouts > 0.Changed Behaviour
Migration
0(no advisory reason).Validation checklist (--auto run — plan was not pre-approved)
268 passed· smoken/a — heart tooling, no workspace scripts changed· review CLEAN · Heart RED, human-authorized corrective-PR exception forrelease validation FAILED (stage integrate)(the structural reason this change fixes)Generated by the PyAutoLabs agent workflow.