diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.json index 44fbc3c..74fecc9 100644 --- a/.claude-plugin/plugin.json +++ b/.claude-plugin/plugin.json @@ -1,7 +1,7 @@ { "name": "simple-workflow", "description": "The Claude Code plugin for an end-to-end AI development workflow — ticket management, multi-agent code review, security audit, and automated PR creation — built on a Harness for long-running AI agents with strict context management and cross-session learning.", - "version": "8.3.0", + "version": "8.3.1", "author": { "name": "aimsise", "url": "https://github.com/aimsise" diff --git a/CHANGELOG.md b/CHANGELOG.md index c3e86f2..5beaaa2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,28 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [8.3.1] — 2026-06-04 + +**TL;DR.** A fail-open, criticality-gated refinement of v8.3.0's M1 (Gate 8 + the `evidence_floor` ladder) and M5 (criticality floor) — the "Wave A" verification-assurance pass that closes the main residual lead a 2026-06-02 A/B found a human-directed max-effort build still held over the harness. It does NOT add a gate; it sharpens what the EXISTING `thorough` / `exhaustive` floor demands for a **standard-backed computational** AC: **(H1)** the oracle evidence must be ≥2 mutually-validated oracles with ≥1 derived from first principles (the spec formula, no library); **(H2)** un-defers committed seeded fuzz — a fixed-seed property-fuzz loop becomes a depth-gated MUST, not an encouragement; **(H3)** `EC-DIFFERENTIAL` is re-specced to algorithm-vs-algorithm where a second independent algorithm exists (membership is necessary-not-sufficient). **(H13)** ships `skills/impl/references/independent-oracle-harness.md`, the copyable gold-standard four-part oracle module (first-principles block + independent-library block + seeded PRNG + second-algorithm differential helper), wired into the producer rubric, the taxonomy, and the verifier. **(H12)** tightens the M5 criticality cue set (color-science cues + a shared-core / shared-input-boundary trigger) so the class of ACs this rigor targets reliably escalates — the catch only fires if classification fires. Every requirement engages ONLY at the `thorough` / `exhaustive` evidence_floor (already M5-criticality-gated) and ONLY where a published spec / second oracle / second algorithm exists; otherwise it degrades to the single natural channel + a Caveat (never a block). A routine S/M-conservative ticket stays byte-identical to v8.3.0; the `standard` floor and every `constraints.: off` path are unchanged. `v8.4.0` remains reserved for M3 + M4. + +### Added + +- **`skills/impl/references/independent-oracle-harness.md`** (H13) — a new reference encoding the gold-standard four-part independent-oracle module: (a) a from-first-principles formula block (the published spec, no library), (b) an independent-library oracle block (a library that does not share the implementation's core), (c) a seeded `mulberry32` PRNG for reproducible fuzz, and (d) a second-algorithm differential helper — with a worked color/WCAG example transcribed from the A/B reference build. It is read at authoring time and is never a runtime gate. Linked from `test-authoring-guidance.md`, `evidence-channels.md`, `agents/ac-evaluator.md` (+ `ac-evaluator-hi.md`), `agents/test-writer.md`, and `agents/implementer.md`. +- **Multi-oracle mutual validation (H1)** — Gate 7 in `skills/create-ticket/references/ac-quality-criteria.md` gains a depth-gated multi-oracle clause: at the `thorough` / `exhaustive` `evidence_floor`, a standard-backed computational AC's expected value must come from ≥2 oracles independent of the implementation's core, mutually-validated (they agree within an explicit tolerance before either is trusted), with ≥1 derived from first principles. A single oracle still suffices at `standard`. Mirrored into the `EC-ORACLE` definition (`evidence-channels.md`), the producer rubric rule 1 (`test-authoring-guidance.md`) + both producer agents (`implementer`, `test-writer`), the verifier (`ac-evaluator` `## Oracle Independence` point 1 + the V3 lens), and the planner step-8 / ticket-evaluator Gate-7-row authoring guidance. +- **Committed seeded fuzz un-deferred (H2)** — `test-authoring-guidance.md` rule 7 is promoted from "encouraged" to a depth-gated **MUST**: at `thorough` / `exhaustive` a computational AC must ship a committed, fixed-seed property-fuzz loop (reproducible PRNG, tier-scaled case count) over the input distribution, not only deterministic grids. Mirrored into the `EC-PROPERTY` definition, both producer agents, the verifier (`## Oracle Independence` point 5 + the V2/V3 lenses), and the planner/ticket-evaluator Gate-8 authoring guidance. (Closes the v8.2.1 seeded-fuzz deferral, now justified by the 2026-06-02 dogfood.) +- **Algorithm-vs-algorithm differential (H3)** — `EC-DIFFERENTIAL` is re-specced (in `evidence-channels.md`, Gate 7 of `ac-quality-criteria.md`, `test-authoring-guidance.md` rule 3, `agents/ac-evaluator.md` + the V2 lens, and `ac-evaluator-orchestration.md`): when a second INDEPENDENT algorithm for the same contract exists (e.g. CSS-MINDE vs chroma-clamping gamut mapping), the verification compares algorithm-vs-algorithm within tolerance — a membership / invariant check alone is necessary-not-sufficient because a wrong result can still be in-range. +- **`### Standard-backed computational evidence floor` subsection** in `skills/impl/references/verification-depth.md` — the single authoritative statement tying H1/H2/H3 to the `thorough` / `exhaustive` floor, with the fail-open degradation rule, placed without disturbing the pinned effects-ladder cells. +- **Cat EV CT-EV-10..14** in `tests/test-skill-contracts.sh` — H1 multi-oracle symmetry guard (`mutually-validated` across the full 9-file author→verify surface — canonical gate + taxonomy + producer rubric + both producers + planner author + ticket-evaluator grader + verifier + floor doc — plus `first-principles` at the two ends; CT-EV-10), H2 committed-seeded-fuzz symmetry guard (`fixed-seed` across 8 files; CT-EV-11), H3 algorithm-vs-algorithm symmetry guard (`algorithm-vs-algorithm` across 9 files; CT-EV-12), H13 harness file-exists + reference-wired (CT-EV-13), and the H12 criticality cue (`shared-core`; CT-EV-14). Every token is HEAD=0 (a `git stash` of the change flips the assert to FAIL). The symmetry guards include the `planner` author and `ticket-evaluator` grader cells so a future silent revert of either authoring surface cannot ship green. + +### Changed + +- **M5 criticality cue set widened (H12)** — the `## Criticality floor` cue list in `verification-depth.md` adds color-science cues (color-space / gamut / OKLab / luminance / chroma conversion) and a domain-independent **shared-core input-boundary** trigger: a computational AC that reads or must hold an invariant across a parser / validation / constant (e.g. an epsilon / range / gamut guard) shared with sibling tools floors `criticality=critical` even when its surface domain is otherwise routine. This is the exact wrong-but-self-consistent shared-core defect class the floor targets; the trigger only RAISES the tier, never lowers it. + +### Verification + +- `bash tests/test-skill-contracts.sh` 797/797, `bash tests/test-path-consistency.sh` 142/142; full sweep 34/34 suites. +- New **CT-EV-10..14** all pass; each new grep token (`mutually-validated`, `first-principles`, `fixed-seed`, `algorithm-vs-algorithm`, `independent-oracle-harness.md`, `shared-core`) is HEAD=0 — a `git stash` flips the corresponding assert to FAIL, `git stash pop` restores PASS. Every v8.3.0 contract is intact: Gate 7's `## Gate 7: Oracle Independence` section (CT-AR-1/4/8), the `## Verification Lens (high-assurance handoff)` / `multi-verifier` DEPTH-8 anchors, the CT-EV-1..9 + CT-EV-MODEL-1..4 family, the `parse-accepted` / `sibling-guard` / `outputSchema` CT-AR-12/13/14 tokens, the `Gates 1-8` carriers, and the `ac-evaluator` ↔ `ac-evaluator-hi` byte-identical-body invariant (every body edit was mirrored). The feature is additive and fail-open: a routine S/M-conservative ticket resolves `evidence_floor=EC-STATIC+natural` (no multi-oracle, no fuzz mandate, no differential), each `constraints.: off` makes its mechanism a no-op, and the requirements degrade to the natural channel + a Caveat where no published spec / second oracle / second algorithm exists. + ## [8.3.0] — 2026-06-02 **TL;DR.** First of three additive, fail-open minor releases bringing the autonomous Generator-Evaluator harness toward parity with a human-directed max-effort session, after an A/B on a color-math MCP server lost on correctness/safety axes. This release ships the shared scaffolding (**Phase 0**) plus the first two measures. **Phase 0**: a new canonical **Evidence-Channel Taxonomy** (`skills/impl/references/evidence-channels.md`) — five evidence channels `EC-ORACLE` / `EC-DIFFERENTIAL` / `EC-PROPERTY` / `EC-RUNTIME` / `EC-STATIC`, five reserved red-team attack classes `RT-FUZZ` / `RT-ABUSE` / `RT-MALFORMED` / `RT-EXHAUST` / `RT-CONCURRENCY` (M2, v8.5.0), and the irreversibility-axis cue list (M5) — cited by ID, never paraphrased; and a `/impl` Step 3a resolved struct `{depth_tier, criticality, evidence_floor, evaluator_model, redteam_budget, domain_set}` that for a routine S/M-conservative ticket resolves to today's values (a byte-identical no-op). **M5 (effort/model allocation by criticality)**: a single `criticality = blast_radius(Size) × irreversibility` scalar, a new **irreversibility axis** (an AC verifying writes / network / money / destructive / external-system side-effects floors `criticality=critical` even at Size S), and an evaluator-model bump (sonnet→opus at `critical`/`exhaustive`) realized via the byte-identical sibling agent `agents/ac-evaluator-hi.md` (the Agent JSONSchema rejects a per-spawn `model:` override). **M1 (evidence-channel independence)**: a new **Gate 8 "Independent Evidence"** generalizing Gate 7's oracle requirement to every *behavioral* AC (Gate 7 stays a literal, intact section as the strongest `EC-ORACLE` sub-case), the three multi-verifier lenses re-specced from attitude-diverse to **evidence-mode-diverse**, and an `evidence_floor` ladder (standard = the AC's natural channel; thorough = +1 independent channel; exhaustive = ≥2). Default `auto` keeps a routine ticket byte-identical; the per-brief kill switches `constraints.independent_evidence: off`, `constraints.irreversibility_floor: off`, and the master `constraints.verification_depth: off` each restore prior behaviour. Red-team budget is recorded into the struct but has no consumer until M2 (v8.5.0). diff --git a/agents/ac-evaluator-hi.md b/agents/ac-evaluator-hi.md index 32c56d9..3e75969 100644 --- a/agents/ac-evaluator-hi.md +++ b/agents/ac-evaluator-hi.md @@ -206,11 +206,11 @@ A **computational AC** is one whose PASS/FAIL hinges on a COMPUTED numeric or al For every computational AC in scope, a green project test suite is **necessary but NOT sufficient**. You MUST independently establish the expected value and compare it against the implementation's RAW output: -1. **Independent oracle**: compute at least one expected value from an oracle that does NOT share the implementation's core — a third-party reference library (e.g. `colorjs.io` cross-checking a culori-based engine), a published formula / standard you apply from first principles, or a hand-computed truth table with a cited source. The AC body or its Implementation Notes (per Gate 7) names the oracle; use it (a runtime oracle Skill, if one was bound, would also appear in `## Bound capabilities (per AC)`). NEVER take the implementation's own output (directly, via an alias, or by re-reading a field the code already rounded) as the expected value — that is the oracle-circularity defect this gate exists to catch. +1. **Independent oracle**: compute at least one expected value from an oracle that does NOT share the implementation's core — a third-party reference library (e.g. `colorjs.io` cross-checking a culori-based engine), a published formula / standard you apply from first principles, or a hand-computed truth table with a cited source. The AC body or its Implementation Notes (per Gate 7) names the oracle; use it (a runtime oracle Skill, if one was bound, would also appear in `## Bound capabilities (per AC)`). NEVER take the implementation's own output (directly, via an alias, or by re-reading a field the code already rounded) as the expected value — that is the oracle-circularity defect this gate exists to catch. When your spawn prompt's `Evidence floor:` is `+1-independent` or `>=2-independent` (the `thorough` / `exhaustive` tiers) AND the AC is a standard-backed computational AC, require **two or more mutually-validated oracles** with **at least one derived from first principles** (the spec formula, hand-implemented, no library) and confirm they agree within an explicit tolerance before trusting either; FAIL a `thorough` / `exhaustive` standard-backed computational AC whose only independent evidence is a single library oracle. Build the second / first-principles oracle yourself under `.simple-workflow/scratch/` per the carve-out (shape: `skills/impl/references/independent-oracle-harness.md`). Where the domain has no published spec or no second independent oracle, the single-oracle path stands — record a Caveat (PASS-WITH-CAVEATS), never FAIL for an oracle that does not exist. 2. **Raw, pre-rounding comparison**: compare the implementation's raw output (before display rounding / formatting) against the oracle value with an explicit tolerance (e.g. `|raw − oracle| ≤ 1e-6`). If the project's tests assert only on a display-rounded value, or re-threshold a field the code itself rounds (e.g. asserting `result.ratio >= target` on the code's 2-decimal `ratio`), treat the AC as NOT verified by those tests and FAIL it with feedback to compare the raw value against an independent oracle. 3. **Probe permitted**: write a throwaway oracle probe under the gitignored `.simple-workflow/scratch/` directory (per the oracle-probe carve-out above) when a one-off computation is the fastest way to derive the expected value. Discard it after the round; never import-and-rubber-stamp the implementation. Invoke a JS/TS probe via `node .simple-workflow/scratch/probe.mjs` or `npx -y tsx .simple-workflow/scratch/probe.ts`, and a Python probe via `python3 .simple-workflow/scratch/probe.py` (these runtimes are granted in this agent's `tools:` allowlist for scratch probes only). A published-formula or hand-computed-truth-table oracle needs no execution at all — prefer it when the ecosystem's standalone runtime is unavailable. 4. **No-oracle degradation**: when the domain genuinely has no independent oracle (novel business logic), verify via raw-value assertions against hand-computed constants AND property / invariant coverage (monotonicity, symmetry, idempotence, round-trip, containment) AND adversarial / non-finite / out-of-range inputs. Reflect any residual uncertainty in the Caveats field (PASS-WITH-CAVEATS) rather than silently trusting a self-confirming test. -5. **Adversarial coverage (externally-fed computational ACs)**: when the computed value comes from a function that takes external / untrusted input, the AC's tests MUST also exercise adversarial / non-finite / out-of-range inputs (`NaN`, `Infinity`, empty, malformed, out-of-range / out-of-gamut). FAIL a computational AC on such a function that ships zero adversarial coverage, with a feedback note — this is what catches DoS hangs and contract-violating outputs on bad input, not merely wrong values on good input. The coverage MUST include at least one **parse-accepted-then-overflows** vector (a value the parser ACCEPTS that yields a non-finite / out-of-range intermediate, e.g. `oklch(0.5 1e400 30)` → Infinity chroma), not just parse-rejected `NaN` / `Infinity` keyword tokens. You SHOULD independently probe one such vector through the tool under a TIME-BOUNDED watchdog — spawn a child process that calls the tool and SIGKILL it after a few seconds (a hang ⇒ FAIL), using the `.simple-workflow/scratch/` carve-out — and FAIL the AC if the tool hangs or returns a non-error success carrying null / NaN channels. Also confirm the validation guard is present across ALL sibling tools that accept the same input class — probe at least one sibling beyond the AC's primary tool; a guard in one tool but not its siblings is a FAIL. +5. **Adversarial coverage (externally-fed computational ACs)**: when the computed value comes from a function that takes external / untrusted input, the AC's tests MUST also exercise adversarial / non-finite / out-of-range inputs (`NaN`, `Infinity`, empty, malformed, out-of-range / out-of-gamut). FAIL a computational AC on such a function that ships zero adversarial coverage, with a feedback note — this is what catches DoS hangs and contract-violating outputs on bad input, not merely wrong values on good input. The coverage MUST include at least one **parse-accepted-then-overflows** vector (a value the parser ACCEPTS that yields a non-finite / out-of-range intermediate, e.g. `oklch(0.5 1e400 30)` → Infinity chroma), not just parse-rejected `NaN` / `Infinity` keyword tokens. You SHOULD independently probe one such vector through the tool under a TIME-BOUNDED watchdog — spawn a child process that calls the tool and SIGKILL it after a few seconds (a hang ⇒ FAIL), using the `.simple-workflow/scratch/` carve-out — and FAIL the AC if the tool hangs or returns a non-error success carrying null / NaN channels. Also confirm the validation guard is present across ALL sibling tools that accept the same input class — probe at least one sibling beyond the AC's primary tool; a guard in one tool but not its siblings is a FAIL. At the `+1-independent` / `>=2-independent` evidence floor (thorough / exhaustive), additionally confirm a **committed, fixed-seed** property-fuzz loop exists in the project's test files (reproducible PRNG, asserting invariants / oracle agreement across the input distribution, not only a hand-picked grid); FAIL a thorough / exhaustive computational AC whose only coverage is a handful of fixed fixtures with no committed seeded sweep. Where the ecosystem has no PRNG idiom this degrades to a Caveat, never a FAIL. 6. **Pre-Gate-7 / legacy degradation**: when the ticket predates Gate 7 (names no oracle and declares no fallback) OR your spawn prompt carries `Oracle verification: off`, do NOT hard-FAIL a computational AC solely for missing oracle independence — verify it by the pre-v8.2.0 path (project tests + code inspection + whatever property / adversarial coverage is present) and record a one-line Caveat that oracle independence was not verifiable from the ticket (PASS-WITH-CAVEATS), mirroring the pre-Gate-6 capability fallback below. A freshly authored or modified circular test still FAILs via the always-on R4 static rule regardless of this degradation. This requirement is **independent of the verification-depth tier** — it applies in single-verifier (`standard`) mode as well as the partition and multi-verifier (`exhaustive`) branches. The orchestrator resolves `constraints.oracle_verification` at `/impl` Step 3a and inlines it into your spawn prompt as the field `Oracle verification: {auto|off}` — read it from the prompt (like the `## Bound capabilities (per AC)` handoff); do NOT read it from disk, and when the field is absent (older orchestrator or a manual run) default to `auto` (active). When it is `off`, verify computational ACs by the pre-v8.2.0 path (project tests + code inspection) and note it in Caveats (see point 6). The R4 oracle-circularity rule in `skills/impl/references/tautological-assertion-rules.md` is the static counterpart that flags the circular test pattern in the diff; this section is the semantic, runtime counterpart. @@ -220,7 +220,7 @@ This requirement is **independent of the verification-depth tier** — it applie Gate 8 (`skills/create-ticket/references/ac-quality-criteria.md`) generalizes the oracle requirement to EVERY behavioral AC — an AC whose PASS/FAIL hinges on observable runtime behaviour (a returned value, emitted output, status code, rendered surface, wire payload, thrown error, side effect), not a structural fact. For every behavioral AC in scope you MUST establish PASS via at least one evidence channel independent of the implementation's own internals (defined in `skills/impl/references/evidence-channels.md`): - **EC-ORACLE** — an oracle-derived expected value (the `## Oracle Independence (computational ACs)` path above; the strongest sub-case, mandatory for computational ACs). -- **EC-DIFFERENTIAL** — cross-check against a separate reference implementation of the same contract. +- **EC-DIFFERENTIAL** — cross-check against a separate reference implementation of the same contract; strongest as **algorithm-vs-algorithm** (a second, INDEPENDENT algorithm for the same contract, e.g. CSS-MINDE vs chroma-clamping) compared within tolerance — at thorough / exhaustive prefer this where a second algorithm exists, since a membership check is necessary-not-sufficient. - **EC-PROPERTY** — invariants over a seeded input distribution (monotonicity, symmetry, idempotence, round-trip, containment). - **EC-RUNTIME** — black-box observation through the real public / protocol boundary (the real CLI, the real MCP `Client` over a transport, the exported API, a rendered DOM), never internal handlers. - **EC-STATIC** — file-grep / signature / exit-code; the natural channel for a STRUCTURAL AC, but NOT sufficient evidence for a behavioral AC on its own. @@ -337,7 +337,11 @@ attitude — gather evidence through your assigned channel: drive a seeded random sweep (fixed seed → reproducible) and assert the invariants the output must hold — monotonicity, symmetry, idempotence, round-trip, range/gamut containment (EC-PROPERTY). FAIL an AC whose tests - assert only fixed points the code itself could have produced. + assert only fixed points the code itself could have produced. At thorough / + exhaustive, when a second independent ALGORITHM for the same contract exists, + compare algorithm-vs-algorithm within tolerance (membership is + necessary-not-sufficient), and require a committed, fixed-seed property-fuzz + loop across the distribution. - **`3/3 oracle-or-fuzz/EC-ORACLE`** — for any computational AC, independently derive >=1 expected value from an oracle that does NOT share the implementation's core and compare against the RAW pre-rounding output @@ -345,7 +349,10 @@ attitude — gather evidence through your assigned channel: at least one parse-accepted-then-overflows vector (e.g. `oklch(0.5 1e400 30)`) under a time-bounded watchdog; FAIL on a hang or a non-error success carrying null / NaN fields. The scratch carve-out under - `.simple-workflow/scratch/` is permitted for the oracle probe. + `.simple-workflow/scratch/` is permitted for the oracle probe. At thorough / + exhaustive derive two mutually-validated oracles (>=1 first-principles), + trusting a value only when they agree within tolerance, backed by a committed, + fixed-seed seeded fuzz; degrade to one oracle + a Caveat where none exists. Report severity as usual — a [CRITICAL] issue from a single lens is NOT voted away by the merge, so do not soften a genuine security / data-loss / diff --git a/agents/ac-evaluator.md b/agents/ac-evaluator.md index 545f72d..6548506 100644 --- a/agents/ac-evaluator.md +++ b/agents/ac-evaluator.md @@ -206,11 +206,11 @@ A **computational AC** is one whose PASS/FAIL hinges on a COMPUTED numeric or al For every computational AC in scope, a green project test suite is **necessary but NOT sufficient**. You MUST independently establish the expected value and compare it against the implementation's RAW output: -1. **Independent oracle**: compute at least one expected value from an oracle that does NOT share the implementation's core — a third-party reference library (e.g. `colorjs.io` cross-checking a culori-based engine), a published formula / standard you apply from first principles, or a hand-computed truth table with a cited source. The AC body or its Implementation Notes (per Gate 7) names the oracle; use it (a runtime oracle Skill, if one was bound, would also appear in `## Bound capabilities (per AC)`). NEVER take the implementation's own output (directly, via an alias, or by re-reading a field the code already rounded) as the expected value — that is the oracle-circularity defect this gate exists to catch. +1. **Independent oracle**: compute at least one expected value from an oracle that does NOT share the implementation's core — a third-party reference library (e.g. `colorjs.io` cross-checking a culori-based engine), a published formula / standard you apply from first principles, or a hand-computed truth table with a cited source. The AC body or its Implementation Notes (per Gate 7) names the oracle; use it (a runtime oracle Skill, if one was bound, would also appear in `## Bound capabilities (per AC)`). NEVER take the implementation's own output (directly, via an alias, or by re-reading a field the code already rounded) as the expected value — that is the oracle-circularity defect this gate exists to catch. When your spawn prompt's `Evidence floor:` is `+1-independent` or `>=2-independent` (the `thorough` / `exhaustive` tiers) AND the AC is a standard-backed computational AC, require **two or more mutually-validated oracles** with **at least one derived from first principles** (the spec formula, hand-implemented, no library) and confirm they agree within an explicit tolerance before trusting either; FAIL a `thorough` / `exhaustive` standard-backed computational AC whose only independent evidence is a single library oracle. Build the second / first-principles oracle yourself under `.simple-workflow/scratch/` per the carve-out (shape: `skills/impl/references/independent-oracle-harness.md`). Where the domain has no published spec or no second independent oracle, the single-oracle path stands — record a Caveat (PASS-WITH-CAVEATS), never FAIL for an oracle that does not exist. 2. **Raw, pre-rounding comparison**: compare the implementation's raw output (before display rounding / formatting) against the oracle value with an explicit tolerance (e.g. `|raw − oracle| ≤ 1e-6`). If the project's tests assert only on a display-rounded value, or re-threshold a field the code itself rounds (e.g. asserting `result.ratio >= target` on the code's 2-decimal `ratio`), treat the AC as NOT verified by those tests and FAIL it with feedback to compare the raw value against an independent oracle. 3. **Probe permitted**: write a throwaway oracle probe under the gitignored `.simple-workflow/scratch/` directory (per the oracle-probe carve-out above) when a one-off computation is the fastest way to derive the expected value. Discard it after the round; never import-and-rubber-stamp the implementation. Invoke a JS/TS probe via `node .simple-workflow/scratch/probe.mjs` or `npx -y tsx .simple-workflow/scratch/probe.ts`, and a Python probe via `python3 .simple-workflow/scratch/probe.py` (these runtimes are granted in this agent's `tools:` allowlist for scratch probes only). A published-formula or hand-computed-truth-table oracle needs no execution at all — prefer it when the ecosystem's standalone runtime is unavailable. 4. **No-oracle degradation**: when the domain genuinely has no independent oracle (novel business logic), verify via raw-value assertions against hand-computed constants AND property / invariant coverage (monotonicity, symmetry, idempotence, round-trip, containment) AND adversarial / non-finite / out-of-range inputs. Reflect any residual uncertainty in the Caveats field (PASS-WITH-CAVEATS) rather than silently trusting a self-confirming test. -5. **Adversarial coverage (externally-fed computational ACs)**: when the computed value comes from a function that takes external / untrusted input, the AC's tests MUST also exercise adversarial / non-finite / out-of-range inputs (`NaN`, `Infinity`, empty, malformed, out-of-range / out-of-gamut). FAIL a computational AC on such a function that ships zero adversarial coverage, with a feedback note — this is what catches DoS hangs and contract-violating outputs on bad input, not merely wrong values on good input. The coverage MUST include at least one **parse-accepted-then-overflows** vector (a value the parser ACCEPTS that yields a non-finite / out-of-range intermediate, e.g. `oklch(0.5 1e400 30)` → Infinity chroma), not just parse-rejected `NaN` / `Infinity` keyword tokens. You SHOULD independently probe one such vector through the tool under a TIME-BOUNDED watchdog — spawn a child process that calls the tool and SIGKILL it after a few seconds (a hang ⇒ FAIL), using the `.simple-workflow/scratch/` carve-out — and FAIL the AC if the tool hangs or returns a non-error success carrying null / NaN channels. Also confirm the validation guard is present across ALL sibling tools that accept the same input class — probe at least one sibling beyond the AC's primary tool; a guard in one tool but not its siblings is a FAIL. +5. **Adversarial coverage (externally-fed computational ACs)**: when the computed value comes from a function that takes external / untrusted input, the AC's tests MUST also exercise adversarial / non-finite / out-of-range inputs (`NaN`, `Infinity`, empty, malformed, out-of-range / out-of-gamut). FAIL a computational AC on such a function that ships zero adversarial coverage, with a feedback note — this is what catches DoS hangs and contract-violating outputs on bad input, not merely wrong values on good input. The coverage MUST include at least one **parse-accepted-then-overflows** vector (a value the parser ACCEPTS that yields a non-finite / out-of-range intermediate, e.g. `oklch(0.5 1e400 30)` → Infinity chroma), not just parse-rejected `NaN` / `Infinity` keyword tokens. You SHOULD independently probe one such vector through the tool under a TIME-BOUNDED watchdog — spawn a child process that calls the tool and SIGKILL it after a few seconds (a hang ⇒ FAIL), using the `.simple-workflow/scratch/` carve-out — and FAIL the AC if the tool hangs or returns a non-error success carrying null / NaN channels. Also confirm the validation guard is present across ALL sibling tools that accept the same input class — probe at least one sibling beyond the AC's primary tool; a guard in one tool but not its siblings is a FAIL. At the `+1-independent` / `>=2-independent` evidence floor (thorough / exhaustive), additionally confirm a **committed, fixed-seed** property-fuzz loop exists in the project's test files (reproducible PRNG, asserting invariants / oracle agreement across the input distribution, not only a hand-picked grid); FAIL a thorough / exhaustive computational AC whose only coverage is a handful of fixed fixtures with no committed seeded sweep. Where the ecosystem has no PRNG idiom this degrades to a Caveat, never a FAIL. 6. **Pre-Gate-7 / legacy degradation**: when the ticket predates Gate 7 (names no oracle and declares no fallback) OR your spawn prompt carries `Oracle verification: off`, do NOT hard-FAIL a computational AC solely for missing oracle independence — verify it by the pre-v8.2.0 path (project tests + code inspection + whatever property / adversarial coverage is present) and record a one-line Caveat that oracle independence was not verifiable from the ticket (PASS-WITH-CAVEATS), mirroring the pre-Gate-6 capability fallback below. A freshly authored or modified circular test still FAILs via the always-on R4 static rule regardless of this degradation. This requirement is **independent of the verification-depth tier** — it applies in single-verifier (`standard`) mode as well as the partition and multi-verifier (`exhaustive`) branches. The orchestrator resolves `constraints.oracle_verification` at `/impl` Step 3a and inlines it into your spawn prompt as the field `Oracle verification: {auto|off}` — read it from the prompt (like the `## Bound capabilities (per AC)` handoff); do NOT read it from disk, and when the field is absent (older orchestrator or a manual run) default to `auto` (active). When it is `off`, verify computational ACs by the pre-v8.2.0 path (project tests + code inspection) and note it in Caveats (see point 6). The R4 oracle-circularity rule in `skills/impl/references/tautological-assertion-rules.md` is the static counterpart that flags the circular test pattern in the diff; this section is the semantic, runtime counterpart. @@ -220,7 +220,7 @@ This requirement is **independent of the verification-depth tier** — it applie Gate 8 (`skills/create-ticket/references/ac-quality-criteria.md`) generalizes the oracle requirement to EVERY behavioral AC — an AC whose PASS/FAIL hinges on observable runtime behaviour (a returned value, emitted output, status code, rendered surface, wire payload, thrown error, side effect), not a structural fact. For every behavioral AC in scope you MUST establish PASS via at least one evidence channel independent of the implementation's own internals (defined in `skills/impl/references/evidence-channels.md`): - **EC-ORACLE** — an oracle-derived expected value (the `## Oracle Independence (computational ACs)` path above; the strongest sub-case, mandatory for computational ACs). -- **EC-DIFFERENTIAL** — cross-check against a separate reference implementation of the same contract. +- **EC-DIFFERENTIAL** — cross-check against a separate reference implementation of the same contract; strongest as **algorithm-vs-algorithm** (a second, INDEPENDENT algorithm for the same contract, e.g. CSS-MINDE vs chroma-clamping) compared within tolerance — at thorough / exhaustive prefer this where a second algorithm exists, since a membership check is necessary-not-sufficient. - **EC-PROPERTY** — invariants over a seeded input distribution (monotonicity, symmetry, idempotence, round-trip, containment). - **EC-RUNTIME** — black-box observation through the real public / protocol boundary (the real CLI, the real MCP `Client` over a transport, the exported API, a rendered DOM), never internal handlers. - **EC-STATIC** — file-grep / signature / exit-code; the natural channel for a STRUCTURAL AC, but NOT sufficient evidence for a behavioral AC on its own. @@ -337,7 +337,11 @@ attitude — gather evidence through your assigned channel: drive a seeded random sweep (fixed seed → reproducible) and assert the invariants the output must hold — monotonicity, symmetry, idempotence, round-trip, range/gamut containment (EC-PROPERTY). FAIL an AC whose tests - assert only fixed points the code itself could have produced. + assert only fixed points the code itself could have produced. At thorough / + exhaustive, when a second independent ALGORITHM for the same contract exists, + compare algorithm-vs-algorithm within tolerance (membership is + necessary-not-sufficient), and require a committed, fixed-seed property-fuzz + loop across the distribution. - **`3/3 oracle-or-fuzz/EC-ORACLE`** — for any computational AC, independently derive >=1 expected value from an oracle that does NOT share the implementation's core and compare against the RAW pre-rounding output @@ -345,7 +349,10 @@ attitude — gather evidence through your assigned channel: at least one parse-accepted-then-overflows vector (e.g. `oklch(0.5 1e400 30)`) under a time-bounded watchdog; FAIL on a hang or a non-error success carrying null / NaN fields. The scratch carve-out under - `.simple-workflow/scratch/` is permitted for the oracle probe. + `.simple-workflow/scratch/` is permitted for the oracle probe. At thorough / + exhaustive derive two mutually-validated oracles (>=1 first-principles), + trusting a value only when they agree within tolerance, backed by a committed, + fixed-seed seeded fuzz; degrade to one oracle + a Caveat where none exists. Report severity as usual — a [CRITICAL] issue from a single lens is NOT voted away by the merge, so do not soften a genuine security / data-loss / diff --git a/agents/implementer.md b/agents/implementer.md index 275d6cd..a3e78a9 100644 --- a/agents/implementer.md +++ b/agents/implementer.md @@ -26,7 +26,7 @@ After implementing, run the project's lint command (as defined in CLAUDE.md or p After lint passes, run the project's test command (as defined in CLAUDE.md or project conventions). If tests fail, fix and re-run (max 3 attempts). -When writing these tests, you MUST follow the positive rubric in `skills/impl/references/test-authoring-guidance.md` (resolve the path relative to the repository root). For any **computational AC** (one whose PASS/FAIL hinges on a computed numeric/algorithmic value — a ratio, threshold, hash, conversion, round-trip, metric), assert against an **independent oracle** (a reference library / published formula / hand-computed truth table — never the code's own output) on the **raw, pre-rounding** value with an explicit tolerance; add property / invariant tests (monotonicity, symmetry, idempotence, round-trip, containment); and cover adversarial / non-finite / out-of-range inputs by default — including at least one **parse-accepted-then-overflows** vector (a value the parser ACCEPTS that yields a non-finite / out-of-range intermediate, e.g. `oklch(0.5 1e400 30)` → Infinity chroma), not only parse-rejected `NaN` / `Infinity` tokens; and, when the function shares an input parser with sibling tools, ensure the input-validation guard lives in the SHARED boundary OR is replicated AND adversarially tested in EVERY sibling tool (see rule 4 of the guidance). A test that re-measures with the implementation's own rounded value is self-confirming and is rejected by tautological rule R4 (`skills/impl/references/tautological-assertion-rules.md`). +When writing these tests, you MUST follow the positive rubric in `skills/impl/references/test-authoring-guidance.md` (resolve the path relative to the repository root). For any **computational AC** (one whose PASS/FAIL hinges on a computed numeric/algorithmic value — a ratio, threshold, hash, conversion, round-trip, metric), assert against an **independent oracle** (a reference library / published formula / hand-computed truth table — never the code's own output) on the **raw, pre-rounding** value with an explicit tolerance; add property / invariant tests (monotonicity, symmetry, idempotence, round-trip, containment); and cover adversarial / non-finite / out-of-range inputs by default — including at least one **parse-accepted-then-overflows** vector (a value the parser ACCEPTS that yields a non-finite / out-of-range intermediate, e.g. `oklch(0.5 1e400 30)` → Infinity chroma), not only parse-rejected `NaN` / `Infinity` tokens; and, when the function shares an input parser with sibling tools, ensure the input-validation guard lives in the SHARED boundary OR is replicated AND adversarially tested in EVERY sibling tool (see rule 4 of the guidance). A test that re-measures with the implementation's own rounded value is self-confirming and is rejected by tautological rule R4 (`skills/impl/references/tautological-assertion-rules.md`). At the `thorough` / `exhaustive` depth tier, for a standard-backed computational target author **two or more mutually-validated independent oracles** (at least one first-principles, no library) and trust a value only when they agree within tolerance — see `skills/impl/references/independent-oracle-harness.md`; ship a **committed, fixed-seed** property-fuzz loop (reproducible PRNG, tier-scaled case count) over the input distribution, not only fixed fixtures; and where a second INDEPENDENT ALGORITHM for the same contract exists, add an algorithm-vs-algorithm differential within tolerance (membership is necessary-not-sufficient). Do NOT include self-assessment, subjective comments, or quality judgments in your return value. Report only factual information. diff --git a/agents/planner.md b/agents/planner.md index bc0b82d..33c1d2a 100644 --- a/agents/planner.md +++ b/agents/planner.md @@ -52,7 +52,7 @@ This audit makes the "planner forgot to consider `ui-ux-pro-max` / `mcp__context After the Gate 6.5 probe completeness cross-check passes, run this oracle-independence cross-check before emitting. The canonical definition lives in `skills/create-ticket/references/ac-quality-criteria.md` Gate 7; this step is the planner-side enforcement procedure. 8. **Gate 7 oracle independence cross-check**. First read `constraints.oracle_verification` from `{ticket-dir}/autopilot-policy.yaml` if present (absent file / field / unknown value → `auto`); when it is `off`, Gate 7 is `n/a` ticket-wide — skip this step. (During create-ticket Phase 3 the per-ticket `autopilot-policy.yaml` is typically not yet written — it is propagated at Step W-8 — so this read normally resolves to `auto`/active; a brief-level `off` takes effect downstream at `/impl` and `ac-evaluator`. Authoring-time Gate 7 is therefore unconditional unless a policy file already exists in the ticket dir.) Otherwise apply the **computational AC** classifier to every AC drafted in step 2: an AC is computational when its PASS/FAIL hinges on a COMPUTED numeric/algorithmic value (contrast / luminance / color ratio, rounding or precision threshold, hash / checksum / collision rate, financial or unit conversion, parser / serializer round-trip, distance / similarity / statistical metric, any "within X of Y" or "≥/≤ a numeric target"). For each computational AC: - a. Verify the AC body OR its Implementation Notes names an **oracle independent of the implementation** (a third-party reference library that does not share the implementation's core, a published formula / standard, or a hand-computed truth table with a cited source) AND specifies a **raw, pre-rounding** comparison with an explicit tolerance. Re-thresholding a field the implementation itself rounds (e.g. asserting `result.ratio >= target` on the code's 2-decimal `ratio`) is a Gate 7 FAIL. + a. Verify the AC body OR its Implementation Notes names an **oracle independent of the implementation** (a third-party reference library that does not share the implementation's core, a published formula / standard, or a hand-computed truth table with a cited source) AND specifies a **raw, pre-rounding** comparison with an explicit tolerance. Re-thresholding a field the implementation itself rounds (e.g. asserting `result.ratio >= target` on the code's 2-decimal `ratio`) is a Gate 7 FAIL. For a standard-backed computational AC expected to resolve to the `thorough` / `exhaustive` evidence_floor (a critical-domain / shared-core / high-stakes computational AC — see `skills/impl/references/verification-depth.md` criticality floor), prefer naming **two mutually-validated oracles** (one first-principles) so the downstream verifier can satisfy the multi-oracle floor; where only one independent oracle exists, say so. Where a second independent ALGORITHM for the contract exists, prefer naming it so the verifier can run an algorithm-vs-algorithm differential (membership alone is necessary-not-sufficient). This is authoring guidance, not a force-FAIL — Gate 7 still passes a single named oracle at draft time. b. If no independent oracle exists for the domain, declare the **no-oracle fallback** explicitly in the AC / Implementation Notes: raw-value assertions with tolerance against hand-computed constants AND property / invariant coverage (monotonicity, symmetry, idempotence, round-trip, containment) AND adversarial / non-finite / out-of-range inputs. State which path applies. c. If a computational AC names neither an oracle (+ raw-value tolerance) nor the no-oracle fallback, EITHER revise it to do so, OR rewrite it as a static AC (file-grep / counter / exit-code). Emitting a computational AC with no oracle, no fallback, and no static rewrite is a Gate 7 FAIL. Purely structural ACs are `n/a` for Gate 7. d. For a computational AC whose value comes from a function taking external / untrusted input, also require adversarial / non-finite / out-of-range coverage (`NaN`, `Infinity`, empty, malformed, out-of-range); a computational AC on such a function with no adversarial coverage is a Gate 7 FAIL — this catches DoS hangs and bad-input contract violations, not just wrong values on good input. The required coverage MUST include a parse-ACCEPTED-then-overflows vector (e.g. `oklch(0.5 1e400 30)`), not only parse-rejected `NaN` / `Infinity` tokens. If the function shares an input parser with sibling tools, require the guard in the SHARED boundary OR in every sibling (not just one). @@ -63,7 +63,7 @@ After the Gate 7 oracle-independence cross-check passes, run this independent-ev 9. **Gate 8 independent-evidence cross-check**. First read `constraints.independent_evidence` from `{ticket-dir}/autopilot-policy.yaml` if present (absent file / field / unknown value → `auto`); when it is `off`, Gate 8 is `n/a` ticket-wide — skip this step. Otherwise apply the **behavioral AC** classifier to every AC drafted in step 2: an AC is behavioral when its PASS/FAIL hinges on observable runtime behaviour (a returned value, emitted output, status code, rendered surface, wire payload, thrown error, side effect) rather than a structural fact (file exists, symbol exported, flag parsed). For each behavioral AC: a. Verify the AC body OR its Implementation Notes names at least one evidence channel from {EC-ORACLE, EC-DIFFERENTIAL, EC-PROPERTY, EC-RUNTIME} (defined in `skills/impl/references/evidence-channels.md`) that is independent of the implementation's own internals. The AC's natural channel counts: a black-box CLI / exported-API / MCP-Client assertion is EC-RUNTIME; a parser round-trip or invariant sweep is EC-PROPERTY; a reference-library cross-check is EC-ORACLE or EC-DIFFERENTIAL. No EXTRA channel beyond the natural one is required at draft time — the resolved `evidence_floor` at `/impl` Step 3a (thorough = +1 independent channel, exhaustive = >=2) is enforced verification-side, not authoring-side. - b. If a behavioral AC's only stated evidence is the implementation re-asserting itself (an EC-STATIC grep on a behavioral claim, or a test re-reading a value the code produced), EITHER revise it to name an independent channel, OR rewrite it as a structural AC (file-grep / counter / exit-code, i.e. EC-STATIC on a genuinely structural fact). A behavioral AC naming no independent channel and with no structural rewrite is a Gate 8 FAIL. A computational AC satisfies Gate 8 via its Gate 7 oracle (EC-ORACLE) — do NOT double-require a channel name when Gate 7 is already satisfied. Purely structural ACs are `n/a` for Gate 8. + b. If a behavioral AC's only stated evidence is the implementation re-asserting itself (an EC-STATIC grep on a behavioral claim, or a test re-reading a value the code produced), EITHER revise it to name an independent channel, OR rewrite it as a structural AC (file-grep / counter / exit-code, i.e. EC-STATIC on a genuinely structural fact). A behavioral AC naming no independent channel and with no structural rewrite is a Gate 8 FAIL. A computational AC satisfies Gate 8 via its Gate 7 oracle (EC-ORACLE) — do NOT double-require a channel name when Gate 7 is already satisfied. Purely structural ACs are `n/a` for Gate 8. For a computational / algorithmic AC expected at the `thorough` / `exhaustive` evidence_floor, prefer authoring it so the test ships a **committed, fixed-seed** property-fuzz loop (EC-PROPERTY) over the input distribution — name it in Implementation Notes; this is authoring guidance, not a draft-time force-FAIL. Every Pre-emit self-audit above (numeric cross-check, Gate 6 binding, Gate 6.5 probe completeness, Gate 7 oracle independence, Gate 8 independent evidence) applies on the 1st-draft emit and on every retry re-emit. diff --git a/agents/test-writer.md b/agents/test-writer.md index 8ca5abe..68fc9d2 100644 --- a/agents/test-writer.md +++ b/agents/test-writer.md @@ -17,7 +17,7 @@ You are a test engineer. Write and run tests following existing project patterns ## Test Design Rubric -Follow `skills/impl/references/test-authoring-guidance.md` (resolve the path relative to the repository root) as the positive rubric for test authoring. For any **computational** target (a computed numeric/algorithmic value — ratio, threshold, hash, conversion, round-trip, metric), assert against an **independent oracle** (reference library / published formula / hand-computed truth table — never the code's own output) on the **raw, pre-rounding** value with an explicit tolerance; add property / invariant tests (monotonicity, symmetry, idempotence, round-trip, containment); and cover adversarial / non-finite / out-of-range inputs by default — including at least one **parse-accepted-then-overflows** vector (a value the parser ACCEPTS that yields a non-finite / out-of-range intermediate, e.g. `oklch(0.5 1e400 30)` → Infinity chroma), not only parse-rejected `NaN` / `Infinity` tokens; and, when the function shares an input parser with sibling tools, ensure the input-validation guard lives in the SHARED boundary OR is replicated AND adversarially tested in EVERY sibling tool (see rule 4 of the guidance). A test that re-measures with the implementation's own rounded value is self-confirming and is rejected by tautological rule R4 (`skills/impl/references/tautological-assertion-rules.md`). +Follow `skills/impl/references/test-authoring-guidance.md` (resolve the path relative to the repository root) as the positive rubric for test authoring. For any **computational** target (a computed numeric/algorithmic value — ratio, threshold, hash, conversion, round-trip, metric), assert against an **independent oracle** (reference library / published formula / hand-computed truth table — never the code's own output) on the **raw, pre-rounding** value with an explicit tolerance; add property / invariant tests (monotonicity, symmetry, idempotence, round-trip, containment); and cover adversarial / non-finite / out-of-range inputs by default — including at least one **parse-accepted-then-overflows** vector (a value the parser ACCEPTS that yields a non-finite / out-of-range intermediate, e.g. `oklch(0.5 1e400 30)` → Infinity chroma), not only parse-rejected `NaN` / `Infinity` tokens; and, when the function shares an input parser with sibling tools, ensure the input-validation guard lives in the SHARED boundary OR is replicated AND adversarially tested in EVERY sibling tool (see rule 4 of the guidance). A test that re-measures with the implementation's own rounded value is self-confirming and is rejected by tautological rule R4 (`skills/impl/references/tautological-assertion-rules.md`). At the `thorough` / `exhaustive` depth tier, for a standard-backed computational target author **two or more mutually-validated independent oracles** (at least one first-principles, no library) and trust a value only when they agree within tolerance — see `skills/impl/references/independent-oracle-harness.md`; ship a **committed, fixed-seed** property-fuzz loop (reproducible PRNG, tier-scaled case count) over the input distribution, not only fixed fixtures; and where a second INDEPENDENT ALGORITHM for the same contract exists, add an algorithm-vs-algorithm differential within tolerance (membership is necessary-not-sufficient). ## Context Conservation Protocol diff --git a/agents/ticket-evaluator.md b/agents/ticket-evaluator.md index acd062b..d89dd82 100644 --- a/agents/ticket-evaluator.md +++ b/agents/ticket-evaluator.md @@ -56,8 +56,8 @@ Return format: - [x] Size Fit: description - [x] Capability Mapping: description (Gate 6 — applies only when at least one AC is runtime/visual per the canonical classifier; mark `n/a` when no AC triggers the classifier) - [x] Probe Completeness: description (Gate 6.5 — verifies every entry in the planner's `Available user skills:` / `Available MCP servers:` probe is classified Bound / Advisory / Skipped in the ticket; mark `n/a` when both probes report `(none)` or when the ticket pre-dates Gate 6.5) -- [x] Oracle Independence: description (Gate 7 — applies only when at least one AC is computational (a computed numeric/algorithmic value) per the canonical classifier; verifies each computational AC names an oracle independent of the implementation + a raw-value tolerance, OR declares the no-oracle fallback, AND (for a computational AC on a function taking external input) requires adversarial / non-finite / out-of-range coverage — including at least one parse-accepted-then-overflows vector (e.g. `oklch(0.5 1e400 30)`), not only parse-rejected `NaN` / `Infinity` tokens — plus the sibling-guard requirement (a shared input-validation guard required across every sibling tool sharing the input boundary, not just one); mark `n/a` when no AC is computational, when the ticket sets `constraints.oracle_verification: off`, or when the ticket pre-dates Gate 7) -- [x] Independent Evidence: description (Gate 8 — applies only when at least one AC is behavioral (PASS/FAIL hinges on observable runtime behaviour) per the canonical classifier; verifies each behavioral AC names at least one evidence channel independent of the implementation — EC-ORACLE / EC-DIFFERENTIAL / EC-PROPERTY / EC-RUNTIME per `skills/impl/references/evidence-channels.md` — OR is rewritten as a structural AC (EC-STATIC). A computational AC satisfies Gate 8 via its Gate 7 oracle. The AC's natural channel counts (a black-box CLI / API assertion is EC-RUNTIME) — do NOT FAIL a behavioral AC whose natural evidence is already independent. Mark `n/a` when every AC is purely structural, when the ticket sets `constraints.independent_evidence: off`, or when the ticket pre-dates Gate 8) +- [x] Oracle Independence: description (Gate 7 — applies only when at least one AC is computational (a computed numeric/algorithmic value) per the canonical classifier; verifies each computational AC names an oracle independent of the implementation + a raw-value tolerance, OR declares the no-oracle fallback, AND (for a computational AC on a function taking external input) requires adversarial / non-finite / out-of-range coverage — including at least one parse-accepted-then-overflows vector (e.g. `oklch(0.5 1e400 30)`), not only parse-rejected `NaN` / `Infinity` tokens — plus the sibling-guard requirement (a shared input-validation guard required across every sibling tool sharing the input boundary, not just one); and, for a standard-backed computational AC expected at the `thorough` / `exhaustive` evidence_floor, prefer two mutually-validated oracles with at least one first-principles (single oracle suffices at `standard`; absence of a second oracle is a Caveat, not a FAIL); where a second independent algorithm exists, prefer an algorithm-vs-algorithm differential (membership is necessary-not-sufficient; its absence is a Caveat, not a FAIL); mark `n/a` when no AC is computational, when the ticket sets `constraints.oracle_verification: off`, or when the ticket pre-dates Gate 7) +- [x] Independent Evidence: description (Gate 8 — applies only when at least one AC is behavioral (PASS/FAIL hinges on observable runtime behaviour) per the canonical classifier; verifies each behavioral AC names at least one evidence channel independent of the implementation — EC-ORACLE / EC-DIFFERENTIAL / EC-PROPERTY / EC-RUNTIME per `skills/impl/references/evidence-channels.md` — OR is rewritten as a structural AC (EC-STATIC). A computational AC satisfies Gate 8 via its Gate 7 oracle. The AC's natural channel counts (a black-box CLI / API assertion is EC-RUNTIME) — do NOT FAIL a behavioral AC whose natural evidence is already independent. For a computational / algorithmic AC expected at the `thorough` / `exhaustive` evidence_floor, prefer a committed, fixed-seed property-fuzz loop over the input distribution (EC-PROPERTY); its absence is a Caveat, not a FAIL. Mark `n/a` when every AC is purely structural, when the ticket sets `constraints.independent_evidence: off`, or when the ticket pre-dates Gate 8) **Issues**: [gate] description (one per line) **Feedback**: [specific, actionable improvements for the planner] ``` diff --git a/skills/create-ticket/references/ac-quality-criteria.md b/skills/create-ticket/references/ac-quality-criteria.md index 2b4a0f5..90acbc4 100644 --- a/skills/create-ticket/references/ac-quality-criteria.md +++ b/skills/create-ticket/references/ac-quality-criteria.md @@ -194,6 +194,8 @@ A probe entry that appears in none of the three buckets is a Gate 6.5 FAIL — t **Raw-value requirement**: the comparison MUST be made on the implementation's **raw, pre-rounding / pre-formatting** output against the oracle value with an **explicit tolerance** (e.g. `|raw − oracle| ≤ 1e-6`). Asserting on a display-rounded value, or re-thresholding the rounded field the code returns, is a Gate 7 FAIL — display rounding can mask a sub-threshold miss. +**Multi-oracle requirement (depth-gated, standard-backed computational ACs)**: when the resolved `evidence_floor` is `thorough` or `exhaustive` (see [`../../impl/references/verification-depth.md`](../../impl/references/verification-depth.md) effects ladder) AND the AC sits in a domain with a **published spec** (color / WCAG, crypto, dates, units, money rounding, spec parsers, accessibility ratios), the expected value MUST be derived from **two or more oracles independent of the implementation's core, mutually-validated** — they MUST agree within an explicit tolerance BEFORE either is trusted — with **at least one derived from first principles** (the published spec formula, hand-implemented, no library at all). A single independent oracle suffices only at the `standard` floor. Two agreeing channels, one of them first-principles, is what makes the evidence independent of BOTH the implementation AND any single reference library (a lone library oracle silently inherits that library's conventions). The canonical shape of such an oracle module is [`../../impl/references/independent-oracle-harness.md`](../../impl/references/independent-oracle-harness.md). **Degradation (never a block)**: where the domain has no published spec or no second independent oracle, the single-oracle path stands and the AC / Implementation Notes record a one-line Caveat that multi-oracle mutual validation was not available — Gate 7 never force-FAILs an AC for the absence of a second oracle that does not exist. + **No-oracle degradation**: when no independent oracle exists for the domain (novel business logic with no reference), Gate 7 is satisfied by an explicit fallback in the AC / Implementation Notes: (a) raw-value assertions with tolerance against hand-computed constants, AND (b) property / invariant coverage (monotonicity, symmetry, idempotence, round-trip, containment), AND (c) adversarial / non-finite / out-of-range inputs. The planner MUST state which path applies. Gate 7 never force-FAILs a ticket where no oracle is possible, but it DOES FAIL a computational AC that names neither an oracle nor the fallback. See [`../../impl/references/test-authoring-guidance.md`](../../impl/references/test-authoring-guidance.md) for the authoring rubric and `agents/ac-evaluator.md` `## Oracle Independence (computational ACs)` for verifier-side enforcement. **Binding rule** (per computational AC): each computational AC MUST either (a) name an independent oracle + a raw-value tolerance, OR (b) declare the no-oracle fallback above, OR (c) be rewritten as a static AC. A computational AC with none of these is a Gate 7 FAIL. @@ -202,6 +204,8 @@ A probe entry that appears in none of the three buckets is a Gate 6.5 FAIL — t **Shared / sibling-guard requirement** (computational ACs sharing an input boundary): when a computational AC's function shares an input parser / validation boundary with sibling tools (e.g. several MCP tools that each parse the same CSS-color string), the input-validation guard (finiteness, range, gamut) MUST either (a) live in the SHARED boundary so every sibling inherits it, OR (b) be replicated AND adversarially tested in EVERY sibling tool that accepts that input class. A guard wired into one consumer but absent from its analogous siblings is a Gate 7 FAIL — this is the `## Modifications` sibling-artifact rule (`CLAUDE.md`) enforced at the AC level. (Dogfood evidence: a finite-components guard added to the solver but not to the analogous `gamut_map` / `parse_color` tools left a live DoS hang reachable through the unguarded siblings.) +**Algorithm-vs-algorithm differential (depth-gated, EC-DIFFERENTIAL)**: when a computational AC's contract admits a **second, independent ALGORITHM** for the same result (e.g. gamut mapping by CSS-MINDE vs chroma-clamping; two independent sorts; two serializers), at the `thorough` / `exhaustive` evidence_floor the verification SHOULD cross-check the implementation against that second algorithm **within an explicit tolerance**, not merely assert the output satisfies a membership / invariant test. A membership check (`inGamut`, "result is sorted") is **necessary-not-sufficient** — a wrong-but-in-range result passes it. Where no second independent algorithm exists for the contract, the membership / property coverage stands and the AC records a one-line Caveat — this is never a force-FAIL. + **Kill switch**: when `{ticket-dir}/autopilot-policy.yaml` sets `constraints.oracle_verification: off`, Gate 7 is graded `n/a` ticket-wide (restores pre-v8.2.0 authoring behaviour). Absent field / absent policy / unknown value → `auto` (Gate 7 active — fail-safe). **Evaluator note**: Gate 7 activates only after this section ships. Tickets drafted before Gate 7 are pre-Gate-7 and are graded `n/a` for Gate 7 (PASS verdicts on those tickets do not imply Gate 7 conformance). diff --git a/skills/impl/references/ac-evaluator-orchestration.md b/skills/impl/references/ac-evaluator-orchestration.md index 3b1dda5..3c39f4d 100644 --- a/skills/impl/references/ac-evaluator-orchestration.md +++ b/skills/impl/references/ac-evaluator-orchestration.md @@ -109,8 +109,8 @@ on the raw value + a parse-accepted-overflow vector). Substitute `{i}` ∈ {1,2,3} and persist to `eval-round-{n}-v{i}.md`: - **V1 — runtime / black-box lens (EC-RUNTIME)**: `--- lens: 1/3 runtime/EC-RUNTIME --- Gather evidence through the REAL public / protocol boundary only: drive the actual CLI, the actual MCP Client over a transport, or the exported public API — never internal handlers reached by reflection or imports the real consumer cannot use. A green project suite is necessary but NOT sufficient: confirm the AC's behaviour is observable at the public boundary, and FAIL any AC whose only evidence is a white-box test that bypasses the schema / serialization / transport layer where real consumers fail.` -- **V2 — differential / property lens (EC-DIFFERENTIAL or EC-PROPERTY)**: `--- lens: 2/3 differential-or-property/EC-DIFFERENTIAL,EC-PROPERTY --- Establish evidence INDEPENDENT of the implementation's own output. When a reference implementation of the same contract exists, cross-check the implementation against it (EC-DIFFERENTIAL). Otherwise drive a seeded random sweep over the input space (fixed seed → reproducible) and assert the invariants the output must hold — monotonicity, symmetry, idempotence, round-trip, range/gamut containment (EC-PROPERTY). FAIL an AC whose tests assert only a handful of fixed points the code itself could have produced, with no reference cross-check and no property coverage across the distribution.` -- **V3 — independent-oracle / targeted-fuzz lens (EC-ORACLE + adversarial)**: `--- lens: 3/3 oracle-or-fuzz/EC-ORACLE --- For any computational AC, independently derive at least one expected value from an oracle that does NOT share the implementation's core (a third-party reference library, a published formula applied from first principles, or a cited hand-computed constant) and compare against the implementation's RAW, pre-rounding output with an explicit tolerance — this is the Gate 7 oracle probe, applied with full force. Additionally fuzz at least one parse-accepted-then-overflows vector (a value the parser ACCEPTS that yields a non-finite / out-of-range intermediate, e.g. `oklch(0.5 1e400 30)`) through the tool under a time-bounded watchdog; FAIL if it hangs or returns a non-error success carrying null / NaN fields. A scratch oracle probe under .simple-workflow/scratch/ is permitted.` +- **V2 — differential / property lens (EC-DIFFERENTIAL or EC-PROPERTY)**: `--- lens: 2/3 differential-or-property/EC-DIFFERENTIAL,EC-PROPERTY --- Establish evidence INDEPENDENT of the implementation's own output. When a reference implementation of the same contract exists, cross-check the implementation against it (EC-DIFFERENTIAL). Otherwise drive a seeded random sweep over the input space (fixed seed → reproducible) and assert the invariants the output must hold — monotonicity, symmetry, idempotence, round-trip, range/gamut containment (EC-PROPERTY). FAIL an AC whose tests assert only a handful of fixed points the code itself could have produced, with no reference cross-check and no property coverage across the distribution. At thorough / exhaustive, where a SECOND independent ALGORITHM for the same contract exists, compare algorithm-vs-algorithm within tolerance (membership / invariant satisfaction is necessary-not-sufficient — a wrong result can still be in-range); and require a committed, fixed-seed property-fuzz loop across the distribution.` +- **V3 — independent-oracle / targeted-fuzz lens (EC-ORACLE + adversarial)**: `--- lens: 3/3 oracle-or-fuzz/EC-ORACLE --- For any computational AC, independently derive at least one expected value from an oracle that does NOT share the implementation's core (a third-party reference library, a published formula applied from first principles, or a cited hand-computed constant) and compare against the implementation's RAW, pre-rounding output with an explicit tolerance — this is the Gate 7 oracle probe, applied with full force. Additionally fuzz at least one parse-accepted-then-overflows vector (a value the parser ACCEPTS that yields a non-finite / out-of-range intermediate, e.g. `oklch(0.5 1e400 30)`) through the tool under a time-bounded watchdog; FAIL if it hangs or returns a non-error success carrying null / NaN fields. A scratch oracle probe under .simple-workflow/scratch/ is permitted. At thorough / exhaustive for a standard-backed computational AC, derive two mutually-validated oracles (>=1 first-principles) and trust a value only when they agree within tolerance, backed by a committed, fixed-seed seeded fuzz; degrade to one oracle + a Caveat where no second oracle / second algorithm exists.` The lens header mirrors the `--- partition: /2 ---` convention so the agent recognises its role; the `ac-evaluator` body documents the three diff --git a/skills/impl/references/evidence-channels.md b/skills/impl/references/evidence-channels.md index a4274c3..f998c1e 100644 --- a/skills/impl/references/evidence-channels.md +++ b/skills/impl/references/evidence-channels.md @@ -40,14 +40,34 @@ but is NOT independent evidence for a *behavioral* AC on its own. implementation's core: a third-party reference library, a published formula applied from first principles, or a hand-computed truth table. *Independent because* the expected value exists without running the SUT. This is the strongest channel and the - one Gate 7 (oracle independence for computational ACs) requires. + one Gate 7 (oracle independence for computational ACs) requires. At the + `thorough` / `exhaustive` evidence_floor a standard-backed computational AC + must use **two or more mutually-validated** EC-ORACLE channels with **at least + one first-principles** (spec formula, no library) — they agree within + tolerance before either is trusted; a single oracle suffices only at + `standard`, and the requirement degrades to one oracle + a Caveat where no + second independent oracle exists. See + [`independent-oracle-harness.md`](independent-oracle-harness.md) for the + canonical shape. - **EC-DIFFERENTIAL** — the implementation's output cross-checked against a SEPARATE reference implementation of the same contract. *Independent because* a second - implementation would have to share the exact same bug to agree. + implementation would have to share the exact same bug to agree. Strongest as + **algorithm-vs-algorithm**: when a second, INDEPENDENT ALGORITHM for the same + contract exists (e.g. CSS-MINDE vs chroma-clamping gamut mapping), cross-check + the two within an explicit tolerance — a membership / invariant test alone + (`inGamut`, "is sorted") is **necessary-not-sufficient** because a wrong + result can still satisfy it. At `thorough` / `exhaustive` use the + second-algorithm differential where one is identifiable; degrade to membership + / property coverage + a Caveat where no second algorithm exists. - **EC-PROPERTY** — invariants the output must satisfy across a seeded input distribution (monotonicity, symmetry, idempotence, round-trip, range/gamut containment), independent of any single expected value. *Independent because* the - invariant is a law the correct answer obeys, not a value the SUT produced. + invariant is a law the correct answer obeys, not a value the SUT produced. At + the `thorough` / `exhaustive` evidence_floor this seeded distribution MUST be a + **committed, fixed-seed** loop (reproducible PRNG, tier-scaled case count), not + a hand-picked grid — see + [`independent-oracle-harness.md`](independent-oracle-harness.md); it degrades + to deterministic coverage + a Caveat where no PRNG idiom exists. - **EC-RUNTIME** — black-box observation through the real public / protocol boundary: the real CLI, the real MCP `Client` over a transport, the exported public API, a rendered DOM — never internal handlers reached by reflection or by imports a real diff --git a/skills/impl/references/independent-oracle-harness.md b/skills/impl/references/independent-oracle-harness.md new file mode 100644 index 0000000..e99920c --- /dev/null +++ b/skills/impl/references/independent-oracle-harness.md @@ -0,0 +1,96 @@ +# Independent-oracle harness (gold-standard shape) + +Binding parties: the `implementer` and `test-writer` agents (authoring), and the +`ac-evaluator` (when building a scratch oracle probe). This file gives the +**copyable SHAPE** of an independent-oracle module for a standard-backed +computational target, so the multi-oracle (H1), committed-seeded-fuzz (H2), and +algorithm-vs-algorithm differential (H3) requirements at the `thorough` / +`exhaustive` evidence_floor can be satisfied by transcribing a known-good +structure instead of re-deriving it. See +[`test-authoring-guidance.md`](test-authoring-guidance.md) rules 1 / 3 / 7 and +[`evidence-channels.md`](evidence-channels.md) (EC-ORACLE / EC-DIFFERENTIAL / +EC-PROPERTY) for the obligations this shape satisfies. + +A trustworthy oracle module has FOUR parts. An expected value is trusted only +when the first-principles formula and the independent library agree within +tolerance — neither alone is enough, because a single library silently shares +its own conventions. + +## (a) First-principles block — the spec formula, NO library + +Implement the published spec directly, with no dependency on the +implementation's core or on any third-party library. This is the channel that +is independent of EVERY library's conventions. + +```js +// WCAG 2.x relative luminance / contrast, hand-implemented from the spec. +function srgbToLinear(c) { return c <= 0.04045 ? c / 12.92 : ((c + 0.055) / 1.055) ** 2.4; } +function relativeLuminanceFP(r8, g8, b8) { + const r = srgbToLinear(r8 / 255), g = srgbToLinear(g8 / 255), b = srgbToLinear(b8 / 255); + return 0.2126 * r + 0.7152 * g + 0.0722 * b; +} +function contrastRatioFP(a, b) { + const la = relativeLuminanceFP(...a), lb = relativeLuminanceFP(...b); + const hi = Math.max(la, lb), lo = Math.min(la, lb); + return (hi + 0.05) / (lo + 0.05); +} +``` + +## (b) Independent-library block — a DIFFERENT library than the SUT + +Wrap a reference library that does NOT share the implementation's core (the SUT +uses culori → the oracle uses colorjs.io). This catches spec-implementation +mistakes the first-principles block might also make. + +```js +import Color from 'colorjs.io'; +const cjsContrast = (a, b) => new Color(a).contrast(new Color(b), 'WCAG21'); +``` + +## (c) Seeded PRNG — reproducible fuzz + +A deterministic PRNG seeded by a literal makes a fuzz run reproducible AND +exploratory. Commit the loop; do not hand-pick a grid. + +```js +function makeRng(seed) { // mulberry32 + let a = seed >>> 0; + return () => { + a |= 0; a = (a + 0x6d2b79f5) | 0; + let t = Math.imul(a ^ (a >>> 15), 1 | a); + t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t; + return ((t ^ (t >>> 14)) >>> 0) / 4294967296; + }; +} +// const rng = makeRng(101); for (let i = 0; i < 1000; i++) { /* assert invariant + oracle agree */ } +``` + +## (d) Second-algorithm differential helper — algorithm-vs-algorithm + +Where a SECOND independent ALGORITHM for the same contract exists, expose it so +the test can compare algorithm-vs-algorithm within tolerance — a membership +check (`inGamut`) is necessary-not-sufficient, because a wrong result can still +be in-range. + +```js +// colorjs.io CSS (MINDE) gamut mapping — a DIFFERENT algorithm than culori's clampChroma. +const cjsToGamutCssHex = (input) => + new Color(input).toGamut({ space: 'srgb', method: 'css' }).to('srgb').toString({ format: 'hex' }); +const cjsDeltaEOK = (a, b) => new Color(a).deltaE(new Color(b), { method: 'ok' }); +// expect(cjsDeltaEOK(engineResult, cjsToGamutCssHex(input))).toBeLessThan(0.1); +``` + +## Mutual validation before trust (H1) + +```js +// The two independent oracles must agree BEFORE either is used as truth: +expect(Math.abs(contrastRatioFP(rgbA, rgbB) - cjsContrast(a, b))).toBeLessThan(2e-3); +// ...only then compare the implementation's RAW (pre-rounding) value against them. +``` + +## Degradation (fail-open) + +Where the domain has no published spec, no second independent library, or no +second algorithm, use whichever channels DO exist and record a one-line Caveat — +the harness never fabricates an oracle, and a missing oracle is never a FAIL. +This shape is read at authoring time; it is not a runtime gate. diff --git a/skills/impl/references/test-authoring-guidance.md b/skills/impl/references/test-authoring-guidance.md index 75c0e54..bdfe39a 100644 --- a/skills/impl/references/test-authoring-guidance.md +++ b/skills/impl/references/test-authoring-guidance.md @@ -31,6 +31,14 @@ exit code) needs only a direct assertion. table with a cited source. NEVER take the implementation's own output — directly, via an alias, or by re-reading a field the code already rounded — as the expected value. That circularity is rejected by tautological rule R4. + At the `thorough` / `exhaustive` depth tier, for a standard-backed + computational target use **two or more mutually-validated independent + oracles** with **at least one first-principles** (the spec formula, + hand-implemented, no library) and trust a value only when they agree within + an explicit tolerance — one library oracle silently shares that library's + conventions. Copy the shape from + [`independent-oracle-harness.md`](independent-oracle-harness.md). Where no + second independent oracle exists, the single-oracle path stands (note it). 2. **Raw before rounded, with explicit tolerance.** Assert on the implementation's RAW, pre-rounding / pre-formatting value against the oracle @@ -45,6 +53,13 @@ exit code) needs only a direct assertion. round-trip (`decode(encode(x)) == x`), gamut / range containment, and conservation. A property holds across a distribution; a point test holds at one point. + When a second INDEPENDENT ALGORITHM for the same contract exists (e.g. + CSS-MINDE vs chroma-clamping gamut mapping, two independent sorts), at the + `thorough` / `exhaustive` depth tier add an **algorithm-vs-algorithm** + differential within an explicit tolerance — a membership / containment check + alone is necessary-not-sufficient, because a wrong result can still be + in-range. Degrade to property coverage + a note where no second algorithm + exists. 4. **Adversarial / non-finite / out-of-range inputs by default.** For any function taking external or untrusted input, include empty, `NaN`, @@ -92,6 +107,18 @@ exit code) needs only a direct assertion. of hard-coded fixtures, and cross-check each generated case against the oracle / invariant. A dozen fixed inputs is a dozen points; a seeded sweep is a distribution. + At the `thorough` / `exhaustive` depth tier this is a **MUST**, not an + encouragement: a computational / algorithmic AC MUST ship a **committed, + fixed-seed** property-fuzz loop (the seed is hard-coded so the run is + reproducible; a `mulberry32`-style PRNG seeded by a literal is the canonical + shape — see + [`independent-oracle-harness.md`](independent-oracle-harness.md)) over the + input distribution, asserting the invariants / oracle agreement across a + **tier-scaled minimum** number of cases (rule of thumb: `thorough` >= a few + hundred per invariant family, `exhaustive` >= ~1000). The loop must be in the + committed test file, not an ad-hoc external probe. Where the ecosystem has no + PRNG idiom, or the AC is not computational, this degrades to the existing + deterministic-grid coverage + a one-line note — never a block. ## No-oracle fallback diff --git a/skills/impl/references/verification-depth.md b/skills/impl/references/verification-depth.md index 022c738..7e49ebe 100644 --- a/skills/impl/references/verification-depth.md +++ b/skills/impl/references/verification-depth.md @@ -92,10 +92,17 @@ Size × `risk_tolerance`, so the `/audit` skeptical third-pass is forced even on an `S` / `conservative` ticket. The critical-domain determination is a model-judgment read of the AC text (there is no deterministic ticket field); keyword cues that should trigger it: WCAG / contrast / a11y / focus-order -(accessibility); auth / crypto / token / signature / input-validation +/ color-space / gamut / OKLab / luminance / chroma conversion (accessibility & +color-science); auth / crypto / token / signature / input-validation (security); currency / decimal / rounding / money (money); checksum / hash / dedup / referential-integrity (data-integrity); RFC / ISO / spec-conformance -(standard-compliance). +(standard-compliance); and any **computational AC over a shared-core input +boundary** — a parser / validation / constant (e.g. an epsilon / range / gamut +guard) shared with sibling tools, where an inconsistency in one sibling is the +exact wrong-but-self-consistent defect class the floor targets. The shared-core +trigger is independent of the named domains above: a computational AC that reads +or must hold an invariant across a shared input boundary floors `critical` even +when its surface domain is otherwise routine. The floor only RAISES the tier (`standard` → `thorough`); it never lowers a tier the matrix already resolved higher (`exhaustive` stays `exhaustive`). It is @@ -163,6 +170,32 @@ evaluator_model, redteam_budget, domain_set}` that `/impl` Step 3a materialises [`ac-evaluator-orchestration.md`](ac-evaluator-orchestration.md) `## High-assurance multi-verifier branch` for the lenses and merge. +### Standard-backed computational evidence floor (Wave A: multi-oracle / seeded-fuzz / algorithm-differential) + +At the `thorough` and `exhaustive` floors, the independent channels mandated by +the `evidence_floor` column above are SHARPENED for a **standard-backed +computational AC** (one in a domain with a published spec / a second independent +oracle / a second algorithm — color/WCAG, crypto, dates, units, money rounding, +spec parsers, accessibility ratios): + +- **multi-oracle (H1)** — the EC-ORACLE evidence MUST be **two or more + mutually-validated oracles** with **at least one derived from first + principles** (the spec formula, no library), trusted only when they agree + within an explicit tolerance. A single library oracle suffices only at + `standard`. +- **committed seeded fuzz (H2)** — a **committed, fixed-seed** property-fuzz + loop (EC-PROPERTY) over the input distribution, tier-scaled, not a hand-picked + grid. +- **algorithm-vs-algorithm (H3)** — where a second independent ALGORITHM for the + contract exists, an EC-DIFFERENTIAL cross-check within tolerance (membership is + necessary-not-sufficient). + +These RAISE the bar only for standard-backed computational ACs at +`thorough` / `exhaustive`; they **degrade to the single natural channel + a +Caveat — never a block** where no published spec / second oracle / second +algorithm exists, and the `standard` floor is unchanged. The canonical oracle +shape is [`independent-oracle-harness.md`](independent-oracle-harness.md). + ### Evaluator model + red-team budget (M5 / v8.3.0+) The resolver also fills two struct fields the floor RAISES: diff --git a/tests/test-skill-contracts.sh b/tests/test-skill-contracts.sh index 5254cbc..345eda3 100644 --- a/tests/test-skill-contracts.sh +++ b/tests/test-skill-contracts.sh @@ -9505,6 +9505,113 @@ assert_true \ "CT-EV-9 (gate-count carriers -> Gates 1-8, MF3/MF6): agent-spawn-prompts ($ev9_asp>=1) + ticket-evaluator L15 ($ev9_tev>=1) + create-ticket SKILL ($ev9_ctskill>=1)" \ "$ev9_result" +# ----------------------------------------------------------------------------- +# Wave A (v8.3.1): ultracode-superiority verification-assurance refinement. +# H1 multi-oracle / H2 committed seeded fuzz / H3 algorithm-vs-algorithm / +# H13 independent-oracle-harness.md / H12 criticality cue tighten. Every token +# below is HEAD=0 (verified net-new), so a revert flips the assert to FAIL. +# Fail-open is a runtime property (engages only at thorough/exhaustive for a +# standard-backed computational AC) and is proven by the §6 fail-open diff, not +# by a contract test. New file-path vars (collision-checked against L9379-9390): +TAG_EV="$REPO_DIR/skills/impl/references/test-authoring-guidance.md" +TW_EV="$REPO_DIR/agents/test-writer.md" +IMPLAGENT_EV="$REPO_DIR/agents/implementer.md" +HARNESS_EV="$REPO_DIR/skills/impl/references/independent-oracle-harness.md" + +# CT-EV-10 (H1 multi-oracle, symmetry guard): the >=2-mutually-validated-oracle +# (>=1 first-principles) obligation is wired across the FULL author->verify +# surface — canonical gate (ac-quality-criteria), taxonomy (evidence-channels), +# producer rubric (test-authoring-guidance + implementer + test-writer), +# authoring gate (planner step 8 + ticket-evaluator Gate-7 row), verifier +# (ac-evaluator), and the floor doc (verification-depth). 'mutually-validated' is +# the H1 signature token (pinned in all 9 binding-party files so a silent revert +# of ANY cell — incl. the planner author + ticket-evaluator grader cells the +# appendix symmetry matrix names — flips this to FAIL); 'first-principles' +# (hyphenated) is pinned at the two ends (authoring gate + verifier). +ev10_acqc=$(grep -ciF 'mutually-validated' "$ACQC_EV" || true) +ev10_ech=$(grep -ciF 'mutually-validated' "$ECH_EV" || true) +ev10_tag=$(grep -ciF 'mutually-validated' "$TAG_EV" || true) +ev10_acev=$(grep -ciF 'mutually-validated' "$ACEV_EV" || true) +ev10_vd=$(grep -ciF 'mutually-validated' "$VD_EV" || true) +ev10_impl=$(grep -ciF 'mutually-validated' "$IMPLAGENT_EV" || true) +ev10_tw=$(grep -ciF 'mutually-validated' "$TW_EV" || true) +ev10_planner=$(grep -ciF 'mutually-validated' "$PLANNER_EV" || true) +ev10_tev=$(grep -ciF 'mutually-validated' "$TEV_EV" || true) +ev10_fp_acqc=$(grep -ciF 'first-principles' "$ACQC_EV" || true) +ev10_fp_acev=$(grep -ciF 'first-principles' "$ACEV_EV" || true) +ev10_result="false" +if [ "$ev10_acqc" -ge 1 ] && [ "$ev10_ech" -ge 1 ] && [ "$ev10_tag" -ge 1 ] && [ "$ev10_acev" -ge 1 ] && [ "$ev10_vd" -ge 1 ] && [ "$ev10_impl" -ge 1 ] && [ "$ev10_tw" -ge 1 ] && [ "$ev10_planner" -ge 1 ] && [ "$ev10_tev" -ge 1 ] && [ "$ev10_fp_acqc" -ge 1 ] && [ "$ev10_fp_acev" -ge 1 ]; then ev10_result="true"; fi +assert_true \ + "CT-EV-10 (H1 multi-oracle wired): 'mutually-validated' in ac-quality-criteria/evidence-channels/test-authoring/ac-evaluator/verification-depth/implementer/test-writer/planner/ticket-evaluator ($ev10_acqc/$ev10_ech/$ev10_tag/$ev10_acev/$ev10_vd/$ev10_impl/$ev10_tw/$ev10_planner/$ev10_tev each>=1) + 'first-principles' in gate+verifier ($ev10_fp_acqc/$ev10_fp_acev each>=1)" \ + "$ev10_result" + +# CT-EV-11 (H2 committed seeded fuzz, symmetry guard): rule 7 promoted from +# "encouraged" to a depth-gated MUST + the committed fixed-seed PRNG obligation +# wired across producer rubric, taxonomy, verifier, and floor doc. 'fixed-seed' +# (hyphenated; the SPACED 'fixed seed' pre-existed) is the H2 signature token. A +# revert (rule 7 back to encouragement, or dropping the fixed-seed clause) flips it. +ev11_tag=$(grep -ciF 'fixed-seed' "$TAG_EV" || true) +ev11_ech=$(grep -ciF 'fixed-seed' "$ECH_EV" || true) +ev11_acev=$(grep -ciF 'fixed-seed' "$ACEV_EV" || true) +ev11_vd=$(grep -ciF 'fixed-seed' "$VD_EV" || true) +ev11_impl=$(grep -ciF 'fixed-seed' "$IMPLAGENT_EV" || true) +ev11_tw=$(grep -ciF 'fixed-seed' "$TW_EV" || true) +ev11_planner=$(grep -ciF 'fixed-seed' "$PLANNER_EV" || true) +ev11_tev=$(grep -ciF 'fixed-seed' "$TEV_EV" || true) +ev11_result="false" +if [ "$ev11_tag" -ge 1 ] && [ "$ev11_ech" -ge 1 ] && [ "$ev11_acev" -ge 1 ] && [ "$ev11_vd" -ge 1 ] && [ "$ev11_impl" -ge 1 ] && [ "$ev11_tw" -ge 1 ] && [ "$ev11_planner" -ge 1 ] && [ "$ev11_tev" -ge 1 ]; then ev11_result="true"; fi +assert_true \ + "CT-EV-11 (H2 committed seeded fuzz wired): 'fixed-seed' in test-authoring/evidence-channels/ac-evaluator/verification-depth/implementer/test-writer/planner/ticket-evaluator ($ev11_tag/$ev11_ech/$ev11_acev/$ev11_vd/$ev11_impl/$ev11_tw/$ev11_planner/$ev11_tev each>=1)" \ + "$ev11_result" + +# CT-EV-12 (H3 algorithm-vs-algorithm, symmetry guard): EC-DIFFERENTIAL re-spec'd +# to algorithm-vs-algorithm (membership is necessary-not-sufficient) across the +# canonical gate, taxonomy, producer rubric, verifier, the lens directives, and +# both producers. 'algorithm-vs-algorithm' is the H3 signature token (HEAD=0). +ev12_acqc=$(grep -ciF 'algorithm-vs-algorithm' "$ACQC_EV" || true) +ev12_ech=$(grep -ciF 'algorithm-vs-algorithm' "$ECH_EV" || true) +ev12_tag=$(grep -ciF 'algorithm-vs-algorithm' "$TAG_EV" || true) +ev12_acev=$(grep -ciF 'algorithm-vs-algorithm' "$ACEV_EV" || true) +ev12_orch=$(grep -ciF 'algorithm-vs-algorithm' "$ORCH_EV" || true) +ev12_impl=$(grep -ciF 'algorithm-vs-algorithm' "$IMPLAGENT_EV" || true) +ev12_tw=$(grep -ciF 'algorithm-vs-algorithm' "$TW_EV" || true) +ev12_planner=$(grep -ciF 'algorithm-vs-algorithm' "$PLANNER_EV" || true) +ev12_tev=$(grep -ciF 'algorithm-vs-algorithm' "$TEV_EV" || true) +ev12_result="false" +if [ "$ev12_acqc" -ge 1 ] && [ "$ev12_ech" -ge 1 ] && [ "$ev12_tag" -ge 1 ] && [ "$ev12_acev" -ge 1 ] && [ "$ev12_orch" -ge 1 ] && [ "$ev12_impl" -ge 1 ] && [ "$ev12_tw" -ge 1 ] && [ "$ev12_planner" -ge 1 ] && [ "$ev12_tev" -ge 1 ]; then ev12_result="true"; fi +assert_true \ + "CT-EV-12 (H3 algorithm-vs-algorithm wired): 'algorithm-vs-algorithm' in ac-quality-criteria/evidence-channels/test-authoring/ac-evaluator/orchestration/implementer/test-writer/planner/ticket-evaluator ($ev12_acqc/$ev12_ech/$ev12_tag/$ev12_acev/$ev12_orch/$ev12_impl/$ev12_tw/$ev12_planner/$ev12_tev each>=1)" \ + "$ev12_result" + +# CT-EV-13 (H13 independent-oracle-harness.md exists + reference-wired): the new +# gold-standard oracle harness file exists AND is linked from the producer rubric, +# taxonomy, verifier, and both producers (CT-AR-6 file-exists + reference-wired +# idiom). A revert deleting the file or any inbound link flips this to FAIL. +ev13_exists=0 +if [ -f "$HARNESS_EV" ]; then ev13_exists=1; fi +ev13_tag=$(grep -cF 'independent-oracle-harness.md' "$TAG_EV" || true) +ev13_ech=$(grep -cF 'independent-oracle-harness.md' "$ECH_EV" || true) +ev13_acev=$(grep -cF 'independent-oracle-harness.md' "$ACEV_EV" || true) +ev13_tw=$(grep -cF 'independent-oracle-harness.md' "$TW_EV" || true) +ev13_impl=$(grep -cF 'independent-oracle-harness.md' "$IMPLAGENT_EV" || true) +ev13_result="false" +if [ "$ev13_exists" -eq 1 ] && [ "$ev13_tag" -ge 1 ] && [ "$ev13_ech" -ge 1 ] && [ "$ev13_acev" -ge 1 ] && [ "$ev13_tw" -ge 1 ] && [ "$ev13_impl" -ge 1 ]; then ev13_result="true"; fi +assert_true \ + "CT-EV-13 (H13 harness file + wired): independent-oracle-harness.md exists ($ev13_exists=1) + linked from test-authoring/evidence-channels/ac-evaluator/test-writer/implementer ($ev13_tag/$ev13_ech/$ev13_acev/$ev13_tw/$ev13_impl each>=1)" \ + "$ev13_result" + +# CT-EV-14 (H12 criticality cue tighten): verification-depth.md criticality floor +# carries the new shared-core escalation trigger so a computational AC over a +# shared input boundary reliably escalates to thorough/exhaustive (the catch +# mechanism only fires if classification fires). 'shared-core' is HEAD=0 +# ('gamut'/'color-space' pre-exist, so they are not usable as the unique token). +ev14_vd=$(grep -cF 'shared-core' "$VD_EV" || true) +ev14_result="false" +if [ "$ev14_vd" -ge 1 ]; then ev14_result="true"; fi +assert_true \ + "CT-EV-14 (H12 criticality cue): verification-depth criticality floor carries 'shared-core' trigger ($ev14_vd>=1)" \ + "$ev14_result" + echo "" # =============================================================================