From e7949c02995085d298d77c017ef36fee7093d90a Mon Sep 17 00:00:00 2001 From: Konrad Heimel Date: Wed, 19 Aug 2026 05:52:07 +0200 Subject: [PATCH 1/4] :bug: fix(taskfile): -count=1 on the cache-blind dogfood-comparison and e2e gates MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `task dogfood-comparison` ran `go test ./examples/comparison/...` with no -count=1, so it could print `ok ... (cached)` and exit 0 on a tree where the same test genuinely fails. Mechanism: the Go test cache keys on the test BINARY's content ID. examples/comparison/validate_test.go is an EXTERNAL test package that references only exported constants from internal/compare (GateZeroMissedDestructive, ExitCodeForGate), so the comparison logic is unreachable from that binary and the linker strips it — the content ID does not move when internal/compare changes. The logic under test actually runs in a subprocess the test builds at runtime (`go build -o ... ./cmd/assent`), entirely outside the cache key. Corpus DATA edits do invalidate the cache (copyDir os.ReadFile's them, so testlog records them); production code changes do not. Measured before the fix: deleting `|| e == aggregate.EffectChallenge` from isStricterInterventionEffect left `task dogfood-comparison` green-and-cached (exit 0) while `go test -count=1` on the identical tree failed (exit status 6). After the fix the same mutated tree reddens the stage. `task e2e` gets the same treatment: its subject is a PREBUILT bin/assent driven against a live GitLab endpoint — neither the binary bytes nor the forge state is in the cache key. It is not part of `task check`, so this costs nothing. Left alone deliberately: `test` (whole-tree, in-process subjects, honest cache and a real time saver), `coverage` (in-process subjects; -coverprofile results do cache but the profile is reproduced faithfully) and `determinism` (-count=2 is uncacheable by construction). A -count=1 someone deletes later is the D-124 "gate invoked by nothing" species, so it is pinned in hack/examples/dogfood_wiring_test.sh — an existing `task check` stage that already checks Taskfile wiring, which keeps CHECK_STAGES at 19. The pin matches the `go test` COMMAND LINE, never a comment, and carries three controls: a comment-only fixture that must NOT satisfy it, a mutation that strips the flag from the command lines while leaving the explanatory comments intact, and the existing wiring mutation for `- task: dogfood-comparison`. Known residual, out of this lane's write paths: .github/workflows/verify.yaml line 102 runs the same uncounted `go test ./examples/comparison/...`, and actions/setup-go@v7 restores GOCACHE by default, so CI can serve a cached PASS across commits too. --- Taskfile.yml | 27 +++++++- hack/examples/dogfood_wiring_test.sh | 97 +++++++++++++++++++++++++++- 2 files changed, 121 insertions(+), 3 deletions(-) diff --git a/Taskfile.yml b/Taskfile.yml index 6c3ac41..eee0cb0 100644 --- a/Taskfile.yml +++ b/Taskfile.yml @@ -66,7 +66,22 @@ tasks: dogfood-comparison: desc: "Dogfood gate (PCS-S08): comparison corpus validates and runs green under assent compare --suite" cmds: - - go test ./examples/comparison/... + # -count=1 IS THE GATE. Without it this stage reports `ok … (cached)` and + # exits 0 on a tree where the same test genuinely fails. The Go test cache + # keys on the test BINARY's content ID, and examples/comparison is an + # EXTERNAL test package that references only exported constants from + # internal/compare (GateZeroMissedDestructive, ExitCodeForGate) — the + # comparison logic itself is unreachable from that binary and the linker + # strips it, so the content ID does not move when internal/compare + # changes. The real logic runs in a subprocess the test builds at runtime + # (`go build -o … ./cmd/assent`), entirely outside the cache key. Measured + # 2026-08-19: deleting `|| e == aggregate.EffectChallenge` from + # isStricterInterventionEffect left this stage green-and-cached while + # `-count=1` on the identical tree failed with exit status 6. Corpus DATA + # edits do invalidate (copyDir os.ReadFile's them, so testlog records + # them); production code changes do not. Pinned by + # hack/examples/dogfood_wiring_test.sh. + - go test -count=1 ./examples/comparison/... check: desc: "Gate: fmt + vet + lint + test + build (must pass before every commit)" @@ -192,7 +207,15 @@ tasks: e2e: desc: E2E tests against a real GitLab (build tag e2e) — see test/e2e/README.md cmds: - - go test -tags e2e ./test/e2e/... + # -count=1 for the same reason as dogfood-comparison: this suite's subject + # is a PREBUILT bin/assent (test/e2e/skeleton_test.go) driven against a + # live GitLab endpoint named by ASSENT_E2E_GITLAB. Neither the binary's + # bytes nor the forge's state is in the test cache key, so a cached PASS + # would assert nothing about the run the operator just asked for. Costs + # nothing: this task is never part of `task check` and the suite skips + # unless the endpoint env var is set. Pinned by + # hack/examples/dogfood_wiring_test.sh. + - go test -count=1 -tags e2e ./test/e2e/... docs: desc: Serve the docs site locally diff --git a/hack/examples/dogfood_wiring_test.sh b/hack/examples/dogfood_wiring_test.sh index b081b8e..24b50c0 100755 --- a/hack/examples/dogfood_wiring_test.sh +++ b/hack/examples/dogfood_wiring_test.sh @@ -7,6 +7,16 @@ # REQ-EX-S08-03: `task check` runs dogfood-examples (after build); deleting # that line from check: must redden this pin. # +# Section 5 extends the same discipline to a SECOND way a dogfood gate can be +# invoked-by-nothing: `task dogfood-comparison` runs `go test` over an EXTERNAL +# test package whose real subject is a binary the test builds at runtime, so the +# Go test cache key is blind to production changes and the stage can print +# `ok … (cached)` and exit 0 on a tree where the test genuinely fails (measured +# 2026-08-19). `-count=1` is what makes that stage a gate at all — so it is +# pinned here, with a mutation control that strips the flag from the COMMAND +# LINE while leaving the explanatory comment in place, which is exactly how a +# human would regress it. +# # Follows the hack/release/changelog_gate_test.sh / example_format_inventory_test.sh # discipline: every "is it wired" assertion is re-run against a mutated copy # with the wiring deleted, so the assertion is proven capable of failing @@ -131,4 +141,89 @@ echo "== 4. hack/dogfood-examples.sh exists and is executable ==" [[ -x "$ROOT/$SCRIPT" ]] || fail "$SCRIPT is not executable" echo "OK: $SCRIPT present and executable" -echo "PASS: dogfood wiring (REQ-EX-S08-02, REQ-EX-S08-03) — task check runs dogfood-examples after build; Taskfile.yml and verify.yaml both delegate to the shared discovery script" +# --------------------------------- 5. -count=1 on cache-blind `go test` gates -- +# +# COUNT1_TASKS: Taskfile tasks whose `go test` subject is NOT the test binary the +# cache keys on — it is a binary built (or prebuilt) at runtime and driven as a +# subprocess. For those the cache key is blind to the code under test, so a +# cached PASS is vacuous and `-count=1` is load-bearing: +# +# dogfood-comparison examples/comparison/validate_test.go is an EXTERNAL test +# package referencing only exported constants from +# internal/compare; the compare logic is unreachable from +# the test binary (linker strips it) and runs instead in a +# `go build -o … ./cmd/assent` subprocess. +# e2e test/e2e drives a PREBUILT bin/assent against a live +# GitLab endpoint; neither the binary bytes nor the forge +# state is in the cache key. +# +# Deliberately NOT listed: `test` (whole-tree `go test -race ./...` — in-process +# subjects, honest cache, and a real time saver) and `coverage` (in-process +# subjects; `-coverprofile` results do cache but the profile is reproduced +# faithfully). `determinism` needs no pin: `-count=2` is uncacheable by +# construction. +COUNT1_TASKS=(dogfood-comparison e2e) + +# count1_pinned — the task's `go test` COMMAND LINE carries +# -count=1. Anchored on the `- go test` command shape and stopping at `#`, so a +# comment that merely mentions -count=1 can never satisfy it. +count1_pinned() { + extract_block "$1" "$2" \ + | grep -qE '^[[:space:]]*-[[:space:]]+go test[^#]*[[:space:]]-count=1([[:space:]]|$)' +} + +echo "== 5. -count=1 pins on cache-blind go test gates ==" +for t in "${COUNT1_TASKS[@]}"; do + extract_block "$TASKFILE" "$t" >"$WORK/def.$t" + [[ -s "$WORK/def.$t" ]] || fail "Taskfile task '$t' is missing or extracted EMPTY — the -count=1 assertion below would be vacuous" + grep -qE '^[[:space:]]*-[[:space:]]+go test' "$WORK/def.$t" \ + || fail "Taskfile task '$t' no longer runs a 'go test' command — re-derive whether it is still cache-blind before deleting this pin" + count1_pinned "$TASKFILE" "$t" \ + || fail "Taskfile task '$t' runs 'go test' WITHOUT -count=1 — its subject is a runtime-built/prebuilt binary the Go test cache key cannot see, so the stage can report 'ok … (cached)' and exit 0 on a tree where the test genuinely fails" + echo "OK: $t runs go test with -count=1" +done + +echo "== 5b. the -count=1 assertion is comment-blind (positive control) ==" +cat >"$WORK/comment-only.yml" <<'EOF' + dogfood-comparison: + cmds: + # -count=1 is required here, honest + - go test ./examples/comparison/... + + next-task: +EOF +if count1_pinned "$WORK/comment-only.yml" dogfood-comparison; then + fail "count1_pinned accepted a task whose -count=1 appears ONLY in a comment — the assertion is satisfiable by prose and therefore vacuous" +fi +echo "OK: a -count=1 that lives only in a comment does NOT satisfy the pin" + +echo "== 5c. the -count=1 assertion can fail (mutation: strip the flag, keep the comment) ==" +mutant_c1="$WORK/Taskfile.nocount1.yml" +sed -E 's/^([[:space:]]*-[[:space:]]+go test)[[:space:]]+-count=1/\1/' "$TASKFILE" >"$mutant_c1" +if cmp -s "$TASKFILE" "$mutant_c1"; then + fail "mutation did not land: $mutant_c1 is byte-identical to Taskfile.yml" +fi +grep -qE '^[[:space:]]*#.*-count=1' "$mutant_c1" \ + || fail "mutation is not the intended one: it removed the explanatory -count=1 COMMENTS too, so 5c would not prove comment-blindness under mutation" +for t in "${COUNT1_TASKS[@]}"; do + if grep -qE '^[[:space:]]*-[[:space:]]+go test[^#]*[[:space:]]-count=1' <(extract_block "$mutant_c1" "$t"); then + fail "mutation did not land for '$t': the command line still carries -count=1 in $mutant_c1" + fi + if count1_pinned "$mutant_c1" "$t"; then + fail "count1_pinned reports '$t' pinned in a Taskfile with the flag stripped from its command line — the assertion is vacuous" + fi +done +echo "OK: stripping -count=1 from the command lines (comments intact) turns the pin red for: ${COUNT1_TASKS[*]}" + +echo "== 5d. task check still runs dogfood-comparison (an unwired gate cannot be saved by -count=1) ==" +check_lists_task "$TASKFILE" dogfood-comparison \ + || fail "'task check' does not run 'dogfood-comparison' — the gate is defined but invoked by nothing" +mutant_dc="$WORK/Taskfile.no-comparison.yml" +grep -vE '^[[:space:]]+- task: dogfood-comparison$' "$TASKFILE" >"$mutant_dc" +[[ "$(wc -l <"$mutant_dc")" -lt "$(wc -l <"$TASKFILE")" ]] || fail "mutation did not land: $mutant_dc has the same line count as Taskfile.yml" +if check_lists_task "$mutant_dc" dogfood-comparison; then + fail "check_lists_task reports dogfood-comparison wired in a Taskfile with that line deleted — the assertion is vacuous" +fi +echo "OK: check runs dogfood-comparison, and deleting that line turns the assertion red" + +echo "PASS: dogfood wiring (REQ-EX-S08-02, REQ-EX-S08-03) — task check runs dogfood-examples after build; Taskfile.yml and verify.yaml both delegate to the shared discovery script; the cache-blind go test gates (${COUNT1_TASKS[*]}) carry -count=1" From f648691d67fe35658a750bb2956e2b1a996a8a08 Mon Sep 17 00:00:00 2001 From: Konrad Heimel Date: Wed, 19 Aug 2026 06:03:06 +0200 Subject: [PATCH 2/4] :bug: fix(ci): -count=1 on verify.yaml's comparison corpus dogfood MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The previous commit closed the LOCAL half of the stale-cache hole. This closes the half that matters more: .github/workflows/verify.yaml ran its own `go test ./examples/comparison/...` with no -count=1, and actions/setup-go@v7.0.0 restores GOCACHE (where test results live) across commits with no `cache: false` anywhere in the repo — so CI could serve `ok ... (cached)` and turn a genuinely red tree into a green PR. Same mechanism as the Taskfile half: the external test package references only exported constants from internal/compare, the linker strips the compare logic out of the test binary the cache keys on, and the real subject is a `go build -o ... ./cmd/assent` subprocess outside the cache key entirely. Fixed per invocation, deliberately. Do NOT "fix" this by setting `cache: false` on setup-go: that disables the Go build cache for every step of every job in every workflow — a broad, permanent speed regression to close one blind invocation. -count=1 on the affected command is the targeted fix. Audited and left alone in the workflows, with reasons: * verify.yaml `go test -race ./...` — whole-tree, in-process subjects; the cache key is honest and it is a real time saver. Its separate (-race) cache entry for examples/comparison is covered because the step below now re-runs that package with -count=1 in the same job. * verify.yaml determinism block — `-count=2` is uncacheable by construction. * schemas.yml `go test -race ./schemas/...` — in-process compile+validate; fixtures are read with os.ReadFile, which testlog records, so edits do invalidate. No subprocess, no external binary. The pin now covers both halves. hack/examples/dogfood_wiring_test.sh section 6 asserts that EVERY `go test ... ./examples/comparison...` command in verify.yaml carries -count=1, matching the command line and never a comment, and distinguishes "flag missing" (rc=1) from "command absent" (rc=2) so the assertion cannot pass by going vacuous. Three controls, all observed red for their own reason: a comment-only fixture; the flag stripped from the real command line with the explanatory comment left intact; and the command deleted outright. No new `task check` stage — CHECK_STAGES stays 19. Evidence this closed a real hole rather than adding a defensive flag: openspec/specs/p5-aud2-audit-remediation/spec.md:477 reads `Verify: delete the EffectChallenge term; task dogfood-comparison reddens`. That Verify step was FALSE before this lane (measured: the stage stayed green and cached) and is TRUE after it. The spec file is Integrator-owned and left untouched. --- .github/workflows/verify.yaml | 13 +++++- hack/examples/dogfood_wiring_test.sh | 70 +++++++++++++++++++++++++++- 2 files changed, 80 insertions(+), 3 deletions(-) diff --git a/.github/workflows/verify.yaml b/.github/workflows/verify.yaml index 921df69..b513dd7 100644 --- a/.github/workflows/verify.yaml +++ b/.github/workflows/verify.yaml @@ -98,8 +98,19 @@ jobs: - run: CGO_ENABLED=0 go build ./... - name: dogfood examples (EX-S08 — shared discovery script; every examples/packs/ with .assent/tests gates itself green under the real `assent test` CLI) run: bash hack/dogfood-examples.sh + # -count=1 IS THE GATE, exactly as in Taskfile.yml's dogfood-comparison. + # examples/comparison is an EXTERNAL test package referencing only exported + # constants from internal/compare, so the compare logic is unreachable from + # the test binary the Go cache keys on (the linker strips it) and the real + # logic runs in a `go build -o … ./cmd/assent` subprocess outside the cache + # key. actions/setup-go restores GOCACHE across commits, so without this + # flag CI can print `ok … (cached)` and pass a PR on a tree where the + # corpus genuinely fails. Fix it HERE, per invocation — do NOT "fix" it by + # setting `cache: false` on setup-go: that would slow every job in every + # workflow to close one blind invocation. Pinned by + # hack/examples/dogfood_wiring_test.sh. - name: comparison corpus dogfood (PCS-S08 — examples/comparison validates and runs green) - run: go test ./examples/comparison/... + run: go test -count=1 ./examples/comparison/... - name: PCS compare exit gate (RELSE-03 — REQ-PCS-S09) run: bash hack/compare/exitgate_test.sh # AUD-S02 (REQ-AUD-S02-02) — CHANGELOG.md drift gate, main + schedule only. diff --git a/hack/examples/dogfood_wiring_test.sh b/hack/examples/dogfood_wiring_test.sh index 24b50c0..3e448dc 100755 --- a/hack/examples/dogfood_wiring_test.sh +++ b/hack/examples/dogfood_wiring_test.sh @@ -15,7 +15,10 @@ # 2026-08-19). `-count=1` is what makes that stage a gate at all — so it is # pinned here, with a mutation control that strips the flag from the COMMAND # LINE while leaving the explanatory comment in place, which is exactly how a -# human would regress it. +# human would regress it. Section 6 pins the SAME flag on verify.yaml's copy of +# that invocation: actions/setup-go restores GOCACHE across commits, so the CI +# half is blind in exactly the same way, and it is the half where a vacuous PASS +# turns a red tree into a green PR. # # Follows the hack/release/changelog_gate_test.sh / example_format_inventory_test.sh # discipline: every "is it wired" assertion is re-run against a mutated copy @@ -226,4 +229,67 @@ if check_lists_task "$mutant_dc" dogfood-comparison; then fi echo "OK: check runs dogfood-comparison, and deleting that line turns the assertion red" -echo "PASS: dogfood wiring (REQ-EX-S08-02, REQ-EX-S08-03) — task check runs dogfood-examples after build; Taskfile.yml and verify.yaml both delegate to the shared discovery script; the cache-blind go test gates (${COUNT1_TASKS[*]}) carry -count=1" +# ------------------------- 6. the CI half of the -count=1 pin (verify.yaml) -- +# +# Taskfile.yml is only the local half. verify.yaml runs its OWN +# `go test … ./examples/comparison/…`, and actions/setup-go restores the Go +# build cache (where test results live) across commits, so an uncounted CI +# invocation can serve `ok … (cached)` and turn a genuinely red tree into a +# green PR. Same discipline as section 5: match the COMMAND, never a comment. + +# workflow_comparison_go_test_lines — every line that RUNS `go test` over +# examples/comparison. Anchored on the command shape (optional `- `, optional +# `run: `) and stopping at `#`, so both `run:` steps and `run: |` block lines +# match while a YAML comment mentioning the command cannot. +workflow_comparison_go_test_lines() { + grep -nE '^[[:space:]]*(-[[:space:]]+)?(run:[[:space:]]*)?go test[^#]*\./examples/comparison' "$1" || true +} + +# workflow_count1_pinned — 0: every such command carries -count=1. +# 1: at least one does not. 2: NO such command exists (vacuity, not success). +workflow_count1_pinned() { + local line found=0 + while IFS= read -r line; do + [[ -n "$line" ]] || continue + found=1 + grep -qE '[[:space:]]-count=1([[:space:]]|$)' <<<"$line" || return 1 + done < <(workflow_comparison_go_test_lines "$1") + ((found == 1)) || return 2 + return 0 +} + +echo "== 6. verify.yaml runs the comparison corpus with -count=1 ==" +rc=0; workflow_count1_pinned "$WORKFLOW" || rc=$? +case "$rc" in + 0) echo "OK: every 'go test … ./examples/comparison…' in verify.yaml carries -count=1" ;; + 2) fail "verify.yaml contains NO 'go test … ./examples/comparison…' command — either the CI dogfood step was deleted or it was respelled past this matcher; re-derive the pin rather than leaving it silently vacuous" ;; + *) fail "verify.yaml runs 'go test' over examples/comparison WITHOUT -count=1 — setup-go restores GOCACHE across commits, so that step can report 'ok … (cached)' and pass a PR on a tree where the corpus genuinely fails" ;; +esac + +echo "== 6b. the workflow assertion is comment-blind (positive control) ==" +cat >"$WORK/wf-comment-only.yaml" <<'EOF' + # -count=1 is required here, honest + - name: comparison corpus dogfood + run: go test ./examples/comparison/... +EOF +rc=0; workflow_count1_pinned "$WORK/wf-comment-only.yaml" || rc=$? +((rc != 0)) || fail "workflow_count1_pinned accepted a workflow whose -count=1 appears ONLY in a comment — the assertion is satisfiable by prose" +((rc == 1)) || fail "the comment-only workflow control failed for the WRONG reason (rc=$rc, want 1 = flag missing from a present command)" +echo "OK: a -count=1 that lives only in a YAML comment does NOT satisfy the workflow pin" + +echo "== 6c. the workflow assertion can fail (mutation: strip the flag, keep the comment) ==" +mutant_wf_c1="$WORK/verify.nocount1.yaml" +sed -E 's/^([[:space:]]*(-[[:space:]]+)?(run:[[:space:]]*)?go test)[[:space:]]+-count=1/\1/' "$WORKFLOW" >"$mutant_wf_c1" +if cmp -s "$WORKFLOW" "$mutant_wf_c1"; then + fail "mutation did not land: $mutant_wf_c1 is byte-identical to verify.yaml" +fi +grep -qE '^[[:space:]]*#.*-count=1' "$mutant_wf_c1" \ + || fail "mutation is not the intended one: it stripped the explanatory -count=1 COMMENTS too, so 6c would not prove comment-blindness under mutation" +[[ -n "$(workflow_comparison_go_test_lines "$mutant_wf_c1")" ]] \ + || fail "mutation removed the comparison command entirely — the mutant would go red for vacuity (rc=2), not for the missing flag" +rc=0; workflow_count1_pinned "$mutant_wf_c1" || rc=$? +((rc != 0)) || fail "workflow_count1_pinned reports verify.yaml pinned with the flag stripped from its command line — the assertion is vacuous" +((rc == 1)) || fail "the verify.yaml mutant went red for the WRONG reason (rc=$rc, want 1 = flag missing from a present command)" +echo "OK: stripping -count=1 from verify.yaml's command line (comment intact) turns the pin red, for the flag and not for vacuity" + +echo "PASS: dogfood wiring (REQ-EX-S08-02, REQ-EX-S08-03) — task check runs dogfood-examples after build; Taskfile.yml and verify.yaml both delegate to the shared discovery script; the cache-blind go test gates (${COUNT1_TASKS[*]}) carry -count=1, in Taskfile.yml AND in verify.yaml" From 760dab9683e39526cee0162300813c6ffd9dd898 Mon Sep 17 00:00:00 2001 From: Konrad Heimel Date: Wed, 19 Aug 2026 06:27:09 +0200 Subject: [PATCH 3/4] :bug: fix(ci): run the -count=1 wiring pin in the PR-visible verify job MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review F1 — the lane's own thesis did not apply to the lane. The wiring pin reached CI only through `task check` -> release-exitgate, which carries `if: github.event_name != 'pull_request'`. A PR that stripped -count=1 from verify.yaml would have merged green and reddened main afterwards: RELSE-08, the blind spot verify.yaml already names as what let AUD-S18's stale CHECK_STAGES pin merge green four times. Fixed the way AUD2-S05 fixed it — the gate now runs as a step in the pull-request-visible `verify` job, beside workflow_pins_test.sh and before setup-go, since it is pure text over Taskfile.yml and this workflow and needs no toolchain. That step is itself pinned (section 7), mirroring AUD2-S05's check_pr_wiring, with a distinct return code per failure mode so a mutation control proves the branch it claims. Six controls, each observed red for its own code: step deleted (5); invocation COMMENTED OUT while the step comments still name the script (5); invoked with an argument (7); made advisory with continue-on-error (8); the verify JOB given release-exitgate's push-only guard (4); the workflow's `on:` losing pull_request entirely (2). The commented-out control caught a real bug while it was being written: a fixed-string presence check was satisfied by verify.yaml's own "Pinned by hack/examples/dogfood_wiring_test.sh" comment, so the check is now anchored on a `run: bash