Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,125 @@
# Builtins benchmark identity pins stale since efcfa0f: preflight cannot pass on main

## Approved current scope — 2026-09-11

Correct the CURRENT manifest-only mismatch, not the historical three-pin description.
Actual fixtures/builtins/v1/MANIFEST.tsv SHA-256 is
31798260263396c242c0b90042e01abb18624f383fd88029341dffecde662796; tools/bench already
uses it and required CI exercises its input-hash test. PCM508c8e94 and meter958a7026
identities already match; preserve all fixture bytes. Remaining outdated ddb4b201
consumers are scripts/preflight-builtins-benchmark.sh, scripts/test-builtins-benchmark.sh
(including its stub/seal rows), scripts/builtins-benchmark-record-validator.jq and
tools/audit/src/builtins_graph.rs. Reconcile those exact provenance values. Add one
cheap real-tree consumer-hash regression with a stale-constant negative control,
using existing test/preflight machinery and required CI wiring only if needed. Do
not repin unrelated validator/source seals or launch timed work; if existing runner
defects demand broader repair, preserve evidence and report before expanding scope.
Verify real-tree freshness, existing hermetic serializer/validator controls and
focused audit tests/Clippy appropriate to the constant change. Explain current
versus historical provenance; no claim that all benchmark infrastructure is fixed.

Astra LOW implements; Astra XHIGH independently verifies. Five attempts maximum.
Root commits exact paths at each coherent compiling/focused-green tranche and pushes
promptly before more edits. At most two active issues (#285/#176); isolated worktrees
and no overlapping paths. No timed benchmark, fixture regeneration, compiler captures,
DSP/runtime change, or performance claim. Preserve actual commands/environment/source/
exit/output evidence externally; ordinary compiler feedback is corrected within the
unfinished pass, while substantive failed gates receive bounded adversarial review.
Required exact-head PR and main qualification, upstream evidence and verified GitHub
closure precede clean delivered-worktree removal. Root owns delivery and any artifact
qualification; these tooling-only slices should not require a new shipped pin.

## Attempt 1 implementation record — 2026-09-11

The current mismatch is manifest provenance only: the four outdated consumers now
use `31798260263396c242c0b90042e01abb18624f383fd88029341dffecde662796`,
matching the unchanged real manifest and the already-current bench consumer. The
historical three-pin account below describes an earlier tree, not this repair.
Fixture bytes and graph PCM/meter pins are unchanged.

The existing benchmark test script now exposes `--check-manifest-consumers`, a
no-build real-tree check of seven declarations across five consumers (including
the synthetic preflight hash stub and seal row). It computes the manifest digest,
requires exactly one matching declaration per selector, and rejects a separately
mutated stale constant through the same predicate for every declaration. Required
qualification runs only this cheap mode; the full existing hermetic lifecycle
suite remains available without running a real benchmark.

No historical lock, validator, source, or artifact seals were repinned. This repair
does not claim that the historical exactly-once preflight is runnable on the current
branch or that all benchmark infrastructure is repaired. No fixture generation,
timed benchmark, DSP change, or performance claim belongs to this checkpoint.
Validation command/output evidence is preserved externally by the implementation
workflow; independent review and delivery evidence follow separately.

## Historical issue body

## Summary

Three builtins-benchmark identity pins have been stale since commit `efcfa0f` ("wip: re-pin builtins fixtures from the reference oracles"), which moved `fixtures/builtins/v1/` but did not update the consumers that pin its hashes. On `origin/main` at `edbbeeb` the builtins benchmark preflight could not have passed.

## Evidence (measured on `origin/main`, before any #163 phase-2 change)

| consumer pin | expected | actual file hash on main |
|---|---|---|
| `scripts/preflight-builtins-benchmark.sh` `manifest_sha256` | `bfcc7bbe…` | `c33781cf…` |
| `scripts/preflight-builtins-benchmark.sh` `graph_pcm_sha256` | `508c8e94…` | `e6294eba…` |
| `scripts/preflight-builtins-benchmark.sh` `graph_meter_sha256` | `958a7026…` | `03cc3979…` |

The same three values are duplicated in `scripts/test-builtins-benchmark.sh`, `scripts/builtins-benchmark-record-validator.jq`, `tools/miso-engine-audit/src/builtins_graph.rs` and `tools/miso-engine-bench/src/builtins.rs`.

`preflight-builtins-benchmark.sh` calls `require_hash fixtures/builtins/v1/MANIFEST.tsv "$manifest_sha256"`, so it fails immediately. It went unnoticed because `preflight-*.sh` is deliberately outside `scripts/sweep.sh` (it is minutes rather than seconds, and every gate it calls is an independent sweep row), and no builtins benchmark has been authorised since `efcfa0f`.

## The corroboration that identifies the cause

Regenerating the fixtures under the #163 phase-2 unfused numeric contract returns **two of the three** to the exact values the consumers still expected, byte for byte and without those constants being touched:

* `fixtures/builtins/v1/pcm/graph-taps.f32le` → `508c8e94…`
* `fixtures/builtins/v1/meters/graph-taps.jsonl` → `958a7026…`

Both are byte-identical to the fixture as it stood at commit `092ded7`, one commit before `efcfa0f`. That is independent evidence that `efcfa0f` re-pinned the fixtures against **fused** arithmetic while its consumers kept the pre-fusion bits, rather than the two sides having drifted for unrelated reasons.

Only `MANIFEST.tsv` needed a genuinely new value, because it also covers `pcm/filters-asymmetric.f32le`, `pcm/partition.f32le` and the four benchmark descriptors, which differ from the pre-`efcfa0f` era for other reasons.

## Disposition

The stale pins are repaired on branch `floor-phase2-contract` in commit `6fa5453`, because that commit's fixture regeneration touches the same files and leaving them knowingly divergent across it was not an option. They are **not** a phase-2 change and should be reviewed as a separate concern: the phase-2 contract change did not cause them, and would not have surfaced them if it had not regenerated the same corpus.

## Follow-up worth considering

1. The three values live in five places each. A single source (the manifest row itself, read at run time) would make this class of rot impossible.
2. `preflight-*.sh` being outside the sweep is a deliberate, documented cost decision. It is correct for runtime, but it means a preflight can rot silently for months. A cheap hermetic row that checks only the `require_hash` constants against the working tree — no build, no launch — would have caught this the day it landed.

## Independent review and joint delivery checkpoint

Astra XHIGH attempt1 SOURCE PASS atcc11fea32076d275647256f5028c959424d64963.
The real CLI rejects the original stale preflight pin and passes after exact
restoration. Required CI routing contract/mutations pass. All builtins fixture
tree objects and PCM/meter identities are unchanged; LOW lifecycle/audit/Clippy
receipts match the committed diff and carry forward. Source/GH/current-base
parity verified. Evidence: /tmp/issue176-xhigh-cc11fea3/review.md.

Root merges the two accepted histories for one tooling delivery. No shipped
engine/SDK source or artifact pin changed. Combined exact-head review, required
PR/main qualification and synchronized closure remain pending.

## PR qualification failure and bounded attempt 2

PR730 qualification34566763819 failed after the graph realtime trace and all
lifecycle/zero-violation assertions passed: the exact audit report hash was
c1baf7d5a6b598310c991844cdafa7896967041dd7e451cfea94d78312f3e929,
while scripts/trace-builtins-graph-audit.sh still expected
7a960a01270a67a430ee2db03d189f71b36f762f6b30ea555e16afbb42c917b0.
Astra XHIGH records attempt1 delivery FAIL. Its independent reconstruction of
the exact1006-byte report matches CI; changing only accepted_manifest_sha256
back to ddb4b201 reproduces the old pin byte-for-byte. No audio, counter or
lifecycle field changed. Failed log: /tmp/tooling285-176-pr-ci-failure.log.

Amend #176's exact paths only to include scripts/trace-builtins-graph-audit.sh.
Attempt2 updates that directly dependent report checksum, preserving all trace,
jq and exact-hash gates. This is related manifest provenance, not permission
for broad historical seal repair. Derivation/source/old-new byte proof must be
retained and independently rechecked against the final source. No timed
benchmark, fixture/runtime change, new issue or blind CI rerun. #285 source
PASS remains earned. Root checkpoints the coherent correction before renewed
required PR/main qualification.
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
# interchange source-path scan prunes only ./target — npm trees red it

## Approved current scope — 2026-09-11

Fix only the generated-artifact traversal in
scripts/check-effect-interchange-qualification.sh to prune node_modules directories
at any depth, preserving the existing root target exclusion and fail-closed traversal
errors. Extend scripts/test-effect-interchange-policy.sh with root and nested npm
Wasm acceptance and genuine source-path Wasm refusal. Do not exempt arbitrary source
paths or files merely named node_modules. Reuse the existing mutation harness; verify
normal baseline and all existing policy controls. No generic scan framework.

Astra LOW implements; Astra XHIGH independently verifies. Five attempts maximum.
Root commits exact paths at each coherent compiling/focused-green tranche and pushes
promptly before more edits. At most two active issues (#285/#176); isolated worktrees
and no overlapping paths. No timed benchmark, fixture regeneration, compiler captures,
DSP/runtime change, or performance claim. Preserve actual commands/environment/source/
exit/output evidence externally; ordinary compiler feedback is corrected within the
unfinished pass, while substantive failed gates receive bounded adversarial review.
Required exact-head PR and main qualification, upstream evidence and verified GitHub
closure precede clean delivered-worktree removal. Root owns delivery and any artifact
qualification; these tooling-only slices should not require a new shipped pin.

## Historical issue body

check-effect-interchange-qualification.sh's 'generated artifact exists under a source path' scan prunes only ./target, so any node_modules containing a .wasm (playwright-core's webp_codec.wasm) turns it red. Cost two agents a sweep run each (#272, #278). Fix: prune node_modules/ (any depth) in the scan; self-test row proving a planted wasm under src/ still reds.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Astra LOW attempt 1 focused checkpoint

Only the scan and existing policy harness changed. Actual node_modules directories
are pruned at any depth; root target exclusion and traversal-status refusal remain.
Root/nested npm Wasm and root-target acceptance, plus src/module.wasm,
src/node_modules.wasm and nested src/target/module.wasm rejection controls passed.
Shell syntax, real-tree baseline and full policy harness exited0. The harness
includes synthetic benchmark108/fake-launch lifecycle controls, not timed work.
Receipts: /tmp/issue285-attempt1 (actual argv/env/source/exit/logs). Independent
review and required PR/main qualification remain pending.

## Independent review and joint delivery checkpoint

Astra XHIGH attempt1 SOURCE PASS at3625c5d2. Fourteen independent controls
prove old/new npm regression, source-artifact refusal, root-target behavior and
empty/nonempty traversal-error refusal. Successful LOW baseline/full-policy
receipts reused. Exact head/base and GitHub parity verified. Review and actual
commands/source/exits/logs: /tmp/issue285-xhigh-o4odaeua/review.md.

Root merges the two accepted histories for one tooling delivery. No shipped
engine/SDK source or artifact pin changed. Combined exact-head review, required
PR/main qualification and synchronized closure remain pending.
2 changes: 2 additions & 0 deletions .github/workflows/qualification.yml
Original file line number Diff line number Diff line change
Expand Up @@ -315,6 +315,8 @@ jobs:
# binaries; only the hermetic mutation-test suite stays here.
- name: Standing builtins fixture manifest mutation tests
run: bash scripts/test-builtins-fixtures.sh
- name: Builtins benchmark real-tree manifest consumers
run: bash scripts/test-builtins-benchmark.sh --check-manifest-consumers
- name: Wasm console benchmark validator mutation tests
run: bash scripts/test-wasm-console-benchmark.sh
# test-builtins-benchmark.sh and test-graph-benchmark.sh moved to nightly's
Expand Down
2 changes: 1 addition & 1 deletion scripts/builtins-benchmark-record-validator.jq
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ def usable_text: type == "string" and length > 0 and . != "unknown" and . != "de
def render_workload:
. == "full_chain_filters" or . == "identity_chain" or . == "matrix_ramp" or . == "meter_success_full";
def frozen_manifest_sha256:
"ddb4b201dcd4cc00ad445013c9a1b29d9d5f6071f018e649748963c74af4c55b";
"31798260263396c242c0b90042e01abb18624f383fd88029341dffecde662796";
def frozen_input_sha256:
if .workload_kind == "full_chain_filters" and .sample_rate_hz == 48000 then
"4e5e2c9fc8e2c2400b816715273879f3635f2374133e5775ade18dabee1f6ad9"
Expand Down
2 changes: 1 addition & 1 deletion scripts/check-effect-interchange-qualification.sh
Original file line number Diff line number Diff line change
Expand Up @@ -238,7 +238,7 @@ if [[ -s "$fixture_scan" ]]; then
fail 'untracked/generated corpus appeared in interchange fixture directory'
fi
artifact_scan="$scratch/artifacts.out"; artifact_error="$scratch/artifacts.err"
if find . -path './target' -prune -o -type f \( -name '*.o' -o -name '*.a' -o -name '*.so' -o -name '*.dylib' -o -name '*.wasm' -o -name '*.profraw' -o -name '*.jsonl.raw' \) -print >"$artifact_scan" 2>"$artifact_error"; then status=0; else status=$?; fi
if find . \( -path './target' -o \( -type d -name node_modules \) \) -prune -o -type f \( -name '*.o' -o -name '*.a' -o -name '*.so' -o -name '*.dylib' -o -name '*.wasm' -o -name '*.profraw' -o -name '*.jsonl.raw' \) -print >"$artifact_scan" 2>"$artifact_error"; then status=0; else status=$?; fi
if [[ "$status" -ne 0 ]]; then cat "$artifact_scan" "$artifact_error" >&2; fail "generated artifact traversal failed (status $status)"; fi
if [[ -s "$artifact_scan" ]]; then
fail 'generated artifact exists under a source path'
Expand Down
2 changes: 1 addition & 1 deletion scripts/preflight-builtins-benchmark.sh
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ for path in "$sealed_binary" "$seal" "$raw_output" "$accepted_output" "$stderr_o
done

lock_sha256=4213efd775d1d1207fea805ccdc01392acb015ae36d1bf2eba783f938f19916a
manifest_sha256=ddb4b201dcd4cc00ad445013c9a1b29d9d5f6071f018e649748963c74af4c55b
manifest_sha256=31798260263396c242c0b90042e01abb18624f383fd88029341dffecde662796
graph_pcm_sha256=508c8e94244b99ae1ee59e4863088ba69c6462127eb0256f85ec72e775a17a19
graph_meter_sha256=958a702612b76353ae2dbb0f8a03a2e41aafbd90ed72857bc0c39a10b5d1935f
issue068_source_sha256=0c71b71d864fbdd01aa918c6825abea78c38f0486535bc914af92142a5080d19
Expand Down
47 changes: 44 additions & 3 deletions scripts/test-builtins-benchmark.sh
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,53 @@ set -euo pipefail

script_directory="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
repository_root="$(cd "$script_directory/.." && pwd)"
# Required CI runs only this real-tree check; the lifecycle suite below remains hermetic.
[[ "$#" == 0 || ( "$#" == 1 && "$1" == --check-manifest-consumers ) ]] || {
printf 'usage: %s [--check-manifest-consumers]\n' "$0" >&2; exit 2;
}
if [[ "${1:-}" == --check-manifest-consumers ]]; then
python3 - "$repository_root" <<'PYTHON'
import hashlib
from pathlib import Path
import re
import sys

root = Path(sys.argv[1])
expected = hashlib.sha256((root / "fixtures/builtins/v1/MANIFEST.tsv").read_bytes()).hexdigest()
# Match each declaration, including the synthetic preflight hash stub and seal row.
consumers = [
("scripts/preflight-builtins-benchmark.sh", r"^manifest_sha256=([0-9a-f]{64})$"),
("scripts/test-builtins-benchmark.sh", r'^manifest64="([0-9a-f]{64})"$'),
("scripts/test-builtins-benchmark.sh", r'MANIFEST\.tsv\)\s+hash=([0-9a-f]{64})'),
("scripts/test-builtins-benchmark.sh", r'fixture_manifest_sha256:"([0-9a-f]{64})"'),
("scripts/builtins-benchmark-record-validator.jq", r'def frozen_manifest_sha256:\s+"([0-9a-f]{64})";'),
("tools/audit/src/builtins_graph.rs", r'const ACCEPTED_MANIFEST_SHA256: &str =\s+"([0-9a-f]{64})";'),
("tools/bench/src/builtins.rs", r'const INPUT_MANIFEST_SHA256: &str =\s+"([0-9a-f]{64})";'),
]

def fresh(source, pattern):
return re.findall(pattern, source, re.MULTILINE) == [expected]

for name, pattern in consumers:
source = (root / name).read_text()
if not fresh(source, pattern):
sys.exit(f"stale or missing builtins manifest consumer: {name} ({pattern})")
match = re.search(pattern, source, re.MULTILINE)
# Mutate just the captured constant, then exercise the same freshness predicate.
stale = "0" * 64 if expected != "0" * 64 else "1" * 64
mutated = source[:match.start(1)] + stale + source[match.end(1):]
if fresh(mutated, pattern):
sys.exit(f"stale-constant negative control accepted: {name}")
print(f"builtins manifest consumers: PASS ({len(consumers)} declarations; stale controls rejected)")
PYTHON
exit 0
fi
command -v jq >/dev/null || { printf 'jq is required for benchmark validator tests\n' >&2; exit 1; }

hash64="4e5e2c9fc8e2c2400b816715273879f3635f2374133e5775ade18dabee1f6ad9"
binary64="bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
output64="cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc"
manifest64="ddb4b201dcd4cc00ad445013c9a1b29d9d5f6071f018e649748963c74af4c55b"
manifest64="31798260263396c242c0b90042e01abb18624f383fd88029341dffecde662796"
commit40="0123456789abcdef0123456789abcdef01234567"

record="$(jq -cn --arg hash "$hash64" --arg binary "$binary64" --arg output "$output64" --arg manifest "$manifest64" --arg commit "$commit40" '
Expand Down Expand Up @@ -600,7 +641,7 @@ case "$1" in
*Cargo.lock)
hash=4213efd775d1d1207fea805ccdc01392acb015ae36d1bf2eba783f938f19916a ;;
*fixtures/builtins/v1/MANIFEST.tsv)
hash=ddb4b201dcd4cc00ad445013c9a1b29d9d5f6071f018e649748963c74af4c55b ;;
hash=31798260263396c242c0b90042e01abb18624f383fd88029341dffecde662796 ;;
*fixtures/builtins/v1/pcm/graph-taps.f32le)
hash=508c8e94244b99ae1ee59e4863088ba69c6462127eb0256f85ec72e775a17a19 ;;
*fixtures/builtins/v1/meters/graph-taps.jsonl)
Expand Down Expand Up @@ -660,7 +701,7 @@ write_fake_nonbenchmark() {
lifecycle_sha256:$lifecycle,
record_validator_sha256:"45f2e0196b4e457a633980653536bb397af7f8ebc82ea69f49c8812dfa7dd9a6",
aggregate_validator_sha256:"6085e740f15d7902fca4443d761cfb8e29df7168ba12f632c7946db56a3e1b63",
fixture_manifest_sha256:"ddb4b201dcd4cc00ad445013c9a1b29d9d5f6071f018e649748963c74af4c55b",
fixture_manifest_sha256:"31798260263396c242c0b90042e01abb18624f383fd88029341dffecde662796",
graph_pcm_sha256:"508c8e94244b99ae1ee59e4863088ba69c6462127eb0256f85ec72e775a17a19",
graph_meter_sha256:"958a702612b76353ae2dbb0f8a03a2e41aafbd90ed72857bc0c39a10b5d1935f",
accepted_issue068_source_sha256:"0c71b71d864fbdd01aa918c6825abea78c38f0486535bc914af92142a5080d19",
Expand Down
Loading
Loading