Skip to content

Conformance: add adversarial Trace mutation benchmark - #6

Closed
pathscale wants to merge 1 commit into
feat/trace-conformance-foundationfrom
feat/trace-adversarial-benchmark
Closed

Conformance: add adversarial Trace mutation benchmark#6
pathscale wants to merge 1 commit into
feat/trace-conformance-foundationfrom
feat/trace-adversarial-benchmark

Conversation

@pathscale

Copy link
Copy Markdown
Owner

Summary

  • add a versioned candidate adversarial benchmark over all 18 eligible Trace-semantic negative cases
  • pair the mutations with 10 unique conformant controls and require exact diagnostic-set matches
  • materialize every control and mutant, compute observed JSON-pointer differences, and fail when a pair escapes its declared mutation paths
  • enforce exhaustive selection so eligible cases cannot be omitted and ineligible cases cannot pad the score
  • publish a deterministic machine-readable report and verify it byte-for-byte on Linux and macOS
  • add one positive R2 control and align existing tamper fixtures into true controlled twins
  • normalize two conformance JSON artifacts so the repository-wide Biome gate passes

Current candidate result

  • Trace suite: 35/35 cases pass
  • controls accepted: 10/10
  • controlled mutations: 18/18
  • exact mutation detections: 18/18
  • eligible semantic mutants scored: 18/18
  • benchmark diagnostics: 0

Scope

This is stacked on #5 and should be reviewed after the Trace conformance foundation. The benchmark is intentionally bounded to semantic mutations determined by the candidate profile. It does not claim cryptographic receipt authenticity, truthfulness without independent producer facts, resource-exhaustion resistance, transport-decoder security, or coverage of unfrozen semantics.

Validation

  • cargo fmt --manifest-path conformance/runner/Cargo.toml -- --check
  • cargo clippy --locked --manifest-path conformance/runner/Cargo.toml --all-targets -- -D warnings
  • cargo test --locked --manifest-path conformance/runner/Cargo.toml
  • deterministic suite report comparison
  • deterministic adversarial report comparison
  • bun run typecheck
  • bun run lint
  • bun run build

@pathscale

Copy link
Copy Markdown
Owner Author

Landed on master via rebase (commits rewritten, so GitHub could not auto-detect the merge).

@pathscale pathscale closed this Aug 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant