Conformance: add adversarial Trace mutation benchmark

.github/workflows/conformance.yml

@@ -26,8 +26,12 @@ permissions:
 
 jobs:
   suite:
-    name: Trace suite
-    runs-on: ubuntu-latest
+    name: Trace suite (${{ matrix.os }})
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, macos-latest]
+    runs-on: ${{ matrix.os }}
     steps:
       - name: Checkout standard and corpus
         uses: actions/checkout@v4
@@ -52,6 +56,11 @@ jobs:
           cargo run --quiet --locked --manifest-path conformance/runner/Cargo.toml -- run-suite . > /tmp/promptsyntax-conformance-report.json
           cmp conformance/reports/0.1.0-rc.1.json /tmp/promptsyntax-conformance-report.json
 
+      - name: Verify adversarial benchmark snapshot
+        run: |
+          cargo run --quiet --locked --manifest-path conformance/runner/Cargo.toml -- run-adversarial-benchmark . > /tmp/promptsyntax-trace-adversarial.json
+          cmp conformance/reports/trace-adversarial-0.1.0-rc.1.json /tmp/promptsyntax-trace-adversarial.json
+
   cross-language-core:
     name: Cross-language Core and Trace adapters
     runs-on: ubuntu-latest

conformance/README.md

@@ -7,7 +7,7 @@ are the oracle. Implementations consume the corpus but do not define its expecte
 ## Current status
 
 Version `0.1.0-rc.1` is a candidate suite against the specification draft at commit
-`7456634817f4eb68c8909a58b476505e6fd7c063`. It executes 34 document and producer cases
+`7456634817f4eb68c8909a58b476505e6fd7c063`. It executes 35 document and producer cases
 across four Trace families and covers all 22 requirements in its named candidate profile.
 It also owns 101 Core parser vectors and a full normalized-output differential protocol for
 the Rust and TypeScript implementations. A deterministic grammar generator adds a
@@ -32,6 +32,34 @@ The normative runner does not require Python, Node, Bun, a provider SDK, a netwo
 connection, or model credentials. The TypeScript implementation may be invoked separately
 with Bun when producing cross-language evidence.
 
+## Adversarial mutation benchmark
+
+The candidate suite also defines a scored adversarial benchmark over specification-owned
+Trace cases. Each mutation is paired with a conformant control. A mutation earns credit
+only when the control still passes and the mutant fails with the exact intended diagnostic
+set; merely rejecting the document is insufficient. The `0.1.0-rc.1` snapshot accepts all
+10 unique controls and exactly detects all 18 mutations across assembly and provenance,
+content integrity, routing and resolution, event and boundary integrity, and independent
+producer replay. The runner also enforces the declared selection rule: every negative case
+in the candidate families whose expected diagnostics are all Trace-specific must appear
+exactly once in the score. This makes the current result exhaustive over that bounded
+semantic-negative corpus rather than a hand-selected subset. The runner materializes each
+control and mutant, computes their observed JSON-pointer differences, and rejects a pair if
+any difference escapes its declared controlled mutation paths.
+
+Run and reproduce the frozen report:
+
+```bash
+cargo run --manifest-path conformance/runner/Cargo.toml -- \
+  run-adversarial-benchmark .
+```
+
+This benchmark measures detection of the bounded semantic mutations in its manifest. It
+does not establish cryptographic receipt authenticity, truthfulness without independent
+producer facts, resistance to resource exhaustion, transport-decoder security, or behavior
+for semantics outside the candidate profile. Those exclusions are machine-readable in the
+benchmark document and must remain explicit in paper claims.
+
 Run the repository-owned checker from the repository root:
 
 ```bash
@@ -40,6 +68,9 @@ cargo run --manifest-path conformance/runner/Cargo.toml -- \
 
 cargo run --manifest-path conformance/runner/Cargo.toml -- run-suite .
 
+cargo run --manifest-path conformance/runner/Cargo.toml -- \
+  run-adversarial-benchmark .
+
 cargo run --manifest-path conformance/runner/Cargo.toml -- \
   compare-core-adapters conformance/cases/core-parser.json \
   /tmp/promptsyntax-rs-core.json /tmp/promptsyntax-ts-core.json
@@ -58,16 +89,23 @@ cargo run --manifest-path conformance/runner/Cargo.toml -- \
 
 The command emits the deterministic report stored in
 `conformance/reports/0.1.0-rc.1.json`. CI regenerates it and requires a byte-for-byte match.
+The adversarial command likewise regenerates
+`conformance/reports/trace-adversarial-0.1.0-rc.1.json` byte-for-byte.
+The suite job performs both snapshot checks on Linux and macOS so diagnostic and report
+stability is tested across operating systems and differing checkout paths.
 
 ## Layout
 
 - `profiles/` freezes the named scope and candidate semantic decisions.
+- `benchmarks/` defines scored threat classes and paired semantic mutations.
 - `families/` contains language-independent cases and expected diagnostics.
 - `fixtures/` contains reusable Trace and deterministic transcript inputs.
 - `cases/core-parser.json` is the single 101-case Core fixture consumed by both adapters.
 - `core-differential-input.schema.json` freezes the generator-to-adapter input shape.
 - `core-adapter-result.schema.json` freezes the normalized cross-language output shape.
 - `core-differential-report.schema.json` freezes the independent comparison report shape.
+- `adversarial-benchmark.schema.json` and `adversarial-report.schema.json` freeze the
+  benchmark and result contracts.
 - `cases/trace-producer.json` lists six valid and four rejection cases for real producers.
 - `trace-producer-input.schema.json` defines deterministic facts without expected output.
 - `trace-producer-cases.schema.json` freezes the external producer adapter protocol.

conformance/adversarial-benchmark.schema.json

@@ -0,0 +1,142 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "$id": "https://promptsyntax.org/conformance/adversarial-benchmark/0.1-draft",
+  "title": "PromptSyntax Trace adversarial mutation benchmark",
+  "type": "object",
+  "additionalProperties": false,
+  "required": [
+    "format_version",
+    "id",
+    "version",
+    "status",
+    "suite",
+    "profile",
+    "evidence_unit",
+    "selection_rule",
+    "scoring",
+    "scope",
+    "threat_classes",
+    "mutations"
+  ],
+  "properties": {
+    "format_version": { "const": "0.1-draft" },
+    "id": { "const": "trace-adversarial" },
+    "version": { "type": "string", "minLength": 1 },
+    "status": { "enum": ["candidate", "released"] },
+    "suite": {
+      "type": "object",
+      "additionalProperties": false,
+      "required": ["id", "version"],
+      "properties": {
+        "id": { "const": "promptsyntax-conformance" },
+        "version": { "type": "string", "minLength": 1 }
+      }
+    },
+    "profile": { "type": "string", "minLength": 1 },
+    "evidence_unit": { "const": "paired-semantic-mutation" },
+    "selection_rule": { "const": "all-negative-cases-with-trace-diagnostics" },
+    "scoring": {
+      "type": "object",
+      "additionalProperties": false,
+      "required": ["control_acceptance", "mutation_detection", "aggregation"],
+      "properties": {
+        "control_acceptance": { "const": "baseline-must-pass-as-conformant" },
+        "mutation_detection": {
+          "const": "mutant-must-fail-with-exact-diagnostic-set"
+        },
+        "aggregation": { "const": "unweighted-case-count" }
+      }
+    },
+    "scope": {
+      "type": "object",
+      "additionalProperties": false,
+      "required": ["claim", "attacker_capabilities", "excluded"],
+      "properties": {
+        "claim": { "type": "string", "minLength": 1 },
+        "attacker_capabilities": {
+          "type": "array",
+          "minItems": 1,
+          "uniqueItems": true,
+          "items": { "type": "string", "minLength": 1 }
+        },
+        "excluded": {
+          "type": "array",
+          "minItems": 1,
+          "uniqueItems": true,
+          "items": { "type": "string", "minLength": 1 }
+        }
+      }
+    },
+    "threat_classes": {
+      "type": "array",
+      "minItems": 1,
+      "items": {
+        "type": "object",
+        "additionalProperties": false,
+        "required": ["id", "description"],
+        "properties": {
+          "id": { "$ref": "#/$defs/kebab_id" },
+          "description": { "type": "string", "minLength": 1 }
+        }
+      }
+    },
+    "mutations": {
+      "type": "array",
+      "minItems": 1,
+      "items": {
+        "type": "object",
+        "additionalProperties": false,
+        "required": [
+          "id",
+          "threat_class",
+          "operator",
+          "baseline_case",
+          "mutant_case",
+          "controlled_paths",
+          "attack",
+          "intended_diagnostics"
+        ],
+        "properties": {
+          "id": { "type": "string", "pattern": "^A[0-9]{3}$" },
+          "threat_class": { "$ref": "#/$defs/kebab_id" },
+          "operator": {
+            "enum": [
+              "content-corruption",
+              "cross-source-divergence",
+              "duplication",
+              "field-omission",
+              "field-replacement",
+              "metadata-corruption",
+              "policy-conflict"
+            ]
+          },
+          "baseline_case": { "$ref": "#/$defs/kebab_id" },
+          "mutant_case": { "$ref": "#/$defs/kebab_id" },
+          "controlled_paths": {
+            "type": "array",
+            "minItems": 1,
+            "uniqueItems": true,
+            "items": {
+              "type": "string",
+              "pattern": "^/(trace|transcript)(/([^/~]|~0|~1)+)+$"
+            }
+          },
+          "attack": { "type": "string", "minLength": 1 },
+          "intended_diagnostics": { "$ref": "#/$defs/diagnostic_set" }
+        }
+      }
+    }
+  },
+  "$defs": {
+    "kebab_id": {
+      "type": "string",
+      "pattern": "^[a-z0-9][a-z0-9-]*$"
+    },
+    "diagnostic_set": {
+      "type": "array",
+      "minItems": 1,
+      "uniqueItems": true,
+      "items": { "type": "string", "pattern": "^[A-Z][A-Z0-9_]+$" }
+    }
+  }
+}