feat(agent): add semantic graph evaluation experiment

[ynadge]

Primary change type

• Bug fix
• Documentation
• Feature
• Tests
• Dependency update
• Maintenance or refactor

Problem and motivation

SAP-3002 needs evidence that a Sapiom-hosted Luna model can recover semantic-only feeds relationships from immutable whole-project evidence before any production contract or rollout is designed. The experiment must distinguish structural validity from semantic correctness, preserve the Protocol-1 boundary, and remain deterministic and network-free in CI.

Summary and scope

• Adds the private, unpublished @sapiom/semantic-graph-eval workspace package with strict internal fixture, request, snapshot, scoring, and report contracts.
• Adds an immutable 18-case synthetic corpus (10 calibration, 8 holdout) covering positive, negative, mixed, malformed, failure, truncation, and prompt-injection behavior, with raw-byte hashes and deterministic regeneration.
• Builds stable whole-project packets, versioned precision-first prompts, a one-call provider boundary, Sapiom gpt-luna adapter (neverFail: false), raw mock replay, strict per-candidate quarantine, hidden-oracle scoring, normalized fingerprints, reports, and guarded CLIs.
• Runs the full deterministic 72-identity mock matrix and records the bounded Luna calibration/holdout evidence and rollout gates in DECISION.md.
• Keeps all evaluator schemas private. This PR intentionally adds no production persistence, scheduling, deployment, Graph API/UI behavior, render-time model call, or public semantic contract.

The evidence supports gpt-luna with the private bounded-source.v2 policy as the next shadow-evaluation starting point, but not user exposure: combined selected-configuration precision was 0.818 (9 TP, 2 FP, 1 FN), while the small frozen holdout was 1.0 precision/recall (6 TP, 0 FP, 0 FN). The public synchronous response has no authoritative per-call price and selected-config p95 latency was 16,934 ms, so the recorded rollout decision is no-go.

Related work

Related issue or discussion: SAP-3002

Stacked on #755 (yashnadge/sap-2986-infrastructure-adapt-direct-invocations-to-graph-evidence) at its current head ee72cd56ff34328c2ebb1701e81dcc7f80aea9ee.

Validation

pnpm --filter @sapiom/semantic-graph-eval fixtures:generate — regenerated 18 fixtures byte-for-byte
pnpm --filter @sapiom/semantic-graph-eval test — 10 suites, 47 tests passed
pnpm --filter @sapiom/semantic-graph-eval typecheck — passed
pnpm --filter @sapiom/semantic-graph-eval lint — passed
pnpm --filter @sapiom/semantic-graph-eval build — CJS and ESM builds passed
pnpm --filter @sapiom/agent test — 11 suites, 195 tests passed
pnpm --filter @sapiom/tools test — 34 suites, 609 tests passed
pnpm --filter @sapiom/semantic-graph-eval eval:mock — 72 runs; byte-locked aggregate passed; zero network calls
pnpm build — all 19 selected workspace packages passed
pnpm typecheck (after dependency build) — all 19 selected workspace packages passed
pnpm lint — all 19 selected workspace packages passed (existing warnings only)
pnpm test — every reached workspace suite, including SAP-3002, passed except one unchanged agent-core permission test; this sandbox retains read capability after chmod(0111), so `bundle-error.spec.ts` cannot stage its unreadable-directory case here

Real-model verification used only Sapiom gpt-luna, with no fallback, retries, or repair calls. All 48 calls in the final evidence dataset succeeded and disclosed serving class medium / lane run_now. A preliminary 48-call dataset was discarded after a packet-identity review correction; the decision record discloses all 96 paid invocations and the holdout limitation.

Tests and documentation

Added exact packet/snapshot goldens, table-driven validator tests, schema/fingerprint mutation tests, provider boundary tests, scoring/report tests, and full CLI end-to-end tests. Added package operating/safety documentation and the evidence-backed DECISION.md.

Automated review hardening removed speculative internal cost-header parsing and narrowed provider failures to actual client invocation errors. An HTTP-200 response without the forced structured output remains a successful provider attempt and is quarantined by deterministic malformed-output validation. Post-response harness faults now receive their own sanitized rejection code, so the CLI preserves paid-run evidence without mislabeling them as provider failures; both paths have explicit regression tests.

Review disposition: the recommendation to remove the gpt-luna label and measured decision record was not applied because both are explicit SAP-3002 deliverables and silently substituting or abstracting the requested model would invalidate the experiment. The SDK's RoutingLabel permits server-configured string labels, while the recorded class/lane values use its documented serving-disclosure vocabulary rather than a provider identity.

Compatibility and release impact

• Breaking or externally visible changes: None. The package is private and has no exports or production consumer.
• Changeset: N/A; no published package behavior or public API changes.

Security

• I have not included secrets, credentials, private data, or unsanitized logs.
• This pull request does not publicly disclose a suspected vulnerability. I will follow the Security Policy for private reporting.

AI assistance

• I did not use AI assistance for this change.
• I used AI assistance and have described it below.

OpenAI Codex implemented the private evaluator, synthetic fixtures, tests, and documentation from the approved SAP-3002 implementation plan. The complete diff was self-reviewed; review findings received red/green regression tests; fixture hashes, deterministic reports, filtered requirements, and repository-wide build/typecheck/lint were independently rerun before commit. No customer or production data was supplied to the experiment.

Checklist

• I read CONTRIBUTING.md, and this contribution follows the issue-first policy through SAP-3002.
• This pull request addresses one focused problem and contains no unrelated cleanup.
• I added or updated tests, or explained above why tests are not applicable.
• I ran the relevant build, typecheck, lint, and test commands, including the environment-specific root-test exception above.
• I updated documentation for the private experiment and recorded its measured decision.
• A Changeset is not applicable because no published package changes.
• I can explain and maintain every submitted change, including the AI-assisted work described above.

[github-actions]

Review — PR #760 (round 1)

🔒 CONFIDENTIALITY — the experiment record publishes Sapiom routing/gateway internals the SDK deliberately abstracts

The package is private: true, so nothing reaches npm — but this repo is public, and DECISION.md
and README.md are read by anyone browsing it. They disclose internals the platform's own public
contract says are never surfaced:

• gpt-luna as a named routing label. packages/tools/src/llm/index.ts:71 types RoutingLabel
  and documents the user-facing vocabulary as capability labels (smart | small | medium | large);
  packages/tools/src/llm/index.ts:346-357 states the label resolution is reported "in the SKU
  vocabulary the platform bills in … never a model or provider id". gpt-luna is a model-shaped
  identifier that appears nowhere else in the repo. It is hardcoded at src/provider.ts:3,
  src/report.ts:200, README.md:24, DECISION.md:8, and asserted on the wire in
  src/__tests__/provider.test.ts:75.
• The label → serving-class mapping. DECISION.md:19 records that gpt-luna resolves to class
  medium / lane run_now; src/__tests__/provider.test.ts:47 bakes in served_class: "luna".
  That is exactly the routing detail the disclosure contract exists to keep opaque.
• Unshipped gateway cost headers. src/providers/sapiom-luna.ts:109-113 names
  x-sapiom-cost-usd, x-sapiom-price-usd, x-sapiom-request-cost-usd. None exist anywhere in
  @sapiom/tools; these are internal/planned header names published as a guess.
• Measured quality and roadmap of a Sapiom-hosted model. DECISION.md:49-54 publishes precision
  0.444–0.6 and 16,934 ms p95 for a named Sapiom model, plus an internal go/no-go, ramp gates, and a
  per-snapshot dollar-cap plan for unshipped work.

Fix: take the concrete label out of committed prose and code — read it from configuration
(SEMANTIC_GRAPH_EVAL_MODEL_LABEL, required, no default) and refer to it as "the selected routing
label" throughout. If the measured class mapping, latency/precision numbers, and rollout gates need
to be recorded, record them where the experiment lives, not in the public SDK repo. The fixture
corpus itself is clean (generic Alpha/Producer/Consumer agents, agents/* paths, no URLs) —
this is the only leak surface.

costUsd is null by contract, so the cost gate can never be met by this code path

src/providers/sapiom-luna.ts:109-113 reads three cost headers that do not exist in the public
gateway surface, so usage.costUsd is null on every real call — which is exactly what
DECISION.md:82 observed (0% disclosure coverage). sumNullable (src/report.ts:57-67) then
collapses the aggregate costUsd to null whenever any run is missing it, i.e. always. Rollout
gate 8 ("cost metadata present on 100% of calls", DECISION.md:119) is therefore unreachable by
this harness, and the follow-up it schedules is chasing a header the disclosure contract explicitly
excludes ("never a provider price"). Either drop the field and say cost is out of scope, or point it
at the real billing surface rather than three guessed header names.

Client-side faults are recorded as provider failures

src/providers/sapiom-luna.ts:151 catches every throw from the whole try block — llm.run, but
also structuredOf, readDisclosure, and usageFromResponse — and returns
status: "failure" with a sanitized provider-error code. captured.status is only set if a fetch
resolved, so a client-side parse bug or a bug in the usage mapper is indistinguishable from a
gateway 5xx. src/cli.ts:254-262 then aborts the run reporting it as a provider failure, and
DECISION.md:19 cites "no fallback, provider failure, or malformed attempt was observed" as
evidence of gateway reliability, while DECISION.md:117 proposes a <1% provider-failure gate.
That metric currently measures the harness as much as the provider. Narrow the catch to the
llm.run call, or record the error class so the two are separable.

Failure scenario: client.llm.structuredOf throws on an unexpected content block shape → the run is
scored as a provider failure, inflating the failure rate and masking a harness defect.

---

Verdict: Request changes — the gpt-luna label, its serving-class mapping, the invented cost
headers, and the model's measured quality/roadmap must come out of the public repo before merge; the
cost and failure-rate metrics that DECISION.md builds gates on are not measuring what they claim.
Package hygiene is otherwise correct (private, no exports, no changeset needed, config matches
sibling packages, fixtures are generic, mock path is network-free and baseline-checked in CI).

[github-actions]

Review — PR #760 (round 2, follow-up)

Delta reviewed: 44edc719..d632cebd (3 files — DECISION.md, providers/sapiom-luna.ts, provider.test.ts).

Not fixed — 🔒 CONFIDENTIALITY (round 1, finding 1)

The push did not touch the leak surface. gpt-luna is still a committed literal in public
prose and code — README.md:24, DECISION.md:8,18,92, src/provider.ts:3,
src/report.ts:200, and pinned into the public type union at src/contracts.ts:373,488 —
and DECISION.md:19 still records the label → class medium / lane run_now mapping plus
the measured precision/latency and rollout gates for a Sapiom-hosted model. packages/tools/src/llm/index.ts:346-357
states serving disclosure is reported in billing vocabulary, "never a model or provider id";
gpt-luna is not in the public provider vocabulary pinned in scripts/provider-neutral-copy-check.mjs.
The PR body now repeats the label, the class mapping, the precision numbers and the paid-call
count as well. Fix is unchanged: read the label from required configuration, say "the selected
routing label" in prose, and keep the class mapping, measured quality and ramp gates out of the
public repo.

Fixed

• Invented x-sapiom-cost-* header parsing removed; costUsd is explicitly null with a
  comment, and gate 8 / the limitations now cite an authoritative billing surface instead of
  header disclosure coverage (round 1, finding 2).
• The catch is narrowed to the client.llm.run call, so gateway failures and client-side faults
  are separable (round 1, finding 3). The fabricated served_class: "luna" test value is now
  "medium", and the HTTP-200/missing-forced-output case has a real regression test.

New (low severity)

• sapiom-luna.ts:129-143: a throw from structuredOf / readDisclosure / usageFromResponse
  now escapes invoke instead of being recorded, and cli.ts:210-217 has no per-run guard — one
  such throw aborts the whole matrix and writes no report, discarding every prior paid call.
  Wrap the post-run block so the harness fault is recorded, not fatal.

Verdict: Request changes — the confidentiality finding is untouched and blocks merge; the two
correctness findings are resolved.

[ynadge]

Closing as superseded by the plan-first Agent Map direction.

Disposition: preserve the no-go decision, DECISION.md, corpus, and useful fixtures as research references. Do not merge the private semantic-evaluation package into the production workspace.

Frozen head: 9dfc3bed8d0f081277e4f4961c09292ceb913752
Base at closure: yashnadge/sap-2986-infrastructure-adapt-direct-invocations-to-graph-evidence

Replacement direction: Agent Studio: Plan-First Agent Map
Cleanup record: SAP-3045

No branch or commit history is being deleted. Any future extraction should be proposed in a new, consumer-scoped PR.