Skip to content

Fix fail-open release gate behavior - #10

Merged
YinkaMetrics merged 6 commits into
mainfrom
agent/fail-closed-release-gate
Aug 5, 2026
Merged

Fix fail-open release gate behavior#10
YinkaMetrics merged 6 commits into
mainfrom
agent/fail-closed-release-gate

Conversation

@YinkaMetrics

Copy link
Copy Markdown
Owner

Summary

  • add a fail-closed INCONCLUSIVE gate verdict for paraphrasing, unverified adapters, and all-NOT-TESTED runs
  • reject --seeds 0 and enforce the same floor in the runner
  • make the HTTP pilot confirm write/delete convergence through /query
  • replace lossy Mem0/LangGraph slugs with reversible encoding
  • add a unique per-invocation run_id to namespaces and report headers
  • rerun the seven-arm live diagnostic and complete the Mem0 15×2 evidence regeneration

Public claim changes requiring founder review

This PR changes README.md and published evidence, so it must remain unmerged pending founder approval. Specifically it:

  1. states that paraphrasing and unverified runs are INCONCLUSIVE and exit non-zero rather than appearing green;
  2. corrects the Mem0 namespace claim to explain lossless identifiers and the new per-invocation run_id;
  3. documents bounded HTTP /query convergence after writes and deletes;
  4. distinguishes gate verdicts (PASS/FAIL/INCONCLUSIVE) from check state NOT TESTED;
  5. replaces “full regeneration pending” with the completed 2026-08-05 Mem0 regeneration and its run ID.

Root causes fixed

The gate treated an exact-match judge’s inability to observe paraphrased absence as evidence of absence. Deterministic run namespaces also allowed separate invocations to reuse provider namespaces, while native adapter slugification merged punctuation-distinct scopes. The HTTP runner additionally advanced immediately after accepted mutations even though doctor already treated the shim as eventually consistent.

Verification

  • offline suite: 87 passed, 4 credential-gated tests skipped
  • 15 scenarios validated, 0 warnings
  • strict reference full regeneration: 15 scenarios × 2 seeds, PASS
  • naive reference control: FAIL, exit 1
  • --seeds 0: argparse rejection, exit 2
  • delayed HTTP write/delete convergence covered end to end
  • seven-arm live Mem0 diagnostic: all arms visible in this execution; prior namespace loss plus current visibility establishes intermittency
  • full live Mem0 regeneration: 170 operations completed, run ID 1986edd5512147dca783bc513029b4f3

Mem0 result

The published lifecycle metrics are unchanged:

  • current-fact accuracy 46/46
  • stale reuse 10/10 (FAIL)
  • scope leakage 0/22
  • deletion residue 0/18
  • expiry NOT TESTED
  • utility delta +1.00
  • seed-stable, quoting answering layer

This matches the expected unaffected scope: pack identifiers do not slug-collide, the run used seeds=2, there was no concurrency, and the Mem0 answering layer quotes exact stored values.

Required label: needs-founder-review. Do not merge until explicit founder approval is recorded.

@YinkaMetrics

Copy link
Copy Markdown
Owner Author

Approved. Regen verified independently: SEARCH quota 885 → 599, consistent with a completed 15 × 2 executing against the live platform. All five external-review fixes verified on the branch — seeds floor, INCONCLUSIVE with non-zero exit, lossless identifiers, HTTP convergence, run_id. 87 tests green. Merge.

@YinkaMetrics
YinkaMetrics marked this pull request as ready for review August 5, 2026 01:08
@YinkaMetrics
YinkaMetrics merged commit f32adf2 into main Aug 5, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant