Skip to content

feat: expand benchmark matrix and external evidence model - #6

Merged
Hubujiu merged 12 commits into
mainfrom
feat/expand-benchmark-matrix
Aug 24, 2026
Merged

feat: expand benchmark matrix and external evidence model#6
Hubujiu merged 12 commits into
mainfrom
feat/expand-benchmark-matrix

Conversation

@Hubujiu

@Hubujiu Hubujiu commented Aug 24, 2026

Copy link
Copy Markdown
Owner

Summary

  • expand Router from 16 to 28 public regression cases, with new cases covering all six routes
  • expand Decision from 4 to 10 full-profile two-turn scenarios (6 in standard)
  • expand Debug from 4 to 12 full-profile cases (8 in standard)
  • add eight new deterministic shared-boundary Debug fixtures covering header normalization, tenant cache isolation, pagination, duration units, blank-row parsing, inventory invariants, zero TTL semantics, and nullable sorting
  • add oracle fixes for every new Debug case and CI tests proving each seed fails while its oracle passes
  • keep the core runner separate from the evolving case catalog via run_catalog.py + case_catalog.py
  • update standard/full matrix counts and reproduction docs
  • document the external benchmark landscape: SkillsBench, LangChain skill benchmarks, SWE-bench, Terminal-Bench, FeatureBench, Senior SWE-Bench, ProgramBench
  • explicitly separate public regression evidence from external benchmarks and a future private held-out set

Current matrix

  • standard: 9 Delivery / 28 Router / 6 Decision / 8 Debug, n=3 by default (222 cells without previous/no-Skill arms)
  • full: 18 Delivery / 28 Router / 10 Decision / 12 Debug, n=3 by default (324 cells without previous/no-Skill arms)

Benchmark-design rationale

The new cases are deliberately different failure mechanisms and routing boundaries rather than paraphrases of the cases that already influenced SKILL.md. Public cases remain regression tests, not proof of unseen-task generalization. The next external-validation target is SkillsBench because it directly measures with-Skills vs without-Skills under paired deterministic evaluation.

@Hubujiu
Hubujiu merged commit ca75ad7 into main Aug 24, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant