We have gsm8k (grade-school math). A harder math suite (AIME-style) would extend coverage for stronger
models where gsm8k saturates. Add an aime benchmark following the existing pattern: implement the
Benchmark protocol (tjbench/benchmarks/base.py), register it in tjbench/benchmarks/__init__.py
(get_benchmark), and use exact-match scoring (tjbench/benchmarks/scoring.py).
Acceptance criteria: tjb run --benchmark aime … runs; exact-match graded; tests + a docs entry in
docs/benchmarks.md. Use a properly licensed task source and note it.
We have
gsm8k(grade-school math). A harder math suite (AIME-style) would extend coverage for strongermodels where gsm8k saturates. Add an
aimebenchmark following the existing pattern: implement theBenchmarkprotocol (tjbench/benchmarks/base.py), register it intjbench/benchmarks/__init__.py(
get_benchmark), and use exact-match scoring (tjbench/benchmarks/scoring.py).Acceptance criteria:
tjb run --benchmark aime …runs; exact-match graded; tests + a docs entry indocs/benchmarks.md. Use a properly licensed task source and note it.