feat(benchmarks): one runner for four benchmarks, decider made visible - #425

Open

YueAWu wants to merge 22 commits into

mainfrom

feat/llm-multiround-retrieval

Commits on Aug 27, 2026

feat(search): make the decider tunable and its fallback visible
juwei.yue
committed
8b6c85d
fix(embedding): stop retrying permanent provider rejections
juwei.yue
committed
eb7503a
fix(runtime): bound OME runs, make the pool explicit, allow LLM extras
juwei.yue
committed
937571f
refactor(memory): decouple profile extraction from clustering
juwei.yue
committed
7919969
feat(benchmarks): one reproducible runner for all four benchmarks
juwei.yue
committed
e9840a7

Commits on Aug 28, 2026

docs(benchmarks): describe what the decider does with no endpoint
juwei.yue
committed
ca8aab6
docs(benchmarks): rewrite the guide, fold the converter into its adapter
juwei.yue
committed
65de8b3
feat(benchmarks): ship each benchmark's best config, and gate all four
juwei.yue
committed
90853c4

Commits on Aug 31, 2026

fix(tests): skip the private-shard test instead of raising on EACCES

juwei.yue
and
claude
committed
cb6aa76
build(deps): restore pypi.org as the lockfile's package index

juwei.yue
and
claude
committed
cc31c06
fix(search): pass the decider's max_tokens and extra to the LLM call

juwei.yue
and
claude
committed
9b63d46
chore: ignore the rendered benchmark result tables

juwei.yue
and
claude
committed
630add5

Commits on Sep 1, 2026

fix(search): let a degraded search reach the response it degraded

juwei.yue
and
claude
committed
f12fe16
fix(llm): build the decider a client that honours sdk_max_retries

juwei.yue
and
claude
committed
ce9a21c
fix(benchmarks): probe the decider the servers will actually use

juwei.yue
and
claude
committed
60f0aab
fix(benchmarks): score nDCG against the gold, and core per question

juwei.yue
and
claude
committed
0f0633f
fix(benchmarks): take --config as a path, and survive a missing /proc

juwei.yue
and
claude
committed
5b7f01c
fix(benchmarks): do not read a store whose ingest failed

juwei.yue
and
claude
committed
4d5bc08
docs(benchmarks): say the headline is micro, because it is

juwei.yue
and
claude
committed
f826076
fix(benchmarks): make resume prove it is the same experiment

juwei.yue
and
claude
committed
49ee22e
fix(benchmarks): digest a directory dataset, not just a file

juwei.yue
and
claude
committed
2711dbf
style(tests): keep the run-identity test names under the line limit

juwei.yue
and
claude
committed
92fc5e5