[BENCH-846] Make pooled WER the headline with a mean fallback - #617
Merged
Conversation
On the normalized path avg_value for WER is the pooled ratio when every clip in the window carries counts and the per-clip mean otherwise, with the mean always on mean_value; the S/I/D split and series value follow the same rule. Series points expose error_sum and reference_word_sum so consumers can pool across buckets. /v1/leaderboard gains a normalized path behind the reads flag and ranks on the same headline. Methodology doc gains the aggregation section, a dated not-comparable note, and ADR-024.
coval-cale
enabled auto-merge
September 9, 2026 20:48
kdmelon
approved these changes
Sep 9, 2026
kdmelon
left a comment
kdmelon
left a comment
Contributor
There was a problem hiding this comment.
Reviewed together with the normalized-table cutover at d90e335. The pooled WER calculation and mixed-history mean fallback check out. All 57 existing aggregate/leaderboard API tests, 22 readiness tests, and changed-file Ruff checks passed. Two targeted PostgreSQL-backed API probes reproduced the issues documented inline; a separate deterministic probe confirmed the inherited #612 validator mismatch. Production cutover readiness was not revalidated.
…n 30d LLM results have no normalized writer, so the reads flag now applies only to benchmarks the normalized store covers; the leaderboard and all three aggregates endpoints share the guard. The 30-day compact projection now carries error_sum and reference_word_sum alongside pooled_value.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On the normalized path avg_value for WER is the pooled ratio when every clip in the window carries counts and the per-clip mean otherwise, with the mean always on mean_value; the S/I/D split and series value follow the same rule. Series points expose error_sum and reference_word_sum so consumers can pool across buckets. /v1/leaderboard gains a normalized path behind the reads flag and ranks on the same headline. Methodology doc gains the aggregation section, a dated not-comparable note, and ADR-024.