Skip to content

[docs] Benchmark suite and ruleset proposal #468

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
arekay-nv wants to merge 1 commit into
base: main
Choose a base branch
from
Open

[docs] Benchmark suite and ruleset proposal #468

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
320 changes: 320 additions & 0 deletions docs/bench_suite.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,320 @@
# Benchmark Suite (status:proposed)

The Benchmark Suite is a Git-versioned record of reusable models, datasets, and the complete contract for each benchmark. A Ruleset is the approved grouping published for one cohort: benchmark IDs, a benchmark suite version, and seed sets. A submission names a cohort, seed set and benchmark, and the checker verifies the run against that benchmark contract.

## Definitions

- Benchmark suite - the git-versioned collection of models, datasets, and benchmarks available in the repository. The suite provides specifications for entire benchmarks (models, datasets, requirements etc), but can also be used for non-submission benchmarking. It is meant as a repository for supported items - everything in the suite at any given time should be supported by the client to conduct benchmarking runs.
- Cohort - a logical collection of submissions made during a certain duration. A cohort is a submission round defined by an identifier such as `2026-09-C0` . The current proposal is to have two cohorts per month, so `C0` and `C1` are the two cohorts for each month.
- Ruleset - defines the valid submissions for a cohort. Each ruleset is tied to a cohort and defines the benchmarks that are valid for that cohort. While datasets, models, benchmark updates can happen at any time, the inclusion in a ruleset for a cohort signals that the update is now a valid submission via rolling submission. A ruleset also includes other cohort specific items such as the seed sets that are valid for submissions in that cohort.
- Phase - each benchmark execution is defined as a series of phases. Currently they can be divided into four groups:
- Pre-processing - executed to generate the complete configuration files, as well as perform any sanity checks (for instance current repo matches the SHA for the cohort tag ).
- Performance - measured execution of the performance dataset with the user specified concurrency.
- Accuracy - correctness checks performed over the accuracy dataset and scored by the designated scorer and aggregated to component scores. Each accuracy dataset can be run multiple times (repeats) and there can be different ways of aggregating the scores from different runs (best of K or average over K). As there can be multiple accuracy datasets, we can also specify aggregation across accuracy datasets (unweighted average), or alternatively leave the scores per dataset separate.
- Post-processing - performs integrity checks on the datasets (SHA verification), number of samples issued, model checkpoints. It will also ensure that the accuracy scores satisfies the gates specified in the benchmark specification. There can also be requirements on the performance phase such as minimum duration, minimum number of samples issued, time-to-first-token constraints etc.

## Submitter file

The submitter will use a minimalist file to specify the submission:

```yaml
config.yaml # submission configuration; replaces today's submission_ref{model, ruleset}
type: submission
submission_ref:
cohort: 2026-09-C0 # frozen selection (YYYY-MM-C0/C1, submission rules §4.2)
seed_set_id: B
benchmark: gpt-oss-120b
# TODO - add path overrides.
endpoint_config:
endpoints: [ https://my-host/v1 ]
api_type: openai # must equal the benchmark's locked api_type
# response_mode comes from benchmark.execution
settings:
load_pattern: # submission-owned; concurrency is the only supported online type
type: concurrency
target_concurrency: 64 # one Pareto point; re-run across values to trace the curve
```

### Folder structure:

The `type: submission` entry in the yaml file above directs the front-end to load the submission details from the benchmark suite which is stored in the following folder structure:

```
benchmark_suite/
├── suite.yaml version; the loader globs the three directories below
├── models/<id>.yaml id, reference_id, tokenizer?, processor?, chat_template?,
│ metadata{}
├── datasets/<id>.yaml id, description, samples, source{url, sha256},
│ prompts{}, ground_truth_or_tests{}, evaluation_recipes{}?,
│ metadata{}
└── benchmarks/<id>.yaml id, family, model, api_type,
execution{} # response_mode - streaming vs non-streaming
preprocessing{} # input transform + optional validation hooks
performance{} # datasets, generation, runtime/sample/latency
# constraints
accuracy{} # datasets, recipes, generation,
# repeat_aggregation
# dataset_aggregation, thresholds
postprocessing{} # report, integrity, and compliance gates
rulesets/<cohort>.yaml
├── cohort, benchmark_suite_version
├── seed_sets[] # id, scheduler_rng_seed, sample_index_rng_seed, model_seed
└── benchmarks[] # complete benchmark IDs; no accuracy overrides


```

## Sample entries:

### Cohort specification(rulesets/2026-09-C0.yaml):

```yaml
rulesets/2026-09-C0.yaml
cohort: 2026-09-C0 # submission round; YYYY-MM-C0 (1st Wed) / -C1 (3rd Wed)
benchmark_suite_version: "1.0"

seed_sets: # a submission selects exactly one, by id
- id: A
scheduler_rng_seed: 16159082839903944936
sample_index_rng_seed: 2747215439041700203
model_seed: 42
- id: B
scheduler_rng_seed: <generated at ruleset publication>
sample_index_rng_seed: <generated at ruleset publication>
model_seed: <generated at ruleset publication>

benchmarks:
- deepseek-r1
- gpt-oss-120b

```

## DeepSeek-R1

### Dataset:

```yaml
datasets:
- id: mlperf_deepseek_r1
description: >-
Custom dataset curated by MLCommons for DeepSeek R1, specifically for the
MLPerf Inference benchmark
samples: 4388
source:
# provenance; not fetch-enforced
url: https://<public-host>/mlperf_deepseek_r1_dataset_4388_fp8_eval.jsonl
# verified against bytes loaded
sha256: <64 hex>
prompts: { column: question }
ground_truth_or_tests: { column: ground_truth }
evaluation_recipes:
mlperf_exact_match:
scorer: legacy_mlperf_deepseek_r1
ground_truth: ground_truth
```

### Model

```yaml
models:
- id: deepseek-r1
reference_id: deepseek-ai/DeepSeek-R1-0528
```

### Benchmark:

```yaml
benchmarks:
- id: deepseek-r1
family: text
model: deepseek-r1
api_type: openai
execution: { response_mode: streaming } # benchmark-wide; applies to every phase

preprocessing:
input_transform: chat_messages

performance:
datasets: [{ dataset: mlperf_deepseek_r1 }]
generation:
temperature: 0.0
top_p: 1.0
top_k: 1
max_new_tokens: 20000
requirements:
min_duration_ms: 600000
min_sample_count:
<TBD> # authoritative online value;
# freeze before the Ruleset is published

accuracy:
datasets:
- dataset: mlperf_deepseek_r1
evaluation_recipe: mlperf_exact_match
num_repeats: 1
generation:
temperature: 0.0
top_p: 1.0
top_k: 1
max_new_tokens: 20000
dataset_aggregation: { method: single_dataset }
thresholds:
exact_match: { reference: 81.3582, lower_factor: 0.99 }
TOKENS_PER_SAMPLE:
{ reference: 3886.2274, lower_factor: 0.9, upper_factor: 1.1 }

postprocessing:
gates:
[
dataset_integrity,
workload_integrity,
accuracy_gate,
requirements_lock,
]
```

## GPT-OSS-120b full specification

```yaml
# MODEL
models:
- id: gpt-oss-120b
reference_id: openai/gpt-oss-120b

# DATASETS
datasets:
- id: aime25
description: AIME 2025 (AIME2025-I + AIME2025-II, test split)
samples: 30
source: { url: <TBD>, sha256: <TBD> }
prompts: { column: question }
ground_truth_or_tests: { column: answer }
evaluation_recipes:
mlperf_exact_match:
{
scorer: pass_at_1,
extractor: boxed_math_extractor,
ground_truth: answer,
}

- id: gpqa
description: GPQA Diamond
samples: 198
source: { url: <TBD>, sha256: <TBD> }
prompts: { column: question }
ground_truth_or_tests: { column: ground_truth }
evaluation_recipes:
mlperf_exact_match:
{
scorer: pass_at_1,
extractor: abcd_extractor,
ground_truth: ground_truth,
}

- id: livecodebench
description: LiveCodeBench release_v6
samples: 1055
source: { url: <TBD>, sha256: <TBD> }
prompts: { column: question }
ground_truth_or_tests: { test_cases: test }
evaluation_recipes:
mlperf_code:
{ scorer: code_bench_scorer, extractor: python_code_extractor }

- id: gpt_oss_perf
description: <TBD>
samples: 6396
source: { url: <TBD>, sha256: <TBD> }
prompts: { column: input }

# BENCHMARK
benchmarks:
- id: gpt-oss-120b
family: text
model: gpt-oss-120b
api_type: openai
execution: { response_mode: streaming } # benchmark-wide; applies to every phase

preprocessing:
input_transform: chat_messages

performance:
datasets: [{ dataset: gpt_oss_perf }]
generation:
temperature: 1.0
top_p: 1.0
top_k: 0
reasoning_effort: low
max_new_tokens: 10240
requirements:
min_duration_ms: 600000
min_sample_count: 6396
min_context_length: 65536
max_ttft_latency_ms: 3000
max_tpot_latency_ms: 80

accuracy:
datasets:
- dataset: aime25
evaluation_recipe: mlperf_exact_match
generation: # sparse merge over the phase generation block
reasoning_effort: high
max_new_tokens: 32768
num_repeats: 8
repeat_aggregation: { method: mean_correctness } # illustrative; WG freezes
- dataset: gpqa
evaluation_recipe: mlperf_exact_match
generation:
reasoning_effort: high
max_new_tokens: 32768
num_repeats: 5
repeat_aggregation: { method: mean_correctness }
- dataset: livecodebench
evaluation_recipe: mlperf_code
generation:
reasoning_effort: high
max_new_tokens: 32768
num_repeats: 3
repeat_aggregation: { method: mean_correctness }
dataset_aggregation:
method: micro_average
weighting: base_questions
components: [aime25, gpqa, livecodebench]
thresholds:
exact_match:
scale: percentage
reference: 83.13
lower_factor: 0.99 # resolved threshold: 82.2987 (display 82.30)
requirements:
min_accuracy_sample_count: 4395

postprocessing:
gates:
[
dataset_integrity,
workload_integrity,
accuracy_gate,
requirements_lock,
]
```

## Submission flow:

1. Submitter writes a config yaml file that identifies the cohort, benchmark, seed and optional overrides (model paths, dataset paths etc) as well as the client specific settings (endpoint urls, worker configs, network configs, api keys etc).
2. The front end will perform initial validation on the submission metadata to check valid cohort, benchmark membership, and verify repository versions match the remote tags for that cohort.
3. Once validated, the specification is inlined into the yaml file to reveal an expanded configuration file which specifies exactly what needs to be executed. This ensures that the benchmark execution is correct by construction. The generated yaml file will be saved as part of the output artifacts.
4. The generated yaml is passed on to the execution phases which will run the phases in order and produce associated artifacts. For instance, the dataloader will produce an implementation specific SHA for the data loaded which will be logged.
5. Towards the end of the execution of the benchmark phases, post-processing step will apply the checks needed to verify submission integrity:
a) Dataset integrity - verify that the dataset loaded include a SHA from the dataloader and that matches the reference SHA. This requires the dataloader to output a SHA for each dataset loaded that would be format dependent. 
b) Model integrity - verify that the model id corresponds to the same model as specified in the benchmark suite. This should not be a hard error as we might have quantized models here. If this pass fails, we can flag it as a warning and advise the submitter to submit a document explaining the difference. (TODO - need a reliable way of determining the version of the model, computing the SHA is too expensive for large models.) For huggingface models, we can use the git SHA for the model root as an identifier.
c) Generation integrity - ensure that for each dataset, the generation config used in the config file is compliant with the benchmark specification. This includes api-type, generation parameters (top k, top p, max output sequence length, reasoning effort etc).
d) Accuracy gating - verify that the accuracy setup conforms to the benchmark specification including the aggregation methods. Compare the reported scores(s) to the benchmark specification.
e) Requirement gating - verify that the run satisfies the run requirements for the benchmark. This includes the duration requirements, sample counts, metrics (TPOT/TTFT) availability.

## Next steps

1. Address review comments.
2. Implementation plan (schema design + POC).
3. Reference implementations (DeepSeek-R1, GPT-OSS-120b, Llama3.1-8b).
4. Submission checker testing.
5. Adoption of submission process.
Loading