Summary
In the LiveCodeBench grading service, an empty test suite causes every submission for that question to be scored as passing, silently inflating pass@1. Surfaced during review of #433 but pre-existing — not introduced by that PR.
Where
src/inference_endpoint/evaluation/livecodebench/lcb_serve.py, in _LCBWorker.__call__:
results[qid][code_idx] = all(case_result > 0 for case_result in res)
all([]) is vacuously True, so whenever res == [] the sample is marked PASS.
How res becomes empty
- A
{qid}.json test-case file that exists but has empty public_test_cases + private_test_cases yields a suite with inputs: []. LCBTestLoader(strict=True) — the default LCBServe uses — only raises on a missing file, not on empty content.
run_test over inputs: [] runs grade_stdio/grade_call_based across zero inputs, appends nothing, and returns ([], metadata).- The no-result branches in
run_code_subprocess likewise build res = [-1] * len(suite["inputs"]), which degenerates to [] when inputs == [].
Impact
A question with an empty/degenerate suite is a free PASS for every code sample → reported pass@1 is inflated with no error, no infra_error, and no raise. On an MLPerf accuracy path this is a silent correctness/validity bug.
Suggested fix
Treat an empty suite as an error/skip rather than a vacuous pass — e.g.
results[qid][code_idx] = bool(res) and all(c > 0 for c in res)
and/or reject empty suites explicitly upstream (raise in the loader, or classify as an infra-style error in run_code_subprocess so it is visible to the all-infra guard) rather than letting them reach all().
Provenance
Found by a multi-model review council (Claude accuracy lens + Grok 4.5) during review of #433; both independently rated it high/critical. Verified reachable under the default strict=True.
Summary
In the LiveCodeBench grading service, an empty test suite causes every submission for that question to be scored as passing, silently inflating pass@1. Surfaced during review of #433 but pre-existing — not introduced by that PR.
Where
src/inference_endpoint/evaluation/livecodebench/lcb_serve.py, in_LCBWorker.__call__:all([])is vacuouslyTrue, so wheneverres == []the sample is marked PASS.How
resbecomes empty{qid}.jsontest-case file that exists but has emptypublic_test_cases + private_test_casesyields a suite withinputs: [].LCBTestLoader(strict=True)— the defaultLCBServeuses — only raises on a missing file, not on empty content.run_testoverinputs: []runsgrade_stdio/grade_call_basedacross zero inputs, appends nothing, and returns([], metadata).run_code_subprocesslikewise buildres = [-1] * len(suite["inputs"]), which degenerates to[]wheninputs == [].Impact
A question with an empty/degenerate suite is a free PASS for every code sample → reported pass@1 is inflated with no error, no
infra_error, and no raise. On an MLPerf accuracy path this is a silent correctness/validity bug.Suggested fix
Treat an empty suite as an error/skip rather than a vacuous pass — e.g.
and/or reject empty suites explicitly upstream (raise in the loader, or classify as an infra-style error in
run_code_subprocessso it is visible to the all-infra guard) rather than letting them reachall().Provenance
Found by a multi-model review council (Claude accuracy lens + Grok 4.5) during review of #433; both independently rated it high/critical. Verified reachable under the default
strict=True.