swe-bench-lite is an experimental scaffold whose scoring is intentionally disabled — score() raises
NotImplementedError because the earlier version only checked whether a tool (e.g. bash) was called, which
is not real bug-fix verification (tjbench/benchmarks/swe_bench_lite.py, tools in
tjbench/agents/swe_bench_tools.py).
Implement genuine verification: check out the target repo at the task's base commit, apply the model's edit,
and run the FAIL_TO_PASS / PASS_TO_PASS test sets to decide pass/fail. Only remove the disabled gate once
scoring reflects real test outcomes.
Acceptance criteria: real per-task pass/fail from executed tests; sandbox/security notes in
docs/swe-bench-lite.md updated; the scoring-disabled gate removed only when verification is genuine; tests
cover the scoring path. This is a large task — scope a PR plan in the issue thread before implementing.
swe-bench-liteis an experimental scaffold whose scoring is intentionally disabled —score()raisesNotImplementedErrorbecause the earlier version only checked whether a tool (e.g. bash) was called, whichis not real bug-fix verification (
tjbench/benchmarks/swe_bench_lite.py, tools intjbench/agents/swe_bench_tools.py).Implement genuine verification: check out the target repo at the task's base commit, apply the model's edit,
and run the
FAIL_TO_PASS/PASS_TO_PASStest sets to decide pass/fail. Only remove the disabled gate oncescoring reflects real test outcomes.
Acceptance criteria: real per-task pass/fail from executed tests; sandbox/security notes in
docs/swe-bench-lite.mdupdated; the scoring-disabled gate removed only when verification is genuine; testscover the scoring path. This is a large task — scope a PR plan in the issue thread before implementing.