RABench

1. Setup

1. Install Python 3.12.12, uv, and Docker.
2. Install the locked dependencies.

uv sync --frozen

2. Reproduce the Paper Results

1. Generate all tables, figures, and supplementary files.

uv run python scripts/reproduce.py

2. Check that outputs/reproduction_report.json reports "status": "PASS" with no CSV mismatches.

3. Data and Pipeline

1. The final dataset is data/rabench.jsonl.gz, the construction records are in data/construction_index.json.gz, and replay manifests are in data/manifests/.
2. Prepare the source benchmarks with the scripts/download_* and scripts/build_* scripts.
3. Generate source agent trajectories with the five scripts/run_* scripts.
4. Build and combine repair windows with scripts/bench_build.py and scripts/bench_combine.py.
5. Run repair experiments with scripts/bench_repair_run.py.
6. Generate paper artifacts with scripts/paper_tables_1.py and scripts/paper_supplement.py.

4. Live Replay

1. Copy .env.example to .env and provide the model credentials and endpoints.
2. List replay tasks without calling a model.

uv run python scripts/bench_repair_run.py \
  --datasets appworld,gaia,swe_bench_pro,terminal_bench,openrca2 \
  --methods rarepair \
  --modes joint \
  --agent-llm-name MODEL_NAME \
  --agent-llm-base-url https://YOUR_ENDPOINT/v1 \
  --agent-llm-api-key dummy \
  --limit 2 \
  --first-window-only \
  --list

3. Remove --list and set the required method, mode, model, and worker arguments to start a replay.