Hi, thank you for creating and open-sourcing ResearchCodeBench!
I am conducting a complementary diagnostic analysis inspired by the benchmark and am looking for two artifacts that I could not locate in the repository:
-
Task-level, model-level pass/fail results for the without-paper condition, corresponding to the runs produced by scripts/run_greedy_wo_paper.sh. These would allow us to reconstruct the "better with paper (86)" and "better without paper (75)" subsets from Figure 4 at the task level.
-
Raw code generations for each task and model, if saved. These would support analyses measuring overlap with the pseudocode and other implementation-relevant content in the source papers.
The directories under outputs/20llms_greedy/ appear to contain only overall_stats.json. Are the artifacts above available elsewhere, or would it be possible to add them to the release? The original files or a CSV/JSON export would both be very helpful.
We will appropriately cite and acknowledge ResearchCodeBench in any resulting work. Thank you for your time!
Hi, thank you for creating and open-sourcing ResearchCodeBench!
I am conducting a complementary diagnostic analysis inspired by the benchmark and am looking for two artifacts that I could not locate in the repository:
Task-level, model-level pass/fail results for the without-paper condition, corresponding to the runs produced by
scripts/run_greedy_wo_paper.sh. These would allow us to reconstruct the "better with paper (86)" and "better without paper (75)" subsets from Figure 4 at the task level.Raw code generations for each task and model, if saved. These would support analyses measuring overlap with the pseudocode and other implementation-relevant content in the source papers.
The directories under
outputs/20llms_greedy/appear to contain onlyoverall_stats.json. Are the artifacts above available elsewhere, or would it be possible to add them to the release? The original files or a CSV/JSON export would both be very helpful.We will appropriately cite and acknowledge ResearchCodeBench in any resulting work. Thank you for your time!