BrainPilotBench

Evaluating agents for brain science research.

BrainPilotBench is an open evaluation framework and curated task suite for testing whether agents can complete real brain science workflows and produce verifiable research artifacts.

Results · Leaderboard · Tasks · Quick Start · Evaluate Your Agent · Citation · Public Data

---

📰 News

• 2026-07-18 — BrainPilot was showcased at the “Intelligence in the Physical World” Science Forum at WAIC 2026. Follow us for the latest updates.
• 2026-07-17 — BrainPilotBench-v0 was released as open source. It provides four real brain-science tasks for agent-agnostic, artifact-based evaluation of the code, figures, models, and reports produced by scientific agents.

BrainPilotBench evaluates systems that search and synthesize neuroscience literature, analyze neural and behavioral data, write code, and produce figures, models, and reports. The interface is agent-agnostic: any system that completes a task and returns a valid submission bundle can be evaluated.

Important

BrainPilotBench-v0 is preliminary because the current public suite contains four tasks. All four have completed runs in the technical report. The package is not yet published to npm; use a source checkout.

Why BrainPilotBench

Real research workflows	Artifact-first evaluation	Comparable scoring	Evaluation integrity
Tasks use neuroscience literature, neural data, code, figures, and structured results.	Systems are judged by the deliverables they produce, not by a required architecture.	Each task defines its required artifacts, metrics, and grader.	Pinned data, held-out evaluation, provenance, and frozen task versions reduce leakage and drift.

The benchmark asks one central question:

Did the agent produce useful, verifiable brain science research artifacts under the task contract?

Reference evaluation

The BrainPilot technical report evaluates seven harness–backbone configurations across all four BrainPilotBench-v0 tasks. BrainPilot matched or approached the strongest evaluated configurations on multiple tasks, with a performance–cost trade-off across backbones.

Except for the BrainPilot RSC result, which follows the report's stated selection of the higher of two runs, all results are single runs. F denotes the absence of a valid primary score and is distinct from 0.00. These are reference runs from the technical report.

Explore the BrainPilotBench-v0 evaluation →

Task suite

BrainPilotBench-v0 contains four canonical public tasks:

Task	Capability	Compute	Primary scoring
neuro-rsc-place-cell	RSC calcium imaging, virtual-reality behavior, place-cell analysis, and decoding	CPU	Deterministic checks + human rubric
tops-fmri	Functional-connectivity modeling and held-out tonic-pain prediction	CPU	Study4 Pearson r + Study5 AUC
bciciv-2a	Four-class motor-imagery EEG decoding	GPU	Held-out accuracy + Cohen's kappa
sleep-edf	Five-class sleep staging	GPU	Held-out Cohen's kappa + per-class recall

Task-specific setup, data requirements, prompts, expected artifacts, and scoring details are documented in each task directory.

Public task data are available from BrainPilot-Bench/Tasks-Data-Public. Evaluator-only data remain gated and are never exposed to an agent run.

Quick start

Requirements

• Node.js 22 or newer
• npm
• Git
• Any task-specific Python, CPU, GPU, or storage requirements listed in its task README

Clone and build:

git clone https://github.com/NeuroAIHub/BrainPilotBench.git
cd BrainPilotBench
npm ci
npm run build
npm link

Run the zero-key example:

bp-bench submit verify examples/submission
bp-bench score examples/submission
bp-bench leaderboard examples

List the current tasks:

bp-bench list

Before starting a full task, read its linked README and run:

bp-bench doctor tops-fmri
bp-bench fetch tops-fmri --public

How it works

flowchart LR
    A["Task specification"] --> B["Agent or harness"]
    B --> C["Submission bundle"]
    C --> D["Contract verification"]
    D --> E["Task-declared scorer"]
    E --> F["Scores and coverage"]

Loading

1. A task defines the research goal, inputs, prompts, and required artifacts.
2. An agent works in its native environment and produces a submission bundle.
3. BrainPilotBench verifies that the bundle satisfies the task contract.
4. The task-declared grader evaluates the submitted artifacts.
5. Scores retain coverage and run-state information so missing measurements are not silently converted to zero.

Evaluate your agent

BrainPilotBench provides three adapters:

Adapter	Use case
brainpilot	A running BrainPilot deployment
command	A local agent command
manual	Any other harness using a prepared workspace and resume step

For an agent-independent manual handoff:

bp-bench run <task-id> --adapter manual --agent my-agent@1

Run your agent in the printed workspace, then execute the printed resume command. Verify and score the resulting bundle:

bp-bench submit verify "runs/<run-id>"
bp-bench score "runs/<run-id>"

To run against BrainPilot:

bp-bench run <task-id> \
  --adapter brainpilot \
  --base-url http://127.0.0.1:9001 \
  --workspace-root /absolute/path/to/workspaces \
  --agent brainpilot@<commit>

The built-in adapters all produce the same submission-bundle contract. This keeps task scoring independent of the agent language, model provider, and orchestration framework.

Scoring and integrity

• Each task declares its required artifacts and grader.
• Deterministic metrics, held-out evaluation, and expert rubrics are used where appropriate.
• A valid score of 0.00, an unscored run, and a metric that does not apply are represented as different states.
• Public inputs are pinned by content hash; evaluator-only inputs remain outside the agent workspace.
• Official results are produced from preserved artifacts and run metadata rather than accepted as self-reported numbers.

Maintainers with gated evaluator access follow the task-specific instructions; for example, bp-bench fetch tops-fmri --private. Private inputs are never staged in the agent workspace.

Documentation and contributing

• Start with the task-specific READMEs for setup and evaluation instructions.
• See CONTRIBUTING.md to propose tasks or contribute framework changes.
• See SECURITY.md to report a vulnerability.
• Open a Task Proposal for a new brain science workflow.

BrainPilotBench is a curated benchmark. Maintainers review scientific construct validity, data provenance, scoring design, and contamination risk before adding a task.

Citation

If BrainPilotBench has helped your work, we welcome you to cite our work!

If you use BrainPilotBench-v0 or its reference evaluation, cite the technical report:

@misc{li2026brainpilotautomatingbraindiscovery,
  title={BrainPilot: Automating Brain Discovery with Agentic Research},
  author={Haoxuan Li and Tianci Gao and Jianhe Li and Yang Fan and Runze Shi
    and Weiran Wang and Tianxiang Zhao and Zezhao Wu and Xiaoyang Jiang
    and Qihui Zhang and Jia Li and Xiao Xiao and Kai Du and Xiaoxuan Jia
    and Chao Xie and Lu Mi},
  year={2026},
  eprint={2607.15079},
  archivePrefix={arXiv},
  primaryClass={cs.AI},
  url={https://arxiv.org/abs/2607.15079}
}

For reproducibility, also report the repository commit, task IDs and versions, and the evaluated harness–model configuration.

Community

Questions, ideas, or just want to say hi? Join the BrainPilot community:

• 💬 Join the BrainPilot Slack →
• 🪶 Join the BrainPilot Feishu group →
• 📧 Contact: thu_neuroai@mail.tsinghua.edu.cn

You can also open an issue or start a discussion.

License

The framework package is licensed under MIT.

---

Build agents for brain science research that can be tested, compared, and trusted.