English | 简体中文 | 繁體中文 | 日本語 | 한국어 | Deutsch | Español | Français
🧪 We are actively seeking external reviewers and predictor authors.
15–30 min sanity checks, reproduction attempts, dataset suggestions and adversarial critiques are welcome.
NeoRepro is a leakage-aware, patient-level, reproducible benchmark resource for public MHC-I peptide–HLA neoantigen predictors. It packages pinned predictor artifacts, record-level provenance, training-overlap audits, common-support comparisons, patient-level uncertainty, support-matched random baselines and machine-generated results.
This is a benchmark/resource contribution, not a new predictor and not a claim of a universal model winner or clinical utility.
- Current manuscript: resource-positioned manuscript.
- Two-minute evidence summary: bilingual expert brief and independent-cohort extension summary.
- Reproduce the frozen results: use the command in the Reproduce section below.
- Benchmark your own predictor: run
neorepro benchmark predictions.csv --output-dir neorepro-results; see the standard one-file contract. - Add a third-party dataset or predictor: follow the plug-in contract.
- Cite a fixed version: CITATION.cff, the v0.1.1 release and Zenodo DOI 10.5281/zenodo.22037064.
For the scientific contract and scope, see RESEARCH_SPEC.md.
- Current-literature audit: complete; decision
RESCOPE, then GO - Benchmarked predictors: MHCflurry 2.2.1, BigMHC v1.0, PRIME 2.0, DeepImmuno-CNN and DeepHLApan; seven additional public tools have versioned profile-only, non-comparable or failed reproduction records
- TESLA pilot: complete; reclassified as a training-overlap-positive control
- Primary benchmark: IMPROVE, 17,475 leakage-filtered rows, 70 patients, 3 cohorts
- Primary IMPROVE inference: complete; 52,425 fixed-tool predictions with no missing rows
- External domains: Zhao vaccine cohort plus a separately frozen 129-record, 9-patient RCC vaccine cohort
- Reusable extension interface: machine-validated Dataset Card, Predictor Card and prediction-artifact contracts
- Manuscript: resource-positioned version, generated from frozen result files; independent statistical and biological review complete
The official PRIME2 supplement showed that all 520 records in the initial TESLA fixture were exact training overlaps, so they are retained only as a leakage-positive control. On the common exact-overlap-filtered, presentation-prefiltered IMPROVE benchmark, PRIME achieved AUROC 0.597 and mean patient-pMHC Recall@20 0.260; BigMHC achieved 0.546 and 0.146. In the independent Zhao vaccine cohort, BigMHC patient NDCG@5 was 0.658 versus a support-matched random reference of 0.578; DeepHLApan was 0.580 versus 0.578, while DeepImmuno-CNN was 0.755 versus 0.759 on 43.8% coverage. These results support an auditable, task- and support-aware evaluation contract, not a universal leaderboard.
Install uv, then rebuild every analysis, figure, table and manuscript artifact with the project-pinned CPython 3.11.15 from the versioned benchmark and prediction files:
make -j4 reproduce-resultsIndependent bootstrap analyses are parallelized by Make. Use make reproduce-results without -j4 when CPU or memory is constrained. make -j4 full-reproduce additionally downloads the pinned public source data and installs/runs the third-party predictors. It requires explicit acceptance of the academic-only BigMHC and PRIME terms, several gigabytes of disk space, and substantially more runtime.
- Primary scientific record: current manuscript, final report, and review record.
- Auditable outputs: final result table, figures, training-overlap audit, and SHA-256 manifest.
- Submission planning: target-venue strategy.
The independent Zhao 2026 vaccine-cohort extension is reproduced with make -j4 extension. Its concise evidence summary is in reports/extension_summary.md, with the frozen pre-inference contract in research/extension_protocol.json. The separately frozen RCC extension is documented in research/extension_protocol_rcc_v1.json, and exploratory three-domain stability outputs are under results/analysis/stability/. Both external endpoints are post-vaccination assays and must not be interpreted as natural tumor presentation or clinical efficacy.
Original NeoRepro code and documentation use the MIT License. Third-party predictors and datasets retain their own terms; inclusion in the study does not imply redistribution permission.