pdf-parser-bench

Eight open-source PDF parsers over the same test documents, every capability graded against the source: MinerU 2.5, Granite-Docling, PaddleOCR-VL (all three served warm on one 24 GB L4 via hexread.com's production API), XBerg 1.0 (CPU), HURIDOCS PDLA v0.0.35 (VGT), LiteParse 2.11 (Tesseract), Chandra and LightOnOCR-1B (both on a rented L4).

Layout

• docs/ - the test inputs. Five of the six PDFs are synthetic, built to stress specific failure modes; the sixth (two pages of arXiv:1512.03385) is not redistributed - regenerate it with gen/02-fetch.sh. Plus two scan tests from archive.org, both public domain: 07-handwriting.png (a 1909 family letter) and 08-historical-print.png (a 1904 journal page full of italic book titles).
• outputs/<doc>/<model>.md - each model's raw output: the receipts behind every table cell. auto.md is hexread's router choosing per document. An empty file means the parser returned nothing.
• gen/ - generator scripts for the synthetic PDFs.
• run.sh - re-runs the three VLM rows + auto against the hexread API (HEXREAD_API_KEY env; a full pass bills ~40 quota pages).
• run-xberg.py - re-runs the XBerg row locally (pip install xberg==1.0.11, no GPU).

How the other rows ran

• PDLA: docker run huridocs/pdf-document-layout-analysis:v0.0.35. Scans go through its /ocr endpoint first (language=de), then /markdown. Speed measured on an L4; outputs are identical to CPU.
• LiteParse: lit parse <file> --format markdown --ocr-language deu.
• Chandra: pip install chandra-ocr[hf], then chandra <input> <outdir> --method hf. Note: without the [hf] extra it exits 0 with empty output. Time per page includes model load per document. Weights are OpenRAIL-M.
• LightOnOCR-1B: served with vLLM. Needs exactly vllm==0.11.2.
• Also attempted: MinerU-Diffusion V1. NaN crash in bf16, unusable output in fp32 (~6 min/page). Nothing to grade, so it has no row.

The handwriting ground truth

The 1909 letter has no published transcription - chosen deliberately, so models must read the pixels, not their training data. docs/07-handwriting-ground-truth.txt was established from two independent readings that agree. Words neither reading could resolve are marked [illegible] and excluded from scoring; a model that confidently fills them is graded as inventing text.

The MinerU footer note

MinerU's own .md generator discards blocks it classifies as page furniture (footers, page numbers, fine print) - on an invoice, that is the block with the IBAN. The model transcribes those blocks; the markdown lane drops them, and there is no option to keep them. hexread's pipeline rebuilds the document markdown from MinerU's block list (content_list), which retains them - the mineru-2.5.md outputs here come from that pipeline. mineru-2.5.pre-fix.md is the stock behavior for comparison: same model, footer gone.

License

Scripts and the synthetic documents: MIT. The arXiv paper's content belongs to its authors and is not included; gen/02-fetch.sh fetches and slices it locally. The letter (1909) and the journal page (1904) are public domain, via archive.org.