scilons-packaging

Bundle GROBID outputs (TEI-XML, JSON, Markdown) into compressed Parquet shards suitable for distribution and downstream reading via HF datasets, Spark, DuckDB, or pyarrow.dataset.

Designed for very large trees (tens of millions of files): planning does no per-file stat(), shards are slices of a single directory, and the plan stage runs as a SLURM job, never on the login node.

Content is stored verbatim. TEI extraction lives in harvesting/scilons_harvesting/.

Two pipelines

Which one you use depends on how the harvest landed on disk:

Source layout	Pipeline	Entry points	Wrapper
<root>/<dir_id>/*.zip (each zip holds *.tei.xml)	TEI-from-zip	grobid-pkg-tei-plan / grobid-pkg-tei-build	scripts/submit_tei_all.sh
<root>/<dir_id>/*.md, *.json	file-tree	grobid-pkg-plan / grobid-pkg-build	scripts/submit_all.sh

Both emit the same schema (below), so their outputs are joinable on (dir_id, doc_id) regardless of which produced them.

Legacy: the file-tree pipeline also accepts --tei-root / FORMAT=tei, packaging TEI from a tree of loose *.tei.xml files. It still works and is kept for old harvests, but current TEI harvests arrive as zips — use the TEI-from-zip pipeline for those.

Footgun: the two pipelines write different plan schemas under similar names. shard_plan.json is {"shards": {fmt: [...]}} (a dict keyed by format); tei_shard_plan.json is {"shards": [...]} (a flat list). They are not interchangeable — each *-build command only reads its own.

How the units of work differ

• File-tree: a shard is a (format, dir_id, slice_start, slice_end) range over the sorted filenames of one directory, sized to ~256 MB compressed. One big directory becomes many shards.
• TEI-from-zip: a shard is one whole zip. Zips are never split — their entries are DEFLATE-compressed individually but laid out sequentially, so reading half a zip already costs most of the I/O of reading all of it. Shard size is therefore whatever the zip holds; --target-shard-mb does not apply.

Schema (identical for every dataset)

field	type	notes
doc_id	string	basename without extension
dir_id	string	first-level subdirectory name (e.g. 0042)
relpath	string	path relative to format root
content	large_string	full file contents (UTF-8)
size_bytes	uint64	original on-disk size

No tagged format column: the three datasets live in separate directories and join on (dir_id, doc_id). For zip-sourced TEI, relpath is normalised to <dir_id>/<filename>.tei.xml — the zip origin is recorded in plan metadata, never in the data path, so it joins identically to file-tree output.

Install

cd scilons-packaging
pip install -e .

Usage — A. TEI from zips

A1. Plan

cd /netscratch/lfoppiano/scilons/packaging   # has .venv/ here
sbatch \
    --export=ALL,\
ROOT=/netscratch/.../grobid_tei_zips,\
OUTPUT_DIR=/netscratch/.../parquet \
    scripts/submit_tei_plan.sbatch

Writes ${OUTPUT_DIR}/tei_shard_plan.json (one shard per zip found) and tei_shard_plan_summary.txt.

Directories with no zip are skipped silently. If a directory contains more than one zip, discovery picks the alphabetically-first and logs a warning — check the plan log if your counts look low.

A2. Build

./scripts/submit_tei_all.sh \
    /netscratch/.../parquet/tei_shard_plan.json \
    /netscratch/.../grobid_tei_zips \
    /netscratch/.../parquet/tei

Note the output path. Unlike the file-tree pipeline, this one writes tei-NNNNN.parquet straight into the directory you give it — it does not append a format subdirectory. Pass .../parquet/tei explicitly if you want the uniform parquet/{tei,json,md}/ layout.

Single-shard smoke test:

grobid-pkg-tei-build \
    --plan /netscratch/.../parquet/tei_shard_plan.json \
    --shard-id 0 \
    --source-root /netscratch/.../grobid_tei_zips \
    --output-dir /tmp/teitest -v

A zip that has vanished between plan and build is a hard error (FileNotFoundError), not a warning — the array task fails loudly. Unreadable entries within a zip are logged and skipped.

Usage — B. Markdown + JSON file trees

The JSON and Markdown roots may point at the same physical tree; the per-format suffix filter (.json vs .md) keeps them independent.

B1. Plan — submit as a SLURM job

The plan stage walks every directory under the source roots. On a multi-million-file tree this would be killed by the login-node reaper, so it must run on a compute node.

The sbatch script activates the venv that holds grobid-pkg-* so the job has the package on PATH (SLURM compute nodes don't inherit a source activate from the login shell). The default is <submit-dir>/.venv — i.e. if you cd into the package root and keep your venv as .venv, no override is needed:

cd /netscratch/lfoppiano/scilons/packaging   # has .venv/ here
sbatch \
    --export=ALL,\
JSON_ROOT=/netscratch/.../grobid_json_md,\
MD_ROOT=/netscratch/.../grobid_json_md,\
OUTPUT_DIR=/netscratch/.../parquet,\
TARGET_MB=256 \
    scripts/submit_plan.sbatch

If your venv lives elsewhere, override with VENV_DIR=....

After ~1–5 minutes (depending on tree size and FS speed) the job writes ${OUTPUT_DIR}/shard_plan.json and a human-readable shard_plan_summary.txt.

Optional environment overrides for submit_plan.sbatch:

env var	default	meaning
VENV_DIR	${SLURM_SUBMIT_DIR}/.venv	Path to the venv with grobid-pkg-plan installed. Defaults to the .venv next to where you ran sbatch. The script sources ${VENV_DIR}/bin/activate.
TARGET_MB	256	Target compressed shard size (MB).
AVG_KB_TEI	45	Avg compressed bytes per TEI file (KB).
AVG_KB_JSON	25	Avg compressed bytes per JSON file (KB).
AVG_KB_MD	15	Avg compressed bytes per Markdown file (KB).

The avg-compressed-kb knobs drive target_files_per_shard = target_bytes / avg_kb. Real compression ratios get logged at build time — adjust these for round 2 if shards are systematically over- or under-target.

B2. Build shards

submit_all.sh resolves VENV_DIR to the package's .venv by default and forwards it (plus MAX_CONCURRENT) to each sbatch invocation. It takes all three roots positionally, in the order <plan> <tei_root> <json_root> <md_root> <output_dir>:

cd /netscratch/lfoppiano/scilons/packaging
./scripts/submit_all.sh \
    /netscratch/.../parquet/shard_plan.json \
    "" \
    /netscratch/.../grobid_json_md \
    /netscratch/.../grobid_json_md \
    /netscratch/.../parquet

This submits one SLURM array job per format that has shards in the plan. Output goes to ${OUTPUT_DIR}/${FORMAT}/.

For a smoke test before the full fan-out, build one shard:

cd /netscratch/lfoppiano/scilons/packaging   # so .venv is found by default
sbatch --array=0-0 \
    --export=ALL,FORMAT=md,\
SOURCE_ROOT=/netscratch/.../grobid_json_md,\
PLAN_FILE=/netscratch/.../parquet/shard_plan.json,\
OUTPUT_DIR=/netscratch/.../parquet \
    scripts/submit_pegasus.sbatch

Packaging a single format

Format selection happens at plan time. Pass only the roots you want; any root left unset (or set to the empty string) is skipped, and its entry in the plan is an empty list.

Markdown only, end to end:

# 1. Plan — omit TEI_ROOT and JSON_ROOT entirely.
sbatch --export=ALL,\
MD_ROOT=/netscratch/.../grobid_json_md,\
OUTPUT_DIR=/netscratch/.../parquet \
    scripts/submit_plan.sbatch

# 2. Build — pass "" for the roots you don't want.
./scripts/submit_all.sh \
    /netscratch/.../parquet/shard_plan.json \
    "" "" \
    /netscratch/.../grobid_json_md \
    /netscratch/.../parquet

submit_all.sh skips a format when either its root is "" ("skipping tei: no source root given") or the plan has 0 shards for it ("skipping tei: 0 shards in plan"). The empty-root check comes first, so the command above works against any plan — including a full three-format one. A root that is given but isn't a directory is a hard error, so a typo'd path fails at submit time instead of producing a silently missing dataset.

shard_id is numbered 0..N-1 within each format, so a single-format plan produces the same shard IDs for that format as a full plan would. Dropping a format never renumbers the others.

To submit just one format's array by hand instead:

PLAN=/netscratch/.../parquet/shard_plan.json
N=$(python3 -c "import json;print(len(json.load(open('$PLAN'))['shards']['md']))")
sbatch --array=0-$((N-1))%256 \
    --export=ALL,VENV_DIR=$PWD/.venv,FORMAT=md,\
SOURCE_ROOT=/netscratch/.../grobid_json_md,\
PLAN_FILE=$PLAN,\
OUTPUT_DIR=/netscratch/.../parquet \
    scripts/submit_pegasus.sbatch

This manual route bypasses the MAX_ARRAY_SIZE chunking below — see the next section if N is large.

Large fan-outs: MaxArraySize and SHARD_OFFSET

SLURM caps the number of indices in one array job (scontrol show config | grep MaxArraySize). Both wrappers handle this transparently: when a format has more shards than MAX_ARRAY_SIZE (default 1000), they issue several sbatch calls, each using indices 0..chunk-1 and exporting a different SHARD_OFFSET. The sbatch scripts recover the real shard ID as SHARD_OFFSET + SLURM_ARRAY_TASK_ID.

MAX_ARRAY_SIZE=4000 MAX_CONCURRENT=512 ./scripts/submit_all.sh ...

If you submit arrays by hand, you must do this chunking yourself — sbatch will reject --array=0-30000 outright.

env var	default	applies to
MAX_CONCURRENT	256	in-flight array tasks (%N suffix)
MAX_ARRAY_SIZE	1000	indices per sbatch; chunks past this
SHARD_OFFSET	0	set by the wrappers; only set it manually when hand-chunking
COMPRESSION_LEVEL	9	zstd level for the build tasks

COMPRESSION_LEVEL is not a positional argument of the wrappers — it reaches the array tasks through the ALL in their --export, so exporting it in the shell that runs the wrapper is enough:

COMPRESSION_LEVEL=3 ./scripts/submit_all.sh ...

Read back

import pyarrow.dataset as ds

tei = ds.dataset("/netscratch/.../parquet/tei", format="parquet")
print(tei.schema)
print(tei.count_rows())

# Filter by directory, project columns, scan lazily.
table = tei.to_table(
    filter=ds.field("dir_id") == "0042",
    columns=["doc_id", "content"],
)

# Join with JSON on (dir_id, doc_id):
js = ds.dataset("/netscratch/.../parquet/json", format="parquet").to_table()
joined = tei.to_table().join(js, keys=["dir_id", "doc_id"])

Local development

tests/make_sample.py builds a fixture that exercises every architectural property — slice-of-dir shards, JSON+MD shared root, byte-identical round-trip, slice non-overlap, and zip extraction:

flag	default	effect
--body-size-kb	4	size of each generated document's body
--multi-slice-files	0	if >0, add TEI dir 0099 with N files to force multi-slice shards
--tei-zip-dirs	0	if >0, add N dirs under <out>/tei_zip/, each with one .zip
--tei-zip-docs	10	TEI documents per zip

It writes <out>/tei/, <out>/json_md/ (shared JSON+MD tree) and, when requested, <out>/tei_zip/.

File-tree pipeline

python -m venv .venv && .venv/bin/pip install -e .

.venv/bin/python tests/make_sample.py /tmp/test/in --multi-slice-files 64

.venv/bin/grobid-pkg-plan \
    --json-root /tmp/test/in/json_md \
    --md-root   /tmp/test/in/json_md \
    --output-dir /tmp/test/out \
    --target-shard-mb 1   # tiny to force slicing

for fmt in json md; do
    n=$(.venv/bin/python -c "import json; \
print(len(json.load(open('/tmp/test/out/shard_plan.json'))['shards']['$fmt']))")
    for sid in $(seq 0 $((n-1))); do
        .venv/bin/grobid-pkg-build \
            --plan /tmp/test/out/shard_plan.json \
            --format $fmt --shard-id $sid \
            --source-root /tmp/test/in/json_md \
            --output-dir /tmp/test/out/$fmt
    done
done

TEI-from-zip pipeline

.venv/bin/python tests/make_sample.py /tmp/test/in --tei-zip-dirs 3 --tei-zip-docs 10

.venv/bin/grobid-pkg-tei-plan \
    --root /tmp/test/in/tei_zip \
    --output-dir /tmp/test/out

n=$(.venv/bin/python -c "import json; \
print(len(json.load(open('/tmp/test/out/tei_shard_plan.json'))['shards']))")
for sid in $(seq 0 $((n-1))); do
    .venv/bin/grobid-pkg-tei-build \
        --plan /tmp/test/out/tei_shard_plan.json \
        --shard-id $sid \
        --source-root /tmp/test/in/tei_zip \
        --output-dir /tmp/test/out/tei
done

Note the different JSON path in the two len(...) one-liners — ['shards']['$fmt'] vs ['shards'] — matching the two plan schemas.

Layout

scilons-packaging/
├── pyproject.toml
├── README.md
├── grobid_pkg/
│   ├── __init__.py
│   ├── schema.py         # PyArrow schema + per-format suffix table
│   ├── discover.py       # scandir-based, count-only, threaded
│   ├── planner.py        # slice-of-dir shards, file-count packing
│   ├── shard_builder.py  # re-list dir, take slice, write parquet
│   ├── tei_zip.py        # TEI-from-zip: discovery, plan I/O, build
│   └── cli.py            # all four grobid-pkg-* entry points
├── scripts/
│   ├── submit_plan.sbatch      # file-tree plan stage on a compute node
│   ├── submit_pegasus.sbatch   # file-tree: one task per shard
│   ├── submit_all.sh           # file-tree: one array job per format
│   ├── submit_tei_plan.sbatch  # zip plan stage on a compute node
│   ├── submit_tei.sbatch       # zip: one task per zip
│   └── submit_tei_all.sh       # zip: submits the array job
└── tests/
    └── make_sample.py

cli.py also supports script-style dispatch, useful when the console scripts aren't on PATH:

python -m grobid_pkg.cli {plan|build|tei-plan|tei-build} ...

Tuning

• --target-shard-mb — fewer/larger shards (better compression, fewer SLURM tasks) vs more/smaller shards (more parallelism, more metadata). 256 MB is a good default for HF datasets and Spark. File-tree pipeline only — zip shards are one-zip-per-shard.
• --avg-compressed-kb-{tei,json,md} — sets target_files_per_shard. Defaults are conservative (45/25/15 KB). Since planning never stats files, shard size is purely count × estimate: if your real files compress to twice the estimate, you silently get shards twice the target. Inspect the actual ratios logged at build time and re-plan for round 2.
• --workers — discovery thread pool size (default 16). Increase if your storage backend tolerates parallel metadata requests; lower if admins complain.
• COMPRESSION_LEVEL — zstd level, default 9, honoured by both submit_pegasus.sbatch and submit_tei.sbatch. Drop to 3 for ~2× faster builds at ~10 % larger output.
• MAX_CONCURRENT / MAX_ARRAY_SIZE — see the fan-out section above.

Troubleshooting

SOURCE_ROOT: SOURCE_ROOT must be set in an array task's log — the job was submitted with an empty root. ${VAR:?} fires on empty as well as unset. Current submit_all.sh refuses to submit in that case; if you see this, you are running an older copy of the wrapper against a plan that still has shards for the format whose root you passed as "". scancel the array and re-run.

A format is skipped with "0 shards in plan" — discovery found no files with that format's suffix. Check, in order:

# 1. What the plan actually contains
python3 -c "
import json; p=json.load(open('.../shard_plan.json'))
print({k: len(v) for k,v in p['shards'].items()})"

# 2. Which roots that plan was built from
python3 -c "
import json; print(json.load(open('.../shard_plan.json'))['metadata']['roots'])"

# 3. Whether the files are really there, at the expected depth
find <root> -maxdepth 2 -name '*.md' | head
find <root> -maxdepth 4 -name '*.md' | head

The suffix match is exact and literal (.md, .json, .tei.xml — see FORMAT_SUFFIX in schema.py). .markdown, .MD, or paper.md.gz will not match.

Layout depth matters. Discovery is exactly two levels: it lists <root>/*/ and counts matching files directly inside each of those directories. Files nested deeper are invisible to the planner:

<root>/0001/paper.md          ✅ found
<root>/0001/sub/paper.md      ❌ not found
<root>/paper.md               ❌ not found (files must be in a subdir)

If step 3 above finds files only at -maxdepth 4, that's your cause — the tree needs flattening, or the plan needs a root one level deeper.

The plan is stale. The plan records file index ranges, not names. If the tree changed after planning, re-plan rather than reusing the old JSON — see Design notes.

Design notes

The file-tree plan stores file index ranges, not filenames. Build workers re-list the directory at run time, sort filenames ascending, and take their slice. This keeps the plan small (~hundreds of KB even for tens of millions of files), at the cost of one assumption: directory contents must be stable between plan and build. For an offline batch this is fine. If files are added/removed in between, the build worker will warn and truncate the slice.

The zip plan stores the zip's relative path, so it has no equivalent index-drift problem — but a zip that moves or disappears between plan and build fails the task outright rather than degrading quietly.