Add a Parquet-based graph dump/load serializer - #667
Open
rjb32 wants to merge 32 commits into
Open
Conversation
rjb32
commented
Jun 8, 2026
Serialize the four GraphMetadata schema maps (labels, edge-types, property-types, labelsets) to one Parquet file each, replayed through getOrCreate on load with id-match assertions; round-trip-tested via GraphMetadataComparator::same.
Serialize the journal's node and edge write sets to journal-nodes.parquet and journal-edges.parquet, read straight back into the raw write-set vectors in dumped order; round-trip-tested via WriteSetComparator. Property write sets are transient and not persisted, matching the binary format.
Serialize the node and edge tombstone sets to tombstone-nodes.parquet and tombstone-edges.parquet, inserted back into the unordered sets on load; round-trip-tested over GraphWriter-produced deletions via TombstoneSetComparator.
Serialize a commit's metadata to commit-metadata.parquet — the all-datapart id list as a column with num_nodes/num_edges/num_commit_dataparts in file key-value metadata — and decode it into a source-independent CommitParquetMetaData struct for the upcoming wiring step; round-trip-tested over a real GraphWriter graph.
Wire the graph- and commit-level orchestration on top of the Phase-1 DataPart
and Phase-2 commit adapters: GraphParquet{Dumper,Loader} and
CommitParquet{Dumper,Loader} emit/read a full Parquet graph directory
(graph-info, commit-log, commits/, dataparts/), the analogues of the binary
GraphDumper/CommitDumper/GraphLoader/CommitLoader. Loaders materialize only the
head commit, mirroring the binary path; shell ancestors dump/load their counters
only. DataPartParquetLoader gains a load(DataPart&) overload to fill a
VersionController-created part.
Validated by a GraphComparator::same round-trip over the Reactome sample
(extracted to examples/ReactomeSampleGraph, now shared with ReactomeTest) plus a
multi-commit-with-deletion case, and by the samples/parquet-roundtrip tool which
round-trips the full Reactome graph (2.97M nodes, 11.5M edges) and confirms it
matches the binary loader.
Align the Parquet EdgeIndexer path with the binary one: stop writing edge-indexer-patch.parquet and rebuild _patchNodeOffsets on load by recovering each patch node's id from its first edge (out-edge, else in-edge), exactly as EdgeIndexerLoader does. It is a cheap O(patch nodes) lookup with no sort or tree build, so it stays within principle 1's intent while keeping the two loaders identical and dropping one file per datapart. Validated by the DataPart patch round-trip (forces a patch node and checks its in-edges resolve through the rebuilt map) and the full Reactome graph round-trip against the binary loader.
Loaders declare their expected schema to ParquetReader and validate cross-column lengths, edge-range bounds and embedding dimensions before building anything, throwing FatalException instead of reading out of bounds. The graph dump takes the version-controller lock like the binary GraphDumper, writes into a sibling temp directory renamed into place on success, and stamps a format version checked first on load. Column names and metadata keys move into shared *ParquetLayout.h headers so the dumper and loader sides cannot drift.
The StringPropertyIndexer loader resolves property-type ids, node ids and child indices with clean FatalExceptions instead of escaping std::out_of_range; GraphParquetLoader refuses a target graph that already has commits; graph-info and datapart info files must have exactly one row. CommitParquetMetaData becomes a class with accessors, readInfo fills its visitor by reference, and PropertyManagerComparator compares container counts both ways.
… count A shared parseMetadataUint64 helper (std::from_chars) replaces the unguarded std::stoull/stoul calls in the loaders, so corrupt metadata values raise FatalException instead of escaping std::invalid_argument. CommitParquetLoader refuses a commit datapart count larger than the dump carries rather than underflowing the commit-datapart span, and PrefixTreeNode::getChild gets the same >= bound setChild and indexToChar already received.
The DataPart loader now checks every property indexer range against the loaded container's value count and refuses indexer entries that have no container; the EdgeContainer loader refuses out/in files whose first-id metadata is missing or disagrees. PropertyIndexerComparator compares labelset and range counts before zipping, indexToChar subtracts the alphabetical character count for numerals, and the Reactome sample doc states the exact node and edge counts.
A full dump of an in-memory graph (as opposed to the incremental dumpMissingCommits path taken on re-dump) walks every commit. Two cases were mishandled: - Shell commits (lazily-loaded ancestors whose CommitData has expired) have no in-memory data; CommitDumper and CommitMetaDataDumper dereferenced commit.data() unconditionally, segfaulting the binary dumper. They now dump only the metadata file with empty datapart lists, mirroring CommitParquetDumper. - Once a loaded graph is committed to, the previous head becomes a shell whose dataparts are still alive and referenced through the new head's allDataparts, but the per-commit walk no longer dumps them. Both the binary and Parquet dumpers now dump any datapart referenced by the head that the walk did not already write, so the dump stays loadable.
The edge and embedding paths materialized whole columns before touching the writer, and reassembled whole files before building containers, so peak memory was roughly double the data on large graphs (tens of GiB of transient buffers on top of the resident graph). - Edge and embedding dumpers now flatten one bounded row group at a time (512 MiB scratch) instead of the full column. - The embedding loader streams each row-group batch straight into the container; the edge loader writes each column's batches into the final EdgeRecord vector through a per-column cursor, dropping the separate int64 columns entirely. - The embedding container skips the reorder in sort() when the ids are already ascending (the common bulk-add case), which otherwise copied every embedding into a fresh container at commit time. Measured on ogbn_papers100m + 256-dim embeddings: zstd dump peak dropped from 273.7 GiB to 228.9 GiB, fitting in RAM without swap.
-dump-only and -load-parquet time a dump or load in isolation, -load-only and -dump-binary exercise the binary path, and -add-random-embeddings attaches a deterministic random embedding of the given dimension to every node. Used to benchmark dump size, dump/load time and peak memory against the binary serializer.
The Parquet property loader accumulated the whole value column (and the whole entity-id column) in a std::vector and copied it into the container in a buildX pass, doubling that column transiently on load. Only the embedding loader streamed. Each value type now streams straight into its TypedPropertyContainer<T> as the reader delivers it, through a single addValueBatch helper: the entity-id column (column 0) is read ahead of the value column (column 1) within every chunk, so each value batch can be added against ids already seen. This removes the per-column accumulation vectors and the buildX copies for int64, uint64, double, bool and string, matching the edge and embedding loaders. Bounded for any graph whose bulk is a large scalar or string property, not just edge/embedding-heavy ones.
Brings the plan level with the hardening, memory-bounding and compression work committed on top of Phase 2: expected-schema validation and load bounds checks, safe-publish dump (lock + temp dir + rename + format version), shell-commit/orphaned-datapart handling, memory-bounded row-group streaming, ZSTD, and the benchmark sample. Marks the resolved risks and refreshes the source references.
… its column scratch buffers, and give the loader SAX visitors caller-owned output instead of public members
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces the bespoke binary graph dump format with a Parquet serializer that dumps every container faithfully and reconstructs nothing on load, streaming columns in bounded row groups and compressing with zstd.
Lots of tradeoff before making this default. I am keeping this PR open for discussion sake for now.
OGBN 100M nodes
Compressed parquet has better load time for 2x dump time compared to uncompressed parquet.
OGBN + embeddings dimension 256
114.6 GB embedding payload compressed only ~11%