WIP: feat(nds): implement on-disk gc - #1201
Draft
emturner wants to merge 11 commits into
Draft
Conversation
|
Durable storage space metrics for revision b780fe8, at pull request
Full reportThese figures are for catching a regression before it merges. Compare them with |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## emturner@working-tip #1201 +/- ##
=======================================================
Coverage ? 90.07%
=======================================================
Files ? 159
Lines ? 35178
Branches ? 35178
=======================================================
Hits ? 31686
Misses ? 2387
Partials ? 1105 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
emturner
force-pushed
the
emturner@working-tip
branch
from
September 7, 2026 16:59
3b3b26d to
c927784
Compare
emturner
force-pushed
the
emturner@disk-gc-fullstack
branch
from
September 7, 2026 16:59
a636fb7 to
8929ec2
Compare
Committing rewrote every node resolved in memory, changed or not, because the only thing that stopped the recursion was a child that had never been loaded - the whole tree for a state just built, and everything touched since for one checked out. The bytes were identical to what was already stored, but they are new writes as far as RocksDB is concerned, so each commit flushed them into fresh files that no earlier checkpoint shares. A node now records which store its body reached, and committing skips it and the subtree beneath: changing a descendant changes every hash above it, so an unchanged node cannot have a changed descendant. invalidate_hash clears the memo, so it is dropped by exactly the mutations that change what a node hashes to, and loading records the store read from, so a checked-out tree does not rewrite what it touches. It records the store rather than a bare flag because copy_database hands the destination a copy of the source store and the same in-memory nodes. A node still dirty then owes a copy to both, and a flag let the first commit satisfy the obligation and the second skip it, leaving a commit referring to nodes that never reached its store. Its low bit records whether the value data went too, so a commit that must write it is not skipped on the strength of one that need not. Also invalidates the hash in balance_factor_mut, so correctness does not depend on rebalancing happening to run after something else already invalidated it. What remains is compaction: the trigger counts level-zero files and every commit forces a flush, so the base level is rewritten however little each commit wrote. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Compaction is unusually expensive when commits are retained. A checkpoint hard-links the files live when it was taken, so rewriting a file ends sharing for every checkpoint taken before it, and because keys are spread by hash a level-zero file spans the whole key space and overlaps everything below - making each compaction a rewrite of the entire base level. Worse, the trigger counts files rather than bytes, and creating a checkpoint flushes the memtable, so one level-zero file appears per commit however little it holds. At RocksDB's default of four, compaction therefore ran every four commits regardless of how much had been written, which is why storing only changed nodes did not help on its own. Raising the trigger makes the cadence coarser, and it helps reads too, so there is no trade to balance: compacting often does not keep level zero small, because flushes accumulate behind a compaction while it runs. Bloom filters pay for the extra files. Both column families are looked up by whole key and never scanned by range - blobs by content hash, values by their key - so a filter lets a lookup skip a file outright, where before it consulted the index of every file whose key range covered the key, which for hash-spread keys is most of them. What the filters contribute on their own is unmeasured. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Collection needs to know which commits are newer than the one it is asked to collect at. That cannot be read off the commits themselves: a commit id is the registry root hash, so it says what a state contains and nothing about when it was reached. A repository now records each committed root in a journal beside the manifests, as a sequence number that increases with every commit. Identity is untouched, so committing the same state twice still produces the same id and the same manifest; only the journal grows. Retention follows the recorded order rather than ancestry, which keeps states whose provenance is unclear - one committed without a parent is kept like any other. Collecting at a root keeps every root recorded at or after it, and a root committed more than once keeps its highest position, so re-committing an old state can only extend how long it survives. Entries are fixed width and appended, so recording a commit does not rewrite the journal and a write a crash cut short is recognisable by length alone. Such an entry is dropped when read. The journal is written after the manifest, so an interrupted commit leaves a manifest with no journal entry rather than a recorded root with nothing behind it - and since the caller never received that commit id, nothing can reference it and collection is free to reclaim it. Nothing is deleted yet. This is the shared prerequisite for collecting either half of a commit, and it sits on the repository trait so that the in-memory backend records commits the same way the on-disk one does. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fmv6auJua8dSB18iGy6rES
Collecting at a root drops the registry commits recorded before it, together with the database commits none of the surviving registry commits still reference. Database commits are content-addressed and therefore shared - unrelated registry commits name the same one whenever their states agree - so removal follows reachability rather than the order commits were made in. This reclaims the value side of a commit. A commit directory is a set of hard links, so unlinking the last directory referencing a file frees it, and values are keyed by user key, so overwrites become obsolete versions compaction discards. Merkle nodes are content-addressed and every version of a node is a distinct live key, so no amount of directory removal reaches them; that needs deletion of the node keys themselves and is the other half of the design. Collection is safe to interrupt and repeat. It prunes the journal to the retained roots, removes the manifests it dropped, then removes the unreachable database commits, in that order, so everything retained is intact at each point in between. Each step after the first enumerates what is present rather than what the journal says should be, so a round left unfinished is picked up by the next one. Pruning first is also what makes a stale target safe: a root an earlier round already collected past is no longer in the journal, so collecting at it is refused rather than taken as a floor that would retain commits whose data has gone. Collection must not overlap a commit or another collection, and that is the caller's responsibility for now. The operations sit on the repository trait, so the in-memory backend collects the same way the on-disk one does, and the space harness now drives the real implementation rather than a copy of it - the reclaim figures it reports measure what ships. The reachability logic the long test carried its own copy of is now the one function. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fmv6auJua8dSB18iGy6rES
Node bodies lived in a blob column family inside each database's own instance. A node reachable from several databases was stored once per database, and every commit directory hard-linked the files holding every node its database had ever written. Node data is the large majority of what a retained commit pins, so that cost was paid again for each commit kept. They now live in one store shared by the whole repository. Nodes are content-addressed, so the same node written from anywhere is the same key and is stored once. The store sits outside the per-commit checkpoints, so its compaction churn is paid once rather than once per retained commit, and it gives one place from which a node can be deleted - which no amount of directory removal can do, and which collecting the Merkle side will need. Measured at 20k keys over six commits: what the history pins falls from 13.0 to 3.9 MiB, all of it values and none of it Merkle; growth per commit from 0.7 to 0.1 MiB; sharing between consecutive commits from 82% to 93%; commit latency from 57 to 21 ms. The number of nodes written is unchanged at 14119, which is the signal that the same work is being done and merely stored once. Nodes reach storage through their own operations rather than through the blob ones. Those two had the same shape but want opposite things now: the PVM uses a persistence layer as its own blob store and persists it by checkpointing, so its blobs must stay in the instance, while nodes must leave it. Separating them leaves that use untouched. Since a commit directory no longer holds everything the state is made of, checking one out takes the repository as well as the path. Committing syncs the store before taking the checkpoint, so a commit never refers to a node that a crash could take with it. The long tests' failure artifacts carry a checkpoint of the store, without which they could only be replayed against the repository that produced them. RocksDB allows one writer per directory, so instances are shared per canonical path and a repository handle holds its store open for as long as it lives. Without that the store would close when the last database dropped and be reopened by the next, which races: a handle stops being findable when its last reference goes, but the directory lock it holds is released later. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fmv6auJua8dSB18iGy6rES
Collecting a node means knowing whether anything still holds it. Answering that by traversing every retained root downwards costs the whole live set on every round, which is the wrong shape: most nodes are not in question, and the ones that are, are asked about individually. As a node is stored, an edge is now recorded from each of its children to it, in a `refs` column family of the Merkle store keyed by `child || parent`. Child first, so every edge into one child is contiguous and its parents are found by walking forward from its hash. Empty trees are never stored, so none is pointed at. Reverse edges rather than reference counts, because they are idempotent: writing an edge twice, or removing it twice, leaves the same state, so a collection interrupted part-way can simply be run again. A count incremented twice after a crash is silently wrong for good. Each edge carries a stamp, being how recently its child was proven reachable from a root. Edges are written without one: the sequence number of the commit being made is not settled while its nodes are being written, and an absent stamp is below every floor, so it forces the walk that settles the question rather than claiming anything. Filling them in is the sweep's job, and neither exists yet. Edges piggyback on the dirty tracking that decides whether a node is written at all, so a node already in the store re-records nothing. Measured at 20k keys over six commits: 1.76 edges per node, adding 47.5% to the Merkle store. That is a real cost and worth revisiting - but it falls on the shared store rather than on retention, which is unchanged at 3.9 MiB of pinned history, all of it values. In the per-database layout the same edges would have been duplicated into every retained commit. Nothing reads them yet, so behaviour is unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fmv6auJua8dSB18iGy6rES
Removing commit directories can never reclaim node bodies. They are content-addressed, so every version of every node is a distinct live key that nothing deletes, and they live in a store shared by the repository rather than in any commit directory. Left alone that store grows with the total number of node writes over a repository's lifetime, not with the size of the state. Collecting at a root now deletes them, and the edges that mentioned them. Whether a node is still held is asked from the node upwards, by walking the reverse edges, rather than by traversing every retained root downwards. A stamp at or above the collection floor ends the walk at once: a retained root already holds that child. Otherwise the walk climbs, and what it learns is written back onto the edges that led to a live root, so the next collection reads the answer rather than walking for it. Answers are memoised within a round, negative ones too, since a dead node's children ask the same question next. A stamp is only ever written where the node is provably reachable from the stamped root. Nodes are immutable, so if a parent is reachable from a root and refers to a child, the child is reachable from that root permanently. A stale stamp costs a walk and an over-generous one retains garbage; neither can drop something live. There are no cycles to guard against, a node's hash being derived from its children's. A database commit is held by every registry commit naming it, so it is retained until the most recent of those goes; taking the highest sequence number is what stops an older reference from deciding it. Deleting a key only writes a tombstone, so collection compacts the store afterwards - otherwise what it reports as freed would still be on disk. Measured at 20k keys over six commits: 6772 nodes and 17233 edges removed, 1.7 MiB of node data, taking the repository from 17.6 to 14.9 MiB. The retained root still checks out with every key readable, which is the property the tests hold onto: subtrees shared between a dropped and a retained commit survive, because an unchanged subtree keeps its hash and the retained root still reaches it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fmv6auJua8dSB18iGy6rES
Committing puts a database's values in a commit directory and syncs the nodes they refer to, which is enough that a commit never outlives its nodes, but it leaves the store as the only copy of them. A full commit takes another: a checkpoint of the store into a numbered slot beside it. Recovery opens the most recent slot. A slot is made by a checkpoint, so it is self-contained and needs no write-ahead log, and hard-links rather than copies, so taking one costs almost nothing. Slots are numbered so that each name is fresh, and one is created by renaming a finished directory to a name that does not exist yet, which is atomic and cannot half-happen. A fixed name with a `CURRENT` pointer beside it would need the pointer swapped separately, and renaming over an existing directory is not available in any case: `rename(2)` fails with `ENOTEMPTY` on a non-empty target. A directory left by an interrupted full commit is not a slot and the next attempt replaces it. Taking one is the caller's to schedule, not something committing does. It bounds what a crash costs, and reaping the slot before it is what returns the disk that compaction has since made redundant: a slot's hard links hold the files that were live when it was taken, so the high-water mark is the live set plus about one full-commit period of garbage. That makes the full-commit cadence the reclamation cadence. Reaping never removes the last slot, so there is always something to recover from. The live store carries on unchanged across a full commit; whether to check anything out again is the caller's decision, not this layer's. Recovery copies a slot into place rather than moving it, so a recovery that fails part-way leaves the slot intact, and refuses to run over a store that is still there rather than mixing the two. It has to happen before anything opens the store, since files appearing underneath an open RocksDB instance are not its own. Leases, so that other processes can read a slot while reaping respects them, are not here yet - nothing outside this process reads one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fmv6auJua8dSB18iGy6rES
Three things a collection needs before anything else depends on it. Full commits are what other processes read, since the live store cannot be opened by a second writer, and reaping was free to remove one while a reader had it open. A reader now takes a lease on the slot it opens and reaping leaves a leased slot for a later round. Leases are `flock`s, so the kernel releases one held by a process that dies and a crashed reader cannot pin a slot forever, which a lockfile checked by hand would allow. Several readers may hold the same slot; only reaping needs it to itself, and it never waits for a reader. The lease is taken beside the slot rather than inside it, so the remover can hold it across the removal - which leaves a reader arriving at that moment able to take a lease on a slot that has just gone, so opening one can still fail. Collection is the longest thing the storage does, and a full commit or a reap should not wait for one. It can now be asked to stop, and both halves check often enough to stop promptly. Stopping is safe wherever it happens: what was removed stays removed, each step leaves everything retained intact, and every step enumerates what is present rather than what it expected, so the next round finishes what the last left. A round that stopped says so, and stops before the Merkle sweep rather than sweeping against a commit set that is still half collected. Reading a state while the nodes behind it are reclaimed used to be indistinguishable from an inconsistent store. A store now records that it has collected, and reports an absent node as collected rather than as missing commit data. The two call for different responses: missing commit data means the storage is inconsistent and there is nothing to be done, while a collected node means checking the state out again resolves it. A store cannot know which of its absent keys it deleted, so this says a collection is the likely explanation rather than a certain one, and only ever in a store that has collected. `libc` becomes a direct dependency for `flock`; it was already in the tree. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fmv6auJua8dSB18iGy6rES
A round considered every node in the store, whether or not there was any reason to think it dead. The stamps on the edges give an early exit inside the walk, but the walk was not the cost - the scan was, and it is paid in full however little there is to reclaim. Measured at 9.5 microseconds and 250 bytes per node in the store, which at a few hundred million entries is the better part of an hour and tens of gigabytes for a round that might free almost nothing. Each node is now listed under the commit it was last known to be held by, in a column family keyed by sequence number and hash, the number first and big-endian so that ordering by key is ordering by commit. A round reads only the entries below its floor. A node a retained commit wrote, or one an earlier round proved still held, sorts at or above the floor and is never looked at. A candidate that turns out to be live is relisted under the newest retained root, which lifts it out of the range future rounds scan until the floor passes that root. Candidates are read in batches and settled as they are reached, and the answers worked out for one batch are dropped with it, so a round holds a batch rather than a store. This needs the sequence number while the nodes are being written, so a commit now reserves it before writing rather than taking it afterwards. A commit that then fails leaves the number unused, which costs nothing: numbers order commits and need not be contiguous. Asked of the store rather than the repository, so a backend that records nothing can say so without callers knowing which kind of repository they hold. Measured at 200k keys, twenty commits, collecting five behind the tip: the round that catches up on a store nothing has collected before examines 297,461 nodes in 6.4 s, and a later round over five commits of churn examines 34,823 in 0.81 s. Across an eightfold range of store sizes the steady-state round grows 0.68 to 1.17 s - the tree-depth term, not the store - and the memory it needs does not move at all. Two things follow from this that did not before. Collection must run behind the tip: collecting at the newest commit leaves no gap between the floor and the newest retained root, so everything relisted falls below the next round's floor and the live set is examined again every time. And compaction, which the round does not force, is now the part that costs the store rather than the garbage, so reclaiming disk belongs on the full-commit cadence. A round that deletes anything also gives the store a new identity. A node is skipped at commit time when it records being in the store, and after a deletion that record is no longer evidence the body is there. Changing the identity makes every such record stop matching, so the next commit writes what it touches. A working state whose lineage a round had dropped would otherwise skip writing nodes the round had just deleted and commit something referring to bodies that are gone - and unlike a read, which errors, a commit would not have said so. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Deleting a node writes a tombstone; the bytes come back when compaction rewrites the files without it. That rewrite costs the whole store rather than the garbage and takes far longer than anything that would want to trigger it, so it is now a background task started on its own. Taking a full commit is a flush and a rename. Pairing it with a rewrite of the store would make the cheap operation as slow as the expensive one, so it does not, and neither does collecting. Starting a reclaim returns at once; a second request while one is running is a no-op rather than a queued rewrite, since two of them achieve nothing one does not. The rewrite no longer excludes RocksDB's own compaction, which it did by default. Disabling automatic compaction and waiting for what is already running, for the length of a full-store rewrite, stalls writes behind it to no purpose: the point is to reclaim disk, not to be the only thing doing so. Reads, writes and commits carry on throughout, which the test holds onto: a full commit and a further commit both go through while a reclaim is running, and the commit made during it checks out afterwards. The measurement harness still waits, because a run has to know what the rewrite freed in order to report it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fmv6auJua8dSB18iGy6rES
emturner
force-pushed
the
emturner@working-tip
branch
from
September 8, 2026 09:06
c927784 to
8fa5eee
Compare
emturner
force-pushed
the
emturner@disk-gc-fullstack
branch
from
September 8, 2026 09:06
8929ec2 to
0f555bb
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Why
How
Manually Testing
Regressions
Tasks for the Author