feat(nds): garbage collect commit directories older than a given registry commit

[emturner]

Part of TZX-209

What

Record the order of commits made, in an append only journal. Then allow for commits to be collected that are older than a given root (and not referenced since). Finally, bound the registry long test with that collection, in place of the copy of it the test kept.

Why

octez-smart-rollup-node needs the storage to reclaim disk, and today it reclaims
nothing: a repository grows with the total number of commits over its lifetime.

Currently, a commit id is just the registry root hash, so contains no information about when the commit occurred. Two unrelated commits could share the same id, and ancestry is unavailable.

Simply removing directories removes a large proportion of what a repository holds. As commit directories are sets of hard links, removing the last directory containing a given hard link frees it. This is sufficient for garbage collection of the 'key-value' column family.

How

Commit Journal

Each committed root is recorded as a sequence number that increases with
every commit. Identity is untouched, so committing the same state twice still produces
the same id and the same manifest; only the journal grows. Retention follows the recorded
order rather than ancestry, which keeps states whose provenance is unclear. Collecting at
a root keeps every root recorded at or after it, and a root committed more than once
keeps its highest position, so re-committing an old state can only extend how long it
survives. The target is read the other way about: its earliest position is the floor,
so collecting at a state that has been committed twice drops only what preceded the first
time it was reached, and never the states committed in between. Both directions err
towards keeping a state.

Collection

Database commits are content-addressed and therefore shared — unrelated
registry commits name the same one whenever their states agree — so their removal follows
reachability rather than the order commits were made in.

It is safe to interrupt and repeat. It prunes the journal to the retained roots, removes
the manifests it dropped, then removes the unreachable database commits, in that order, so
everything retained is intact at each point in between. Each step after the first
enumerates what is present rather than what the journal says should be, so a round left
unfinished is picked up by the next one.

Both sit on the repository trait, so the in-memory backend records and collects the same
way the on-disk one does.

For now, collection must not overlap a commit or another collection; that is the caller's
responsibility for now, and is stated in the docs page.

Long Test Retention

The registry long test advances a shared base once per epoch, then runs a proptest of many
cases against that base. A case checks the base out and applies operations with proofs; it
never commits. Only the epoch advance does. So the only commits a run records are epoch
bases, in the order the epochs ran.

Bounding the repository over a long run therefore means dropping old epoch bases, and the
test used to do that with its own copy of collection: a window of recent bases, and on
eviction a reachability walk over the retained manifests, then database directory and
manifest removals, mirrored by hand onto both backends. That is the mechanism this PR
ships, written a second time, so the retention a run exercised was never quite the
retention that ships — and the two could drift apart without anything noticing.

Now the window is all the test keeps. It trims to the keep most recent bases and calls
collect() at the oldest one still in the window, once per backend. Retention by recorded
order coincides with a window over epoch bases exactly because nothing between them is ever
recorded, so "every root recorded at or after this one" and "the bases still in the window"
are the same set.

Three things follow from the swap. A long randomised run now exercises the real
collect(), including the journal pruning the copy never touched. The bound is tighter,
because the copy only ever considered the database commits of the base it was evicting,
leaving anything else unreachable on disk. And it is the only place the in-memory
repository's side of collection runs at all — see the note under Manually Testing.

Ordering is safe: collection must not overlap a commit, and the call sits between epochs,
after the proptest for that epoch has finished, so nothing else is touching the repository.

Manually Testing

make -C durable-storage check
make -C durable-storage test
cargo nextest run -p octez-riscv-durable-storage

The space harness now drives the real collect() rather than a copy of it, so the
reclaim figures it reports measure what ships:

cargo run --features unstable-test-utils --bin gc_space -- --keys 200000 --commits 20 --collect

So does the registry long test, on both backends:

make -C durable-storage run-registry-long-test

That run is also the only coverage the in-memory repository has for its side of collection.
The collect tests reach the trait through DirectoryManager, and the crate's two test
flavours are the same build in this respect — the self dev-dependency takes default
features, and default = ["rocksdb"], so --no-default-features does not turn rocksdb off
for the test binary — which left nothing running InMemoryRepo's journal and removal
methods.

Regressions

None. Nothing is deleted by the first commit, and the second only removes commit
directories and manifests that are no longer retained.

Tasks for the Author

• Link all Linear issues related to this MR using magic words (e.g. part of, relates to, closes).
• Eliminate dead code and other spurious artefacts introduced in your changes.
• Document new public functions, methods and types.
• Make sure the documentation for updated functions, methods, and types is correct.
• Add tests for bugs that have been fixed.
• Explain changes to regression test captures when applicable.
• Write commit messages in agreement with our guidelines.
• Self-review your changes to ensure they are high-quality.
• Complete all of the above before assigning this MR to reviewers.

[github-actions]

Durable storage space metrics for revision c663de3, at pull request
scale. The scale itself lives in durable-storage/Makefile.

Metric	Value	Unit
dead_node_bytes_per_commit	1.48	MiB
dead_node_fraction	37.4	%
repo_bytes_per_commit	15.17	MiB
checkpoint_shared_fraction	76.8	%
checkpoint_shared_fraction_steady	91.6	%

Full report

repository: /tmp/nix-shell.1WstFv/.tmpnstU4i
shape: 1 database(s) x 100000 keys x 32 B values, 10 commits x 1000 modified keys
prepopulated 100000 keys in 0.6s
committing the base state...
base state committed as bbcc53be48c9f9285a3607046839524e6d2152561d2e95d9e760e6a665d72d44 in 1.0s

 commit    blob MiB    live MiB    dead MiB   dead %   value MiB    disk MiB  commit ms  commits MiB     new MiB  shared %  rewrote MiB
      0        24.8        24.8         0.0     0.0%        15.4        46.1          0         42.0           -         -            -
      1        26.3        24.8         1.5     5.6%        15.4        47.8         91         43.7         1.7     96.1%          0.0
      2        27.7        24.8         2.9    10.6%        15.4        50.6         97         46.5         2.7     94.1%          0.0
      3        29.2        24.8         4.4    15.2%        15.4       100.7        119         50.1         3.6     92.8%          0.0
      4        30.7        24.8         5.9    19.2%        15.4       105.2        120        101.1        50.9      0.0%         50.0
      5        32.2        24.8         7.4    22.9%        15.4       110.3        135        106.1         5.1     90.9%          0.0
      6        33.7        24.8         8.9    26.3%        15.4       116.0        152        111.9         5.7     90.8%          0.0
      7        35.1        24.8        10.3    29.4%        15.4       159.2        167        118.6         6.7     90.1%          0.5
      8        36.6        24.8        11.8    32.3%        15.4       166.1        168        161.9        43.3     28.5%         50.7
      9        38.1        24.8        13.3    34.9%        15.4       173.4        185        169.2         7.3     89.2%          0.0
     10        39.6        24.8        14.8    37.4%        15.4       197.8        200        177.0         7.8     89.7%          0.0

over 10 commit(s):
  dead node data grew by 14.8 MiB, 1.48 MiB per commit
  blob column family grew by 14.8 MiB, 1.48 MiB per commit
  59689 node(s) written, 5968 per commit, averaging 260 B stored per node
  history pins 176.9 MiB (141.5 MiB Merkle, 35.3 MiB values, 0.1 MiB other), growing 13.5 MiB per commit
  79.9% of what the history pins is Merkle node data, which a shared store would stop duplicating per commit
  sharing: 76.8% of the bytes checkpoints pinned were already on disk (445.6 MiB of 580.5 MiB), adding 13.5 MiB in 2 new file(s) per commit
  3 of 10 measured commit(s) followed a compaction; without them sharing is 91.6%, so compaction costs 14.8 points of it, rewriting 101.2 MiB the earlier checkpoints still pin
  levels at the last commit:
    L0: 7 file(s), 22.5 MiB
    L6: 2 file(s), 53.0 MiB
  37.4% of the blob column family is now dead (14.8 MiB of 39.6 MiB)
  repository occupies 197.8 MiB across 11 commit directories (697.2 MiB apparent, and hard links save 503.7 MiB)

directory-level collection: removed 9 database commit(s) and 9 manifest(s)
  repository went from 197.8 MiB to 138.4 MiB, freeing 59.4 MiB
  retained history now occupies 117.6 MiB
  still dead and now unreachable by any directory deletion: 14.8 MiB of node data (37.4% of the surviving blob column family)

These figures are for catching a regression before it merges. Compare them with
the longer run on the default branch, which measures more commits.

[codecov]

Codecov Report

❌ Patch coverage is 91.24088% with 48 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.54%. Comparing base (48db913) to head (279fe69).

Files with missing lines	Patch %	Lines
durable-storage/src/repo.rs	88.52%	8 Missing and 13 partials ⚠️
durable-storage/src/collect.rs	93.63%	2 Missing and 8 partials ⚠️
durable-storage/src/storage/in_memory.rs	91.37%	0 Missing and 5 partials ⚠️
durable-storage/src/gc_space/scenario.rs	62.50%	0 Missing and 3 partials ⚠️
durable-storage/src/long_test/registry/gc.rs	66.66%	1 Missing and 2 partials ⚠️
durable-storage/src/long_test/registry/mod.rs	50.00%	0 Missing and 2 partials ⚠️
durable-storage/src/registry.rs	77.77%	0 Missing and 2 partials ⚠️
durable-storage/src/gc_space/prune.rs	75.00%	0 Missing and 1 partial ⚠️
durable-storage/src/journal.rs	99.13%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1207      +/-   ##
==========================================
+ Coverage   90.44%   90.54%   +0.09%     
==========================================
  Files         153      155       +2     
  Lines       32995    33427     +432     
  Branches    32995    33427     +432     
==========================================
+ Hits        29843    30266     +423     
+ Misses       2152     2150       -2     
- Partials     1000     1011      +11     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

Benchmark results for revision c663de3:

Metric	Duration	TPS
Mean	1.478014689s	27.064
Worst	1.485169533s	26.933
Best	1.470477489s	27.202
Standard Deviation	±4.786979ms	±0.088

Full results

Run	Transfers	Duration	TPS
1	40	1.48493886s	26.937
2	40	1.482809691s	26.976
3	40	1.485169533s	26.933
4	40	1.483821665s	26.957
5	40	1.475937936s	27.101
6	40	1.484751871s	26.941
7	40	1.47367073s	27.143
8	40	1.472696552s	27.161
9	40	1.470477489s	27.202
10	40	1.478936971s	27.046
11	40	1.484431473s	26.946
12	40	1.477230342s	27.078
13	40	1.480300278s	27.022
14	40	1.47704588s	27.081
15	40	1.474031858s	27.136
16	40	1.470551998s	27.201
17	40	1.477546167s	27.072
18	40	1.476797584s	27.086
19	40	1.475807701s	27.104
20	40	1.473339204s	27.149

Compare the results above with those for the default branch.