You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Aug 3, 2026. It is now read-only.
From the 2026-07-27 concurrency-fix series (patch 0010, "vacuum: persist row moves through CDC"). Vacuum on persisted tables is now correct — row moves flow through the CDC stream via the VacuumPersistence sink, so the on-disk indexes follow the moves and the event-id stream stays gapless (tests/persistence/vacuum.rs proves reload works, including secondary indexes) — but it is consistency-only.
The gap
EmptyDataVacuum::defragment frees pages in memory (stats.pages_freed), and the moved rows' bytes are persisted at their new links. Nothing ever shrinks or truncates the on-disk .wt.data file:
Pages emptied by vacuum remain in the data file with stale bytes; the indexes simply stop pointing at them.
SpaceData::save_batch_data (src/persistence/space/data.rs) only ever writes pages; there is no free-page bookkeeping on disk beyond SpaceInfoPage.empty_links_list.
After many vacuum cycles the file is dominated by dead pages.
So "vacuum" on a persisted table today = in-memory compaction + disk consistency, not disk space reclamation.
Suggested direction
Options, roughly in order of ambition:
Persist the freed-page set (extend SpaceInfoPage / TOC) so freed pages are reused by later save_batch_data page creation instead of appending — stops growth without shrinking.
Physical compaction: rewrite the tail of the data file into freed pages and truncate — needs link rewrites, i.e. the same CDC move machinery vacuum already uses, so VacuumPersistence::apply_move is the natural hook.
Offline compaction on load (cheapest to reason about; no concurrency).
Verification hints
tests/persistence/vacuum.rs::test_vacuum_on_persisted_table_survives_reload is the harness to extend: assert on the data file size (or page count) after vacuum + reload. Full suite runs take ~4s warm; 30-50 run loops are cheap for flake checks.
Context
From the 2026-07-27 concurrency-fix series (patch 0010, "vacuum: persist row moves through CDC"). Vacuum on persisted tables is now correct — row moves flow through the CDC stream via the
VacuumPersistencesink, so the on-disk indexes follow the moves and the event-id stream stays gapless (tests/persistence/vacuum.rsproves reload works, including secondary indexes) — but it is consistency-only.The gap
EmptyDataVacuum::defragmentfrees pages in memory (stats.pages_freed), and the moved rows' bytes are persisted at their new links. Nothing ever shrinks or truncates the on-disk.wt.datafile:SpaceData::save_batch_data(src/persistence/space/data.rs) only ever writes pages; there is no free-page bookkeeping on disk beyondSpaceInfoPage.empty_links_list.So "vacuum" on a persisted table today = in-memory compaction + disk consistency, not disk space reclamation.
Suggested direction
Options, roughly in order of ambition:
SpaceInfoPage/ TOC) so freed pages are reused by latersave_batch_datapage creation instead of appending — stops growth without shrinking.VacuumPersistence::apply_moveis the natural hook.Verification hints
tests/persistence/vacuum.rs::test_vacuum_on_persisted_table_survives_reloadis the harness to extend: assert on the data file size (or page count) after vacuum + reload. Full suite runs take ~4s warm; 30-50 run loops are cheap for flake checks.