fix: run the frame-record gap backfill on every role, at runtime - #620
Open
blacks1ne wants to merge 2 commits into
Open
Conversation
A non-archive never ran the frame-record gap backfill at all, and no node ran
it after bootstrap. Verified on a live mainnet regular: zero `gap scan`,
`restart gap scan` or `record-only` lines in six hours of uptime. Both gates
did that — `run_all_gap_backfill` was behind `if sync_archive_mode`, and it was
spawned once inside the bootstrap closure.
That matters because the hole it would have closed is large and permanent. A
freshly-wiped mainnet node holds the genesis record at 244200, then whatever
gossip delivers once it adopts the head — on the node examined, a contiguous
block at 701050..=701741 — and nothing in between. 456,849 absent frames. The
poller cannot help: forward-fill only climbs from its cursor, which sits at the
head. A regular's app-shard storage attestation anchors rho_N to an exact
global frame that must be present locally, and below the cursor it is not.
The same shape is reached by several routes — a successful state jump (which
writes its target frame and nothing beneath it, at boot or via the runtime
far-behind rescue), a failed jump plus gossip head adoption, or a restart. The
scan already reported this as an internal hole. Nothing ever ran the scan.
Changes:
- The scan runs for every role, on a loop (`GAP_RESCAN_INTERVAL`, 10 min).
Serving ranges is the archive's reason to want contiguity, not the only one;
`forward_fill` is already enabled network-wide on the same argument.
Bootstrap-only missed the runtime jump entirely.
- Gaps are descended in chunks of 512 rather than handed over whole.
`run_record_only_backfill` materializes `(lo..=hi)` into a `Vec<u64>` and
fetches serially — fine for a 3-frame restart leftover, not for 456k.
Descending, because each chunk's `parent_selector` anchor is the record the
chunk above it just filled.
- A chunk nobody can serve stops that gap's descent, since below the migration
boundary every frame fails the genesis-prover allowlist. The signal is
`filled == 0 && unresolved > 0`, not `filled == 0`: a chunk the poller closed
between the scan and the backfill also fills nothing, and stopping on it
would abandon a fillable descent. Hence `BackfillOutcome`.
- Non-archives bound the descent to three epochs below a gap's TOP —
the slots a leaf-root registration vertex retains ({prev, current, next}),
so every epoch an in-flight opening can be validated against. Measured from
the top because the heights needed soonest are those just under the records
the node already has. Without it a regular would serially fetch ~31k frames
down to the migration boundary.
- `find_global_frame_record_gaps` also reports the hole below the store's
lowest record, which it structurally could not see before (it emitted a range
only between two iterated keys). The caller MUST clamp that to the network's
genesis frame and now does: `bootstrap_genesis` writes the genesis record on
every node, so `earliest` IS genesis and the reported floor range is entirely
fictional — 244,199 mainnet heights that never existed. Unclamped it costs a
chunk of pointless fetches at every startup and then blacklists itself.
Tests: six fail before this change and pass after — the scan reporting the
floor hole, that hole filled from local candidates, a descent spanning more
than one chunk, the depth bound applying to an internal gap measured from its
top, and the fictional sub-genesis range being discarded rather than fetched.
The rest pass on both sides, including the pair separating an already-complete
range (`{filled: 0, unresolved: 0}`) from an unservable one
(`{filled: 0, unresolved: 3}`). `-p quil-store -p quil-node` 131/131.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
blacks1ne
force-pushed
the
blacks1ne/backfill-below-the-lowest-stored-frame
branch
from
August 18, 2026 22:54
9492a76 to
2d9231b
Compare
Field report on the parent commit, from a wiped mainnet non-archive: the descent filled ~29k records down to the flag day, then stalled there and never moved again. Frames at or below GLOBAL_FLAG_DAY_LAST_LEGACY_FRAME (669975) were produced by pre-migration provers, so step 1 of `archive_frame_is_valid` — the genesis-prover allowlist — drops every one of them. Its own comment already said so: "expected for legacy pre-migration frames". Archives serve those records correctly; this node fetches them and then discards them locally. The parent commit created the stall. Before it, only archives ran the scan, and archives migrate from the flag-day head so they never descend below it. Un-gating the scan for non-archives pointed the new 3-epoch depth cap at [667816, 669975] — 2160 heights that are 100% unfillable by construction. - Floor the descent at the flag day on mainnet, folded into the existing `floor_frame` so the clamp itself is unchanged. Other networks never rewound and keep genesis; the constant sits above every frame they have. - Clamp before cap, via one shared helper, so a cap deeper than the distance to the floor cannot re-admit legacy heights. - Report `intended_frames` next to `missing_frames`. The raw hole size is not the objective: on the reporting node it read 669,974 and never moved, of which 244,199 is the fictional sub-genesis range and 425,775 is the legacy range. The true objective was zero. - Tally WHY heights went unresolved (invalid / unavailable / timed_out / store_failed / connect_failed). A locally-rejected frame and a frame no peer holds were one indistinguishable number, which sent the reporter hunting an archive-side gap that does not exist. - Drop the "likely uncommitted/orphaned (correctly not canonical)" inference. It was written for the reseed path, which fetches above the canonical head; it is provably wrong for a gap-scan range, whose hi+1 is a present canonical record — the chain is contiguous by frame number, so heights beneath it cannot be orphaned. The log field `unrecoverable` is now `unresolved`. Proven RED without the fix: the floor test reports 244200 vs 669976, and the intent count reports 2160 vs 0 — 2160 being exactly the reporter's stalled window (669975 − bounded_lo 667816 + 1). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Base:
2b96656eA non-archive never ran the frame-record gap backfill, and no node ran it after
bootstrap:
run_all_gap_backfillsat behindif sync_archive_mode, spawnedonce inside the bootstrap closure. Measured on a live mainnet regular — zero
gap scan/record-onlylines in six hours.The hole it would close is large and permanent. A wiped mainnet node holds
genesis at 244200, then whatever gossip delivers once it adopts the head (on
the node examined,
701050..=701741), and nothing in between — 456,849absent frames. Forward-fill only climbs from its cursor, so the poller cannot
close it, and a regular's storage attestation anchors ρ_N to an exact global
frame that must be present locally. A state jump, a failed jump plus gossip
head adoption, and a restart all reach this shape. The scan already reported
it; nothing ran the scan.
#587 / #609 are the empty-store entry into the same end state, fixed at the
poller in
quil-rpc. Disjoint files, either can land first.Changes
jump entirely.
parent_selectoranchor is the record the chunk above just filled.
run_record_only_backfillmaterializes
(lo..=hi)and fetches serially — fine for a 3-frame restartleftover, not for 456k.
bootstrap_genesiswrites that record onevery node, making the range the scan reports beneath it fictional (244,199
mainnet heights that never existed). And, on mainnet, the 2.1.0 flag day
(669975): those frames were produced by pre-migration provers, so
archive_frame_is_valid's genesis-prover allowlist drops every one — thearchive serves the record, this node fetches it, then discards it locally.
slots a leaf-root registration retains (
{prev, current, next}). From thetop, because the heights needed soonest are those just under the records the
node already has.
filled == 0 && unresolved > 0, notfilled == 0. A chunk thepoller closed between the scan and the backfill also fills nothing, and
stopping on that would abandon a fillable descent. Hence
BackfillOutcome.frame no peer holds were one indistinguishable number.
unrecoverableis nowunresolved(grep-visible), and the scan logsintended_framesbesidemissing_frames.RESEED_GAP_BACKFILL_ENABLEDisfalsetoday because an unbounded backfillover unservable heights produced
attempted=4970, unrecoverable=4970— whichis why the bound and the stop rule are load-bearing.
Field result
Overnight on a wiped mainnet non-archive: ~29k records recovered descending
698605 → 669422, then a permanent stall at 669975 — the flag day, exact. The
chunk straddling it split at 669976/669975, filling the 470 heights above and
none of the 42 below. It was reported as an archive-side gap; the archives were
serving those frames correctly.
Tests
Eight red before, green after. The headline-count test reports
2160withoutthe fix and
0with it —2160is exactly the window the reporting nodestalled in.
-p quil-node -p quil-store -p quil-crypto190/190.Not tested
The role gating and the rescan loop —
archive_sync.rshas no harness for thatbootstrap closure. The live-node observation above is what measures them.