Skip to content

fix: run the frame-record gap backfill on every role, at runtime - #620

Open
blacks1ne wants to merge 2 commits into
QuilibriumNetwork:v2.1.0.25from
blacks1ne:blacks1ne/backfill-below-the-lowest-stored-frame
Open

fix: run the frame-record gap backfill on every role, at runtime#620
blacks1ne wants to merge 2 commits into
QuilibriumNetwork:v2.1.0.25from
blacks1ne:blacks1ne/backfill-below-the-lowest-stored-frame

Conversation

@blacks1ne

@blacks1ne blacks1ne commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Base: 2b96656e

A non-archive never ran the frame-record gap backfill, and no node ran it after
bootstrap: run_all_gap_backfill sat behind if sync_archive_mode, spawned
once inside the bootstrap closure. Measured on a live mainnet regular — zero
gap scan / record-only lines in six hours.

The hole it would close is large and permanent. A wiped mainnet node holds
genesis at 244200, then whatever gossip delivers once it adopts the head (on
the node examined, 701050..=701741), and nothing in between — 456,849
absent frames
. Forward-fill only climbs from its cursor, so the poller cannot
close it, and a regular's storage attestation anchors ρ_N to an exact global
frame that must be present locally. A state jump, a failed jump plus gossip
head adoption, and a restart all reach this shape. The scan already reported
it; nothing ran the scan.

#587 / #609 are the empty-store entry into the same end state, fixed at the
poller in quil-rpc. Disjoint files, either can land first.

Changes

  • Every role, on a 10-minute loop. Bootstrap-only missed the runtime state
    jump entirely.
  • Chunked descent (512), descending, so each chunk's parent_selector
    anchor is the record the chunk above just filled. run_record_only_backfill
    materializes (lo..=hi) and fetches serially — fine for a 3-frame restart
    leftover, not for 456k.
  • Two floors. Genesis, because bootstrap_genesis writes that record on
    every node, making the range the scan reports beneath it fictional (244,199
    mainnet heights that never existed). And, on mainnet, the 2.1.0 flag day
    (669975): those frames were produced by pre-migration provers, so
    archive_frame_is_valid's genesis-prover allowlist drops every one — the
    archive serves the record, this node fetches it, then discards it locally.
  • Non-archives bound the descent to three epochs below a gap's TOP — the
    slots a leaf-root registration retains ({prev, current, next}). From the
    top, because the heights needed soonest are those just under the records the
    node already has.
  • Stop on filled == 0 && unresolved > 0, not filled == 0. A chunk the
    poller closed between the scan and the backfill also fills nothing, and
    stopping on that would abandon a fillable descent. Hence BackfillOutcome.
  • Unresolved heights are tallied by cause. A frame this node rejected and a
    frame no peer holds were one indistinguishable number. unrecoverable is now
    unresolved (grep-visible), and the scan logs intended_frames beside
    missing_frames.

RESEED_GAP_BACKFILL_ENABLED is false today because an unbounded backfill
over unservable heights produced attempted=4970, unrecoverable=4970 — which
is why the bound and the stop rule are load-bearing.

Field result

Overnight on a wiped mainnet non-archive: ~29k records recovered descending
698605 → 669422, then a permanent stall at 669975 — the flag day, exact. The
chunk straddling it split at 669976/669975, filling the 470 heights above and
none of the 42 below. It was reported as an archive-side gap; the archives were
serving those frames correctly.

Tests

Eight red before, green after. The headline-count test reports 2160 without
the fix and 0 with it — 2160 is exactly the window the reporting node
stalled in. -p quil-node -p quil-store -p quil-crypto 190/190.

Not tested

The role gating and the rescan loop — archive_sync.rs has no harness for that
bootstrap closure. The live-node observation above is what measures them.

A non-archive never ran the frame-record gap backfill at all, and no node ran
it after bootstrap. Verified on a live mainnet regular: zero `gap scan`,
`restart gap scan` or `record-only` lines in six hours of uptime. Both gates
did that — `run_all_gap_backfill` was behind `if sync_archive_mode`, and it was
spawned once inside the bootstrap closure.

That matters because the hole it would have closed is large and permanent. A
freshly-wiped mainnet node holds the genesis record at 244200, then whatever
gossip delivers once it adopts the head — on the node examined, a contiguous
block at 701050..=701741 — and nothing in between. 456,849 absent frames. The
poller cannot help: forward-fill only climbs from its cursor, which sits at the
head. A regular's app-shard storage attestation anchors rho_N to an exact
global frame that must be present locally, and below the cursor it is not.

The same shape is reached by several routes — a successful state jump (which
writes its target frame and nothing beneath it, at boot or via the runtime
far-behind rescue), a failed jump plus gossip head adoption, or a restart. The
scan already reported this as an internal hole. Nothing ever ran the scan.

Changes:

- The scan runs for every role, on a loop (`GAP_RESCAN_INTERVAL`, 10 min).
  Serving ranges is the archive's reason to want contiguity, not the only one;
  `forward_fill` is already enabled network-wide on the same argument.
  Bootstrap-only missed the runtime jump entirely.

- Gaps are descended in chunks of 512 rather than handed over whole.
  `run_record_only_backfill` materializes `(lo..=hi)` into a `Vec<u64>` and
  fetches serially — fine for a 3-frame restart leftover, not for 456k.
  Descending, because each chunk's `parent_selector` anchor is the record the
  chunk above it just filled.

- A chunk nobody can serve stops that gap's descent, since below the migration
  boundary every frame fails the genesis-prover allowlist. The signal is
  `filled == 0 && unresolved > 0`, not `filled == 0`: a chunk the poller closed
  between the scan and the backfill also fills nothing, and stopping on it
  would abandon a fillable descent. Hence `BackfillOutcome`.

- Non-archives bound the descent to three epochs below a gap's TOP —
  the slots a leaf-root registration vertex retains ({prev, current, next}),
  so every epoch an in-flight opening can be validated against. Measured from
  the top because the heights needed soonest are those just under the records
  the node already has. Without it a regular would serially fetch ~31k frames
  down to the migration boundary.

- `find_global_frame_record_gaps` also reports the hole below the store's
  lowest record, which it structurally could not see before (it emitted a range
  only between two iterated keys). The caller MUST clamp that to the network's
  genesis frame and now does: `bootstrap_genesis` writes the genesis record on
  every node, so `earliest` IS genesis and the reported floor range is entirely
  fictional — 244,199 mainnet heights that never existed. Unclamped it costs a
  chunk of pointless fetches at every startup and then blacklists itself.

Tests: six fail before this change and pass after — the scan reporting the
floor hole, that hole filled from local candidates, a descent spanning more
than one chunk, the depth bound applying to an internal gap measured from its
top, and the fictional sub-genesis range being discarded rather than fetched.
The rest pass on both sides, including the pair separating an already-complete
range (`{filled: 0, unresolved: 0}`) from an unservable one
(`{filled: 0, unresolved: 3}`). `-p quil-store -p quil-node` 131/131.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@blacks1ne
blacks1ne force-pushed the blacks1ne/backfill-below-the-lowest-stored-frame branch from 9492a76 to 2d9231b Compare August 18, 2026 22:54
@blacks1ne blacks1ne changed the title fix: backfill the frame records below the store's floor fix: run the frame-record gap backfill on every role, at runtime Aug 18, 2026
Field report on the parent commit, from a wiped mainnet non-archive: the
descent filled ~29k records down to the flag day, then stalled there and
never moved again.

Frames at or below GLOBAL_FLAG_DAY_LAST_LEGACY_FRAME (669975) were produced
by pre-migration provers, so step 1 of `archive_frame_is_valid` — the
genesis-prover allowlist — drops every one of them. Its own comment already
said so: "expected for legacy pre-migration frames". Archives serve those
records correctly; this node fetches them and then discards them locally.

The parent commit created the stall. Before it, only archives ran the scan,
and archives migrate from the flag-day head so they never descend below it.
Un-gating the scan for non-archives pointed the new 3-epoch depth cap at
[667816, 669975] — 2160 heights that are 100% unfillable by construction.

- Floor the descent at the flag day on mainnet, folded into the existing
  `floor_frame` so the clamp itself is unchanged. Other networks never
  rewound and keep genesis; the constant sits above every frame they have.
- Clamp before cap, via one shared helper, so a cap deeper than the distance
  to the floor cannot re-admit legacy heights.
- Report `intended_frames` next to `missing_frames`. The raw hole size is not
  the objective: on the reporting node it read 669,974 and never moved, of
  which 244,199 is the fictional sub-genesis range and 425,775 is the legacy
  range. The true objective was zero.
- Tally WHY heights went unresolved (invalid / unavailable / timed_out /
  store_failed / connect_failed). A locally-rejected frame and a frame no
  peer holds were one indistinguishable number, which sent the reporter
  hunting an archive-side gap that does not exist.
- Drop the "likely uncommitted/orphaned (correctly not canonical)" inference.
  It was written for the reseed path, which fetches above the canonical head;
  it is provably wrong for a gap-scan range, whose hi+1 is a present
  canonical record — the chain is contiguous by frame number, so heights
  beneath it cannot be orphaned. The log field `unrecoverable` is now
  `unresolved`.

Proven RED without the fix: the floor test reports 244200 vs 669976, and the
intent count reports 2160 vs 0 — 2160 being exactly the reporter's stalled
window (669975 − bounded_lo 667816 + 1).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant