Skip to content

Documentation gap: api re-genesis runbook (Railway Postgres wipe + indexer cursor reset) #16

Description

@proofmancer

Context

When ligate-chain went through the v0.2.0 re-genesis on 2026-05-17, we discovered that the existing chain re-genesis runbook (ligate-chain/docs/development/runbooks/chain-id-bump.md) doesn't cover the api / indexer Postgres reset.

The chain wipe → restart → bootstrap-cli sequence worked, but the api kept serving stale state because:

  1. The api's attestations table TRUNCATE migration only covers ONE table; the other 6 (slots, transactions, attestor_sets, schemas, address_summaries, indexer_state) keep their pre-wipe rows.
  2. The api's indexer cursor (indexer_state.last_indexed_height) cached the pre-wipe height. After the chain re-genesised to height 0, the api still thought it was at height 30018 and stopped ingesting.
  3. Required intervention: manual TRUNCATE on all 7 tables + Railway redeploy to clear the in-memory cursor.

What's needed in docs.ligate.io

A new operator runbook section (probably under nodes.mdx or a new operators/re-genesis.mdx) covering:

  1. When re-genesis happens (wire-format break, chain_hash change, state corruption recovery).
  2. What the chain side does (link to ligate-chain runbook).
  3. What the operator must ALSO do on the api side:
    • psql $DATABASE_URL -c "TRUNCATE TABLE slots, transactions, attestor_sets, schemas, attestations, address_summaries, indexer_state CASCADE;"
    • railway redeploy --service ligate-api to clear the in-memory cursor
    • Verify /v1/info shows indexer_height: 0 then growing
  4. Optional: if you want a TRULY clean restart (not re-process historical Celestia blobs), advance genesis_da_height in devnet-1/celestia.toml past the prior chain's blob range, OR rotate to a new DA namespace.

Related

  • chain re-genesis 2026-05-17 (we're tracking this in the ligate-chain repo's chain-id-bump.md runbook as a sibling section)
  • Audit-fix PR for docs: ligate-io/docs#13

Priority

Important but not launch-blocking. File now so a future operator (or v0.3.0 re-genesis) doesn't relive this debugging session.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions