WIP Shadow test ai#1803
Draft
lispc wants to merge 92 commits into
Draft
Conversation
…d in current code base)
…_block repair script - AGENTS.md: Add verifier wrapper deployment pitfalls - anvil_setCode does NOT reset immutables (wrong digests) - Do not 'fix' production Solidity without evidence - finalizeBundlePostEuclidV2 access control notes - estimategas.go: Fix Anvil eth_estimateGas failure - Strip GasFeeCap/GasTipCap from CallMsg before EstimateGas - Set high Gas limit to prevent Gas=0 rejection - shadow-testing docs: Update Sepolia traps and lessons learned - L1MessageQueueV2 slot 104 verification - Anvil gas estimation and sender balance traps - psql timeout vs TCP connectivity diagnosis - Add fix_l2_block_transactions.py script for repairing missing transaction data in shadow DB from L2 RPC.
…s, fix numbering Merge scripts/shadow-testing/ into tests/shadow-testing/: - Move fetch-l2-blocks.py to tests/shadow-testing/scripts/ - Move .env.example to tests/shadow-testing/.env.example - Remove redundant scripts/shadow-testing/ directory entirely (setup.sh, import-production-data.sh, configs/ superseded by Makefile targets and 00-import-bundle-range.sh) Deduplicate documentation: - README.md: Slim down to ~100 lines (entry + quick start + index) - docs/README.md → docs/GUIDE.md (full detailed guide) - docs/QUICKSTART.md: deleted (content merged into README.md) - AGENTS.md → docs/TROUBLESHOOTING.md (structured traps) - Update all internal links to point to new names Fix script numbering collision: - 02-deploy-verifier.sh → 03-deploy-verifier.sh - Shift all subsequent scripts (+1) - Update Makefile references and internal script calls Update root AGENTS.md cross-references.
- Add export RUST_MIN_STACK=16777216 to 04-prover-up.sh - Document the batch proof stack overflow issue in LESSONS_LEARNED.md - Chunk proofs work with default stack, but batch/bundle proofs need 16MB
…ndle 13451 - Document the root cause: Anvil fork's L1MessageQueueV2 messageRollingHashes were stale, causing messageQueueHash mismatch and VerificationFailed. - Add recovery steps: sync messageRollingHashes from production RPC, set nextCrossDomainMessageIndex/nextUnfinalizedQueueIndex correctly. - Document OnlyProver requirement for finalizeBundlePostEuclidV2. - Update pre-finalize checklist with L1MessageQueueV2 sync step. - Fix 03-deploy-verifier.sh EXISTING_START parsing. - Update relayer config: increase fusaka_timestamp, set max_batches=1. - Update 06-run-relayer.sh to use correct commit key.
Document the key finding that S3 digest_1.hex/digest_2.hex are stored in Montgomery form and must be converted to canonical form before deploying ZkEvmVerifierPostFeynman. - GUIDE.md: Add conversion script and deployment instructions - LESSONS_LEARNED.md: Update verifier mismatch section with S3 digest note - TROUBLESHOOTING.md: Update Trap 1 to include digest form mismatch
…ness lessons Add critical lessons learned from shadow proving bundles 13470-13474: - MVRV routing must match proof digests (Cause C of VerificationFailed) - bundle.batch_proofs_status must be Ready(2) before finalization Updates all 3 docs: - GUIDE.md: Add 'Verify MVRV Routing' and 'Verify bundle readiness' sections - LESSONS_LEARNED.md: Add 2 new dated entries with root cause analysis - TROUBLESHOOTING.md: Expand Trap 1 with MVRV routing verification steps
Co-authored-by: georgehao <hongfan@scroll.io>
…ternal to develop baseline
…SONS_LEARNED.md, update for v0.9.0 S3 paths and S3 digest extraction
…er to test infra, update docs for S3 digests - Deploy ZkEvmVerifierPostFeynman with S3 digests and protocolVersion=10 - Move shadow-only verifier wrapper out of scroll-contracts submodule into tests/shadow-testing/contracts/ - Update GUIDE/TROUBLESHOOTING/README: prefer S3 digest files, remove proof-extraction fallback - Prover-up script: asset detours for v0.9.0 S3 layout - fetch-l2-blocks: store full header/transactions for coordinator chunk-task generation
- libzkp: add openvm-sdk/openvm-stark-sdk, input_commits, batch_root_verifier_vk loading - prover-bin: adapt dumper and universal handler to v0.9.0 ProverConfig/StarkProof - common/message: introduce OpenVMStarkProof with UserPvsProof, switch chunk/batch to StarkProof - coordinator: update proof receiver, mock verifier and tests for StarkProof - rust-toolchain: bump to nightly-2025-11-20 + rust-src - regenerate Cargo.lock to resolve v0.9.0 dependency graph
- prover.rs: load child circuit handler and call enable_deferral for Batch/Bundle tasks - UniversalHandler: expose enable_deferral wrapper around Prover::enable_deferral - CircuitConfig: add child_circuit_vks for chunk/batch child lookup - 04-prover-up.sh: populate child_circuit_vks from openVmVk.json
…rifier, add v0.9.0 lessons
formatProverTask/applyUniversal/InsertProverTask failures after Update*Attempts previously refunded only active_attempts, permanently burning total_attempts. After SessionAttempts (5) such tasks became invisible to GetUnassigned* (total_attempts < maxAttempts) with no prover_task row, so the cron timeout checker could never mark them failed either — silent starvation (shadow-testing Trap 22/24). Add RefundAttemptsByHash (chunk/batch/bundle orm) and recoverAttempts: when the charge happened in this Assign call (hasAssignedTask == nil), refund both counters and reset proving_status to unassigned; re-polls of already-assigned tasks keep the old active-only recovery. Also cover the previously unrecovered fixCompatibility failure sites in batch/bundle assign.
- lib/: scripts and configs shared by both modes (anvil/verifier/prover/ relayer setup, sync-queue-hashes, anvil-utils) - follow/: follow-mode scripts, configs, Makefile, GUIDE, TROUBLESHOOTING (poll-sync follow of mainnet tip, finalize on Anvil fork) - snapshot/: snapshot-replay scripts, configs, Makefile, GUIDE, TROUBLESHOOTING (historical fork + fixed bundle range) - docs/COMMON-TROUBLESHOOTING.md: mode-independent traps; per-mode TROUBLESHOOTING files carry the mode-tagged traps verbatim - root Makefile becomes a thin dispatcher; README rewritten as mode chooser - 10-follow-up.sh: drop follow-run.env on --reset so a fresh run gets a fresh report window; assert postgres log_statement='mod' on every setup (also in 02-prepare-db.sh) for write attribution - update AGENTS.md references and .gitignore for per-mode config dirs
- l2_config.disable_l2_watcher: skip the L2 missing-blocks fetch loop. In shadow mode the DB is populated by import/poll-sync; an empty l2_block table previously made the watcher crawl from genesis (shadow Trap 26). Default false keeps production behavior. - sender_config.chain_nonce_only: initialize the sender nonce from the chain pending nonce only, ignoring pending_transaction rows, which are not chain/fork-scoped and poisoned nonces on fresh forks (shadow Traps 7/27). Default false keeps max(db+1, chain) behavior. - estimategas: set an explicit 30M gas cap in the estimation CallMsg instead of 0; Anvil rejects fee-capped eth_estimateGas with Gas=0 (shadow Trap 9). The cap only bounds the binary search, estimates are unchanged. - l2_relayer: move full-calldata dumps on commit/finalize send errors from Error to Debug (was flooding logs every retry tick), and log a loud Error when a bundle is marked RollupFinalizeFailed — status 7 is never retried by ProcessPendingBundles and needs manual reset.
- relayer.json.template: disable_l2_watcher + chain_nonce_only - 10-follow-up.sh --reset: TRUNCATE pending_transaction (stale rows poison sender nonces and can replay old calldata on a fresh fork) - mark Traps 26/7/9 as fixed upstream; document the new config knobs in follow/GUIDE.md
…hase - 10-follow-up.sh: FOLLOW_FORK_HOURS_BACK (default 5) — fork at tip - hours*300 blocks so the run opens with a real backlog (committed-but-not-finalized at fork + everything produced since); lastFinalized/lastCommitted now read AT the fork block; 0 = old fork-at-tip behavior. follow-run.env records FORK_HOURS_BACK and MAINNET_BUNDLE_TIP_AT_START. - generate-catchup-report.py: split report into catch-up phase (fork finalized index climbing to the recorded mainnet bundle tip) and steady-state phase; use the max_finalized_bundle watermark for finalized counts; backward-compatible with old env files. - docs: follow README/GUIDE + AGENTS.md describe catch-up-then-follow.
…over The impersonated eth_sendTransaction for updateVerifier could still be in the txpool when addProver signed with the same owner nonce, failing with 'replacement transaction underpriced' (hit twice at bring-up). Poll for the receipt before continuing.
…rover in follow-up (traps 28/29)
… blob-base-fee); accept Fusaka-era fee floor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.