Skip to content

ci: make Nethermind E2E failures actionable - #3

Merged
davidtaikocha merged 4 commits into
masterfrom
codex/harden-nethermind-e2e
Sep 3, 2026
Merged

ci: make Nethermind E2E failures actionable#3
davidtaikocha merged 4 commits into
masterfrom
codex/harden-nethermind-e2e

Conversation

@davidtaikocha

Copy link
Copy Markdown
Member

🧩 What was the bug?

The Nethermind Shasta E2E workflow treated any L2 JSON-RPC response as readiness. In the failed run, block production remained at genesis, so the first transaction timed out and the rest of the suite cascaded into misleading nonce errors. The workflow artifact only contained pytest output, leaving the service-level cause hidden.

Failed run: https://github.com/taikoxyz/Catalyst/actions/runs/33500814591/job/99833478483


🔧 What’s been fixed?

  • Pin the external Nethermind devnet checkout to commit 841f041fcfe5248351530cda6874f4fc4bc09fcd.
  • Validate both L2 RPC response payloads using bounded requests.
  • Run RPC, transaction, and cross-node propagation tests as a 20-minute smoke gate with --maxfail=1.
  • Stop immediately if the smoke gate fails, then run the remaining suite without duplicate tests only after it succeeds.
  • Capture Docker state and logs for all relevant services within a five-minute diagnostic step.
  • Upload pytest and diagnostic logs and report the real job result.
  • Add regression tests for the workflow helpers and run them plus actionlint in CI.

💬 Anything else reviewers should know?

The devnet sets PRECONF_MIN_TXS=1, so passive block-number growth is not a valid readiness requirement. The smoke gate deliberately submits transactions through the existing E2E tests.

This PR makes the failure precise, bounded, and observable. It does not claim the external block-production stall itself is resolved until the full hosted E2E workflow reruns with the new diagnostics.


✅ Checklist

  • Existing tests pass
  • New regression tests added
  • Workflows validated with actionlint

@davidtaikocha
davidtaikocha merged commit d461fbf into master Sep 3, 2026
6 checks passed
@davidtaikocha
davidtaikocha deleted the codex/harden-nethermind-e2e branch September 3, 2026 01:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant