fix(infra): avoid huge warp backtraces in aggregator and relay logs - #3444
Open
jpraynaud wants to merge 3 commits into
Open
fix(infra): avoid huge warp backtraces in aggregator and relay logs#3444jpraynaud wants to merge 3 commits into
jpraynaud wants to merge 3 commits into
Conversation
jpraynaud
force-pushed
the
jpraynaud/fix-warp-huge-backtraces
branch
from
July 30, 2026 15:21
464f506 to
1d55528
Compare
jpraynaud
force-pushed
the
jpraynaud/fix-warp-huge-backtraces
branch
from
July 30, 2026 15:39
1d55528 to
a4a4499
Compare
jpraynaud
marked this pull request as ready for review
July 30, 2026 15:39
There was a problem hiding this comment.
Pull request overview
This PR addresses multi-megabyte log entries caused by anyhow capturing extremely large warp route-tree backtraces (notably since Rust 1.97 / v0 symbol mangling), by disabling library backtraces in the affected container environments while preserving panic backtraces.
Changes:
- Bump infra asset version from
0.5.17to0.5.18. - Add
RUST_LIB_BACKTRACE=0to aggregator and relay-related docker-compose environments to preventanyhowbacktrace capture. - Apply the same
RUST_LIB_BACKTRACE=0setting to the signer relay docker-compose override as well.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| mithril-infra/assets/infra.version | Bumps infra asset version to reflect the rollout of the backtrace logging mitigation. |
| mithril-infra/assets/docker/docker-compose-signer-p2p-base-no-dmq-override.yaml | Disables library backtraces for the signer relay container via environment variable. |
| mithril-infra/assets/docker/docker-compose-aggregator-p2p-base-override.yaml | Disables library backtraces for the aggregator relay container via environment variable. |
| mithril-infra/assets/docker/docker-compose-aggregator-base.yaml | Disables library backtraces for the aggregator container via environment variable. |
Set RUST_LIB_BACKTRACE=0 so anyhow errors logged by spawned nodes do not embed full backtraces.
- 'mithril-infra' from '0.5.17' to '0.5.18' - 'mithril-end-to-end' from '0.5.7' to '0.5.8'.
jpraynaud
force-pushed
the
jpraynaud/fix-warp-huge-backtraces
branch
from
July 30, 2026 16:39
a4a4499 to
d3f4c07
Compare
jpraynaud
temporarily deployed
to
testing-preview
July 30, 2026 16:57 — with
GitHub Actions
Inactive
jpraynaud
temporarily deployed
to
testing-2-preview
July 30, 2026 16:57 — with
GitHub Actions
Inactive
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Content
This PR includes a fix for multi-megabyte log entries emitted by the aggregator and relays when an error is logged from a
warproute.Since Rust
1.97.0, thev0symbol mangling scheme is the default: backtrace frames now embed the fully monomorphized type of warp's route tree, inflating each frame to ~65 KB. WithRUST_BACKTRACE=1,anyhowcaptures such a backtrace on every error, producing log entries of over 2 MB.RUST_LIB_BACKTRACE=0to the aggregator and relay docker compose files: this disablesanyhowerror backtraces while keeping full panic backtraces fromRUST_BACKTRACE=1anyhowcontext chainwarp(only the signer test relay is updated)RUST_LIB_BACKTRACE=0to the end to end testPre-submit checklist
Issue(s)
Relates to #3271