Skip to content

fix(infra): avoid huge warp backtraces in aggregator and relay logs - #3444

Open
jpraynaud wants to merge 3 commits into
mainfrom
jpraynaud/fix-warp-huge-backtraces
Open

fix(infra): avoid huge warp backtraces in aggregator and relay logs#3444
jpraynaud wants to merge 3 commits into
mainfrom
jpraynaud/fix-warp-huge-backtraces

Conversation

@jpraynaud

@jpraynaud jpraynaud commented Jul 30, 2026

Copy link
Copy Markdown
Member

Content

This PR includes a fix for multi-megabyte log entries emitted by the aggregator and relays when an error is logged from a warp route.

Since Rust 1.97.0, the v0 symbol mangling scheme is the default: backtrace frames now embed the fully monomorphized type of warp's route tree, inflating each frame to ~65 KB. With RUST_BACKTRACE=1, anyhow captures such a backtrace on every error, producing log entries of over 2 MB.

  • Add RUST_LIB_BACKTRACE=0 to the aggregator and relay docker compose files: this disables anyhow error backtraces while keeping full panic backtraces from RUST_BACKTRACE=1
  • Error logs keep the anyhow context chain
  • The signer is left unchanged as it does not embed warp (only the signer test relay is updated)
  • Add RUST_LIB_BACKTRACE=0 to the end to end test

Pre-submit checklist

  • Branch
    • Crates versions are updated (if relevant)
    • Commit sequence broadly makes sense
    • Key commits have useful messages
  • PR
    • All check jobs of the CI have succeeded
    • Self-reviewed the diff
    • Useful pull request description
    • Reviewer requested

Issue(s)

Relates to #3271

@jpraynaud jpraynaud self-assigned this Jul 30, 2026
@jpraynaud
jpraynaud force-pushed the jpraynaud/fix-warp-huge-backtraces branch from 464f506 to 1d55528 Compare July 30, 2026 15:21
@github-actions

github-actions Bot commented Jul 30, 2026

Copy link
Copy Markdown

Test Results

     5 files  ±0     209 suites  ±0   59m 29s ⏱️ + 4m 26s
 3 330 tests ±0   3 330 ✅ ±0  0 💤 ±0  0 ❌ ±0 
11 090 runs  ±0  11 090 ✅ ±0  0 💤 ±0  0 ❌ ±0 

Results for commit d3f4c07. ± Comparison against base commit bf1c9b3.

♻️ This comment has been updated with latest results.

@jpraynaud
jpraynaud force-pushed the jpraynaud/fix-warp-huge-backtraces branch from 1d55528 to a4a4499 Compare July 30, 2026 15:39
@jpraynaud
jpraynaud marked this pull request as ready for review July 30, 2026 15:39
Copilot AI review requested due to automatic review settings July 30, 2026 15:39

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR addresses multi-megabyte log entries caused by anyhow capturing extremely large warp route-tree backtraces (notably since Rust 1.97 / v0 symbol mangling), by disabling library backtraces in the affected container environments while preserving panic backtraces.

Changes:

  • Bump infra asset version from 0.5.17 to 0.5.18.
  • Add RUST_LIB_BACKTRACE=0 to aggregator and relay-related docker-compose environments to prevent anyhow backtrace capture.
  • Apply the same RUST_LIB_BACKTRACE=0 setting to the signer relay docker-compose override as well.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File Description
mithril-infra/assets/infra.version Bumps infra asset version to reflect the rollout of the backtrace logging mitigation.
mithril-infra/assets/docker/docker-compose-signer-p2p-base-no-dmq-override.yaml Disables library backtraces for the signer relay container via environment variable.
mithril-infra/assets/docker/docker-compose-aggregator-p2p-base-override.yaml Disables library backtraces for the aggregator relay container via environment variable.
mithril-infra/assets/docker/docker-compose-aggregator-base.yaml Disables library backtraces for the aggregator container via environment variable.

Set RUST_LIB_BACKTRACE=0 so anyhow errors logged by spawned nodes
do not embed full backtraces.
- 'mithril-infra' from '0.5.17' to '0.5.18'
- 'mithril-end-to-end' from '0.5.7' to '0.5.8'.
@jpraynaud
jpraynaud force-pushed the jpraynaud/fix-warp-huge-backtraces branch from a4a4499 to d3f4c07 Compare July 30, 2026 16:39
@jpraynaud
jpraynaud temporarily deployed to testing-preview July 30, 2026 16:57 — with GitHub Actions Inactive
@jpraynaud
jpraynaud temporarily deployed to testing-2-preview July 30, 2026 16:57 — with GitHub Actions Inactive
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants