apollo_dashboard: exclude observers from strk_to_usd_rate_frozen alert - #14934
apollo_dashboard: exclude observers from strk_to_usd_rate_frozen alert#14934matanl-starkware wants to merge 1 commit into
Conversation
The frozen-rate alert hardcoded ObserverApplicability::Applicable for both exchange-rate oracles. That is correct for eth_to_strk, which every node queries when validating a proposal, but wrong for strk_to_usd, which is queried only on the proposer build path (SNIP-35 fee proposal). An observer never proposes, so its snip35_strk_usd_rate gauge is registered at 0 and never updated — present-but-flat by construction — making the alert a permanent false positive on observer nodes (seen on apollo-sepolia-alpha-0). Parameterize oracle_rate_frozen_alert with ObserverApplicability and pass NotApplicable for strk_to_usd, matching the oracle success/error-count alerts which already exclude observers. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
PR SummaryLow Risk Overview
Reviewed by Cursor Bugbot for commit 0211d16. Bugbot is set up for automated code reviews on this repo. Configure here. |
Problem
strk_to_usd_rate_frozenfires permanently on observer nodes (seen onapollo-sepolia-alpha-0, the one sepolia-alpha node whosevalidator_idis outside the committee).Root cause:
oracle_rate_frozen_alerthardcodesObserverApplicability::Applicablefor both exchange-rate oracles, but the two oracles are queried on different consensus paths:validate_proposal.rs→get_l1_prices_in_fri_and_wei), so every node — observers included — keeps its gauge moving.Applicableis correct.build_proposal→compute_proposer_fee_proposal→resolve_fee_target), for the SNIP-35 fee proposal. Validation bounds-checks the proposer'sfee_proposal_friagainst the local median without an oracle read. An observer never proposes, so itssnip35_strk_usd_rategauge is registered at 0 (ExchangeRateOracleClient::newregisters metrics unconditionally) and never updated — present-but-flat by construction, sosum(changes(...[1h])) < 1is permanently true.Not an outage: on the observer,
snip35_strk_usd_success_countand_error_countare both flat 0 (zero attempts), while all validators read the rate normally via round-robin proposership.Fix
Parameterize
oracle_rate_frozen_alertwithObserverApplicabilityand passNotApplicablefor strk_to_usd (Applicablefor eth_to_strk, unchanged behavior). This matches the oracle success/error-count alerts, which already exclude observers. Regenerateddev_grafana_alerts.json; the only expression change isstrk_to_usd_rate_frozengaining theand on() (is_observer == 0)clause.Verification
SEED=0 cargo test -p apollo_dashboard— 16 passed (includes the generated-JSON drift test)cargo clippy -p apollo_dashboard --all-targets -- -D warnings— cleanscripts/rust_fmt.sh— clean🤖 Generated with Claude Code