Skip to content

apollo_dashboard: exclude observers from strk_to_usd_rate_frozen alert - #14934

Open
matanl-starkware wants to merge 1 commit into
mainfrom
matanl/strk-to-usd-rate-frozen-observer
Open

apollo_dashboard: exclude observers from strk_to_usd_rate_frozen alert#14934
matanl-starkware wants to merge 1 commit into
mainfrom
matanl/strk-to-usd-rate-frozen-observer

Conversation

@matanl-starkware

Copy link
Copy Markdown
Collaborator

Problem

strk_to_usd_rate_frozen fires permanently on observer nodes (seen on apollo-sepolia-alpha-0, the one sepolia-alpha node whose validator_id is outside the committee).

Root cause: oracle_rate_frozen_alert hardcodes ObserverApplicability::Applicable for both exchange-rate oracles, but the two oracles are queried on different consensus paths:

  • eth_to_strk is queried on the proposal validation path (validate_proposal.rsget_l1_prices_in_fri_and_wei), so every node — observers included — keeps its gauge moving. Applicable is correct.
  • strk_to_usd is queried only on the proposer build path (build_proposalcompute_proposer_fee_proposalresolve_fee_target), for the SNIP-35 fee proposal. Validation bounds-checks the proposer's fee_proposal_fri against the local median without an oracle read. An observer never proposes, so its snip35_strk_usd_rate gauge is registered at 0 (ExchangeRateOracleClient::new registers metrics unconditionally) and never updated — present-but-flat by construction, so sum(changes(...[1h])) < 1 is permanently true.

Not an outage: on the observer, snip35_strk_usd_success_count and _error_count are both flat 0 (zero attempts), while all validators read the rate normally via round-robin proposership.

Fix

Parameterize oracle_rate_frozen_alert with ObserverApplicability and pass NotApplicable for strk_to_usd (Applicable for eth_to_strk, unchanged behavior). This matches the oracle success/error-count alerts, which already exclude observers. Regenerated dev_grafana_alerts.json; the only expression change is strk_to_usd_rate_frozen gaining the and on() (is_observer == 0) clause.

Verification

  • SEED=0 cargo test -p apollo_dashboard — 16 passed (includes the generated-JSON drift test)
  • cargo clippy -p apollo_dashboard --all-targets -- -D warnings — clean
  • scripts/rust_fmt.sh — clean

🤖 Generated with Claude Code

The frozen-rate alert hardcoded ObserverApplicability::Applicable for both
exchange-rate oracles. That is correct for eth_to_strk, which every node
queries when validating a proposal, but wrong for strk_to_usd, which is
queried only on the proposer build path (SNIP-35 fee proposal). An observer
never proposes, so its snip35_strk_usd_rate gauge is registered at 0 and
never updated — present-but-flat by construction — making the alert a
permanent false positive on observer nodes (seen on apollo-sepolia-alpha-0).

Parameterize oracle_rate_frozen_alert with ObserverApplicability and pass
NotApplicable for strk_to_usd, matching the oracle success/error-count
alerts which already exclude observers.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@cursor

cursor Bot commented Aug 9, 2026

Copy link
Copy Markdown

PR Summary

Low Risk
Alert-definition-only change that reduces noise on observers; validator monitoring for frozen STRK/USD rates is unchanged.

Overview
Stops false-positive strk_to_usd_rate_frozen alerts on observer nodes by scoping that frozen-rate check to non-observers only.

oracle_rate_frozen_alert now takes ObserverApplicability: eth_to_strk stays Applicable (rate updates on validation for all nodes); strk_to_usd uses NotApplicable because SNIP-35 USD rate is only read when building proposals, so observers keep a flat gauge. Regenerated dev_grafana_alerts.json so strk_to_usd_rate_frozen adds and on() (is_observer == 0); inline docs explain the split.

Reviewed by Cursor Bugbot for commit 0211d16. Bugbot is set up for automated code reviews on this repo. Configure here.

@reviewable-StarkWare

Copy link
Copy Markdown

This change is Reviewable

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants