apollo_dashboard: alert on the Chainlink oracle guard trips - #14948
apollo_dashboard: alert on the Chainlink oracle guard trips#14948asaf-sw wants to merge 1 commit into
Conversation
Surfaces two of the Chainlink client's guards to on-call, on top of the config switch that makes the client selectable. - P4 when a feed reading is rejected for falling outside the freshness window. This is Ohad's "alert if price isn't fresh from the last day": the feeds guarantee an update every 24h, so a rejection means the feed stopped publishing or its `updated_at` is implausible. The counter covers both directions, since a timestamp too far in the future is the poisoning case the guard exists to reject, an unbounded `updated_at` otherwise reading as permanently fresh. - P3 when a rate is rejected for falling outside the configured sanity bounds. Higher, because this is the wrong-feed and poisoned-feed detector, and the one guard consensus cannot substitute for: validators check only that they agree with each other, and every node reads the same chain state, so a plausible but wrong price produces unanimous agreement. The other two guards get no alert. Every guard trip also increments the pair's `error_count`, which already carries a paging alert, so invalid-answer and contract-call trips would page twice for a case that alert covers. These two earn their own because they name a cause the aggregate cannot. `or vector(0)` keeps each query defined before its counter first exists, since `sum` over an empty vector returns empty rather than zero and the generated rules page on their no-data state. That is also why these apply to observers where the sibling oracle alerts do not: `NotApplicable` wraps the whole expression in an `is_observer` gate that discards the fallback wherever that metric is absent. Observers read the same feeds, so a trip is environment-wide either way. The counters stay flat while a feed is served over HTTP, so these alerts are silent until a feed is switched to Chainlink. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
This change is |
|
Warning This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
This stack of pull requests is managed by Graphite. Learn more about stacking. |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 1c03576. Configure here.
| EvaluationRate::Default, | ||
| format!("{} or vector(0)", sum_increase(guard_trip_metric, GUARD_SAMPLING_WINDOW)), | ||
| vec![AlertCondition::new(AlertComparisonOp::GreaterThan, 0.0, AlertLogicalOp::And)], | ||
| GUARD_PENDING_DURATION, |
There was a problem hiding this comment.
Pending duration fails to filter singles
Medium Severity
GUARD_PENDING_DURATION is documented to keep a single guard trip quiet, and guard_trip_alert is described as firing while rejections continue, but sum(increase(...[1h])) > 0 stays true for about an hour after one increment. The 10m for only delays the page; it does not require persistence, and the alert can remain firing long after the feed recovers.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 1c03576. Configure here.
2 participants
Surfaces two of the Chainlink client's guards to on-call, on top of the config
switch that makes the client selectable.
This is Ohad's "alert if price isn't fresh from the last day": the feeds
guarantee an update every 24h, so a rejection means the feed stopped publishing
or its
updated_atis implausible. The counter covers both directions, sincea timestamp too far in the future is the poisoning case the guard exists to
reject, an unbounded
updated_atotherwise reading as permanently fresh.Higher, because this is the wrong-feed and poisoned-feed detector, and the one
guard consensus cannot substitute for: validators check only that they agree
with each other, and every node reads the same chain state, so a plausible but
wrong price produces unanimous agreement.
The other two guards get no alert. Every guard trip also increments the pair's
error_count, which already carries a paging alert, so invalid-answer andcontract-call trips would page twice for a case that alert covers. These two earn
their own because they name a cause the aggregate cannot.
or vector(0)keeps each query defined before its counter first exists, sincesumover an empty vector returns empty rather than zero and the generated rulespage on their no-data state. That is also why these apply to observers where the
sibling oracle alerts do not:
NotApplicablewraps the whole expression in anis_observergate that discards the fallback wherever that metric is absent.Observers read the same feeds, so a trip is environment-wide either way.
The counters stay flat while a feed is served over HTTP, so these alerts are
silent until a feed is switched to Chainlink.
Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com