feat(observability): SLOs, alerting, and on-call runbook - #448
Open
peaceshallom37-rgb wants to merge 1 commit into
Open
feat(observability): SLOs, alerting, and on-call runbook#448peaceshallom37-rgb wants to merge 1 commit into
peaceshallom37-rgb wants to merge 1 commit into
Conversation
Define SLOs for the enrollment, verification, and playback journeys, extend the backend Prometheus metrics with journey-level counters and latency histograms, and ship Grafana dashboards, burn-rate alert rules, and an Alertmanager config wired to the on-call rotation. - backend: add src/metrics/slo.ts and instrument the three journeys - infra: PrometheusRule (recording + burn-rate alerts), Grafana dashboard ConfigMap, Alertmanager routes/receivers with on-call webhook - docs: SLO definitions, alerting guide, and on-call runbook Closes AetherEdu#415
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #415
Summary
Adds the observability layer requested in #415: SLOs for the core user
journeys, backend metrics to measure them, Grafana dashboards, burn-rate
alerting, and an on-call runbook with escalation paths.
Changes
SLOs (
docs/observability/)slo.md— availability and latency targets for the enrollment(99.5% / p95 < 2s), verification (99.9% / p95 < 1s), and playback
(99.9% / p95 < 1s) journeys, with error budgets and measurement details.
alerting.md— alert semantics and the escalation contract.runbook.md— incident response process, severity model, escalationpaths, and per-alert runbooks.
Backend metrics (
backend/src/metrics/slo.ts)aethermint_slo_requests_total{journey,result}),a latency histogram for successful requests, and a last-success gauge,
registered on the existing Prometheus registry (exposed at
/api/metrics).eventLoggerService.logCourseEnrollment()eventLoggerService.verifyEvent()routes/content.jsGET /:cidInfra (
infra/observability/)prometheus/slo-alerts.yaml— recording rules (error ratios, errorbudget) plus multi-window multi-burn-rate alerts (page fast/long burn,
ticket burn, critical error rate, journey outage, budget depleted).
grafana/— dashboard ConfigMap visualising error budgets, SLOcompliance, latency, and traffic per journey.
alertmanager/— routespagealerts to the on-call rotation webhookand
ticketalerts to the team channel, with email fallback; credentialsare injected via environment variables, none are committed.
README.md— deployment steps and receiver configuration.Notes
reported as "not authentic", so backend errors are no longer
indistinguishable from a genuinely invalid credential.
service failures).
tests/routes/events.test.jssuite failure (undefinedevaluateForUserin
routes/admin/featureFlags.ts) is unrelated and reproduces onmain.