diff --git a/.github/workflows/slo-drift.yml b/.github/workflows/slo-drift.yml new file mode 100644 index 0000000..215a1f7 --- /dev/null +++ b/.github/workflows/slo-drift.yml @@ -0,0 +1,37 @@ +name: SLO rule drift check + +# Fails if applications/infrastructure/sloth/prometheusservicelevels.yaml was +# edited without re-running `just gen-slos` — the standalone `prometheus` +# chart used in this lab reads no PrometheusRule CRs, so the rendered rule +# groups embedded in applications/infrastructure/prometheus/values.yaml are +# the only copy Prometheus ever loads. See provisioning/scripts/gen_slos.sh. +on: + pull_request: + paths: + - "applications/infrastructure/sloth/prometheusservicelevels.yaml" + - "applications/infrastructure/prometheus/values.yaml" + - "provisioning/scripts/gen_slos.sh" + +jobs: + check-drift: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - name: Install sloth CLI (pinned v0.16.0, matches the vendored chart version) + run: | + curl -sSL -o /usr/local/bin/sloth \ + https://github.com/slok/sloth/releases/download/v0.16.0/sloth-linux-amd64 + chmod +x /usr/local/bin/sloth + sloth version + + - name: Install yq + uses: mikefarah/yq@v4.53.3 + + - name: Regenerate and diff + run: | + ./provisioning/scripts/gen_slos.sh + git diff --exit-code -- applications/infrastructure/prometheus/values.yaml || { + echo "::error::applications/infrastructure/prometheus/values.yaml is stale — run 'just gen-slos' and commit the result." + exit 1 + } diff --git a/CLAUDE.md b/CLAUDE.md index 7a3aa2b..bd49008 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -54,7 +54,8 @@ applications/ config/ # gitops.env — single source of truth for all IPs, versions, hostnames infrastructure/ # one dir per infra component; each has config.json + kustomization.yaml ceph-latency-bridge/ # native OSD histogram reconstruction (wave 30) - sloth/ # PrometheusServiceLevel CRs + SLO burn-rate rules (wave 33) + sloth/ # PrometheusServiceLevel specs — NOT a deployed app; build-time input to + # `just gen-slos` (renders into infrastructure/prometheus/values.yaml). See gotcha #24. topology-catalog/ # static catalog-info.yaml topology map, ConfigMap (wave 5) l7-policies/ # CiliumNetworkPolicies, organized by namespace subdirectory argocd/ # cnp-argocd.yaml @@ -144,7 +145,6 @@ All policies live in `applications/infrastructure/l7-policies/`, organized into | 30 | rook-storage | true | true | | 30 | ceph-latency-bridge (SLO metrics) | true | true | | 31 | rook-dashboards (Grafana ConfigMaps) | true | true | -| 33 | sloth (`PrometheusServiceLevel` CRs + SLO burn-rate rules) | true | true | | 35 | rook-gateway | true | true | Rook operator and rook-cluster prune/selfHeal settings are intentional — they protect Ceph data from accidental ArgoCD deletes. @@ -165,7 +165,7 @@ Rook operator and rook-cluster prune/selfHeal settings are intentional — they 10. **`--enable-helm` is required** — patched into `argocd-cm` via `cluster-bootstrap/argocd/kustomization.yaml`. Without it, `helmCharts:` stanzas are silently ignored. 11. **OSD histogram `le` labels use high floating-point precision** — `ceph-latency-bridge` exports bucket boundaries like `le="0.099999"` not `le="0.1"`. PromQL queries in dashboards and recording rules must match these exact labels or return no data. 12. **Histogram buckets must be in strictly ascending order** — custom exporters (like `ceph-latency-bridge`) must emit `le` values ascending with `+Inf` last. Out-of-order buckets cause Prometheus to silently discard the metric. -13. **`ceph-observability-mach-2.json` is the definitive SLO dashboard** — 3-row narrative: Health → SLI → SLO. Uses per-OSD P99 lines and burn-rate alerting. Located in `applications/rook/dashboards/`. Row 3 is Sloth-backed (`sloth_id="ceph-osd-latency"`, **write** latency 99%/50ms — it was read latency 99.9%/100ms before the Sloth cutover, a real semantic change) — see gotcha #21/#22 before editing its queries. +13. **`ceph-observability-mach-2.json` is the definitive SLO dashboard** — 3-row narrative: Health → SLI → SLO. Uses per-OSD P99 lines and burn-rate alerting. Located in `applications/rook/dashboards/`. Row 3 is Sloth-backed (`sloth_id="ceph-osd-latency"`, **write** latency 99%/100ms — it was read latency 99.9%/100ms before the Sloth cutover, a real semantic change) — see gotcha #21/#22 before editing its queries. The write-latency threshold was originally spec'd as <50ms (`le="0.049999"`) but that bucket doesn't exist; verified live 2026-07-03 against `ceph_osd_op_w_latency_bucket` and corrected to <100ms (`le="0.099999"`) — see gotcha #24. 14. **Disabling ArgoCD selfHeal during manual fixes** — `selfHeal: true` will immediately revert `kubectl patch/apply` changes. Disable sync on `ceph-lab-root` first, then the specific Application, apply the fix, then push to git and re-enable. 15. **virtiofs mounts are async** — Lima YAML provision steps poll for `/ceph-lab/provisioning/provision.env` (up to 60s) before running scripts. Scripts must not assume `/ceph-lab` is immediately available at VM boot. 16. **`socket_vmnet` must be at `/opt/socket_vmnet`** — required for the `ceph-lab` host-only network (192.168.56.0/24) that gives VMs static IPs. Run `make setup` once after installing Lima. @@ -183,6 +183,8 @@ Rook operator and rook-cluster prune/selfHeal settings are intentional — they 21. **Sloth's `sloth_id` label is NOT the CR's `metadata.name`** — it's computed as `"{spec.service}-{slo.name}"` (confirmed against Sloth's own source, `internal/storage/io/k8s_sloth.go`). To land on a specific `sloth_id` (rattle's `SLO.ID` contract requires this — see `applications/infrastructure/sloth/prometheusservicelevels.yaml`), split `service`/`name` accordingly (e.g. `service: ceph-osd` + `name: latency` → `sloth_id: ceph-osd-latency`), not by renaming the CR. 22. **Sloth's `slo:current_burn_rate:ratio` is a single un-windowed series**, computed off the page-alert's short window — there is no separate metadata metric per window (1h vs 6h). Anything wanting an explicit windowed burn rate (like `ceph-observability-mach-2.json`'s Row 3) must use `slo:sli_error:ratio_rate{1h,6h}{sloth_id=...} / (1 - objective)` instead — Sloth does generate per-window SLI error-ratio series, just not per-window burn-rate ones. Also: Sloth's metadata recording rules have no on/off flag — they're always generated, there's nothing to "enable." 23. **Loki/Tempo/Alloy were already deployed here once and removed for OOM'ing `ceph-control`** (see `docs/trim-observability-stack.md`) — but that was on the old Vagrant/VMware topology at 3 GB. The Lima rewrite already doubled `ceph-control` to 6 GiB, and Lima's `vz` backend has substantially less virtualization overhead than Vagrant/VMware on top of that — real headroom exists now. Still: size any reintroduction (Loki/Promtail, Tempo/OTel Collector) with explicit resource requests/limits (the original incident's root cause was *unbounded* usage, not merely "too much") and a post-deploy OOM-watch verification gate, not by assuming the extra headroom makes limits unnecessary. +24. **The standalone `prometheus` chart reads NO `monitoring.coreos.com` CRs** — no ServiceMonitor/PrometheusRule controller is deployed (`prometheus-operator-crds` ships CRD schemas only, no controller), so a live Sloth Deployment generating `PrometheusRule` CRs was pure overhead nothing consumed. Fix: `applications/infrastructure/sloth/prometheusservicelevels.yaml` is now a **build-time spec only** (the Sloth runtime chart + CRDs were retired from the cluster) — `provisioning/scripts/gen_slos.sh` (`just gen-slos`) runs the pinned `sloth` CLI (`brew install sloth-cli`, version must match the spec's last-tested chart version) to render it into plain rule groups spliced into `applications/infrastructure/prometheus/values.yaml`'s `serverFiles.recording_rules.yml` / `.alerting_rules.yml` (between `# --- BEGIN/END sloth-generated ... ---` markers — do not hand-edit inside them). `.github/workflows/slo-drift.yml` fails CI if the spec and the rendered rules disagree. Also folded in here: the `ceph-osd-latency` SLO's `le="0.049999"` bucket boundary was a guess that turned out wrong — verified live 2026-07-03 against `ceph_osd_op_w_latency_bucket`, corrected to `le="0.099999"` (objective is p99 **< 100ms**, not <50ms — update `ceph-observability-mach-2.json` Row 3's panel description too if this ever changes again). +25. **Any new scrape target in a CNP'd namespace needs its port added to that namespace's `l7-visibility` allowlist** — the l7-policies make selected pods default-deny for unmatched ingress (`endpointSelector: {}` + explicit `toPorts` allowlist). Diagnosed via kube-state-metrics: its scrape job existed and KSM was healthy, but `monitoring/l7-visibility` only allowlisted 9090/3000/80, so every scrape timed out (`context deadline exceeded`) despite being same-node. Symptom looks like "no ServiceMonitor" or "target down"; check the CNP allowlist before anything else. --- diff --git a/applications/infrastructure/l7-policies/monitoring/cnp-monitoring.yaml b/applications/infrastructure/l7-policies/monitoring/cnp-monitoring.yaml index 0810625..04005ce 100644 --- a/applications/infrastructure/l7-policies/monitoring/cnp-monitoring.yaml +++ b/applications/infrastructure/l7-policies/monitoring/cnp-monitoring.yaml @@ -15,6 +15,8 @@ spec: - { port: "9090", protocol: TCP } # Prometheus server - { port: "3000", protocol: TCP } # Grafana - { port: "80", protocol: TCP } # Prometheus service port + - { port: "8080", protocol: TCP } # kube-state-metrics (metrics) + - { port: "8081", protocol: TCP } # kube-state-metrics (telemetry) - fromEntities: [ host, remote-node ] egress: - toEndpoints: diff --git a/applications/infrastructure/l7-policies/rook-ceph/cnp-rook-ceph.yaml b/applications/infrastructure/l7-policies/rook-ceph/cnp-rook-ceph.yaml index 6c2dcbb..ed215e3 100644 --- a/applications/infrastructure/l7-policies/rook-ceph/cnp-rook-ceph.yaml +++ b/applications/infrastructure/l7-policies/rook-ceph/cnp-rook-ceph.yaml @@ -8,7 +8,8 @@ metadata: spec: endpointSelector: {} ingress: - # RGW S3, MGR dashboard, Prometheus metrics, Rook operator metrics + # RGW S3, MGR dashboard, Prometheus metrics, Rook exporter metrics + # (2112 removed — no rook-ceph-operator Prometheus endpoint ever existed) - fromEntities: [cluster] toPorts: - ports: @@ -16,7 +17,6 @@ spec: - { port: "7000", protocol: TCP } - { port: "9283", protocol: TCP } - { port: "8080", protocol: TCP } - - { port: "2112", protocol: TCP } - { port: "9926", protocol: TCP } # Rook operator webhook (TLS, L4 only) - fromEntities: [kube-apiserver] diff --git a/applications/infrastructure/prometheus/values.yaml b/applications/infrastructure/prometheus/values.yaml index ef60aaa..fc9b883 100644 --- a/applications/infrastructure/prometheus/values.yaml +++ b/applications/infrastructure/prometheus/values.yaml @@ -1,6 +1,9 @@ -# Standalone Prometheus — single-process, no operator. -# Picks up all ServiceMonitors and PrometheusRules cluster-wide -# (including the ones Rook's operator creates in rook-ceph). +# Standalone Prometheus — single-process, no operator. This chart reads NO +# monitoring.coreos.com CRs (no ServiceMonitor/PrometheusRule controller is +# deployed) — all scraping below is static/annotation-based, and Sloth's +# generated rules are rendered to plain rule groups at commit time (see +# serverFiles.recording_rules.yml / .alerting_rules.yml below, and +# provisioning/scripts/gen_slos.sh) rather than picked up live. server: replicaCount: 1 @@ -71,8 +74,9 @@ kube-state-metrics: prometheus-pushgateway: enabled: false -# Tell Prometheus to also discover ServiceMonitors (CRD-based scraping). -# The standalone chart supports this natively via serverFiles. +# serverFiles keys become files in the server ConfigMap, mounted at +# /etc/config/. There is no ServiceMonitor/PrometheusRule discovery here — +# see the header comment above. serverFiles: prometheus.yml: rule_files: @@ -91,14 +95,17 @@ serverFiles: - target_label: job replacement: rook-ceph-mgr - # Rook operator metrics (port 2112). - - job_name: "rook-ceph-operator" + # Rook exporter metrics (port 9926) — the real Ceph-cluster-wide + # exporter Rook ships. (There is no "rook-ceph-operator:2112" endpoint; + # the operator itself exposes no Prometheus metrics — a prior version + # of this job scraped a Service that never existed.) + - job_name: "rook-ceph-exporter" static_configs: - targets: - - "rook-ceph-operator-metrics.rook-ceph.svc.cluster.local:2112" + - "rook-ceph-exporter.rook-ceph.svc.cluster.local:9926" relabel_configs: - target_label: job - replacement: rook-ceph-operator + replacement: rook-ceph-exporter # Hubble metrics (Cilium flow observability). - job_name: "hubble-metrics" @@ -127,3 +134,738 @@ serverFiles: relabel_configs: - target_label: job replacement: argocd-metrics + + # --- BEGIN sloth-generated recording rules (regenerate: just gen-slos) --- + # Source: applications/infrastructure/sloth/prometheusservicelevels.yaml + # DO NOT EDIT BY HAND — run `just gen-slos` after changing the source spec. + recording_rules.yml: + groups: + - name: sloth-slo-sli-recordings-ceph-rgw-availability + rules: + - expr: | + (sum(rate(ceph_rgw_failed_req{job="rook-ceph-mgr"}[5m]))) + / + (sum(rate(ceph_rgw_req{job="rook-ceph-mgr"}[5m]))) + labels: + lab: ceph-lab + sloth_id: ceph-rgw-availability + sloth_service: ceph-rgw + sloth_slo: availability + sloth_window: 5m + record: slo:sli_error:ratio_rate5m + - expr: | + (sum(rate(ceph_rgw_failed_req{job="rook-ceph-mgr"}[30m]))) + / + (sum(rate(ceph_rgw_req{job="rook-ceph-mgr"}[30m]))) + labels: + lab: ceph-lab + sloth_id: ceph-rgw-availability + sloth_service: ceph-rgw + sloth_slo: availability + sloth_window: 30m + record: slo:sli_error:ratio_rate30m + - expr: | + (sum(rate(ceph_rgw_failed_req{job="rook-ceph-mgr"}[1h]))) + / + (sum(rate(ceph_rgw_req{job="rook-ceph-mgr"}[1h]))) + labels: + lab: ceph-lab + sloth_id: ceph-rgw-availability + sloth_service: ceph-rgw + sloth_slo: availability + sloth_window: 1h + record: slo:sli_error:ratio_rate1h + - expr: | + (sum(rate(ceph_rgw_failed_req{job="rook-ceph-mgr"}[2h]))) + / + (sum(rate(ceph_rgw_req{job="rook-ceph-mgr"}[2h]))) + labels: + lab: ceph-lab + sloth_id: ceph-rgw-availability + sloth_service: ceph-rgw + sloth_slo: availability + sloth_window: 2h + record: slo:sli_error:ratio_rate2h + - expr: | + (sum(rate(ceph_rgw_failed_req{job="rook-ceph-mgr"}[6h]))) + / + (sum(rate(ceph_rgw_req{job="rook-ceph-mgr"}[6h]))) + labels: + lab: ceph-lab + sloth_id: ceph-rgw-availability + sloth_service: ceph-rgw + sloth_slo: availability + sloth_window: 6h + record: slo:sli_error:ratio_rate6h + - expr: | + (sum(rate(ceph_rgw_failed_req{job="rook-ceph-mgr"}[1d]))) + / + (sum(rate(ceph_rgw_req{job="rook-ceph-mgr"}[1d]))) + labels: + lab: ceph-lab + sloth_id: ceph-rgw-availability + sloth_service: ceph-rgw + sloth_slo: availability + sloth_window: 1d + record: slo:sli_error:ratio_rate1d + - expr: | + (sum(rate(ceph_rgw_failed_req{job="rook-ceph-mgr"}[3d]))) + / + (sum(rate(ceph_rgw_req{job="rook-ceph-mgr"}[3d]))) + labels: + lab: ceph-lab + sloth_id: ceph-rgw-availability + sloth_service: ceph-rgw + sloth_slo: availability + sloth_window: 3d + record: slo:sli_error:ratio_rate3d + - expr: | + sum_over_time(slo:sli_error:ratio_rate5m{sloth_id="ceph-rgw-availability", sloth_service="ceph-rgw", sloth_slo="availability"}[30d]) + / ignoring (sloth_window) + count_over_time(slo:sli_error:ratio_rate5m{sloth_id="ceph-rgw-availability", sloth_service="ceph-rgw", sloth_slo="availability"}[30d]) + labels: + lab: ceph-lab + sloth_id: ceph-rgw-availability + sloth_service: ceph-rgw + sloth_slo: availability + sloth_window: 30d + record: slo:sli_error:ratio_rate30d + - name: sloth-slo-meta-recordings-ceph-rgw-availability + rules: + - expr: vector(0.9990000000000001) + labels: + lab: ceph-lab + sloth_id: ceph-rgw-availability + sloth_service: ceph-rgw + sloth_slo: availability + record: slo:objective:ratio + - expr: vector(1-0.9990000000000001) + labels: + lab: ceph-lab + sloth_id: ceph-rgw-availability + sloth_service: ceph-rgw + sloth_slo: availability + record: slo:error_budget:ratio + - expr: vector(30) + labels: + lab: ceph-lab + sloth_id: ceph-rgw-availability + sloth_service: ceph-rgw + sloth_slo: availability + record: slo:time_period:days + - expr: | + slo:sli_error:ratio_rate5m{sloth_id="ceph-rgw-availability", sloth_service="ceph-rgw", sloth_slo="availability"} + / on(sloth_id, sloth_slo, sloth_service) group_left + slo:error_budget:ratio{sloth_id="ceph-rgw-availability", sloth_service="ceph-rgw", sloth_slo="availability"} + labels: + lab: ceph-lab + sloth_id: ceph-rgw-availability + sloth_service: ceph-rgw + sloth_slo: availability + record: slo:current_burn_rate:ratio + - expr: | + slo:sli_error:ratio_rate30d{sloth_id="ceph-rgw-availability", sloth_service="ceph-rgw", sloth_slo="availability"} + / on(sloth_id, sloth_slo, sloth_service) group_left + slo:error_budget:ratio{sloth_id="ceph-rgw-availability", sloth_service="ceph-rgw", sloth_slo="availability"} + labels: + lab: ceph-lab + sloth_id: ceph-rgw-availability + sloth_service: ceph-rgw + sloth_slo: availability + record: slo:period_burn_rate:ratio + - expr: 1 - slo:period_burn_rate:ratio{sloth_id="ceph-rgw-availability", sloth_service="ceph-rgw", sloth_slo="availability"} + labels: + lab: ceph-lab + sloth_id: ceph-rgw-availability + sloth_service: ceph-rgw + sloth_slo: availability + record: slo:period_error_budget_remaining:ratio + - expr: vector(1) + labels: + lab: ceph-lab + sloth_id: ceph-rgw-availability + sloth_mode: cli-gen-k8s + sloth_objective: "99.9" + sloth_service: ceph-rgw + sloth_slo: availability + sloth_spec: sloth.slok.dev/v1 + sloth_version: 0.16.0 + record: sloth_slo_info + - name: sloth-slo-sli-recordings-ceph-osd-latency + rules: + - expr: | + (sum(rate(ceph_osd_op_w_latency_count[5m])) - sum(rate(ceph_osd_op_w_latency_bucket{le="0.099999"}[5m]))) + / + (sum(rate(ceph_osd_op_w_latency_count[5m]))) + labels: + lab: ceph-lab + sloth_id: ceph-osd-latency + sloth_service: ceph-osd + sloth_slo: latency + sloth_window: 5m + record: slo:sli_error:ratio_rate5m + - expr: | + (sum(rate(ceph_osd_op_w_latency_count[30m])) - sum(rate(ceph_osd_op_w_latency_bucket{le="0.099999"}[30m]))) + / + (sum(rate(ceph_osd_op_w_latency_count[30m]))) + labels: + lab: ceph-lab + sloth_id: ceph-osd-latency + sloth_service: ceph-osd + sloth_slo: latency + sloth_window: 30m + record: slo:sli_error:ratio_rate30m + - expr: | + (sum(rate(ceph_osd_op_w_latency_count[1h])) - sum(rate(ceph_osd_op_w_latency_bucket{le="0.099999"}[1h]))) + / + (sum(rate(ceph_osd_op_w_latency_count[1h]))) + labels: + lab: ceph-lab + sloth_id: ceph-osd-latency + sloth_service: ceph-osd + sloth_slo: latency + sloth_window: 1h + record: slo:sli_error:ratio_rate1h + - expr: | + (sum(rate(ceph_osd_op_w_latency_count[2h])) - sum(rate(ceph_osd_op_w_latency_bucket{le="0.099999"}[2h]))) + / + (sum(rate(ceph_osd_op_w_latency_count[2h]))) + labels: + lab: ceph-lab + sloth_id: ceph-osd-latency + sloth_service: ceph-osd + sloth_slo: latency + sloth_window: 2h + record: slo:sli_error:ratio_rate2h + - expr: | + (sum(rate(ceph_osd_op_w_latency_count[6h])) - sum(rate(ceph_osd_op_w_latency_bucket{le="0.099999"}[6h]))) + / + (sum(rate(ceph_osd_op_w_latency_count[6h]))) + labels: + lab: ceph-lab + sloth_id: ceph-osd-latency + sloth_service: ceph-osd + sloth_slo: latency + sloth_window: 6h + record: slo:sli_error:ratio_rate6h + - expr: | + (sum(rate(ceph_osd_op_w_latency_count[1d])) - sum(rate(ceph_osd_op_w_latency_bucket{le="0.099999"}[1d]))) + / + (sum(rate(ceph_osd_op_w_latency_count[1d]))) + labels: + lab: ceph-lab + sloth_id: ceph-osd-latency + sloth_service: ceph-osd + sloth_slo: latency + sloth_window: 1d + record: slo:sli_error:ratio_rate1d + - expr: | + (sum(rate(ceph_osd_op_w_latency_count[3d])) - sum(rate(ceph_osd_op_w_latency_bucket{le="0.099999"}[3d]))) + / + (sum(rate(ceph_osd_op_w_latency_count[3d]))) + labels: + lab: ceph-lab + sloth_id: ceph-osd-latency + sloth_service: ceph-osd + sloth_slo: latency + sloth_window: 3d + record: slo:sli_error:ratio_rate3d + - expr: | + sum_over_time(slo:sli_error:ratio_rate5m{sloth_id="ceph-osd-latency", sloth_service="ceph-osd", sloth_slo="latency"}[30d]) + / ignoring (sloth_window) + count_over_time(slo:sli_error:ratio_rate5m{sloth_id="ceph-osd-latency", sloth_service="ceph-osd", sloth_slo="latency"}[30d]) + labels: + lab: ceph-lab + sloth_id: ceph-osd-latency + sloth_service: ceph-osd + sloth_slo: latency + sloth_window: 30d + record: slo:sli_error:ratio_rate30d + - name: sloth-slo-meta-recordings-ceph-osd-latency + rules: + - expr: vector(0.99) + labels: + lab: ceph-lab + sloth_id: ceph-osd-latency + sloth_service: ceph-osd + sloth_slo: latency + record: slo:objective:ratio + - expr: vector(1-0.99) + labels: + lab: ceph-lab + sloth_id: ceph-osd-latency + sloth_service: ceph-osd + sloth_slo: latency + record: slo:error_budget:ratio + - expr: vector(30) + labels: + lab: ceph-lab + sloth_id: ceph-osd-latency + sloth_service: ceph-osd + sloth_slo: latency + record: slo:time_period:days + - expr: | + slo:sli_error:ratio_rate5m{sloth_id="ceph-osd-latency", sloth_service="ceph-osd", sloth_slo="latency"} + / on(sloth_id, sloth_slo, sloth_service) group_left + slo:error_budget:ratio{sloth_id="ceph-osd-latency", sloth_service="ceph-osd", sloth_slo="latency"} + labels: + lab: ceph-lab + sloth_id: ceph-osd-latency + sloth_service: ceph-osd + sloth_slo: latency + record: slo:current_burn_rate:ratio + - expr: | + slo:sli_error:ratio_rate30d{sloth_id="ceph-osd-latency", sloth_service="ceph-osd", sloth_slo="latency"} + / on(sloth_id, sloth_slo, sloth_service) group_left + slo:error_budget:ratio{sloth_id="ceph-osd-latency", sloth_service="ceph-osd", sloth_slo="latency"} + labels: + lab: ceph-lab + sloth_id: ceph-osd-latency + sloth_service: ceph-osd + sloth_slo: latency + record: slo:period_burn_rate:ratio + - expr: 1 - slo:period_burn_rate:ratio{sloth_id="ceph-osd-latency", sloth_service="ceph-osd", sloth_slo="latency"} + labels: + lab: ceph-lab + sloth_id: ceph-osd-latency + sloth_service: ceph-osd + sloth_slo: latency + record: slo:period_error_budget_remaining:ratio + - expr: vector(1) + labels: + lab: ceph-lab + sloth_id: ceph-osd-latency + sloth_mode: cli-gen-k8s + sloth_objective: "99" + sloth_service: ceph-osd + sloth_slo: latency + sloth_spec: sloth.slok.dev/v1 + sloth_version: 0.16.0 + record: sloth_slo_info + - name: sloth-slo-sli-recordings-ceph-health + rules: + - expr: (avg_over_time(clamp_max(ceph_health_status, 1)[5m:1m])) + labels: + lab: ceph-lab + sloth_id: ceph-health + sloth_service: ceph + sloth_slo: health + sloth_window: 5m + record: slo:sli_error:ratio_rate5m + - expr: (avg_over_time(clamp_max(ceph_health_status, 1)[30m:1m])) + labels: + lab: ceph-lab + sloth_id: ceph-health + sloth_service: ceph + sloth_slo: health + sloth_window: 30m + record: slo:sli_error:ratio_rate30m + - expr: (avg_over_time(clamp_max(ceph_health_status, 1)[1h:1m])) + labels: + lab: ceph-lab + sloth_id: ceph-health + sloth_service: ceph + sloth_slo: health + sloth_window: 1h + record: slo:sli_error:ratio_rate1h + - expr: (avg_over_time(clamp_max(ceph_health_status, 1)[2h:1m])) + labels: + lab: ceph-lab + sloth_id: ceph-health + sloth_service: ceph + sloth_slo: health + sloth_window: 2h + record: slo:sli_error:ratio_rate2h + - expr: (avg_over_time(clamp_max(ceph_health_status, 1)[6h:1m])) + labels: + lab: ceph-lab + sloth_id: ceph-health + sloth_service: ceph + sloth_slo: health + sloth_window: 6h + record: slo:sli_error:ratio_rate6h + - expr: (avg_over_time(clamp_max(ceph_health_status, 1)[1d:1m])) + labels: + lab: ceph-lab + sloth_id: ceph-health + sloth_service: ceph + sloth_slo: health + sloth_window: 1d + record: slo:sli_error:ratio_rate1d + - expr: (avg_over_time(clamp_max(ceph_health_status, 1)[3d:1m])) + labels: + lab: ceph-lab + sloth_id: ceph-health + sloth_service: ceph + sloth_slo: health + sloth_window: 3d + record: slo:sli_error:ratio_rate3d + - expr: | + sum_over_time(slo:sli_error:ratio_rate5m{sloth_id="ceph-health", sloth_service="ceph", sloth_slo="health"}[30d]) + / ignoring (sloth_window) + count_over_time(slo:sli_error:ratio_rate5m{sloth_id="ceph-health", sloth_service="ceph", sloth_slo="health"}[30d]) + labels: + lab: ceph-lab + sloth_id: ceph-health + sloth_service: ceph + sloth_slo: health + sloth_window: 30d + record: slo:sli_error:ratio_rate30d + - name: sloth-slo-meta-recordings-ceph-health + rules: + - expr: vector(0.9990000000000001) + labels: + lab: ceph-lab + sloth_id: ceph-health + sloth_service: ceph + sloth_slo: health + record: slo:objective:ratio + - expr: vector(1-0.9990000000000001) + labels: + lab: ceph-lab + sloth_id: ceph-health + sloth_service: ceph + sloth_slo: health + record: slo:error_budget:ratio + - expr: vector(30) + labels: + lab: ceph-lab + sloth_id: ceph-health + sloth_service: ceph + sloth_slo: health + record: slo:time_period:days + - expr: | + slo:sli_error:ratio_rate5m{sloth_id="ceph-health", sloth_service="ceph", sloth_slo="health"} + / on(sloth_id, sloth_slo, sloth_service) group_left + slo:error_budget:ratio{sloth_id="ceph-health", sloth_service="ceph", sloth_slo="health"} + labels: + lab: ceph-lab + sloth_id: ceph-health + sloth_service: ceph + sloth_slo: health + record: slo:current_burn_rate:ratio + - expr: | + slo:sli_error:ratio_rate30d{sloth_id="ceph-health", sloth_service="ceph", sloth_slo="health"} + / on(sloth_id, sloth_slo, sloth_service) group_left + slo:error_budget:ratio{sloth_id="ceph-health", sloth_service="ceph", sloth_slo="health"} + labels: + lab: ceph-lab + sloth_id: ceph-health + sloth_service: ceph + sloth_slo: health + record: slo:period_burn_rate:ratio + - expr: 1 - slo:period_burn_rate:ratio{sloth_id="ceph-health", sloth_service="ceph", sloth_slo="health"} + labels: + lab: ceph-lab + sloth_id: ceph-health + sloth_service: ceph + sloth_slo: health + record: slo:period_error_budget_remaining:ratio + - expr: vector(1) + labels: + lab: ceph-lab + sloth_id: ceph-health + sloth_mode: cli-gen-k8s + sloth_objective: "99.9" + sloth_service: ceph + sloth_slo: health + sloth_spec: sloth.slok.dev/v1 + sloth_version: 0.16.0 + record: sloth_slo_info + - name: sloth-slo-sli-recordings-argocd-sync + rules: + - expr: (avg_over_time((count(argocd_app_info{sync_status!="Synced"}) or vector(0))[5m:1m]) / avg_over_time(count(argocd_app_info)[5m:1m])) + labels: + lab: ceph-lab + sloth_id: argocd-sync + sloth_service: argocd + sloth_slo: sync + sloth_window: 5m + record: slo:sli_error:ratio_rate5m + - expr: (avg_over_time((count(argocd_app_info{sync_status!="Synced"}) or vector(0))[30m:1m]) / avg_over_time(count(argocd_app_info)[30m:1m])) + labels: + lab: ceph-lab + sloth_id: argocd-sync + sloth_service: argocd + sloth_slo: sync + sloth_window: 30m + record: slo:sli_error:ratio_rate30m + - expr: (avg_over_time((count(argocd_app_info{sync_status!="Synced"}) or vector(0))[1h:1m]) / avg_over_time(count(argocd_app_info)[1h:1m])) + labels: + lab: ceph-lab + sloth_id: argocd-sync + sloth_service: argocd + sloth_slo: sync + sloth_window: 1h + record: slo:sli_error:ratio_rate1h + - expr: (avg_over_time((count(argocd_app_info{sync_status!="Synced"}) or vector(0))[2h:1m]) / avg_over_time(count(argocd_app_info)[2h:1m])) + labels: + lab: ceph-lab + sloth_id: argocd-sync + sloth_service: argocd + sloth_slo: sync + sloth_window: 2h + record: slo:sli_error:ratio_rate2h + - expr: (avg_over_time((count(argocd_app_info{sync_status!="Synced"}) or vector(0))[6h:1m]) / avg_over_time(count(argocd_app_info)[6h:1m])) + labels: + lab: ceph-lab + sloth_id: argocd-sync + sloth_service: argocd + sloth_slo: sync + sloth_window: 6h + record: slo:sli_error:ratio_rate6h + - expr: (avg_over_time((count(argocd_app_info{sync_status!="Synced"}) or vector(0))[1d:1m]) / avg_over_time(count(argocd_app_info)[1d:1m])) + labels: + lab: ceph-lab + sloth_id: argocd-sync + sloth_service: argocd + sloth_slo: sync + sloth_window: 1d + record: slo:sli_error:ratio_rate1d + - expr: (avg_over_time((count(argocd_app_info{sync_status!="Synced"}) or vector(0))[3d:1m]) / avg_over_time(count(argocd_app_info)[3d:1m])) + labels: + lab: ceph-lab + sloth_id: argocd-sync + sloth_service: argocd + sloth_slo: sync + sloth_window: 3d + record: slo:sli_error:ratio_rate3d + - expr: | + sum_over_time(slo:sli_error:ratio_rate5m{sloth_id="argocd-sync", sloth_service="argocd", sloth_slo="sync"}[30d]) + / ignoring (sloth_window) + count_over_time(slo:sli_error:ratio_rate5m{sloth_id="argocd-sync", sloth_service="argocd", sloth_slo="sync"}[30d]) + labels: + lab: ceph-lab + sloth_id: argocd-sync + sloth_service: argocd + sloth_slo: sync + sloth_window: 30d + record: slo:sli_error:ratio_rate30d + - name: sloth-slo-meta-recordings-argocd-sync + rules: + - expr: vector(0.99) + labels: + lab: ceph-lab + sloth_id: argocd-sync + sloth_service: argocd + sloth_slo: sync + record: slo:objective:ratio + - expr: vector(1-0.99) + labels: + lab: ceph-lab + sloth_id: argocd-sync + sloth_service: argocd + sloth_slo: sync + record: slo:error_budget:ratio + - expr: vector(30) + labels: + lab: ceph-lab + sloth_id: argocd-sync + sloth_service: argocd + sloth_slo: sync + record: slo:time_period:days + - expr: | + slo:sli_error:ratio_rate5m{sloth_id="argocd-sync", sloth_service="argocd", sloth_slo="sync"} + / on(sloth_id, sloth_slo, sloth_service) group_left + slo:error_budget:ratio{sloth_id="argocd-sync", sloth_service="argocd", sloth_slo="sync"} + labels: + lab: ceph-lab + sloth_id: argocd-sync + sloth_service: argocd + sloth_slo: sync + record: slo:current_burn_rate:ratio + - expr: | + slo:sli_error:ratio_rate30d{sloth_id="argocd-sync", sloth_service="argocd", sloth_slo="sync"} + / on(sloth_id, sloth_slo, sloth_service) group_left + slo:error_budget:ratio{sloth_id="argocd-sync", sloth_service="argocd", sloth_slo="sync"} + labels: + lab: ceph-lab + sloth_id: argocd-sync + sloth_service: argocd + sloth_slo: sync + record: slo:period_burn_rate:ratio + - expr: 1 - slo:period_burn_rate:ratio{sloth_id="argocd-sync", sloth_service="argocd", sloth_slo="sync"} + labels: + lab: ceph-lab + sloth_id: argocd-sync + sloth_service: argocd + sloth_slo: sync + record: slo:period_error_budget_remaining:ratio + - expr: vector(1) + labels: + lab: ceph-lab + sloth_id: argocd-sync + sloth_mode: cli-gen-k8s + sloth_objective: "99" + sloth_service: argocd + sloth_slo: sync + sloth_spec: sloth.slok.dev/v1 + sloth_version: 0.16.0 + record: sloth_slo_info + # --- END sloth-generated recording rules --- + # --- BEGIN sloth-generated alerting rules (regenerate: just gen-slos) --- + # Source: applications/infrastructure/sloth/prometheusservicelevels.yaml + # DO NOT EDIT BY HAND — run `just gen-slos` after changing the source spec. + alerting_rules.yml: + groups: + - name: sloth-slo-alerts-ceph-rgw-availability + rules: + - alert: CephRGWAvailabilitySLO + annotations: + summary: '{{$labels.sloth_service}} {{$labels.sloth_slo}} SLO error budget burn rate is over expected.' + title: (page) {{$labels.sloth_service}} {{$labels.sloth_slo}} SLO error budget burn rate is too fast. + expr: | + ( + max(slo:sli_error:ratio_rate5m{sloth_id="ceph-rgw-availability", sloth_service="ceph-rgw", sloth_slo="availability"} > (14.4 * 0.0009999999999999432)) without (sloth_window) + and + max(slo:sli_error:ratio_rate1h{sloth_id="ceph-rgw-availability", sloth_service="ceph-rgw", sloth_slo="availability"} > (14.4 * 0.0009999999999999432)) without (sloth_window) + ) + or + ( + max(slo:sli_error:ratio_rate30m{sloth_id="ceph-rgw-availability", sloth_service="ceph-rgw", sloth_slo="availability"} > (6 * 0.0009999999999999432)) without (sloth_window) + and + max(slo:sli_error:ratio_rate6h{sloth_id="ceph-rgw-availability", sloth_service="ceph-rgw", sloth_slo="availability"} > (6 * 0.0009999999999999432)) without (sloth_window) + ) + labels: + severity: critical + sloth_severity: page + - alert: CephRGWAvailabilitySLO + annotations: + summary: '{{$labels.sloth_service}} {{$labels.sloth_slo}} SLO error budget burn rate is over expected.' + title: (ticket) {{$labels.sloth_service}} {{$labels.sloth_slo}} SLO error budget burn rate is too fast. + expr: | + ( + max(slo:sli_error:ratio_rate2h{sloth_id="ceph-rgw-availability", sloth_service="ceph-rgw", sloth_slo="availability"} > (3 * 0.0009999999999999432)) without (sloth_window) + and + max(slo:sli_error:ratio_rate1d{sloth_id="ceph-rgw-availability", sloth_service="ceph-rgw", sloth_slo="availability"} > (3 * 0.0009999999999999432)) without (sloth_window) + ) + or + ( + max(slo:sli_error:ratio_rate6h{sloth_id="ceph-rgw-availability", sloth_service="ceph-rgw", sloth_slo="availability"} > (1 * 0.0009999999999999432)) without (sloth_window) + and + max(slo:sli_error:ratio_rate3d{sloth_id="ceph-rgw-availability", sloth_service="ceph-rgw", sloth_slo="availability"} > (1 * 0.0009999999999999432)) without (sloth_window) + ) + labels: + severity: warning + sloth_severity: ticket + - name: sloth-slo-alerts-ceph-osd-latency + rules: + - alert: CephOSDLatencySLO + annotations: + summary: '{{$labels.sloth_service}} {{$labels.sloth_slo}} SLO error budget burn rate is over expected.' + title: (page) {{$labels.sloth_service}} {{$labels.sloth_slo}} SLO error budget burn rate is too fast. + expr: | + ( + max(slo:sli_error:ratio_rate5m{sloth_id="ceph-osd-latency", sloth_service="ceph-osd", sloth_slo="latency"} > (14.4 * 0.01)) without (sloth_window) + and + max(slo:sli_error:ratio_rate1h{sloth_id="ceph-osd-latency", sloth_service="ceph-osd", sloth_slo="latency"} > (14.4 * 0.01)) without (sloth_window) + ) + or + ( + max(slo:sli_error:ratio_rate30m{sloth_id="ceph-osd-latency", sloth_service="ceph-osd", sloth_slo="latency"} > (6 * 0.01)) without (sloth_window) + and + max(slo:sli_error:ratio_rate6h{sloth_id="ceph-osd-latency", sloth_service="ceph-osd", sloth_slo="latency"} > (6 * 0.01)) without (sloth_window) + ) + labels: + severity: critical + sloth_severity: page + - alert: CephOSDLatencySLO + annotations: + summary: '{{$labels.sloth_service}} {{$labels.sloth_slo}} SLO error budget burn rate is over expected.' + title: (ticket) {{$labels.sloth_service}} {{$labels.sloth_slo}} SLO error budget burn rate is too fast. + expr: | + ( + max(slo:sli_error:ratio_rate2h{sloth_id="ceph-osd-latency", sloth_service="ceph-osd", sloth_slo="latency"} > (3 * 0.01)) without (sloth_window) + and + max(slo:sli_error:ratio_rate1d{sloth_id="ceph-osd-latency", sloth_service="ceph-osd", sloth_slo="latency"} > (3 * 0.01)) without (sloth_window) + ) + or + ( + max(slo:sli_error:ratio_rate6h{sloth_id="ceph-osd-latency", sloth_service="ceph-osd", sloth_slo="latency"} > (1 * 0.01)) without (sloth_window) + and + max(slo:sli_error:ratio_rate3d{sloth_id="ceph-osd-latency", sloth_service="ceph-osd", sloth_slo="latency"} > (1 * 0.01)) without (sloth_window) + ) + labels: + severity: warning + sloth_severity: ticket + - name: sloth-slo-alerts-ceph-health + rules: + - alert: CephHealthSLO + annotations: + summary: '{{$labels.sloth_service}} {{$labels.sloth_slo}} SLO error budget burn rate is over expected.' + title: (page) {{$labels.sloth_service}} {{$labels.sloth_slo}} SLO error budget burn rate is too fast. + expr: | + ( + max(slo:sli_error:ratio_rate5m{sloth_id="ceph-health", sloth_service="ceph", sloth_slo="health"} > (14.4 * 0.0009999999999999432)) without (sloth_window) + and + max(slo:sli_error:ratio_rate1h{sloth_id="ceph-health", sloth_service="ceph", sloth_slo="health"} > (14.4 * 0.0009999999999999432)) without (sloth_window) + ) + or + ( + max(slo:sli_error:ratio_rate30m{sloth_id="ceph-health", sloth_service="ceph", sloth_slo="health"} > (6 * 0.0009999999999999432)) without (sloth_window) + and + max(slo:sli_error:ratio_rate6h{sloth_id="ceph-health", sloth_service="ceph", sloth_slo="health"} > (6 * 0.0009999999999999432)) without (sloth_window) + ) + labels: + severity: critical + sloth_severity: page + - alert: CephHealthSLO + annotations: + summary: '{{$labels.sloth_service}} {{$labels.sloth_slo}} SLO error budget burn rate is over expected.' + title: (ticket) {{$labels.sloth_service}} {{$labels.sloth_slo}} SLO error budget burn rate is too fast. + expr: | + ( + max(slo:sli_error:ratio_rate2h{sloth_id="ceph-health", sloth_service="ceph", sloth_slo="health"} > (3 * 0.0009999999999999432)) without (sloth_window) + and + max(slo:sli_error:ratio_rate1d{sloth_id="ceph-health", sloth_service="ceph", sloth_slo="health"} > (3 * 0.0009999999999999432)) without (sloth_window) + ) + or + ( + max(slo:sli_error:ratio_rate6h{sloth_id="ceph-health", sloth_service="ceph", sloth_slo="health"} > (1 * 0.0009999999999999432)) without (sloth_window) + and + max(slo:sli_error:ratio_rate3d{sloth_id="ceph-health", sloth_service="ceph", sloth_slo="health"} > (1 * 0.0009999999999999432)) without (sloth_window) + ) + labels: + severity: warning + sloth_severity: ticket + - name: sloth-slo-alerts-argocd-sync + rules: + - alert: ArgoCDSyncSLO + annotations: + summary: '{{$labels.sloth_service}} {{$labels.sloth_slo}} SLO error budget burn rate is over expected.' + title: (page) {{$labels.sloth_service}} {{$labels.sloth_slo}} SLO error budget burn rate is too fast. + expr: | + ( + max(slo:sli_error:ratio_rate5m{sloth_id="argocd-sync", sloth_service="argocd", sloth_slo="sync"} > (14.4 * 0.01)) without (sloth_window) + and + max(slo:sli_error:ratio_rate1h{sloth_id="argocd-sync", sloth_service="argocd", sloth_slo="sync"} > (14.4 * 0.01)) without (sloth_window) + ) + or + ( + max(slo:sli_error:ratio_rate30m{sloth_id="argocd-sync", sloth_service="argocd", sloth_slo="sync"} > (6 * 0.01)) without (sloth_window) + and + max(slo:sli_error:ratio_rate6h{sloth_id="argocd-sync", sloth_service="argocd", sloth_slo="sync"} > (6 * 0.01)) without (sloth_window) + ) + labels: + severity: critical + sloth_severity: page + - alert: ArgoCDSyncSLO + annotations: + summary: '{{$labels.sloth_service}} {{$labels.sloth_slo}} SLO error budget burn rate is over expected.' + title: (ticket) {{$labels.sloth_service}} {{$labels.sloth_slo}} SLO error budget burn rate is too fast. + expr: | + ( + max(slo:sli_error:ratio_rate2h{sloth_id="argocd-sync", sloth_service="argocd", sloth_slo="sync"} > (3 * 0.01)) without (sloth_window) + and + max(slo:sli_error:ratio_rate1d{sloth_id="argocd-sync", sloth_service="argocd", sloth_slo="sync"} > (3 * 0.01)) without (sloth_window) + ) + or + ( + max(slo:sli_error:ratio_rate6h{sloth_id="argocd-sync", sloth_service="argocd", sloth_slo="sync"} > (1 * 0.01)) without (sloth_window) + and + max(slo:sli_error:ratio_rate3d{sloth_id="argocd-sync", sloth_service="argocd", sloth_slo="sync"} > (1 * 0.01)) without (sloth_window) + ) + labels: + severity: warning + sloth_severity: ticket + # --- END sloth-generated alerting rules --- diff --git a/applications/infrastructure/sloth/config.json b/applications/infrastructure/sloth/config.json deleted file mode 100644 index 79783f6..0000000 --- a/applications/infrastructure/sloth/config.json +++ /dev/null @@ -1,6 +0,0 @@ -{ - "appName": "sloth", - "syncWave": "33", - "namespace": "monitoring", - "localPath": "applications/infrastructure/sloth" -} diff --git a/applications/infrastructure/sloth/kustomization.yaml b/applications/infrastructure/sloth/kustomization.yaml deleted file mode 100644 index b888c50..0000000 --- a/applications/infrastructure/sloth/kustomization.yaml +++ /dev/null @@ -1,15 +0,0 @@ -apiVersion: kustomize.config.k8s.io/v1beta1 -kind: Kustomization -namespace: monitoring - -resources: - - prometheusservicelevels.yaml - -helmCharts: - - name: sloth - repo: https://slok.github.io/sloth - version: 0.16.0 - releaseName: sloth - namespace: monitoring - includeCRDs: true - valuesFile: values.yaml diff --git a/applications/infrastructure/sloth/prometheusservicelevels.yaml b/applications/infrastructure/sloth/prometheusservicelevels.yaml index ba6688f..c62a59a 100644 --- a/applications/infrastructure/sloth/prometheusservicelevels.yaml +++ b/applications/infrastructure/sloth/prometheusservicelevels.yaml @@ -1,3 +1,12 @@ +# BUILD-TIME SPEC, NOT A LIVE K8S OBJECT — the Sloth runtime chart (and its +# CRDs) were retired from this cluster: the standalone `prometheus` chart +# reads no monitoring.coreos.com CRs, so a live Sloth controller generating +# PrometheusRule CRs nobody consumes was pure overhead. This CRD-shaped YAML +# is now only an input to `sloth generate` (provisioning/scripts/gen_slos.sh +# / `just gen-slos`), which renders it to plain rule groups embedded in +# applications/infrastructure/prometheus/values.yaml. Edit this file, run +# `just gen-slos`, commit both. +# # sloth_id is NOT metadata.name — Sloth computes it as "{spec.service}-{slo.name}" # (confirmed against sloth/internal/storage/io/k8s_sloth.go). service/name below # are chosen so the concatenation matches rattle's SLO.ID exactly, per the @@ -52,26 +61,22 @@ spec: - name: latency objective: 99 description: >- - 99% of OSD writes complete under 50ms. Supersedes the hand-rolled - elk-slo-dashboard burn-rate rule (same underlying metrics, same - threshold) — see applications/infrastructure/elk-slo-dashboard's - removal in this same change. - EXECUTION-TIME CHECKPOINT (gotcha #11/#12): le="0.049999" below is a - reasoned guess, not a confirmed value — provisioning/scripts/ceph_latency_exporter.py - computes le = range_info["max"]/1e6 from Ceph's own irregular histogram - boundaries, and mach-2's read-latency SLO shows the 100ms bucket is - actually "0.099999" (99999 microseconds, not 100000) — by the same - pattern the 50ms bucket is expected at 49999us = "0.049999", but this - MUST be confirmed against a live cluster - (curl '.../api/v1/query?query=ceph_osd_op_w_latency_bucket' | jq - '.data.result[].metric.le') before trusting this SLO's numbers, and - corrected here if wrong. + 99% of OSD writes complete under 100ms. Supersedes the hand-rolled + elk-slo-dashboard burn-rate rule (same underlying metrics) — see + applications/infrastructure/elk-slo-dashboard's removal in this same + change. + VERIFIED LIVE 2026-07-03 (see docs/ceph-lab-running-notes F4): the + le="0.049999" guess was wrong — ceph_osd_op_w_latency_bucket has no + 50ms bucket. provisioning/scripts/ceph_latency_exporter.py's 2x + ladder starts at le="0.099999" (99999us, not 100000us), same as + mach-2's existing read-latency SLO. Objective corrected from + "<50ms" to "<100ms" to match the actual bucket boundary. sli: events: totalQuery: sum(rate(ceph_osd_op_w_latency_count[{{.window}}])) errorQuery: >- sum(rate(ceph_osd_op_w_latency_count[{{.window}}])) - - sum(rate(ceph_osd_op_w_latency_bucket{le="0.049999"}[{{.window}}])) + - sum(rate(ceph_osd_op_w_latency_bucket{le="0.099999"}[{{.window}}])) alerting: name: CephOSDLatencySLO labels: diff --git a/applications/infrastructure/sloth/values.yaml b/applications/infrastructure/sloth/values.yaml deleted file mode 100644 index 0b80ff3..0000000 --- a/applications/infrastructure/sloth/values.yaml +++ /dev/null @@ -1,17 +0,0 @@ -# Sloth's own SLI/SLO plugins cover everything the four PrometheusServiceLevel -# CRs below use (events + raw). The chart's default git-sync sidecar pulls -# community plugins from github.com/slok/sloth-common-sli-plugins on a poll -# loop — an extra container and an unnecessary external dependency for plugins -# this lab doesn't use. Disabled. -commonPlugins: - enabled: false - -# Pinned explicitly (matches the chart's own default) so a future chart bump -# can't silently change this — Sloth's footprint should stay near-zero. -resources: - requests: - cpu: 5m - memory: 75Mi - limits: - cpu: 50m - memory: 150Mi diff --git a/applications/rook/dashboards/ceph-observability-mach-2.json b/applications/rook/dashboards/ceph-observability-mach-2.json index 5498dfd..040a3fa 100644 --- a/applications/rook/dashboards/ceph-observability-mach-2.json +++ b/applications/rook/dashboards/ceph-observability-mach-2.json @@ -219,7 +219,7 @@ "type": "stat", "id": 9, "title": "Error budget remaining — the clock", - "description": "SLO (Sloth-backed, sloth_id=ceph-osd-latency): 99% of writes < 50ms, 30-day period. Was read-latency (99.9%/100ms) before this SLO's Sloth cutover — a real semantic change, not a rename. When it hits zero, you've blown the SLO.", + "description": "SLO (Sloth-backed, sloth_id=ceph-osd-latency): 99% of writes < 100ms, 30-day period (corrected from a guessed <50ms bucket boundary — le=\"0.049999\" doesn't exist, verified live 2026-07-03). Was read-latency (99.9%/100ms) before this SLO's Sloth cutover — a real semantic change, not a rename. When it hits zero, you've blown the SLO.", "gridPos": { "x": 0, "y": 31, "w": 8, "h": 4 }, "options": { "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }, diff --git a/justfile b/justfile index 722be36..fc9635e 100644 --- a/justfile +++ b/justfile @@ -28,3 +28,7 @@ ssh: # merge kubeconfig + SSH aliases after a restart kubeconfig: python3 provisioning/scripts/manage_k8s_config.py add + +# regenerate Prometheus rule groups from the Sloth SLO specs (requires sloth-cli, yq) +gen-slos: + provisioning/scripts/gen_slos.sh diff --git a/provisioning/scripts/gen_slos.sh b/provisioning/scripts/gen_slos.sh new file mode 100755 index 0000000..348bdee --- /dev/null +++ b/provisioning/scripts/gen_slos.sh @@ -0,0 +1,76 @@ +#!/usr/bin/env bash +# ceph-lab — gen_slos.sh +# +# Renders Sloth's PrometheusServiceLevel CRs into plain Prometheus rule +# groups and splices them into applications/infrastructure/prometheus/ +# values.yaml's serverFiles block. The standalone community `prometheus` +# chart used in this lab reads NO PrometheusRule CRs (no controller watches +# monitoring.coreos.com/v1 here) — Sloth's runtime Deployment generates +# correct PrometheusRules, but Prometheus never sees them. Rendering the +# CRs to plain rule groups at commit time (this script) and embedding them +# in the same ConfigMap Prometheus already loads (`rule_files:` in +# serverFiles.prometheus.yml already points at these two paths) is the fix. +# +# Run this after editing applications/infrastructure/sloth/prometheusservicelevels.yaml, +# then commit the diff. CI (see .github/workflows/slo-drift.yml) fails the +# build if this script's output would differ from what's committed. +set -euo pipefail + +REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" +SLO_SPEC="$REPO_ROOT/applications/infrastructure/sloth/prometheusservicelevels.yaml" +VALUES_FILE="$REPO_ROOT/applications/infrastructure/prometheus/values.yaml" + +command -v sloth >/dev/null 2>&1 || { echo "gen_slos.sh: 'sloth' CLI required (brew install sloth-cli)" >&2; exit 1; } +command -v yq >/dev/null 2>&1 || { echo "gen_slos.sh: 'yq' CLI required (brew install yq)" >&2; exit 1; } + +tmpdir="$(mktemp -d)" +trap 'rm -rf "$tmpdir"' EXIT + +sloth generate -i "$SLO_SPEC" --disable-alerts --no-color -o "$tmpdir/recording.k8s.yaml" +sloth generate -i "$SLO_SPEC" --disable-recordings --no-color -o "$tmpdir/alerting.k8s.yaml" + +# Sloth emits one PrometheusRule (k8s-wrapped) document per SLO. Prometheus's +# plain rule_files format wants a single top-level `groups:` list — merge +# every SLO's .spec.groups into one list, in source order. +yq eval-all '[.spec.groups[]] as $g ireduce ([]; . + $g) | {"groups": .}' \ + "$tmpdir/recording.k8s.yaml" > "$tmpdir/recording_rules.yml" +yq eval-all '[.spec.groups[]] as $g ireduce ([]; . + $g) | {"groups": .}' \ + "$tmpdir/alerting.k8s.yaml" > "$tmpdir/alerting_rules.yml" + +# Re-indent to nest as serverFiles.recording_rules.yml / .alerting_rules.yml +# siblings of serverFiles.prometheus.yml (2-space indent already used there). +{ + echo " # Source: applications/infrastructure/sloth/prometheusservicelevels.yaml" + echo " # DO NOT EDIT BY HAND — run \`just gen-slos\` after changing the source spec." + echo " recording_rules.yml:" + sed 's/^/ /' "$tmpdir/recording_rules.yml" +} > "$tmpdir/recording_block.yml" +{ + echo " # Source: applications/infrastructure/sloth/prometheusservicelevels.yaml" + echo " # DO NOT EDIT BY HAND — run \`just gen-slos\` after changing the source spec." + echo " alerting_rules.yml:" + sed 's/^/ /' "$tmpdir/alerting_rules.yml" +} > "$tmpdir/alerting_block.yml" + +splice_block() { + local begin_marker="$1" end_marker="$2" block_file="$3" + awk -v begin="$begin_marker" -v end="$end_marker" -v blockfile="$block_file" ' + $0 == begin { print; while ((getline line < blockfile) > 0) print line; skipping = 1; next } + $0 == end { skipping = 0; print; next } + skipping { next } + { print } + ' "$VALUES_FILE" > "$VALUES_FILE.tmp" + mv "$VALUES_FILE.tmp" "$VALUES_FILE" +} + +splice_block \ + " # --- BEGIN sloth-generated recording rules (regenerate: just gen-slos) ---" \ + " # --- END sloth-generated recording rules ---" \ + "$tmpdir/recording_block.yml" + +splice_block \ + " # --- BEGIN sloth-generated alerting rules (regenerate: just gen-slos) ---" \ + " # --- END sloth-generated alerting rules ---" \ + "$tmpdir/alerting_block.yml" + +echo "gen_slos.sh: regenerated recording_rules.yml / alerting_rules.yml in $VALUES_FILE"