Skip to content

fix(telemetry): fix SLO integrity issues (PR1.6) #11

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
IanEff merged 1 commit into from
Jul 3, 2026
Merged

fix(telemetry): fix SLO integrity issues (PR1.6) #11

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions CLAUDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -163,7 +163,7 @@ Rook operator and rook-cluster prune/selfHeal settings are intentional — they
8. **k3s uses SQLite, not etcd** — `kubeEtcd` scraper is disabled in Prometheus values.
9. **Gateway API CRDs must precede Cilium** — wave `-15` mirrors `install_cilium.sh` which applies CRDs first with `kubectl wait`.
10. **`--enable-helm` is required** — patched into `argocd-cm` via `cluster-bootstrap/argocd/kustomization.yaml`. Without it, `helmCharts:` stanzas are silently ignored.
11. **OSD histogram `le` labels use high floating-point precision** — `ceph-latency-bridge` exports bucket boundaries like `le="0.099999"` not `le="0.1"`. PromQL queries in dashboards and recording rules must match these exact labels or return no data.
11. **OSD histogram `le` labels use high floating-point precision** — `ceph-latency-bridge` exports bucket boundaries with millisecond units because the exporter divides raw nanosecond values by `1e6` (e.g. `le="102.399999"` represents ~102.4ms). PromQL queries in dashboards and recording rules must match these exact labels, or they will return no data.
12. **Histogram buckets must be in strictly ascending order** — custom exporters (like `ceph-latency-bridge`) must emit `le` values ascending with `+Inf` last. Out-of-order buckets cause Prometheus to silently discard the metric.
13. **`ceph-observability-mach-2.json` is the definitive SLO dashboard** — 3-row narrative: Health → SLI → SLO. Uses per-OSD P99 lines and burn-rate alerting. Located in `applications/rook/dashboards/`. Row 3 is Sloth-backed (`sloth_id="ceph-osd-latency"`, **write** latency 99%/100ms — it was read latency 99.9%/100ms before the Sloth cutover, a real semantic change) — see gotcha #21/#22 before editing its queries. The write-latency threshold was originally spec'd as <50ms (`le="0.049999"`) but that bucket doesn't exist; verified live 2026-07-03 against `ceph_osd_op_w_latency_bucket` and corrected to <100ms (`le="0.099999"`) — see gotcha #24.
14. **Disabling ArgoCD selfHeal during manual fixes** — `selfHeal: true` will immediately revert `kubectl patch/apply` changes. Disable sync on `ceph-lab-root` first, then the specific Application, apply the fix, then push to git and re-enable.
Expand All @@ -183,8 +183,9 @@ Rook operator and rook-cluster prune/selfHeal settings are intentional — they
21. **Sloth's `sloth_id` label is NOT the CR's `metadata.name`** — it's computed as `"{spec.service}-{slo.name}"` (confirmed against Sloth's own source, `internal/storage/io/k8s_sloth.go`). To land on a specific `sloth_id` (rattle's `SLO.ID` contract requires this — see `applications/infrastructure/sloth/prometheusservicelevels.yaml`), split `service`/`name` accordingly (e.g. `service: ceph-osd` + `name: latency` → `sloth_id: ceph-osd-latency`), not by renaming the CR.
22. **Sloth's `slo:current_burn_rate:ratio` is a single un-windowed series**, computed off the page-alert's short window — there is no separate metadata metric per window (1h vs 6h). Anything wanting an explicit windowed burn rate (like `ceph-observability-mach-2.json`'s Row 3) must use `slo:sli_error:ratio_rate{1h,6h}{sloth_id=...} / (1 - objective)` instead — Sloth does generate per-window SLI error-ratio series, just not per-window burn-rate ones. Also: Sloth's metadata recording rules have no on/off flag — they're always generated, there's nothing to "enable."
23. **Loki/Tempo/Alloy were already deployed here once and removed for OOM'ing `ceph-control`** (see `docs/trim-observability-stack.md`) — but that was on the old Vagrant/VMware topology at 3 GB. The Lima rewrite already doubled `ceph-control` to 6 GiB, and Lima's `vz` backend has substantially less virtualization overhead than Vagrant/VMware on top of that — real headroom exists now. Still: size any reintroduction (Loki/Promtail, Tempo/OTel Collector) with explicit resource requests/limits (the original incident's root cause was *unbounded* usage, not merely "too much") and a post-deploy OOM-watch verification gate, not by assuming the extra headroom makes limits unnecessary.
24. **The standalone `prometheus` chart reads NO `monitoring.coreos.com` CRs** — no ServiceMonitor/PrometheusRule controller is deployed (`prometheus-operator-crds` ships CRD schemas only, no controller), so a live Sloth Deployment generating `PrometheusRule` CRs was pure overhead nothing consumed. Fix: `applications/infrastructure/sloth/prometheusservicelevels.yaml` is now a **build-time spec only** (the Sloth runtime chart + CRDs were retired from the cluster) — `provisioning/scripts/gen_slos.sh` (`just gen-slos`) runs the pinned `sloth` CLI (`brew install sloth-cli`, version must match the spec's last-tested chart version) to render it into plain rule groups spliced into `applications/infrastructure/prometheus/values.yaml`'s `serverFiles.recording_rules.yml` / `.alerting_rules.yml` (between `# --- BEGIN/END sloth-generated ... ---` markers — do not hand-edit inside them). `.github/workflows/slo-drift.yml` fails CI if the spec and the rendered rules disagree. Also folded in here: the `ceph-osd-latency` SLO's `le="0.049999"` bucket boundary was a guess that turned out wrong — verified live 2026-07-03 against `ceph_osd_op_w_latency_bucket`, corrected to `le="0.099999"` (objective is p99 **< 100ms**, not <50ms — update `ceph-observability-mach-2.json` Row 3's panel description too if this ever changes again).
24. **The standalone `prometheus` chart reads NO `monitoring.coreos.com` CRs** — no ServiceMonitor/PrometheusRule controller is deployed (`prometheus-operator-crds` ships CRD schemas only, no controller), so a live Sloth Deployment generating `PrometheusRule` CRs was pure overhead nothing consumed. Fix: `applications/infrastructure/sloth/prometheusservicelevels.yaml` is now a **build-time spec only** (the Sloth runtime chart + CRDs were retired from the cluster) — `provisioning/scripts/gen_slos.sh` (`just gen-slos`) runs the pinned `sloth` CLI (`brew install sloth-cli`, version must match the spec's last-tested chart version) to render it into plain rule groups spliced into `applications/infrastructure/prometheus/values.yaml`'s `serverFiles.recording_rules.yml` / `.alerting_rules.yml` (between `# --- BEGIN/END sloth-generated ... ---` markers — do not hand-edit inside them). `.github/workflows/slo-drift.yml` fails CI if the spec and the rendered rules disagree. Also folded in here: the `ceph-osd-latency` SLO's `le="0.049999"` bucket boundary was a guess that turned out wrong — verified live 2026-07-03 against `ceph_osd_op_w_latency_bucket`, corrected to `le="102.399999"` (objective is p99 **< 100ms**, not <50ms; also requires filtering count/bucket to `{job="ceph-latency-bridge"}` to avoid count inflation from double scrapes).
25. **Any new scrape target in a CNP'd namespace needs its port added to that namespace's `l7-visibility` allowlist** — the l7-policies make selected pods default-deny for unmatched ingress (`endpointSelector: {}` + explicit `toPorts` allowlist). Diagnosed via kube-state-metrics: its scrape job existed and KSM was healthy, but `monitoring/l7-visibility` only allowlisted 9090/3000/80, so every scrape timed out (`context deadline exceeded`) despite being same-node. Symptom looks like "no ServiceMonitor" or "target down"; check the CNP allowlist before anything else.
26. **Sloth SLIs must aggregate to a singleton series** — any scrape source scraped multiple times (e.g., `mgr` pod double-scraped by the annotation-based `kubernetes-pods` job and the explicit `rook-ceph-mgr` job) will duplicate metric streams. If the SLI doesn't aggregate (e.g., wrapping raw queries in `max()` or `sum()`), multiple series are written per `sloth_id` (like `slo:current_burn_rate:ratio`). Downstream systems like rattle's `BurnSamples` take the first series (`Result[0]`), silently ignoring duplicates, which leads to flapping behavior if scrape paths go out of sync.

---

Expand Down
3 changes: 2 additions & 1 deletion GEMINI.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,11 +81,12 @@ This project contains custom Gemini CLI skills in `.github/skills/`. For GitOps
4. **`CephFilesystemSubVolumeGroup` is required** (Rook ≥ v1.17) — without it, CephFS dynamic provisioning silently fails.
5. **ArgoCD runs insecure** — TLS terminates at the Cilium Gateway. Service URLs work at https://*.ceph.lab.
6. **L7 Policies (CNP) are in effect** — Cilium Network Policies in `applications/infrastructure/l7-policies/` enforce strict network isolation.
7. **OSD Performance Histograms precision** — Ceph OSD histograms (via `ceph-latency-bridge`) export bucket boundaries with high floating-point precision (e.g., `le="0.099999"`). PromQL queries MUST match these exact labels.
7. **OSD Performance Histograms precision & units** — Ceph OSD histograms (via `ceph-latency-bridge`) export bucket boundaries with millisecond units because the exporter divides raw nanosecond values by `1e6` (e.g., `le="102.399999"` represents ~102.4ms). PromQL queries MUST match these exact labels.
8. **`virtiofs` mounts are async** — Lima YAML provision steps poll for `/ceph-lab/provisioning/provision.env` (up to 60s) before running scripts.
9. **`socket_vmnet` requirement** — Must be at `/opt/socket_vmnet` for the 192.168.56.0/24 host-only network. Run `make setup` once.
10. **`lima0` default route break** — DHCP on `lima0` must have `use-routes: false` to prevent pod egress breakage (already in templates).
11. **Definitive Observability Dashboard** — `ceph-observability-mach-2.json` is the definitive reference for Health → SLI → SLO.
12. **Sloth SLIs must aggregate to a singleton series** — any scrape source scraped multiple times (e.g., `mgr` pod double-scraped by the annotation-based `kubernetes-pods` job and the explicit `rook-ceph-mgr` job) will duplicate metric streams. If the SLI doesn't aggregate (e.g., wrapping raw queries in `max()`), multiple series are written per `sloth_id` (like `slo:current_burn_rate:ratio`), which causes downstream systems like rattle to behave unpredictably.

---

Expand Down
12 changes: 12 additions & 0 deletions _system/log.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,16 @@
# Operations Log

## [2026-07-03] - PR1.6: SLO Integrity Fixes

### Action
- Cut branch `fix/slo-integrity-pr1.6`.
- Modified `applications/infrastructure/sloth/prometheusservicelevels.yaml`:
- Aligned OSD write latency SLO to query `job="ceph-latency-bridge"` and use `le="102.399999"` (since the exporter divides raw nanosecond values by `1e6`, making values millisecond-based).
- Wrapped `ceph-health` raw SLI query in `max()` to aggregate away multiple series resulting from double-scraped MGR target.
- Updated `applications/rook/dashboards/prototype-observability.json` to use `le="102.399999"` bucket and filter to `job="ceph-latency-bridge"` for OSD latency panels.
- Regenerated Prometheus rules by running `just gen-slos` which rendered rule groups inside `applications/infrastructure/prometheus/values.yaml`.
- Updated `CLAUDE.md` and `GEMINI.md` gotchas to document both the millisecond-based OSD latency bucket configuration and the requirement for Sloth SLIs to aggregate to singleton series.

## [2026-07-03] - Deploy Tempo & OTel Collector, Enable Tracing

### Action
Expand All @@ -17,3 +28,4 @@
- Identified that `ceph-rgw-availability` SLO rules were querying `job="rook-ceph-mgr"` but the metrics are exported under `job="rook-ceph-exporter"`.
- Removed the incorrect `{job="rook-ceph-mgr"}` filter in `applications/infrastructure/sloth/prometheusservicelevels.yaml`.
- Regenerated Prometheus rules via `just gen-slos`.
- Amended PR1.6 to drop explicit duplicate jobs `rook-ceph-mgr` and `rook-ceph-exporter` in Prometheus values since the `kubernetes-pods` annotation path covers them.
61 changes: 21 additions & 40 deletions applications/infrastructure/prometheus/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,26 +86,7 @@ serverFiles:
# The chart's default scrape_configs already cover kubernetes-service-endpoints,
# kubernetes-nodes, kubernetes-pods, etc. Only add custom targets below.

# Rook-ceph MGR metrics (port 9283) — scraped directly regardless of annotation.
- job_name: "rook-ceph-mgr"
static_configs:
- targets:
- "rook-ceph-mgr.rook-ceph.svc.cluster.local:9283"
relabel_configs:
- target_label: job
replacement: rook-ceph-mgr

# Rook exporter metrics (port 9926) — the real Ceph-cluster-wide
# exporter Rook ships. (There is no "rook-ceph-operator:2112" endpoint;
# the operator itself exposes no Prometheus metrics — a prior version
# of this job scraped a Service that never existed.)
- job_name: "rook-ceph-exporter"
static_configs:
- targets:
- "rook-ceph-exporter.rook-ceph.svc.cluster.local:9926"
relabel_configs:
- target_label: job
replacement: rook-ceph-exporter

# Hubble metrics (Cilium flow observability).
- job_name: "hubble-metrics"
Expand Down Expand Up @@ -294,9 +275,9 @@ serverFiles:
- name: sloth-slo-sli-recordings-ceph-osd-latency
rules:
- expr: |
(sum(rate(ceph_osd_op_w_latency_count[5m])) - sum(rate(ceph_osd_op_w_latency_bucket{le="0.099999"}[5m])))
(sum(rate(ceph_osd_op_w_latency_count{job="ceph-latency-bridge"}[5m])) - sum(rate(ceph_osd_op_w_latency_bucket{job="ceph-latency-bridge",le="102.399999"}[5m])))
/
(sum(rate(ceph_osd_op_w_latency_count[5m])))
(sum(rate(ceph_osd_op_w_latency_count{job="ceph-latency-bridge"}[5m])))
labels:
lab: ceph-lab
sloth_id: ceph-osd-latency
Expand All @@ -305,9 +286,9 @@ serverFiles:
sloth_window: 5m
record: slo:sli_error:ratio_rate5m
- expr: |
(sum(rate(ceph_osd_op_w_latency_count[30m])) - sum(rate(ceph_osd_op_w_latency_bucket{le="0.099999"}[30m])))
(sum(rate(ceph_osd_op_w_latency_count{job="ceph-latency-bridge"}[30m])) - sum(rate(ceph_osd_op_w_latency_bucket{job="ceph-latency-bridge",le="102.399999"}[30m])))
/
(sum(rate(ceph_osd_op_w_latency_count[30m])))
(sum(rate(ceph_osd_op_w_latency_count{job="ceph-latency-bridge"}[30m])))
labels:
lab: ceph-lab
sloth_id: ceph-osd-latency
Expand All @@ -316,9 +297,9 @@ serverFiles:
sloth_window: 30m
record: slo:sli_error:ratio_rate30m
- expr: |
(sum(rate(ceph_osd_op_w_latency_count[1h])) - sum(rate(ceph_osd_op_w_latency_bucket{le="0.099999"}[1h])))
(sum(rate(ceph_osd_op_w_latency_count{job="ceph-latency-bridge"}[1h])) - sum(rate(ceph_osd_op_w_latency_bucket{job="ceph-latency-bridge",le="102.399999"}[1h])))
/
(sum(rate(ceph_osd_op_w_latency_count[1h])))
(sum(rate(ceph_osd_op_w_latency_count{job="ceph-latency-bridge"}[1h])))
labels:
lab: ceph-lab
sloth_id: ceph-osd-latency
Expand All @@ -327,9 +308,9 @@ serverFiles:
sloth_window: 1h
record: slo:sli_error:ratio_rate1h
- expr: |
(sum(rate(ceph_osd_op_w_latency_count[2h])) - sum(rate(ceph_osd_op_w_latency_bucket{le="0.099999"}[2h])))
(sum(rate(ceph_osd_op_w_latency_count{job="ceph-latency-bridge"}[2h])) - sum(rate(ceph_osd_op_w_latency_bucket{job="ceph-latency-bridge",le="102.399999"}[2h])))
/
(sum(rate(ceph_osd_op_w_latency_count[2h])))
(sum(rate(ceph_osd_op_w_latency_count{job="ceph-latency-bridge"}[2h])))
labels:
lab: ceph-lab
sloth_id: ceph-osd-latency
Expand All @@ -338,9 +319,9 @@ serverFiles:
sloth_window: 2h
record: slo:sli_error:ratio_rate2h
- expr: |
(sum(rate(ceph_osd_op_w_latency_count[6h])) - sum(rate(ceph_osd_op_w_latency_bucket{le="0.099999"}[6h])))
(sum(rate(ceph_osd_op_w_latency_count{job="ceph-latency-bridge"}[6h])) - sum(rate(ceph_osd_op_w_latency_bucket{job="ceph-latency-bridge",le="102.399999"}[6h])))
/
(sum(rate(ceph_osd_op_w_latency_count[6h])))
(sum(rate(ceph_osd_op_w_latency_count{job="ceph-latency-bridge"}[6h])))
labels:
lab: ceph-lab
sloth_id: ceph-osd-latency
Expand All @@ -349,9 +330,9 @@ serverFiles:
sloth_window: 6h
record: slo:sli_error:ratio_rate6h
- expr: |
(sum(rate(ceph_osd_op_w_latency_count[1d])) - sum(rate(ceph_osd_op_w_latency_bucket{le="0.099999"}[1d])))
(sum(rate(ceph_osd_op_w_latency_count{job="ceph-latency-bridge"}[1d])) - sum(rate(ceph_osd_op_w_latency_bucket{job="ceph-latency-bridge",le="102.399999"}[1d])))
/
(sum(rate(ceph_osd_op_w_latency_count[1d])))
(sum(rate(ceph_osd_op_w_latency_count{job="ceph-latency-bridge"}[1d])))
labels:
lab: ceph-lab
sloth_id: ceph-osd-latency
Expand All @@ -360,9 +341,9 @@ serverFiles:
sloth_window: 1d
record: slo:sli_error:ratio_rate1d
- expr: |
(sum(rate(ceph_osd_op_w_latency_count[3d])) - sum(rate(ceph_osd_op_w_latency_bucket{le="0.099999"}[3d])))
(sum(rate(ceph_osd_op_w_latency_count{job="ceph-latency-bridge"}[3d])) - sum(rate(ceph_osd_op_w_latency_bucket{job="ceph-latency-bridge",le="102.399999"}[3d])))
/
(sum(rate(ceph_osd_op_w_latency_count[3d])))
(sum(rate(ceph_osd_op_w_latency_count{job="ceph-latency-bridge"}[3d])))
labels:
lab: ceph-lab
sloth_id: ceph-osd-latency
Expand Down Expand Up @@ -444,55 +425,55 @@ serverFiles:
record: sloth_slo_info
- name: sloth-slo-sli-recordings-ceph-health
rules:
- expr: (avg_over_time(clamp_max(ceph_health_status, 1)[5m:1m]))
- expr: (max(avg_over_time(clamp_max(ceph_health_status, 1)[5m:1m])))
labels:
lab: ceph-lab
sloth_id: ceph-health
sloth_service: ceph
sloth_slo: health
sloth_window: 5m
record: slo:sli_error:ratio_rate5m
- expr: (avg_over_time(clamp_max(ceph_health_status, 1)[30m:1m]))
- expr: (max(avg_over_time(clamp_max(ceph_health_status, 1)[30m:1m])))
labels:
lab: ceph-lab
sloth_id: ceph-health
sloth_service: ceph
sloth_slo: health
sloth_window: 30m
record: slo:sli_error:ratio_rate30m
- expr: (avg_over_time(clamp_max(ceph_health_status, 1)[1h:1m]))
- expr: (max(avg_over_time(clamp_max(ceph_health_status, 1)[1h:1m])))
labels:
lab: ceph-lab
sloth_id: ceph-health
sloth_service: ceph
sloth_slo: health
sloth_window: 1h
record: slo:sli_error:ratio_rate1h
- expr: (avg_over_time(clamp_max(ceph_health_status, 1)[2h:1m]))
- expr: (max(avg_over_time(clamp_max(ceph_health_status, 1)[2h:1m])))
labels:
lab: ceph-lab
sloth_id: ceph-health
sloth_service: ceph
sloth_slo: health
sloth_window: 2h
record: slo:sli_error:ratio_rate2h
- expr: (avg_over_time(clamp_max(ceph_health_status, 1)[6h:1m]))
- expr: (max(avg_over_time(clamp_max(ceph_health_status, 1)[6h:1m])))
labels:
lab: ceph-lab
sloth_id: ceph-health
sloth_service: ceph
sloth_slo: health
sloth_window: 6h
record: slo:sli_error:ratio_rate6h
- expr: (avg_over_time(clamp_max(ceph_health_status, 1)[1d:1m]))
- expr: (max(avg_over_time(clamp_max(ceph_health_status, 1)[1d:1m])))
labels:
lab: ceph-lab
sloth_id: ceph-health
sloth_service: ceph
sloth_slo: health
sloth_window: 1d
record: slo:sli_error:ratio_rate1d
- expr: (avg_over_time(clamp_max(ceph_health_status, 1)[3d:1m]))
- expr: (max(avg_over_time(clamp_max(ceph_health_status, 1)[3d:1m])))
labels:
lab: ceph-lab
sloth_id: ceph-health
Expand Down
Loading
Loading