PR1.5 (P0): fix rules-consumer gap + scrape drops blocking Sloth SLO verification

[IanEff]

Summary

Live verification of PR1 (#6, Sloth SLOs) failed: slo:current_burn_rate:ratio returned nothing for all four SLOs. This PR is the diagnosed fix (the "F1-F4" findings from 2026-07-03's live diagnosis) that unblocks re-running that verification.

• F1 — Sloth's rules never loaded, root-caused: this lab runs the standalone community prometheus chart, which reads no monitoring.coreos.com CRs (no ServiceMonitor/PrometheusRule controller — prometheus-operator-crds ships CRD schemas only). Sloth's Deployment was generating perfectly correct PrometheusRule CRs that nothing ever consumed.

  • Fix: retired the Sloth runtime chart + CRDs entirely (deleted applications/infrastructure/sloth/{config.json,kustomization.yaml,values.yaml}). prometheusservicelevels.yaml is now a build-time-only spec.
  • Added provisioning/scripts/gen_slos.sh (just gen-slos) — runs the pinned sloth CLI (v0.16.0, matching the retired chart version) against the spec and splices the rendered rule groups into applications/infrastructure/prometheus/values.yaml's serverFiles.recording_rules.yml / .alerting_rules.yml, which is exactly where serverFiles.prometheus.yml's rule_files: already pointed (wired and waiting since PR1).
  • Added .github/workflows/slo-drift.yml — fails PRs where the spec and rendered rules disagree.
  • Verified locally: kustomize build --enable-helm applications/infrastructure/prometheus renders a ConfigMap with real groups: content under both keys; gen_slos.sh is idempotent (reran twice, zero diff).

• F2 — kube-state-metrics scrape silently dropped by CNP: KSM's job existed, KSM was healthy, but monitoring/l7-visibility's CiliumNetworkPolicy only allowlisted ports 9090/3000/80 — every same-node scrape to KSM's 8080/8081 timed out (context deadline exceeded). Added 8080/8081 to the allowlist.

• F3 — fictional scrape job removed: rook-ceph-operator:2112 targeted a Service that has never existed (the Rook operator ships no Prometheus endpoint). Deleted the scrape job and the matching CNP rule; added the real rook-ceph-exporter:9926 job instead (verified live: the Service exists with that exact port).

• F4 — wrong histogram bucket, verified live: ceph-osd-latency's le="0.049999" was a reasoned-but-wrong guess — that bucket doesn't exist. ceph_osd_op_w_latency_bucket is a 2× ladder starting at le="0.099999". Corrected the SLI query; the SLO's real objective is p99 < 100ms, not <50ms. Updated the SLO description and ceph-observability-mach-2.json Row 3's panel description to match (previously said "< 50ms").

Also updated CLAUDE.md: removed the sloth wave-33 row from the sync-wave table (no longer a deployed app), added gotcha #24 (build-time SLO rendering pipeline) and #25 (CNP port allowlisting for new scrape targets in CNP'd namespaces).

Cluster-affecting note

The live sloth ArgoCD Application has the resources-finalizer.argocd.argoproj.io finalizer set, so once this is merged and ceph-lab-root syncs, ArgoCD will cascade-delete the Sloth Deployment, its CRDs, and the 4 PrometheusServiceLevel CRs from the cluster. This is intentional (that's the "retire the runtime app" part of F1) but worth knowing before you sync.

Test plan

• kustomize build --enable-helm applications/infrastructure/prometheus succeeds, renders recording_rules.yml/alerting_rules.yml with real rule groups
• kustomize build --enable-helm applications/infrastructure/l7-policies succeeds
• gen_slos.sh re-run twice produces zero diff (idempotent)
• rook-ceph-exporter Service confirmed live at rook-ceph-exporter.rook-ceph.svc.cluster.local:9926
• Confirmed no remaining le="0.049999" references outside historical/explanatory comments
• After merge + sync: confirm all four slo:current_burn_rate:ratio{sloth_id=...} series appear in Prometheus (the actual PR1 verification gate)
• After merge + sync: confirm KSM targets show up in Prometheus
• After merge + sync: confirm the Sloth Deployment/CRDs were cleanly pruned (no orphaned resources)

🤖 Generated with Claude Code