PR1.5 (P0): fix rules-consumer gap + scrape drops blocking Sloth SLO verification

.github/workflows/slo-drift.yml

@@ -0,0 +1,37 @@
+name: SLO rule drift check
+
+# Fails if applications/infrastructure/sloth/prometheusservicelevels.yaml was
+# edited without re-running `just gen-slos` — the standalone `prometheus`
+# chart used in this lab reads no PrometheusRule CRs, so the rendered rule
+# groups embedded in applications/infrastructure/prometheus/values.yaml are
+# the only copy Prometheus ever loads. See provisioning/scripts/gen_slos.sh.
+on:
+  pull_request:
+    paths:
+      - "applications/infrastructure/sloth/prometheusservicelevels.yaml"
+      - "applications/infrastructure/prometheus/values.yaml"
+      - "provisioning/scripts/gen_slos.sh"
+
+jobs:
+  check-drift:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Install sloth CLI (pinned v0.16.0, matches the vendored chart version)
+        run: |
+          curl -sSL -o /usr/local/bin/sloth \
+            https://github.com/slok/sloth/releases/download/v0.16.0/sloth-linux-amd64
+          chmod +x /usr/local/bin/sloth
+          sloth version
+
+      - name: Install yq
+        uses: mikefarah/yq@v4.53.3
+
+      - name: Regenerate and diff
+        run: |
+          ./provisioning/scripts/gen_slos.sh
+          git diff --exit-code -- applications/infrastructure/prometheus/values.yaml || {
+            echo "::error::applications/infrastructure/prometheus/values.yaml is stale — run 'just gen-slos' and commit the result."
+            exit 1
+          }

CLAUDE.md

@@ -54,7 +54,8 @@ applications/
   config/             # gitops.env — single source of truth for all IPs, versions, hostnames
   infrastructure/     # one dir per infra component; each has config.json + kustomization.yaml
     ceph-latency-bridge/  # native OSD histogram reconstruction (wave 30)
-    sloth/            # PrometheusServiceLevel CRs + SLO burn-rate rules (wave 33)
+    sloth/            # PrometheusServiceLevel specs — NOT a deployed app; build-time input to
+                      # `just gen-slos` (renders into infrastructure/prometheus/values.yaml). See gotcha #24.
     topology-catalog/ # static catalog-info.yaml topology map, ConfigMap (wave 5)
     l7-policies/      # CiliumNetworkPolicies, organized by namespace subdirectory
       argocd/         # cnp-argocd.yaml
@@ -144,7 +145,6 @@ All policies live in `applications/infrastructure/l7-policies/`, organized into
 | 30 | rook-storage | true | true |
 | 30 | ceph-latency-bridge (SLO metrics) | true | true |
 | 31 | rook-dashboards (Grafana ConfigMaps) | true | true |
-| 33 | sloth (`PrometheusServiceLevel` CRs + SLO burn-rate rules) | true | true |
 | 35 | rook-gateway | true | true |
 
 Rook operator and rook-cluster prune/selfHeal settings are intentional — they protect Ceph data from accidental ArgoCD deletes.
@@ -165,7 +165,7 @@ Rook operator and rook-cluster prune/selfHeal settings are intentional — they
 10. **`--enable-helm` is required** — patched into `argocd-cm` via `cluster-bootstrap/argocd/kustomization.yaml`. Without it, `helmCharts:` stanzas are silently ignored.
 11. **OSD histogram `le` labels use high floating-point precision** — `ceph-latency-bridge` exports bucket boundaries like `le="0.099999"` not `le="0.1"`. PromQL queries in dashboards and recording rules must match these exact labels or return no data.
 12. **Histogram buckets must be in strictly ascending order** — custom exporters (like `ceph-latency-bridge`) must emit `le` values ascending with `+Inf` last. Out-of-order buckets cause Prometheus to silently discard the metric.
-13. **`ceph-observability-mach-2.json` is the definitive SLO dashboard** — 3-row narrative: Health → SLI → SLO. Uses per-OSD P99 lines and burn-rate alerting. Located in `applications/rook/dashboards/`. Row 3 is Sloth-backed (`sloth_id="ceph-osd-latency"`, **write** latency 99%/50ms — it was read latency 99.9%/100ms before the Sloth cutover, a real semantic change) — see gotcha #21/#22 before editing its queries.
+13. **`ceph-observability-mach-2.json` is the definitive SLO dashboard** — 3-row narrative: Health → SLI → SLO. Uses per-OSD P99 lines and burn-rate alerting. Located in `applications/rook/dashboards/`. Row 3 is Sloth-backed (`sloth_id="ceph-osd-latency"`, **write** latency 99%/100ms — it was read latency 99.9%/100ms before the Sloth cutover, a real semantic change) — see gotcha #21/#22 before editing its queries. The write-latency threshold was originally spec'd as <50ms (`le="0.049999"`) but that bucket doesn't exist; verified live 2026-07-03 against `ceph_osd_op_w_latency_bucket` and corrected to <100ms (`le="0.099999"`) — see gotcha #24.
 14. **Disabling ArgoCD selfHeal during manual fixes** — `selfHeal: true` will immediately revert `kubectl patch/apply` changes. Disable sync on `ceph-lab-root` first, then the specific Application, apply the fix, then push to git and re-enable.
 15. **virtiofs mounts are async** — Lima YAML provision steps poll for `/ceph-lab/provisioning/provision.env` (up to 60s) before running scripts. Scripts must not assume `/ceph-lab` is immediately available at VM boot.
 16. **`socket_vmnet` must be at `/opt/socket_vmnet`** — required for the `ceph-lab` host-only network (192.168.56.0/24) that gives VMs static IPs. Run `make setup` once after installing Lima.
@@ -183,6 +183,8 @@ Rook operator and rook-cluster prune/selfHeal settings are intentional — they
 21. **Sloth's `sloth_id` label is NOT the CR's `metadata.name`** — it's computed as `"{spec.service}-{slo.name}"` (confirmed against Sloth's own source, `internal/storage/io/k8s_sloth.go`). To land on a specific `sloth_id` (rattle's `SLO.ID` contract requires this — see `applications/infrastructure/sloth/prometheusservicelevels.yaml`), split `service`/`name` accordingly (e.g. `service: ceph-osd` + `name: latency` → `sloth_id: ceph-osd-latency`), not by renaming the CR.
 22. **Sloth's `slo:current_burn_rate:ratio` is a single un-windowed series**, computed off the page-alert's short window — there is no separate metadata metric per window (1h vs 6h). Anything wanting an explicit windowed burn rate (like `ceph-observability-mach-2.json`'s Row 3) must use `slo:sli_error:ratio_rate{1h,6h}{sloth_id=...} / (1 - objective)` instead — Sloth does generate per-window SLI error-ratio series, just not per-window burn-rate ones. Also: Sloth's metadata recording rules have no on/off flag — they're always generated, there's nothing to "enable."
 23. **Loki/Tempo/Alloy were already deployed here once and removed for OOM'ing `ceph-control`** (see `docs/trim-observability-stack.md`) — but that was on the old Vagrant/VMware topology at 3 GB. The Lima rewrite already doubled `ceph-control` to 6 GiB, and Lima's `vz` backend has substantially less virtualization overhead than Vagrant/VMware on top of that — real headroom exists now. Still: size any reintroduction (Loki/Promtail, Tempo/OTel Collector) with explicit resource requests/limits (the original incident's root cause was *unbounded* usage, not merely "too much") and a post-deploy OOM-watch verification gate, not by assuming the extra headroom makes limits unnecessary.
+24. **The standalone `prometheus` chart reads NO `monitoring.coreos.com` CRs** — no ServiceMonitor/PrometheusRule controller is deployed (`prometheus-operator-crds` ships CRD schemas only, no controller), so a live Sloth Deployment generating `PrometheusRule` CRs was pure overhead nothing consumed. Fix: `applications/infrastructure/sloth/prometheusservicelevels.yaml` is now a **build-time spec only** (the Sloth runtime chart + CRDs were retired from the cluster) — `provisioning/scripts/gen_slos.sh` (`just gen-slos`) runs the pinned `sloth` CLI (`brew install sloth-cli`, version must match the spec's last-tested chart version) to render it into plain rule groups spliced into `applications/infrastructure/prometheus/values.yaml`'s `serverFiles.recording_rules.yml` / `.alerting_rules.yml` (between `# --- BEGIN/END sloth-generated ... ---` markers — do not hand-edit inside them). `.github/workflows/slo-drift.yml` fails CI if the spec and the rendered rules disagree. Also folded in here: the `ceph-osd-latency` SLO's `le="0.049999"` bucket boundary was a guess that turned out wrong — verified live 2026-07-03 against `ceph_osd_op_w_latency_bucket`, corrected to `le="0.099999"` (objective is p99 **< 100ms**, not <50ms — update `ceph-observability-mach-2.json` Row 3's panel description too if this ever changes again).
+25. **Any new scrape target in a CNP'd namespace needs its port added to that namespace's `l7-visibility` allowlist** — the l7-policies make selected pods default-deny for unmatched ingress (`endpointSelector: {}` + explicit `toPorts` allowlist). Diagnosed via kube-state-metrics: its scrape job existed and KSM was healthy, but `monitoring/l7-visibility` only allowlisted 9090/3000/80, so every scrape timed out (`context deadline exceeded`) despite being same-node. Symptom looks like "no ServiceMonitor" or "target down"; check the CNP allowlist before anything else.
 
 ---
 

applications/infrastructure/l7-policies/monitoring/cnp-monitoring.yaml

@@ -15,6 +15,8 @@ spec:
             - { port: "9090", protocol: TCP } # Prometheus server
             - { port: "3000", protocol: TCP } # Grafana
             - { port: "80", protocol: TCP }   # Prometheus service port
+            - { port: "8080", protocol: TCP } # kube-state-metrics (metrics)
+            - { port: "8081", protocol: TCP } # kube-state-metrics (telemetry)
     - fromEntities: [ host, remote-node ]
   egress:
     - toEndpoints:

applications/infrastructure/l7-policies/rook-ceph/cnp-rook-ceph.yaml

@@ -8,15 +8,15 @@ metadata:
 spec:
   endpointSelector: {}
   ingress:
-    # RGW S3, MGR dashboard, Prometheus metrics, Rook operator metrics
+    # RGW S3, MGR dashboard, Prometheus metrics, Rook exporter metrics
+    # (2112 removed — no rook-ceph-operator Prometheus endpoint ever existed)
     - fromEntities: [cluster]
       toPorts:
         - ports:
             - { port: "80",   protocol: TCP }
             - { port: "7000", protocol: TCP }
             - { port: "9283", protocol: TCP }
             - { port: "8080", protocol: TCP }
-            - { port: "2112", protocol: TCP }
             - { port: "9926", protocol: TCP }
     # Rook operator webhook (TLS, L4 only)
     - fromEntities: [kube-apiserver]