diff --git a/CLAUDE.md b/CLAUDE.md index bd49008..9a86521 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -163,7 +163,7 @@ Rook operator and rook-cluster prune/selfHeal settings are intentional — they 8. **k3s uses SQLite, not etcd** — `kubeEtcd` scraper is disabled in Prometheus values. 9. **Gateway API CRDs must precede Cilium** — wave `-15` mirrors `install_cilium.sh` which applies CRDs first with `kubectl wait`. 10. **`--enable-helm` is required** — patched into `argocd-cm` via `cluster-bootstrap/argocd/kustomization.yaml`. Without it, `helmCharts:` stanzas are silently ignored. -11. **OSD histogram `le` labels use high floating-point precision** — `ceph-latency-bridge` exports bucket boundaries like `le="0.099999"` not `le="0.1"`. PromQL queries in dashboards and recording rules must match these exact labels or return no data. +11. **OSD histogram `le` labels use high floating-point precision** — `ceph-latency-bridge` exports bucket boundaries with millisecond units because the exporter divides raw nanosecond values by `1e6` (e.g. `le="102.399999"` represents ~102.4ms). PromQL queries in dashboards and recording rules must match these exact labels, or they will return no data. 12. **Histogram buckets must be in strictly ascending order** — custom exporters (like `ceph-latency-bridge`) must emit `le` values ascending with `+Inf` last. Out-of-order buckets cause Prometheus to silently discard the metric. 13. **`ceph-observability-mach-2.json` is the definitive SLO dashboard** — 3-row narrative: Health → SLI → SLO. Uses per-OSD P99 lines and burn-rate alerting. Located in `applications/rook/dashboards/`. Row 3 is Sloth-backed (`sloth_id="ceph-osd-latency"`, **write** latency 99%/100ms — it was read latency 99.9%/100ms before the Sloth cutover, a real semantic change) — see gotcha #21/#22 before editing its queries. The write-latency threshold was originally spec'd as <50ms (`le="0.049999"`) but that bucket doesn't exist; verified live 2026-07-03 against `ceph_osd_op_w_latency_bucket` and corrected to <100ms (`le="0.099999"`) — see gotcha #24. 14. **Disabling ArgoCD selfHeal during manual fixes** — `selfHeal: true` will immediately revert `kubectl patch/apply` changes. Disable sync on `ceph-lab-root` first, then the specific Application, apply the fix, then push to git and re-enable. @@ -183,8 +183,9 @@ Rook operator and rook-cluster prune/selfHeal settings are intentional — they 21. **Sloth's `sloth_id` label is NOT the CR's `metadata.name`** — it's computed as `"{spec.service}-{slo.name}"` (confirmed against Sloth's own source, `internal/storage/io/k8s_sloth.go`). To land on a specific `sloth_id` (rattle's `SLO.ID` contract requires this — see `applications/infrastructure/sloth/prometheusservicelevels.yaml`), split `service`/`name` accordingly (e.g. `service: ceph-osd` + `name: latency` → `sloth_id: ceph-osd-latency`), not by renaming the CR. 22. **Sloth's `slo:current_burn_rate:ratio` is a single un-windowed series**, computed off the page-alert's short window — there is no separate metadata metric per window (1h vs 6h). Anything wanting an explicit windowed burn rate (like `ceph-observability-mach-2.json`'s Row 3) must use `slo:sli_error:ratio_rate{1h,6h}{sloth_id=...} / (1 - objective)` instead — Sloth does generate per-window SLI error-ratio series, just not per-window burn-rate ones. Also: Sloth's metadata recording rules have no on/off flag — they're always generated, there's nothing to "enable." 23. **Loki/Tempo/Alloy were already deployed here once and removed for OOM'ing `ceph-control`** (see `docs/trim-observability-stack.md`) — but that was on the old Vagrant/VMware topology at 3 GB. The Lima rewrite already doubled `ceph-control` to 6 GiB, and Lima's `vz` backend has substantially less virtualization overhead than Vagrant/VMware on top of that — real headroom exists now. Still: size any reintroduction (Loki/Promtail, Tempo/OTel Collector) with explicit resource requests/limits (the original incident's root cause was *unbounded* usage, not merely "too much") and a post-deploy OOM-watch verification gate, not by assuming the extra headroom makes limits unnecessary. -24. **The standalone `prometheus` chart reads NO `monitoring.coreos.com` CRs** — no ServiceMonitor/PrometheusRule controller is deployed (`prometheus-operator-crds` ships CRD schemas only, no controller), so a live Sloth Deployment generating `PrometheusRule` CRs was pure overhead nothing consumed. Fix: `applications/infrastructure/sloth/prometheusservicelevels.yaml` is now a **build-time spec only** (the Sloth runtime chart + CRDs were retired from the cluster) — `provisioning/scripts/gen_slos.sh` (`just gen-slos`) runs the pinned `sloth` CLI (`brew install sloth-cli`, version must match the spec's last-tested chart version) to render it into plain rule groups spliced into `applications/infrastructure/prometheus/values.yaml`'s `serverFiles.recording_rules.yml` / `.alerting_rules.yml` (between `# --- BEGIN/END sloth-generated ... ---` markers — do not hand-edit inside them). `.github/workflows/slo-drift.yml` fails CI if the spec and the rendered rules disagree. Also folded in here: the `ceph-osd-latency` SLO's `le="0.049999"` bucket boundary was a guess that turned out wrong — verified live 2026-07-03 against `ceph_osd_op_w_latency_bucket`, corrected to `le="0.099999"` (objective is p99 **< 100ms**, not <50ms — update `ceph-observability-mach-2.json` Row 3's panel description too if this ever changes again). +24. **The standalone `prometheus` chart reads NO `monitoring.coreos.com` CRs** — no ServiceMonitor/PrometheusRule controller is deployed (`prometheus-operator-crds` ships CRD schemas only, no controller), so a live Sloth Deployment generating `PrometheusRule` CRs was pure overhead nothing consumed. Fix: `applications/infrastructure/sloth/prometheusservicelevels.yaml` is now a **build-time spec only** (the Sloth runtime chart + CRDs were retired from the cluster) — `provisioning/scripts/gen_slos.sh` (`just gen-slos`) runs the pinned `sloth` CLI (`brew install sloth-cli`, version must match the spec's last-tested chart version) to render it into plain rule groups spliced into `applications/infrastructure/prometheus/values.yaml`'s `serverFiles.recording_rules.yml` / `.alerting_rules.yml` (between `# --- BEGIN/END sloth-generated ... ---` markers — do not hand-edit inside them). `.github/workflows/slo-drift.yml` fails CI if the spec and the rendered rules disagree. Also folded in here: the `ceph-osd-latency` SLO's `le="0.049999"` bucket boundary was a guess that turned out wrong — verified live 2026-07-03 against `ceph_osd_op_w_latency_bucket`, corrected to `le="102.399999"` (objective is p99 **< 100ms**, not <50ms; also requires filtering count/bucket to `{job="ceph-latency-bridge"}` to avoid count inflation from double scrapes). 25. **Any new scrape target in a CNP'd namespace needs its port added to that namespace's `l7-visibility` allowlist** — the l7-policies make selected pods default-deny for unmatched ingress (`endpointSelector: {}` + explicit `toPorts` allowlist). Diagnosed via kube-state-metrics: its scrape job existed and KSM was healthy, but `monitoring/l7-visibility` only allowlisted 9090/3000/80, so every scrape timed out (`context deadline exceeded`) despite being same-node. Symptom looks like "no ServiceMonitor" or "target down"; check the CNP allowlist before anything else. +26. **Sloth SLIs must aggregate to a singleton series** — any scrape source scraped multiple times (e.g., `mgr` pod double-scraped by the annotation-based `kubernetes-pods` job and the explicit `rook-ceph-mgr` job) will duplicate metric streams. If the SLI doesn't aggregate (e.g., wrapping raw queries in `max()` or `sum()`), multiple series are written per `sloth_id` (like `slo:current_burn_rate:ratio`). Downstream systems like rattle's `BurnSamples` take the first series (`Result[0]`), silently ignoring duplicates, which leads to flapping behavior if scrape paths go out of sync. --- diff --git a/GEMINI.md b/GEMINI.md index 510ba37..d6fb203 100644 --- a/GEMINI.md +++ b/GEMINI.md @@ -81,11 +81,12 @@ This project contains custom Gemini CLI skills in `.github/skills/`. For GitOps 4. **`CephFilesystemSubVolumeGroup` is required** (Rook ≥ v1.17) — without it, CephFS dynamic provisioning silently fails. 5. **ArgoCD runs insecure** — TLS terminates at the Cilium Gateway. Service URLs work at https://*.ceph.lab. 6. **L7 Policies (CNP) are in effect** — Cilium Network Policies in `applications/infrastructure/l7-policies/` enforce strict network isolation. -7. **OSD Performance Histograms precision** — Ceph OSD histograms (via `ceph-latency-bridge`) export bucket boundaries with high floating-point precision (e.g., `le="0.099999"`). PromQL queries MUST match these exact labels. +7. **OSD Performance Histograms precision & units** — Ceph OSD histograms (via `ceph-latency-bridge`) export bucket boundaries with millisecond units because the exporter divides raw nanosecond values by `1e6` (e.g., `le="102.399999"` represents ~102.4ms). PromQL queries MUST match these exact labels. 8. **`virtiofs` mounts are async** — Lima YAML provision steps poll for `/ceph-lab/provisioning/provision.env` (up to 60s) before running scripts. 9. **`socket_vmnet` requirement** — Must be at `/opt/socket_vmnet` for the 192.168.56.0/24 host-only network. Run `make setup` once. 10. **`lima0` default route break** — DHCP on `lima0` must have `use-routes: false` to prevent pod egress breakage (already in templates). 11. **Definitive Observability Dashboard** — `ceph-observability-mach-2.json` is the definitive reference for Health → SLI → SLO. +12. **Sloth SLIs must aggregate to a singleton series** — any scrape source scraped multiple times (e.g., `mgr` pod double-scraped by the annotation-based `kubernetes-pods` job and the explicit `rook-ceph-mgr` job) will duplicate metric streams. If the SLI doesn't aggregate (e.g., wrapping raw queries in `max()`), multiple series are written per `sloth_id` (like `slo:current_burn_rate:ratio`), which causes downstream systems like rattle to behave unpredictably. --- diff --git a/_system/log.md b/_system/log.md index b82014b..d740b33 100644 --- a/_system/log.md +++ b/_system/log.md @@ -1,5 +1,16 @@ # Operations Log +## [2026-07-03] - PR1.6: SLO Integrity Fixes + +### Action +- Cut branch `fix/slo-integrity-pr1.6`. +- Modified `applications/infrastructure/sloth/prometheusservicelevels.yaml`: + - Aligned OSD write latency SLO to query `job="ceph-latency-bridge"` and use `le="102.399999"` (since the exporter divides raw nanosecond values by `1e6`, making values millisecond-based). + - Wrapped `ceph-health` raw SLI query in `max()` to aggregate away multiple series resulting from double-scraped MGR target. +- Updated `applications/rook/dashboards/prototype-observability.json` to use `le="102.399999"` bucket and filter to `job="ceph-latency-bridge"` for OSD latency panels. +- Regenerated Prometheus rules by running `just gen-slos` which rendered rule groups inside `applications/infrastructure/prometheus/values.yaml`. +- Updated `CLAUDE.md` and `GEMINI.md` gotchas to document both the millisecond-based OSD latency bucket configuration and the requirement for Sloth SLIs to aggregate to singleton series. + ## [2026-07-03] - Deploy Tempo & OTel Collector, Enable Tracing ### Action @@ -17,3 +28,4 @@ - Identified that `ceph-rgw-availability` SLO rules were querying `job="rook-ceph-mgr"` but the metrics are exported under `job="rook-ceph-exporter"`. - Removed the incorrect `{job="rook-ceph-mgr"}` filter in `applications/infrastructure/sloth/prometheusservicelevels.yaml`. - Regenerated Prometheus rules via `just gen-slos`. +- Amended PR1.6 to drop explicit duplicate jobs `rook-ceph-mgr` and `rook-ceph-exporter` in Prometheus values since the `kubernetes-pods` annotation path covers them. diff --git a/applications/infrastructure/prometheus/values.yaml b/applications/infrastructure/prometheus/values.yaml index e85dccf..acc5911 100644 --- a/applications/infrastructure/prometheus/values.yaml +++ b/applications/infrastructure/prometheus/values.yaml @@ -86,26 +86,7 @@ serverFiles: # The chart's default scrape_configs already cover kubernetes-service-endpoints, # kubernetes-nodes, kubernetes-pods, etc. Only add custom targets below. - # Rook-ceph MGR metrics (port 9283) — scraped directly regardless of annotation. - - job_name: "rook-ceph-mgr" - static_configs: - - targets: - - "rook-ceph-mgr.rook-ceph.svc.cluster.local:9283" - relabel_configs: - - target_label: job - replacement: rook-ceph-mgr - # Rook exporter metrics (port 9926) — the real Ceph-cluster-wide - # exporter Rook ships. (There is no "rook-ceph-operator:2112" endpoint; - # the operator itself exposes no Prometheus metrics — a prior version - # of this job scraped a Service that never existed.) - - job_name: "rook-ceph-exporter" - static_configs: - - targets: - - "rook-ceph-exporter.rook-ceph.svc.cluster.local:9926" - relabel_configs: - - target_label: job - replacement: rook-ceph-exporter # Hubble metrics (Cilium flow observability). - job_name: "hubble-metrics" @@ -294,9 +275,9 @@ serverFiles: - name: sloth-slo-sli-recordings-ceph-osd-latency rules: - expr: | - (sum(rate(ceph_osd_op_w_latency_count[5m])) - sum(rate(ceph_osd_op_w_latency_bucket{le="0.099999"}[5m]))) + (sum(rate(ceph_osd_op_w_latency_count{job="ceph-latency-bridge"}[5m])) - sum(rate(ceph_osd_op_w_latency_bucket{job="ceph-latency-bridge",le="102.399999"}[5m]))) / - (sum(rate(ceph_osd_op_w_latency_count[5m]))) + (sum(rate(ceph_osd_op_w_latency_count{job="ceph-latency-bridge"}[5m]))) labels: lab: ceph-lab sloth_id: ceph-osd-latency @@ -305,9 +286,9 @@ serverFiles: sloth_window: 5m record: slo:sli_error:ratio_rate5m - expr: | - (sum(rate(ceph_osd_op_w_latency_count[30m])) - sum(rate(ceph_osd_op_w_latency_bucket{le="0.099999"}[30m]))) + (sum(rate(ceph_osd_op_w_latency_count{job="ceph-latency-bridge"}[30m])) - sum(rate(ceph_osd_op_w_latency_bucket{job="ceph-latency-bridge",le="102.399999"}[30m]))) / - (sum(rate(ceph_osd_op_w_latency_count[30m]))) + (sum(rate(ceph_osd_op_w_latency_count{job="ceph-latency-bridge"}[30m]))) labels: lab: ceph-lab sloth_id: ceph-osd-latency @@ -316,9 +297,9 @@ serverFiles: sloth_window: 30m record: slo:sli_error:ratio_rate30m - expr: | - (sum(rate(ceph_osd_op_w_latency_count[1h])) - sum(rate(ceph_osd_op_w_latency_bucket{le="0.099999"}[1h]))) + (sum(rate(ceph_osd_op_w_latency_count{job="ceph-latency-bridge"}[1h])) - sum(rate(ceph_osd_op_w_latency_bucket{job="ceph-latency-bridge",le="102.399999"}[1h]))) / - (sum(rate(ceph_osd_op_w_latency_count[1h]))) + (sum(rate(ceph_osd_op_w_latency_count{job="ceph-latency-bridge"}[1h]))) labels: lab: ceph-lab sloth_id: ceph-osd-latency @@ -327,9 +308,9 @@ serverFiles: sloth_window: 1h record: slo:sli_error:ratio_rate1h - expr: | - (sum(rate(ceph_osd_op_w_latency_count[2h])) - sum(rate(ceph_osd_op_w_latency_bucket{le="0.099999"}[2h]))) + (sum(rate(ceph_osd_op_w_latency_count{job="ceph-latency-bridge"}[2h])) - sum(rate(ceph_osd_op_w_latency_bucket{job="ceph-latency-bridge",le="102.399999"}[2h]))) / - (sum(rate(ceph_osd_op_w_latency_count[2h]))) + (sum(rate(ceph_osd_op_w_latency_count{job="ceph-latency-bridge"}[2h]))) labels: lab: ceph-lab sloth_id: ceph-osd-latency @@ -338,9 +319,9 @@ serverFiles: sloth_window: 2h record: slo:sli_error:ratio_rate2h - expr: | - (sum(rate(ceph_osd_op_w_latency_count[6h])) - sum(rate(ceph_osd_op_w_latency_bucket{le="0.099999"}[6h]))) + (sum(rate(ceph_osd_op_w_latency_count{job="ceph-latency-bridge"}[6h])) - sum(rate(ceph_osd_op_w_latency_bucket{job="ceph-latency-bridge",le="102.399999"}[6h]))) / - (sum(rate(ceph_osd_op_w_latency_count[6h]))) + (sum(rate(ceph_osd_op_w_latency_count{job="ceph-latency-bridge"}[6h]))) labels: lab: ceph-lab sloth_id: ceph-osd-latency @@ -349,9 +330,9 @@ serverFiles: sloth_window: 6h record: slo:sli_error:ratio_rate6h - expr: | - (sum(rate(ceph_osd_op_w_latency_count[1d])) - sum(rate(ceph_osd_op_w_latency_bucket{le="0.099999"}[1d]))) + (sum(rate(ceph_osd_op_w_latency_count{job="ceph-latency-bridge"}[1d])) - sum(rate(ceph_osd_op_w_latency_bucket{job="ceph-latency-bridge",le="102.399999"}[1d]))) / - (sum(rate(ceph_osd_op_w_latency_count[1d]))) + (sum(rate(ceph_osd_op_w_latency_count{job="ceph-latency-bridge"}[1d]))) labels: lab: ceph-lab sloth_id: ceph-osd-latency @@ -360,9 +341,9 @@ serverFiles: sloth_window: 1d record: slo:sli_error:ratio_rate1d - expr: | - (sum(rate(ceph_osd_op_w_latency_count[3d])) - sum(rate(ceph_osd_op_w_latency_bucket{le="0.099999"}[3d]))) + (sum(rate(ceph_osd_op_w_latency_count{job="ceph-latency-bridge"}[3d])) - sum(rate(ceph_osd_op_w_latency_bucket{job="ceph-latency-bridge",le="102.399999"}[3d]))) / - (sum(rate(ceph_osd_op_w_latency_count[3d]))) + (sum(rate(ceph_osd_op_w_latency_count{job="ceph-latency-bridge"}[3d]))) labels: lab: ceph-lab sloth_id: ceph-osd-latency @@ -444,7 +425,7 @@ serverFiles: record: sloth_slo_info - name: sloth-slo-sli-recordings-ceph-health rules: - - expr: (avg_over_time(clamp_max(ceph_health_status, 1)[5m:1m])) + - expr: (max(avg_over_time(clamp_max(ceph_health_status, 1)[5m:1m]))) labels: lab: ceph-lab sloth_id: ceph-health @@ -452,7 +433,7 @@ serverFiles: sloth_slo: health sloth_window: 5m record: slo:sli_error:ratio_rate5m - - expr: (avg_over_time(clamp_max(ceph_health_status, 1)[30m:1m])) + - expr: (max(avg_over_time(clamp_max(ceph_health_status, 1)[30m:1m]))) labels: lab: ceph-lab sloth_id: ceph-health @@ -460,7 +441,7 @@ serverFiles: sloth_slo: health sloth_window: 30m record: slo:sli_error:ratio_rate30m - - expr: (avg_over_time(clamp_max(ceph_health_status, 1)[1h:1m])) + - expr: (max(avg_over_time(clamp_max(ceph_health_status, 1)[1h:1m]))) labels: lab: ceph-lab sloth_id: ceph-health @@ -468,7 +449,7 @@ serverFiles: sloth_slo: health sloth_window: 1h record: slo:sli_error:ratio_rate1h - - expr: (avg_over_time(clamp_max(ceph_health_status, 1)[2h:1m])) + - expr: (max(avg_over_time(clamp_max(ceph_health_status, 1)[2h:1m]))) labels: lab: ceph-lab sloth_id: ceph-health @@ -476,7 +457,7 @@ serverFiles: sloth_slo: health sloth_window: 2h record: slo:sli_error:ratio_rate2h - - expr: (avg_over_time(clamp_max(ceph_health_status, 1)[6h:1m])) + - expr: (max(avg_over_time(clamp_max(ceph_health_status, 1)[6h:1m]))) labels: lab: ceph-lab sloth_id: ceph-health @@ -484,7 +465,7 @@ serverFiles: sloth_slo: health sloth_window: 6h record: slo:sli_error:ratio_rate6h - - expr: (avg_over_time(clamp_max(ceph_health_status, 1)[1d:1m])) + - expr: (max(avg_over_time(clamp_max(ceph_health_status, 1)[1d:1m]))) labels: lab: ceph-lab sloth_id: ceph-health @@ -492,7 +473,7 @@ serverFiles: sloth_slo: health sloth_window: 1d record: slo:sli_error:ratio_rate1d - - expr: (avg_over_time(clamp_max(ceph_health_status, 1)[3d:1m])) + - expr: (max(avg_over_time(clamp_max(ceph_health_status, 1)[3d:1m]))) labels: lab: ceph-lab sloth_id: ceph-health diff --git a/applications/infrastructure/sloth/prometheusservicelevels.yaml b/applications/infrastructure/sloth/prometheusservicelevels.yaml index 6d97be7..3404f48 100644 --- a/applications/infrastructure/sloth/prometheusservicelevels.yaml +++ b/applications/infrastructure/sloth/prometheusservicelevels.yaml @@ -73,10 +73,10 @@ spec: "<50ms" to "<100ms" to match the actual bucket boundary. sli: events: - totalQuery: sum(rate(ceph_osd_op_w_latency_count[{{.window}}])) + totalQuery: sum(rate(ceph_osd_op_w_latency_count{job="ceph-latency-bridge"}[{{.window}}])) errorQuery: >- - sum(rate(ceph_osd_op_w_latency_count[{{.window}}])) - - sum(rate(ceph_osd_op_w_latency_bucket{le="0.099999"}[{{.window}}])) + sum(rate(ceph_osd_op_w_latency_count{job="ceph-latency-bridge"}[{{.window}}])) + - sum(rate(ceph_osd_op_w_latency_bucket{job="ceph-latency-bridge",le="102.399999"}[{{.window}}])) alerting: name: CephOSDLatencySLO labels: @@ -105,7 +105,7 @@ spec: ERR (2) both count as a violation for this window (clamped to 1). sli: raw: - errorRatioQuery: avg_over_time(clamp_max(ceph_health_status, 1)[{{.window}}:1m]) + errorRatioQuery: max(avg_over_time(clamp_max(ceph_health_status, 1)[{{.window}}:1m])) alerting: name: CephHealthSLO labels: diff --git a/applications/rook/dashboards/prototype-observability.json b/applications/rook/dashboards/prototype-observability.json index 5550bfd..a351b06 100644 --- a/applications/rook/dashboards/prototype-observability.json +++ b/applications/rook/dashboards/prototype-observability.json @@ -254,7 +254,7 @@ } } }, - "targets": [{ "expr": "sum(rate(ceph_osd_op_w_latency_count[1m])) - sum(rate(ceph_osd_op_w_latency_bucket{le=\"0.099999\"}[1m]))", "legendFormat": "Slow Writes/s", "refId": "A" }] + "targets": [{ "expr": "sum(rate(ceph_osd_op_w_latency_count{job=\"ceph-latency-bridge\"}[1m])) - sum(rate(ceph_osd_op_w_latency_bucket{job=\"ceph-latency-bridge\",le=\"102.399999\"}[1m]))", "legendFormat": "Slow Writes/s", "refId": "A" }] }, { "type": "stat", @@ -280,7 +280,7 @@ } } }, - "targets": [{ "expr": "sum(rate(ceph_native_osd_op_r_latency_seconds_bucket{le=\"0.099999\"}[5m])) / sum(rate(ceph_native_osd_op_r_latency_seconds_count[5m]))", "legendFormat": "Success Rate", "refId": "A" }] + "targets": [{ "expr": "sum(rate(ceph_native_osd_op_r_latency_seconds_bucket{job=\"ceph-latency-bridge\",le=\"102.399999\"}[5m])) / sum(rate(ceph_native_osd_op_r_latency_seconds_count{job=\"ceph-latency-bridge\"}[5m]))", "legendFormat": "Success Rate", "refId": "A" }] }, { "type": "stat", @@ -306,7 +306,7 @@ } } }, - "targets": [{ "expr": "sum(rate(ceph_native_osd_op_w_latency_seconds_bucket{le=\"0.099999\"}[5m])) / sum(rate(ceph_native_osd_op_w_latency_seconds_count[5m]))", "legendFormat": "Success Rate", "refId": "A" }] + "targets": [{ "expr": "sum(rate(ceph_native_osd_op_w_latency_seconds_bucket{job=\"ceph-latency-bridge\",le=\"102.399999\"}[5m])) / sum(rate(ceph_native_osd_op_w_latency_seconds_count{job=\"ceph-latency-bridge\"}[5m]))", "legendFormat": "Success Rate", "refId": "A" }] }, { "type": "stat",