fix(JDWLABS-464): verify drift-scan verdicts against Prometheus before escalating

[jdwillmsen]

Problem

ClusterDriftDetected (fingerprint drift-scan-20260829) fired from the hourly holmes-drift-scan CronJob in ai-sre with VERDICT: ISSUE: Pending pods detected in the cluster. while the cluster had zero Pending pods (scheduler_pending_pods 0, no phase=Pending pods, no events). The job forwarded whatever Holmes returned straight to the alert relay, so an LLM false positive became a page.

Today's runs show the same class of error in another category: all three of this morning's jobs escalated "restart spikes" that were lifetime restart counts (7-9) on pods that are weeks old, with increase(kube_pod_container_status_restarts_total[1h]) at 0 across the cluster.

Fix

Scope: the CronJob manifest only (tenants/platform/services/holmes/postInstall/drift-scan-cronjob.yaml).

1. Prompt tightened. Each of the four conditions gets an explicit definition (pending >5m; >=3 restarts in the last hour, "lifetime count is NOT a spike"; node memory >90%; cert-manager expiry <30d). Holmes must report each confirmed condition as its own VERDICT: ISSUE: <category>: <summary> (evidence: <names and metric values>) line with <category> from a fixed set, and must not report conditions it could not check.
2. Deterministic verification. Before escalating, every VERDICT: ISSUE: line is parsed for its category and re-checked against Prometheus with a fixed PromQL query using the same threshold given to the model. Only claims Prometheus confirms are escalated; the alert description gains a Ground truth: section with the query result.
3. Observability of false positives. Every verification prints one JSON line {"event":"drift-scan-verify","category":...,"outcome":confirmed|rejected|unverified,"groundTruth":...,"claim":...}; a run where nothing is confirmed logs the count and exits 0.
4. Fail-open on missing ground truth. If Prometheus cannot be queried the claim is escalated as unverified rather than dropped. An unlabelled claim (model ignores the format) is checked against all four categories combined rather than dropped.

Evidence

• sh -n and shellcheck -s sh on the extracted script: clean. yamllint tenants/ bootstrap/, kubeconform -strict, check-image-pins.py, check-orphaned-manifests.py: pass.
• All four PromQL queries executed against the live Prometheus (platform-kube-prometheus-s-prometheus) and return a scalar; the metrics exist (certmanager_certificate_expiration_timestamp_seconds has 7 series, min expiry 49d).
• Script exercised end-to-end against live Prometheus with stubbed Holmes/relay:
  • the exact false-positive verdict from the incident -> outcome=rejected, groundTruth=0, exit 0, no relay POST
  • today's restart-spike + node-memory verdicts -> both rejected, no relay POST
  • legacy unlabelled VERDICT: ISSUE: Pending pods... -> category=unknown, rejected
  • VERDICT: CLEAN -> clean exit
  • tls-expiry with the threshold widened to 60d -> confirmed, groundTruth=3, relay payload built with Ground truth: appended
  • Prometheus unreachable -> outcome=unverified, escalated
• Egress from ai-sre to monitoring is allowed by the namespace's existing allow-all-egress policy; Holmes itself already uses the same Prometheus URL.

Left for a human

• Thresholds (5m / 3 restarts per hour / 90% / 30d) are a first cut; tune if the drift-scan-verify lines show the model and the queries disagreeing on definitions rather than facts.
• The underlying model weakness noted in the manifest header (gpt-oss-20b) is unchanged.

🤖 Generated with Claude Code

https://claude.ai/code/session_01B6FWEmAs2beQvKnX4LwWN2