feat(observability): deploy github-repo-health-exporter with dashboard and alerts (JDWLABS-388) - #385
Open
jdwillmsen wants to merge 1 commit into
Open
feat(observability): deploy github-repo-health-exporter with dashboard and alerts (JDWLABS-388)#385jdwillmsen wants to merge 1 commit into
jdwillmsen wants to merge 1 commit into
Conversation
…d and alerts Cross-repo CI health had no durable sink: a scheduled workflow on main failed 13 of 15 daily runs and the only place that showed was a step summary nobody opened. The exporter image now exists; this deploys it as a tenant service scraped by kube-prometheus-stack, gives it one Git-Synced dashboard, and alerts on the series it actually emits. - Deployment/Service/ServiceMonitor in monitoring, digest-pinned image, same hardening as the sibling exporters. - ExternalSecret reuses the org agent-bot App key already in Vault; the exporter only lists Actions runs, so no new credential. - PrometheusRule: consecutive-failure streak on main (warning at 3, critical at 7), exporter stale / refresh failing / target down. Unit tests under tests/prometheus-rules. - Dashboard observability/dashboards/platform/repo-health.json. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016XRddumw4ZNvFqHSd9KSf6
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Comment on lines
+31
to
+74
| - name: exporter | ||
| image: jdwlabs/github-repo-health-exporter:0.0.1@sha256:3f697eefb82e9cd1266efee0ba13e32f82bcc9a1956c08cb1d51e31f670b33c4 | ||
| ports: | ||
| - name: metrics | ||
| containerPort: 9090 | ||
| protocol: TCP | ||
| envFrom: | ||
| - secretRef: | ||
| name: github-repo-health-exporter | ||
| env: | ||
| - name: PORT | ||
| value: "9090" | ||
| - name: GITHUB_ORG | ||
| value: "jdwlabs" | ||
| # 15m is the cadence the originating investigation asked for; at | ||
| # one request per repo per cycle this is far below the App's | ||
| # rate limit and Prometheus scrapes it at 60s regardless. | ||
| - name: REFRESH_INTERVAL | ||
| value: "15m" | ||
| - name: GOMEMLIMIT | ||
| value: "48MiB" | ||
| readinessProbe: | ||
| httpGet: | ||
| path: /healthz | ||
| port: metrics | ||
| initialDelaySeconds: 3 | ||
| periodSeconds: 10 | ||
| livenessProbe: | ||
| httpGet: | ||
| path: /healthz | ||
| port: metrics | ||
| initialDelaySeconds: 10 | ||
| periodSeconds: 30 | ||
| securityContext: | ||
| allowPrivilegeEscalation: false | ||
| readOnlyRootFilesystem: true | ||
| capabilities: | ||
| drop: ["ALL"] | ||
| resources: | ||
| requests: | ||
| cpu: 10m | ||
| memory: 32Mi | ||
| limits: | ||
| memory: 64Mi |
Comment on lines
+31
to
+74
| - name: exporter | ||
| image: jdwlabs/github-repo-health-exporter:0.0.1@sha256:3f697eefb82e9cd1266efee0ba13e32f82bcc9a1956c08cb1d51e31f670b33c4 | ||
| ports: | ||
| - name: metrics | ||
| containerPort: 9090 | ||
| protocol: TCP | ||
| envFrom: | ||
| - secretRef: | ||
| name: github-repo-health-exporter | ||
| env: | ||
| - name: PORT | ||
| value: "9090" | ||
| - name: GITHUB_ORG | ||
| value: "jdwlabs" | ||
| # 15m is the cadence the originating investigation asked for; at | ||
| # one request per repo per cycle this is far below the App's | ||
| # rate limit and Prometheus scrapes it at 60s regardless. | ||
| - name: REFRESH_INTERVAL | ||
| value: "15m" | ||
| - name: GOMEMLIMIT | ||
| value: "48MiB" | ||
| readinessProbe: | ||
| httpGet: | ||
| path: /healthz | ||
| port: metrics | ||
| initialDelaySeconds: 3 | ||
| periodSeconds: 10 | ||
| livenessProbe: | ||
| httpGet: | ||
| path: /healthz | ||
| port: metrics | ||
| initialDelaySeconds: 10 | ||
| periodSeconds: 30 | ||
| securityContext: | ||
| allowPrivilegeEscalation: false | ||
| readOnlyRootFilesystem: true | ||
| capabilities: | ||
| drop: ["ALL"] | ||
| resources: | ||
| requests: | ||
| cpu: 10m | ||
| memory: 32Mi | ||
| limits: | ||
| memory: 64Mi |
Comment on lines
+31
to
+74
| - name: exporter | ||
| image: jdwlabs/github-repo-health-exporter:0.0.1@sha256:3f697eefb82e9cd1266efee0ba13e32f82bcc9a1956c08cb1d51e31f670b33c4 | ||
| ports: | ||
| - name: metrics | ||
| containerPort: 9090 | ||
| protocol: TCP | ||
| envFrom: | ||
| - secretRef: | ||
| name: github-repo-health-exporter | ||
| env: | ||
| - name: PORT | ||
| value: "9090" | ||
| - name: GITHUB_ORG | ||
| value: "jdwlabs" | ||
| # 15m is the cadence the originating investigation asked for; at | ||
| # one request per repo per cycle this is far below the App's | ||
| # rate limit and Prometheus scrapes it at 60s regardless. | ||
| - name: REFRESH_INTERVAL | ||
| value: "15m" | ||
| - name: GOMEMLIMIT | ||
| value: "48MiB" | ||
| readinessProbe: | ||
| httpGet: | ||
| path: /healthz | ||
| port: metrics | ||
| initialDelaySeconds: 3 | ||
| periodSeconds: 10 | ||
| livenessProbe: | ||
| httpGet: | ||
| path: /healthz | ||
| port: metrics | ||
| initialDelaySeconds: 10 | ||
| periodSeconds: 30 | ||
| securityContext: | ||
| allowPrivilegeEscalation: false | ||
| readOnlyRootFilesystem: true | ||
| capabilities: | ||
| drop: ["ALL"] | ||
| resources: | ||
| requests: | ||
| cpu: 10m | ||
| memory: 32Mi | ||
| limits: | ||
| memory: 64Mi |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Deploys the
github-repo-health-exporter(released by apps as0.0.1, JDWLABS-423) as a platform tenant service, scraped by kube-prometheus-stack, with one Git-Synced Grafana dashboard and a PrometheusRule on the series it emits. Modelled ontruenas-graphite-exporter(deployment + service + servicemonitor) with the App key delivered by ExternalSecret the wayai-sre-relaydoes.Image:
jdwlabs/github-repo-health-exporter:0.0.1@sha256:3f697eefb82e9cd1266efee0ba13e32f82bcc9a1956c08cb1d51e31f670b33c4(OCI index digest, multi-arch).Files
tenants/platform/services/github-repo-health-exporter/postInstall/{deployment,service,servicemonitor,externalsecret}.yamltenants/platform/tenant.yaml— registered as rawManifests inmonitoring, syncWave 2tenants/platform/services/kube-prometheus-stack/postInstall/rules-github-repo-health.yamltests/prometheus-rules/rules-github-repo-health_test.yamlobservability/dashboards/platform/repo-health.json(uidplatform-repo-health; Git Sync already bound to this folder)Metrics the exporter emits (all read by dashboard/rules — nothing invented)
jdwlabs_repo_ci_consecutive_failures{repo,workflow}jdwlabs_repo_ci_main_failure_ratio{repo,workflow}jdwlabs_repo_health_exporter_sample_size{repo,workflow}jdwlabs_repo_health_exporter_refreshes_total{outcome}jdwlabs_repo_health_exporter_last_refresh_timestamp_secondsAlerts
RepoCiWorkflowFailingOnMainjdwlabs_repo_ci_consecutive_failures >= 3for 1hRepoCiWorkflowFailingOnMainWeek>= 7for 1hRepoHealthExporterStaleRepoHealthExporterRefreshFailingincrease(refreshes_total{outcome="failed"}[1h]) >= 3for 15mRepoHealthExporterTargetDownup == 0 or absent(up)for 15mThe ticket also asks for an alert on merged PRs without approving review — the exporter's current slice does not emit that metric (only the CI-health family shipped in apps#190), so there is nothing to alert on yet. Follow-up when that slice lands.
Manual pre-steps
jdwlabs-agent-botGitHub AppActions: Read. The Vaultagent-botsecret is reused (no new credential), but the org installation currently holds onlycontents: write, metadata: read, pull_requests: write(checked viagh api /orgs/jdwlabs/installations).GET /repos/{owner}/{repo}/actions/runsneedsactions: read; without it every refresh 403s andRepoHealthExporterStalefires ~1h15m after rollout. Add the permission in the App settings, then accept the updated permissions on the org installation.Test plan
tests/prometheus-rules/run.sh—promtool check rules+ unit tests green (19 rule sets, new tests cover healthy-silent, 3→7 escalation, short streak never fires, never-refreshed stale, one-repo-failing, target down + absent)yamllintwith the CI config on the touched fileskubeconform -stricton the new manifests (5/5 valid)tools/check-image-pins.py,tools/check-gitsync-tenant-folders.py,tools/testsunitteststests/alertmanager-routing/run.sh(24 cases, 0 failures)platformctl tenants validate tenants/up{job="github-repo-health-exporter"}==1,/metricslists all four repos, dashboard appears in the platform Git Sync folder,RepoCiWorkflowFailingOnMainfires fordeployments / prd Driftif it is still redRollback: delete the tenant directory, the tenant.yaml entry, the rules file + its test, and the dashboard JSON.
Refs JDWLABS-388.
🤖 Generated with Claude Code
https://claude.ai/code/session_016XRddumw4ZNvFqHSd9KSf6