Prometheus/Grafana monitoring integration

[karutoil]

Summary

Integrate Prometheus metrics exposition and Grafana dashboards for production observability. Currently, metrics are only available via the internal API and WebSocket streams.

Current State

• Agent sends health_report (node-level: CPU%, memory, disk, containers) and resource_stats_batch (per-server: CPU%, memory, network, disk) every 5s via WebSocket
• Backend stores metrics in PostgreSQL (ServerMetrics, NodeMetrics models) with retention workers
• No Prometheus /metrics endpoint exists
• No Grafana dashboards exist
• Docker Compose stack includes Redis but no Prometheus/Grafana services

Requirements

Backend

• GET /metrics endpoint exposing Prometheus-format metrics (using prom-client or fastify-metrics)
• System metrics: active servers, connected agents, WebSocket connections, API request rates, error rates
• Per-server metrics where applicable
• Optional: expose via a separate metrics port for security isolation

Agent

• Optional Prometheus metrics endpoint on the agent (for direct node scraping)
• Container-level metrics: CPU, memory, network I/O per container
• Go-style runtime metrics for the Rust agent itself

Grafana

• Pre-built dashboards for:
  • Overview: cluster health, active servers, agent connectivity
  • Node detail: CPU, memory, disk, network per node
  • Server detail: per-server resource usage over time
  • Alerting: alert rule triggers, delivery status
• Docker Compose overlay (docker-compose.monitoring.yml) with Prometheus + Grafana
• Provisioned datasources and dashboards (JSON models checked into repo)

Documentation

• Setup guide for enabling monitoring
• Metric naming conventions and labels
• Alert rule examples in Grafana

Acceptance Criteria

• Backend /metrics endpoint with standard Prometheus format
• Docker Compose monitoring overlay with Prometheus + Grafana
• At least 3 pre-built Grafana dashboards (overview, node, server)
• Documentation for setup and configuration
• Metrics endpoint is opt-in (disabled by default, enabled via env var)