Distributed HTTP Telemetry

README.md

@@ -1122,6 +1122,7 @@ Routes:
 |---|---|---|
 | `POST` | `/:command` | Dispatch a command. Body = JSON input, headers = session variables. |
 | `GET` | `/health` | Health check: `{ "ok": true, "commands": ["counter.initialize", ...] }` |
+| `GET` | `/metrics` | Prometheus text metrics when the `metrics` feature is enabled. |
 
 ```bash
 curl -X POST http://localhost:3000/counter.initialize \
@@ -1486,6 +1487,12 @@ Bus-only and worker services can expose the same registry on a side port with
 `distributed::metrics::serve_http`. See [`docs/metrics.md`](docs/metrics.md) for
 metric names, label rules, and GitOps details.
 
+HTTP services record request-count and duration metrics for Distributed-owned
+route templates (`/health`, `/metrics`, `/:command` as `/{command}`, `/`, and
+`/cloudevent/{type}`) separately from dispatch metrics. Request metrics capture
+pre-dispatch failures such as malformed JSON or body-limit rejection without
+recording raw paths, query strings, headers, session variables, or payload data.
+
 `describe`/`schema` compile your crate and call its `distributed_manifest()`
 entrypoint (override with `--entrypoint`), which registers the [read
 models](#read-models) and tables that define the schema:

docs/metrics.md

@@ -24,7 +24,8 @@ distributed::microsvc::serve(service, "0.0.0.0:3000").await?;
 
 Prometheus can scrape `GET /metrics` on the service's HTTP port. Hops
 ObserveStack should use the same scrape target and labels; dashboards can join
-these metrics with trace data by the stable `service` and message labels.
+HTTP request metrics with dispatch metrics and trace data by the stable
+`service` label.
 
 The scrape route is unauthenticated by design. Do not expose `/metrics` on a
 public listener; keep it behind a private network, ingress policy, security
@@ -50,6 +51,12 @@ This listener exposes only `GET /metrics`; it does not expose command dispatch.
 Metric labels are intentionally bounded:
 
 - `service`: `Service::named(...)`, or `unnamed` when no service name was set.
+- `method`: normalized HTTP method. Standard methods keep their uppercase token;
+  non-standard methods are bucketed as `_OTHER`.
+- `route`: Distributed-owned HTTP route template only: `/health`, `/metrics`,
+  `/{command}`, `/`, `/cloudevent/{type}`, or `unmatched`. Raw paths are never
+  recorded.
+- `status_code`: numeric HTTP response status as a string.
 - `message_kind`: `command` or `event`.
 - `message`: registered command/event name; unknown-command failures are
   bucketed as `unknown` rather than recording the unrecognized input.
@@ -65,8 +72,9 @@ Metric labels are intentionally bounded:
   `log_and_ack`, `stop`, `recv_error`, or a settle failure label such as
   `settle_ack`.
 
-Do not add IDs, trace IDs, user IDs, aggregate IDs, or other high-cardinality
-values as framework labels.
+Do not add raw URI paths, query strings, IDs, trace IDs, request IDs, user IDs,
+aggregate IDs, headers, cookies, session variables, payload data, arbitrary
+metadata values, or other high-cardinality values as framework labels.
 
 The label names, framework status values, transport outcomes, failure actions,
 and outbox outcomes live in one internal telemetry vocabulary. New framework
@@ -75,6 +83,10 @@ metrics should use that vocabulary rather than introducing ad hoc labels.
 ## Metric Families
 
 - `distributed_service_info{service,version}` gauge.
+- `distributed_http_server_requests_total{service,method,route,status_code}`
+  counter.
+- `distributed_http_server_request_duration_seconds{service,method,route,status_code}`
+  histogram.
 - `distributed_microsvc_dispatch_total{service,message_kind,message,status}`
   counter.
 - `distributed_microsvc_dispatch_duration_seconds{service,message_kind,message,status}`
@@ -96,8 +108,16 @@ shape.
 ## Boundaries
 
 The `metrics` feature owns Prometheus text exposition for framework metrics. It
-does not install an OpenTelemetry metrics SDK, emit request-level HTTP metrics,
-or record user payload data.
+does not install an OpenTelemetry metrics SDK or record user payload data.
+
+HTTP request metrics measure the Distributed-owned HTTP boundary: health checks,
+metrics scrapes, command ingress, CloudEvents ingress, and framework rejections
+that happen before dispatch such as malformed JSON or body-limit failures.
+Dispatch metrics measure the command/event routing and handler boundary after a
+request has been accepted. A single successful `POST /{command}` normally
+records both HTTP request metrics and `distributed_microsvc_dispatch_*`; a bad
+JSON body records only HTTP request metrics because no command dispatch occurs.
+Unknown command dispatches still use `message="unknown"` in dispatch metrics.
 
 Direct transport receive paths emit receive/settle counters and failure
 counters. Outbox dispatch emits publish outcomes and backlog gauges. Direct

docs/observability.md

@@ -59,10 +59,11 @@ metadata already stored with events.
 
 ## Optional Span Feature
 
-The `otel` feature adds framework-owned `tracing` spans around dispatch,
-handler execution, transport receive, and outbox publish boundaries. When the
-incoming message carries W3C `traceparent` / `tracestate`, Distributed extracts
-that context and sets it as the OpenTelemetry parent for the framework span:
+The `otel` feature adds framework-owned `tracing` spans around HTTP request,
+dispatch, handler execution, transport receive, and outbox publish boundaries.
+When the incoming HTTP request or message carries W3C `traceparent` /
+`tracestate`, Distributed extracts that context and sets it as the OpenTelemetry
+parent for the framework span:
 
 ```toml
 [dependencies]
@@ -77,20 +78,42 @@ setup in the generated `main.rs` for the common case.
 
 Framework span names are intentionally bounded:
 
+- `GET /health`
+- `GET /metrics`
+- `POST /{command}`
+- `POST /`
+- `POST /cloudevent/{type}`
 - `distributed.microsvc.dispatch`
 - `distributed.handler`
 - `distributed.transport.receive`
 - `distributed.outbox.publish`
 
-Each span uses the same framework-owned message attributes:
+HTTP server spans are created only for Distributed-owned routers and are span
+kind `SERVER`. They use bounded HTTP attributes:
+
+- `http.request.method`
+- `http.route`
+- `http.response.status_code`
+- `network.protocol.name = "http"`
+
+Distributed intentionally emits `http.route` and omits `url.path` and
+`url.query` in this first pass so route telemetry cannot leak raw paths or query
+strings. A command HTTP request carrying `traceparent` creates a request span
+parented to that incoming context; `distributed.microsvc.dispatch` is then a
+child of the HTTP span. If a caller has already opened a local span before
+entering Distributed, the existing ambient-span rule still wins and framework
+spans nest under that local span.
+
+Message-oriented spans use these framework-owned message attributes:
 
 - `distributed.message.name`
 - `distributed.message.kind`
 - `messaging.message.id`
 
-Future spans should use the same helper path so new attributes are reviewed in
-one place. Do not add payload fields, user ids, aggregate ids, or raw metadata
-values as framework span attributes.
+Future spans should use the same helper paths so new attributes are reviewed in
+one place. Do not add payload fields, user ids, aggregate ids, raw URI paths,
+query strings, headers, cookies, session variables, or raw metadata values as
+framework span attributes.
 
 Recommended environment variables for OTLP exporters:
 
@@ -151,10 +174,12 @@ CI verifies the observability surface at the docker level (see
   `GET /metrics`, and lints the exposition with `promtool check metrics`
   (skips when `PROMTOOL` is unset).
 - `tests/otel_export` builds a real OTLP pipeline the way a service binary
-  would, dispatches a message carrying a W3C `traceparent`, and asserts a real
-  OpenTelemetry Collector received `distributed.microsvc.dispatch` parented to
-  the incoming span (skips when `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` /
-  `OTEL_COLLECTOR_TRACES_FILE` are unset).
+  would, dispatches a message carrying a W3C `traceparent`, sends an HTTP
+  command request carrying `traceparent`, and asserts a real OpenTelemetry
+  Collector received the request span and `distributed.microsvc.dispatch` with
+  the expected parent/child relationships (skips when
+  `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` / `OTEL_COLLECTOR_TRACES_FILE` are
+  unset).
 - The scaffolded `ServiceMonitor` / `PrometheusRule` / OTLP env output is
   rendered with `helm template` and validated against published CRD schemas
   with `kubeconform`.
@@ -167,11 +192,11 @@ platform concern, verified on a live cluster rather than per PR.
 The default feature set propagates W3C trace context metadata only.
 
 The `metrics` feature records framework metrics and renders Prometheus text. It
-does not expose OpenTelemetry metrics or request-level HTTP telemetry.
+does not expose OpenTelemetry metrics.
 
-The `otel` feature creates framework spans and parents them from W3C metadata
-when available. It does not install subscribers, exporters, sampling rules, or
-resource attributes.
+The `otel` feature creates framework spans and parents them from W3C HTTP
+headers or message metadata when available. It does not install subscribers,
+exporters, sampling rules, or resource attributes.
 
 Future `logs` or private `diagnostics` features should build on the same
 bounded telemetry vocabulary. Logs may carry structured failure context, but

docs/transports.md

@@ -180,6 +180,10 @@ in-process consume loop: `KnativeBus::manifests(&plan, &subscriptions)` renders
 the role-based `Broker` + per-name `Trigger` YAML (subscriber URIs
 `/cloudevent/<type>`, with a `.local(addr)` kubefwd variant), and the service
 mounts `cloud_events_router` so those Triggers reach `dispatch_message`.
+When `metrics` is enabled, the router also exposes `GET /metrics` and records
+bounded HTTP request metrics for `/`, `/cloudevent/{type}`, and `/metrics`;
+dispatch metrics remain separate and describe the command/event handler
+boundary after a CloudEvent is parsed.
 
 `PostgresBus` uses the claim-lease work queue (not `sqlxmq`) for the same reason
 the low-level adapter does — sqlxmq's always-on push runner doesn't compose with

src/http_telemetry.rs

@@ -0,0 +1,128 @@
+//! Shared telemetry middleware for Distributed-owned HTTP routers.
+
+#[cfg(feature = "metrics")]
+use std::time::Instant;
+
+#[cfg(any(feature = "metrics", feature = "otel"))]
+use axum::extract::MatchedPath;
+use axum::extract::{Request, State};
+#[cfg(any(feature = "metrics", feature = "otel"))]
+use axum::http::Method;
+use axum::middleware::Next;
+use axum::response::Response;
+
+#[derive(Clone, Debug, Default)]
+pub(crate) struct HttpTelemetryState {
+    #[cfg(feature = "metrics")]
+    service: Option<String>,
+}
+
+impl HttpTelemetryState {
+    pub(crate) fn new(service: Option<String>) -> Self {
+        #[cfg(not(feature = "metrics"))]
+        let _ = service;
+        #[cfg(not(feature = "metrics"))]
+        return Self {};
+
+        #[cfg(feature = "metrics")]
+        Self { service }
+    }
+
+    #[cfg(feature = "metrics")]
+    fn service(&self) -> Option<&str> {
+        self.service.as_deref()
+    }
+}
+
+pub(crate) async fn middleware(
+    State(state): State<HttpTelemetryState>,
+    req: Request,
+    next: Next,
+) -> Response {
+    #[cfg(not(any(feature = "metrics", feature = "otel")))]
+    {
+        let _ = state;
+        return next.run(req).await;
+    }
+
+    #[cfg(any(feature = "metrics", feature = "otel"))]
+    {
+        #[cfg(not(feature = "metrics"))]
+        let _ = state;
+
+        let method = normalize_method(req.method());
+        let route = route_label(req.extensions().get::<MatchedPath>());
+        #[cfg(feature = "metrics")]
+        let started = Instant::now();
+
+        #[cfg(feature = "otel")]
+        let span = http_server_span(method, route);
+        #[cfg(feature = "otel")]
+        crate::trace_context::set_span_parent_from_headers_if_no_current_span(&span, req.headers());
+
+        #[cfg(feature = "otel")]
+        let response = {
+            use tracing::Instrument as _;
+
+            next.run(req).instrument(span.clone()).await
+        };
+        #[cfg(not(feature = "otel"))]
+        let response = next.run(req).await;
+
+        let status = response.status();
+        #[cfg(feature = "otel")]
+        span.record("http.response.status_code", i64::from(status.as_u16()));
+        #[cfg(feature = "metrics")]
+        crate::metrics::record_http_server_request(
+            state.service(),
+            method,
+            route,
+            status.as_u16(),
+            started.elapsed(),
+        );
+
+        response
+    }
+}
+
+#[cfg(any(feature = "metrics", feature = "otel"))]
+fn normalize_method(method: &Method) -> &'static str {
+    match method.as_str() {
+        "GET" => "GET",
+        "HEAD" => "HEAD",
+        "POST" => "POST",
+        "PUT" => "PUT",
+        "PATCH" => "PATCH",
+        "DELETE" => "DELETE",
+        "OPTIONS" => "OPTIONS",
+        "TRACE" => "TRACE",
+        "CONNECT" => "CONNECT",
+        _ => "_OTHER",
+    }
+}
+
+#[cfg(any(feature = "metrics", feature = "otel"))]
+fn route_label(matched_path: Option<&MatchedPath>) -> &'static str {
+    match matched_path.map(MatchedPath::as_str) {
+        Some("/health") => "/health",
+        Some("/metrics") => "/metrics",
+        Some("/{command}") => "/{command}",
+        Some("/") => "/",
+        Some("/cloudevent/{type}") => "/cloudevent/{type}",
+        _ => "unmatched",
+    }
+}
+
+#[cfg(feature = "otel")]
+fn http_server_span(method: &'static str, route: &'static str) -> tracing::Span {
+    let span_name = format!("{method} {route}");
+    tracing::info_span!(
+        "distributed.http.server",
+        otel.name = span_name.as_str(),
+        otel.kind = "server",
+        http.request.method = method,
+        http.route = route,
+        http.response.status_code = tracing::field::Empty,
+        network.protocol.name = "http",
+    )
+}

src/lib.rs

@@ -12,6 +12,8 @@ pub mod repository;
 mod commit_builder;
 #[cfg(feature = "emitter")]
 pub mod emitter;
+#[cfg(feature = "http")]
+mod http_telemetry;
 mod in_memory_repo;
 pub mod lock;
 pub mod manifest;