A dependency-free Prometheus /metrics endpoint (#91)

CHANGELOG.md

@@ -6,8 +6,75 @@ policy in [docs/versioning.md](docs/versioning.md).
 
 ## [Unreleased]
 
+### Added
+
+- **A dependency-free Prometheus `/metrics` endpoint** (#91, first work
+  item). `smithy::server::MetricsRegistry` aggregates the existing `Observe`
+  hooks into three families —
+  the five `http_server_*` families labeled by `service_name`, `http_method`
+  and `route` — and `MetricsEndpoint` serves them in the
+  text exposition format, which needs no client library and so costs zero new
+  dependencies. `RecordMetrics` is `Observe` wired to a registry, so request
+  timing keeps one implementation and the scraped numbers cannot drift from
+  the logged ones; compose the endpoint outside the recorder and scrapes
+  answer without inflating the request rate they report. Label cardinality is
+  bounded by construction: `target` is never a label (path parameters and
+  query strings would mint a series per request id), an off-wire `method`
+  outside the standard set collapses to `other`, and a series cap backstops
+  anything unforeseen while counting what it refused in
+  `metrics_observations_dropped_total`. Application metrics join the
+  same scrape through `NewCounter` / `NewGauge` / `NewHistogram`, so one
+  Prometheus target covers the service; the registry keeps owning escaping,
+  label ordering, and the per-family cap. `Declare` exports a known series at
+  zero from startup, so the first event is a visible step rather than a
+  counter's invisible first sample. `RecordRejections` feeds
+  `BeastServerTransport::Options::on_rejected`, so the 413/431 the transport
+  writes before any middleware exists are counted too — without filing a
+  zero latency that would flatter the panel during an over-limit flood.
+
+  The whole stack is **off unless `MetricsOptions::enabled` says otherwise**,
+  and off means absent rather than idle: `RecordMetrics` and `MetricsEndpoint`
+  compose to the identity, so a disabled registry puts no wrapper on the
+  request path and `/metrics` 404s through to the router rather than serving
+  an empty scrape that would read as a live target with nothing to report.
+  Handles from a disabled registry are inert rather than unusable, so
+  application code never branches on the flag. Registration is deliberately
+  *not* conditional on it — an invalid name, a type collision, or a bad bucket
+  ladder aborts at startup either way, so enabling metrics in production is
+  never the first time those checks run.
+
+  The exposition is [MoonBase](https://github.com/muchq/MoonBase)'s shared
+  HTTP serving contract, and is deliberately not configurable: the five
+  `http_server_*` families with the descriptions its three emitter rails pin,
+  the `service_name`/`http_method`/`route` label set, the `unmatched` and
+  `/health` route sentinels, the `CUSTOM` and `(unparsed)` method sentinels,
+  and the microsecond bucket ladder `//domains/platform/libs/otel_contract`
+  pins equal across them. Those services are who scrapes this, so a service
+  here can replace an aura/futility, yodel, or server_pal one without touching
+  a dashboard — and a knob would only be a way to drift off the contract
+  silently, since the failure renders as an empty panel rather than an error.
+  `service_name` is required when enabled and an empty one aborts, because
+  every dashboard query selects on it. Status is not a label: the success and
+  failure counters carry the outcome and are derived from the same tally the
+  total sums, so the three cannot disagree. The active gauge carries no route,
+  which is where every rail leaves it — it moves before dispatch. See the
+  Observability section of
+  [docs/production-guide.md](docs/production-guide.md).
+
 ### Fixed
 
+- **Health probes are distinguishable from dispatch failures in observability
+  hooks.** `HealthEndpoint` built its response without stamping
+  `HttpResponse::operation`, so every probe reached `Observe` (and so any
+  metrics or logging backend) as the empty operation — the same value
+  404/405/400 use. Since Kubernetes polls a probe every few seconds, that is
+  usually the highest-volume path a service has, and merging the two meant
+  the 404 rate could not be read, the probe's own latency contaminated the
+  service's duration histogram, and no filter could separate them.
+  `HealthEndpoint` and `MetricsEndpoint` now report their own configured path
+  as the operation (`/livez`, `/readyz`, `/metrics`), which is fixed at
+  composition and so adds one series per composed endpoint. Chains that do
+  not observe probe traffic are unaffected.
 - **Numeric wire values no longer truncate into generated narrow types**
   (#109). Three holes in the otherwise-uniform range-check posture: an
   `intEnum` member in a document body cast the raw wire int64 straight into

docs/production-guide.md

@@ -203,8 +203,9 @@ transport.Start(smithy::server::Chain(
      smithy::server::PerClientRateLimit(
          [limiter](const std::string& client) { return limiter->Allow(client); },
          trusted, std::chrono::seconds(30)),
-     // Observe everything admitted — health probes included. on_start
-     // (optional) enables an in-flight gauge; on_complete carries
+     // Observe everything admitted — health probes included, reporting
+     // `operation` as their own path so a dashboard can filter them out.
+     // on_start (optional) enables an in-flight gauge; on_complete carries
      // method/target/operation/status/duration/trace_parent.
      smithy::server::Observe(
          [](const smithy::server::RequestObservation& o) {
@@ -339,7 +340,8 @@ OpenTelemetry — plugs in without the core taking a telemetry dependency.
 
 **Server:** `Observe` (above) reports, per request: `method`, `target`,
 `operation` (the Smithy operation that handled it, stamped by the generated
-router; empty for 404/405/400 dispatch failures), `status`, `duration`, and
+router; the endpoint's own path for `HealthEndpoint` and `MetricsEndpoint`;
+empty for 404/405/400 dispatch failures), `status`, `duration`, and
 `trace_parent` — the request's W3C `traceparent` header, which always parses:
 a valid inbound one continues verbatim, and the transport ingress mints a
 fresh root when the client sent none or sent garbage (ADR-0011). The same
@@ -370,6 +372,175 @@ config.interceptors.push_back(smithy::PropagateTraceContext());
 `GenerateSpanId` — for building richer integrations (e.g. a server
 middleware that opens a span from `RequestObservation::trace_parent`).
 
+**Prometheus:** the one bundled backend, because the text exposition format
+needs no client library — it is a few lines of text over HTTP, so it costs
+zero dependencies. Two middleware compose around the generated handler:
+
+```cpp
+auto metrics = std::make_shared<smithy::server::MetricsRegistry>(
+    smithy::server::MetricsOptions{.enabled = true});
+transport.Start(smithy::server::Chain({smithy::server::MetricsEndpoint(metrics),
+                                       smithy::server::RecordMetrics(metrics)},
+                                      server.Handler()));
+```
+
+**It is off unless you say otherwise.** `MetricsOptions::enabled` defaults to
+false, and a disabled registry is not a registry that records into a void: the
+two middleware compose to the *identity*, so nothing wraps the request path —
+no timing, no lock, not even an extra call frame — and `/metrics` reaches the
+router like any other unmodeled path and 404s. (A disabled endpoint answering
+an empty 200 would read to Prometheus as a live target reporting no series,
+which is exactly what a service whose metrics have gone silent looks like.)
+Handles from a disabled registry are inert rather than unusable, so
+application code never branches on the flag; only their *arguments* still
+cost anything, so guard a hot call site whose labels are themselves expensive
+with `metrics->enabled()`.
+
+Registration is not conditional on the flag. An invalid metric name, a type
+collision, or a bad bucket ladder aborts at startup either way (ADR-0009), so
+switching metrics on in production is never the first time those checks run.
+
+`RecordMetrics` is `Observe` wired to the registry, so request timing has one
+implementation and the scraped numbers cannot drift from the logged ones. The
+order above is deliberate: the endpoint sits *outside* the recorder, so
+scrapes answer without being counted as served traffic — swap them and every
+scrape inflates your own request rate, at whatever interval Prometheus polls.
+
+Five families are exposed on `/metrics` (path configurable), labeled by
+`service_name`, `http_method` and `route`:
+
+| Family | Type | Notes |
+| --- | --- | --- |
+| `http_server_requests_total` | counter | every completed request |
+| `http_server_requests_success_total` | counter | status < 400 |
+| `http_server_requests_failure_total` | counter | status >= 400 |
+| `http_server_requests_active_gauge` | gauge | no `route` label; see below |
+| `http_server_request_duration_microseconds` | histogram | `_bucket`/`_sum`/`_count` |
+
+plus `metrics_observations_dropped_total`, the registry's own health.
+
+This is not a vocabulary of our own invention, and it is deliberately **not
+configurable**. It is [MoonBase](https://github.com/muchq/MoonBase)'s shared
+HTTP serving contract — spoken identically by its Java (yodel), Rust
+(server_pal) and C++ (futility/otel, behind aura) emitters, and pinned across
+them by `//domains/platform/libs/otel_contract`: names, descriptions, label
+sets, route sentinels and bucket boundaries alike. Those services are who
+scrapes this, and their dashboards (prom_proxy) query exactly these names with
+exactly these labels. A knob here would be a way for one service to drift off
+that contract, and the drift is silent — the panel renders empty, which looks
+like a quiet service rather than a misconfigured one. If a second fleet ever
+needs a different dialect, that is the point to design one.
+
+`service_name` is required whenever metrics are enabled, and an empty one
+aborts at construction: every dashboard query selects on it, so a service
+reporting the empty string is scraped, stored, and invisible.
+
+Three consequences of the contract worth knowing:
+
+- **Status is not a label.** The outcome rides on the success and failure
+  counters, which are two views of the same tally the total sums — so the
+  three can never disagree, and no series is multiplied by the codes a
+  service happens to return.
+- **The active gauge carries no route.** It moves at request start, before
+  dispatch, where nothing bounded is known about the path. Every rail leaves
+  the route off it for that reason, and prom_proxy's negative
+  `route!="/health"` matcher passes a series without the label through
+  untouched — which is what makes the same filter safe on it.
+- **Durations are microseconds**, on the ladder the rails pin equal.
+  `histogram_quantile` reads `le` off bucket counts, so a service on a
+  different ladder charts a quantile computed against different bins than
+  everything beside it.
+
+The label set is bounded by construction, because cardinality is what
+actually kills a metrics endpoint. `target` is deliberately *not* a label —
+it carries path parameters and query strings, so one series per distinct URL
+is one series per request id; `route` is the bounded stand-in the router
+stamps from the model, and a request that reached no operation reports the
+`unmatched` sentinel rather than the empty string (`route!="/health"` matches
+the empty string, so unrouted traffic would silently join the serving
+figures). `HealthEndpoint` and `MetricsEndpoint` answer paths the model does
+not define, so they stamp that path as their route (`route="/health"`):
+probes are usually a service's highest-volume route, and left unlabeled they
+would bury the 404 rate in the sentinel they share with it, and mix their own
+latency into the same duration histogram. The path is fixed at composition,
+so it is one series per composed endpoint — compose the probes inside
+`RecordMetrics` if you want them counted, outside it if you do not.
+`http_method` arrives from the wire, so anything outside the nine RFC 9110
+verbs collapses to `CUSTOM` rather than minting a series per invented verb,
+and a request rejected before its method parsed reports `(unparsed)`. Past
+`max_series` combinations the registry stops minting and counts what it
+refused in `metrics_observations_dropped_total` — alert on that being
+non-zero rather than discovering the cap as an OOM.
+
+Two things composition still has to get right for the MoonBase dashboards:
+compose `HealthEndpoint()` on its default `/health` path and *inside*
+`RecordMetrics`, because prom_proxy subtracts `route!="/health"` from every
+serving number and charts that route on its own tile — a service that never
+reports it reads as having no probe rather than as a healthy one. And point
+Prometheus at the service directly: this produces the collector's output
+shape without the collector.
+
+Your own metrics share the same scrape — one Prometheus target covers the
+service, rather than the built-in families sitting behind one endpoint and
+your domain numbers behind another. Mint a family once and keep the handle:
+
+```cpp
+auto orders = metrics->NewCounter("orders_processed_total", "Orders processed.");
+auto latency = metrics->NewHistogram("order_pipeline_seconds", "Pipeline time.");
+auto depth = metrics->NewGauge("queue_depth", "Pending jobs.");
+
+orders.Increment({{"region", "us-east"}});
+latency.Observe(elapsed.count());
+depth.Set(pending);
+```
+
+Declare the series whose labels are known at startup — `orders.Declare({{"region",
+"us-east"}})`, or `depth.Declare()` for an unlabeled one. A series nobody has
+touched is simply absent from the scrape, and a counter whose first exported
+sample is its first event's value hides that event for good: `increase()` and
+`rate()` measure the change *between* samples, so with nothing earlier the
+first one shows no increase at all and the panel reads zero — worse than a
+missing tile, because it looks like an answer. Declaring is idempotent and
+never disturbs a series that already has events. A declared histogram is
+genuinely empty rather than an observation of zero, so `rate(_sum)/rate(_count)`
+stays unbiased. Labels carrying request data have no series to declare (and
+are the cardinality problem above); bound them to a known kind and declare
+that instead.
+
+Handles are cheap to copy and address the same family, so a handler can hold
+them as members. The registry keeps owning the parts that are easy to get
+wrong: label values are escaped, labels are sorted so `{a,b}` and `{b,a}` are
+one series rather than two, and the same per-family cap applies — a label
+taken from unbounded data (a user id) costs that family its series budget and
+is attributed on
+`metrics_observations_dropped_total{metric="..."}` instead of taking
+the process down. A metric name that isn't a valid Prometheus name, or that
+collides with an existing family under a different type, aborts at
+registration: both produce a scrape Prometheus rejects in full, and nothing
+in-process would notice.
+
+One more hook is worth wiring, because middleware cannot reach it. The
+transport answers over-limit requests (413/431) itself, while the parser is
+still reading and before any handler chain exists — so `RecordMetrics` never
+sees them and an over-limit flood would be invisible in the counters:
+
+```cpp
+options.on_rejected = smithy::server::RecordRejections(metrics);
+```
+
+These count as requests (`operation=""`, with `413`/`431` as the signature)
+but file no latency and never move the in-flight gauge: a request refused at
+parse time has no service latency to report, and recording it as a zero
+observation would drag `rate(_sum)/rate(_count)` down — flattering the
+latency panel during exactly the flood it should be exposing. A method that
+never parsed (a 431 can fire mid-headers) is labeled `unparsed` rather than
+`other`, since "never parsed" and "client invented a verb" are different
+diagnoses.
+
+The endpoint is unauthenticated: it is middleware, so gate it the way you
+gate anything else — compose `Guard` or `RequireBearerAuth` outside it, or
+bind the scrape listener somewhere the internet cannot reach.
+
 **OpenTelemetry:** not bundled, by design — opentelemetry-cpp's dependency
 tree (protobuf, gRPC for OTLP) would violate the runtime's dep-light rule.
 The hooks above map 1:1 onto OTel spans and metrics; an optional