A dependency-free Prometheus /metrics endpoint (#91)

[aaylward]

What

The first work item of #91, server side. MetricsRegistry aggregates the existing Observe hooks; MetricsEndpoint serves them in the Prometheus text exposition format, which needs no client library — so this costs zero new dependencies and lives in :server directly. RecordMetrics is Observe wired to a registry, built on it rather than beside it so request timing keeps one implementation.

auto metrics = std::make_shared<smithy::server::MetricsRegistry>(
    smithy::server::MetricsOptions{.enabled = true, .service_name = "todo-service"});
transport.Start(smithy::server::Chain({MetricsEndpoint(metrics),
                                       RecordMetrics(metrics),
                                       HealthEndpoint()},
                                      server.Handler()));

Off unless asked for, and absent rather than idle when off

MetricsOptions::enabled defaults to false. Disabled, RecordMetrics and MetricsEndpoint compose to the identity — not a wrapper that checks a flag, but no wrapper at all, so a served request runs the call chain it would have run had this never been written. /metrics then reaches the router like any other unmodeled path and 404s; an empty 200 there would read to Prometheus as a live target reporting no series, which is indistinguishable from a service whose metrics have gone silent. Handles from a disabled registry are inert rather than unusable, so application code never branches on the flag; only their arguments still cost anything, so enabled() is exposed for a hot call site whose labels are expensive to build.

Registration is deliberately not conditional on the flag. An invalid metric name, a type collision, or a missing service_name aborts at startup either way (ADR-0009), so switching metrics on in production is never the first time those checks run.

The exposition is MoonBase's, and is not configurable

Five families labeled by service_name, http_method, route:

Family	Type
http_server_requests_total	counter
http_server_requests_success_total	counter (status < 400)
http_server_requests_failure_total	counter (status >= 400)
http_server_requests_active_gauge	gauge (no route)
http_server_request_duration_microseconds	histogram

plus metrics_observations_dropped_total, the registry's own health.

This is not a vocabulary invented here. It is MoonBase's shared HTTP serving contract, spoken identically by its Java (yodel), Rust (server_pal) and C++ (futility/otel, behind aura) emitters and pinned across them by //domains/platform/libs/otel_contract — names, descriptions, label sets, route sentinels and bucket boundaries alike. Those services are who scrapes this, and prom_proxy queries exactly these names with exactly these labels.

An earlier revision of this branch made all of it configurable with a preset. That was surface with no user, and worse: a knob is a way for one service to drift off a contract the fleet shares, and the drift is silent — the panel renders empty, which looks like a quiet service rather than a misconfigured one. MetricsOptions is now three fields that are still genuine choices: whether it runs, who it says it is, and the cardinality cap.

Things the diff doesn't show:

• Status is not a label. The outcome rides on the success and failure counters, which are two views of the same status-keyed tally the total sums — so the three can never disagree, they need no drop accounting of their own, and no series is multiplied by the codes a service happens to return.
• The active gauge carries no route. It moves at request start, before dispatch, where nothing bounded is known about the path. Every rail leaves the route off it for that reason, and prom_proxy's negative route!="/health" matcher passes a series without the label through untouched — which is what makes the same filter safe on it.
• service_name is required when enabled, and empty aborts. Every dashboard query selects on it, so a service reporting the empty string is scraped, stored, and absent from all of them: success everywhere except the panel nobody is watching yet. The transport tests found this the moment it landed.
• Labels are bounded by construction. target is never a label — path parameters and query strings make it one series per request id. An unrouted request reports unmatched rather than the empty string, because route!="/health" matches the empty string and unrouted traffic would silently join the serving figures. http_method arrives off the wire, so anything outside the nine RFC 9110 verbs collapses to CUSTOM (case-sensitively — a rejected get must not report as served GET), and a request rejected before its method parsed reports (unparsed). A per-family cap backstops the rest, attributed on metrics_observations_dropped_total{metric="..."}.
• NewHistogram has no default buckets. The built-in ladder is microseconds and shaped for request latency; inheriting it silently for a histogram of bytes would produce a chart whose bins mean nothing.
• The endpoint is unauthenticated — it is middleware, so gate it like anything else, or bind the scrape listener privately. Noted in the guide.

Folded in: probes stop reporting as the unrouted sentinel

HealthEndpoint built its response without stamping HttpResponse::operation, so every probe reached Observe — and so any backend, not just this one — under the same value 404/405/400 dispatch failures use. An orchestrator polls liveness every few seconds, making it usually the highest-volume path a service has, so the 404 rate was buried under probe volume, the probe's latency contaminated the service duration histogram, and no query could separate them. MetricsEndpoint had the same hole.

Both now report their composed path. It is fixed at composition, never off the wire, so this costs one series per composed endpoint rather than reopening the cardinality question target was excluded for. It is also the prerequisite for prom_proxy's fleet-wide route!="/health" subtraction: compose HealthEndpoint() on its default path and inside RecordMetrics, and the Probes tile finds it.

Testing

Unit tests over the registry, the application families, and the composed middleware: exposition format (HELP/TYPE, cumulative buckets, +Inf, sum/count), sub-millisecond latencies surviving the microsecond hook, concurrent recording, method clamping, cap accounting, label escaping and ordering, registration aborts, a handle outliving its registry, and in-flight pairing including the throwing-handler path.

The ExpositionContract suite pins the MoonBase literals — the five names with their descriptions, the label set, the outcome split at 400, the gauge keyed by method and never by route, the route and method sentinels, and the 15-bound microsecond ladder. Those tests are what stops this rail drifting off the pin, and silence is how such a drift would otherwise show up.

The DisabledRegistry suite pins the other half. The identity-composition test uses a plain function as the terminal so std::function::target is non-null exactly while nothing has wrapped it, and asserts the enabled case does wrap — without that second half the first proves nothing.

Two tests over a real Beast socket. Nine in the out-of-tree consumer module, driving the generated router so the route label is the model's and not a fixture's: the outcome counters and duration under the model's route, probes separable from unrouted traffic, application metrics on the same scrape, declared series at zero before traffic, and the disabled path 404ing while the service still works.

Locally: make verify (127 tests, lockfiles, codegen, goldens, clang-format), make noexcept, gcc ASan and TSan, clang-tidy, buildifier via npx, and the 14-test consumer suite. Two sandbox gaps, both verified another way: make lint's buildifier has no system binary here (run CI's way via npx @bazel/buildifier@8.2.1 it passes), and clang-tidy cannot parse beast_transport.cc without Boost headers, which CI installs — every remaining finding is downstream of that one file not found, and the files this PR touches are clean.

Deliberately not covered

The client-side registry #91 item 1 also asks for. AttemptObservation carries no duration, so the per-attempt latency histogram can't be built from ObserveAttempts today — that's the open API-shape question in the issue's own comment, and item 3 sequences hook changes separately and first. Items 2–4 remain, so this does not close #91.

Second item carried: one commit updates docs/working-agreement.md from MoonBase's fork of it (adapted, not copied — its receipts, squash-merge rationale, and "no ADRs/CHANGELOG" section are replaced or dropped). Unrelated to the metrics work, kept here by request.

Follow-up filed: #201, on removing smithy from runtime surfaces that have nothing to do with the IDL. The metric names were the visible instance and are fixed here; the namespace, include root and Bazel labels are a breaking rename that needs a decision first.

Checklist

• Tests added/updated for the change
• bazel test //... and (cd codegen && gradle build spotlessCheck) pass locally
• Formatting clean (clang-format, buildifier via npx, spotless)
• Architectural decisions recorded as an ADR (not applicable — a new middleware on the existing composition pattern; no new architecture)

🤖 Generated with Claude Code

https://claude.ai/code/session_014C7WdBD99mUFWGxMvGSjSU