Turnkey observability backends: OTel adapter and a dependency-free /metrics endpoint

[aaylward]

Broken out of #49 (its last unchecked generated-API item) and graduating #29's "OpenTelemetry adapter" roadmap line, per that issue's convention.

What exists today (hooks, no backends)

• Client: Observe/ObserveAttempts (smithy/client/observability.h) — per-call and per-attempt records.
• Server: RequestObservation middleware, Guard, HealthEndpoint (smithy/server/middleware.h).
• Tracing: W3C traceparent propagation client-side, smithy/http/trace_context.h helpers (ParseTraceparent/FormatTraceparent/GenerateTraceContext), server observation carries the incoming traceparent.

Everything maps cleanly onto spans/metrics — but every consumer hand-rolls the export. The production guide documents this posture explicitly: opentelemetry-cpp's dependency tree (protobuf, gRPC for OTLP) would violate the runtime's dep-light rule, so nothing is bundled.

Work items

1. Dependency-free Prometheus /metrics first — the exposition text format needs no client library: a smithy::server::MetricsEndpoint middleware (same composition pattern as HealthEndpoint) aggregating request counts/latencies/status classes from the existing observation hooks, plus a client-side registry fed by ObserveAttempts. Zero new deps, so it can live in :server/:client directly — likely most of the consumer value at a fraction of the OTel effort.
2. //runtime:otel adapter — bridges the hooks to opentelemetry-cpp behind its own target (the //runtime:http_beast isolation pattern: heavy deps enter the graph only for consumers that opt in). Spans from RequestObservation/ObserveAttempts + traceparent continuity; OTLP exporter config left to the consumer.
3. Hook-shape check before committing — the guide gates this on "the hook shapes have survived production use"; the adapter work doubles as that audit. Any hook change lands separately, first, with pins.
4. Production-guide observability section gains the turnkey paths (metrics endpoint + OTel wiring), with the compiled-and-mirror-pinned example convention.

Notes

• Module-graph caution from Boost 1.90: beast/asio to 1.90.0.bcr.1, boringssl pinned to match #89: opentelemetry-cpp on BCR drags large transitive graphs — same drill as the boost bump (explicit pins for anything security-relevant, check what toolchains enter the graph, expect proxy-blind local dev per docs/development.md).
• Post-0.1.0 roadmap (tracking) #29's OTel line should be annotated as graduated here (done in the same pass as this issue).

Refs: #49 (origin), #29 (roadmap line graduating), docs/production-guide.md Observability section, PLAN §7c.

[aaylward]

First concrete input to the hook-shape audit (work item 3), from real migration feedback:

• Fixed in Server Observe reports microsecond durations — milliseconds zeroed the fast path #92: RequestObservation::duration was std::chrono::milliseconds, so sub-millisecond requests (cache hits, loopback) observed as 0 — the consumer measured around the hook instead of with it. Now std::chrono::microseconds.
• Still open, needs design: the client side has no duration at all — AttemptObservation carries method/target/attempt/status but no latency, so a per-attempt latency histogram (the core client metric this issue's registry item needs) can't be built from ObserveAttempts today. The blocker is structural: ObserveAttempts is an Interceptor, and interceptors are shared across a client's concurrent calls with no per-call context — there's nowhere thread-safe to stash the attempt's start time between ModifyBeforeTransmit and ReadAfterTransmit. Options include timing in the retry loop itself (SendWithRetries owns the attempt boundaries) and passing the duration into the observation record, or growing the interceptor API a per-call context. Either way it's an API-shape decision that belongs to this issue, not a drive-by field.

---

Generated by Claude Code

[aaylward]

Progress and scope corrections on work item 1, from #199.

Landed in #199 (server half)

MetricsRegistry + MetricsEndpoint + RecordMetrics in :server, zero new deps. Built-in families: smithy_http_requests_total{method,operation,status}, smithy_http_request_duration_seconds{method,operation}, smithy_http_requests_in_flight.

Three things settled while building it, recorded here so they don't get re-litigated:

• target cannot be a label — path parameters and query strings make it one series per request id. operation is the bounded stand-in the generated router already stamps. Only an out-of-tree test against the generated server proves that label is populated; in-tree tests use hand-written handlers that stamp it themselves and would keep passing if the router stopped.
• method is attacker-controlled. It arrives off the wire, so a loop of curl -X <random> mints a series per invented verb. Anything outside the RFC 9110 set collapses to other, case-sensitively — a rejected get must not report as served GET.
• Status is the exact code, not a class. Bounded either way, and {status=~"5.."} recovers the class at query time while the reverse loses information. This diverges from the issue text ("status classes") deliberately.

Also added beyond the issue text: application metrics on the same scrape (NewCounter/NewGauge/NewHistogram), because an endpoint that can't carry the service's own domain metrics isn't turnkey — a consumer would otherwise need a second registry behind a second endpoint, and a Prometheus target scrapes one.

Ported from MoonBase

futility/otel's zero baseline (MoonBase #1323): a series nobody has touched is absent from the scrape, so a counter's first exported sample is its first event's value — increase() has nothing earlier to measure against and hides that event for good, leaving a panel that reads zero. Declare() materializes a series at its baseline.

One improvement over the original: MoonBase #1384 records that a histogram cannot be declared at zero, since recording 0 is an observation that biases rate(_sum)/rate(_count). That constraint comes from OTel's record-only API. This implementation writes the exposition directly, so a declared histogram is genuinely empty — every bucket, _sum and _count at 0, contributing nothing to the mean.

Still open on item 1

The client-side registry is blocked, exactly as the comment above describes: AttemptObservation carries no duration, so the per-attempt latency histogram can't be built from ObserveAttempts. That is the API-shape decision this issue owns, and work item 3 sequences hook changes separately and first. #199 does not touch it.

New follow-up found

Transport-level rejections are invisible to metrics. BeastServerTransport writes 413/431 before any handler chain exists (Options::on_rejected), so RecordMetrics — being middleware — never sees them. An over-limit flood would not appear in the request counters at all. MoonBase's aura closes this with a RejectionMetrics sink feeding the same instruments, plus a distinct (unparsed) method label for requests rejected before a method existed (a 431 can fire mid-headers, so "never parsed" and "invented verb" are different diagnoses). Not folded into #199 — it is a new capability rather than a fix to that diff — but it is a real hole in the metrics story and belongs to this issue.

Items 2–4 remain untouched, so #199 references this issue without closing it.

---

Generated by Claude Code