What happened
POST /listSandboxMetrics catches every rejection from executor.metrics() and removes that sandbox from the response. The documented skip case is a container that vanished during measurement, but the catch-all also swallows Docker daemon outages, docker stats failures, mount/statfs failures, and other accounting errors.
Focused reproduction:
- Acquire one active sandbox in a route test.
- Replace
executor.metrics with a rejection such as Error('dockerd unavailable').
- Call
POST /listSandboxMetrics.
Expected: An unrelated runtime failure should produce an error response, or an explicit per-sandbox error representation. Only the narrow vanished-container race should be omitted.
Actual: The endpoint responds HTTP 200 with samples: []. With more sandboxes it returns a silently partial fleet.
Impact: Monitoring clients cannot distinguish an unhealthy Docker/runtime host from a healthy host with no measurable sandboxes, so outages can look like successful empty telemetry.
Relevant code and contract:
packages/server/src/routes/sandboxes.ts:455-487 catches all metrics errors and filters them to null.packages/shared/src/metrics.ts:46-52 reserves absence for non-measurable or vanished containers.packages/server/src/executor/docker.ts:533-577 shows other fallible operations covered by the same catch, including Docker stats and filesystem accounting.packages/server/src/routes/observability.test.ts:686-699 verifies only the narrow vanished-container case.website/content/docs/metrics.mdx:42-54 describes the endpoint as answering for the whole measurable fleet.
dor doctor output
Not applicable: the defect is an API error-classification path. A Docker outage may also be visible to Doctor, but this endpoint still returns success.
Environment
- Dormice commit:
4417ae14938273f5d937c476cf7cdc57e22e53a5
- API route:
POST /listSandboxMetrics
- Verification: focused route behavior with a rejected metrics provider, plus source-path inspection
What happened
POST /listSandboxMetricscatches every rejection fromexecutor.metrics()and removes that sandbox from the response. The documented skip case is a container that vanished during measurement, but the catch-all also swallows Docker daemon outages,docker statsfailures, mount/statfs failures, and other accounting errors.Focused reproduction:
executor.metricswith a rejection such asError('dockerd unavailable').POST /listSandboxMetrics.Expected: An unrelated runtime failure should produce an error response, or an explicit per-sandbox error representation. Only the narrow vanished-container race should be omitted.
Actual: The endpoint responds HTTP 200 with
samples: []. With more sandboxes it returns a silently partial fleet.Impact: Monitoring clients cannot distinguish an unhealthy Docker/runtime host from a healthy host with no measurable sandboxes, so outages can look like successful empty telemetry.
Relevant code and contract:
packages/server/src/routes/sandboxes.ts:455-487catches all metrics errors and filters them tonull.packages/shared/src/metrics.ts:46-52reserves absence for non-measurable or vanished containers.packages/server/src/executor/docker.ts:533-577shows other fallible operations covered by the same catch, including Docker stats and filesystem accounting.packages/server/src/routes/observability.test.ts:686-699verifies only the narrow vanished-container case.website/content/docs/metrics.mdx:42-54describes the endpoint as answering for the whole measurable fleet.dor doctor output
Not applicable: the defect is an API error-classification path. A Docker outage may also be visible to Doctor, but this endpoint still returns success.
Environment
4417ae14938273f5d937c476cf7cdc57e22e53a5POST /listSandboxMetrics