[RUM] Auth endpoints started 503ing on 2026-09-01 — 17 of 26 /Login attempts failed in 24h, and sign-in shows the same dead end as a wrong password

[dawsontoth]

Summary

Since 2026-09-01 20:27 UTC, the central-manager API has started returning 503 on its
unauthenticated auth resources, and only those. In the last 24h /Login answered 503 more often
than it answered anything else, and Studio's sign-in form cannot tell the user this is a server
problem rather than a bad password — so people retype their credentials until they give up.

Two halves, one issue: the 503 itself is server-side, the dead-end UX is ours.

The 503s (RUM, last 24h, prod v2.169.3)

Status mix per auth endpoint on fabric.harper.fast:

Endpoint	2xx	401/403	503
POST /Login	9	6	17
POST /User (sign-up)	1	2	5
POST /ForgotPassword	1	–	3
GET /User/current	121	10	11

• 17 of the 26 non-credential /Login responses were 503 — a majority of sign-in attempts failed.
• 9 sessions affected out of 104 in the window (~9%).
• Of the 4 sessions that hit a /Login 503, 3 never got a successful login at all (session-id
  set intersection of 503-vs-200 on /Login, 48h window).

This is new. Zero 503s on any of these paths in the preceding 30 days — the first one in the
whole 30-day window is 2026-09-01T20:27:14Z. It is also growing: 5 events in the 48h–24h window,
36 in the last 24h.

It is not an outage, and it is not the client's connectivity

Scoped to the auth resources only. In the same 24h the same host served 14,708 200s, and
every affected session had between 32 and 2,604 successful requests to that same host alongside
its 503s. So the API is up; the auth path specifically is rejecting.

Two duration populations, which may be two causes:

• fast rejects — /Login 503s at 18–21 ms, repeated
• slow rejects — /User/current 390–1,254 ms, POST /User 662–756 ms, and one /ForgotPassword
  that took 50.6 s before its 503

Spread across five countries and both real Chrome sessions and one crawler, so it is not one
client or one region.

central-manager's Login.post (src/resources/Login.js) throws only ClientError with
400/401/403/409 — it has no 503 path — so the 503 is coming from below or in front of the resource
(Harper core or the ingress), not from a credential decision. A server-side look is needed to say
which; Harper core has several ServerError(…, 503) emitters reachable from a User.search
(queue-limit and worker-thread ones), but RUM can't distinguish them.

The client half: a retryable failure is indistinguishable from a wrong password

SignIn.tsx renders no inline error at all. Every sign-in failure goes through
useCloudSignIn's onError → errorHandler (src/react-query/queryClient.ts), i.e. a generic
toast that fades, away from the inputs. A 503 with no usable body skips describeError's Axios
branch and falls back to the Axios message, so the user is shown the bare
Request failed with status code 503 with nothing saying try again in a moment — or whether
the request even got far enough to check their credentials.

Corrected 2026-09-03 (this section originally said the user sees "We had some trouble!").
That string is the next fallback down and requires message to be absent too, which a real
AxiosError never is — it appears only in tests whose fixture omits it. Verified by running
describeError against a realistic payload. The defect is the same either way: an unactionable
string that drives credential retries. Worse, a 503 served by an edge proxy returns an HTML
body, which describeError treats as the server's message and renders verbatim.

The observed behaviour is exactly what that predicts: one session submitted /Login nine times
over 17 minutes, another four times in twelve seconds.

This is the same defect #1612 described for
sign-up, which #1613 fixed by reporting the
failure in the form. Sign-in never got that treatment — and sign-in is the endpoint now
failing. ForgotPassword and the sign-up path are worth the same audit.

Client fix

Mirror #1613 on sign-in: render the failure inline, and answer a retryable failure (5xx, 429,
network error, timeout) with our own copy instead of the server's unactionable text. Keep the
server's own message for 4xx, which is genuinely actionable (invalid credentials, unverified email,
deactivated account, SSO-required) and which isEmailNotVerifiedError already keys off.

Note added 2026-09-03. Cross-model review found a second reason not to render a 5xx body here,
beyond it being unactionable: these pages are anonymous and the new inline alert persists where
the old toast faded, so a 5xx body would show our own infrastructure to signed-out visitors —
Harper's exceeded request queue limit for resolving cache record, or an upstream
connect ECONNREFUSED <internal-address>. The fix therefore gates on status, never on whether
the body looks presentable; two earlier attempts to gate on body content both leaked.

Queries

APP='@application.id:<studio-rum-app-id>'
pup rum aggregate --query "$APP @type:resource @resource.url:*/Login*" \
  --compute count --group-by '@resource.status_code' --from 24h --to now
pup rum events --query "$APP @type:resource @resource.status_code:503" --from 30d --to now

Note --group-by silently caps at 10 buckets — an ungrouped @resource.status_code:503 count is
how the 503 bucket was found at all, since grouping all statuses on the host dropped it.

[dawsontoth]

Still live: four more /Login 503s in the two hours before this comment (2026-09-03 ~14:00 UTC), so this is ongoing rather than a one-off window.

One correction/addition to the version note above, checked while working the client half: production runs v2.169.3 (tagged 2026-08-28) while v2.169.7 (2026-09-01) is the newest tag — four releases behind, the same shape as #1609. That gap is why the Canceled/Canceled unhandled family still shows in prod (50 events across 2 sessions in one window): a37c6344 drops it and is not an ancestor of v2.169.3. Nothing to fix there — just worth knowing that the deployed build predates several RUM fixes, so absence of a fix in prod telemetry is not evidence the fix is wrong.

[dawsontoth]

The client half is up as a draft: #1678

It does not touch the 503 — that is still the server-side half this issue tracks. What it changes is that a signed-out visitor hitting one now gets copy that says whether to retry and whether their credentials were even checked, inline beside the inputs rather than in a toast that fades away from them.

One thing the review surfaced that is worth recording here rather than only on the PR: because that alert persists where the toast faded, rendering a 5xx body on these anonymous pages would expose infrastructure detail to unauthenticated visitors (Harper's exceeded request queue limit for resolving cache record, an upstream connect ECONNREFUSED <internal-address>). The fix therefore gates on status and never renders a 5xx body at all. Two earlier attempts to gate on whether the body "looked presentable" both leaked.

The server-side investigation is unchanged and still open here.