MQTT: anonymous-client disconnect fails will cleanup with AccessViolation ("Must login") on system.hdb_session_will

[harper-joseph]

Summary

With mqtt.requireAuthentication: false and anonymous MQTT clients, an abrupt (non-clean) client disconnect makes Harper read the internal system.hdb_session_will table using the disconnecting client's own request context — which has user: null and authorize: true. The authorization check fails with AccessViolation ("Must login", statusCode 401):

[warn]: Error publishing MQTT will for <sessionId> AccessViolation: Must login
    at authorizeActionOnResource (resources/Resource.ts)
    at hdb_session_will.applyContext (resources/Resource.ts)
    at <anonymous> (server/DurableSubscriptionsSession.ts)
    at transaction (resources/transaction.ts)
    at SubscriptionsSession.disconnect (server/DurableSubscriptionsSession.ts)
    at WebSocket.onClose (server/mqtt.ts)
  statusCode: 401

Under connect/disconnect churn (e.g. a reconnect storm after a restart), the transactions wrapping this cleanup pile up on hdb_session_will and get aborted:

[error]: Transaction was open too long and has been aborted after exceeding the open-transaction limit, from table: hdb_session_will/

Affected versions

• Confirmed on harper-pro 5.1.22 (current latest stable).
• Present on harper main as of a0f4a51 — server/DurableSubscriptionsSession.ts.

Root cause

SubscriptionsSession.disconnect() builds its context from createContext(), which sets user: this.user and authorize: true, then uses that context to touch the internal will store:

// server/DurableSubscriptionsSession.ts
createContext(): any {
  const context: any = { session: this, socket: this.socket, user: this.user, authorize: true };
  ...
}

disconnect(clientTerminated) {
  const context = this.createContext();
  transaction(context, async () => {
    try {
      if (!clientTerminated) {
        const will = await getLastWill().get(this.sessionId);   // authorized read of internal system.hdb_session_will
        if (will) await publishMessage(will, will.data, context);
      }
    } finally {
      await getLastWill().delete(this.sessionId);               // same problem
    }
  }).catch((error) => { warn(`Error publishing MQTT will for ${this.sessionId}`, error); });
}

getLastWill() is the internal system.hdb_session_will table. Accessing it under a context with authorize: true and user: null (anonymous session) runs authorizeActionOnResource → allowRead(null) on a system table → AccessViolation. The failure is on the .get(), before any will is published, so it fires on every non-clean anonymous disconnect regardless of whether the client actually registered a will.

For contrast, the startup-recovery path in the same file handles this correctly — it reconstructs the stored user before publishing and does not run under an anonymous authorizing context:

for await (const will of getLastWill().search({})) {
  const message = { ...will };
  if (message.user?.username) message.user = await (server as any).getUser(message.user.username);
  await publishMessage(message, data, message);
  getLastWill().delete(will.id);
}

Reproduction

1. Set mqtt.requireAuthentication: false.
2. Connect an anonymous MQTT client (with or without a will) over WS/TLS.
3. Drop the connection abruptly (non-clean disconnect).
4. Observe the AccessViolation: Must login warning. Under many concurrent disconnects, observe the hdb_session_will "transaction open too long" aborts.

Impact

• Log noise proportional to anonymous non-clean disconnects; spikes during restart-driven reconnect storms.
• Anonymous clients' Last-Will messages are never delivered on abrupt disconnect.
• Transaction pressure on system.hdb_session_will during storms (the open-transaction-limit aborts).

Severity is low in steady state (the cleanup path only warns and the failed transaction unwinds), but it is constant noise for any deployment that permits anonymous MQTT, and it silently breaks will delivery for those clients.

Suggested fix

Access the internal hdb_session_will store with an internal/system context (e.g. authorize: false, or a dedicated internal context) for the disconnect get/delete, rather than the client's authorizing context. When (re)publishing the will, reconstruct the stored user (as the startup-recovery path already does) so the publish is authorized as the will's owner rather than the live — possibly anonymous — session. The will-store bookkeeping is internal plumbing and shouldn't be gated on the disconnecting client's permissions.

Related

• MQTT: implement duplicate-clientId session takeover (will-message + durable-session races, esp. multi-worker) #1604 (MQTT duplicate-clientId session takeover; will-message + durable-session races) — same subsystem, distinct issue. The recent last-will persistence-race fix (a0f4a51) does not touch this authorization path.

_{Filed by an agent (Claude Code) on behalf of @harper-joseph, from a live investigation.}

[kriszyp]

Exploratory QA hit this three times across separate waves, and the measurements together widen the scope beyond what the title says. Consolidating them here rather than filing more issues.

It is not anonymous-only — it is any non-superuser

The title and original report scope this to anonymous clients. Measured: 12/12 connect/disconnect cycles leak for a non-anonymous, non-superuser client as well. The failing path is the same — disconnect()'s cleanup delete runs on the client's own context, which lacks permission on system.hdb_session_will, throws AccessViolation, and is swallowed by .catch(warn()).

So the trigger is "the disconnecting client's own identity cannot delete from a system table", and anonymous is just the most obvious instance of that. A fix scoped to anonymous clients would leave the majority of the surface leaking.

The will message is also dropped

Beyond the leaked row: the will does not publish for the affected clients. The earlier arm of this investigation reported "the good news is the will does NOT spuriously publish — no spec violation, purely a leak", and the wider arm contradicts that: for non-superusers the will is silently dropped, which is an MQTT spec violation, not just a housekeeping leak.

Why it looked intermittent

The orphan system.hdb_session_will row persists indefinitely on a running instance and is cleared only by the restart sweep. That means:

• instances that restart regularly appear to self-heal, and
• long-uptime instances accumulate one row per connect/disconnect cycle without bound.

This is what masked the leak in an earlier wave's clean-disconnect arm — the arm ran after a restart, so the sweep had already cleared it.

Suggested direction

The cleanup delete should run with the privilege the operation actually needs rather than the disconnecting client's context, and the .catch(warn()) should not be the only thing standing between a failed system-table write and silent data accumulation — a swallowed AccessViolation here is indistinguishable from success.

Numbers: 20/20 anonymous cycles leaked vs 0/20 authenticated-superuser (original arm); 12/12 non-superuser cycles leaked (wider arm). No re-measurement on current main in this pass — the three arms are QA-run measurements, and the code path was not re-read today, so treat the scope claim as high-confidence and the current-main status as unverified.

_{Consolidated from three exploratory-QA findings during a queue triage pass. Claude Opus 5.}

[kriszyp]

From dispatch finding (task qa-wave-2026072121):

• gh#1874 is scoped too narrowly: MQTT will-cleanup fails for ANY non-superuser, not just anonymous clients, leaking one permanent system.hdb_session_will row per connect/disconnect cycle (12/12 measured) and silently dropping the will message

[kriszyp]

From dispatch finding (task qa-wave-2026072121):

• Worker 0 deletes every row of system.hdb_session_will in a one-time startup IIFE (DurableSubscriptionsSession.ts, ~2 s after component load) outside any per-session auth context — unclear whether durable wills are meant to be discarded at boot, and it can mask the MQTT: anonymous-client disconnect fails will cleanup with AccessViolation ("Must login") on system.hdb_session_will #1874 leak on instances that restart regularly

[kriszyp]

Follow-up to the 2026-08-26 comment on this issue, which flagged the worker-0 startup sweep of
system.hdb_session_will as running outside any per-session auth context and noted it "can mask the
#1874 leak". A dispatch QA run has now reproduced the stronger consequence, so that open question can
be closed: the sweep does not merely mask a leak, it destroys a live client's will.

Reproduced 2/2: roughly 2s after boot the sweep publishes a still-connected client's will and
consumes the row, so that client's real ungraceful disconnect later delivers nothing. A broker
restart's reconnect stampede lands clients squarely in this window.

Mechanism on origin/main (2e65550d7c), server/DurableSubscriptionsSession.ts ~:70-89 — the
if (getWorkerIndex() === 0) IIFE (whenComponentsLoaded + 2000ms):

for await (const will of getLastWill().search({})) { … await publishMessage(…); await getLastWill().delete(will.id); }

No liveness check and no connection-identity check: every hdb_session_will row is published and
deleted unconditionally. git log --since=2026-07-20 on that file shows only unrelated changes
(#2341, #2040, #2176, TLS work) — the loop is unchanged.

Open design question this raises, which is really the fix decision: are durable wills meant to be
discarded at boot at all? If yes, the sweep still must skip rows whose owning session reconnects; if
no, the sweep should be removed and the rows left for the normal disconnect path.

— Claude (Opus 5), dispatch QA finding-triage pass