fix(security): honor the SQL permission denial processAST computes

DESIGN.md

@@ -391,6 +391,80 @@ early-returns (super_user, structure_user, system-table allowances): persisted r
 `super_user` with other permission keys, but inline roles can combine `structure_user` with an
 allowlist, and the gate ordering is what keeps unlisted schema ops unreachable.
 
+## The dispatched API operation is carried on async context, never on the request (`server/serverHelpers/operationAuthorizationState.ts`)
+
+`verifyPermsAST`'s token-scope check has to be told which top-level API operation the caller
+invoked, because the scope is written in that namespace (`sql`, `export_local`, ...). Two things
+make that awkward:
+
+1. On the **direct-SQL** path, the object handed to `checkASTPermissions` _is_ the client's request
+   body, and this check is the only gate there (`chooseOperation`'s `sql` branch is mutually
+   exclusive with its `verifyPerms` call). Any field read off that object is therefore a way to
+   name whichever operation the caller's scope happens to allow and run arbitrary SQL under it.
+   `jsonMessage.operation` is safe only because dispatch already routed on that same field, so it
+   cannot disagree with the operation running. Never add another.
+2. A **job** re-parses its SQL from the nested `search_operation` in a _different_ async context —
+   `executeJob` persists the request and hands off to the job runner, and `jobProcess.ts` re-enters
+   from the `hdb_job` record. So a store established around the originating request cannot reach
+   it, and the re-parse would be judged as `sql` rather than as the job's own operation.
+
+The carrier is therefore established **in the job worker**, by `runWithDispatchedOperation`, from
+the same `request.operation` that `getOperationFunction` just resolved the handler from. That
+identity is the whole basis for trusting it: the value naming the operation and the value selecting
+the code cannot diverge. A new carrier must preserve that property — an added request property, a
+`search_operation` field, or a persisted `parsed_sql_object` would not.
+
+This lives in the same `AsyncLocalStorage` as the auth bypass rather than a second store, so
+`processAST` reads the state once. `runWithOperationAuthorizationBypass` **preserves** an existing
+carrier on both branches. That is deliberate and was initially got wrong: its enforced branch is not
+a bypass, so a job handler dispatching a nested _authorized_ operation lands there, and dropping the
+carrier would judge that job's re-parsed SQL as the inner `sql` and refuse it partway through its own
+work. The consequence to know is the other direction — a nested dispatch inside a job is judged
+against the **outer** job's operation for any `evaluateSQL` that does not pass through
+`chooseOperation`. It allocates only when a carrier is present; with none, two shared frozen objects
+serve the common path. All four stores are frozen, so `getOperationAuthorizationState()` cannot hand
+a mutable one to a caller.
+
+It has four call sites, and they are not all dispatch wrappers: `server.operation()`
+(`serverUtilities.ts`), the ITC path (`registeredOperations.ts`), the legacy SQL engine
+(`sqlEngine/diff/differential.ts`), and Harper's own `hdb_job` query (`server/jobs/jobs.ts`) — that
+last one **is** reached from the ops-API dispatch, via `search_jobs_by_start_date` →
+`handleGetJobsByStartDate` → `getJobsInDateRange`.
+
+Harper's own internal SQL takes the bypass, not the carrier. `getJobsInDateRange` runs a fixed
+`system.hdb_job` query through `evaluateSQL` beneath a handler the caller was already authorized for,
+and `SqlSearchObject` hardcodes `operation: 'sql'` — so the same mismatch applies, but the answer
+differs, and the reason is easy to get backwards. `verifyPermsAST`'s super_user early return is
+`isSuperUser && !isSuSystemOperation`, so a `system` schema is **exempt** from it and the table check
+genuinely runs. A carrier would therefore put Harper's own query through `hasPermissions` on
+`system.hdb_job`, which passes only because `appendSystemTablesToRole` grants `system.*.read` to a
+hydrated super_user — a super_user principal without an appended `permission.system` (an
+impersonation payload, or any path that skips user-cache hydration) would start getting 403s on an
+operation it is entitled to. The bypass also states the actual intent: the statement is Harper's, not
+the caller's. Wrap the individual statement, not the function — a later caller-dependent statement
+must not inherit it.
+
+A second body field has to be neutralized for any of this to hold: `evaluateSQL` trusts a supplied
+`parsed_sql_object` verbatim and skips parsing, `chooseOperation` overwrites only the **top-level**
+one, and `dataLayer/export.ts` hands the nested `search_operation` straight to `evaluateSQL`. So a
+body-supplied `search_operation.parsed_sql_object` carrying `permissions_checked: true` would run an
+arbitrary AST with the check skipped. `chooseOperation` deletes it, forcing the worker to re-parse
+from the `sql` string that dispatch authorized — the nested object is never overwritten the way the
+top-level one is, because nothing downstream should read one at all.
+
+What is untestable is not the carrier's contract — unit tests cover that by calling
+`runWithDispatchedOperation` directly — but that `jobProcess` is what establishes it. Delete that call
+and those tests stay green. The carrier only changes an outcome through `tokenScopeDenial`, which is
+inert unless the principal carries `tokenOperations`, and that property has exactly one origin: an
+OIDC trust-policy exchange, for which there is no integration harness.
+
+Three different mechanisms are easy to conflate here. `tokenOperations` above is the **OIDC token
+operation scope** (#2174). An **inline-role scoped token** (`create_authentication_tokens` with a
+`role` object) is not the same thing and cannot substitute, because `createScopedToken` mints it
+`super_user: false`, so it cannot invoke a `requires_su` operation such as `export_local` at all.
+**Table permissions** are a third, and also cannot substitute — see the system-schema exemption
+above. See #2298.
+
 ## TLS hot-reload: cert vs. private key follow two different propagation paths (`security/keys.ts`)
 
 A renewed **certificate** and a renewed **private key** reach a worker's live TLS secure context

server/jobs/jobProcess.ts

@@ -9,6 +9,8 @@ import harperLogger from '../../utility/logging/harper_logger.ts';
 import * as globalSchema from '../../utility/globalSchema.ts';
 import * as user from '../../security/user.ts';
 import * as serverUtils from '../serverHelpers/serverUtilities.ts';
+import { runWithDispatchedOperation } from '../serverHelpers/operationAuthorizationState.ts';
+import { stripSuppliedParsedSqlObject } from '../serverHelpers/requestSanitization.ts';
 import moment from 'moment';
 import * as jobs from './jobs.ts';
 import { cloneDeep } from 'lodash';
@@ -54,12 +56,16 @@ const JOB_ID = JOB_NAME.substring(4);
 			throw new Error('Did not find job request in hdb_job table, unable to proceed');
 		}
 		request = cloneDeep(request);
+		// The worker re-enters from the persisted row rather than re-dispatching, so a row queued before
+		// the dispatch-time strip existed — or written directly — is sanitized here.
+		stripSuppliedParsedSqlObject(request);
 
 		const operation = serverUtils.getOperationFunction(request);
 		harperLogger.trace('Running operation:', request.operation, 'for job', JOB_ID);
 
-		// Run the job operation.
-		const results = await operation.job_operation_function(request);
+		const results = await runWithDispatchedOperation(request.operation, () =>
+			operation.job_operation_function(request)
+		);
 		harperLogger.trace('Result from job:', JOB_ID, results);
 
 		jobObj.status = hdbTerms.JOB_STATUS_ENUM.COMPLETE;

server/jobs/jobs.ts

@@ -11,6 +11,7 @@ import * as search from '../../dataLayer/search.ts';
 import Search_Object from '../../dataLayer/SearchObject.ts';
 import searchByHashObj from '../../dataLayer/SearchByHashObject.ts';
 import SQL_Search_Object from '../../dataLayer/SqlSearchObject.ts';
+import { runWithOperationAuthorizationBypass } from '../serverHelpers/operationAuthorizationState.ts';
 import * as hdbTerms from '../../utility/hdbTerms.ts';
 import JobObject from './JobObject.ts';
 import UpdateObject from '../../dataLayer/UpdateObject.ts';
@@ -249,7 +250,9 @@ export async function getJobsInDateRange(jsonBody: any) {
 			const hdbSql = require('../../sqlTranslator/index');
 			pSqlEvaluate = promisify(hdbSql.evaluateSQL);
 		}
-		return await pSqlEvaluate(sqlSearchObj);
+		// Harper's own statement, not the caller's. A carrier is not equivalent here — see DESIGN.md.
+		// Wraps only this statement: a later caller-dependent one must not inherit the bypass.
+		return await runWithOperationAuthorizationBypass(true, () => pSqlEvaluate(sqlSearchObj));
 	} catch (e) {
 		log.error(
 			`there was a problem searching for jobs from date ${jsonBody.from_date} to date ${jsonBody.to_date} ${e}`

server/serverHelpers/operationAuthorizationState.ts

@@ -1,11 +1,42 @@
 import { AsyncLocalStorage } from 'node:async_hooks';
 
-const operationAuthorizationState = new AsyncLocalStorage<boolean>();
+interface OperationAuthorizationState {
+	bypassAuth: boolean;
+	apiOperation?: string;
+}
+
+const operationAuthorizationState = new AsyncLocalStorage<OperationAuthorizationState>();
+
+// Shared, for the common case of no carrier to preserve.
+const BYPASSED: OperationAuthorizationState = Object.freeze({ bypassAuth: true });
+const ENFORCED: OperationAuthorizationState = Object.freeze({ bypassAuth: false });
 
 export function runWithOperationAuthorizationBypass<T>(bypassAuth: boolean, callback: () => T): T {
-	return operationAuthorizationState.run(bypassAuth === true, callback);
+	// An existing carrier survives: the enforced branch is not a bypass, and a job handler dispatching
+	// a nested authorized operation must not lose its own operation identity.
+	const apiOperation = operationAuthorizationState.getStore()?.apiOperation;
+	if (apiOperation === undefined) {
+		return operationAuthorizationState.run(bypassAuth === true ? BYPASSED : ENFORCED, callback);
+	}
+	return operationAuthorizationState.run(Object.freeze({ bypassAuth: bypassAuth === true, apiOperation }), callback);
+}
+
+/**
+ * `apiOperation` MUST come from the same value that selected the code now running, so it cannot name
+ * an operation other than the one executing. That is the entire basis for trusting it: a request
+ * property would be forgeable, and on the direct-SQL path this check is the only gate.
+ */
+export function runWithDispatchedOperation<T>(apiOperation: string, callback: () => T): T {
+	return operationAuthorizationState.run(
+		Object.freeze({ bypassAuth: operationAuthorizationState.getStore()?.bypassAuth === true, apiOperation }),
+		callback
+	);
+}
+
+export function getOperationAuthorizationState(): OperationAuthorizationState | undefined {
+	return operationAuthorizationState.getStore();
 }
 
 export function isOperationAuthorizationBypassed(): boolean {
-	return operationAuthorizationState.getStore() === true;
+	return operationAuthorizationState.getStore()?.bypassAuth === true;
 }

server/serverHelpers/requestSanitization.ts

@@ -0,0 +1,15 @@
+/**
+ * `evaluateSQL` honors a supplied `parsed_sql_object` verbatim and skips parsing, so one carrying
+ * `permissions_checked: true` executes an AST that no authorization check ever saw. Neither position
+ * is ever legitimately client-supplied: dispatch parses the statement itself, and the job worker
+ * re-parses from the `sql` string dispatch authorized.
+ *
+ * Both positions, because the two entry points differ: `chooseOperation` overwrites the top-level
+ * object with its own parse, but only inside its SQL branch, so a non-SQL job (`export_local` with a
+ * `search_by_value` search) can still carry a client-supplied one through to the persisted row.
+ */
+export function stripSuppliedParsedSqlObject(request: any): void {
+	if (!request) return;
+	delete request.parsed_sql_object;
+	delete request.search_operation?.parsed_sql_object;
+}

server/serverHelpers/serverUtilities.ts

@@ -51,6 +51,7 @@ import {
 	setLocalOperationDispatch,
 } from './registeredOperations.ts';
 import { runWithOperationAuthorizationBypass } from './operationAuthorizationState.ts';
+import { stripSuppliedParsedSqlObject } from './requestSanitization.ts';
 
 const pSearchSearch = util.promisify(search.search);
 let pEvaluateSql: (sql: string) => Promise<any>;
@@ -247,14 +248,10 @@ export function chooseOperation(json: OperationRequestBody, bypassAuth = false)
 		if (json.operation === 'sql' || (json.search_operation && json.search_operation.operation === 'sql')) {
 			const sql = require('../../sqlTranslator/index');
 			const sqlStatement = json.operation === 'sql' ? json.sql : json.search_operation.sql;
+			// Before this dispatch's own parse is assigned, so a body-supplied object cannot survive it.
+			stripSuppliedParsedSqlObject(json);
 			const parsedSqlObject = sql.convertSQLToAST(sqlStatement);
 			json.parsed_sql_object = parsedSqlObject;
-			// NOTE: a job's SQL is re-parsed from its nested search_operation when the job runs, so the
-			// check there sees `sql` rather than the job's own operation. Carrying the real one on the
-			// request was tried and reverted: on the direct-SQL path the request is the client's body,
-			// so any property consulted by that check is forgeable, and it is the only gate on that
-			// path. This changes no outcome today — the branch that would act on the denial is dead
-			// (#2202) — but #2202 needs an unforgeable carrier before making it live.
 			if (!bypassAuth) {
 				// The SQL path never reaches verifyPerms, so the role `operations` allowlist must be
 				// enforced here — otherwise an allowlisted role reaches unlisted operations via `sql`.

sqlTranslator/index.ts

@@ -17,7 +17,7 @@ import * as terms from '../utility/hdbTerms.ts';
 import { handleHDBError } from '../utility/errors/hdbError.ts';
 import { HTTP_STATUS_CODES } from '../utility/errors/commonErrors.ts';
 import * as sqlEngineRouter from '../sqlEngine/router.ts';
-import { isOperationAuthorizationBypassed } from '../server/serverHelpers/operationAuthorizationState.ts';
+import { getOperationAuthorizationState } from '../server/serverHelpers/operationAuthorizationState.ts';
 
 //here we call to define and import custom functions to alasql
 alasqlFunctionImporter(alasql);
@@ -90,20 +90,20 @@ export function checkASTPermissions(jsonMessage: any, parsedSqlObject: any, apiO
 			// a caller to name whichever operation their token scope happens to allow and run arbitrary
 			// SQL under it.
 			//
-			// That rules out carrying the job's real operation on the request too. A job re-parses from
-			// its nested search_operation, so this sees `sql` rather than `export_local` there — which
-			// today changes nothing, because the branch in processAST that would act on the denial is
-			// dead (see #2202). When #2202 makes it live, the job's operation needs a carrier that a
-			// client cannot forge; a request property is not one, however carefully it is stripped.
+			// A job's re-parse would otherwise land here with the nested search_operation, whose
+			// `operation` is the inner `sql`; the job worker supplies the real one out of async context
+			// instead (operationAuthorizationState), which a request cannot set.
 			apiOperation ?? jsonMessage.operation
 		);
-		parsedSqlObject.permissions_checked = true;
 	} catch (e) {
 		throw e;
 	}
 	if (verifyResult) {
 		return verifyResult;
 	}
+	// Only after a pass: this flag is what processAST trusts to skip the gate, so a denied AST
+	// carrying it would read as already authorized.
+	parsedSqlObject.permissions_checked = true;
 	return null;
 }
 
@@ -150,15 +150,17 @@ export function processAST(jsonMessage: any, parsedSqlObject: any, callback: any
 		// runWithOperationAuthorizationBypass), never body state — jsonMessage.bypass_auth is
 		// caller-controlled and is stripped before operation code sees it (see
 		// server/serverHelpers/serverHandlers.js and components/mcp/tools/operations.ts).
-		if (!isOperationAuthorizationBypassed() && !parsedSqlObject.permissions_checked) {
-			let permissionsCheck = checkASTPermissions(jsonMessage, parsedSqlObject);
-			// NOTE: this guard is dead — PermissionResponseObject has no `length`, so `undefined > 0`
-			// discards a denial that was computed correctly. Pre-existing and not specific to this
-			// feature, so it is fixed separately in #2202 rather than bundled here. This PR does not
-			// depend on it: the outer gate in serverUtilities refuses an out-of-scope job operation,
-			// and sqlWriteScopeDenial refuses write SQL, both through correct truthiness tests.
-			if (permissionsCheck && permissionsCheck.length > 0) {
-				return callback(UNAUTHORIZED_RESPONSE, permissionsCheck);
+		if (!parsedSqlObject.permissions_checked) {
+			const authorizationState = getOperationAuthorizationState();
+			if (authorizationState?.bypassAuth !== true) {
+				let permissionsCheck = checkASTPermissions(jsonMessage, parsedSqlObject, authorizationState?.apiOperation);
+				// A denial is a PermissionResponseObject, which has no `length`.
+				if (permissionsCheck) {
+					// Logged here because this is the last place the reason exists: evaluateSQL drops the
+					// second callback argument, so a job worker records only the bare status.
+					logger.warn('SQL statement refused by AST permission check:', permissionsCheck);
+					return callback(UNAUTHORIZED_RESPONSE, permissionsCheck);
+				}
 			}
 		}