Skip to content

Record every tool call in an audit log - #1554

Open
midego1 wants to merge 7 commits into
UsefulSoftwareCo:mainfrom
midego1:claude/tool-call-audit-log
Open

Record every tool call in an audit log#1554
midego1 wants to merge 7 commits into
UsefulSoftwareCo:mainfrom
midego1:claude/tool-call-audit-log

Conversation

@midego1

@midego1 midego1 commented Aug 7, 2026

Copy link
Copy Markdown

The gap

Executor keeps no record of tool usage. A run that calls GitHub or Search Console leaves one HTTP line —

INFO http.span: Sent HTTP response { "http.method": "POST", "http.url": "/mcp", "http.status": 200 }

— and nothing about which integration, which tool, or what came back. There is no executions/audit table in the schema, and packages/core/analytics is anonymous by construction: its own header forbids tool addresses, connection names and arguments, which are exactly the fields an audit needs. The executor.tool.execute span carries the right data, but only where an OTel exporter is configured, and it can't answer "which connection did this agent use last week".

Two questions were therefore unanswerable after the fact: what did this agent touch, and what did my policies actually stop.

What this adds

execute in packages/core/sdk/src/executor.ts is the one place every call passes through — every plugin kind, every host, MCP and REST alike — so the row is written there, from an Effect.onExit wrapper that sees every way a call can end:

outcome meaning
ok reached the upstream and succeeded
fail the tool's own error result — which rides the success channel by design, so a channel-only reading records an upstream 404 as a healthy call
blocked a block policy stopped it
declined a human refused the approval
error tool/connection missing, plugin not loaded, transport broke

blocked and declined are the rows that make this worth having: both end before any request is made, so nothing upstream ever saw them and no HTTP-level observation can.

Each row carries the address as called, its integration/connection/tool, the governing policy (action + pattern), the duration, and the top-level argument names.

Readable three ways:

  • executor.toolCalls.list({ integration, connection, outcome, since, search, limit, offset })
  • GET /api/tool-calls with the same filters
  • the Activity page in the console

The Activity page

Activity overview

Filtering follows the console's existing patterns: outcome as FilterTabs, an integration dropdown drawn with the same favicon pipeline as the Integrations page, and a search box over the tool address (DB-level contains, deferred input). Every filter change resets to page 1.

Integration dropdown

Pagination is the Admin · Users pattern: 25 rows per page, one extra row fetched to answer "is there a next page" without a count query, Previous/Next with the mono page label. offset runs through all three layers, bounded at the HTTP edge like every other numeric input.

The blocked view — the row that exists nowhere else, because the call ended before any request was made:

Blocked filter

Read-only by construction: a log a caller can edit is not evidence, so there is no write or delete endpoint. executor.toolCalls.prune({ before }) exists for retention, and nothing schedules it — an audit log that silently deletes itself on a default nobody chose seemed worse than one that grows.

What it deliberately does not store

Arguments, results, and any text that came from outside:

  • Messages are never persisted. A failed call keeps its code; plugins derive error.message from upstream response bodies (the OpenAPI plugin lifts it straight out), which routinely echo the request back — token included.
  • Codes must look like codes. ToolError.code is typed as any string, so a plugin can forward a body into it. Anything not matching an identifier shape is dropped; the outcome column already says what happened.
  • Argument names, never values — and only names that look like parameters. execute takes unknown args, so { "ghp_realtoken": null } is a reachable shape; names are length-bounded, identifier-shaped, and credential-shaped ones are dropped.

Failure behaviour

Writing a row can never change the outcome of the call it describes. The write is wrapped in catchCause, so an insert failure — or a defect — is logged and swallowed. It is awaited on purpose (a forked write would be interrupted when a per-request host tears the executor down, and a silently missing row is the one thing an audit log may not do), but awaited under a 2s cap, so a sick database costs a tool call that timeout rather than the driver's own.

Scope and follow-ups

  • Local/self-host/Cloudflare need no migration: ensureDrizzleRuntimeSchemaFromTables creates the table at boot from coreTables. Cloud gets 0016_nosy_expediter.sql.
  • apps/local/src/db/executor-schema.ts is left alone — it already predates artifact/subject, and it only drives a generate-time baseline.
  • Known limitation: a subject's view spans two partitions (its own rows plus the org's), so a newest-first read across both still sorts. Serving it takes a (tenant, created_at) index and the schema layer has no non-unique index API yet. Worth adding before this table gets large; I left a note at the table definition rather than adding a unique index whose leading columns wouldn't serve the query.
  • Branch is current with main (merged through the MCP spec 2026-07-28 / SDK v2 work; clean merge, all gates re-run on the merged base).

Verification

  • New suite packages/core/sdk/src/tool-call-log.test.ts — 19 tests: outcome classification per ending (including a decline raised inside a handler, which arrives wrapped in ToolInvocationError), the redaction rules with real-looking secrets, list filtering/search/paging tiling without overlap, and executor-level assertions that a blocked call and a declined approval each leave a row.
  • format:check, lint, typecheck (44/44) clean; packages/core/sdk 613+, packages/core/api 98, packages/react 320 tests pass.
  • Running in production on a self-hosted instance since 2026-08-08 (self-built image from this branch); the screenshots above are that build with seeded demo rows, and the live instance has been recording real agent traffic — including policy-blocked calls — since.
  • Reviewed with codex review over two rounds; every finding from both is addressed in the branch.

🤖 Generated with Claude Code

midego1 and others added 7 commits August 7, 2026 19:39
Executor kept no record of tool usage. A run that called GitHub or Search
Console left one HTTP line (POST /mcp 200) and nothing about which integration,
which tool, or what came back — and the analytics catalog is anonymous by
construction, so it deliberately drops exactly those fields. Two questions were
therefore unanswerable after the fact: what did this agent touch, and what did
my policies actually stop.

`execute` is the one place every call passes through, whatever the plugin kind
and whatever the host, so the row is written there, from an `onExit` wrapper
that sees every way a call can end:

- ok / fail — reached the upstream. `fail` is a tool's own error result, which
  rides the SUCCESS channel by design and would otherwise be recorded as a
  healthy call.
- blocked / declined — never left the gateway. A policy stopped it, or a human
  refused the approval. These leave no other trace anywhere: they end before
  any request is made.
- error — the tool or connection did not exist, the plugin failed to load, the
  transport broke.

Arguments and results are never stored: an argument can be a credential. The
row keeps the top-level argument NAMES, which is what an audit needs without
the table becoming a place secrets accumulate.

Writing a row can never change the outcome of the call it describes — a failed
write is logged and swallowed, because an audit trail that can take the gateway
down with it is worse than one with a gap in it.

Readable three ways: `executor.toolCalls.list()`, `GET /api/tool-calls`
(filter by integration, connection, outcome, time), and an Activity page in the
console. Read-only by construction — a log a caller can edit is not evidence,
so there is no write or delete endpoint.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A day of agent traffic made the page one long scroll. Same pattern as
Admin · Users: 25 rows per page, one extra row fetched to know whether a
next page exists (splitPage), Previous/Next with the mono page label.

`offset` runs through all three layers — the executor list, the HTTP query
(bounded, like every numeric input on this API), and an Atom.family keyed on
the offset so paging back is instant while the front page keeps its 5s TTL.
The log is append-only at the top, so a page can shift while browsing; that
is fine for eyeballing, and programmatic sweeps use `since`.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Outcome filter as FilterTabs (All / Ok / Failed / Blocked / Declined /
Error) and a search box over the tool address — the one free-text field a
row has that is safe to search, because this codebase wrote it rather than
an upstream. Both reset to page 1: a new filter is a new list.

`search` runs through the same three layers as the other filters: a
DB-level `contains` on the executor list (LIKE wildcards in the query are
harmless — the match never leaves the caller's own partition), a bounded
string on the HTTP query, and the Atom.family key, which now carries
`offset|outcome|search`. The input defers the query, not the keystroke.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A Select between the outcome tabs and the search box, listing the tenant's
own integration catalog with the same brand marks the Integrations page
draws (integrationPresetIconUrl → IntegrationFavicon), so gsc and github
tell apart at a glance. Wired to the `integration` filter the API had from
the start; "" means all, carried as a sentinel because Select cannot hold
an empty value. Like every other filter change it resets to page 1.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant