Give a Bot its computer when nobody is watching - #298
Draft
jerelvelarde wants to merge 1 commit into
Draft
Conversation
The computer tools were registered in the browser with `useFrontendTool`, and every handler was a `fetch` back to `/api/computers/:botId/...` — a round trip into this same process. That made an open tab load-bearing. A Bot whose person had closed the window had no browser, no workspace and no shell, because the only thing that could carry out a tool call had gone away, and an unattended run was out of reach entirely. This repository has already made that move once, for MCP, and `plugins/tools.ts` still says why: it "made a browser a hard requirement for a Bot to do anything, which rules out an embedded widget, a run nobody is watching, and any surface that is not our own app." So twelve of the fourteen move to `computer/tools.ts` and execute here, offered through the same per-run capability seam that already hands a Bot `message_bot` and `ask_person`. Nothing about governance moves with them: every acting tool still goes through ComputerGateway, which resolves the ref against the snapshot this server took, evaluates the policy, writes the audit row, and only then acts. Two deliberately stay in the browser. `computer_request_secret` and `computer_request_help` both end with a person typing into a masked box or taking the wheel, so both need somebody present by definition; moving them would produce a tool a headless run can call and can never have answered. Such a run is not left mute — it still holds `ask_person`. Rendering stays in the browser too, through `useRenderTool`, which draws a call without claiming to execute it. The renderers are unchanged: they already parsed a JSON result, which is what these return. The activity pane had to change with it. It was written from the handlers, "which run exactly once per call", and `render` runs on every paint. So recordActivity now takes the tool call's id and is idempotent on it, and its listeners are notified in a microtask rather than synchronously inside another component's render.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
A Bot's computer is the thing this product is about: its own browser, its own logins, its own files,
and every action decided before it happens. Today none of it exists unless somebody has a tab open.
The fourteen computer tools were registered in the browser with
useFrontendTool, and each handlerwas a
fetchback to/api/computers/:botId/...— a round trip out of this process and into itagain. So the browser was not a client of the feature, it was a required component of it. Close the
window mid-task and the Bot loses its shell, its workspace and its browser, because the only thing
that could carry out a tool call has gone away. A run nobody is watching cannot use a computer at
all, which is the wall every unattended surface meets: a routine, a scheduled sweep, a delivery to a
Bot that a person is not sitting in front of.
This repository has already made this move once, for a different tool family, and
plugins/tools.tsstill says why in its opening paragraph:
Every word of that applies to the computer tools, which were left behind.
The approach
Twelve of the fourteen move to
server/src/computer/tools.tsand execute here. They are offeredthrough the per-run capability seam that already hands a Bot
message_botandask_person, sothere is no new plumbing: a
GrantedToolis already this codebase's word for "a tool the model maycall, executed here", and
escalation.tsandhandoff-tool.tsalready build them without goingnear a plugin grant.
Nothing about governance moves with them. Every acting tool still goes through
ComputerGateway,which resolves the ref against the snapshot this server took, evaluates the policy, writes the audit
row, and only then acts. The tools module hands the model a description of what it may call; the
gateway remains what decides whether a call happens.
Two deliberately stay in the browser.
computer_request_secretandcomputer_request_helpbothend with a person typing into a masked box or taking the wheel, so both need somebody present by
definition. Moving them would produce a tool a headless run can call and can never have answered — a
worse failure than not offering it, because the run stalls instead of adapting. Such a run is not
left mute: it still holds
ask_person, which is the honest exit for a Bot that needs a human.Rendering stays in the browser, through
useRenderTool, which draws a call without claiming toexecute it. The renderers themselves are unchanged, because they already parsed a JSON result and
that is what these return. The transcript still names the element the gateway resolved rather than
the ref the model sent.
The activity pane is the part that cost something. It was written from the handlers, and its own
docblock explained why: they "run exactly once per call", where a tool's
renderruns on everypaint. With the handlers gone,
renderis the only place the browser still sees a call, sorecordActivitynow takes the tool call's id and is idempotent on it. Its listeners are notified ina microtask rather than synchronously, because notifying from inside one component's render sets
state in another that is subscribed through
useSyncExternalStore. The cost is a required argumentat every call site and a
seenset that has to be cleared when a computer is wiped; both arecovered by tests, and the second is the one that would have rotted quietly.
What is not covered
own by design — the renderers are deliberately unchanged — so the only honest recording is a Bot
running a command with no tab attached, which needs a model credential this environment does not
have. It follows before this leaves draft.
and a workspace, but cannot ask for a password or hand over the wheel.
that walks through it.
report_refusalwrites its audit row through the capability seam rather than the existingPOST /api/agents/:id/declinedroute. Both now exist and write the same event type; the routeis still what the browser-side path would use, and collapsing them is left alone here rather than
changed in a commit about something else.
Verification
Run against a local Postgres with migrations applied.
mainatfb0c797: 2064 pass, 23 skip, 0 fail, 170 files.difference; nothing existing changed its result.
tsc --noEmitclean for bothserverandapp.biome lint --error-on-warnings .clean across 492 files.New and changed test files:
server/tests/computer-tools.test.ts— holds down that the Bot and the actor reach the gateway(the audit row is written from them), that a refusal comes back as an answer carrying the rule
rather than a throw that ends the run, that the failure modes stay distinguishable so a model can
tell a retry from a dead end, that bad arguments are refused before the gateway is touched, that
optional arguments are omitted rather than sent as
undefined, and that the two person-presenttools are not offered.
app/tests/computer-activity.test.ts— extended with the property the keyed signature exists for:one call is one entry however many times a render records it, and a wiped computer can record the
same call id again without it being swallowed as a duplicate.