Skip to content

Give a Bot its computer when nobody is watching - #298

Draft
jerelvelarde wants to merge 1 commit into
CopilotKit:mainfrom
jerelvelarde:jerel/computer-tools-server-side
Draft

Give a Bot its computer when nobody is watching#298
jerelvelarde wants to merge 1 commit into
CopilotKit:mainfrom
jerelvelarde:jerel/computer-tools-server-side

Conversation

@jerelvelarde

Copy link
Copy Markdown
Contributor

The problem

A Bot's computer is the thing this product is about: its own browser, its own logins, its own files,
and every action decided before it happens. Today none of it exists unless somebody has a tab open.

The fourteen computer tools were registered in the browser with useFrontendTool, and each handler
was a fetch back to /api/computers/:botId/... — a round trip out of this process and into it
again. So the browser was not a client of the feature, it was a required component of it. Close the
window mid-task and the Bot loses its shell, its workspace and its browser, because the only thing
that could carry out a tool call has gone away. A run nobody is watching cannot use a computer at
all, which is the wall every unattended surface meets: a routine, a scheduled sweep, a delivery to a
Bot that a person is not sitting in front of.

This repository has already made this move once, for a different tool family, and plugins/tools.ts
still says why in its opening paragraph:

The loop used to run in the browser: every MCP tool was registered with useFrontendTool and its
handler posted back to /api/plugins/call. That made a browser a hard requirement for a Bot to do
anything, which rules out an embedded widget, a run nobody is watching, and any surface that is
not our own app.

Every word of that applies to the computer tools, which were left behind.

The approach

Twelve of the fourteen move to server/src/computer/tools.ts and execute here. They are offered
through the per-run capability seam that already hands a Bot message_bot and ask_person, so
there is no new plumbing: a GrantedTool is already this codebase's word for "a tool the model may
call, executed here", and escalation.ts and handoff-tool.ts already build them without going
near a plugin grant.

Nothing about governance moves with them. Every acting tool still goes through ComputerGateway,
which resolves the ref against the snapshot this server took, evaluates the policy, writes the audit
row, and only then acts. The tools module hands the model a description of what it may call; the
gateway remains what decides whether a call happens.

Two deliberately stay in the browser. computer_request_secret and computer_request_help both
end with a person typing into a masked box or taking the wheel, so both need somebody present by
definition. Moving them would produce a tool a headless run can call and can never have answered — a
worse failure than not offering it, because the run stalls instead of adapting. Such a run is not
left mute: it still holds ask_person, which is the honest exit for a Bot that needs a human.

Rendering stays in the browser, through useRenderTool, which draws a call without claiming to
execute it. The renderers themselves are unchanged, because they already parsed a JSON result and
that is what these return. The transcript still names the element the gateway resolved rather than
the ref the model sent.

The activity pane is the part that cost something. It was written from the handlers, and its own
docblock explained why: they "run exactly once per call", where a tool's render runs on every
paint. With the handlers gone, render is the only place the browser still sees a call, so
recordActivity now takes the tool call's id and is idempotent on it. Its listeners are notified in
a microtask rather than synchronously, because notifying from inside one component's render sets
state in another that is subscribed through useSyncExternalStore. The cost is a required argument
at every call site and a seen set that has to be cleared when a computer is wiped; both are
covered by tests, and the second is the one that would have rotted quietly.

What is not covered

  • No recording yet, and this is a draft because of it. The change has no visible surface of its
    own by design — the renderers are deliberately unchanged — so the only honest recording is a Bot
    running a command with no tab attached, which needs a model credential this environment does not
    have. It follows before this leaves draft.
  • The two assistance tools still require a browser, so a headless run can drive a page, a shell
    and a workspace, but cannot ask for a password or hand over the wheel.
  • Nothing yet consumes the new capability. This removes the wall; it does not add the surface
    that walks through it.
  • report_refusal writes its audit row through the capability seam rather than the existing
    POST /api/agents/:id/declined route.
    Both now exist and write the same event type; the route
    is still what the browser-side path would use, and collapsing them is left alone here rather than
    changed in a commit about something else.

Verification

Run against a local Postgres with migrations applied.

  • Full suite on main at fb0c797: 2064 pass, 23 skip, 0 fail, 170 files.
  • Full suite on this branch: 2073 pass, 23 skip, 0 fail, 171 files. The nine new tests are the
    difference; nothing existing changed its result.
  • tsc --noEmit clean for both server and app.
  • biome lint --error-on-warnings . clean across 492 files.

New and changed test files:

  • server/tests/computer-tools.test.ts — holds down that the Bot and the actor reach the gateway
    (the audit row is written from them), that a refusal comes back as an answer carrying the rule
    rather than a throw that ends the run, that the failure modes stay distinguishable so a model can
    tell a retry from a dead end, that bad arguments are refused before the gateway is touched, that
    optional arguments are omitted rather than sent as undefined, and that the two person-present
    tools are not offered.
  • app/tests/computer-activity.test.ts — extended with the property the keyed signature exists for:
    one call is one entry however many times a render records it, and a wiped computer can record the
    same call id again without it being swallowed as a duplicate.

The computer tools were registered in the browser with `useFrontendTool`, and
every handler was a `fetch` back to `/api/computers/:botId/...` — a round trip
into this same process. That made an open tab load-bearing. A Bot whose person
had closed the window had no browser, no workspace and no shell, because the
only thing that could carry out a tool call had gone away, and an unattended
run was out of reach entirely.

This repository has already made that move once, for MCP, and `plugins/tools.ts`
still says why: it "made a browser a hard requirement for a Bot to do anything,
which rules out an embedded widget, a run nobody is watching, and any surface
that is not our own app."

So twelve of the fourteen move to `computer/tools.ts` and execute here, offered
through the same per-run capability seam that already hands a Bot `message_bot`
and `ask_person`. Nothing about governance moves with them: every acting tool
still goes through ComputerGateway, which resolves the ref against the snapshot
this server took, evaluates the policy, writes the audit row, and only then
acts.

Two deliberately stay in the browser. `computer_request_secret` and
`computer_request_help` both end with a person typing into a masked box or
taking the wheel, so both need somebody present by definition; moving them
would produce a tool a headless run can call and can never have answered. Such
a run is not left mute — it still holds `ask_person`.

Rendering stays in the browser too, through `useRenderTool`, which draws a call
without claiming to execute it. The renderers are unchanged: they already
parsed a JSON result, which is what these return.

The activity pane had to change with it. It was written from the handlers,
"which run exactly once per call", and `render` runs on every paint. So
recordActivity now takes the tool call's id and is idempotent on it, and its
listeners are notified in a microtask rather than synchronously inside another
component's render.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant