RFC — Phase 6: Remote Agent Backend (over SSH / WebSocket)

[GoDiao]

RFC — Phase 6: Remote Agent Backend (over SSH / WebSocket)

Status: Draft — open for discussion
Champion: @GoDiao
Related: Original feature request #3 — Remote Development Support (mirrored from openai/codex#10450)
Sibling RFC: #19 — Phase 4 IM adapters

---

Why this RFC

DreamCoder v0.4.0 shipped H5 Remote Access (Phase 3): you sit in front of a desktop running DreamCoder and reach into it from your phone. That solves "I'm away from my desk."

It does not solve a different problem that #3 (and many real-world workflows) raise: "my code lives on a remote machine and I don't want to clone it locally." Typical setups:

• VS Code Remote-SSH style daily workflow
• GPU / specialized hardware boxes (training, inference, simulation)
• Cloud / production-like dev VMs
• Security/compliance contexts where code may not leave the server

What's needed is the opposite topology of H5:

	H5 Remote Access (v0.4.0)	Remote Agent Backend (this RFC)
Agent runtime	local desktop	remote host
Codebase location	local	remote
Remote endpoint	phone / browser (thin)	server / GPU box (heavy)
Problem solved	"I'm away from my desk"	"Code isn't on my desk"

The two are complementary — if we land both, DreamCoder works wherever you are and wherever your code is.

---

Goals

• Run the DreamCoder agent (Claude Code CLI + tools + PTY + MCP) on a remote machine, against a remote codebase.
• Use the local DreamCoder desktop GUI as the client UI — chat panel, sessions, settings, terminal pass-through.
• Authenticate cleanly. Idle remote sessions should not become an attack surface.
• Be opt-in and reversible — local-only mode stays the default, the remote backend is an additional connection mode.
• Reuse as much of the existing sidecar / H5 protocol as possible. Avoid forking the codebase.

Non-goals (for v1)

• Full IDE features (LSP indexing, language servers, refactor across files via remote). Out of scope — Claude Code already calls remote tools fine via PTY.
• Multi-user collaborative editing on the same remote session.
• Reinventing SSH / VPN. We assume the user already has SSH access to the host.
• File sync / mirroring like VS Code Remote-SSH's vfs. The agent operates server-side; the client renders chat + file diffs.

---

Candidate paths

Three architectural shapes are on the table. They are not mutually exclusive — Path C is arguably a superset.

Path A — SSH reverse tunnel to a remote sidecar

[ Local DreamCoder GUI ] ──(localhost http/ws)──► [ ssh -R tunnel ] ──► [ dreamcoder-sidecar on remote ]
                                                                              │
                                                                              ▼
                                                                       remote codebase + claude CLI

How: User configures a remote host. DreamCoder spawns ssh -R 0:127.0.0.1:<port> user@host 'dreamcoder-sidecar --headless --bind 127.0.0.1:<port>'. The local GUI talks to localhost as if it were the local sidecar.

• ✅ Auth is "free" — SSH does it.
• ✅ No new ports exposed on the public internet.
• ✅ Works through corporate firewalls if SSH already does.
• ❌ Need to bootstrap dreamcoder-sidecar onto the remote (one-time install or auto-upload binary).
• ❌ Tunnel lifetime tied to a process — reconnect logic needed.
• ❌ Windows SSH server story is messier than Linux.

Path B — Remote-installed sidecar, local GUI connects via WebSocket

[ remote host ]  dreamcoder-sidecar --serve --auth=token  ◄──(wss + token)── [ Local DreamCoder GUI ]

How: User installs dreamcoder-sidecar on the remote (binary or cargo install or bun install -g), starts it as a service, and pairs the local GUI to it (host + token, like the H5 pairing flow but in reverse direction).

• ✅ Survives across multiple local clients (laptop + work desktop pairing to the same remote box).
• ✅ Reuses H5's token rotation, QR pairing UX, request classifier.
• ❌ Now there's a port to expose on the remote — needs TLS, ideally behind reverse-proxy or only on a private network.
• ❌ Operational story: who keeps the sidecar daemon alive? (systemd template? launchd? pm2?)

Path C — Reuse H5 protocol, run headless DreamCoder on remote, local GUI as the "phone"

[ remote host ]  dreamcoder --headless (full server, H5 endpoint)  ◄──(wss, H5 protocol)── [ local GUI in "client" mode ]

How: We already have connectHeadless.ts and createDirectConnectSession.ts from v0.4.0. The remote box runs the full DreamCoder server in headless mode (no Tauri shell, just the bun server). The local desktop GUI gets a "Connect to remote DreamCoder" mode and talks the same H5 protocol the phone speaks today.

• ✅ Maximum code reuse — protocol, auth, token rotation, request classifier are already live.
• ✅ One mental model for both directions: the desktop is just a richer H5 client.
• ✅ Symmetrical — same binary works as host or client.
• ❌ Headless-mode server needs hardening: process lifecycle, log rotation, multi-session isolation, resource limits.
• ❌ The desktop GUI's existing assumption that the sidecar is 127.0.0.1 needs auditing — anywhere we trust loopback today must be re-checked under remote topology.

Tentative recommendation

Default: Path C with Path A as a 5-minute fallback.

• C maximizes leverage from the v0.4.0 H5 stack and gives users a single coherent story.
• A is trivially achievable for power users today as a documented recipe ("run DreamCoder over SSH tunnel") even before C ships — could be the v0.5 stop-gap while C bakes.
• B is what C looks like in production when paired with auto-start service templates; it's a deployment shape, not a separate architecture.

---

Open questions

1. Auth model: Token + TLS cert pinning? mTLS? Reuse H5's rotating token? OAuth-device-code? What's the minimum bar that won't get someone fired by their security team?
2. Discovery: How does the local GUI find the remote? Manual host:port + token? mDNS on private networks? A small "DreamCoder Hub" relay (opt-in)?
3. File operations: Do we surface remote file diffs in the GUI (currently we read locally)? Stream them over the wire? Cache?
4. Terminal: portable-pty lives on the remote in this design — does the existing xterm.js client need protocol changes for remote PTY resize / signal forwarding?
5. MCP servers: Remote-side or local-side? Probably remote (they need filesystem access), but some MCP servers are local-only by design (e.g. Apple Notes). Do we support both ends simultaneously?
6. Updater: How does dreamcoder-sidecar (or headless DreamCoder) get updated on the remote? Pinned version + manual update? Auto-update from a configured endpoint?
7. Computer Use: On a headless Linux box without a display server, screenshot mode is moot. UIA tree mode is Windows-only. Does Computer Use just disable itself in remote mode, or do we ship something like xvfb-based headless screenshotting?
8. Resource limits: One remote box, multiple DreamCoder sessions, multiple users — what's the isolation story? Per-session subprocess? cgroup? Just trust the OS user account?

---

Proposed work breakdown (subject to scope)

1. Spike: prove Path C in a branch — local desktop GUI connects to a headless DreamCoder over WebSocket on a LAN box. (1-2 days)
2. Auth hardening: extend H5 token model to remote topology; remove any lingering "trust loopback" assumptions. (~1 week)
3. Headless lifecycle: systemd unit + launchd plist + Windows service templates; log rotation; PID file. (~3 days)
4. GUI side: "Add remote connection" flow — host + paste token, save profile, switch profile. Mirror the existing H5 setup UX. (~1 week)
5. Path A docs: write the SSH-tunnel recipe for users who want it today. (~half a day)
6. Computer Use mode: gate / disable on headless Linux for v1. (~1 day)
7. MCP: decide local-vs-remote per server in settings; default = remote. (~3 days)
8. Docs + sample setup: "Install DreamCoder on a GPU server" walkthrough. (~2 days)

---

How to participate

• 💬 Comment on this issue with answers to any of the open questions, or push back on the candidate paths.
• 🔬 Try the SSH-tunnel recipe once we publish it (Path A) and report back what breaks — that data informs the auth/lifecycle hardening for Path C.
• 🛠 Pick up a sub-task from the work breakdown if you want to ship something. Mentor available — @GoDiao.

cc @v2less (original requester in #3) — you're the source for this RFC. Would love your take on the candidate paths and which workflow shape best matches your day-to-day.

cc @pocca2048 — credit to your codex#10450 issue, which #3 mirrored. Cross-pollination welcome if you're around.

Tagging interested area: rfc, enhancement, area:remote, phase-6.