Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,40 @@ tail -f .vm/mitmdump.log # all proxy traffic
cat .vm/blocked.jsonl | jq . # blocked requests
```

## Host tool bridges

Some tools can't run inside the sandbox because they only exist on the host — Xcode being the obvious one. `vm.py start --bridge PORT=COMMAND` exposes a host-side stdio program to the guest at `10.0.2.101:PORT`. Every TCP connection the guest opens to that address spawns a fresh copy of `COMMAND` on the host with its stdin/stdout wired to the socket, which is exactly the shape of an [MCP](https://modelcontextprotocol.io/) stdio server. The flag is repeatable, and bridges last only for the lifetime of that `vm.py start`.

> **A bridge is a hole in the sandbox boundary.** Bridge traffic does **not** pass through mitmproxy and is **not** checked against `allowlist.txt`. It is a raw pipe from the guest to a process running on your host, outside the VM, with your own user's privileges — and a process the agent can restart at will, once per connection. Only bridge tools you are willing to let the agent drive unsupervised, and leave the flag off the rest of the time.

### Example: driving Xcode from inside the VM

Xcode exposes its tools to external agents over an MCP stdio server, which `xcrun mcpbridge` launches — so the agent in the guest can drive the Xcode running on your host. Three steps, and the first is the one that gets missed:

**1. On the macOS host, in Xcode:** Settings → Intelligence → enable **"Allow external agents to use Xcode tools"**. Xcode also has to be running with your project open. Without that toggle the bridge connects fine and hands back an empty tool list, which looks like a networking failure and isn't one.

**2. Start the VM with the bridge:**

```bash
./vm.py start --bridge '8110=xcrun mcpbridge'
```

The port is arbitrary — pick anything free. `10.0.2.101` is internal to the QEMU process; nothing is opened on your host's real network interfaces.

**3. Inside the guest, from your project folder:**

```bash
claude mcp add --transport stdio xcode -- nc 10.0.2.101 8110
```

`vm.py start` prints a ready-to-paste version of this line (named `bridge-8110`) when it brings the bridge up, so you don't have to remember the address. `nc` is preinstalled in the guest. Registration lives in the guest's Claude config, so re-run it after `vm.py reset` — that config is on the ephemeral overlay disk.

Notes and limits:

- **`COMMAND` must not contain commas.** QEMU splits its netdev argument on commas; spaces are fine.
- **The host command inherits your host environment**, including anything on your `PATH` and any credentials in it.
- **Any MCP stdio server works.** The bridge doesn't care what the command is, so Xcode is just the obvious example — substitute any host-side tool that speaks MCP over stdio.

## Credentials and the shared directory

The `shared/` directory is mounted read-write inside the guest. Be deliberate about what you place there.
Expand Down
68 changes: 68 additions & 0 deletions TODO.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,3 +4,71 @@
2. ~~Evaluate alternatives to using the `com.apple/agent-vm` pf anchor on macOS.~~ Evaluated — keeping `com.apple/agent-vm`. Alternatives (custom anchor in `/etc/pf.conf`, `pfctl -f`, LaunchDaemon) are all more invasive or dangerous. Added rationale comment and runtime verification that warns if rules aren't active.
4. ~~Simple is harder than complex. Review all code and propose 3 ways it can be simplified based on what you know now.~~ Done — (a) replaced MIME multipart user-data merge with cloud-init's native `#cloud-config-archive` format, (b) unified EFI preparation into a shared `_prepare_efi()` helper, (c) replaced 3-way ISO tool cascade with platform-explicit `_build_iso()`.
5. ~~Explore going sudoless by using QEMU's `-netdev user` (slirp) with `guestfwd` to connect the guest to the proxy.~~ Done — replaced socket_vmnet (macOS) and TAP/bridge (Linux) with slirp `restrict=on` + `guestfwd=tcp:10.0.2.100:PORT-cmd:nc 127.0.0.1 PORT`. Removed all pf/iptables firewall code. Zero sudo prompts. The `cmd:nc` guestfwd mode (revisited with QEMU 10.x) works correctly — each guest TCP connection spawns a fresh `nc` that connects to the host-side proxy. `restrict=on` blocks all other outbound traffic (TCP, UDP, ICMP) at the slirp layer, providing stronger isolation than the previous firewall-based approach.
6. Make the `--bridge` MCP wiring self-serve. Today `vm.py start --bridge '8110=xcrun mcpbridge'` opens the socket, but the user still has to run `claude mcp add --transport stdio xcode -- nc 10.0.2.101 8110` inside the guest by hand. Options, roughly in order of increasing magic: (a) document it in README.md next to the `--bridge` flag — see the required contents below; (b) have `--bridge` print the exact `claude mcp add` line to paste after SSH, e.g.

Host-tool bridges (bypass the allowlist — raw pipe to host):
10.0.2.101:8110 -> xcrun mcpbridge

To use from within the guest, run this command from within your
project folder:

claude mcp add --transport stdio bridge-8110 -- nc 10.0.2.101 8110

(a) is required regardless of whether we do (b) or (c) — the README needs an end-to-end Xcode walkthrough covering all three steps, because the host-side one is not discoverable:

1. **On the macOS host, in Xcode:** Settings → Intelligence → enable **"Allow external agents to use Xcode tools"**. Without this the bridge connects but the tools aren't exposed, which looks like a networking failure and isn't one.
2. **Start the VM with the bridge:**

./vm.py start --bridge '8110=xcrun mcpbridge'

3. **Inside the guest, from your project folder:**

claude mcp add --transport stdio xcode -- nc 10.0.2.101 8110

Also state plainly in that section what the flag's `--help` already warns: this channel bypasses the mitmproxy allowlist and is a raw pipe to a host process. Someone reading the README to wire up Xcode should not have to read `--help` to learn they've opened a hole in the sandbox boundary. Verify the `xcrun mcpbridge` invocation against whatever Xcode actually ships before publishing it — it's currently just the example string in the `--bridge` help text.

(c) add a `--xcode` convenience flag that implies the bridge and writes the MCP server entry into the guest's Claude config via cloud-init or a post-boot SSH command. Note (c) means `vm.py` starts owning guest-side agent config, and the registration has to be idempotent across reboots and survive `vm.py reset`; decide whether that belongs in this tool at all.
7. Fix mode bits being lost on the shared directory, and revisit the 9p driver while we're in there. Symptom: editing a file from inside the guest drops its exec bit on the host — `vm.py` went from `100755` to `100644` in the working tree with no `chmod` anywhere. Suspect the 9p transport and/or the `bindfs --force-user=vm --force-group=vm` layer not round-tripping permissions; worth confirming which of the two is responsible before changing anything. The likely fix is switching the share from 9p to virtiofs, which we want regardless for the performance win — but check that it doesn't reintroduce the UID-mapping problem bindfs exists to solve, and that it doesn't pull us back to the "generic" vs "genericcloud" image constraint for the wrong reason (see CLAUDE.md). Until it's fixed, check `git diff` for spurious mode changes before committing.
8. Update the default guest Claude Code settings (`cloud-init/user-data`, the `/opt/provision/claude-settings.json` block around line 80) to match what we actually run with:

{
"permissions": {
"defaultMode": "bypassPermissions"
},
"model": "opus[1m]",
"effortLevel": "high",
"skipDangerousModePermissionPrompt": true,
"theme": "dark",
"tui": "default"
}

Changes from the current block: `model` moves from the pinned `claude-opus-4-6` to the `opus[1m]` alias (large context window, and it tracks the current Opus instead of going stale), plus new `effortLevel`, `skipDangerousModePermissionPrompt`, and `theme` keys. `skipDangerousModePermissionPrompt` suppresses the startup confirmation that `bypassPermissions` otherwise triggers — appropriate here precisely *because* the VM is the sandbox, but it does mean a fresh guest goes straight to autonomous execution with no interstitial. Worth verifying each key is spelled the way the installed Claude Code version expects; unknown keys are ignored silently, so a typo shows up as "the setting just didn't apply."
9. Also add `"tui": "default"` to the guest settings (part of the same block as item 8). This is the `/tui` setting — `"fullscreen"` is the alternate renderer, `"default"` is the normal one. The fullscreen renderer's OSC-based copy/paste doesn't round-trip through the SSH session `vm.py start` drops you into, so the guest should ship with the normal renderer rather than leaving each user to discover `/tui` themselves. Key name confirmed against a live `~/.claude/settings.json` written by `/tui`, not guessed.
10. **Bug:** changes to `cloud-init/` don't rebuild `seed.iso`. `build_seed_iso()` (`vm.py:379`) returns immediately if `.vm/seed.iso` exists, so after editing `cloud-init/user-data`, `meta-data`, or `network-config` the next `vm.py start` boots the *old* config with no warning — you have to know to run `vm.py reset` first. This is a genuine footgun: the edit looks applied, the VM boots fine, and the change silently isn't there. Detect staleness by hashing the fully assembled user-data (not just the `cloud-init/` file mtimes — that misses injected values like the SSH pubkey, git identity, and CA cert) and storing the digest next to the ISO.

**Do not auto-reset.** Blowing away the user's disk because a config file changed is far more disruptive than the bug it fixes — there's real work in that VM. The fix is *detect and tell*, not *detect and act*: on a digest mismatch, rebuild the ISO and print a clear warning that the new cloud-init config won't take effect until `vm.py reset`, then boot normally. Leave the decision to the user.

Related wrinkle worth confirming while implementing: cloud-init keys off the `instance-id` in `meta-data`, so even a correctly-rebuilt seed.iso attached to an existing disk generally won't re-run provisioning. That makes the warning the actual deliverable here — the rebuild alone doesn't fix anything on a VM that's already been provisioned.
11. Expose CPU count and disk size as `vm.py start` options. `-smp` is hardcoded in `build_qemu_args()` (`vm.py:440`) and the qcow2 overlay is created at a hardcoded `20G` in `ensure_disk()` (`vm.py:374`). Add `--cpus` and `--disk-size` alongside the existing `--memory`. Note the disk is only sized at creation time, so `--disk-size` has no effect on an existing `.vm/disk.qcow2` — either say so in the help text or detect the mismatch and tell the user to `vm.py reset`.
12. Persist `~/.claude` across `vm.py reset` by backing it with a host-mapped directory. Today it lives on the ephemeral overlay disk, so a reset takes the whole agent history with it: `projects/`, `sessions/`, `history.jsonl`, `todos`/`tasks`, `file-history`, and any per-project config. That's the main thing that makes reset feel expensive.

Placement: it has to live outside `.vm/`, which is exactly what `cmd_reset()` deletes — mirror how `.images/` survives. shared/.claude probably works fine if Claude itself can be convinced to look there, or otherwise a dedicated gitignored .claude in this same repo.

Things to work out:
- **Conflicts with items 8/9.** `runcmd` currently does an unconditional `cp /opt/provision/claude-settings.json /home/vm/.claude/settings.json`, instead, copy that file only if a copy doesn't already exist, so it gets bootstrapped but not clobbered.
- **File locking over 9p.** `~/.claude` has `daemon.lock`, append-heavy `history.jsonl`, and a live `daemon/` — advisory locking and rename semantics are where 9p is weakest. This is a strong argument for doing item 7 (virtiofs) *first* and building on that rather than on 9p+bindfs.
- **`.credentials.json` leaves the sandbox.** This is fine since these credentials come from the host box in the first place.
- Consider persisting a subset (`projects/`, `sessions/`, `history.jsonl`, `settings.json`) via symlinks rather than mapping the whole directory — it sidesteps both the daemon-state and credentials questions.
13. Extend the guest's global `CLAUDE.md` (the `/opt/provision/CLAUDE.md` block in `cloud-init/user-data`, currently network-proxy guidance only) with a section on installing things. Two points:

- **Passwordless sudo is available.** The `vm` user has `sudo: ALL=(ALL) NOPASSWD:ALL`, but nothing tells the agent that, so it may hedge around installs or ask the user to run commands it could run itself.
- **`apt` is the right default.** Prefer `apt install` over language-specific version managers, source builds, or vendored toolchains. The exception is a project that clearly pins versions — a `.tool-versions`, `.nvmrc`, `.python-version`, `rust-toolchain.toml`, or equivalent standard lockfile — in which case honor the pin. The rule should be evidence-based: a committed version file is the signal, not a guess about what the project "probably" wants.

- **Python is a hard carve-out: never use Debian-provided Python packages.** No `apt install python3-<anything>` — not the interpreter, not libraries, not `python3-pip`/`python3-venv`. Python interpreters and packages come from **uv**, which manages interpreter versions too, and uv itself is installed only from its official binary release (never pip/conda/pipx). Debian's Python packaging is system-owned, externally-managed, and version-frozen to the release; mixing it with uv-managed environments is how you get a broken interpreter that's hard to unpick. State this as an absolute, not a preference — the apt-first default must not read as licensing `apt install python3-requests`.

Framing: "apt is the default for system packages, **except** Python." Both halves need to be in the same section, or an agent that reads only one will do the wrong thing in exactly the case that matters.

Since all standard Debian apt repos are allowlisted, `apt install` works without any proxy friction — no need to caveat it. Do mention that a 418 on a *third-party* apt repo means that repo isn't allowlisted rather than the package not existing, since that's the one case where an install fails for a reason the agent can't infer from apt's error.
14. Study `grant.py` and make it a first-class capability that is available by default inside the VM, probably as a skill. (Obviously, a key step of the skill is still asking the user to review the proposed policy file and run a command on their host machine to grant it. The command the user runs needs to be committed outside the shared dir so it can be reviewed and can't be modified by a malicious agent between the user reviewing it and running it in a TOCTOU vuln.) Cover how to use this safely in the README (the agent realizes it needs AWS permissions, proposes IAM policy, notifies the user with the PushNotification tool that they need to review and grant, and then the agent waits for the outputted credentials to appear.)
15. Add some kind of Notification support, even if it's just terminal bell, that works through SSH.
16. Add support for a netrc-style credentials file as a sibling to allowlist.txt that the proxy will automatically add to Authorization headers so the VM/agent never see the actual credentials being used. Include a sample file.
3 changes: 3 additions & 0 deletions cloud-init/user-data
Original file line number Diff line number Diff line change
Expand Up @@ -182,6 +182,9 @@ packages:
- bindfs
- git
- docker.io
# nc: used to bridge a stdio MCP server to a host-side tool exposed via
# `vm.py start --bridge` (e.g. Xcode's `xcrun mcpbridge` on a macOS host).
- netcat-openbsd

runcmd:
- mkdir -p /mnt/9p /home/vm/shared
Expand Down
Loading
Loading