Bunsen

Bunsen is a general-purpose experiment runner for agentic systems. Give an agent an environment, run it reproducibly, capture artifacts and traces, then evaluate the result. Environment-first, agent-agnostic.

🌐 Live at bunsen.dev — with full docs.

Idea to insight, fast. Run agents in reproducible environments, inspect traces and artifacts deeply, and turn behavior into evidence-backed insight humans and AI can build on.

Familiar shape: if you've used devcontainer features, Bunsen's environment model will feel similar — declarative units that compose into a shared container, multi-source (inline / file / registry), a schema centered on what does this install rather than what's the full environment shape. Bunsen applies the same pattern to a slightly different runtime model (artifacts mounted at agent invocation rather than baked into images per agent). See The Environment Model for the full lineage.

Features

• Reproducible runs — each run executes in its own Docker container with workspace snapshotting for idempotent runs (a reproducibility boundary, not a security sandbox — see Trust & Safety).
• Agent and experiment variants — first-class variants on both, with clear merge semantics. Model is a separate axis — pick it with bn run --model <id> instead of a per-model variant.
• Multiple agent sources — load agents from local dirs, git repos, npm packages, or binaries.
• Suites — consume external benchmark suites at a pinned git ref (e.g. Terminal Bench).
• Automatic capture — logs, artifacts, workspace diffs, AI traces, and cost, captured automatically. Optional terminal recording for visual replay.
• Five criterion types — script, judge, agent, browser-agent, aggregate — plus a dedicated evaluation.report synthesis step.
• Gate pattern — short-circuit expensive LLM scoring when cheap script tests fail.
• CLI-first — a noun-grouped command tree (bn run, bn experiments, bn agents, bn runs, bn suites, bn eval, …) with stable exit codes and machine-readable output.

Quick Start

Prerequisites: a running Docker daemon. An ANTHROPIC_API_KEY is needed for Claude-based evaluation and for Claude-powered agents; the no-AI echo-agent with script-only scoring runs fully offline (agent invocation is composed deterministically — no model in the path). No Node toolchain required — the bn binary embeds its own runtime.

curl -fsSL https://bunsen.dev/install.sh | sh   # standalone binary onto your PATH
bn doctor                  # verify Docker + environment

mkdir my-lab && cd my-lab
bn init --example          # scaffold a project + a hello-world experiment + echo-agent
echo "ANTHROPIC_API_KEY=sk-ant-..." > .env

bn run hello-world echo-agent
bn runs show               # score, cost, status
bn runs open               # open the run in the local web viewer

That's the whole loop. The echo-agent needs no API key, so bn run hello-world echo-agent works offline. To run a real coding agent, copy in a starter:

bn agents add              # add claude-code, codex-cli, gemini-cli (or `bn init --starter-agents`)
bn run hello-world claude-code

Set the matching key in .env first (ANTHROPIC_API_KEY for claude-code, OPENAI_API_KEY for codex-cli, GEMINI_API_KEY for gemini-cli). The copied agents are yours to edit — pin a different CLI version, add a variant, swap the model. Next: Getting Started walks it in detail and forks into two paths — Run a Terminal Bench Task or Bring Your Own Task.

Distribution. bn ships as a single self-contained binary — one download onto your PATH, no Node and no version skew (the binary embeds its own runtime). Install with curl -fsSL https://bunsen.dev/install.sh | sh, or via a Homebrew tap (brew install bunsen-dev/tap/bunsen) / Scoop on Windows. The one unavoidable host prerequisite is a Docker daemon — experiments run in reproducible containers, and the first run pulls a couple of images (the mitmproxy trace-capture sidecar + your experiment's base image); bn doctor reports both honestly. @bunsen-dev/sdk + @bunsen-dev/types (the programmatic surface) stay on npm. Experiments using a custom Dockerfile or non-bunsen base image work too: Bunsen fetches the small container Node runtime it mounts into those containers on first use, verifies it against a pinned sha256, and caches it per-user (glibc bases — debian/ubuntu/CUDA/etc.; musl/Alpine bases aren't yet supported). Set BUNSEN_NODE_OFFLINE=1 to forbid that fetch (CI / air-gapped), or pre-seed it with BUNSEN_NODE_RUNTIME_DIR. The binary is ~55–110 MB per platform — that's the embedded-runtime floor, not app size.

From a source checkout (for contributing)

pnpm install
pnpm build
# then invoke the CLI through pnpm by prefixing each command: `pnpm bn …`

A checkout ships example agents and experiments under examples/, so you can run bn run fix-the-bug claude-code or bn run fizzbuzz basic-coding-agent directly.

Documentation

Full documentation lives at bunsen.dev/docs (sources under docs/):

Start here

• Introduction · Getting Started · Run a Terminal Bench Task · Bring Your Own Task

Concepts

• How Bunsen Works · The Environment Model · Trust Model

Authoring

• experiment.yaml Reference · agent.yaml Reference · Agent Dependencies Cookbook · System Prompts · Running as Root · Supervised Mode · Agent Skills

Evaluation

• Scorers & Evaluation · Scoring in the Agent Container · Scoring Service Tasks

Suites & Reference

• Suites · CLI Reference · Project Configuration · Run Manifest & Events · Exporting a Run Workspace · Cost Accounting · Platforms & Architecture · Packages & Schemas · Changelog · Glossary

Examples

A source checkout includes sample agents and experiments under examples/:

Agent	Description	Starter?
echo-agent	Minimal test agent (no LLM) — echoes the task, for smoke tests.	--example
basic-coding-agent	Full coding agent with file/bash tools (ships its own Python).
claude-code	Claude Code — Anthropic's agentic CLI; headless by default (:headed for supervised), non-root.	✓
claude-sdk-agent	Claude Agent SDK wrapper; programmatic tool allowlist, runs as root.
codex-cli	OpenAI Codex CLI.	✓
gemini-cli	Google Gemini CLI (Gemini Direct API).	✓

The three frontier coding CLIs marked Starter are also bundled inside the published @bunsen-dev/cli, so an npm-only install can drop them into a project with bn agents add (or bn init --starter-agents) — no checkout required. The others are checkout-only references to learn from.

Any agent can be steered with a system prompt without changing the platform — see System Prompts. To scaffold your own from a blank template, use bn agents new <name> (the bunsen-new-agent skill from bn skills install walks you through the agent.yaml); to start from a working one, bn agents add then edit (see the CLI Reference).

Example experiments live under examples/experiments/, grouped by category:

Category	What the agent does
zero-to-one/	Build a small app or program from a prompt.
fix-bugs/	Fix a real bug in a verbatim snapshot of an upstream repo.
games/	Author a game-playing bot scored from authoritative game state, with an auto-rendered replay. games/battlesnake (write a Battlesnake bot, scored by win-rate against a hidden held-out ladder, emits a replay GIF) is the first.

Base images

Image	Description
bunsen/headless	Default — Python + Node for CLI agents.
bunsen/visual	Headless + Playwright/Chromium for visual scoring.
bunsen/desktop	Full desktop environment for GUI agents.

Most experiments use a Bunsen image (the default); you can also point at any Docker image or a Dockerfile in the experiment directory. See The Environment Model for image selection, runtimes, packages, and custom Dockerfiles.

Development

pnpm build      # build all packages
pnpm test       # run tests
pnpm typecheck  # type check
pnpm lint       # lint

Testing the compiled binary locally

pnpm bn <cmd> runs the bundled CLI under Bun (the everyday dev loop — it has all assets, so bn run works). To exercise the actual standalone binary built from your working tree:

pnpm -r build                                            # incl. dist/assets + agent bundles
pnpm --filter @bunsen-dev/cli build:binary darwin-arm64  # or darwin-x64 / linux-x64 / linux-arm64 / all
./packages/cli/dist/binaries/bn-darwin-arm64 doctor

The binary extracts its embedded assets to ~/.cache/bunsen/assets/<version>/, keyed by version — so across rebuilds at the same version the cache won't refresh. When iterating, point BUNSEN_ASSET_DIR at the freshly-built assets (it bypasses extraction entirely), or clear the cache:

BUNSEN_ASSET_DIR="$PWD/packages/cli/dist/assets" ./packages/cli/dist/binaries/bn-darwin-arm64 run …
# or after each rebuild:  rm -rf ~/.cache/bunsen/assets

Trust & Safety

Running an experiment, agent, or suite means running its author's code on your machine. That's the nature of the tool — Bunsen exists to execute agents against environments and capture what happens. Runs execute in a Docker container, but the container is a reproducibility and accident boundary, not a security sandbox: it uses Docker's default capabilities and seccomp, sets no resource limits, has open network egress, and has your real provider API keys inside it. There's no known container-escape vector and the agent runs non-root by default — but a deliberately hostile agent can still exfiltrate keys and data over the network. Saved run directories and AI traces can also contain secrets and full prompt/workspace content, and there is no automatic redaction yet, so scrub before sharing.

Treat it like git clone && make on an unfamiliar repo: run what you trust, or run untrusted code on a disposable host with throwaway, scoped API keys. See Trust Model for the full trust model — where code executes (host vs container), the exact container posture, how to share runs safely, and the trace body-vs-headers nuance.

You are responsible for what your runs do and what they cost. Bunsen does not cap spend — a run is bounded by run.timeout, not by a dollar or token budget, so a looping or fast agent can run up a large provider bill. You are solely responsible for all charges you incur with model and tool providers, and for ensuring your use — and anything an agent does over the open network — complies with those providers' terms and with applicable law. Use scoped, low-limit keys and provider-side budgets to bound spend. See Cost Accounting and the Additional Terms in LICENSE.

License

Bunsen is source-available under the PolyForm Shield License 1.0.0 — it is not an OSI "open source" license. In plain English:

• Yes, you can use it at work. Internal commercial use is allowed.
• Yes, you can run it anywhere — your laptop, your servers, your cloud — for your own use.
• Yes, you can read, modify, fork, and redistribute the source.
• Yes, you can build, test, and evaluate your own products with it.
• No, you cannot offer a competing Bunsen-like product or service without a separate commercial license. That applies whether the competing product is hosted, local, embedded, forked, or free.

Bunsen has been source-available since day one; there was no prior OSI open-source release.

See LICENSING.md for the full terms, the licenses of bundled third-party components, and the rationale. Want to do something the license doesn't allow (e.g. offer a competing or hosted commercial service)? A commercial license is available — contact licensing@bunsen.dev.

Contact

General questions, feedback, and collaboration: hello@bunsen.dev.