diff --git a/docs/benchmark.md b/docs/benchmark.md index 9666749..de0b517 100644 --- a/docs/benchmark.md +++ b/docs/benchmark.md @@ -174,6 +174,7 @@ python3 -m evaluation.swe_bench_pro \ --config-yaml "$RUN_ROOT/config.yaml" \ --preflight-output "$RUN_ROOT/preflight.json" \ --on-demand-image-status "$RUN_ROOT/images.json" \ + --native-trace-dir "$RUN_ROOT/traces" \ --report-prefix "$RUN_ID" \ --agent-model-name gpt-5.4 \ --sample-offset 0 \ @@ -190,6 +191,21 @@ python3 -m evaluation.swe_bench_pro \ --persistent-cache-mode rw ``` +The native runner exports each task container's multiagent state before the +container closes. Traces are stored under +`$RUN_ROOT/traces/official-row-NNNNNN/` as a hash-verified +`multiagent-trace.tar.gz` plus `manifest.json`. The archive contains the +orchestrator log, subagent transcripts, structured workflow/checkpoint state, +runtime identity, and native runner stdout/stderr. It is written on successful, +failed, and timed-out solver exits, remains outside `/app`, and is never part of +the submitted patch or official scoring. Treat the raw archives as private +artifacts because agent transcripts can contain source and environment details. + +Parallel shard runs accept the same `--native-trace-dir`; all workers write +unique directories keyed by the absolute official row number. If the option is +omitted, the single-run command uses `evaluation/reports/swe-bench-pro-traces`, +while the parallel launcher uses `REPORT_DIR/traces`. + After the run completes, capture a relocatable evidence bundle. The command fails if any source checkout is dirty, any row lacks official verifier/native outcome evidence, image identity is incomplete, runtime Codex/Node identity is diff --git a/docs/control-plane-boundary.md b/docs/control-plane-boundary.md index 0c96588..0af365e 100644 --- a/docs/control-plane-boundary.md +++ b/docs/control-plane-boundary.md @@ -11,6 +11,8 @@ Rust owns production decisions and durable state: - write-policy checks and approvals; - assignments, checkpoints, and Git worktree metadata; - findings, repair TODOs, resolution and closure evidence; +- durable reviewer findings, which cannot be replaced by a later pass on the + same candidate without first entering the repair loop; - validation leases and bounded validation subprocesses; - launch configuration, tmux subprocess orchestration, status, watch, and recovery behavior. @@ -21,6 +23,23 @@ allocate or emulate a PTY; tmux continues to own terminal lifecycle and interactive process semantics. This keeps PTY behavior without preserving shell implementations. +In the production Linux-container boundary, tmux runs as the read-only +orchestrator UID. A raw tmux window therefore cannot acquire repository writes. +Worker/reviewer transitions use the Rust binary's narrowly gated +`role-agent-exec` entrypoint: it accepts only a persisted named Codex agent, +validates the trusted bridge, and starts Codex in a dedicated process group +under the role's UID. A minimal wait-only parent retains no workflow discretion; +it exists solely to forward pane termination to the complete role process tree. +`subagent kill` waits for that boundary to close, preventing detached or late +worker output from modifying the workspace after cancellation. The setuid +privilege gate drops privilege for every other command, including generic +`role-exec`, so bypassing the high-level CLI cannot create an arbitrary writer +shell. Lifecycle enforcement is also derived from the orchestrator's real UID, +not solely from its mutable environment. Before the privileged bridge starts a +writer it revalidates the assignment against the live workflow phase and +approved implementation context; setting +`MULTIAGENT_LIFECYCLE_ENFORCEMENT=0` cannot reopen a completed workflow. + Python under `evaluation/` is limited to benchmark adapters, status readers, and provenance. SWE Bench adapters launch the production workflow and pass the current workspace diff to the official scorer. They neither derive a second diff --git a/docs/getting-started.md b/docs/getting-started.md index bcb6b86..4213518 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -72,15 +72,28 @@ workers and generic named subagents too: ORCHESTRATOR_CLI=codex WORKER_CLI=codex SUBAGENT_CLI=codex ./launch.sh ``` -Codex launches with `--cd`, `--dangerously-bypass-approvals-and-sandbox`, and -`--no-alt-screen`. Claude launches from the target worktree/root with -`claude --dangerously-skip-permissions`; Codex-only flags are intentionally not -passed to Claude. - -`--root` selects the target project repo for `MULTIAGENT_ROOT`, state, write -policy, and the orchestrator CLI working directory. The default orchestrator -prompt is still loaded from this launcher's directory, so cross-repo launches do -not need an `orchestrator_prompt.md` in the target repo. Set +The Rust supervisor assigns Codex access from trusted process roles. On hosts +where Codex's native sandbox is available, the orchestrator starts in the +durable state directory with `workspace-write`, workers start in the target +repository with `workspace-write`, and scouts/authority reviewers use +`read-only`. The production Linux-container adapter uses separate unprivileged +Unix identities instead because nested bubblewrap is unavailable under Docker's +default seccomp profile. Its tmux server runs as the non-writing orchestrator +identity. A narrowly gated, setuid Rust entrypoint may only start the fixed +Codex subagent command recorded for a named role; all other invocations +permanently drop back to the caller UID. Each role also receives a private +Codex runtime home so one role's private lock/config files cannot stall another. +The isolated orchestrator's real UID makes lifecycle enforcement mandatory, so +shell-level environment overrides cannot authorize a writer after completion. +In both environments the orchestrator can read the target but cannot write it, +while workers can. Claude remains a compatibility path and does not provide +Codex's native role boundary outside the production adapter. + +`--root` selects the target project repo for `MULTIAGENT_ROOT`, state, and write +policy. The orchestrator CLI works from the durable state directory and reads +the target repository without write access. The default orchestrator prompt is +still loaded from this launcher's directory, so cross-repo launches do not need +an `orchestrator_prompt.md` in the target repo. Set `MULTIAGENT_PROMPT=/path/to/prompt.md` to override that default. ## System Flow @@ -451,11 +464,19 @@ orchestrator/user decision: multiagent policy approve /tmp --actor orchestrator --assignment-id build-logs --reason "user approved shared temp output" --force ``` -Mechanical enforcement is limited to the helper's policy checks and startup -visibility. Codex is still launched with -`--dangerously-bypass-approvals-and-sandbox`, so shell sandboxing is not -enforcing the boundary. The orchestrator and worker instructions require agents -to check and follow the policy before writes. +For Codex roles, the OS boundary mechanically prevents the orchestrator, +authority reviewers, and scouts from writing the target repository. On native +hosts that boundary is Codex's sandbox; in the production Linux container it is +Unix ownership plus a permanent role UID drop. The tmux server itself has the +orchestrator UID, so bypassing the Rust CLI to open a raw pane still produces a +non-writing process. The only privileged transition is the fixed +`role-agent-exec` path, which validates persisted role metadata and a +root-owned, non-group-writable Codex bridge before dropping to the writer or +reader UID. Generic `role-exec` calls from the orchestrator lose setuid +privilege before dispatch. The write-policy helper remains responsible for +explicit writes outside the normal role root. Claude +compatibility processes do not receive this mechanical boundary on native +hosts. ## Assignment Metadata and Acceptance @@ -465,9 +486,11 @@ work starts: ```bash multiagent subagent assignment-create worker-01-docs \ --assignment-id docs-001 \ - --branch worker/docs-001 \ + --branch "$(git rev-parse --abbrev-ref HEAD)" \ --owned README.md,orchestrator_prompt.md -multiagent subagent worktree-create worker-01-docs +SUBAGENT_CLI="$WORKER_CLI" multiagent subagent spawn worker-01-docs \ + --role worker --instruction-file /path/to/worker-instruction.md +multiagent subagent wait worker-01-docs --timeout 1800 multiagent subagent assignment-show worker-01-docs multiagent subagent assignment-status worker-01-docs running multiagent subagent checkpoint-update worker-01-docs --step "started implementation" --status running @@ -531,7 +554,7 @@ Use `multiagent subagent` for named subagents that should keep working or monito ```bash multiagent subagent spawn subagent-ci-monitor --instruction "Monitor CI and report status changes." SUBAGENT_CLI=claude multiagent subagent spawn subagent-ci-monitor --instruction "Monitor CI and report status changes." -multiagent subagent poll subagent-ci-monitor +multiagent subagent wait subagent-ci-monitor --timeout 900 multiagent subagent inspect subagent-ci-monitor --lines 160 multiagent subagent recover-plan multiagent subagent restore subagent-ci-monitor diff --git a/evaluation/evalscope_multiagent_native_runner.py b/evaluation/evalscope_multiagent_native_runner.py index 09c891a..1d0d6bf 100644 --- a/evaluation/evalscope_multiagent_native_runner.py +++ b/evaluation/evalscope_multiagent_native_runner.py @@ -13,9 +13,12 @@ from __future__ import annotations import base64 +import datetime as dt +import hashlib import json import os import shlex +import uuid from pathlib import Path from typing import Any, Dict @@ -30,6 +33,8 @@ _STDOUT_FILE = "/tmp/evalscope-native-multiagent-stdout.log" _STDERR_FILE = "/tmp/evalscope-native-multiagent-stderr.log" _RUNTIME_IDENTITY_FILE = "/tmp/multiagent-prod-swe/runtime-identity.json" +_TRACE_ARCHIVE_FILE = "/tmp/evalscope-native-multiagent-trace.tar.gz" +_TRACE_CHUNK_BYTES = 256 * 1024 _DEFAULT_SOLVER_COMMAND = "/tmp/evalscope-native-multiagent-solver.sh" _PUBLIC_METADATA_KEYS = { "language", @@ -94,9 +99,10 @@ def __init__( working_dir: str = "/app", model_name: str = "gpt-5", codex_auth_json: str = "", - codex_auth_container_home: str = "/root/.codex-multiagent-prod", + codex_auth_container_home: str = "/tmp/multiagent-prod-swe/codex-home", swe_bench_pro_repo_path: str = "", swe_bench_pro_sample_offset: int = 0, + trace_output_dir: str = "", **_: Any, ) -> None: self._working_dir = working_dir or "/app" @@ -104,9 +110,12 @@ def __init__( self._codex_auth_json = codex_auth_json.strip() if not self._codex_auth_json: raise ValueError("multiagent-native requires runtime Codex auth JSON") - self._codex_auth_container_home = codex_auth_container_home.rstrip("/") or "/root/.codex-multiagent-prod" + self._codex_auth_container_home = ( + codex_auth_container_home.rstrip("/") or "/tmp/multiagent-prod-swe/codex-home" + ) self._swe_bench_pro_repo_path = swe_bench_pro_repo_path.strip() self._swe_bench_pro_sample_offset = swe_bench_pro_sample_offset + self._trace_output_dir = Path(trace_output_dir).expanduser().resolve() if trace_output_dir.strip() else None async def setup(self, env: AgentEnvironment) -> None: await self._write_file(env, _DEFAULT_SOLVER_COMMAND, _SOLVER_LAUNCHER) @@ -158,6 +167,8 @@ async def run( f"cwd={self._working_dir} command={command!r}" ) runtime_identity: dict[str, Any] = {} + trace_export: dict[str, Any] = {} + trace_export_error: Exception | None = None try: result = await env.exec(["bash", "-lc", shell_command], timeout=task.timeout, env=env_vars, cwd=self._working_dir) finally: @@ -165,6 +176,21 @@ async def run( runtime_identity = await self._read_json_file(env, _RUNTIME_IDENTITY_FILE) except Exception as exc: logger.warning(f"multiagent-native could not read runtime identity: {exc!r}") + if self._trace_output_dir is not None: + try: + trace_export = await self._export_trace_bundle( + env, + sample_id=sample_id, + sample_index=sample_index, + instance_id=raw_metadata.get("instance_id"), + ) + except Exception as exc: + trace_export_error = exc + logger.error( + "multiagent-native could not export trace for official_index=%s: %r", + sample_index, + exc, + ) await self._scrub_codex_auth(env) logger.info( f"multiagent-native exited: sample={sample_id} rc={result.returncode} " @@ -178,6 +204,11 @@ async def run( stderr = await env.exec(["bash", "-lc", f"tail -c 4000 {shlex.quote(_STDERR_FILE)} 2>/dev/null || true"]) stdout_tail = (stdout.stdout or "")[-4000:] stderr_tail = (stderr.stdout or "")[-4000:] + if trace_export_error is not None: + raise RuntimeError( + f"multiagent-native could not export the configured trace for official_index={sample_index}: " + f"{trace_export_error}" + ) from trace_export_error if result.timed_out: raise RunnerTimeoutError(f"multiagent-native timed out after {task.timeout}s") elif result.returncode != 0: @@ -191,9 +222,135 @@ async def run( "timed_out": result.timed_out, "stderr_tail": stderr_tail, "runtime_identity": runtime_identity, + "trace_export": trace_export, }, ) + async def _export_trace_bundle( + self, + env: AgentEnvironment, + *, + sample_id: Any, + sample_index: int, + instance_id: Any, + ) -> dict[str, Any]: + """Copy the container-local multiagent trace into a host-side row archive.""" + + if self._trace_output_dir is None: + return {} + + prepare_script = f""" +set -euo pipefail +stage=/tmp/evalscope-native-multiagent-trace-stage +archive={shlex.quote(_TRACE_ARCHIVE_FILE)} +rm -rf -- "$stage" +mkdir -p "$stage/runner" +if [[ -d /tmp/multiagent-prod-swe/state ]]; then + cp -a /tmp/multiagent-prod-swe/state "$stage/state" + rm -f -- "$stage/state/runtime_state/tmux.sock" +else + printf 'multiagent state directory was not created\n' > "$stage/state-missing.txt" +fi +for source in {_STDOUT_FILE} {_STDERR_FILE} {_RUNTIME_IDENTITY_FILE}; do + if [[ -f "$source" ]]; then + cp -a "$source" "$stage/runner/$(basename "$source")" + fi +done +tar -C "$stage" -czf "$archive" . +size=$(wc -c < "$archive" | tr -d '[:space:]') +digest=$(sha256sum "$archive" | awk '{{print $1}}') +printf '%s\t%s\n' "$size" "$digest" +""" + prepared = await env.exec(["bash", "-lc", prepare_script], timeout=120) + if prepared.returncode != 0: + detail = ((prepared.stderr or "") + "\n" + (prepared.stdout or "")).strip()[-4000:] + raise RuntimeError(f"could not prepare container trace archive: {detail}") + fields = (prepared.stdout or "").strip().splitlines()[-1].split("\t") + if len(fields) != 2: + raise RuntimeError(f"container trace archive metadata is malformed: {prepared.stdout!r}") + try: + expected_size = int(fields[0]) + except ValueError as exc: + raise RuntimeError(f"container trace archive size is invalid: {fields[0]!r}") from exc + expected_digest = fields[1].strip().lower() + if expected_size < 1 or len(expected_digest) != 64: + raise RuntimeError( + f"container trace archive metadata is invalid: size={expected_size} sha256={expected_digest!r}" + ) + + row_dir = self._trace_output_dir / f"official-row-{sample_index:06d}" + row_dir.mkdir(mode=0o700, parents=True, exist_ok=True) + archive_path = row_dir / "multiagent-trace.tar.gz" + temporary_path = row_dir / f".{archive_path.name}.{uuid.uuid4().hex}.tmp" + digest = hashlib.sha256() + written = 0 + try: + with temporary_path.open("wb") as handle: + for chunk_index in range((expected_size + _TRACE_CHUNK_BYTES - 1) // _TRACE_CHUNK_BYTES): + chunk_script = ( + "set -o pipefail; " + f"dd if={shlex.quote(_TRACE_ARCHIVE_FILE)} bs={_TRACE_CHUNK_BYTES} " + f"skip={chunk_index} count=1 status=none | base64" + ) + chunk_result = await env.exec(["bash", "-lc", chunk_script], timeout=120) + if chunk_result.returncode != 0: + detail = ((chunk_result.stderr or "") + "\n" + (chunk_result.stdout or "")).strip()[-2000:] + raise RuntimeError(f"could not read trace archive chunk {chunk_index}: {detail}") + try: + chunk = base64.b64decode((chunk_result.stdout or "").encode("ascii"), validate=False) + except (UnicodeEncodeError, ValueError) as exc: + raise RuntimeError(f"trace archive chunk {chunk_index} is not valid base64") from exc + if not chunk: + raise RuntimeError(f"trace archive chunk {chunk_index} is empty") + handle.write(chunk) + digest.update(chunk) + written += len(chunk) + if written != expected_size: + raise RuntimeError(f"trace archive size mismatch: expected {expected_size}, copied {written}") + actual_digest = digest.hexdigest() + if actual_digest != expected_digest: + raise RuntimeError( + f"trace archive digest mismatch: expected {expected_digest}, copied {actual_digest}" + ) + temporary_path.chmod(0o600) + temporary_path.replace(archive_path) + finally: + temporary_path.unlink(missing_ok=True) + + manifest = { + "captured_at": dt.datetime.now(dt.timezone.utc).isoformat(timespec="seconds"), + "official_index": sample_index, + "sample_id": None if sample_id is None else str(sample_id), + "instance_id": None if instance_id is None else str(instance_id), + "archive": archive_path.name, + "archive_bytes": expected_size, + "archive_sha256": expected_digest, + "container_state_dir": "/tmp/multiagent-prod-swe/state", + "submission_workspace": self._working_dir, + } + manifest_path = row_dir / "manifest.json" + manifest_tmp = row_dir / f".{manifest_path.name}.{uuid.uuid4().hex}.tmp" + try: + manifest_tmp.write_text(json.dumps(manifest, indent=2) + "\n", encoding="utf-8") + manifest_tmp.chmod(0o600) + manifest_tmp.replace(manifest_path) + finally: + manifest_tmp.unlink(missing_ok=True) + + logger.info( + "multiagent-native trace exported: official_index=%s path=%s bytes=%s sha256=%s", + sample_index, + archive_path, + expected_size, + expected_digest, + ) + return { + "path": str(archive_path), + "manifest": str(manifest_path), + "bytes": expected_size, + "sha256": expected_digest, + } + async def _read_json_file(self, env: AgentEnvironment, path: str) -> dict[str, Any]: result = await env.exec(["bash", "-lc", f"cat {shlex.quote(path)} 2>/dev/null || true"], timeout=30) raw = (result.stdout or "").strip() @@ -283,7 +440,11 @@ async def _install_codex_auth(self, env: AgentEnvironment) -> None: async def _scrub_codex_auth(self, env: AgentEnvironment) -> None: home = shlex.quote(self._codex_auth_container_home) - result = await env.exec(["bash", "-lc", f"rm -rf -- {home}"], timeout=30) + role_homes = shlex.quote("/tmp/multiagent-prod-swe/role-codex-homes") + result = await env.exec( + ["bash", "-lc", f"rm -rf -- {home} {role_homes}"], + timeout=30, + ) if result.returncode != 0: tail = ((result.stderr or "") + "\n" + (result.stdout or "")).strip()[-1000:] logger.warning(f"multiagent-native failed to scrub Codex auth home: {tail}") diff --git a/evaluation/native_solver/swe_prod_bootstrap.py b/evaluation/native_solver/swe_prod_bootstrap.py index 99bfc2c..93e731a 100644 --- a/evaluation/native_solver/swe_prod_bootstrap.py +++ b/evaluation/native_solver/swe_prod_bootstrap.py @@ -39,7 +39,7 @@ def write_codex_bridge(real_codex: str, model: str, auth_mode: str) -> None: CODEX_WRAPPER.write_text( f"""#!/usr/bin/env bash set -euo pipefail -export CODEX_HOME={str(CODEX_HOME)!r} +export CODEX_HOME="${{CODEX_HOME:-{str(CODEX_HOME)!s}}}" {codex_exec} -c 'model_provider="openai"' \\ -c 'model="{model}"' \\ @@ -53,7 +53,7 @@ def write_codex_bridge(real_codex: str, model: str, auth_mode: str) -> None: CODEX_WRAPPER.write_text( f"""#!/usr/bin/env bash set -euo pipefail -export CODEX_HOME={str(CODEX_HOME)!r} +export CODEX_HOME="${{CODEX_HOME:-{str(CODEX_HOME)!s}}}" {codex_exec} -c 'model_provider="evalscope"' \\ -c 'model_providers.evalscope.name="EvalScope Bridge"' \\ diff --git a/evaluation/native_solver/swe_prod_contracts.py b/evaluation/native_solver/swe_prod_contracts.py index 938cd78..7b076b5 100644 --- a/evaluation/native_solver/swe_prod_contracts.py +++ b/evaluation/native_solver/swe_prod_contracts.py @@ -16,7 +16,9 @@ RUNTIME_IDENTITY_PATH = RUNTIME_ROOT / "runtime-identity.json" TASK_METADATA_PATH = Path(os.environ.get("EVAL_TASK_METADATA_FILE", "/tmp/evalscope-native-multiagent-metadata.json")) CODEX_WRAPPER = RUNTIME_ROOT / "codex-bridge" -CODEX_HOME = Path(os.environ.get("CODEX_HOME", "/root/.codex-multiagent-prod")) +CODEX_HOME = Path(os.environ.get("CODEX_HOME", "/tmp/multiagent-prod-swe/codex-home")) +ROLE_CODEX_HOME_ROOT = RUNTIME_ROOT / "role-codex-homes" +TMUX_SOCKET = RUNTIME_ROOT / "state" / "runtime_state" / "tmux.sock" APPLY_PATCH_WRAPPER = RUNTIME_ROOT / "apply_patch" STABLE_APPLY_PATCH = Path("/usr/local/bin/apply_patch") diff --git a/evaluation/native_solver/swe_prod_lifecycle.py b/evaluation/native_solver/swe_prod_lifecycle.py index 0e0d730..43c3d98 100644 --- a/evaluation/native_solver/swe_prod_lifecycle.py +++ b/evaluation/native_solver/swe_prod_lifecycle.py @@ -3,6 +3,7 @@ import json import os import shutil +import stat import time from pathlib import Path @@ -17,8 +18,10 @@ from .swe_prod_contracts import ( CODEX_HOME, CODEX_WRAPPER, + ROLE_CODEX_HOME_ROOT, RUNTIME_IDENTITY_PATH, RUNTIME_ROOT, + TMUX_SOCKET, log, read_prompt, read_task_metadata, @@ -32,6 +35,84 @@ ) +ORCHESTRATOR_UID = 10001 +WRITER_UID = 10002 +READER_UID = 10003 +ROLE_GID = 10001 + + +def prepare_role_filesystem(workdir: Path, role_launcher: Path) -> None: + """Give worker processes source writes without giving them to the orchestrator.""" + + def prepare_tree(root: Path, uid: int, *, group_write: bool) -> None: + paths = [root] + paths.extend(root.rglob("*")) + for path in paths: + try: + info = path.lstat() + os.chown(path, uid, ROLE_GID, follow_symlinks=False) + if stat.S_ISLNK(info.st_mode): + continue + mode = stat.S_IMODE(info.st_mode) + mode &= ~stat.S_IWOTH + mode |= stat.S_IRGRP + if stat.S_ISDIR(info.st_mode): + mode |= stat.S_IXGRP + if group_write: + mode |= stat.S_IWGRP + else: + mode &= ~stat.S_IWGRP + # Some benchmark images expose Python builds where chmod does + # not implement follow_symlinks=False. lstat above already + # proves this is not a symlink, so the portable call is safe. + os.chmod(path, mode) + except FileNotFoundError: + continue + + prepare_tree(workdir, WRITER_UID, group_write=False) + + os.chown(role_launcher, 0, 0) + os.chmod(role_launcher, 0o4755) + + # Codex creates private config, lock, and SQLite files at runtime. Sharing a + # single CODEX_HOME across different role UIDs lets the orchestrator make + # its own home unreadable to later workers. Seed an independent home for + # each identity instead; auth is copied at runtime and is never baked into + # the task image or trace bundle. + ROLE_CODEX_HOME_ROOT.mkdir(parents=True, exist_ok=True) + seed_files = [path for path in (CODEX_HOME / "auth.json", CODEX_HOME / "config.toml") if path.is_file()] + for role, uid in ( + ("orchestrator", ORCHESTRATOR_UID), + ("writer", WRITER_UID), + ("reader", READER_UID), + ): + home = ROLE_CODEX_HOME_ROOT / role + home.mkdir(parents=True, exist_ok=True) + for source in seed_files: + shutil.copyfile(source, home / source.name) + (home / ".gitconfig").write_text( + f"[safe]\n\tdirectory = {workdir}\n", + encoding="utf-8", + ) + prepare_tree(home, uid, group_write=False) + os.chmod(home, 0o700) + + for cache in (RUNTIME_ROOT / "go-build-cache", RUNTIME_ROOT / "go-mod-cache"): + prepare_tree(cache, WRITER_UID, group_write=True) + + +def restore_workspace_owner(workdir: Path) -> None: + """Return the frozen workspace to the container owner for patch transport.""" + + paths = [workdir] + paths.extend(workdir.rglob("*")) + for path in paths: + try: + os.chown(path, 0, 0, follow_symlinks=False) + except FileNotFoundError: + continue + + def find_codex_cli() -> str | None: found = shutil.which("codex") if found: @@ -64,11 +145,15 @@ def toolchain_path_prefixes() -> list[str]: def tmux_has_session(session: str) -> bool: - return run(["tmux", "has-session", "-t", session], timeout=10).returncode == 0 + command = ["tmux", "-S", str(TMUX_SOCKET), "has-session", "-t", session] + return run(command, timeout=10).returncode == 0 def tmux_has_orchestrator(session: str) -> bool: - result = run(["tmux", "list-windows", "-t", session, "-F", "#W"], timeout=10) + result = run( + ["tmux", "-S", str(TMUX_SOCKET), "list-windows", "-t", session, "-F", "#W"], + timeout=10, + ) return result.returncode == 0 and "orchestrator" in result.stdout.splitlines() @@ -136,7 +221,7 @@ def run_prod_solver(prompt_path: str | None, workdir: Path, repo_root: Path, tim autonomous_prompt = make_prompt(repo_root, workdir, issue, task_metadata) session = f"swe-prod-{os.getpid()}" toolchain_prefix = ":".join(toolchain_path_prefixes()) - path_parts = [str(RUNTIME_ROOT)] + path_parts = [str(RUNTIME_ROOT), str(repo_root / "bin")] if toolchain_prefix: path_parts.append(toolchain_prefix) path_parts.append(os.environ.get("PATH", "")) @@ -158,8 +243,11 @@ def run_prod_solver(prompt_path: str | None, workdir: Path, repo_root: Path, tim "VERIFIER_CLI": "codex", "CODEX_BIN": str(CODEX_WRAPPER), "CODEX_HOME": str(CODEX_HOME), + "MULTIAGENT_CODEX_HOME_ROOT": str(ROLE_CODEX_HOME_ROOT), "MULTIAGENT_CODEX_EXEC": os.environ.get("MULTIAGENT_CODEX_EXEC", "1"), "MULTIAGENT_EXTRA_PATH": str(RUNTIME_ROOT), + "MULTIAGENT_ROLE_SHARED_WRITE_DIR": str(RUNTIME_ROOT), + "MULTIAGENT_UID_SANDBOX": "1", "PATH": ":".join(part for part in path_parts if part), "GOCACHE": ensure_cache_dir(RUNTIME_ROOT / "go-build-cache"), "GOMODCACHE": ensure_cache_dir(RUNTIME_ROOT / "go-mod-cache"), @@ -168,6 +256,8 @@ def run_prod_solver(prompt_path: str | None, workdir: Path, repo_root: Path, tim } ) + prepare_role_filesystem(workdir, Path(multiagent_command(repo_root)[0])) + launch_args = [str(repo_root / "launch.sh"), "--session", session, "--root", str(workdir), "--no-attach"] log(f"launching production multiagent session={session} root={workdir} repo={repo_root}") launch = run(launch_args, env=env, timeout=120) @@ -181,8 +271,9 @@ def run_prod_solver(prompt_path: str | None, workdir: Path, repo_root: Path, tim time.sleep(5) finally: if tmux_has_session(session): - run(["tmux", "kill-session", "-t", session], timeout=30) + run(["tmux", "-S", str(TMUX_SOCKET), "kill-session", "-t", session], timeout=30) + restore_workspace_owner(workdir) materialize_committed_changes(workdir, start_head) mark_untracked_intent_to_add(workdir) log("workspace prepared for EvalScope submission") diff --git a/evaluation/native_solver/templates/swe_autonomous_appendix.md b/evaluation/native_solver/templates/swe_autonomous_appendix.md index 0623dfe..5a31563 100644 --- a/evaluation/native_solver/templates/swe_autonomous_appendix.md +++ b/evaluation/native_solver/templates/swe_autonomous_appendix.md @@ -6,6 +6,14 @@ repository is `/app` and the framework is installed at `/opt/multiagent`. Use only the public task and visible repository contents. Do not use hidden tests, expected patches, benchmark scores, row identity, or private metadata. +This run has no interactive user. Treat every behavior explicitly stated in the +public task as already user-approved. Do not stop to ask the user to reselect an +explicit requirement because the repository exposes aliases, legacy APIs, or +additional possible behavior. When the public task leaves an implementation +detail open, use the narrowest backward-compatible interpretation supported by +visible source/tests, record the assumption, and continue. Stop only for a true +contradiction that makes the public task impossible to implement safely. + Leave the final working-tree changes in `/app`. The adapter only transports that workspace to EvalScope; the official SWE-bench verifier evaluates it. diff --git a/evaluation/swe_bench_pro.py b/evaluation/swe_bench_pro.py index b41fb3d..1e5e7f4 100644 --- a/evaluation/swe_bench_pro.py +++ b/evaluation/swe_bench_pro.py @@ -30,6 +30,7 @@ DEFAULT_CONFIG_YAML = DEFAULT_REPORT_DIR / "swe-bench-pro-production-task-config.yaml" DEFAULT_PREFLIGHT_OUTPUT = DEFAULT_REPORT_DIR / "swe-bench-pro-official-preflight.json" DEFAULT_ON_DEMAND_IMAGE_STATUS = DEFAULT_REPORT_DIR / "swe-bench-pro-on-demand-image-status.json" +DEFAULT_NATIVE_TRACE_DIR = DEFAULT_REPORT_DIR / "swe-bench-pro-traces" DEFAULT_IMAGE_ARCHIVE_DIR = Path("/private/tmp/swe-bench-pro-image-preload") DEFAULT_PERSISTENT_CACHE_ROOT = Path("/private/tmp/swe-bench-pro-persistent-cache") DEFAULT_NATIVE_SOLVER_SOURCE = Path(__file__).resolve().parents[1] @@ -154,6 +155,7 @@ def evaluation_config(args: argparse.Namespace) -> dict[str, Any]: "working_dir": args.agent_working_dir, "swe_bench_pro_repo_path": str(args.swe_bench_pro_repo_path), "swe_bench_pro_sample_offset": args.sample_offset, + "trace_output_dir": str(args.native_trace_dir), }, }, "work_dir": str(args.work_dir), @@ -631,6 +633,7 @@ def summarize_result( "native_solver_source": str(args.native_solver_source), "native_codex_auth_mode": "chatgpt-auth-json", "native_codex_auth_container_home": args.native_codex_auth_container_home, + "native_trace_dir": str(args.native_trace_dir), "submission_policy": "pass current workspace diff to the official SWE-bench verifier", }, "on_demand_image_status": ( @@ -752,7 +755,16 @@ def main() -> int: default=Path(auth_from_env).expanduser() if auth_from_env else None, help="host path to Codex auth.json copied into each live task container at runtime; never baked into images", ) - parser.add_argument("--native-codex-auth-container-home", default="/root/.codex-multiagent-prod") + parser.add_argument( + "--native-codex-auth-container-home", + default="/tmp/multiagent-prod-swe/codex-home", + ) + parser.add_argument( + "--native-trace-dir", + type=Path, + default=DEFAULT_NATIVE_TRACE_DIR, + help="host directory for per-official-row multiagent trace archives exported before task containers close", + ) parser.add_argument("--api-url", default=os.environ.get("EVALSCOPE_MODEL_API_URL", "http://127.0.0.1:8765/v1")) parser.add_argument("--api-key", default=os.environ.get("EVALSCOPE_MODEL_API_KEY", "EMPTY")) parser.add_argument("--limit", type=parse_limit, default=1) @@ -807,6 +819,8 @@ def main() -> int: parser.error("--native-codex-auth-json or NATIVE_CODEX_AUTH_JSON is required for a production evaluation") if args.native_codex_auth_json and not args.native_codex_auth_json.is_file(): parser.error(f"Codex auth file does not exist: {args.native_codex_auth_json}") + if args.native_trace_dir.exists() and not args.native_trace_dir.is_dir(): + parser.error(f"--native-trace-dir must be a directory: {args.native_trace_dir}") config = evaluation_config(args) write_config(config, args.config_json, args.config_yaml) diff --git a/evaluation/swe_bench_pro_run_parallel_shards.py b/evaluation/swe_bench_pro_run_parallel_shards.py index 5d9d8c8..938884c 100644 --- a/evaluation/swe_bench_pro_run_parallel_shards.py +++ b/evaluation/swe_bench_pro_run_parallel_shards.py @@ -106,6 +106,8 @@ def build_worker_command( str(args.native_codex_auth_json), "--native-codex-auth-container-home", args.native_codex_auth_container_home, + "--native-trace-dir", + str(args.native_trace_dir), "--on-demand-min-free-gb", str(args.on_demand_min_free_gb), "--no-docker-inspect", @@ -154,7 +156,15 @@ def main() -> int: parser.add_argument("--on-demand-min-free-gb", type=float, default=50.0) parser.add_argument("--native-solver-source", type=Path, default=DEFAULT_NATIVE_SOLVER_SOURCE) parser.add_argument("--native-codex-auth-json", type=Path, required=True) - parser.add_argument("--native-codex-auth-container-home", default="/root/.codex-multiagent-prod") + parser.add_argument( + "--native-codex-auth-container-home", + default="/tmp/multiagent-prod-swe/codex-home", + ) + parser.add_argument( + "--native-trace-dir", + type=Path, + help="shared host directory for per-official-row multiagent trace archives; defaults to REPORT_DIR/traces", + ) parser.add_argument("--persistent-cache", action="store_true") parser.add_argument("--persistent-cache-root", type=Path, default=Path("/private/tmp/swe-bench-pro-persistent-cache")) parser.add_argument("--persistent-cache-mode", default="rw", choices=["rw", "ro"]) @@ -165,6 +175,9 @@ def main() -> int: parser.add_argument("--dry-run", action="store_true") args = parser.parse_args() + if args.native_trace_dir is None: + args.native_trace_dir = args.report_dir / "traces" + if args.workers < 1 or args.shard_size < 1: parser.error("--workers and --shard-size must be >= 1") explicit_offsets = parse_sample_offsets(args.sample_offsets or "") diff --git a/orchestrator_prompt.md b/orchestrator_prompt.md index 6573e05..f985aa0 100644 --- a/orchestrator_prompt.md +++ b/orchestrator_prompt.md @@ -97,10 +97,13 @@ The launch script exports: - `VERIFIER_CLI`: CLI to use for verifier agents, default `codex`. Supported CLI values are `codex` and `claude`. Keep the orchestrator on Codex -unless the user explicitly asks otherwise. Codex commands use `--cd`, -`--dangerously-bypass-approvals-and-sandbox`, and `--no-alt-screen`. Claude -commands start from the target worktree/root and use -`claude --dangerously-skip-permissions`. +unless the user explicitly asks otherwise. The Rust supervisor maps trusted +roles to enforced access profiles: the orchestrator can write durable state but +the target repository is read-only, workers can write the target workspace, +and scouts/authority reviewers are read-only. Native hosts use Codex sandboxes; +production Linux containers use unprivileged role identities. Do not bypass the Rust spawn path +with direct `codex`, `claude`, or `tmux new-window` commands. Claude does not +provide the same role-level OS sandbox and is retained only for compatibility. If a variable is missing, infer the tmux session with: @@ -143,7 +146,10 @@ Use one verifier window per worker assignment at a time. A verifier is a read-only reviewer, not a second implementer. Before spawning a replacement worker for the same owned files, poll the existing -worker and either finalize/kill it or explicitly wait. If validation ownership +worker and either finalize/kill it or explicitly wait. Prefer the bounded +`multiagent subagent wait NAME --timeout SECONDS` command when a result is +required before continuing; one immediate poll is not evidence that an agent is +stalled. If validation ownership is unclear, use the validation coordinator role before adding more workers. ## Role Routing diff --git a/prompts/playbooks/agent-spawning.md b/prompts/playbooks/agent-spawning.md index 08b6781..1a4dcf1 100644 --- a/prompts/playbooks/agent-spawning.md +++ b/prompts/playbooks/agent-spawning.md @@ -31,34 +31,24 @@ multiagent subagent assignment-create worker-01-task \ --plan-id PLAN_ID \ --branch BRANCH \ --owned PATH[,PATH...] -multiagent subagent worktree-create worker-01-task multiagent subagent checkpoint-update worker-01-task --step "assignment created" --status assigned ``` -Use a separate git worktree per worker unless the user explicitly directs -otherwise. Spawn from that worktree path: +For the normal single-writer path, spawn through the Rust supervisor in the +shared target workspace. The trusted worker role receives workspace-write +access while the orchestrator remains unable to edit that workspace: ```bash -WORKTREE_PATH="$(multiagent subagent worktree-show worker-01-task | awk -F= '$1 == "path" {print $2}')" -WORKER_CLI="${WORKER_CLI:-claude}" -case "$WORKER_CLI" in - codex) - WORKER_COMMAND="cd '$WORKTREE_PATH' && ${CODEX_BIN:-codex} --cd '$WORKTREE_PATH' --dangerously-bypass-approvals-and-sandbox --no-alt-screen" - ;; - claude) - WORKER_COMMAND="cd '$WORKTREE_PATH' && ${CLAUDE_BIN:-claude} --dangerously-skip-permissions" - ;; - *) - echo "Unsupported WORKER_CLI: $WORKER_CLI" >&2 - exit 2 - ;; -esac -tmux new-window -d -t "$MULTIAGENT_SESSION" -n "worker-01-task" "$WORKER_COMMAND" +SUBAGENT_CLI="$WORKER_CLI" multiagent subagent spawn worker-01-task \ + --role worker --instruction-file WORKER_INSTRUCTION +multiagent subagent wait worker-01-task --timeout 1800 ``` -Capture repeatedly until the selected CLI prompt is visible. If the pane shows -authentication/setup blockers or never becomes ready, report the blocker -instead of sending instructions. +The supervisor handles readiness and capture. Inspect a terminal `blocked` or +`failed` result instead of treating it as completion. Separate git worktrees +remain available for intentionally parallel, disjoint assignments, but require +an explicit integration step before completion; do not use them for the normal +SWE single-writer path. ## Long-Running Subagent Skill @@ -67,7 +57,7 @@ persists context: ```bash multiagent subagent spawn subagent-build-watch --instruction "FIRST_INSTRUCTION_TEXT" -multiagent subagent poll subagent-build-watch +multiagent subagent wait subagent-build-watch --timeout 900 multiagent subagent inspect subagent-build-watch --lines 160 multiagent subagent finalize subagent-build-watch ``` diff --git a/prompts/playbooks/implementation-lifecycle.md b/prompts/playbooks/implementation-lifecycle.md index 9df36e2..deefc31 100644 --- a/prompts/playbooks/implementation-lifecycle.md +++ b/prompts/playbooks/implementation-lifecycle.md @@ -52,6 +52,14 @@ destructive or difficult-to-reverse behavior, material scope or cost, or a prior explicit user decision. Treat uncertain authority as user-owned. Evidence may clarify a choice but does not transfer authority. +The user's explicit task contract is already approved public behavior. Do not +ask the user to choose it again or reinterpret repository alternatives as an +unanswered product decision. A user-owned question exists only when two +materially different outcomes both remain consistent with the complete request +after bounded source/test inspection. Otherwise select the narrowest +backward-compatible implementation of the stated contract and record that +source-backed choice as orchestrator-owned. + For consequential or uncertain decisions, run the independent `decision-authority-reviewer` role. It must check both the proposed authority and whether the TODOs or proposed assignment contain omitted decisions. Ask the @@ -62,8 +70,13 @@ Spawn that review read-only through the normal subagent path, for example: ```bash SUBAGENT_CLI="$VERIFIER_CLI" multiagent subagent spawn decision-authority-reviewer \ --role reviewer --instruction-file AUTHORITY_REVIEW_INPUT +multiagent subagent wait decision-authority-reviewer --timeout 900 ``` +Do not continue merely because an immediate poll still reports `running`. +Inspect the completed or blocked result after the bounded wait and persist its +actual authority finding before preparing implementation. + Create an approved implementation context document containing the selected plan, decision and plan IDs, authority and approval basis, intended outcome, rejected alternatives and reasons, must-do and must-not-do constraints, migration choice, @@ -112,9 +125,19 @@ Run independent reviews against the frozen candidate diff: - `technical`: verify behavior and the accepted contract; - `reflection`: compare expected and actual results and identify improvements. -Record each review with `multiagent workflow record-review`. Every actionable -finding must be added with `multiagent workflow add-todo`; a review with findings is -not a terminal review. +Every reviewer final message must include an exact durable marker on its own +line: `review-record: type=TYPE verdict=pass|findings diff=DIFF_HASH` (use +`diff=-` for decision-authority). A Markdown list prefix or enclosing backticks +are accepted as cosmetic formatting, but surrounding prose is not. Wait for and finalize that reviewer before +recording its result. Record each review with `multiagent workflow +record-review ... --reviewer REVIEWER_NAME`; the supervisor rejects an +orchestrator-authored verdict that is not backed by the finalized read-only +reviewer's matching marker. The supervisor also scans finalized reviewer +messages: a findings marker for the current candidate cannot be discarded by +launching a replacement reviewer and recording only the later pass. Record the +finding, route it through the repair loop, and begin a new lifecycle iteration +before attempting completion. Every actionable finding must be added with +`multiagent workflow add-todo`; a review with findings is not a terminal review. Technical verifier findings must also use the existing structured `finding-create -> todo-create -> resolution-create -> todo-close` protocol in @@ -156,6 +179,10 @@ multiagent orchestrator complete The final command also runs `multiagent subagent gate-check`, so lifecycle reviews cannot substitute for hash-bound technical finding and TODO closure. +After it succeeds, the candidate is sealed: stop launching workers or reviewers +and do not mutate the repository. The privileged writer bridge independently +rechecks the live lifecycle phase and rejects any post-completion writer, even +if a shell command overrides `MULTIAGENT_LIFECYCLE_ENFORCEMENT`. `MULTIAGENT_VERIFIER_MAX_ITERATIONS` is an escalation threshold, not an acceptance condition. At the threshold, reconsider the route, surface a diff --git a/prompts/roles/decision-authority-reviewer.md b/prompts/roles/decision-authority-reviewer.md index e10d714..5975f2b 100644 --- a/prompts/roles/decision-authority-reviewer.md +++ b/prompts/roles/decision-authority-reviewer.md @@ -23,6 +23,22 @@ destructive or difficult-to-reverse behavior, material scope or cost, and conflict with a prior explicit user decision. Treat uncertain authority as user-owned. +The original request is itself the user's decision for every behavior it +explicitly specifies. Do not reopen that behavior merely because the repository +contains multiple lookup helpers, representations, legacy paths, synonyms, or +possible edge-case policies. Read the task's clauses together; explanatory +parentheticals and named canonical forms refine the contract rather than create +new alternatives. Choosing the narrowest source-backed implementation that +directly realizes an explicit requirement is orchestrator-owned. + +Return `user-choice-required` only when at least two materially different +public outcomes remain compatible with the complete explicit request, bounded +source/test evidence does not select between them, and choosing one would add, +remove, or contradict public behavior. Name the exact unresolved conflict. Do +not escalate hypothetical collisions, normalization policies, compatibility +variants, or other unrequested behavior; preserve existing behavior and use the +narrowest contract-compatible default instead. + Return only: 1. `verdict:` `orchestrator-may-decide`, `user-choice-required`, or @@ -33,6 +49,9 @@ Return only: conditions. 5. `user-question:` compact alternatives and tradeoffs when user choice is required; otherwise `none`. +6. `review-record: type=decision-authority verdict=pass diff=-` when the verdict + is `orchestrator-may-decide`; otherwise + `review-record: type=decision-authority verdict=findings diff=-`. Do not use agent agreement or majority preference as authority. A passing review means the orchestrator may proceed under the recorded authority; it is diff --git a/prompts/roles/scope-guard.md b/prompts/roles/scope-guard.md index b122d1e..f18499e 100644 --- a/prompts/roles/scope-guard.md +++ b/prompts/roles/scope-guard.md @@ -49,6 +49,9 @@ Return only: 4. `must-preserve:` contract items the next worker/verifier must carry forward. 5. `validation-gaps:` exact tests/probes/source inspections still needed. 6. `routing:` recommended next worker or verifier assignment, with owned paths. +7. `review-record: type=scope verdict=pass|findings diff=DIFF_HASH`, using the + candidate diff hash supplied in the assignment. Only an unqualified + `scope-verdict: accept` maps to `pass`. Keep the report compact enough for the orchestrator to paste into verifier or follow-up worker instructions. diff --git a/prompts/verifier.md b/prompts/verifier.md index da60848..b2caa23 100644 --- a/prompts/verifier.md +++ b/prompts/verifier.md @@ -436,6 +436,10 @@ for the exact live final diff. Build acceptance remains a separate build verifier artifact. A missing verdict, stale hash, or unbound acceptance is blocking at the framework gate. +Also include `review-record: type=technical verdict=pass diff=DIFF_HASH` on its +own line for `ACCEPTED`, or +`review-record: type=technical verdict=findings diff=DIFF_HASH` for `BLOCKING`. + Prefer the exact line marker above. If structured JSON is also emitted, use `final_diff_sha256`, `compile_clean: true`, and a non-empty `commands` array whose every entry records `rc: 0`; the gate treats that as equivalent evidence. diff --git a/src/config.rs b/src/config.rs index d49772e..7b296ca 100644 --- a/src/config.rs +++ b/src/config.rs @@ -1,6 +1,40 @@ use std::env; use std::path::PathBuf; +pub const ORCHESTRATOR_UID: u32 = 10001; + +/// Return whether lifecycle gates are mandatory for the current process. +/// +/// The environment flag remains useful for non-isolated local invocations, but +/// it is not an authority boundary. A UID-isolated orchestrator must not be +/// able to disable the supervisor by overriding an environment variable in a +/// shell command. +pub fn lifecycle_enforced() -> bool { + lifecycle_enforced_for( + real_uid(), + env::var("MULTIAGENT_LIFECYCLE_ENFORCEMENT").ok().as_deref(), + env::var("MULTIAGENT_UID_SANDBOX").ok().as_deref(), + ) +} + +fn lifecycle_enforced_for( + real_uid: u32, + requested: Option<&str>, + uid_sandbox: Option<&str>, +) -> bool { + real_uid == ORCHESTRATOR_UID || requested == Some("1") || uid_sandbox == Some("1") +} + +#[cfg(unix)] +fn real_uid() -> u32 { + unsafe { libc::getuid() } +} + +#[cfg(not(unix))] +fn real_uid() -> u32 { + u32::MAX +} + pub fn root() -> Result { match env::var_os("MULTIAGENT_ROOT") { Some(value) if !value.is_empty() => Ok(PathBuf::from(value)), @@ -15,3 +49,21 @@ pub fn state_dir() -> Result { _ => Ok(root()?.join(".multiagent")), } } + +#[cfg(test)] +mod tests { + use super::{lifecycle_enforced_for, ORCHESTRATOR_UID}; + + #[test] + fn orchestrator_cannot_disable_lifecycle_with_environment_override() { + assert!(lifecycle_enforced_for( + ORCHESTRATOR_UID, + Some("0"), + Some("0") + )); + assert!(lifecycle_enforced_for(ORCHESTRATOR_UID, None, None)); + assert!(lifecycle_enforced_for(0, Some("1"), Some("0"))); + assert!(lifecycle_enforced_for(0, Some("0"), Some("1"))); + assert!(!lifecycle_enforced_for(0, Some("0"), Some("0"))); + } +} diff --git a/src/main.rs b/src/main.rs index 382a195..bd67b8a 100644 --- a/src/main.rs +++ b/src/main.rs @@ -3,6 +3,7 @@ mod dag; mod decision; mod policy; mod prompt_bundle; +mod role_sandbox; mod runtime; mod snapshot; mod subagent; @@ -32,6 +33,10 @@ fn main() -> ExitCode { } let command = args.remove(0); + if let Err(message) = role_sandbox::gate_setuid_invocation(&command) { + eprintln!("multiagent: {message}"); + return ExitCode::from(1); + } let result: Result = match command.as_str() { "launch" => runtime::launch(&args).map_err(|message| ("launch", message)), "orchestrator" => runtime::orchestrator(&args).map_err(|message| ("orchestrator", message)), @@ -49,6 +54,10 @@ fn main() -> ExitCode { "prompt-bundle" => prompt_bundle::run(&args) .map(|_| ExitCode::SUCCESS) .map_err(|message| ("prompt-bundle", message)), + "role-exec" => role_sandbox::run(&args).map_err(|message| ("role-exec", message)), + "role-agent-exec" => { + runtime::role_agent_exec(&args).map_err(|message| ("role-agent-exec", message)) + } "snapshot" => snapshot::run(&args) .map(|_| ExitCode::SUCCESS) .map_err(|message| ("snapshot", message)), diff --git a/src/role_sandbox.rs b/src/role_sandbox.rs new file mode 100644 index 0000000..54110a5 --- /dev/null +++ b/src/role_sandbox.rs @@ -0,0 +1,436 @@ +use std::collections::BTreeSet; +use std::fs; +use std::path::PathBuf; +use std::process::{Command, ExitCode}; +#[cfg(unix)] +use std::sync::atomic::{AtomicI32, Ordering}; + +#[cfg(unix)] +static SUPERVISED_CHILD: AtomicI32 = AtomicI32::new(0); + +#[cfg(unix)] +pub fn gate_setuid_invocation(command: &str) -> Result<(), String> { + let real_uid = unsafe { libc::getuid() }; + let effective_uid = unsafe { libc::geteuid() }; + if effective_uid != 0 || real_uid == 0 { + return Ok(()); + } + if privileged_command_allowed(command) { + return Ok(()); + } + if unsafe { libc::setuid(real_uid) } != 0 { + return Err(format!( + "drop setuid privilege for {command}: {}", + std::io::Error::last_os_error() + )); + } + Ok(()) +} + +fn privileged_command_allowed(command: &str) -> bool { + command == "role-agent-exec" +} + +#[cfg(not(unix))] +pub fn gate_setuid_invocation(_command: &str) -> Result<(), String> { + Ok(()) +} + +pub fn run(args: &[String]) -> Result { + let mut write_roots = BTreeSet::new(); + let mut uid = None; + let mut gid = None; + let mut index = 0; + while index < args.len() { + match args[index].as_str() { + "--allow-write" => { + let path = args + .get(index + 1) + .filter(|value| !value.is_empty()) + .ok_or_else(|| "role-exec --allow-write requires PATH".to_string())?; + let canonical = fs::canonicalize(path) + .map_err(|error| format!("resolve role write root {path}: {error}"))?; + write_roots.insert(canonical); + index += 2; + } + "--uid" => { + uid = Some(parse_id(args, index, "--uid")?); + index += 2; + } + "--gid" => { + gid = Some(parse_id(args, index, "--gid")?); + index += 2; + } + "--" => { + index += 1; + break; + } + other => return Err(format!("unknown role-exec argument: {other}")), + } + } + let command = args + .get(index) + .filter(|value| !value.is_empty()) + .ok_or_else(|| "role-exec requires a command after --".to_string())?; + let command_args = &args[index + 1..]; + + if let Some(uid) = uid { + drop_identity( + uid, + gid.ok_or_else(|| "role-exec --uid requires --gid".to_string())?, + )?; + } else { + restrict_writes(&write_roots.into_iter().collect::>())?; + } + exec(command, command_args) +} + +/// Run a role process under a minimal privileged supervisor. +/// +/// The tmux server runs as the orchestrator UID while the role process runs as +/// a different UID. tmux therefore cannot reliably signal a writer after the +/// identity transition. Keep the setuid process as a wait-only parent whose +/// real UID remains the orchestrator UID, and place the writer in its own +/// process group. When tmux closes the pane, the parent receives the signal +/// and terminates the complete writer process group before it exits. +#[cfg(unix)] +pub fn run_supervised( + uid: u32, + gid: u32, + command: &str, + args: &[String], +) -> Result { + install_supervisor_signal_handlers()?; + + let _parent_pid = unsafe { libc::getpid() }; + let child = unsafe { libc::fork() }; + if child < 0 { + return Err(format!( + "fork role process: {}", + std::io::Error::last_os_error() + )); + } + if child == 0 { + if unsafe { libc::setpgid(0, 0) } != 0 { + unsafe { libc::_exit(126) }; + } + #[cfg(target_os = "linux")] + { + if unsafe { libc::prctl(libc::PR_SET_PDEATHSIG, libc::SIGKILL) } != 0 + || unsafe { libc::getppid() } != _parent_pid + { + unsafe { libc::_exit(126) }; + } + } + if drop_identity(uid, gid).is_err() { + unsafe { libc::_exit(126) }; + } + use std::os::unix::process::CommandExt; + let _ = Command::new(command).args(args).exec(); + unsafe { libc::_exit(127) }; + } + + // Close the fork-to-setpgid race so a pane-close signal can always target + // the child's group. EACCES merely means the child already exec'd. + unsafe { + libc::setpgid(child, child); + } + SUPERVISED_CHILD.store(child, Ordering::SeqCst); + + let mut status = 0; + loop { + let waited = unsafe { libc::waitpid(child, &mut status, 0) }; + if waited == child { + break; + } + if waited < 0 && std::io::Error::last_os_error().kind() == std::io::ErrorKind::Interrupted { + continue; + } + SUPERVISED_CHILD.store(0, Ordering::SeqCst); + return Err(format!( + "wait for role process: {}", + std::io::Error::last_os_error() + )); + } + SUPERVISED_CHILD.store(0, Ordering::SeqCst); + + if libc::WIFEXITED(status) { + return Ok(ExitCode::from(libc::WEXITSTATUS(status) as u8)); + } + if libc::WIFSIGNALED(status) { + return Ok(ExitCode::from((128 + libc::WTERMSIG(status)).min(255) as u8)); + } + Ok(ExitCode::FAILURE) +} + +#[cfg(not(unix))] +pub fn run_supervised( + _uid: u32, + _gid: u32, + _command: &str, + _args: &[String], +) -> Result { + Err("supervised role execution requires Unix".into()) +} + +#[cfg(unix)] +extern "C" fn terminate_supervised_child(_signal: libc::c_int) { + let child = SUPERVISED_CHILD.load(Ordering::SeqCst); + if child > 0 { + // A pane close is a cancellation boundary. SIGKILL prevents a detached + // CLI or one of its tool children from writing after kill returns. + unsafe { + libc::kill(-child, libc::SIGKILL); + libc::kill(child, libc::SIGKILL); + } + } +} + +#[cfg(unix)] +fn install_supervisor_signal_handlers() -> Result<(), String> { + for signal in [libc::SIGHUP, libc::SIGINT, libc::SIGTERM, libc::SIGQUIT] { + let mut action: libc::sigaction = unsafe { std::mem::zeroed() }; + action.sa_sigaction = terminate_supervised_child as usize; + if unsafe { libc::sigemptyset(&mut action.sa_mask) } != 0 + || unsafe { libc::sigaction(signal, &action, std::ptr::null_mut()) } != 0 + { + return Err(format!( + "install role supervisor signal handler: {}", + std::io::Error::last_os_error() + )); + } + } + Ok(()) +} + +fn parse_id(args: &[String], index: usize, flag: &str) -> Result { + args.get(index + 1) + .ok_or_else(|| format!("role-exec {flag} requires an integer"))? + .parse() + .map_err(|_| format!("role-exec {flag} requires an integer")) +} + +#[cfg(unix)] +fn drop_identity(uid: u32, gid: u32) -> Result<(), String> { + let groups = [gid as libc::gid_t]; + if unsafe { libc::setgroups(1, groups.as_ptr()) } != 0 { + return Err(format!( + "set role supplementary groups: {}", + std::io::Error::last_os_error() + )); + } + if unsafe { libc::setgid(gid as libc::gid_t) } != 0 { + return Err(format!( + "set role gid {gid}: {}", + std::io::Error::last_os_error() + )); + } + if unsafe { libc::setuid(uid as libc::uid_t) } != 0 { + return Err(format!( + "set role uid {uid}: {}", + std::io::Error::last_os_error() + )); + } + Ok(()) +} + +#[cfg(not(unix))] +fn drop_identity(_uid: u32, _gid: u32) -> Result<(), String> { + Err("role-exec uid isolation requires Unix".into()) +} + +#[cfg(target_os = "linux")] +fn restrict_writes(write_roots: &[PathBuf]) -> Result<(), String> { + linux::restrict_writes(write_roots) +} + +#[cfg(not(target_os = "linux"))] +fn restrict_writes(_write_roots: &[PathBuf]) -> Result<(), String> { + Err( + "role-exec is only available on Linux; use the native Codex sandbox on this platform" + .into(), + ) +} + +#[cfg(unix)] +fn exec(command: &str, args: &[String]) -> Result { + use std::os::unix::process::CommandExt; + + let error = Command::new(command).args(args).exec(); + Err(format!("execute role command {command}: {error}")) +} + +#[cfg(not(unix))] +fn exec(command: &str, args: &[String]) -> Result { + let status = Command::new(command) + .args(args) + .status() + .map_err(|error| format!("execute role command {command}: {error}"))?; + Ok(ExitCode::from(status.code().unwrap_or(1) as u8)) +} + +#[cfg(target_os = "linux")] +mod linux { + use std::fs::File; + use std::io; + use std::os::fd::{AsRawFd, FromRawFd}; + use std::os::unix::fs::OpenOptionsExt; + use std::path::PathBuf; + + const LANDLOCK_CREATE_RULESET_VERSION: u32 = 1; + const LANDLOCK_RULE_PATH_BENEATH: i32 = 1; + + const ACCESS_WRITE_FILE: u64 = 1 << 1; + const ACCESS_REMOVE_DIR: u64 = 1 << 4; + const ACCESS_REMOVE_FILE: u64 = 1 << 5; + const ACCESS_MAKE_CHAR: u64 = 1 << 6; + const ACCESS_MAKE_DIR: u64 = 1 << 7; + const ACCESS_MAKE_REG: u64 = 1 << 8; + const ACCESS_MAKE_SOCK: u64 = 1 << 9; + const ACCESS_MAKE_FIFO: u64 = 1 << 10; + const ACCESS_MAKE_BLOCK: u64 = 1 << 11; + const ACCESS_MAKE_SYM: u64 = 1 << 12; + const ACCESS_REFER: u64 = 1 << 13; + const ACCESS_TRUNCATE: u64 = 1 << 14; + const ACCESS_IOCTL_DEV: u64 = 1 << 15; + + #[repr(C)] + struct RulesetAttr { + handled_access_fs: u64, + } + + #[repr(C)] + struct PathBeneathAttr { + allowed_access: u64, + parent_fd: i32, + } + + pub fn restrict_writes(write_roots: &[PathBuf]) -> Result<(), String> { + let abi = unsafe { + libc::syscall( + libc::SYS_landlock_create_ruleset, + std::ptr::null::(), + 0, + LANDLOCK_CREATE_RULESET_VERSION, + ) + }; + if abi < 1 { + return Err(format!( + "Landlock is unavailable; refusing to run without role write enforcement: {}", + io::Error::last_os_error() + )); + } + + let handled_access = handled_access_for_abi(abi); + let ruleset_attr = RulesetAttr { + handled_access_fs: handled_access, + }; + let ruleset_fd = unsafe { + libc::syscall( + libc::SYS_landlock_create_ruleset, + &ruleset_attr, + std::mem::size_of::(), + 0, + ) + }; + if ruleset_fd < 0 { + return Err(format!( + "create Landlock ruleset: {}", + io::Error::last_os_error() + )); + } + let ruleset = unsafe { File::from_raw_fd(ruleset_fd as i32) }; + + for path in write_roots { + add_path_rule(&ruleset, path, handled_access)?; + } + + let no_new_privileges = unsafe { libc::prctl(libc::PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0) }; + if no_new_privileges != 0 { + return Err(format!( + "enable no-new-privileges for Landlock: {}", + io::Error::last_os_error() + )); + } + let restricted = + unsafe { libc::syscall(libc::SYS_landlock_restrict_self, ruleset.as_raw_fd(), 0) }; + if restricted != 0 { + return Err(format!( + "apply Landlock role ruleset: {}", + io::Error::last_os_error() + )); + } + Ok(()) + } + + fn handled_access_for_abi(abi: libc::c_long) -> u64 { + let mut access = ACCESS_WRITE_FILE + | ACCESS_REMOVE_DIR + | ACCESS_REMOVE_FILE + | ACCESS_MAKE_CHAR + | ACCESS_MAKE_DIR + | ACCESS_MAKE_REG + | ACCESS_MAKE_SOCK + | ACCESS_MAKE_FIFO + | ACCESS_MAKE_BLOCK + | ACCESS_MAKE_SYM; + if abi >= 2 { + access |= ACCESS_REFER; + } + if abi >= 3 { + access |= ACCESS_TRUNCATE; + } + if abi >= 5 { + access |= ACCESS_IOCTL_DEV; + } + access + } + + fn add_path_rule(ruleset: &File, path: &PathBuf, handled_access: u64) -> Result<(), String> { + let file = std::fs::OpenOptions::new() + .read(true) + .custom_flags(libc::O_PATH | libc::O_CLOEXEC) + .open(path) + .map_err(|error| format!("open Landlock write root {}: {error}", path.display()))?; + let is_dir = path.is_dir(); + let allowed_access = if is_dir { + handled_access + } else { + handled_access & (ACCESS_WRITE_FILE | ACCESS_TRUNCATE | ACCESS_IOCTL_DEV) + }; + let rule = PathBeneathAttr { + allowed_access, + parent_fd: file.as_raw_fd(), + }; + let added = unsafe { + libc::syscall( + libc::SYS_landlock_add_rule, + ruleset.as_raw_fd(), + LANDLOCK_RULE_PATH_BENEATH, + &rule, + 0, + ) + }; + if added != 0 { + return Err(format!( + "allow Landlock write root {}: {}", + path.display(), + io::Error::last_os_error() + )); + } + Ok(()) + } +} + +#[cfg(test)] +mod tests { + use super::privileged_command_allowed; + + #[test] + fn setuid_gate_only_retains_privilege_for_fixed_agent_launch() { + assert!(privileged_command_allowed("role-agent-exec")); + for command in ["role-exec", "subagent", "launch", "snapshot", "workflow"] { + assert!(!privileged_command_allowed(command)); + } + } +} diff --git a/src/runtime.rs b/src/runtime.rs index 4f3677f..f84699d 100644 --- a/src/runtime.rs +++ b/src/runtime.rs @@ -1,4 +1,4 @@ -use crate::{config, policy}; +use crate::{config, policy, role_sandbox}; use chrono::{Local, SecondsFormat, Utc}; use sha2::{Digest, Sha256}; use std::collections::{BTreeMap, BTreeSet}; @@ -8,7 +8,7 @@ use std::io::Write; use std::path::{Path, PathBuf}; use std::process::{Command, ExitCode, Output}; use std::thread; -use std::time::Duration; +use std::time::{Duration, Instant}; const STATUS_HEADER: &str = "TYPE\tNAME\tSTATUS\tWINDOW\tLAST_PROGRESS\tSTATE_DIR\tROLE\tDECISION_ID\tPLAN_ID\tWORKFLOW_ID\tNODE_ID\n"; @@ -29,6 +29,26 @@ struct RuntimeConfig { code_exec: bool, } +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +enum CodexAccess { + ReadOnly, + WorkspaceWrite, +} + +const ORCHESTRATOR_UID: u32 = config::ORCHESTRATOR_UID; +const WRITER_UID: u32 = 10002; +const READER_UID: u32 = 10003; +const ROLE_GID: u32 = 10001; + +impl CodexAccess { + fn sandbox(self) -> &'static str { + match self { + Self::ReadOnly => "read-only", + Self::WorkspaceWrite => "workspace-write", + } + } +} + impl RuntimeConfig { fn load() -> Result { let root = config::root()?; @@ -70,6 +90,107 @@ impl RuntimeConfig { } } +pub fn role_agent_exec(args: &[String]) -> Result { + let (name, restored) = match args { + [name] => (name.as_str(), false), + [name, flag] if flag == "--restore" => (name.as_str(), true), + _ => return Err("usage: multiagent role-agent-exec NAME [--restore]".into()), + }; + validate_name(name)?; + if env::var("MULTIAGENT_UID_SANDBOX").as_deref() != Ok("1") { + return Err("role-agent-exec requires MULTIAGENT_UID_SANDBOX=1".into()); + } + #[cfg(unix)] + if unsafe { libc::getuid() } != ORCHESTRATOR_UID { + return Err("role-agent-exec is reserved for the orchestrator UID".into()); + } + + let cfg = RuntimeConfig::load()?; + if !cfg.code_exec { + return Err("role-agent-exec requires MULTIAGENT_CODEX_EXEC=1".into()); + } + let dir = cfg.state.join("subagents").join(name); + let metadata = read_env(&dir.join("meta.env"))?; + if metadata.get("name").map(String::as_str) != Some(name) + || metadata.get("cli").map(String::as_str) != Some("codex") + || metadata.get("cli_bin").map(String::as_str) != Some(cfg.codex_bin.as_str()) + { + return Err("role-agent-exec metadata does not match the requested Codex agent".into()); + } + let access = match metadata.get("codex_access").map(String::as_str) { + Some("read-only") => CodexAccess::ReadOnly, + Some("workspace-write") => CodexAccess::WorkspaceWrite, + _ => return Err("role-agent-exec metadata has invalid codex_access".into()), + }; + validate_privileged_codex_bridge(Path::new(&cfg.codex_bin))?; + let prompt = dir.join(if restored { + "restore-instruction.txt" + } else { + "instruction.txt" + }); + if !prompt.is_file() { + return Err(format!( + "role-agent-exec instruction is missing: {}", + prompt.display() + )); + } + let instruction = fs::read_to_string(&prompt).map_err(io_error("read role instruction"))?; + if access == CodexAccess::WorkspaceWrite { + // The setuid bridge is the final write-authority boundary. Recheck the + // assignment against live lifecycle state here so an orchestrator + // cannot gain a writer by overriding launch-time environment flags. + validate_implementation_context(&cfg, name, Some(&prompt), &instruction)?; + } + let output = dir.join("last-message.txt"); + let command = build_cli_command( + "codex", + &cfg.root, + Some(&prompt), + Some(&output), + &cfg.codex_bin, + &cfg.claude_bin, + true, + access, + )?; + let supervisor_pid = dir.join("supervisor.pid"); + atomic_write( + &supervisor_pid, + &format!("{}\n", std::process::id()), + "role supervisor pid", + )?; + let result = role_sandbox::run_supervised( + if access == CodexAccess::WorkspaceWrite { + WRITER_UID + } else { + READER_UID + }, + ROLE_GID, + "/bin/sh", + &["-c".into(), command], + ); + let _ = fs::remove_file(supervisor_pid); + result +} + +#[cfg(unix)] +fn validate_privileged_codex_bridge(path: &Path) -> Result<(), String> { + use std::os::unix::fs::{MetadataExt, PermissionsExt}; + + let metadata = fs::metadata(path).map_err(io_error("inspect privileged Codex bridge"))?; + if !metadata.is_file() || metadata.uid() != 0 || metadata.permissions().mode() & 0o022 != 0 { + return Err(format!( + "privileged Codex bridge must be a root-owned, non-group-writable executable: {}", + path.display() + )); + } + Ok(()) +} + +#[cfg(not(unix))] +fn validate_privileged_codex_bridge(_path: &Path) -> Result<(), String> { + Err("role-agent-exec requires Unix".into()) +} + pub fn launch(args: &[String]) -> Result { if args .iter() @@ -143,12 +264,23 @@ pub fn launch(args: &[String]) -> Result { { return Err("MULTIAGENT_VERIFIER_MAX_ITERATIONS must be a positive integer".into()); } - let lifecycle_enforcement = - env_nonempty("MULTIAGENT_LIFECYCLE_ENFORCEMENT").unwrap_or_else(|| "1".into()); + let lifecycle_enforcement = if config::lifecycle_enforced() { + "1".into() + } else { + env_nonempty("MULTIAGENT_LIFECYCLE_ENFORCEMENT").unwrap_or_else(|| "1".into()) + }; if !matches!(lifecycle_enforcement.as_str(), "0" | "1") { return Err("MULTIAGENT_LIFECYCLE_ENFORCEMENT must be 0 or 1".into()); } require_command("tmux")?; + if env::var("MULTIAGENT_UID_SANDBOX").as_deref() == Ok("1") { + fs::create_dir_all(state_dir.join("runtime_state")) + .map_err(io_error("create tmux socket directory"))?; + env::set_var( + "MULTIAGENT_TMUX_SOCKET", + state_dir.join("runtime_state/tmux.sock"), + ); + } let orchestrator_bin = if orchestrator_cli == "codex" { &codex_bin } else { @@ -194,6 +326,7 @@ pub fn launch(args: &[String]) -> Result { state_dir.join("assignments"), state_dir.join("worktrees"), state_dir.join("runtime_state"), + state_dir.join("tmp"), log_dir.clone(), ] { fs::create_dir_all(directory).map_err(io_error("create runtime directory"))?; @@ -271,17 +404,33 @@ pub fn launch(args: &[String]) -> Result { &state_dir.join("orchestrator-last-message.txt"), resume, )?; - tmux_checked(&[ + if env::var("MULTIAGENT_UID_SANDBOX").as_deref() == Ok("1") { + prepare_uid_state_permissions(&state_dir)?; + } + let bootstrap_command = format!("bash {}", shell_escape(&bootstrap.display().to_string())); + let new_session = [ "new-session", "-d", "-s", &session, "-n", "orchestrator", - &format!("bash {}", shell_escape(&bootstrap.display().to_string())), - ])?; + &bootstrap_command, + ]; + if env::var("MULTIAGENT_UID_SANDBOX").as_deref() == Ok("1") { + tmux_checked_as_uid(&new_session, &executable, ORCHESTRATOR_UID)?; + } else { + tmux_checked(&new_session)?; + } tmux_checked(&["select-window", "-t", &format!("{session}:orchestrator")])?; pipe_log(&session, "orchestrator", &log_dir)?; + if env::var("MULTIAGENT_UID_SANDBOX").as_deref() == Ok("1") { + atomic_write( + &state_dir.join("runtime_state/tmux-access-ready"), + "ready\n", + "tmux access marker", + )?; + } println!("Started tmux session: {session}"); println!("Attach with: tmux attach -t {session}"); @@ -383,11 +532,31 @@ fn launch_environment( "MULTIAGENT_EXTRA_PATH", env_nonempty("MULTIAGENT_EXTRA_PATH").unwrap_or_default(), ), + ( + "MULTIAGENT_UID_SANDBOX", + env_nonempty("MULTIAGENT_UID_SANDBOX").unwrap_or_else(|| "0".into()), + ), + ( + "MULTIAGENT_CODEX_HOME_ROOT", + env_nonempty("MULTIAGENT_CODEX_HOME_ROOT").unwrap_or_default(), + ), + ( + "MULTIAGENT_TMUX_SOCKET", + env_nonempty("MULTIAGENT_TMUX_SOCKET").unwrap_or_default(), + ), + ("TMPDIR", state.join("tmp").display().to_string()), ("MULTIAGENT_BIN", executable.display().to_string()), ("PATH", env::var("PATH").unwrap_or_default()), ] { values.insert(key.into(), value); } + if env::var("MULTIAGENT_UID_SANDBOX").as_deref() == Ok("1") { + if let Some(root) = env_nonempty("MULTIAGENT_CODEX_HOME_ROOT") { + let home = Path::new(&root).join("orchestrator"); + values.insert("CODEX_HOME".into(), home.display().to_string()); + values.insert("HOME".into(), home.display().to_string()); + } + } values } @@ -410,6 +579,22 @@ fn write_bootstrap( for (key, value) in environment { text.push_str(&format!("export {key}={}\n", shell_escape(value))); } + if environment + .get("MULTIAGENT_UID_SANDBOX") + .map(String::as_str) + == Some("1") + { + text.push_str("umask 0007\n"); + text.push_str(&format!( + "until [[ -f {} ]]; do sleep 0.05; done\n", + shell_escape( + &Path::new(&environment["MULTIAGENT_STATE_DIR"]) + .join("runtime_state/tmux-access-ready") + .display() + .to_string() + ) + )); + } text.push_str(&format!( "printf 'Multiagent launch mode: MULTIAGENT_RESUME=%s (%s)\\n' {} {}\n", u8::from(resume), @@ -417,13 +602,38 @@ fn write_bootstrap( )); let command = build_cli_command( cli, - root, + environment + .get("MULTIAGENT_STATE_DIR") + .map(Path::new) + .unwrap_or(root), Some(prompt), Some(last_message), codex_bin, claude_bin, env::var("MULTIAGENT_CODEX_EXEC").as_deref() == Ok("1"), + CodexAccess::WorkspaceWrite, )?; + let command = if environment + .get("MULTIAGENT_UID_SANDBOX") + .map(String::as_str) + == Some("1") + { + // The tmux server and every ordinary pane already run as the + // non-writing orchestrator UID. Only the narrowly gated setuid role + // launcher may transition a subagent pane to a writer/reader UID. + command + } else { + wrap_linux_role_sandbox( + &command, + Path::new( + environment + .get("MULTIAGENT_BIN") + .ok_or_else(|| "missing MULTIAGENT_BIN in launch environment".to_string())?, + ), + role_write_roots(root, Path::new(&environment["MULTIAGENT_STATE_DIR"]), false), + ORCHESTRATOR_UID, + ) + }; text.push_str(&command); text.push('\n'); atomic_write(path, &text, "orchestrator bootstrap")?; @@ -459,7 +669,7 @@ pub fn orchestrator(args: &[String]) -> Result { if args != ["complete"] { return Err(format!("unknown command: {}", args[0])); } - if env::var("MULTIAGENT_LIFECYCLE_ENFORCEMENT").as_deref() == Ok("1") { + if config::lifecycle_enforced() { let workflow_id = env_nonempty("MULTIAGENT_WORKFLOW_ID") .ok_or_else(|| "lifecycle enforcement requires MULTIAGENT_WORKFLOW_ID".to_string())?; run_self_quiet(&["workflow", "completion-check", &workflow_id])?; @@ -806,6 +1016,7 @@ pub fn subagent(args: &[String]) -> Result { let name = one_name("poll", &args[1..])?; poll(&cfg, name, true)?; } + "wait" => wait(&cfg, &args[1..])?, "inspect" => inspect(&cfg, &args[1..])?, "recover-plan" => recover_plan(&cfg, &args[1..])?, "restore" => restore(&cfg, &args[1..])?, @@ -819,7 +1030,7 @@ pub fn subagent(args: &[String]) -> Result { fn print_subagent_usage() { println!( - "Usage:\n multiagent subagent spawn NAME [--own PATH[,PATH...] ...] [--role ROLE] [--instruction TEXT | --instruction-file PATH | -- TEXT]\n multiagent subagent list|recover-plan|restore-all|gate-check\n multiagent subagent poll|inspect|restore|finalize|kill NAME [OPTIONS]\n\nAll durable state and tmux subprocess orchestration are implemented by the Rust CLI." + "Usage:\n multiagent subagent spawn NAME [--own PATH[,PATH...] ...] [--role ROLE] [--instruction TEXT | --instruction-file PATH | -- TEXT]\n multiagent subagent list|recover-plan|restore-all|gate-check\n multiagent subagent poll|inspect|restore|finalize|kill NAME [OPTIONS]\n multiagent subagent wait NAME [--timeout SECONDS] [--poll-interval SECONDS]\n\nAll durable state and tmux subprocess orchestration are implemented by the Rust CLI." ); } @@ -890,6 +1101,7 @@ fn spawn(cfg: &RuntimeConfig, args: &[String]) -> Result<(), String> { instruction = compose_role_instruction(cfg, name, &role, &instruction)?; instruction = append_verifier_diff_binding(cfg, name, &role, &instruction)?; let assignment_role = assignment_role_for_spawn(cfg, name, &role); + let access = codex_access_for_spawn(cfg, name, &role); require_command("tmux")?; let cli = &cfg.subagent_cli; @@ -947,9 +1159,12 @@ fn spawn(cfg: &RuntimeConfig, args: &[String]) -> Result<(), String> { fs::create_dir_all(&cfg.logs).map_err(io_error("create subagent log directory"))?; let executable = env::current_exe().map_err(io_error("resolve multiagent executable"))?; let metadata = format!( - "name={name}\nsession={}\nroot={}\nwrite_policy={}\nlog_file={}\ncli={cli}\ncli_bin={binary}\nhelper={}\ncreated_at={}\n", + "name={name}\nsession={}\nroot={}\nrole={}\ncodex_access={}\nworkflow_id={}\nwrite_policy={}\nlog_file={}\ncli={cli}\ncli_bin={binary}\nhelper={}\ncreated_at={}\n", cfg.session, cfg.root.display(), + if role.is_empty() { assignment_role } else { &role }, + access.sandbox(), + env_nonempty("MULTIAGENT_WORKFLOW_ID").unwrap_or_default(), cfg.policy.display(), cfg.logs.join(format!("{name}.log")).display(), executable.display(), @@ -970,16 +1185,38 @@ fn spawn(cfg: &RuntimeConfig, args: &[String]) -> Result<(), String> { )?; prompt_file = Some(path); } - let cli_command = build_cli_command( - cli, - &cfg.root, - prompt_file.as_deref(), - Some(&output_file), - &cfg.codex_bin, - &cfg.claude_bin, - cfg.code_exec, - )?; - let command = subagent_shell_command(cfg, name, cli, &executable, &cli_command, false); + let cli_command = if env::var("MULTIAGENT_UID_SANDBOX").as_deref() == Ok("1") { + if cli != "codex" || !cfg.code_exec { + return Err("UID role isolation requires codex exec subagents".into()); + } + format!( + "{} role-agent-exec {}", + shell_escape(&executable.display().to_string()), + shell_escape(name) + ) + } else { + let command = build_cli_command( + cli, + &cfg.root, + prompt_file.as_deref(), + Some(&output_file), + &cfg.codex_bin, + &cfg.claude_bin, + cfg.code_exec, + access, + )?; + wrap_linux_role_sandbox( + &command, + &executable, + role_write_roots(&cfg.root, &cfg.state, access == CodexAccess::WorkspaceWrite), + if access == CodexAccess::WorkspaceWrite { + WRITER_UID + } else { + READER_UID + }, + ) + }; + let command = subagent_shell_command(cfg, name, cli, &executable, &cli_command, access, false); tmux_checked(&["new-window", "-d", "-t", &cfg.session, "-n", name, &command])?; pipe_log(&cfg.session, name, &cfg.logs)?; set_subagent_status(cfg, name, "running")?; @@ -1035,6 +1272,63 @@ fn poll(cfg: &RuntimeConfig, name: &str, report: bool) -> Result<(), String> { } } +fn wait(cfg: &RuntimeConfig, args: &[String]) -> Result<(), String> { + let name = args + .first() + .filter(|value| !value.is_empty()) + .ok_or_else(|| "wait requires NAME".to_string())?; + validate_name(name)?; + let mut timeout = 900.0f64; + let mut interval = 1.0f64; + let mut index = 1; + while index < args.len() { + match args[index].as_str() { + "--timeout" => { + timeout = required_value(args, index, "wait --timeout")? + .parse() + .map_err(|_| "wait --timeout must be a non-negative number".to_string())?; + index += 2; + } + "--poll-interval" => { + interval = required_value(args, index, "wait --poll-interval")? + .parse() + .map_err(|_| { + "wait --poll-interval must be a non-negative number".to_string() + })?; + index += 2; + } + other => return Err(format!("unknown wait argument: {other}")), + } + } + if !timeout.is_finite() || timeout < 0.0 { + return Err("wait --timeout must be a non-negative number".into()); + } + if !interval.is_finite() || interval < 0.0 { + return Err("wait --poll-interval must be a non-negative number".into()); + } + + let deadline = Instant::now() + Duration::from_secs_f64(timeout); + loop { + poll(cfg, name, false)?; + let status = read_trimmed(&cfg.state.join("subagents").join(name).join("status")) + .unwrap_or_else(|| "unknown".into()); + if matches!( + status.as_str(), + "done" | "failed" | "blocked" | "exited" | "finalized" | "killed" | "missing" + ) { + println!("{name}\t{status}"); + return Ok(()); + } + if Instant::now() >= deadline { + println!("{name}\t{status}"); + return Err(format!( + "timed out after {timeout} seconds waiting for subagent: {name}" + )); + } + thread::sleep(Duration::from_secs_f64(interval)); + } +} + fn inspect(cfg: &RuntimeConfig, args: &[String]) -> Result<(), String> { let name = args .first() @@ -1205,6 +1499,10 @@ fn restore(cfg: &RuntimeConfig, args: &[String]) -> Result<(), String> { .cloned() .unwrap_or_else(|| cfg.subagent_cli.clone()); validate_cli(&cli)?; + let access = match metadata.get("codex_access").map(String::as_str) { + Some("read-only") => CodexAccess::ReadOnly, + _ => CodexAccess::WorkspaceWrite, + }; let binary = cfg.cli_bin(&cli)?; require_command(binary)?; if !tmux_success(&["has-session", "-t", &cfg.session]) { @@ -1265,16 +1563,38 @@ fn restore(cfg: &RuntimeConfig, args: &[String]) -> Result<(), String> { None }; let executable = env::current_exe().map_err(io_error("resolve multiagent executable"))?; - let cli_command = build_cli_command( - &cli, - &cfg.root, - prompt_file.as_deref(), - Some(&output_file), - &cfg.codex_bin, - &cfg.claude_bin, - cfg.code_exec, - )?; - let command = subagent_shell_command(cfg, name, &cli, &executable, &cli_command, true); + let cli_command = if env::var("MULTIAGENT_UID_SANDBOX").as_deref() == Ok("1") { + if cli != "codex" || !cfg.code_exec { + return Err("UID role isolation requires codex exec subagents".into()); + } + format!( + "{} role-agent-exec {} --restore", + shell_escape(&executable.display().to_string()), + shell_escape(name) + ) + } else { + let command = build_cli_command( + &cli, + &cfg.root, + prompt_file.as_deref(), + Some(&output_file), + &cfg.codex_bin, + &cfg.claude_bin, + cfg.code_exec, + access, + )?; + wrap_linux_role_sandbox( + &command, + &executable, + role_write_roots(&cfg.root, &cfg.state, access == CodexAccess::WorkspaceWrite), + if access == CodexAccess::WorkspaceWrite { + WRITER_UID + } else { + READER_UID + }, + ) + }; + let command = subagent_shell_command(cfg, name, &cli, &executable, &cli_command, access, true); tmux_checked(&["new-window", "-d", "-t", &cfg.session, "-n", name, &command])?; pipe_log(&cfg.session, name, &cfg.logs)?; set_subagent_status(cfg, name, "running")?; @@ -1345,10 +1665,14 @@ fn finalize(cfg: &RuntimeConfig, args: &[String]) -> Result<(), String> { fn kill(cfg: &RuntimeConfig, args: &[String]) -> Result<(), String> { let name = one_name("kill", args)?; require_command("tmux")?; + let supervisor_pid = read_supervisor_pid(cfg, name); if window_exists(&cfg.session, name) { let _ = capture_subagent(cfg, name); tmux_checked(&["kill-window", "-t", &format!("{}:{name}", cfg.session)])?; } + if let Some(pid) = supervisor_pid { + wait_for_process_exit(pid, name)?; + } set_subagent_status(cfg, name, "killed")?; if cfg .state @@ -1363,6 +1687,36 @@ fn kill(cfg: &RuntimeConfig, args: &[String]) -> Result<(), String> { Ok(()) } +fn read_supervisor_pid(cfg: &RuntimeConfig, name: &str) -> Option { + read_trimmed( + &cfg.state + .join("subagents") + .join(name) + .join("supervisor.pid"), + ) + .and_then(|value| value.parse().ok()) +} + +#[cfg(unix)] +fn wait_for_process_exit(pid: u32, name: &str) -> Result<(), String> { + for _ in 0..100 { + let alive = unsafe { libc::kill(pid as libc::pid_t, 0) } == 0 + || std::io::Error::last_os_error().raw_os_error() == Some(libc::EPERM); + if !alive { + return Ok(()); + } + thread::sleep(Duration::from_millis(50)); + } + Err(format!( + "subagent process did not exit after pane close: {name} pid={pid}" + )) +} + +#[cfg(not(unix))] +fn wait_for_process_exit(_pid: u32, _name: &str) -> Result<(), String> { + Ok(()) +} + fn compose_role_instruction( cfg: &RuntimeConfig, name: &str, @@ -1424,6 +1778,35 @@ fn assignment_role_for_spawn<'a>(cfg: &RuntimeConfig, name: &str, role: &'a str) } } +fn codex_access_for_spawn(cfg: &RuntimeConfig, name: &str, role: &str) -> CodexAccess { + let lower = name.to_ascii_lowercase(); + let prompt = role_prompt_path(cfg, name, role).and_then(|path| { + path.file_name() + .map(|value| value.to_string_lossy().to_string()) + }); + if role == "reviewer" + || role == "scout" + || lower.contains("decision-authority-reviewer") + || matches!( + prompt.as_deref(), + Some( + "acceptance-scout.md" + | "contract-scout.md" + | "decision-authority-reviewer.md" + | "scope-guard.md" + | "validation-coordinator.md" + ) + ) + { + CodexAccess::ReadOnly + } else { + // Workers need source writes. Technical/build verifiers retain workspace + // writes because repository-local compilers and test runners commonly + // create build artifacts; their role prompt still forbids source edits. + CodexAccess::WorkspaceWrite + } +} + fn append_verifier_diff_binding( cfg: &RuntimeConfig, name: &str, @@ -1482,7 +1865,7 @@ fn validate_implementation_context( } fn implementation_context(cfg: &RuntimeConfig, name: &str) -> Result, String> { - if env::var("MULTIAGENT_LIFECYCLE_ENFORCEMENT").as_deref() != Ok("1") { + if !config::lifecycle_enforced() { return Ok(None); } let meta_path = cfg @@ -1576,6 +1959,7 @@ fn reject_parallel_generic_worker_spawn(cfg: &RuntimeConfig, name: &str) -> Resu Ok(()) } +#[allow(clippy::too_many_arguments)] fn build_cli_command( cli: &str, cwd: &Path, @@ -1584,13 +1968,15 @@ fn build_cli_command( codex_bin: &str, claude_bin: &str, codex_exec: bool, + access: CodexAccess, ) -> Result { match cli { "codex" if codex_exec => { let mut command = format!( - "{} exec --cd {} --skip-git-repo-check --dangerously-bypass-approvals-and-sandbox", + "{} exec --cd {} --skip-git-repo-check {}", shell_escape(codex_bin), - shell_escape(&cwd.display().to_string()) + shell_escape(&cwd.display().to_string()), + codex_safety_args(access, true), ); if let Some(path) = output { command.push_str(&format!( @@ -1608,9 +1994,10 @@ fn build_cli_command( } "codex" => { let mut command = format!( - "{} --cd {} --dangerously-bypass-approvals-and-sandbox --no-alt-screen", + "{} --cd {} {} --no-alt-screen", shell_escape(codex_bin), - shell_escape(&cwd.display().to_string()) + shell_escape(&cwd.display().to_string()), + codex_safety_args(access, false), ); if let Some(path) = prompt { command.push_str(&format!( @@ -1639,16 +2026,143 @@ fn build_cli_command( } } +#[cfg(target_os = "linux")] +fn codex_safety_args(_access: CodexAccess, _exec: bool) -> String { + // Docker's default seccomp profile blocks the user namespaces required by + // Codex/bubblewrap. The enclosing role-exec Landlock boundary is inherited + // by Codex and every model-generated child process, so Codex itself must not + // attempt a second sandbox. + "--dangerously-bypass-approvals-and-sandbox".into() +} + +#[cfg(not(target_os = "linux"))] +fn codex_safety_args(access: CodexAccess, exec: bool) -> String { + if exec { + format!("--sandbox {} -c approval_policy=never", access.sandbox()) + } else { + format!("--sandbox {} --ask-for-approval never", access.sandbox()) + } +} + +fn role_write_roots(root: &Path, state: &Path, include_source: bool) -> Vec { + let mut paths = BTreeSet::from([state.to_path_buf()]); + if include_source { + paths.insert(root.to_path_buf()); + } + for key in [ + "CODEX_HOME", + "GOCACHE", + "GOMODCACHE", + "MULTIAGENT_ROLE_SHARED_WRITE_DIR", + ] { + if let Some(path) = env_path(key) { + if path.exists() { + paths.insert(path); + } + } + } + for path in [PathBuf::from("/dev/null"), PathBuf::from("/dev/tty")] { + if path.exists() { + paths.insert(path); + } + } + paths.into_iter().collect() +} + +#[cfg(target_os = "linux")] +fn wrap_linux_role_sandbox( + command: &str, + executable: &Path, + write_roots: Vec, + _uid: u32, +) -> String { + let allowances = write_roots + .into_iter() + .map(|path| { + format!( + "--allow-write {}", + shell_escape(&path.display().to_string()) + ) + }) + .collect::>() + .join(" "); + format!( + "{} role-exec {allowances} -- /bin/sh -c {}", + shell_escape(&executable.display().to_string()), + shell_escape(command) + ) +} + +#[cfg(target_os = "linux")] +fn prepare_uid_state_permissions(state: &Path) -> Result<(), String> { + use std::os::unix::fs::PermissionsExt; + + fn visit(path: &Path) -> Result<(), String> { + let metadata = fs::symlink_metadata(path).map_err(io_error("inspect uid sandbox path"))?; + chown_path(path, ORCHESTRATOR_UID, ROLE_GID)?; + if metadata.is_dir() { + fs::set_permissions(path, fs::Permissions::from_mode(0o2770)) + .map_err(io_error("set uid sandbox directory permissions"))?; + for entry in fs::read_dir(path).map_err(io_error("read uid sandbox directory"))? { + visit(&entry.map_err(io_error("read uid sandbox entry"))?.path())?; + } + } else if metadata.is_file() { + let executable = metadata.permissions().mode() & 0o111 != 0; + fs::set_permissions( + path, + fs::Permissions::from_mode(if executable { 0o770 } else { 0o660 }), + ) + .map_err(io_error("set uid sandbox file permissions"))?; + } + Ok(()) + } + + visit(state) +} + +#[cfg(not(target_os = "linux"))] +fn prepare_uid_state_permissions(_state: &Path) -> Result<(), String> { + Err("MULTIAGENT_UID_SANDBOX is only supported on Linux".into()) +} + +#[cfg(target_os = "linux")] +fn chown_path(path: &Path, uid: u32, gid: u32) -> Result<(), String> { + use std::ffi::CString; + use std::os::unix::ffi::OsStrExt; + + let raw = CString::new(path.as_os_str().as_bytes()) + .map_err(|_| format!("path contains NUL: {}", path.display()))?; + if unsafe { libc::lchown(raw.as_ptr(), uid, gid) } != 0 { + return Err(format!( + "chown {}: {}", + path.display(), + std::io::Error::last_os_error() + )); + } + Ok(()) +} + +#[cfg(not(target_os = "linux"))] +fn wrap_linux_role_sandbox( + command: &str, + _executable: &Path, + _write_roots: Vec, + _uid: u32, +) -> String { + command.into() +} + fn subagent_shell_command( cfg: &RuntimeConfig, name: &str, cli: &str, executable: &Path, cli_command: &str, + access: CodexAccess, restored: bool, ) -> String { let workflow_id = env_nonempty("MULTIAGENT_WORKFLOW_ID").unwrap_or_default(); - let lifecycle = env_nonempty("MULTIAGENT_LIFECYCLE_ENFORCEMENT").unwrap_or_else(|| "0".into()); + let lifecycle = u8::from(config::lifecycle_enforced()).to_string(); let path = env::var("PATH").unwrap_or_default(); let mut values = vec![ ("MULTIAGENT_SESSION", cfg.session.clone()), @@ -1671,13 +2185,25 @@ fn subagent_shell_command( if restored { values.push(("MULTIAGENT_SUBAGENT_RESTORED", "1".into())); } + if env::var("MULTIAGENT_UID_SANDBOX").as_deref() == Ok("1") { + if let Some(root) = env_nonempty("MULTIAGENT_CODEX_HOME_ROOT") { + let role = if access == CodexAccess::WorkspaceWrite { + "writer" + } else { + "reader" + }; + let home = Path::new(&root).join(role).display().to_string(); + values.push(("CODEX_HOME", home.clone())); + values.push(("HOME", home)); + } + } let exports = values .into_iter() .map(|(key, value)| format!("{key}={}", shell_escape(&value))) .collect::>() .join(" "); format!( - "cd {} && export {exports} && {cli_command}; rc=$?; printf '\\nfinal status: codex exec exited rc=%s\\n' $rc; sleep infinity", + "cd {} && umask 0007 && export {exports} && {cli_command}; rc=$?; printf '\\nfinal status: codex exec exited rc=%s\\n' $rc; sleep infinity", shell_escape(&cfg.root.display().to_string()) ) } @@ -1872,14 +2398,32 @@ fn infer_status(cfg: &RuntimeConfig, name: &str) -> String { fn accepted_report(text: &str) -> bool { text.lines().any(|line| { - let value = line.trim().to_ascii_lowercase(); + let value = normalize_report_line(line).to_ascii_lowercase(); value == "accepted" || value.starts_with("accepted ") || value.starts_with("verdict: accepted") || value.starts_with("verdict=accepted") + || (value.starts_with("review-record: type=") && value.contains(" verdict=pass diff=")) }) } +fn normalize_report_line(line: &str) -> &str { + let mut value = line.trim(); + if let Some((prefix, rest)) = value.split_once(' ') { + let numbered = prefix + .strip_suffix('.') + .or_else(|| prefix.strip_suffix(')')) + .is_some_and(|number| !number.is_empty() && number.chars().all(|c| c.is_ascii_digit())); + if numbered || matches!(prefix, "-" | "*") { + value = rest.trim_start(); + } + } + if value.starts_with('`') && value.ends_with('`') && value.len() >= 2 { + value = &value[1..value.len() - 1]; + } + value +} + fn looks_blocked_report(text: &str) -> bool { text.lines().any(|line| { let line = line.trim().to_ascii_lowercase(); @@ -2003,6 +2547,9 @@ fn capture_window(session: &str, name: &str, lines: usize) -> Result Result<(), String> { + #[cfg(target_os = "linux")] + use std::os::unix::fs::PermissionsExt; + fs::create_dir_all(logs).map_err(io_error("create log directory"))?; let log = logs.join(format!("{window}.log")); OpenOptions::new() @@ -2010,6 +2557,12 @@ fn pipe_log(session: &str, window: &str, logs: &Path) -> Result<(), String> { .append(true) .open(&log) .map_err(io_error("create pane log"))?; + #[cfg(target_os = "linux")] + if env::var("MULTIAGENT_UID_SANDBOX").as_deref() == Ok("1") && unsafe { libc::geteuid() } == 0 { + chown_path(&log, ORCHESTRATOR_UID, ROLE_GID)?; + fs::set_permissions(&log, fs::Permissions::from_mode(0o660)) + .map_err(io_error("set orchestrator pane log permissions"))?; + } tmux_checked(&[ "pipe-pane", "-o", @@ -2020,19 +2573,61 @@ fn pipe_log(session: &str, window: &str, logs: &Path) -> Result<(), String> { } fn tmux_success(args: &[&str]) -> bool { - Command::new("tmux") + tmux_command() .args(args) - .status() - .is_ok_and(|status| status.success()) + .output() + .is_ok_and(|output| output.status.success()) } fn tmux_output(args: &[&str]) -> Result { - Command::new("tmux") + tmux_command() .args(args) .output() .map_err(io_error("run tmux")) } +fn tmux_command() -> Command { + let mut command = Command::new("tmux"); + if let Some(socket) = env_path("MULTIAGENT_TMUX_SOCKET") { + command.arg("-S").arg(socket); + } + command +} + +#[cfg(target_os = "linux")] +fn tmux_checked_as_uid(args: &[&str], executable: &Path, uid: u32) -> Result<(), String> { + let mut command = Command::new(executable); + command + .arg("role-exec") + .arg("--uid") + .arg(uid.to_string()) + .arg("--gid") + .arg(ROLE_GID.to_string()) + .arg("--") + .arg("tmux"); + if let Some(socket) = env_path("MULTIAGENT_TMUX_SOCKET") { + command.arg("-S").arg(socket); + } + let output = command + .args(args) + .output() + .map_err(io_error("run tmux as orchestrator"))?; + if output.status.success() { + Ok(()) + } else { + Err(format!( + "tmux {} as orchestrator failed: {}", + args.join(" "), + String::from_utf8_lossy(&output.stderr).trim() + )) + } +} + +#[cfg(not(target_os = "linux"))] +fn tmux_checked_as_uid(_args: &[&str], _executable: &Path, _uid: u32) -> Result<(), String> { + Err("MULTIAGENT_UID_SANDBOX is only supported on Linux".into()) +} + fn tmux_checked(args: &[&str]) -> Result<(), String> { let output = tmux_output(args)?; if output.status.success() { @@ -2405,4 +3000,14 @@ mod tests { assert_eq!(classify_capture("assignment complete"), "done"); assert_eq!(classify_capture("working"), "busy"); } + + #[test] + fn reviewer_pass_marker_is_accepted_evidence() { + let report = "authority-findings: blocked sets remain unchanged\n\ +review-record: type=decision-authority verdict=pass diff=-\n"; + assert!(accepted_report(report)); + assert!(accepted_report( + "3. `review-record: type=scope verdict=pass diff=abc`" + )); + } } diff --git a/src/subagent.rs b/src/subagent.rs index a911b5a..3fd890d 100644 --- a/src/subagent.rs +++ b/src/subagent.rs @@ -468,7 +468,7 @@ fn assignment_create(args: &[String]) -> Result<(), String> { reject_overlap(&assignments, &options.name, &options.role, &owned)?; let start_commit = resolve_commit(&root, &options.start_commit)?; - let lifecycle_enforced = env::var("MULTIAGENT_LIFECYCLE_ENFORCEMENT").as_deref() == Ok("1"); + let lifecycle_enforced = config::lifecycle_enforced(); let workflow_id = if lifecycle_enforced && options.role == "exploitation" && options.workflow_id.is_empty() { env::var("MULTIAGENT_WORKFLOW_ID").unwrap_or_default() @@ -978,7 +978,7 @@ fn verifier_dirs(state: &Path) -> Result, String> { .and_then(|value| value.to_str()) .unwrap_or("") .to_ascii_lowercase(); - name.contains("verifier") || name.contains("review") + name.contains("verifier") && !name.contains("build-verifier") }) .collect()) } diff --git a/src/workflow.rs b/src/workflow.rs index f56b2fa..c894913 100644 --- a/src/workflow.rs +++ b/src/workflow.rs @@ -41,7 +41,8 @@ const ENV_ORDER: &[&str] = &[ "updated_at", ]; const TODO_HEADER: &str = "todo_id\tkind\tsummary\torigin\tstatus\tassignment_id\tresolution\treason_code\treason\tevidence\tauthority\tdestination\tresume_condition\titeration\tupdated_at"; -const REVIEW_HEADER: &str = "review_id\ttype\tverdict\tdiff_hash\tevidence\titeration\trecorded_at"; +const REVIEW_HEADER: &str = + "review_id\ttype\tverdict\tdiff_hash\tevidence\titeration\trecorded_at\treviewer"; const USAGE: &str = r#"Usage: multiagent workflow init WORKFLOW_ID @@ -52,7 +53,7 @@ const USAGE: &str = r#"Usage: multiagent workflow add-todo WORKFLOW_ID TODO_ID --kind KIND --summary TEXT [--origin TEXT] multiagent workflow todo-status WORKFLOW_ID TODO_ID STATUS [--assignment-id ID] multiagent workflow resolve-todo WORKFLOW_ID TODO_ID --resolution STATUS --evidence TEXT [OPTIONS] - multiagent workflow record-review WORKFLOW_ID REVIEW_ID --type TYPE --verdict VERDICT [--diff-hash HASH] --evidence TEXT + multiagent workflow record-review WORKFLOW_ID REVIEW_ID --type TYPE --verdict VERDICT [--diff-hash HASH] --evidence TEXT [--reviewer NAME] multiagent workflow gate WORKFLOW_ID implementation|completion [--decision-id ID] [--plan-id ID] multiagent workflow completion-check WORKFLOW_ID multiagent workflow value WORKFLOW_ID KEY"#; @@ -171,7 +172,7 @@ impl Todo { #[derive(Clone)] struct Review { - fields: [String; 7], + fields: [String; 8], } impl Review { fn parse(line: &str) -> Self { @@ -607,6 +608,7 @@ fn record_review(args: &[String]) -> Result<(), String> { let kind = required(&o, "--type")?; let verdict = required(&o, "--verdict")?; let evidence = required(&o, "--evidence")?; + let reviewer = opt(&o, "--reviewer"); let requested_diff = opt(&o, "--diff-hash"); if !REVIEW_TYPES.contains(&kind) { return Err(format!("invalid review type: {kind}")); @@ -634,6 +636,12 @@ fn record_review(args: &[String]) -> Result<(), String> { } requested_diff }; + if reviewer_evidence_required() { + if reviewer.is_empty() { + return Err("reviewer-backed lifecycle requires --reviewer NAME".into()); + } + validate_reviewer_evidence(&store, reviewer, kind, verdict, diff)?; + } let mut rows = read_reviews(&p.reviews)?; if rows.iter().any(|r| r.get(0) == review_id) { return Err(format!("review already exists: {review_id}")); @@ -647,6 +655,7 @@ fn record_review(args: &[String]) -> Result<(), String> { evidence.into(), state_value(&state, "iteration").into(), timestamp(), + reviewer.into(), ], }); write_reviews(&p.reviews, &rows)?; @@ -788,6 +797,17 @@ fn completion_state(store: &Store, id: &str) -> Result, } let iteration = state_value(&state, "iteration"); let reviews = read_reviews(&p.reviews)?; + let findings: BTreeSet<&str> = reviews + .iter() + .filter(|r| r.get(5) == iteration && r.get(3) == diff && r.get(2) == "findings") + .map(|r| r.get(1)) + .collect(); + if !findings.is_empty() { + return Err(format!( + "completion blocked by current-diff review findings: {}", + findings.into_iter().collect::>().join(",") + )); + } let passed: BTreeSet<&str> = reviews .iter() .filter(|r| r.get(5) == iteration && r.get(3) == diff && r.get(2) == "pass") @@ -804,10 +824,186 @@ fn completion_state(store: &Store, id: &str) -> Result, missing.join(",") )); } + if reviewer_evidence_required() { + let unrecorded = unrecorded_reviewer_findings(store, id, diff, &reviews)?; + if !unrecorded.is_empty() { + return Err(format!( + "completion blocked by unrecorded reviewer findings: {}", + unrecorded.join(",") + )); + } + let unfinished = active_reviewers(store)?; + if !unfinished.is_empty() { + return Err(format!( + "completion blocked by active reviewers: {}", + unfinished.join(",") + )); + } + for review in reviews.iter().filter(|row| { + row.get(5) == iteration + && row.get(3) == diff + && row.get(2) == "pass" + && POST_REVIEWS.contains(&row.get(1)) + }) { + let reviewer = review.get(7); + if reviewer.is_empty() { + return Err(format!( + "passing {} review is missing durable reviewer evidence", + review.get(1) + )); + } + validate_reviewer_evidence(store, reviewer, review.get(1), "pass", diff)?; + } + } validate_context(&state)?; Ok(state) } +fn unrecorded_reviewer_findings( + store: &Store, + workflow_id: &str, + diff: &str, + reviews: &[Review], +) -> Result, String> { + let root = store.state_dir.join("subagents"); + if !root.is_dir() { + return Ok(Vec::new()); + } + let mut unrecorded = Vec::new(); + for entry in fs::read_dir(&root).map_err(io_error("read subagents directory"))? { + let dir = entry.map_err(io_error("read subagent entry"))?.path(); + if !dir.is_dir() { + continue; + } + let metadata = read_simple_env(&dir.join("meta.env"))?; + if state_value(&metadata, "role") != "reviewer" + || state_value(&metadata, "codex_access") != "read-only" + { + continue; + } + let reviewer_workflow = state_value(&metadata, "workflow_id"); + if !reviewer_workflow.is_empty() && reviewer_workflow != workflow_id { + continue; + } + let status = fs::read_to_string(dir.join("status")).unwrap_or_default(); + if status.trim() != "finalized" || !dir.join("finalized_at").is_file() { + continue; + } + let message = fs::read_to_string(dir.join("last-message.txt")).unwrap_or_default(); + let reviewer = dir + .file_name() + .and_then(|value| value.to_str()) + .unwrap_or("unknown"); + for kind in POST_REVIEWS { + let marker = format!("review-record: type={kind} verdict=findings diff={diff}"); + if !message + .lines() + .any(|line| review_marker_matches(line, &marker)) + { + continue; + } + let recorded = reviews.iter().any(|row| { + row.get(1) == *kind + && row.get(2) == "findings" + && row.get(3) == diff + && row.get(7) == reviewer + }); + if !recorded { + unrecorded.push(format!("{reviewer}:{kind}")); + } + } + } + unrecorded.sort(); + unrecorded.dedup(); + Ok(unrecorded) +} + +fn reviewer_evidence_required() -> bool { + config::lifecycle_enforced() +} + +fn validate_reviewer_evidence( + store: &Store, + reviewer: &str, + kind: &str, + verdict: &str, + diff: &str, +) -> Result<(), String> { + valid_id("reviewer name", reviewer)?; + let dir = store.state_dir.join("subagents").join(reviewer); + let metadata = read_simple_env(&dir.join("meta.env"))?; + if state_value(&metadata, "role") != "reviewer" + || state_value(&metadata, "codex_access") != "read-only" + { + return Err(format!( + "reviewer evidence must come from a read-only reviewer role: {reviewer}" + )); + } + let status = fs::read_to_string(dir.join("status")) + .map_err(|_| format!("reviewer status is missing: {reviewer}"))?; + if status.trim() != "finalized" || !dir.join("finalized_at").is_file() { + return Err(format!("reviewer is not finalized: {reviewer}")); + } + let message = fs::read_to_string(dir.join("last-message.txt")) + .map_err(|_| format!("reviewer final message is missing: {reviewer}"))?; + let marker = format!("review-record: type={kind} verdict={verdict} diff={diff}"); + if !message + .lines() + .any(|line| review_marker_matches(line, &marker)) + { + return Err(format!( + "reviewer {reviewer} final message is missing marker: {marker}" + )); + } + Ok(()) +} + +fn review_marker_matches(line: &str, marker: &str) -> bool { + let mut value = line.trim(); + if let Some((prefix, rest)) = value.split_once(' ') { + let numbered = prefix + .strip_suffix('.') + .or_else(|| prefix.strip_suffix(')')) + .is_some_and(|number| !number.is_empty() && number.chars().all(|c| c.is_ascii_digit())); + if numbered || matches!(prefix, "-" | "*") { + value = rest.trim_start(); + } + } + if value.starts_with('`') && value.ends_with('`') && value.len() >= 2 { + value = &value[1..value.len() - 1]; + } + value == marker +} + +fn active_reviewers(store: &Store) -> Result, String> { + let root = store.state_dir.join("subagents"); + if !root.is_dir() { + return Ok(Vec::new()); + } + let mut active = Vec::new(); + for entry in fs::read_dir(&root).map_err(io_error("read subagents directory"))? { + let dir = entry.map_err(io_error("read subagent entry"))?.path(); + if !dir.is_dir() { + continue; + } + let metadata = read_simple_env(&dir.join("meta.env"))?; + if state_value(&metadata, "role") != "reviewer" { + continue; + } + let status = fs::read_to_string(dir.join("status")).unwrap_or_default(); + if matches!(status.trim(), "starting" | "pending" | "running") { + active.push( + dir.file_name() + .and_then(|value| value.to_str()) + .unwrap_or("unknown") + .to_string(), + ); + } + } + active.sort(); + Ok(active) +} + fn validate_context(state: &BTreeMap) -> Result<(), String> { let text = state_value(state, "implementation_context"); if text.is_empty() { @@ -1046,4 +1242,16 @@ mod tests { }; assert_eq!(Todo::parse(&row.line()).fields, row.fields); } + + #[test] + fn review_markers_allow_only_cosmetic_markdown_wrapping() { + let marker = "review-record: type=scope verdict=pass diff=abc"; + assert!(review_marker_matches(marker, marker)); + assert!(review_marker_matches(&format!("3. `{marker}`"), marker)); + assert!(review_marker_matches(&format!("- `{marker}`"), marker)); + assert!(!review_marker_matches( + &format!("evidence includes {marker}"), + marker + )); + } } diff --git a/tests/lifecycle.sh b/tests/lifecycle.sh index 86c0afe..d3a5e73 100755 --- a/tests/lifecycle.sh +++ b/tests/lifecycle.sh @@ -2,7 +2,7 @@ set -euo pipefail FRAMEWORK_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd -P)" -MULTIAGENT="$FRAMEWORK_ROOT/target/debug/multiagent" +MULTIAGENT="${MULTIAGENT_BIN:-$FRAMEWORK_ROOT/target/debug/multiagent}" TEST_TMP="$(mktemp -d)" trap 'rm -rf "$TEST_TMP"' EXIT @@ -27,6 +27,19 @@ git -C "$TEST_REPO" add README.md git -C "$TEST_REPO" commit -q -m initial TEST_BRANCH="$(git -C "$TEST_REPO" branch --show-current)" +BYPASS_STATE="$TEST_TMP/bypass-state" +if MULTIAGENT_ROOT="$TEST_REPO" MULTIAGENT_STATE_DIR="$BYPASS_STATE" \ + MULTIAGENT_UID_SANDBOX=1 MULTIAGENT_LIFECYCLE_ENFORCEMENT=0 \ + "$MULTIAGENT" subagent assignment-create worker-bypass \ + --assignment-id BYPASS-1 --role exploitation \ + --branch "$TEST_BRANCH" --owned README.md \ + >"$TEST_TMP/lifecycle-env-bypass.out" 2>&1; then + echo "expected UID-isolated orchestrator to reject lifecycle environment bypass" >&2 + exit 1 +fi +assert_contains "$TEST_TMP/lifecycle-env-bypass.out" \ + "lifecycle enforcement requires --workflow-id" + IMPLEMENTATION_CONTEXT="$TEST_TMP/approved-implementation-context.md" printf '%s\n' \ '# Approved implementation context' \ @@ -48,6 +61,28 @@ wf() { } wf init WF-LIFECYCLE >/dev/null + +wf init WF-REVIEW-EVIDENCE >/dev/null +if MULTIAGENT_STATE_DIR="$TEST_STATE" MULTIAGENT_LIFECYCLE_ENFORCEMENT=1 \ + "$MULTIAGENT" workflow record-review WF-REVIEW-EVIDENCE AUTH-MISSING \ + --type decision-authority --verdict pass --evidence "claimed pass" \ + >"$TEST_TMP/missing-reviewer-evidence.out" 2>&1; then + echo "expected enforced review without reviewer evidence to fail" >&2 + exit 1 +fi +assert_contains "$TEST_TMP/missing-reviewer-evidence.out" "requires --reviewer NAME" +REVIEWER_STATE="$TEST_STATE/subagents/authority-reviewer-test" +mkdir -p "$REVIEWER_STATE" +printf '%s\n' 'role=reviewer' 'codex_access=read-only' >"$REVIEWER_STATE/meta.env" +printf 'finalized\n' >"$REVIEWER_STATE/status" +printf '2026-08-15T00:00:00Z\n' >"$REVIEWER_STATE/finalized_at" +printf 'review-record: type=decision-authority verdict=pass diff=-\n' \ + >"$REVIEWER_STATE/last-message.txt" +MULTIAGENT_STATE_DIR="$TEST_STATE" MULTIAGENT_LIFECYCLE_ENFORCEMENT=1 \ + "$MULTIAGENT" workflow record-review WF-REVIEW-EVIDENCE AUTH-DURABLE \ + --type decision-authority --verdict pass --evidence "durable reviewer pass" \ + --reviewer authority-reviewer-test >/dev/null + MULTIAGENT_STATE_DIR="$TEST_STATE" "$MULTIAGENT" decision init DEC-1 \ --title "Lifecycle decision" --owner orchestrator >/dev/null MULTIAGENT_STATE_DIR="$TEST_STATE" "$MULTIAGENT" decision add-alternative DEC-1 \ @@ -150,16 +185,67 @@ loop transition WF-LOOP post-implementation --diff-hash DIFF-FINAL >/dev/null loop resolve-todo WF-LOOP TODO-FOLLOWUP \ --resolution completed --evidence "repair and verifier recheck passed" >/dev/null for review_type in decision-drift scope technical reflection; do + if [[ "$review_type" == "technical" ]]; then + reviewer_name="verifier-technical" + else + reviewer_name="reviewer-$review_type" + fi + reviewer_state="$LOOP_STATE/subagents/$reviewer_name" + mkdir -p "$reviewer_state" + printf '%s\n' 'role=reviewer' 'codex_access=read-only' >"$reviewer_state/meta.env" + printf 'finalized\n' >"$reviewer_state/status" + printf '2026-08-15T00:00:00Z\n' >"$reviewer_state/finalized_at" + if [[ "$review_type" == "technical" ]]; then + printf 'ACCEPTED\nreview-record: type=technical verdict=pass diff=DIFF-FINAL\n' \ + >"$reviewer_state/last-message.txt" + else + printf 'review-record: type=%s verdict=pass diff=DIFF-FINAL\n' "$review_type" \ + >"$reviewer_state/last-message.txt" + fi loop record-review WF-LOOP "REVIEW-$review_type" \ --type "$review_type" --verdict pass --diff-hash DIFF-FINAL \ - --evidence "$review_type passed" >/dev/null + --evidence "$review_type passed" --reviewer "$reviewer_name" >/dev/null done +FINDINGS_STATE="$TEST_TMP/findings-state" +cp -R "$LOOP_STATE" "$FINDINGS_STATE" +UNRECORDED_REVIEWER="$FINDINGS_STATE/subagents/reviewer-unrecorded-findings" +mkdir -p "$UNRECORDED_REVIEWER" +printf '%s\n' 'role=reviewer' 'codex_access=read-only' 'workflow_id=WF-LOOP' \ + >"$UNRECORDED_REVIEWER/meta.env" +printf 'finalized\n' >"$UNRECORDED_REVIEWER/status" +printf '2026-08-15T00:00:00Z\n' >"$UNRECORDED_REVIEWER/finalized_at" +printf 'review-record: type=technical verdict=findings diff=DIFF-FINAL\n' \ + >"$UNRECORDED_REVIEWER/last-message.txt" +if MULTIAGENT_STATE_DIR="$FINDINGS_STATE" MULTIAGENT_LIFECYCLE_ENFORCEMENT=1 \ + "$MULTIAGENT" workflow completion-check WF-LOOP \ + >"$TEST_TMP/unrecorded-reviewer-findings.out" 2>&1; then + echo "expected unrecorded reviewer findings to block completion" >&2 + exit 1 +fi +assert_contains "$TEST_TMP/unrecorded-reviewer-findings.out" \ + "completion blocked by unrecorded reviewer findings: reviewer-unrecorded-findings:technical" +MULTIAGENT_STATE_DIR="$FINDINGS_STATE" MULTIAGENT_LIFECYCLE_ENFORCEMENT=1 \ + "$MULTIAGENT" workflow record-review WF-LOOP REVIEW-UNRESOLVED \ + --type technical --verdict findings --diff-hash DIFF-FINAL \ + --evidence "reviewer found a source defect" \ + --reviewer reviewer-unrecorded-findings >/dev/null +if MULTIAGENT_STATE_DIR="$FINDINGS_STATE" MULTIAGENT_LIFECYCLE_ENFORCEMENT=1 \ + "$MULTIAGENT" workflow completion-check WF-LOOP \ + >"$TEST_TMP/current-reviewer-findings.out" 2>&1; then + echo "expected recorded current-diff findings to block completion" >&2 + exit 1 +fi +assert_contains "$TEST_TMP/current-reviewer-findings.out" \ + "completion blocked by current-diff review findings: technical" loop completion-check WF-LOOP >/dev/null loop transition WF-LOOP complete >/dev/null -MULTIAGENT_ROOT="$TEST_REPO" MULTIAGENT_STATE_DIR="$LOOP_STATE" \ +if ! MULTIAGENT_ROOT="$TEST_REPO" MULTIAGENT_STATE_DIR="$LOOP_STATE" \ MULTIAGENT_WORKFLOW_ID=WF-LOOP MULTIAGENT_RUN_ID=RUN-LIFECYCLE \ MULTIAGENT_LIFECYCLE_ENFORCEMENT=1 \ - "$MULTIAGENT" orchestrator complete >"$TEST_TMP/complete.out" + "$MULTIAGENT" orchestrator complete >"$TEST_TMP/complete.out" 2>&1; then + cat "$TEST_TMP/complete.out" >&2 + exit 1 +fi assert_contains "$TEST_TMP/complete.out" $'run completed\tRUN-LIFECYCLE' echo "implementation lifecycle tests passed" diff --git a/tests/run.sh b/tests/run.sh index aabc458..c85ff4f 100755 --- a/tests/run.sh +++ b/tests/run.sh @@ -2,7 +2,9 @@ set -euo pipefail ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" -MULTIAGENT="$ROOT/target/debug/multiagent" +TARGET_DIR="${CARGO_TARGET_DIR:-$ROOT/target}" +MULTIAGENT="$TARGET_DIR/debug/multiagent" +HOST_KERNEL="$(uname -s)" cargo build --offline --locked --manifest-path "$ROOT/Cargo.toml" >/dev/null TMPDIR="$(mktemp -d)" TMPDIR="$(cd "$TMPDIR" && pwd -P)" @@ -254,7 +256,16 @@ assert_file_contains "$TMPDIR/launch.out" "Dashboard: MULTIAGENT_SESSION=launch- LAUNCH_BOOTSTRAP="$LAUNCH_STATE/orchestrator-bootstrap.sh" assert_file_contains "$MOCK_TMUX_LOG" "$(printf '%q' "$LAUNCH_BOOTSTRAP")" assert_file_contains "$MOCK_TMUX_LOG" "pipe-pane launch-cross-repo:orchestrator cat >> $LAUNCH_STATE/logs/orchestrator.log" -assert_file_contains "$LAUNCH_BOOTSTRAP" "--cd $LAUNCH_TARGET" +assert_file_contains "$LAUNCH_BOOTSTRAP" "--cd $LAUNCH_STATE" +if [[ "$HOST_KERNEL" == Linux ]]; then + assert_file_contains "$LAUNCH_BOOTSTRAP" "$MULTIAGENT role-exec" + assert_file_contains "$LAUNCH_BOOTSTRAP" "--allow-write $LAUNCH_STATE" + assert_file_not_contains "$LAUNCH_BOOTSTRAP" "--allow-write $LAUNCH_TARGET" + assert_file_contains "$LAUNCH_BOOTSTRAP" "--dangerously-bypass-approvals-and-sandbox" +else + assert_file_contains "$LAUNCH_BOOTSTRAP" "--sandbox workspace-write" + assert_file_not_contains "$LAUNCH_BOOTSTRAP" "--dangerously-bypass-approvals-and-sandbox" +fi assert_file_contains "$LAUNCH_BOOTSTRAP" "export MULTIAGENT_RESUME=0" assert_file_contains "$LAUNCH_BOOTSTRAP" "export MULTIAGENT_LOG_DIR=$LAUNCH_STATE/logs" assert_file_contains "$LAUNCH_BOOTSTRAP" "export MULTIAGENT_VERIFIER_MAX_ITERATIONS=3" @@ -640,6 +651,7 @@ assert_file_contains "$ROOT/orchestrator_prompt.md" "Do not inspect recovery sta assert_file_contains "$ROOT/orchestrator_prompt.md" 'When `MULTIAGENT_RESUME=1`' assert_file_contains "$ROOT/orchestrator_prompt.md" 'Only in that mode' assert_file_contains "$ROOT/orchestrator_prompt.md" 'MULTIAGENT_VERIFIER_MAX_ITERATIONS' +assert_file_contains "$ROOT/docs/control-plane-boundary.md" "preventing detached or late" assert_file_contains "$ROOT/orchestrator_prompt.md" 'SUBAGENT_CLI="$VERIFIER_CLI" multiagent subagent spawn' assert_file_contains "$ROOT/orchestrator_prompt.md" "Core Disciplines" assert_file_contains "$ROOT/orchestrator_prompt.md" "intent-contract.md" @@ -739,7 +751,8 @@ assert_file_contains "$ROOT/prompts/playbooks/agent-spawning.md" "todo-close" assert_file_contains "$ROOT/prompts/playbooks/agent-spawning.md" "gate-check" assert_file_contains "$ROOT/prompts/playbooks/agent-spawning.md" "required-path-outside-owned:" assert_file_contains "$ROOT/prompts/playbooks/agent-spawning.md" "ownership blocker" -assert_file_contains "$ROOT/prompts/playbooks/agent-spawning.md" 'WORKER_CLI="${WORKER_CLI:-claude}"' +assert_file_contains "$ROOT/prompts/playbooks/agent-spawning.md" 'SUBAGENT_CLI="$WORKER_CLI" multiagent subagent spawn' +assert_file_contains "$ROOT/prompts/playbooks/agent-spawning.md" "multiagent subagent wait worker-01-task" assert_file_contains "$ROOT/prompts/playbooks/orchestration-routing.md" "Orchestration Routing Playbook" assert_file_contains "$ROOT/prompts/playbooks/orchestration-routing.md" "Contract Scout Workflow" assert_file_contains "$ROOT/prompts/playbooks/orchestration-routing.md" "Scope Guard Workflow" @@ -1385,7 +1398,14 @@ assert_file_contains "$MULTIAGENT_STATE_DIR/subagents/verifier-01-docs/instructi assert_file_contains "$MOCK_TMUX_LOG" "send-key test-session:verifier-01-docs Read and follow the assignment in" verifier_spawn_line="$(grep -F "new-window -d test-session verifier-01-docs " "$MOCK_TMUX_LOG")" [[ "$verifier_spawn_line" == *"--cd $ROOT"* ]] -[[ "$verifier_spawn_line" == *"--dangerously-bypass-approvals-and-sandbox --no-alt-screen"* ]] +if [[ "$HOST_KERNEL" == Linux ]]; then + [[ "$verifier_spawn_line" == *"$MULTIAGENT role-exec"* ]] + [[ "$verifier_spawn_line" == *"--dangerously-bypass-approvals-and-sandbox"* ]] + [[ "$verifier_spawn_line" == *"--allow-write $ROOT"* ]] +else + [[ "$verifier_spawn_line" == *"--sandbox workspace-write --ask-for-approval never --no-alt-screen"* ]] + [[ "$verifier_spawn_line" != *"--dangerously-bypass-approvals-and-sandbox"* ]] +fi printf 'Codex prompt ready\n' >"$MOCK_TMUX_CAPTURES/verifier-owned-01.txt" SUBAGENT_CLI="$VERIFIER_CLI" "$MULTIAGENT" subagent spawn verifier-owned-01 \ @@ -1430,8 +1450,42 @@ assert_file_contains "$MULTIAGENT_STATE_DIR/subagents/codex-exec-protocol/instru assert_file_contains "$MULTIAGENT_STATE_DIR/subagents/codex-exec-protocol/instruction.txt" "Inspect /app" codex_exec_spawn_line="$(grep -F "new-window -d test-session codex-exec-protocol " "$MOCK_TMUX_LOG")" [[ "$codex_exec_spawn_line" == *"exec --cd $ROOT"* ]] +if [[ "$HOST_KERNEL" == Linux ]]; then + [[ "$codex_exec_spawn_line" == *"$MULTIAGENT role-exec"* ]] + [[ "$codex_exec_spawn_line" == *"--dangerously-bypass-approvals-and-sandbox"* ]] + [[ "$codex_exec_spawn_line" == *"--allow-write $ROOT"* ]] +else + [[ "$codex_exec_spawn_line" == *"--sandbox workspace-write -c approval_policy=never"* ]] +fi [[ "$codex_exec_spawn_line" == *"--output-last-message"* ]] +printf 'final status: codex exec exited rc=0\n' >"$MOCK_TMUX_CAPTURES/codex-exec-protocol.txt" +codex_wait_output="$(MULTIAGENT_CODEX_EXEC=1 SUBAGENT_CLI=codex "$MULTIAGENT" subagent wait codex-exec-protocol --timeout 1 --poll-interval 0)" +[[ "$codex_wait_output" == $'codex-exec-protocol\tdone' ]] + +printf 'Codex exec prompt ready\n' >"$MOCK_TMUX_CAPTURES/decision-authority-read-only.txt" +MULTIAGENT_CODEX_EXEC=1 SUBAGENT_CLI=codex "$MULTIAGENT" subagent spawn decision-authority-read-only \ + --role reviewer --instruction "Review the proposed authority" +authority_spawn_line="$(grep -F "new-window -d test-session decision-authority-read-only " "$MOCK_TMUX_LOG")" +[[ "$authority_spawn_line" == *"exec --cd $ROOT"* ]] +if [[ "$HOST_KERNEL" == Linux ]]; then + [[ "$authority_spawn_line" == *"$MULTIAGENT role-exec"* ]] + [[ "$authority_spawn_line" == *"--dangerously-bypass-approvals-and-sandbox"* ]] + [[ "$authority_spawn_line" != *"--allow-write $ROOT"* ]] +else + [[ "$authority_spawn_line" == *"--sandbox read-only -c approval_policy=never"* ]] +fi +assert_file_contains "$MULTIAGENT_STATE_DIR/subagents/decision-authority-read-only/meta.env" "role=reviewer" +assert_file_contains "$MULTIAGENT_STATE_DIR/subagents/decision-authority-read-only/meta.env" "codex_access=read-only" + +printf 'Progress update: still running\n' >"$MOCK_TMUX_CAPTURES/decision-authority-read-only.txt" +if MULTIAGENT_CODEX_EXEC=1 SUBAGENT_CLI=codex "$MULTIAGENT" subagent wait decision-authority-read-only --timeout 0 --poll-interval 0 >"$TMPDIR/authority-wait-timeout.out" 2>&1; then + echo "expected bounded subagent wait to time out for a running reviewer" >&2 + exit 1 +fi +assert_file_contains "$TMPDIR/authority-wait-timeout.out" $'decision-authority-read-only\trunning' +assert_file_contains "$TMPDIR/authority-wait-timeout.out" "timed out after 0 seconds" + printf 'Codex exec prompt ready\n' >"$MOCK_TMUX_CAPTURES/verifier-exec-role.txt" VERIFIER_DIFF_ROOT="$TMPDIR/verifier-diff-root" mkdir -p "$VERIFIER_DIFF_ROOT" diff --git a/tests/test_migration_contracts.py b/tests/test_migration_contracts.py index 99aa98f..c80eb28 100644 --- a/tests/test_migration_contracts.py +++ b/tests/test_migration_contracts.py @@ -20,7 +20,9 @@ PROJECT_ROOT = Path(__file__).resolve().parents[1] -MULTIAGENT = PROJECT_ROOT / "target" / "debug" / "multiagent" +MULTIAGENT = Path( + os.environ.get("MULTIAGENT_BIN", PROJECT_ROOT / "target" / "debug" / "multiagent") +) CLI_PREFIX = { "decision": ["decision"], "dag": ["dag"], diff --git a/tests/test_swe_outcomes.py b/tests/test_swe_outcomes.py index ca1b1f6..300e594 100644 --- a/tests/test_swe_outcomes.py +++ b/tests/test_swe_outcomes.py @@ -2,7 +2,10 @@ from __future__ import annotations +import base64 +import hashlib import json +import re import subprocess import sys import tempfile @@ -47,6 +50,24 @@ def _install_evalscope_stubs() -> None: class NativeOutcomeTest(unittest.TestCase): + def test_autonomous_authority_does_not_reopen_explicit_task_behavior(self): + root = Path(__file__).resolve().parents[1] + reviewer = (root / "prompts/roles/decision-authority-reviewer.md").read_text( + encoding="utf-8" + ) + lifecycle = (root / "prompts/playbooks/implementation-lifecycle.md").read_text( + encoding="utf-8" + ) + autonomous = ( + root / "evaluation/native_solver/templates/swe_autonomous_appendix.md" + ).read_text(encoding="utf-8") + + self.assertIn("original request is itself the user's decision", reviewer) + self.assertIn("at least two materially different", reviewer) + self.assertIn("explicit task contract is already approved", lifecycle) + self.assertIn("This run has no interactive user", autonomous) + self.assertIn("narrowest backward-compatible interpretation", autonomous) + def test_runner_has_no_submission_rejection_path(self): self.assertFalse(hasattr(evalscope_multiagent_native_runner, "is_submission_gate_rejection")) self.assertFalse(hasattr(evalscope_multiagent_native_runner.MultiagentNativeRunner, "_score_no_submission")) @@ -54,6 +75,56 @@ def test_runner_has_no_submission_rejection_path(self): hasattr(evalscope_multiagent_native_runner.MultiagentNativeRunner, "_collect_rejection_diagnostics") ) + def test_role_filesystem_seeds_private_codex_home_per_identity(self): + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + workdir = root / "repo" + workdir.mkdir() + (workdir / "source.py").write_text("value = 1\n", encoding="utf-8") + launcher = root / "multiagent" + launcher.write_text("binary", encoding="utf-8") + staging = root / "staging-home" + staging.mkdir() + (staging / "auth.json").write_text('{"token":"test"}', encoding="utf-8") + (staging / "config.toml").write_text("model = 'test'\n", encoding="utf-8") + role_homes = root / "role-homes" + runtime = root / "runtime" + + with mock.patch.multiple( + swe_prod_lifecycle, + CODEX_HOME=staging, + ROLE_CODEX_HOME_ROOT=role_homes, + RUNTIME_ROOT=runtime, + ): + with mock.patch.object(swe_prod_lifecycle.os, "chown"): + with mock.patch.object( + swe_prod_lifecycle.os, + "chmod", + wraps=swe_prod_lifecycle.os.chmod, + ) as chmod: + swe_prod_lifecycle.prepare_role_filesystem(workdir, launcher) + + for role in ("orchestrator", "writer", "reader"): + home = role_homes / role + self.assertEqual((home / "auth.json").read_text(encoding="utf-8"), '{"token":"test"}') + self.assertTrue((home / "config.toml").is_file()) + self.assertIn("directory =", (home / ".gitconfig").read_text(encoding="utf-8")) + self.assertEqual(home.stat().st_mode & 0o777, 0o700) + chmod.assert_any_call(launcher, 0o4755) + self.assertTrue( + all("follow_symlinks" not in call.kwargs for call in chmod.call_args_list), + "role filesystem setup must work on Python builds without chmod follow_symlinks support", + ) + + def test_runner_monitors_the_orchestrator_tmux_socket(self): + completed = SimpleNamespace(returncode=0, stdout="orchestrator\n", stderr="") + with mock.patch.object(swe_prod_lifecycle, "run", return_value=completed) as run: + self.assertTrue(swe_prod_lifecycle.tmux_has_session("session-1")) + self.assertTrue(swe_prod_lifecycle.tmux_has_orchestrator("session-1")) + + for call in run.call_args_list: + self.assertEqual(call.args[0][:3], ["tmux", "-S", str(swe_prod_lifecycle.TMUX_SOCKET)]) + def test_shard_problem_statement_uses_relative_sample_id(self): with tempfile.TemporaryDirectory() as directory: repo = Path(directory) @@ -98,6 +169,8 @@ def test_orchestrator_exit_prepares_workspace_for_official_scorer(self): "make_prompt": mock.Mock(return_value=prompt), "toolchain_path_prefixes": mock.Mock(return_value=[]), "ensure_cache_dir": mock.Mock(return_value=str(root)), + "prepare_role_filesystem": mock.DEFAULT, + "restore_workspace_owner": mock.DEFAULT, "tmux_has_session": mock.Mock(return_value=True), "tmux_has_orchestrator": mock.Mock(return_value=False), "materialize_committed_changes": mock.DEFAULT, @@ -121,10 +194,28 @@ def test_orchestrator_exit_prepares_workspace_for_official_scorer(self): result = swe_prod_lifecycle.run_prod_solver(None, root, root, 60) materialize = swe_prod_lifecycle.materialize_committed_changes expose_untracked = swe_prod_lifecycle.mark_untracked_intent_to_add + prepare_roles = swe_prod_lifecycle.prepare_role_filesystem + restore_owner = swe_prod_lifecycle.restore_workspace_owner + launch_env = next( + call.kwargs["env"] + for call in swe_prod_lifecycle.run.call_args_list + if call.kwargs.get("env") is not None + and call.args + and isinstance(call.args[0], list) + and call.args[0] + and str(call.args[0][0]).endswith("launch.sh") + ) self.assertEqual(result, 0) materialize.assert_called_once_with(root, "a" * 40) expose_untracked.assert_called_once_with(root) + prepare_roles.assert_called_once_with(root, Path("multiagent")) + restore_owner.assert_called_once_with(root) + self.assertEqual(launch_env["MULTIAGENT_UID_SANDBOX"], "1") + self.assertEqual( + launch_env["MULTIAGENT_CODEX_HOME_ROOT"], + str(swe_prod_lifecycle.ROLE_CODEX_HOME_ROOT), + ) def test_workspace_handoff_includes_new_source_and_test_files(self): with tempfile.TemporaryDirectory() as directory: @@ -227,9 +318,69 @@ def _summary_args(root: Path, work_dir: Path) -> SimpleNamespace: native_solver_source=Path(__file__).resolve().parents[1], native_codex_auth_json="", native_codex_auth_container_home="/root/.codex-multiagent-prod", + native_trace_dir=root / "traces", ) +class NativeTraceExportTest(unittest.IsolatedAsyncioTestCase): + async def test_runner_exports_hash_verified_trace_archive_per_official_row(self): + archive = (b"multiagent-trace\n" * 20000) + b"tail" + expected_digest = hashlib.sha256(archive).hexdigest() + + class FakeEnvironment: + def __init__(self): + self.commands = [] + + async def exec(self, cmd, **_kwargs): + script = cmd[-1] + self.commands.append(script) + if "tar -C" in script: + return SimpleNamespace( + returncode=0, + stdout=f"{len(archive)}\t{expected_digest}\n", + stderr="", + ) + match = re.search(r"skip=(\d+)", script) + if match: + index = int(match.group(1)) + start = index * evalscope_multiagent_native_runner._TRACE_CHUNK_BYTES + chunk = archive[start:start + evalscope_multiagent_native_runner._TRACE_CHUNK_BYTES] + return SimpleNamespace( + returncode=0, + stdout=base64.b64encode(chunk).decode("ascii") + "\n", + stderr="", + ) + raise AssertionError(f"unexpected command: {script}") + + with tempfile.TemporaryDirectory() as directory: + root = Path(directory).resolve() + runner = evalscope_multiagent_native_runner.MultiagentNativeRunner( + codex_auth_json=str(root / "auth.json"), + trace_output_dir=str(root / "traces"), + ) + environment = FakeEnvironment() + + exported = await runner._export_trace_bundle( + environment, + sample_id="2", + sample_index=7, + instance_id="instance_qutebrowser", + ) + + archive_path = root / "traces" / "official-row-000007" / "multiagent-trace.tar.gz" + manifest_path = archive_path.with_name("manifest.json") + manifest = json.loads(manifest_path.read_text(encoding="utf-8")) + + self.assertEqual(archive_path.read_bytes(), archive) + self.assertEqual(exported["path"], str(archive_path)) + self.assertEqual(exported["sha256"], expected_digest) + self.assertEqual(manifest["official_index"], 7) + self.assertEqual(manifest["instance_id"], "instance_qutebrowser") + self.assertEqual(manifest["archive_sha256"], expected_digest) + self.assertIn("/tmp/multiagent-prod-swe/state", environment.commands[0]) + self.assertNotIn("/app/.multiagent", environment.commands[0]) + + class AggregateOutcomeTest(unittest.TestCase): def test_parallel_refresh_aggregates_from_configured_report_directory(self): with tempfile.TemporaryDirectory() as directory: @@ -249,6 +400,40 @@ def test_parallel_refresh_aggregates_from_configured_report_directory(self): report_dir_index = command.index("--report-dir") self.assertEqual(command[report_dir_index + 1], str(report_dir)) + def test_parallel_worker_uses_shared_configured_trace_directory(self): + args = SimpleNamespace( + report_prefix_template="run-w{worker}-offset{offset}-count{count}", + work_root=Path("/tmp/work"), + report_dir=Path("/tmp/reports"), + swe_bench_pro_repo_path=Path("/tmp/swe"), + agent_model_name="gpt-5.4", + max_steps=250, + agent_timeout=3600, + native_solver_source=Path("/tmp/solver"), + native_codex_auth_json=Path("/tmp/auth.json"), + native_codex_auth_container_home="/root/.codex", + native_trace_dir=Path("/tmp/run-traces"), + on_demand_min_free_gb=50, + evalscope_path=None, + memory_limit="20g", + cpu_limit="", + persistent_cache=False, + persistent_cache_root=Path("/tmp/cache"), + persistent_cache_mode="rw", + workers=2, + ignore_errors=False, + ) + + command = swe_bench_pro_run_parallel_shards.build_worker_command( + args, + offset=5, + count=5, + worker_index=1, + ) + + trace_index = command.index("--native-trace-dir") + self.assertEqual(command[trace_index + 1], "/tmp/run-traces") + def test_default_discovery_accepts_custom_parallel_report_prefix(self): with tempfile.TemporaryDirectory() as directory: reports = Path(directory)