Enforce role-isolated workflows and persist SWE traces

docs/benchmark.md

@@ -174,6 +174,7 @@ python3 -m evaluation.swe_bench_pro \
   --config-yaml "$RUN_ROOT/config.yaml" \
   --preflight-output "$RUN_ROOT/preflight.json" \
   --on-demand-image-status "$RUN_ROOT/images.json" \
+  --native-trace-dir "$RUN_ROOT/traces" \
   --report-prefix "$RUN_ID" \
   --agent-model-name gpt-5.4 \
   --sample-offset 0 \
@@ -190,6 +191,21 @@ python3 -m evaluation.swe_bench_pro \
   --persistent-cache-mode rw
 ```
 
+The native runner exports each task container's multiagent state before the
+container closes. Traces are stored under
+`$RUN_ROOT/traces/official-row-NNNNNN/` as a hash-verified
+`multiagent-trace.tar.gz` plus `manifest.json`. The archive contains the
+orchestrator log, subagent transcripts, structured workflow/checkpoint state,
+runtime identity, and native runner stdout/stderr. It is written on successful,
+failed, and timed-out solver exits, remains outside `/app`, and is never part of
+the submitted patch or official scoring. Treat the raw archives as private
+artifacts because agent transcripts can contain source and environment details.
+
+Parallel shard runs accept the same `--native-trace-dir`; all workers write
+unique directories keyed by the absolute official row number. If the option is
+omitted, the single-run command uses `evaluation/reports/swe-bench-pro-traces`,
+while the parallel launcher uses `REPORT_DIR/traces`.
+
 After the run completes, capture a relocatable evidence bundle. The command
 fails if any source checkout is dirty, any row lacks official verifier/native
 outcome evidence, image identity is incomplete, runtime Codex/Node identity is

docs/control-plane-boundary.md

@@ -11,6 +11,8 @@ Rust owns production decisions and durable state:
 - write-policy checks and approvals;
 - assignments, checkpoints, and Git worktree metadata;
 - findings, repair TODOs, resolution and closure evidence;
+- durable reviewer findings, which cannot be replaced by a later pass on the
+  same candidate without first entering the repair loop;
 - validation leases and bounded validation subprocesses;
 - launch configuration, tmux subprocess orchestration, status, watch, and
   recovery behavior.
@@ -21,6 +23,23 @@ allocate or emulate a PTY; tmux continues to own terminal lifecycle and
 interactive process semantics. This keeps PTY behavior without preserving shell
 implementations.
 
+In the production Linux-container boundary, tmux runs as the read-only
+orchestrator UID. A raw tmux window therefore cannot acquire repository writes.
+Worker/reviewer transitions use the Rust binary's narrowly gated
+`role-agent-exec` entrypoint: it accepts only a persisted named Codex agent,
+validates the trusted bridge, and starts Codex in a dedicated process group
+under the role's UID. A minimal wait-only parent retains no workflow discretion;
+it exists solely to forward pane termination to the complete role process tree.
+`subagent kill` waits for that boundary to close, preventing detached or late
+worker output from modifying the workspace after cancellation. The setuid
+privilege gate drops privilege for every other command, including generic
+`role-exec`, so bypassing the high-level CLI cannot create an arbitrary writer
+shell. Lifecycle enforcement is also derived from the orchestrator's real UID,
+not solely from its mutable environment. Before the privileged bridge starts a
+writer it revalidates the assignment against the live workflow phase and
+approved implementation context; setting
+`MULTIAGENT_LIFECYCLE_ENFORCEMENT=0` cannot reopen a completed workflow.
+
 Python under `evaluation/` is limited to benchmark adapters, status readers,
 and provenance. SWE Bench adapters launch the production workflow and pass the
 current workspace diff to the official scorer. They neither derive a second

docs/getting-started.md

@@ -72,15 +72,28 @@ workers and generic named subagents too:
 ORCHESTRATOR_CLI=codex WORKER_CLI=codex SUBAGENT_CLI=codex ./launch.sh
 ```
 
-Codex launches with `--cd`, `--dangerously-bypass-approvals-and-sandbox`, and
-`--no-alt-screen`. Claude launches from the target worktree/root with
-`claude --dangerously-skip-permissions`; Codex-only flags are intentionally not
-passed to Claude.
-
-`--root` selects the target project repo for `MULTIAGENT_ROOT`, state, write
-policy, and the orchestrator CLI working directory. The default orchestrator
-prompt is still loaded from this launcher's directory, so cross-repo launches do
-not need an `orchestrator_prompt.md` in the target repo. Set
+The Rust supervisor assigns Codex access from trusted process roles. On hosts
+where Codex's native sandbox is available, the orchestrator starts in the
+durable state directory with `workspace-write`, workers start in the target
+repository with `workspace-write`, and scouts/authority reviewers use
+`read-only`. The production Linux-container adapter uses separate unprivileged
+Unix identities instead because nested bubblewrap is unavailable under Docker's
+default seccomp profile. Its tmux server runs as the non-writing orchestrator
+identity. A narrowly gated, setuid Rust entrypoint may only start the fixed
+Codex subagent command recorded for a named role; all other invocations
+permanently drop back to the caller UID. Each role also receives a private
+Codex runtime home so one role's private lock/config files cannot stall another.
+The isolated orchestrator's real UID makes lifecycle enforcement mandatory, so
+shell-level environment overrides cannot authorize a writer after completion.
+In both environments the orchestrator can read the target but cannot write it,
+while workers can. Claude remains a compatibility path and does not provide
+Codex's native role boundary outside the production adapter.
+
+`--root` selects the target project repo for `MULTIAGENT_ROOT`, state, and write
+policy. The orchestrator CLI works from the durable state directory and reads
+the target repository without write access. The default orchestrator prompt is
+still loaded from this launcher's directory, so cross-repo launches do not need
+an `orchestrator_prompt.md` in the target repo. Set
 `MULTIAGENT_PROMPT=/path/to/prompt.md` to override that default.
 
 ## System Flow
@@ -451,11 +464,19 @@ orchestrator/user decision:
 multiagent policy approve /tmp --actor orchestrator --assignment-id build-logs --reason "user approved shared temp output" --force
 ```
 
-Mechanical enforcement is limited to the helper's policy checks and startup
-visibility. Codex is still launched with
-`--dangerously-bypass-approvals-and-sandbox`, so shell sandboxing is not
-enforcing the boundary. The orchestrator and worker instructions require agents
-to check and follow the policy before writes.
+For Codex roles, the OS boundary mechanically prevents the orchestrator,
+authority reviewers, and scouts from writing the target repository. On native
+hosts that boundary is Codex's sandbox; in the production Linux container it is
+Unix ownership plus a permanent role UID drop. The tmux server itself has the
+orchestrator UID, so bypassing the Rust CLI to open a raw pane still produces a
+non-writing process. The only privileged transition is the fixed
+`role-agent-exec` path, which validates persisted role metadata and a
+root-owned, non-group-writable Codex bridge before dropping to the writer or
+reader UID. Generic `role-exec` calls from the orchestrator lose setuid
+privilege before dispatch. The write-policy helper remains responsible for
+explicit writes outside the normal role root. Claude
+compatibility processes do not receive this mechanical boundary on native
+hosts.
 
 ## Assignment Metadata and Acceptance
 
@@ -465,9 +486,11 @@ work starts:
 ```bash
 multiagent subagent assignment-create worker-01-docs \
   --assignment-id docs-001 \
-  --branch worker/docs-001 \
+  --branch "$(git rev-parse --abbrev-ref HEAD)" \
   --owned README.md,orchestrator_prompt.md
-multiagent subagent worktree-create worker-01-docs
+SUBAGENT_CLI="$WORKER_CLI" multiagent subagent spawn worker-01-docs \
+  --role worker --instruction-file /path/to/worker-instruction.md
+multiagent subagent wait worker-01-docs --timeout 1800
 multiagent subagent assignment-show worker-01-docs
 multiagent subagent assignment-status worker-01-docs running
 multiagent subagent checkpoint-update worker-01-docs --step "started implementation" --status running
@@ -531,7 +554,7 @@ Use `multiagent subagent` for named subagents that should keep working or monito
 ```bash
 multiagent subagent spawn subagent-ci-monitor --instruction "Monitor CI and report status changes."
 SUBAGENT_CLI=claude multiagent subagent spawn subagent-ci-monitor --instruction "Monitor CI and report status changes."
-multiagent subagent poll subagent-ci-monitor
+multiagent subagent wait subagent-ci-monitor --timeout 900
 multiagent subagent inspect subagent-ci-monitor --lines 160
 multiagent subagent recover-plan
 multiagent subagent restore subagent-ci-monitor

evaluation/evalscope_multiagent_native_runner.py

@@ -13,9 +13,12 @@
 from __future__ import annotations
 
 import base64
+import datetime as dt
+import hashlib
 import json
 import os
 import shlex
+import uuid
 from pathlib import Path
 from typing import Any, Dict
 
@@ -30,6 +33,8 @@
 _STDOUT_FILE = "/tmp/evalscope-native-multiagent-stdout.log"
 _STDERR_FILE = "/tmp/evalscope-native-multiagent-stderr.log"
 _RUNTIME_IDENTITY_FILE = "/tmp/multiagent-prod-swe/runtime-identity.json"
+_TRACE_ARCHIVE_FILE = "/tmp/evalscope-native-multiagent-trace.tar.gz"
+_TRACE_CHUNK_BYTES = 256 * 1024
 _DEFAULT_SOLVER_COMMAND = "/tmp/evalscope-native-multiagent-solver.sh"
 _PUBLIC_METADATA_KEYS = {
     "language",
@@ -94,19 +99,23 @@ def __init__(
         working_dir: str = "/app",
         model_name: str = "gpt-5",
         codex_auth_json: str = "",
-        codex_auth_container_home: str = "/root/.codex-multiagent-prod",
+        codex_auth_container_home: str = "/tmp/multiagent-prod-swe/codex-home",
         swe_bench_pro_repo_path: str = "",
         swe_bench_pro_sample_offset: int = 0,
+        trace_output_dir: str = "",
         **_: Any,
     ) -> None:
         self._working_dir = working_dir or "/app"
         self._model_name = model_name.strip() or "gpt-5"
         self._codex_auth_json = codex_auth_json.strip()
         if not self._codex_auth_json:
             raise ValueError("multiagent-native requires runtime Codex auth JSON")
-        self._codex_auth_container_home = codex_auth_container_home.rstrip("/") or "/root/.codex-multiagent-prod"
+        self._codex_auth_container_home = (
+            codex_auth_container_home.rstrip("/") or "/tmp/multiagent-prod-swe/codex-home"
+        )
         self._swe_bench_pro_repo_path = swe_bench_pro_repo_path.strip()
         self._swe_bench_pro_sample_offset = swe_bench_pro_sample_offset
+        self._trace_output_dir = Path(trace_output_dir).expanduser().resolve() if trace_output_dir.strip() else None
 
     async def setup(self, env: AgentEnvironment) -> None:
         await self._write_file(env, _DEFAULT_SOLVER_COMMAND, _SOLVER_LAUNCHER)
@@ -158,13 +167,30 @@ async def run(
             f"cwd={self._working_dir} command={command!r}"
         )
         runtime_identity: dict[str, Any] = {}
+        trace_export: dict[str, Any] = {}
+        trace_export_error: Exception | None = None
         try:
             result = await env.exec(["bash", "-lc", shell_command], timeout=task.timeout, env=env_vars, cwd=self._working_dir)
         finally:
             try:
                 runtime_identity = await self._read_json_file(env, _RUNTIME_IDENTITY_FILE)
             except Exception as exc:
                 logger.warning(f"multiagent-native could not read runtime identity: {exc!r}")
+            if self._trace_output_dir is not None:
+                try:
+                    trace_export = await self._export_trace_bundle(
+                        env,
+                        sample_id=sample_id,
+                        sample_index=sample_index,
+                        instance_id=raw_metadata.get("instance_id"),
+                    )
+                except Exception as exc:
+                    trace_export_error = exc
+                    logger.error(
+                        "multiagent-native could not export trace for official_index=%s: %r",
+                        sample_index,
+                        exc,
+                    )
             await self._scrub_codex_auth(env)
         logger.info(
             f"multiagent-native exited: sample={sample_id} rc={result.returncode} "
@@ -178,6 +204,11 @@ async def run(
         stderr = await env.exec(["bash", "-lc", f"tail -c 4000 {shlex.quote(_STDERR_FILE)} 2>/dev/null || true"])
         stdout_tail = (stdout.stdout or "")[-4000:]
         stderr_tail = (stderr.stdout or "")[-4000:]
+        if trace_export_error is not None:
+            raise RuntimeError(
+                f"multiagent-native could not export the configured trace for official_index={sample_index}: "
+                f"{trace_export_error}"
+            ) from trace_export_error
         if result.timed_out:
             raise RunnerTimeoutError(f"multiagent-native timed out after {task.timeout}s")
         elif result.returncode != 0:
@@ -191,9 +222,135 @@ async def run(
                 "timed_out": result.timed_out,
                 "stderr_tail": stderr_tail,
                 "runtime_identity": runtime_identity,
+                "trace_export": trace_export,
             },
         )
 
+    async def _export_trace_bundle(
+        self,
+        env: AgentEnvironment,
+        *,
+        sample_id: Any,
+        sample_index: int,
+        instance_id: Any,
+    ) -> dict[str, Any]:
+        """Copy the container-local multiagent trace into a host-side row archive."""
+
+        if self._trace_output_dir is None:
+            return {}
+
+        prepare_script = f"""
+set -euo pipefail
+stage=/tmp/evalscope-native-multiagent-trace-stage
+archive={shlex.quote(_TRACE_ARCHIVE_FILE)}
+rm -rf -- "$stage"
+mkdir -p "$stage/runner"
+if [[ -d /tmp/multiagent-prod-swe/state ]]; then
+  cp -a /tmp/multiagent-prod-swe/state "$stage/state"
+  rm -f -- "$stage/state/runtime_state/tmux.sock"
+else
+  printf 'multiagent state directory was not created\n' > "$stage/state-missing.txt"
+fi
+for source in {_STDOUT_FILE} {_STDERR_FILE} {_RUNTIME_IDENTITY_FILE}; do
+  if [[ -f "$source" ]]; then
+    cp -a "$source" "$stage/runner/$(basename "$source")"
+  fi
+done
+tar -C "$stage" -czf "$archive" .
+size=$(wc -c < "$archive" | tr -d '[:space:]')
+digest=$(sha256sum "$archive" | awk '{{print $1}}')
+printf '%s\t%s\n' "$size" "$digest"
+"""
+        prepared = await env.exec(["bash", "-lc", prepare_script], timeout=120)
+        if prepared.returncode != 0:
+            detail = ((prepared.stderr or "") + "\n" + (prepared.stdout or "")).strip()[-4000:]
+            raise RuntimeError(f"could not prepare container trace archive: {detail}")
+        fields = (prepared.stdout or "").strip().splitlines()[-1].split("\t")
+        if len(fields) != 2:
+            raise RuntimeError(f"container trace archive metadata is malformed: {prepared.stdout!r}")
+        try:
+            expected_size = int(fields[0])
+        except ValueError as exc:
+            raise RuntimeError(f"container trace archive size is invalid: {fields[0]!r}") from exc
+        expected_digest = fields[1].strip().lower()
+        if expected_size < 1 or len(expected_digest) != 64:
+            raise RuntimeError(
+                f"container trace archive metadata is invalid: size={expected_size} sha256={expected_digest!r}"
+            )
+
+        row_dir = self._trace_output_dir / f"official-row-{sample_index:06d}"
+        row_dir.mkdir(mode=0o700, parents=True, exist_ok=True)
+        archive_path = row_dir / "multiagent-trace.tar.gz"
+        temporary_path = row_dir / f".{archive_path.name}.{uuid.uuid4().hex}.tmp"
+        digest = hashlib.sha256()
+        written = 0
+        try:
+            with temporary_path.open("wb") as handle:
+                for chunk_index in range((expected_size + _TRACE_CHUNK_BYTES - 1) // _TRACE_CHUNK_BYTES):
+                    chunk_script = (
+                        "set -o pipefail; "
+                        f"dd if={shlex.quote(_TRACE_ARCHIVE_FILE)} bs={_TRACE_CHUNK_BYTES} "
+                        f"skip={chunk_index} count=1 status=none | base64"
+                    )
+                    chunk_result = await env.exec(["bash", "-lc", chunk_script], timeout=120)
+                    if chunk_result.returncode != 0:
+                        detail = ((chunk_result.stderr or "") + "\n" + (chunk_result.stdout or "")).strip()[-2000:]
+                        raise RuntimeError(f"could not read trace archive chunk {chunk_index}: {detail}")
+                    try:
+                        chunk = base64.b64decode((chunk_result.stdout or "").encode("ascii"), validate=False)
+                    except (UnicodeEncodeError, ValueError) as exc:
+                        raise RuntimeError(f"trace archive chunk {chunk_index} is not valid base64") from exc
+                    if not chunk:
+                        raise RuntimeError(f"trace archive chunk {chunk_index} is empty")
+                    handle.write(chunk)
+                    digest.update(chunk)
+                    written += len(chunk)
+            if written != expected_size:
+                raise RuntimeError(f"trace archive size mismatch: expected {expected_size}, copied {written}")
+            actual_digest = digest.hexdigest()
+            if actual_digest != expected_digest:
+                raise RuntimeError(
+                    f"trace archive digest mismatch: expected {expected_digest}, copied {actual_digest}"
+                )
+            temporary_path.chmod(0o600)
+            temporary_path.replace(archive_path)
+        finally:
+            temporary_path.unlink(missing_ok=True)
+
+        manifest = {
+            "captured_at": dt.datetime.now(dt.timezone.utc).isoformat(timespec="seconds"),
+            "official_index": sample_index,
+            "sample_id": None if sample_id is None else str(sample_id),
+            "instance_id": None if instance_id is None else str(instance_id),
+            "archive": archive_path.name,
+            "archive_bytes": expected_size,
+            "archive_sha256": expected_digest,
+            "container_state_dir": "/tmp/multiagent-prod-swe/state",
+            "submission_workspace": self._working_dir,
+        }
+        manifest_path = row_dir / "manifest.json"
+        manifest_tmp = row_dir / f".{manifest_path.name}.{uuid.uuid4().hex}.tmp"
+        try:
+            manifest_tmp.write_text(json.dumps(manifest, indent=2) + "\n", encoding="utf-8")
+            manifest_tmp.chmod(0o600)
+            manifest_tmp.replace(manifest_path)
+        finally:
+            manifest_tmp.unlink(missing_ok=True)
+
+        logger.info(
+            "multiagent-native trace exported: official_index=%s path=%s bytes=%s sha256=%s",
+            sample_index,
+            archive_path,
+            expected_size,
+            expected_digest,
+        )
+        return {
+            "path": str(archive_path),
+            "manifest": str(manifest_path),
+            "bytes": expected_size,
+            "sha256": expected_digest,
+        }
+
     async def _read_json_file(self, env: AgentEnvironment, path: str) -> dict[str, Any]:
         result = await env.exec(["bash", "-lc", f"cat {shlex.quote(path)} 2>/dev/null || true"], timeout=30)
         raw = (result.stdout or "").strip()
@@ -283,7 +440,11 @@ async def _install_codex_auth(self, env: AgentEnvironment) -> None:
 
     async def _scrub_codex_auth(self, env: AgentEnvironment) -> None:
         home = shlex.quote(self._codex_auth_container_home)
-        result = await env.exec(["bash", "-lc", f"rm -rf -- {home}"], timeout=30)
+        role_homes = shlex.quote("/tmp/multiagent-prod-swe/role-codex-homes")
+        result = await env.exec(
+            ["bash", "-lc", f"rm -rf -- {home} {role_homes}"],
+            timeout=30,
+        )
         if result.returncode != 0:
             tail = ((result.stderr or "") + "\n" + (result.stdout or "")).strip()[-1000:]
             logger.warning(f"multiagent-native failed to scrub Codex auth home: {tail}")