Skip to content

fix(harbor): make generated tasks safe on remote sandboxes (e2b) - #346

Open
vaibhavdabas16 wants to merge 3 commits into
TIGER-AI-Lab:mainfrom
vaibhavdabas16:feat/harbor-remote-sandbox
Open

fix(harbor): make generated tasks safe on remote sandboxes (e2b)#346
vaibhavdabas16 wants to merge 3 commits into
TIGER-AI-Lab:mainfrom
vaibhavdabas16:feat/harbor-remote-sandbox

Conversation

@vaibhavdabas16

Copy link
Copy Markdown
Contributor

Advances #331 §3. Does not close it — the end-to-end -e e2b smoke run is still open work, and I have no e2b account to do it with.

Scoping

#331 says §1–§2 are largely done on feat/harbor-hub-registry and that "§3 (e2b) and the parity run in §4 are the open work", so this deliberately touches neither the task format nor the registry — no conflict with that branch. Of §3's checklist:

  • ✅ No dependency on local podman/docker specifics; healthcheck-driven readiness for Chromium + CDP + interceptor
  • ✅ Credentials via --ve/--env-file only; resource floor documented
  • ✅ Prior harbor-0.20-port-notes.md conclusion (E2B needs no local disk) folded into docs/harbor.md
  • harbor run ... -e e2b smoke task — not done, needs an account
  • ❌ Prebuilt public runtime image — belongs to §2 on the hub-registry branch

The bug this fixes

Setup accepted any 200 from /api/status as "runtime ready":

if curl -sf http://127.0.0.1:7878/api/status >/dev/null \
  && curl -sf http://127.0.0.1:9223/json/version >/dev/null; then

But /api/status starts answering as soon as uvicorn binds, while eval_interceptor_ready only flips to true after the CDP handler connects and sends Target.setAutoAttach (runtime-server/server.py:296) — a loop that retries on 1s sleeps. A task that starts in that window runs with interception inactive and cannot score Stage 1, and the result reads as an agent miss rather than a harness fault. The window is small on a warm local daemon and much wider on a cold remote sandbox, which is why it surfaces here.

task.toml's [steps.healthcheck] already checked the right condition. The two now share one runtime_ready_command(), so they cannot drift. That also fixed a latent escaping trap: the healthcheck string is written into a TOML basic string where \" unescapes to ", but the same literal pasted into a shell script keeps the backslash and the grep would never match. The shell form is now canonical and task_toml escapes it once with json.dumpsthe emitted task.toml is byte-identical to before (verified against main).

Also raises the setup wait from 60s to a configurable 180s, and makes a timeout tail the runtime-server log instead of printing one opaque line.

Fixed sleeps → polling

start-runtime.sh slept 1–2s after starting Xvfb, uvicorn, and Chromium. Those constants encode local daemon timing; in remote-browser mode the script declared the CDP bridge ready after a bare sleep 1. Each step now polls for the thing it needs with a bounded timeout that names the component that never came up. CLAWBENCH_RUNTIME_WAIT_TIMEOUT_S overrides the 60s per-component default.

Testing

Five new cases in tests/test_harbor_adapter.py: the healthcheck and setup script share one readiness contract in both local and kernel modes and are escaped correctly for their respective consumers; the setup wait is remote-sized and reports what failed; startup polls rather than sleeps (no bare sleep <n> left standing in for a readiness check); and a generated task carries no host paths and no baked secrets — every credential in task.toml is an env reference.

Full suite passes locally (280 passed, 10 skipped); ruff and pyright clean.

Not verified against a live Harbor run — these are static assertions on the generated dataset plus a shell-syntax check. Worth one local harbor run on the smoke dataset before merge, since the setup gate is now strictly stricter than it was.

Setup accepted any 200 from /api/status as "the runtime is ready". The
runtime server starts answering before its CDP handler attaches and sets
eval_interceptor_ready, so a task could begin in that window with
interception inactive and silently fail to score Stage 1 — the failure
looks like an agent miss, not a harness fault.

The step healthcheck in task.toml already checked the right condition, so
the two now share one `runtime_ready_command()` and cannot drift: runtime
server up, request interceptor armed, CDP endpoint live. The TOML form is
escaped once via json.dumps and the shell form is used verbatim, which
also fixes the escaping that would have made a literal copy of the
healthcheck string never match inside a shell script.

The wait is also raised from 60s to a configurable 180s, and a timeout now
tails the runtime-server log instead of exiting with one opaque line —
remote sandboxes (TIGER-AI-Lab#331 §3) provision slower than a local daemon and were
the case 60s was never sized for.
start-runtime.sh slept a fixed 1-2s after starting Xvfb, uvicorn, and
Chromium before moving on. Those numbers encode local container-daemon
timing; on a remote sandbox the next step can run against a service that
is not listening yet, and in remote-browser mode the script declared the
CDP bridge ready after a bare `sleep 1`.

Each step now polls for the thing it needs — the X socket, the runtime
server, Chromium's CDP endpoint — with a bounded timeout that reports
which component never came up. CLAWBENCH_RUNTIME_WAIT_TIMEOUT_S overrides
the 60s default for slow providers.
Adds a `-e e2b` section covering what the generated tasks assume about a
sandbox (no bind mounts outside the task dir, no local X11 or GPU, polled
readiness), the rule that every credential stays an env reference resolved
by --env-file/--ve, and the per-trial resource floor. States plainly that
the end-to-end e2b smoke run in TIGER-AI-Lab#331 is still open.

Tests assert the healthcheck and setup script share one readiness
contract in both local and kernel modes, that startup polls rather than
sleeps, and that a generated task carries no host paths or baked secrets.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant