feat(cua): expose /v1/cua pure-Holo3 instruction loop on Modal (+H100 fallback) - #960
Open
mercurialsolo wants to merge 8 commits into
Open
feat(cua): expose /v1/cua pure-Holo3 instruction loop on Modal (+H100 fallback)#960mercurialsolo wants to merge 8 commits into
mercurialsolo wants to merge 8 commits into
Conversation
… fallback) Adds an autonomous /v1/cua endpoint to the Modal server so every step is a 100%-student Holo3 action with planner-layer modelio + per-token logprobs — the on-policy substrate SA-OPSD / GRPO training needs. Modal's run_holo3 was suite-only; the pure-Holo3 loop lived only in baseten_server.run_pure_cua and was never reachable from the Modal deployment. - _run_holo3_executor: new `_mode == "pure_cua"` branch (sibling of the _micro_plan fork) that reuses the built brain + env, drives a bare GymRunner(task=instruction, start_url), and — critically — opens a LIVE AugurAdapter onto runner._augur so publish_modelio_context publishes layer="planner" records with logprobs. On Baseten nothing sets runner._augur, so modelio silently no-ops; this path fixes that. - build_api_app: new POST /v1/cua route (contract in docs/client/pure-cua.md) that translates instruction/start_url/... into a _mode="pure_cua" suite and dispatches through a shared _submit_detached_run helper extracted from /v1/predict (one lifecycle, two entrypoints). - Honest terminal_status/halt_reason on the pure_cua envelope so _do_action maps the wire status truthfully. - H100 GPU fallback on run_holo3 (gpu=["A100-80GB","H100"]) — the A100-80GB pool starves; bundled here as it was a stranded working-tree change. Verified live: a /v1/cua run emits 4 layer="planner" modelio records, each carrying populated per-token logprobs (172 tokens/step with top-K alternatives). Requires MANTIS_CAPTURE_LOGPROBS=1 in the deploy env. Tests: /v1/cua request->suite translation + lifecycle; pure_cua branch selection + the runner._augur (logprobs-capture) guarantee. 9 passed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The /v1/cua pure-CUA loop drives Holo3's native "Action: click({...})"
text format (Strategy 2 in _parse_response). That path took a SEPARATE
coord conversion from the tool_calls path that the S09 fix repaired:
``_safe_int(0.324) == 0`` truncated the normalized fraction, ``_has_valid_xy``
then read it as the (0,0) origin and rejected the click, and the harness
substituted WAIT(1s) — silently dropping a sensible, on-policy action the
student had chosen (and often correctly predicted the outcome of). Only
the tool_calls Strategy 1 had been routed through the fraction-aware
``_coords_to_screen``.
Fix: route the native-text (Strategy 2) and JSON (Strategy 3) click /
double_click branches through ``_coords_to_screen`` (same helper the
tool_calls path uses), and validate x/y as FLOAT in ``_has_valid_xy`` so a
fraction isn't truncated to the origin before the no-target gate. The #574
(0,0)-origin reject is preserved (now on the float value).
Belongs with the /v1/cua endpoint (#960): without this, pure-CUA runs emit
WAITs instead of the clicks the policy chose. Logprobs capture is
unaffected/verified.
Tests: normalized fraction now scales on the native + JSON paths
(0.324→415, 0.5→640); explicit (0,0) and resized ints still honor the #574
contract. 68 passed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…me-offset) The deploy runs with MANTIS_ROUTE_SOM_CLICKS=enabled, which flips som_for_unstructured_clicks on globally. That routed pure-CUA brain-clicks through the SoM/CDP path (try_som_click → cdp_click_at_point), which translates screen→viewport by the browser chrome offset (outerHeight - innerHeight ≈ 95px). But the pure-CUA brain grounds on FULL-WINDOW screenshots, so every click landed ~95px too high — the model re-clicked a target it never actually hit and looped out (observed live: 6 iterations all clicking the same "Go to feed" button, 0 progress). Fix: the pure_cua branch forces som_for_unstructured_clicks=False, overriding the global flag, so clicks dispatch via env.step → xdotool in screen-space — the same coordinate space as the full-window screenshot (like the rest of the computer plane). Explicit _route_som_clicks:true still opts back into SoM routing per-request. Tests: pure-CUA forces xdotool even when MANTIS_ROUTE_SOM_CLICKS=enabled; explicit opt-in re-enables SoM. 5 passed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… rollouts The /v1/cua route validated the request (PureCUARequest, extra:allow) but never copied ``_browser_extra_headers`` into the ``_mode="pure_cua"`` task_suite it builds, so the executor's ``setup_env(extra_http_headers=…)`` always got None. The browser therefore never sent Daytona's ``X-Daytona-Skip-Preview-Warning`` / ``X-Daytona-Preview-Token`` headers, and every Daytona-hosted rollout stranded on the preview interstitial at frame one (confirmed live: URL bar showed /feed/ but the frame was the warning page). Modal-hosted envs were unaffected (no header needed), which masked it. Fix: forward the headers into the suite (accept ``_browser_extra_headers`` or a bare ``browser_extra_headers``). The env already applies them persistently via ``Network.setExtraHTTPHeaders`` (xdotool_env), which honors the ``x-daytona-*`` headers on every request. Tests: the route-translation test now asserts the headers land in the suite. 6 passed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…o-inject + imgdim probe Three fixes that unblock the graded E1 baseline (the 0/12 was dominated by the annotation defect, not model incapacity): 1. runner.py — URL-diff guard on the perceptual-diff verifier. A click that NAVIGATED is a real change even when pre/post frames hash identical (the new page renders after the post-action capture, or looks similar). Firing "no observed effect" there contradicts the "page navigated to <url>" line _build_feedback already added, poisoning context until loop-recovery thrashes the step budget — the single largest rollout-quality depressor. Snapshot the pre-action URL and suppress the warning when it changed. Also carries the [cua-offset] grounding-delta WARNING (aim observability). 2. modal_cua_server.py — /v1/cua auto-injects X-Daytona-Skip-Preview-Warning for any daytonaproxy start_url, so callers only pass the preview token and never strand on the interstitial by forgetting the skip header. 3. brain_holo3.py — [cua-imgdim] WARNING logs the model's actual frame size vs the assumed screen_size, to root-cause the Holo3 y-over-emit (pairs with the raw coord in modelio + mapped/grounded in [cua-offset]). Tests: navigation suppresses the false warning + keeps "navigated to"; the /v1/cua auto-inject fires for Daytona URLs, not others. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… visible change"
Root cause of the "clicked (no visible change)" trap (the dominant depressor
of E1 rollout quality): the env's step() built ``info = {}`` and set
``type_verified`` / ``focused_input`` but NEVER ``info["url"]``. Both the
"page navigated to <url>" annotation and the ``clicked (no visible change)``
gate in ``_build_feedback`` read ``gym_result.info.get("url")`` — so with no
URL in info, a navigating click never produced a "navigated to" line, fell
through to "clicked (no visible change)", and the brain thrashed the step
budget re-clicking a target that had in fact navigated.
Fix: step() now populates ``info["url"] = current_url`` (best-effort CDP read,
post-action verification — not target derivation). "page navigated to" fires
on a real nav → parts non-empty → the false "no visible change" is suppressed
by the existing ``not parts`` gate.
Complements the earlier perceptual-diff URL-diff guard (bef7845), which
targeted the separate "no observed effect" string.
Known minor edge (noted for follow-up): step 1 can emit one
"navigated to <initial>" since last_url starts empty — harmless, single step.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The bottleneck moved from clicking to text entry: with grounding, click sub-goals resolve, but plain <textarea>/<input> fields still failed — t01's connect-note textarea "never filled", t02's post composer "lost focus mid-type". Root cause: the TYPE ladder (Input.insertText / clipboard-paste / xdotool) all fire at document.activeElement, and the only focus-hardened CDP insert (cdp_contenteditable_insert) handles contenteditable ONLY — plain form fields had no robust path, so any focus drift between the focusing click and the keystrokes lost the text. Fix: cdp_input_insert — the <textarea>/<input> analogue. Focus-tolerant target resolution (focused field → :focus field → the single visible field on the view, which handles t02's already-lost focus), then an ATOMIC value set via the native value setter + input/change events (React-safe AND plain-textarea-safe, single-shot so no mid-type focus race), then read-back verify. Wired into the existing TYPE retry seam after the contenteditable attempt; no change to the happy path. Tests: value-set lands + read-back confirms → True; no field / empty read-back / no-CDP → False. 16 passed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…isible" one The first cut (d21a3f9) fell back to "the single visible textarea/input" when focus was lost — which filled the SEARCH BAR when the intended composer hadn't opened (verified live: post text landed in search, not the composer). Tighten target resolution to the field the model actually CLICKED: - XdotoolGymEnv records ``_last_click_xy`` (screen coords) on every click. - cdp_input_insert resolves the target as: elementFromPoint(last click, screen→viewport via the chrome offset) → focused field → :focus field. The single-visible guess is removed, so it never fills a wrong field. Tests: targets elementFromPoint(vx, vy) with the chrome-offset-adjusted click, no offsetParent single-visible scan. 17 passed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Exposes an autonomous
/v1/cuaendpoint on the Modal server: a pure-Holo3 instruction loop (goal + screenshot → Holo3 policy picks the action → repeat) where every step is a 100%-student action with planner-layer modelio + per-token logprobs — the on-policy substrate SA-OPSD / GRPO training needs.Modal's
run_holo3was suite-only (it steps a decomposed_micro_plan). The pure-Holo3 loop existed only inbaseten_server.run_pure_cua, running in-process on Baseten — never reachable from the Modal deployment. This wires the loop into the Modal GPU executor and mounts the route.Changes
_run_holo3_executor— new_mode == "pure_cua"branch (sibling of the_micro_planfork) that reuses the already-builtbrain+env, drives a bareGymRunner(task=instruction, start_url=…), and opens a liveAugurAdapterontorunner._augur. This is the make-or-break detail:runner.pypublisheslayer="planner"modelio viapublish_modelio_context(_planner_augur, …)where_planner_augurfalls back togetattr(self, "_augur", None). On Baseten nothing setsrunner._augur, so modelio silently no-ops — this path opens the adapter so the records (and logprobs) actually land.build_api_app— newPOST /v1/cuaroute (documented contract:docs/client/pure-cua.md) that translatesinstruction/start_url/max_steps/settle_time/ground_clicks/decompose/proxy into a_mode="pure_cua"suite and dispatches through a shared_submit_detached_runhelper extracted from/v1/predict(one lifecycle, two entrypoints —GET /v1/runs/{id}/status|resultwork unchanged).terminal_status/halt_reasonso_do_actionmaps the wire status truthfully (success →completed; elsehalted+ reason).run_holo3(gpu=["A100-80GB","H100"]) — the A100-80GB pool starves; bundled here as it was a stranded working-tree change with no other home.Verified live (deployed to
mantis-cua-server)A real
/v1/cuarun (default__default-647c4133, sim-env target) produced:/v1/cuain the live OpenAPI; submit →queued,mode: pure_cua, model holo3iteration_count: 4)layer="planner"modelio records (one per student step)Requires
MANTIS_CAPTURE_LOGPROBS=1in the deploy env (already set).Tests
tests/test_modal_cua_route.py(request→suite translation + auth + shared lifecycle) andtests/test_holo3_pure_cua_branch.py(branch selection + therunner._augurlogprobs-capture guarantee: adapter set,.active is True, closed on terminal;_micro_plansuite does NOT take the pure branch). 9 passed on a cleanmainbase.Notes
main; no shared hunks).tests/test_modal_endpoint.pyfail identically on cleanmain(honest-terminal-status default) — pre-existing, not touched here.🤖 Generated with Claude Code