Web Skill Factory: evolving reusable, verified, code-native skills for web agents#54
Open
DEM1TASSE wants to merge 141 commits into
Open
Web Skill Factory: evolving reusable, verified, code-native skills for web agents#54DEM1TASSE wants to merge 141 commits into
DEM1TASSE wants to merge 141 commits into
Conversation
…tool
A built-in submodule turning solved tasks into reusable, executable code skills:
- skills/{library,retrieve,decide,gate,update,llm}: store / retrieve (relevance) /
decide (use·adapt·skip utility) / admission gate (gold|self_verify|none) /
evolve (incremental growth on existing library) — backend-agnostic via configure_llm
over webwright's own Model abstraction (no hardcoded gateway/key/path)
- tools/skill_use.py: solve-time tool (agent invokes like self_reflection/image_qa) ->
retrieve+decide -> JSON recommendation (use/adapt/skip + source path)
- python -m webwright.skills.update --manifest batch.json --library ./lib : batch growth
- tests/skills: 5 unit tests pass against the migrated module (logic == original)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…skill_use CLI - skills/prompt.with_skill_hint: prepend skill-library usage hint to task prompt (non-invasive; webwright merges system_template by replacement, so prompt-level is the clean way) - config/skill_mode.yaml: optional overlay doc + step budget for skill-reuse runs - llm._model(): bare CLI (python -m webwright.tools.skill_use) builds model from SKILL_MODEL_NAME/ENDPOINT (or OPENAI_*) env -> same backend as agent, no hardcoded gateway Co-Authored-By: Demi Wang <86202027+DEM1TASSE@users.noreply.github.com>
- README: what the module is, the two plug points (skill_use tool + update CLI), components table, gate semantics, backend config, results summary - llm._model(): bare CLI builds model from SKILL_MODEL_NAME/ENDPOINT (or OPENAI_*) env Co-Authored-By: Demi Wang <86202027+DEM1TASSE@users.noreply.github.com>
- README: Skill Library section (what it is, reuse via skill_use tool, grow via update CLI, end-to-end validation summary) - tests/skills: 5 unit tests for library/gate/update/evolve/retrieve+decide Co-Authored-By: Demi Wang <86202027+DEM1TASSE@users.noreply.github.com>
Co-Authored-By: Demi Wang <86202027+DEM1TASSE@users.noreply.github.com>
Remove _grow / update() / _UPDATERS dispatch — evolve() is the single entry now; drop the test_update test that exercised the removed grow path. Keep retrieve/llm fallbacks (useful). Co-Authored-By: Demi Wang <86202027+DEM1TASSE@users.noreply.github.com>
…val) Three bugs hit when update.refine emits a large skill on a slow gateway: - llm() ignored max_tokens -> model default ~4000 truncated the refined skill mid-code - llm() had no timeout override -> model default 120s ReadTimeout'd on the ~16k-token refine (now request_timeout_seconds defaults 600, env SKILL_MODEL_TIMEOUT) - _extract_code returned raw text (with ```python fence) when the closing fence was missing (truncated) -> skill failed to compile; now strips the opening fence anyway Co-Authored-By: Demi Wang <86202027+DEM1TASSE@users.noreply.github.com>
…lve-time reuse, direct skill run) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Author
@microsoft-github-policy-service agree company="Microsoft" |
…te+manifest -> update -> reuse); fix output_schema examples to gate's {type} form
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…bArena numbers Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- traces_from_manifest: 'admit' is now REQUIRED per run — a missing gate verdict raises instead of silently defaulting to admitted (was the main pollution risk) - _slug: templates longer than 48 chars get a short content-hash suffix so two templates sharing a long prefix can no longer overwrite each other's skill - skill_use.recommend: the decision's skill_id must be one of the RETRIEVED candidates; anything else (LLM hallucination, even an existing library id) downgrades to skip - tests for all three Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… is truthy) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tive paths, missing answer file) Independent cleanroom reproduction (fresh clone + venv, README-only, public GitHub tasks) surfaced usability failures; mechanism itself reproduced end-to-end in 25 min. - with_skill_hint resolves the library path to ABSOLUTE (F2): the hint's command runs in the agent's workspace, where a relative ./library silently resolved to a nonexistent dir -> empty library -> every lookup skipped, no error, answer still right - skill_use.recommend: a missing/empty library now answers skip with an explicit 'warning: library empty at <abspath>' BEFORE Library() can mkdir the bogus path (F3) - README: step 1 now tells the agent to write agent_response.json (stock webwright does not produce it; the gate/manifest flow assumed it) with a copyable ANSWER_SPEC (F1); absolute-path + --library-beats-env notes (F2/F4); custom endpoint tip (F5) - skills/__init__ no longer eagerly imports update -> no more runpy RuntimeWarning on 'python -m webwright.skills.update' (F6); import evolve/Trace from the submodule - tests: hint abspath, empty/missing-library warning (incl. no-mkdir side effect) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…s, filled-in inputs - example_library/: the commit-counting skill verbatim as evolve wrote it (runnable standalone via taskspec, no LLM in the loop; functionally verified against a local repo) - README: what a skill looks like (catalog card + the distilled git-log algorithm), measured held-out numbers (33->10 steps; wrong->correct rescues; honest note that reuse costs more than it saves on cheap tasks), three try-it paths - honest coverage-boundary demo: an unseen period shape raises cleanly; on the real held-out run the agent read the source and adapted around it - tasks/batch/taskspec example JSONs matching the how-to-use steps - links from the module README Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…, safe growth, measured results) + data-flow/interfaces diagram
…no manifest) python -m webwright.skills learn <runs_dir> --library ./library - auto: reads task.json/agent_response.json per run, gates (gold via --golds, else self_verify), groups tasks into templates + extracts params with one LLM call per chunk (default 25; existing templates passed in so chunks refine instead of duplicating), infers output_schema from the answer shape, site from start_url - idempotent: processed runs remembered in library/.learned.json; --dry-run plan mode - README: Quickstart (two commands) + use cases up top; old walkthrough demoted to 'Manual mode'; examples/solve_with_library.sh wrapper (hint + answer instruction) - unit tests for the LLM-free plumbing
…ks, leaks) External-user test of the friendly path surfaced that a trivially-easy config mistake (gateway key + unset OPENAI_ENDPOINT) silently disabled ALL reuse. Fixes: - skill_use: a hard error still degrades to skip (never block solving) but now says LOUDLY it is a LOOKUP FAILURE, not a no-match — error field in the JSON, hint about OPENAI_ENDPOINT/SKILL_MODEL_ENDPOINT, and a stderr line (F1+F2) - README Quickstart: gateway users must export OPENAI_ENDPOINT/OPENAI_MODEL for BOTH steps, stated where step-1 users actually look (F2) - learn: grouping-LLM failure now exits with a one-line actionable message instead of a 40-line traceback (F3); skipped-for-missing-answer runs get a visible summary with the correct pointer (the old message named a command that does not exist) (F4) - learn: strips the answer-output instruction from task text so it cannot leak into templates/skill_ids (F7) - solve_with_library.sh: usage check instead of passing empty args into the CLI (F6)
…ers get them too)
…ssion tests - README: "Only verified solves get in" -> "Validation-gated, exactly as strong as the gate you give it" — states plainly that the default self_verify checks shape only and that the WebArena numbers used the gold gate; learn prints the same warning at run time when no --golds is given - examples/learned_library/: a skill produced by "skills learn" from 3 real GitHub solves — n_solves=3, owner/repo lifted to parameters, two extraction strategies as fallbacks; verified standalone on an unseen repo (numpy/numpy -> v2.5.1, no model); test_learned_example.py locks n_solves>=3 + lifted params + no leak - regression tests for the interface-test findings: F1 (skill_use surfaces hard errors as ERROR, not quiet skip), F3 (learn exits with an actionable message)
There was a problem hiding this comment.
Pull request overview
This PR introduces a new webwright.skills subsystem that turns previously solved tasks into reusable, executable “skills”, enabling solve-time reuse (via a CLI tool) and offline library growth (via learn/update pipelines) while keeping the main agent loop unchanged.
Changes:
- Adds a disk-backed skill library (
Skill/Library) plus retrieve/decide/gate/evolve/learn modules to store, select, admit, and incrementally refine skills. - Adds
webwright.tools.skill_useas a solve-time CLI that recommendsuse|adapt|skipand provides the source path for reuse. - Adds docs/config/examples and new tests to validate deterministic plumbing and example artifacts.
Reviewed changes
Copilot reviewed 29 out of 31 changed files in this pull request and generated 13 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/skills/test_retrieve_decide.py | Adds deterministic tests for retrieve/decide + skill_use/prompt behavior (currently not pytest-discoverable). |
| tests/skills/test_library.py | Adds tests for disk persistence of Library (currently not pytest-discoverable). |
| tests/skills/test_learned_example.py | Adds a check that the checked-in learned example is aggregated/parameterized (currently not pytest-discoverable). |
| tests/skills/test_learn.py | Adds tests for learn plumbing + regression handling (currently not pytest-discoverable). |
| tests/skills/test_gate.py | Adds tests for gate admission logic (currently not pytest-discoverable). |
| tests/skills/test_evolve.py | Adds tests for evolve behavior and slug collision avoidance (currently not pytest-discoverable). |
| src/webwright/tools/skill_use.py | Introduces solve-time library recommendation tool with guardrails against missing/empty libraries and hallucinated skill IDs. |
| src/webwright/skills/update.py | Implements incremental library evolution and refinement prompt construction + manifest ingestion. |
| src/webwright/skills/retrieve.py | Implements LLM-based retrieval plus a simple deterministic keyword-overlap fallback. |
| src/webwright/skills/decide.py | Implements LLM-based use/adapt/skip decision over retrieved candidates. |
| src/webwright/skills/gate.py | Implements admission gate (gold/self_verify/none/auto) to prevent wrong solves from entering the library. |
| src/webwright/skills/learn.py | Adds “friendly” pipeline to learn skills from run folders with gating, chunked grouping, and an idempotent ledger. |
| src/webwright/skills/library.py | Adds on-disk skill storage (<id>/skill.py + meta.json) and simple list/get/add APIs. |
| src/webwright/skills/llm.py | Adds backend-agnostic LLM helper using Webwright’s Model abstraction. |
| src/webwright/skills/prompt.py | Adds with_skill_hint() helper that prepends a bash command hint to consult the skill library. |
| src/webwright/skills/init.py | Exposes the public webwright.skills API surface for consumers. |
| src/webwright/skills/main.py | Adds `python -m webwright.skills <learn |
| src/webwright/skills/README.md | Adds comprehensive module documentation, usage patterns, and rationale. |
| src/webwright/skills/pipeline_diagram.svg | Adds diagram documenting data flow and interfaces for the skills pipeline. |
| src/webwright/config/skill_mode.yaml | Adds optional config overlay to increase step budget for skill reuse runs. |
| src/webwright/skills/examples/README.md | Adds examples overview and how-to for running skills/tools and batch pipeline. |
| src/webwright/skills/examples/solve_with_library.sh | Adds helper script to prepend hint + answer spec and run Webwright with a library. |
| src/webwright/skills/examples/taskspec.example.json | Adds example taskspec input for running a skill standalone. |
| src/webwright/skills/examples/tasks.example.json | Adds example batch task list input with params/golds. |
| src/webwright/skills/examples/batch.example.json | Adds example manifest for update (admit/params/schema/etc). |
| src/webwright/skills/examples/example_library/how_many_commits_did_user_make_period_in_the_cur/skill.py | Adds a runnable example skill produced by the pipeline. |
| src/webwright/skills/examples/example_library/how_many_commits_did_user_make_period_in_the_cur/meta.json | Adds metadata for the example skill. |
| src/webwright/skills/examples/learned_library/what_is_the_latest_release_version_of_ow_c29dab8/skill.py | Adds a checked-in “learned” skill example aggregated from multiple solves. |
| src/webwright/skills/examples/learned_library/what_is_the_latest_release_version_of_ow_c29dab8/meta.json | Adds metadata for the learned skill example. |
| README.md | Adds top-level README section linking to the new skill-library capability and docs. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| f"answer={json.dumps(tr.answer, ensure_ascii=False)[:120]})\n```python\n{tr.code}\n```" | ||
| ) | ||
| sys_prompt = _REFINE_SYS + (_REFINE_INCREMENTAL if existing else "") | ||
| code = _extract_code(llm(sys_prompt, "\n\n".join(blocks), max_tokens=16000, timeout=400)) |
Author
There was a problem hiding this comment.
Fixed in 6513d87 — removed the kwarg (it was indeed swallowed by llm()'s **_). Timeout is governed by the model's request_timeout_seconds (600s default in _model()).
Comment on lines
+175
to
+179
| if answer is None and (d / "agent_response.json").exists(): | ||
| try: | ||
| answer = json.load(open(d / "agent_response.json")).get("retrieved_data") | ||
| except Exception: | ||
| pass |
Author
There was a problem hiding this comment.
Fixed in 6513d87 — switched to Path.read_text(encoding="utf-8").
Comment on lines
+198
to
+199
| manifest = json.load(open(a.manifest, encoding="utf-8")) | ||
| traces = traces_from_manifest(manifest) |
Author
There was a problem hiding this comment.
Fixed in 6513d87 — switched to Path.read_text(encoding="utf-8").
Comment on lines
+59
to
+64
| def llm_json(system: str, user: str, **kw: Any) -> dict: | ||
| """Call + parse the first {...} JSON object out of the reply.""" | ||
| txt = llm(system, user, **kw) | ||
| match = re.search(r"\{.*\}", txt, re.S) | ||
| if not match: | ||
| return {} |
Author
There was a problem hiding this comment.
Fixed in 6513d87 — llm_json now walks the reply with json.JSONDecoder().raw_decode and returns the first valid JSON object, skipping non-JSON brace snippets; matches the docstring.
Comment on lines
+48
to
+56
| meta = json.loads((d / "meta.json").read_text()) | ||
| code = (d / "skill.py").read_text() if (d / "skill.py").exists() else "" | ||
| return Skill(skill_id=skill_id, code=code, meta=meta) | ||
|
|
||
| def add(self, skill: Skill) -> None: | ||
| d = self._dir(skill.skill_id) | ||
| d.mkdir(parents=True, exist_ok=True) | ||
| (d / "skill.py").write_text(skill.code) | ||
| (d / "meta.json").write_text(json.dumps(skill.meta, ensure_ascii=False, indent=2)) |
Author
There was a problem hiding this comment.
Fixed in 6513d87 — explicit UTF-8 on all skill/meta reads and writes in library.py (and the same for learn.py's ledger/run files).
Comment on lines
+34
to
+35
| if __name__ == "__main__": | ||
| run() |
Author
There was a problem hiding this comment.
Fixed in 6513d87 — added a pytest entry point (test_all).
Comment on lines
+86
to
+87
| if __name__ == "__main__": | ||
| run() |
Author
There was a problem hiding this comment.
Fixed in 6513d87 — added a pytest entry point (test_all).
Comment on lines
+26
to
+27
| if __name__ == "__main__": | ||
| run() |
Author
There was a problem hiding this comment.
Fixed in 6513d87 — added a pytest entry point (test_all).
Comment on lines
+69
to
+70
| if __name__ == "__main__": | ||
| run() |
Author
There was a problem hiding this comment.
Fixed in 6513d87 — added a pytest entry point (test_all).
Comment on lines
+53
to
+55
| if __name__ == "__main__": | ||
| run() | ||
| run_regressions() |
Author
There was a problem hiding this comment.
Fixed in 6513d87 — added a pytest entry point (test_all) that runs both run() and run_regressions().
The README table (held-out 70% vs 55%, 14.7 vs 17.1 steps; train 26/30 vs
23/30) previously existed only as prose. Now:
- evals/webarena/results/: sanitized per-task records of the exact run behind
the table — task id, answer, gold score, steps, skill verdict; one command
("reproduce.py table --results results") re-derives the table, no setup
- evals/webarena/reproduce.py: self-contained driver that re-runs the whole
experiment (train -> gold-gated update -> held-out with/base -> table)
against your own WebArena deployment via microsoft/webarena-verified;
resumable, parallelizable per template
- run_all.sh + model.eval.yaml (agent-model overrides; gateway pointer)
- tests/skills/test_eval_snapshot.py: CI-locks the records to the published
numbers and enforces snapshot sanitization (no local paths/hosts/keys)
- skills README + examples README now link the records instead of asking for
trust; CI also triggers on evals/webarena/**
- prompt.py: shell-quote task and library in the skill_use hint (shlex.quote) —
$VAR / $(...) / backticks expanded in bash even inside the old double quotes;
regression test added
- llm.py: llm_json now scans for the FIRST valid JSON object (raw_decode) as
documented, instead of a greedy first-{ to last-} regex that could span
unrelated braces
- update.py: drop the misleading llm(..., timeout=400) kwarg (silently
swallowed; the model's request_timeout_seconds already governs); read JSON
files via read_text instead of unclosed open()
- library.py / learn.py: explicit UTF-8 on every skill/meta/ledger read+write
(locale-independent on Windows)
- tests: pytest-discoverable test_all() entry points in all 7 files (CI keeps
running them as scripts too)
adamlu123
reviewed
Jul 9, 2026
Author
There was a problem hiding this comment.
It's the guided tour of the examples/ directory (the module README links here for every "see examples"): two real, checked-in skill libraries — learned_library/ is the Quickstart loop's actual output (3 GitHub solves -> learn -> owner/repo lifted to parameters, runs standalone on unseen repos with no model), example_library/ is verbatim update.evolve output from the WebArena eval — plus the solve wrapper and filled-in copies of every input file the manual pipeline asks you to write. 8ea59be makes this explicit in the file's opening paragraph and adds the learned_library provenance section.
Contributor
There was a problem hiding this comment.
this file looks redundant, can we remove?
Author
There was a problem hiding this comment.
Agreed — removed in 8ea59be. Nothing referenced it: the skill hint is prompt-level (with_skill_hint), and no documented path needed the step_limit bump.
| # 2. turn everything you've solved into skills — no manifest, no fields to learn | ||
| python -m webwright.skills learn outputs/ --library ./library | ||
| ``` | ||
|
|
Contributor
There was a problem hiding this comment.
@DEM1TASSE It is better to add the complete example in the quick start session.
It needs to additionally include how to use the skill library.
Author
There was a problem hiding this comment.
Done in 8ea59be — the Quickstart is now the complete loop on a copy-pasteable example (public GitHub): solve 3 instances -> learn -> an unseen instance reuses the skill (with the expected skill_decision.json shown), plus how to use the library without the agent (querying skill_use directly, and running the learned skill standalone with no model — verified pandas-dev/pandas -> v3.0.4). The same loop's output is checked in at examples/learned_library/.
…larify examples/ - README Quickstart is now the complete loop on a runnable example (public GitHub): solve 3 instances -> learn -> an UNSEEN instance reuses the skill, plus using the library without the agent (skill_use query, running the learned skill standalone, verified: pandas-dev/pandas -> v3.0.4 with a params-only taskspec) - remove config/skill_mode.yaml: nothing referenced it (the skill hint is prompt-level via with_skill_hint; the step_limit bump was never needed by any documented path) - examples/README: state the directory's purpose up front, and document learned_library's provenance (the Quickstart loop's checked-in result) with the unseen-repo runs and the CI test that locks it
…unclutter gpt-4o is real, not a typo: openai_model.py:106 is `model_name: OptStr = "gpt-4o"`, and skill_factory names no model of its own, so that's genuinely what you get when you set neither var. It's an upstream file this PR has no business editing. But quoting it in the table next to Results-at-gpt-5.4 reads as a recommendation that contradicts the experiments. So the table now says "the class's fallback" for both the model and the URL, and one line under it names them and says what they are: inherited, not suggestions, the Results ran on a much newer model, so name the model you want. --jobs was five sentences in one cell nobody can scan. Cell keeps what a lookup needs (solve N at once, solving only, more than you have means all); the throttling advice moves to a line under the table, where "3 to 5" also retires the file's last en-dash. SKILL_MODEL_ENDPOINT and SKILL_MODEL_NAME were crammed into one <br> row while their OPENAI_* twins had a row each. One var per row now, matching every other table. Confirmed, unchanged: skill_use --output writes an extra copy — print(payload) at skill_use.py:97 is unconditional, so stdout gets the verdict either way. The table is right and this is the behaviour the older "also write" wording meant. gate.py's methods (gold | self_verify | none) still match the input-gate levels; `auto` is a fourth, the signature default, which just dispatches to gold when a gold is present and self_verify otherwise — neither section lists it, and they stay consistent.
…fault The moved-page sentence said a failed replay "is real news and reference records it", which reads as automatic. It isn't: learn defaults to --on-fail reject, as its own table two sections down says, so by default a moved page leaves the runs retryable and nothing lands. Now the sentence names the flag and says what the default does instead. Dropped the em-dashes from the paragraphs I added (verification, two-models); the title's is yours, left alone. skill_use is introduced as belonging to neither mode, then appears in the two-models table as a caller of the module's model, which looks like a contradiction until you know it makes a model call at all. It does: skill_use.py:48 calls decide(), and decide.py:38 is an llm_json round trip. Retrieval itself is local. Said so where the command is introduced, which also answers "how much does asking cost".
… gpt-4o Reviewing the docs kept snagging on the same thing: the reference said the fallback model is gpt-4o while Results says gpt-5.4. The docs were right — openai_model.py:106 really does default model_name to "gpt-4o", and skill_factory names no model of its own, so setting neither SKILL_MODEL_NAME nor OPENAI_MODEL genuinely distils on gpt-4o. That default belongs to upstream and isn't this PR's to change; the fix is to stop it happening silently. _model() now prints one line to stderr when no name was set, naming the model it fell back to. Every skill in a library is written by this model, so which one it is shouldn't be something you discover later from a bad skill. It's the same shape as build's existing half-configured-gateway warning: warn, don't fail, so nobody's working setup breaks. stderr specifically, not stdout: skill_use prints JSON on stdout for an agent to parse, and a warning there would corrupt it. There's a test pinning that. Adds tests/skill_factory/test_llm_env.py — the precedence llm.py owns had no coverage at all: SKILL_MODEL_* over OPENAI_*, configure_llm over both, an unset endpoint not clobbering the class default with an empty string, and the warning (fires, names the fallback, stderr only, once rather than per chunk per draw). Each of the six was checked to fail against the bug it describes by mutating llm.py and rerunning it. Docs: the env section now says the warning exists, and says why the name matters rather than just that it's inherited.
…two vars "Set neither and you get..." made the reader stop and work out which two things "neither" meant, right after a sentence naming SKILL_MODEL_NAME, OPENAI_MODEL, and a yaml key. One word fixes the antecedent. Confirmed while here, since the sentence is a claim about behaviour: the warning fires only when both vars are missing. llm.py:35 ors them into one `name` and :42 warns on `not name`, so anyone who set only OPENAI_MODEL — the setup the paragraph above tells them is fine — stays silent. test_llm_env.py's test_naming_a_model_says_nothing pins exactly that case.
The redundancy had a root cause: "a schedule holds still so strict is fair, a fare drifts so it isn't" had no owner, so every section that touched it argued it again from scratch — section 1's closing, section 3's "Verification honestly", section 4's "When the answer moves on its own", and choosing-a-task's second question. Four passes at one idea. Choosing-a-task owns it now; everyone else points there. - Deleted "When the answer moves on its own" whole. It was question 2 with a different example. Its one unique fact — shape catches a broken skill but never a wrong one, which is what --golds is for — moved into question 2. - "Verification, honestly" keeps only what nothing else says: the standalone answer is byte-identical to an independent model-free probe. Two code paths, one answer. The strict-is-fair-here argument around it went. - Section 1's closing paragraph is one sentence and a link. - The timeout warning at the top now lives in section 4's --jobs paragraph, next to the flag it's about, instead of greeting you before you've run anything. Also unpacked the sentences that were carrying three clauses each (the step-1 screenshot line, how-to-read-the-middle-column, vary-the-parameters). Untouched on purpose: the cost table and how to read it (that's quickstart's own measurement, not the README's WebArena numbers), all four choosing-a-task questions, and the rejection table. 296 lines to 264, and what's left is what only this page says.
Updated quickstart instructions for running the skill and agent, clarified output and execution steps.
…" wasn't Your annotations, applied on top of your section 1-2 rewrite: - the cost table's task line names the date it was measured on, 2026-08-15 — pinned, unlike section 1's rolling default, which is why it's worth stating — and keeps "a route the skill never trained on", since held-out is the point of the table; - "This loop 已经提交和展示到" -> checked in at examples/learned_library/, with the provenance pointer; - section 4 becomes the two doors you listed, learn first: 4.1 you have trajectories, 4.2 you have a task but no runs. It used to open with init and build and bury learn at the end, which is backwards — learn is the cheap day-to-day path. Same order and same two doors as the README's 3.1/3.2. One real defect found while I was in there. The spec block says "this is its real output, verbatim" and it wasn't: no draws: 2 at all (I added --draws and never updated this page), and two of init's comments had drifted. Regenerated it by calling init's own _yaml_skeleton, so it's byte-identical now, long comment line and all. quickstart.sh: ask and solve now print the date they're asking about. demo already did. solve's answer depends on that date and it wasn't saying which.
"Step 3" meant two different things depending on which page you were on. The README's three steps are run / agent / build your own; quickstart had four, because "Watch the library get built from nothing" sat between 2 and 4 holding a step number. So quickstart's 4.1/4.2 — which I'd just aligned with the README's 3.1/3.2 — were reachable from a section number that didn't exist over there. The tell was in the title: sections 1, 2 and 4 are things you do, and section 3 was something you watch. It isn't a step. It's the worked example of the step after it, so it moves inside "do it for your own task" as an unnumbered "the same loop, on our task", keeping all of it — the dry-run, the by-hand fold-out, the cost table. Nothing is cut; it just stops claiming to be a step you take. Numbered sections now map 1:1 onto the README's, down to 3.1/3.2, and the header says so and links there. Checked: nothing anywhere linked to the two anchors this retires.
The quickstart was a second telling, not a second half. Its sections 1 and 2 had drifted into near-copies of the README's — same commands, same ten-steps line, same cost table, same probe claim — and once I renumbered it to match the README last commit, the overlap was the whole top of the page. Two files, one story, already disagreeing in places. What it had that the README didn't is now in the README, folded so the section stays scannable: - **the loop by hand** — the solve/learn/reuse commands build wraps, minus its copy of the gateway setup, which the README already documents right below; - **getting a skill that lands** — vary the parameters rather than the count, and which rejection means "wrong verify mode" versus "actually broken", with the .rejected_<id>.py post-mortem path. "What to expect" said rejection is cheap and not your fault; it never said how to read one. Three things the quickstart answered and the README didn't, now answered inline: `./quickstart.sh` with no arguments IS demo mode, on SEA→DEN thirty days out — so the date rolls, which is why it prints the one it picked and why your flight won't be the one in the table. The table's three runs are pinned to 2026-08-15; that's a measurement, and it now says so, because otherwise the first thing a reader does is run the command and get a different flight than the row above. The run directory keeps the trajectory, and QUICKSTART_WORKDIR keeps the run directory. `build` now shows the spec we ship (examples/flights.skill.yaml), so you can dry -run the real thing before writing one. `learn` already had its shipped trajectories in 3.1. Dropped with the file, deliberately: choosing-a-task, and the cost table's training-run spread. Every inbound link updated — the README's two, and reference.md's, which now points at the Quick Start. Nothing in the repo mentions quickstart.md.
Updated README.md to clarify command usage and execution steps.
…tput It showed verify, draws and on_fail. init writes five: verify_rounds and chunk were missing, so a reader would conclude they were CLI-only flags — you can put either in the spec, and build reads both. That was survivable while the quickstart carried a verbatim copy; deleting it made this the only spec anyone sees. It had drifted twice, both times the same way: I added --draws, then --verify-rounds, updated init.py, and never came back here. Nothing pinned the docs to the code. There's a test for init-writes == build-reads, and none for what the docs show. So test_the_readme_spec_shows_every_key_init_writes parses the README's yaml block and compares its build keys against _yaml_skeleton's. Checked both directions by mutation: drop verify_rounds from the README, it fails; make init write a key the README lacks, it fails. While there, the two keys people confuse now explain themselves in the block: draws bins the candidate and distils a fresh one, verify_rounds feeds one candidate its own failures. And the header says spec keys are CLI flags too, and the flag wins.
Updated README to clarify the purpose and behavior of 'init' and 'build' commands, including details on verification and handling of rejected skills.
…ledgered too
"A skill is added to library/.learned.json only after it passes verification" is
wrong twice.
It's runs, not skills: the ledger is {"runs": {<run dir>: {"template": ...}}} and
collect_runs (learn.py:44) uses it to skip dirs it already processed. Skills live
in library/<id>/ and never appear in it. The distinction is the whole point of
the sentence — a rejection is cheap to recover from *because* the runs stay
un-ledgered and a later learn picks them up again. Read it as a skill registry
and the sentence explains nothing.
And "only after it passes verification" isn't true under --on-fail reference:
update.py:322 files that under "reference", not "rejected", so learn.py:167 never
skips the ledger write. A reference skill lands without passing, and its runs are
marked learned anyway — which is correct, the material was spent.
So the --on-fail reference paragraph now says it's a one-way door for that
template. The skill exists and its runs are ledgered, so a later learn skips the
runs and won't replace the skill; going again for an executable one means
deleting both. (I first wrote "delete the skill" — wrong: collect_runs never
looks at the library, so the runs stay skipped and learn finds nothing to do.)
Also made the --golds snippet a command you can run instead of a bare flag in a
```bash fence, and dropped the blank lines inside it.
… this"
Quick Start opened with `cd src/webwright/skill_factory/examples && ./quickstart.sh`
and never said where the repo comes from or what has to be installed first. Run
it on a fresh box and the very first command dies with
./quickstart.sh: line 62: python: command not found
which reads like a broken script and is really a missing sentence: stock Linux
has python3 but no python, and the scripts call python — which exists once a venv
is activated. So the setup block says to activate one, and says why, naming the
error you get if you don't.
Clone/venv/install match what the root README already prescribes (pip install -e .,
playwright install chromium); the module ships inside Webwright, so it's that repo
you clone.
Also notes step 1 is the key-free one. (First draft said steps 1 and 2 were —
step 2 is ask/solve, both of which call a model and say so ten lines down.)
The rewrite kept the runs-not-skills fix and brought back the other half:
"Once a skill passes verification, its runs are marked as learned." A reference
skill is precisely one that didn't pass, and its runs are ledgered all the same —
update.py:322 files it under "reference" rather than "rejected", so learn.py:167
never skips the ledger write. The paragraph 27 lines down says so out loud ("the
skill now exists and its runs are ledgered"), so the page was contradicting
itself; the code agrees with that one.
Same sentence, now: lands — executable or reference — rather than passes.
The gateway fold-out was the last thing in Quick Start — after all three steps had told you to run things. Anyone on a gateway found it once their solves had already gone to api.openai.com. It's setup, so it sits with setup now: right after clone/venv/install, before step 1. So the top now answers the whole question in order: where the repo comes from, how to get a working python, the key, and — folded, because most people don't need it — the endpoint, the model, and the yaml the agent reads instead of these env vars. Dropped the `export OPENAI_API_KEY=...` line from step 2's block: it was the same line the setup section now carries, and the block is about ask/solve, not about keys. Nothing else exports anything anymore.
"Copy examples/model_gateway.example.yaml and point MODEL_CFG at it" only reads as instructions if you already know how. It never said which fields to fill, that the path has to be absolute, or where to put the copy. So it now shows the cp, the two lines that matter (model_name, openai_endpoint — the same values you just exported), and the export. Two things checked in the code while writing it, one of which bit me: -c REPLACES the default configs rather than adding to them — cli.py:42 is `config_spec = config_spec or DEFAULT_CONFIGS`, and DEFAULT_CONFIGS is ["base.yaml", "model_openai.yaml"]. My first draft said `build ... -c my_gateway.yaml`, which would silently drop base.yaml. It's why quickstart.sh and the by-hand block both spell out `-c base.yaml -c <model>`. The command says so now. The warnings are real and say where they are: build.py:137 and quickstart.sh's solve path, not every mode. The copy goes outside the repo. The template is meant to be copied and filled, and a filled one holds your gateway URL — it has no business sitting in a working tree you push from.
I wrote `build examples/flights.skill.yaml` without running it. That path resolves from src/webwright/skill_factory/ — a directory the README never sends you to. Follow the page top to bottom and you're in examples/ (step 1 cds there), where it's FileNotFoundError; start from the repo root, where the other commands in 3.2 run, and it's FileNotFoundError there too. It was wrong from both cwds a reader could plausibly be in. Now it's the full path from the repo root, and says so. Verified by running it: prints the 3-instance plan and the strict/draws=2 policy, exits. Also says what --dry-run costs you, which is nothing — no key, no browser — since that's the reason to offer the shipped spec at all.
… root Fixing the FileNotFoundError, I reached for an absolute-ish path from the repo root and never asked where the page had actually left you. Step 1 does `cd src/webwright/skill_factory/examples` and nothing sends you back, so by 3.2 — after init has written skill.yaml into that same directory, and `build skill.yaml` has read it from there — you are standing in examples/, next to flights.skill.yaml. Only my line demanded a trip to the root and a 60-character path. So it's `build flights.skill.yaml` now, which is also what 3.1's trajectories example already does from the same directory. Ran it from examples/: same 3-instance plan, and library/ and build_outputs/ land beside the other example artifacts instead of at the repo root.
Reported from a fresh box: OPENAI_ENDPOINT/OPENAI_MODEL exported, build run, and all three solves died on 401s from api.openai.com. The warning for exactly this existed — and fired after the "Solve 3 instance(s)? [y/N]" prompt. You said yes, then it told you the run was doomed. That's not a warning, it's a caption. Worse, --dry-run returned two lines above it, so the free look — the whole point of which is to see what will happen before spending anything — was the one path that stayed silent about it. The check now runs before both: dry-run prints it, and a real run prints it above the prompt, where it's still a decision. Its advice was also incomplete: "Pass -c your_model.yaml" drops base.yaml, because -c replaces the defaults rather than adding to them (cli.py:42, `config_spec or DEFAULT_CONFIGS`). Follow it literally and you trade a 401 for a differently broken agent. It says `-c base.yaml -c your_model.yaml` now. Five tests, each mutation-checked against the bug it names: dry-run reports it, the text keeps base.yaml, the warning precedes the prompt (answer "n" and it's already out), and it stays quiet when you passed -c or set no gateway.
I argued the code warning was enough and the fold-out 100 lines up said it. Both true, and neither helps someone typing the command in 3.2 — which is how this got hit in the first place. A warning can only fire after you've typed it wrong; a command you can copy means you don't. So the build line carries it: what to add on a gateway, that the agent never reads the env vars, and that -c replaces the defaults so base.yaml stays. It duplicates the fold-out on purpose — this is the spot where the mistake happens. Not on the --dry-run line below it: dry-run makes no model call, so it needs no model config.
…arning
Asked why build makes you pass two -c for a gateway you've already described in
the environment. It doesn't have to. The CLI accepts inline `model.key=value`
specs, so build can translate OPENAI_ENDPOINT / OPENAI_MODEL into the agent's
config itself — no yaml to write, no MODEL_CFG, no -c:
agent cfg: base.yaml model_openai.yaml
model.openai_endpoint=https://gw.example/api/responses
model.model_name=gpt-5.5
which it prints, on dry-run too, so the backend is part of the plan you approve
rather than something you infer from a traceback.
This replaces the warning I added an hour ago. That was the wrong shape of fix:
it could only fire once you'd typed the command wrong, and its advice was two
things to remember. The failure it warned about — solves 401ing on
api.openai.com while the module talks to your gateway — can't happen now.
Explicit -c still wins, unchanged: you said what you wanted. Nothing named means
nothing invented, and the CLI's own defaults apply. DEFAULT_CONFIGS is imported
from the CLI rather than copied, because -c replaces the defaults and a
hand-copied list is a drift waiting to happen.
Two -c is upstream's config layering, by the way — base.yaml is the agent's
behaviour, model_*.yaml is the model, split so you can swap one without the
other. Not something this module chose, but something it can spare you.
Tests rewritten around the behaviour rather than the warning text: the gateway
reaches _agent_cfg, the defaults come with it, an explicit -c is left alone, no
env means no opinion, it survives the trip into _solve, and dry-run shows it.
Each mutation-checked.
README: the -c line on the build command and the yaml step in the gateway
fold-out are gone — build needs neither now. The yaml stays for quickstart.sh
(which still reads MODEL_CFG) and for pointing the agent at a different model
than the distiller.
A real build died on this today. Google Flights prints AS26; the agent recorded the label as "AS 26". Under strict, replay compares the skill's answer to that label — so every skill that read the page correctly returned "AS26" and was rejected. No draw could ever have passed: the instance was unwinnable by construction, and it cost three solves to find out. _norm already existed for exactly this reasoning — 5 == "5", because "type jitter between a solve's string answer and a skill's numeric one is not a logic error; WebArena's own evaluator normalizes the same way". Spacing and case are the same kind of jitter, so they now fold too. strict still means the same *answer*; it just stopped meaning the same *bytes*. The leniency risk is the one that matters, so it's pinned: the draw that prompted this returned "\b 434" for "B6 434" — a regex-escape bug — and that still fails, as do AS26 vs AS27 and 7:00 AM vs 8:00 AM. Mutation-checked all three ways: drop the folding, fold only case, or fold so hard the digits go — each breaks a test. No LLM judge in the replay gate, which was the other half of the suggestion. The gate's one distinguishing property is that it's mechanical and model-free — "the skill must reproduce its own answers with no model, or it doesn't land". Semantic equality belongs in the input gate, where --golds and a judge already live and where correctness is what's being decided. Docs: strict no longer promises byte equality anywhere it's defined — the gate table, the --verify row, _replay's docstring, and the comment init writes into every spec. Known gap, not touched: gate.py's _gold still compares with ==, so a gold label typed "AS 26" rejects a correct solve the same way. Same fix, different gate.
A rebuild of the shipped flights spec failed every draw, and the reason wasn't
the gate being harsh. One of the three solves ended like this:
RESULT = ["UA 729", "United", "12:10 AM"]
if "UA 729" in text: return "UA 729"
Its answer is right, so self_verify admitted it. But there is no method in it —
it recognises the answer instead of working it out. Distillation is told never to
copy an instance's values (_REFINE_SYS says so, and the repair feedback repeats
it), so it has to invent an extractor the trace never contained, and it fails
replay on that instance every time. Two draws, four draws, it doesn't matter:
that instance is unwinnable, and it takes the whole batch down with it. The input
gate can't see this — it only ever looks at the answer.
So evolve drops those before distilling, and says so.
The signature is narrow because measurement made it narrow. My first idea — "the
answer appears in the code" — fires on all three shipped trajectories: an airline
name belongs in a vocabulary, a time in an assertion. A 100% false-positive rate,
caught by trying it on real solves before writing the code. What a working solve
doesn't have is *every field at once*, verbatim. On the six real solves I have:
0/5 that distilled, 1/1 that couldn't. A single-field answer is never evidence —
"United" is a word.
Tests use those real shapes, and the false-positive direction is mutation-pinned:
change all() back to any() and the "a working solve may mention its answer" test
fails.
Changelog gains dropped_lookup, next to dropped_wrong: one counts answers the
gate rejected, the other counts answers it accepted from scripts that never
earned them.
The gate section described two gates and left a hole between them that cost a real build every draw it had: a solve can hold a right answer and contain no method. self_verify admits it — the answer is right, and the answer is all it looks at — and then distillation, forbidden from copying instance values, has to invent an extractor the trace never had and fails replay on that instance forever. So the section now shows the script that caused it, says what happens to it, and names dropped_lookup next to dropped_wrong: one counts answers the gate rejected, the other counts answers it accepted from scripts that never earned them. It also records why the test is "every field verbatim" and not the obvious "the answer appears in the code" — the obvious one flags all three shipped trajectories, because an airline name is a vocabulary entry and a time lands in an assertion. That's the kind of thing a reader will re-derive and get wrong unless the measurement is written down: 0 of the 5 solves that distilled, 1 of the 1 that couldn't.
The measurement belongs in the commit that made the decision, not in a reference page. Gone: the 0-of-5 / 1-of-1 counts, "the narrowness is the point", and the sentence about times landing in assertions. What's left is the script, what happens to it, and the one clause a maintainer needs before loosening the test — "the answer appears in the code" flags working solves too. 24 lines to 16.
Updated README to enhance clarity and detail about the Web Skill Factory, including integration points and setup instructions.
The README now says "build and quickstart.sh translate the environment variables into the appropriate agent configuration". That was true of build and not of this script, which read MODEL_CFG or nothing — so exporting a gateway sent `ask` there and `solve` to api.openai.com, and the fix was a yaml you had to write yourself. Same trick build uses: the CLI takes inline `model.key=value` specs, so the script passes OPENAI_ENDPOINT/OPENAI_MODEL through as overrides on top of the defaults. No yaml, no MODEL_CFG, nothing to remember. MODEL_CFG still wins when set, which is now what it's for: putting the agent on a different model than the distiller. Checked all three paths: env set -> base.yaml + model_openai.yaml + the two overrides; MODEL_CFG set -> base.yaml + your yaml; neither -> the old defaults. warn_gateway_agent goes with it. It existed to tell you the script was about to ignore your gateway; it can't. That's twice today a warning turned out to be a fix I hadn't written yet. The header comment was also wrong from the day it was written — "MODEL_CFG ... for the agent in solve/build" — build has never read MODEL_CFG, only -c.
Updated instructions for setting up the module and clarified the importance of keeping the virtual environment activated.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds Web Skill Factory (
webwright.skill_factory) — a self-evolving skill factory (MVP): turn solved tasks into reusable,executable code skills, retrieve + judge them at solve time, gate what enters the library, and grow
the library incrementally. A self-evolving loop on top of Webwright's code-as-action solves:
This is the reuse + accumulation layer on top of Webwright's code-as-action solves: it consumes
the
final_script.pyevery solve already produces (plain or crafted mode — both work), accumulatesskills across tasks, judges when a prior skill applies, and improves skills as more solves arrive —
with a gate so wrong solves don't pollute the library. It complements
crafted_cli: wherecrafted_cliparameterizes a single task's script by anticipating what might vary,update.refineparameterizes across multiple verified solves — the differences actually observed between
instances become the parameters.
Modular composition (~810 lines of core code)
Ten small, single-responsibility modules — each with a stable interface and a swappable
implementation:
skill_factory/library.pySkill+Library, skills on disk (skill.py+meta.json)skill_factory/retrieve.pyskill_factory/decide.pyskill_factory/gate.pyskill_factory/update.pyrefineparameterizes + decomposes into primitivesskill_factory/llm.pyModel(no endpoint/key hardcoded)skill_factory/prompt.pywith_skill_hint)skill_factory/learn.pylearn <runs_dir>: auto-group runs into templates, gate, evolve; no manifest to writeskill_factory/__main__.pypython -m webwright.skill_factory <learn|update>dispatchertools/skill_use.pyHow it plugs in (no change to the agent loop or default config)
skill_usetool, invoked from bash likeself_reflection/image_qa:{verdict: use|adapt|skip, skill_id, source_path, how_to_reuse}.updateCLI distills a batch of gate-passed solves into aparameterized, primitive-decomposed skill:
learngroups a folder of finished runs intotemplates (one LLM call per chunk), gates them, and evolves the library — idempotent,
--dry-run:examples/learned_library/checks in the skill this produced from 3 real Google Flightssolves — five parameters lifted (origin/destination city+code, date), verified on an unseen
route three independent ways (from scratch / reuse / standalone, same answer) — with a CI
test locking it.
Validation
WebArena: 10 templates × 3 domains — reuse lifts accuracy +15pp and saves steps on held-out tasks
10 retrieve-type task templates across shopping_admin / gitlab / map. Per template: 3 train
tasks build the library (solved from scratch; only gold-verified solves are admitted), 2 held-out
tasks (unseen instances of the template — different parameter values) measure reuse. Every task is
solved both WITH the library and from scratch (BASE) — 100 solves total.
Per-task records and a reproduction driver are kept in the companion research repo and can be
shipped here on request.
Highlights:
library; net reuse-wins 7 vs 1 regression across the 20 held-out tasks.
33 steps (scratch) to 10 (reuse); a map routing task from 29 to 16.
here. (The gate is exactly as strong as its verifier — the default
self_verifyis a shapecheck only; see the README's "validation-gated" section.)
update.refinelifts per-instance differences into parametersand bakes the aggregation logic (top-n ranking, commit counting, route-time extraction) into
primitives, so unseen instances of the template solve by a direct
useof the skill.skill from the shared library (grown to 10 skills over the run), including telling apart two
near-duplicate gitlab commit-counting skills (by-date vs by-period).
evolvebatches produce 4 independent skills — new templates get added, existing skills arerefined in place (working functions kept), skills with no new traces stay byte-identical, and
zero cross-contamination between skills; held-out reuse against the mixed-built library matches
the per-template-built one.
Real website (public GitHub, read-only): the full loop end-to-end
Solve two repos from scratch ->
updatebuilds a parameterized skill -> a held-out repo is solved byreusing it (the agent calls
skill_use, verdictuse, answer correct). Reuse pays off most onmulti-step tasks where saved exploration outweighs the lookup overhead (see the WebArena numbers);
on short single-page lookups it is roughly break-even.
7 unit-test files under
tests/skill_factory/(library / gate / evolve / retrieve+decide / learn /learned-example lock / eval-snapshot lock) run in CI on every push touching the module
(
.github/workflows/skills-tests.yml).Status: a deliberately simplistic MVP
Most steps are a single LLM call (retrieve = one catalog prompt, decide = one prompt, refine =
one batched prompt) — chosen for clarity, not yet for scale/accuracy. The point is the modular
shape: each stage has a stable interface, so swapping in something stronger (embedding retrieval,
a learned ranker, WebJudge / cross-source consistency for the real-website gate) is a localized
change that does not touch the others or the agent loop.
Scope
Purely additive (zero deletions), confined to
src/webwright/skill_factory/(module + examples,including a checked-in learned skill),
src/webwright/tools/skill_use.py,tests/skill_factory/and one CI workflow. The actual implementation is ~670 lines of logic(non-blank, non-comment, across the skills module + the
skill_usetool); the rest is tests,examples, eval records, and docs. No edits to the agent loop, models, or existing configs.
Module README:
src/webwright/skill_factory/README.md.