@jumptag/refine-skill

Refine an Agent Skill via the skill-forge judge → hitl loop, in a sandboxed Docker container.

Quick start

export ANTHROPIC_API_KEY=sk-...
npx @jumptag/refine-skill ./path/to/skill

Default: 3 iterations max, claude-sonnet-4-5, telemetry written to <path>/.refine/log.json.

Requirements

• Node 20+ (for npx).
• Docker (Engine 20.10+, daemon running).
• An API key for one of: Anthropic, OpenAI, Google, xAI, Mistral, Groq, OpenRouter — matched to your --model choice. See MODELS.md for the full list of supported models, env vars, and where to get keys.

Usage

npx @jumptag/refine-skill <path> [options]

Option	Default	Effect
--iterations N	3	Max passes before stopping at the cap
--model M	claude-sonnet-4-5	Any model pi.dev supports
--image TAG	ghcr.io/barryroodt/refine-skill:<pkg-version>	Override the image
--pull POLICY	missing	always / never / missing
--no-log	off	Skip writing .refine/log.json
--dry-run	off	Print the docker invocation and exit
--verbose	off	Stream pi output uncut
--pi-timeout SECS	600	Per-pi-call timeout; auto-scales to 1800 for opus/fable-class models

The model catalog lives in the image, not the CLI. The pi model catalog is baked into the Docker image (tagged to the package version), so @latest on the npm CLI does not guarantee the newest models. A slug the image predates fails fast with exit 5 and a list of the closest supported slugs. If a specific new model matters, pass a matching/newer --image tag (or --pull always), or run --dry-run first to see the resolved image tag.

Filtering streamed output. Piping the CLI through a redactor or logger makes the shell report the last command's status, masking the CLI's real exit code and defeating the exit-code contract below. Preserve it with pipefail + PIPESTATUS:

set -o pipefail
npx @jumptag/refine-skill@latest ./skill --model claude-opus-4-8 2>&1 \
  | sed -E 's/sk-ant-[A-Za-z0-9_-]+/sk-ant-***REDACTED***/g'
status=${PIPESTATUS[0]}   # capture immediately, before any other command resets $?
exit "$status"

How stopping works

The loop exits at the first matching rule:

1. Pass 1 → never stops.
2. Judge produces zero items → all_obsolete.
3. All items match "already satisfied / no-op / superseded" → all_obsolete.
4. Score fraction (score / max) gains < 0.02 vs. the previous pass → delta_below_threshold. The rubric max can move between passes, so this compares normalized fractions, not raw points — a positive raw delta can still be a normalized regression.
5. All items match trade-off / diminishing / LOW priority → tradeoff_floor.
6. Pass > --iterations → max_iterations (exit 1, still successful).

Telemetry

.refine/log.json contains per-pass score/grade/delta, per-item commit messages + diffs, stop reason, model, image tag, timestamps. Disable with --no-log.

.refine/log.json also carries a summary block — input (pass 1, the original skill), best, and final scores as normalized fractions, plus regressed_below_input. Because the loop can converge on a state scoring below the input, check summary.regressed_below_input before adopting the result, and revert to summary.best (via that pass's per-item commits) if it is true.

Running without npx

docker run --rm -i \
  -v "$PWD/path/to/skill:/work" \
  -e ANTHROPIC_API_KEY \
  ghcr.io/barryroodt/refine-skill:latest \
  /work --iterations 3 --model claude-sonnet-4-5

Exit codes

Code	Meaning
0	Natural convergence (any of rules 2-5)
1	Max iterations reached (still successful)
2	Bad path / missing SKILL.md
3	Missing / mismatched API key
4	Docker not available
5	Model not found in image catalog
10	Pi crash
11	Judge output malformed
12	Hitl partial apply
13	Disk full / OOM
14	Another refine running on the same path
130	SIGINT
143	SIGTERM

Spec

specs/2026-05-20-deftly-refine-cli-design.md

Credits

refine-skill is a thin orchestration harness around two existing pieces of work:

• Skill Forge by @WrathZA — skill-forge-judge + skill-forge-hitl provide all the actual refinement logic (scoring rubric, per-item HITL loop). Apache 2.0; pinned tag 2026.04.30; baked into the image at build time and copied verbatim. See NOTICE.
• pi.dev coding agent by @mariozechner — provider-agnostic LLM harness that runs the two skills inside the container.

This project (@jumptag/refine-skill, MIT) just wires them together: Node CLI + bash outer loop + deterministic stop rules + telemetry.