Skip to content

feat: run Practical Coding against SkillsBench - #7

Merged
Hubujiu merged 13 commits into
mainfrom
feat/skillsbench-external-adapter
Aug 24, 2026
Merged

feat: run Practical Coding against SkillsBench#7
Hubujiu merged 13 commits into
mainfrom
feat/skillsbench-external-adapter

Conversation

@Hubujiu

@Hubujiu Hubujiu commented Aug 24, 2026

Copy link
Copy Markdown
Owner

Summary

  • add benchmarks/run_external.ps1 as a one-command external benchmark entrypoint
  • add a SkillsBench v1.1 adapter using BenchFlow's versioned dataset path
  • compare the same codex-acp / gpt-5.6-luna / medium configuration as no-skill vs a custom Skill directory containing only Practical Coding
  • dynamically select all SkillsBench v1.1 software-engineering tasks for the standard profile; smoke uses three tasks and full uses the complete versioned roster
  • run the SkillsBench oracle before model calls by default
  • run separate paired repetitions and alternate arm order to reduce order/cache bias
  • require n>=3 plus complete healthy task/repetition pairs for stable external evidence
  • report pass-rate lift, mean-reward lift, wins/losses/ties, normalized gain, per-task rates, and task-cluster-bootstrap 95% CIs
  • preserve raw BenchFlow jobs, exact commands, dataset/source pins, adapter hash, and Practical Skill bundle hash
  • add unit tests and CI coverage

Canonical stable run

pwsh -File benchmarks/run_external.ps1 `
  -Benchmark skillsbench `
  -Profile standard `
  -Runs 3 `
  -Workers 3 `
  -RequireStableRanking

Evidence boundary

This is a Practical-owned custom-Skill ablation on the immutable skillsbench@1.1 dataset. It intentionally does not use the benchmark's per-task curated Skills, so it should not be presented as an official SkillsBench leaderboard submission. It answers the narrower question: does mounting Practical Coding improve the same Codex/Luna agent on external software-engineering tasks?

@Hubujiu
Hubujiu merged commit 431aa5b into main Aug 24, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant