The README says these skills give "consistent, high-quality AI assistance". The project has claimed fewer iterations and better token efficiency since the beginning. None of it has ever been measured.
The specification defines how: evals/evals.json inside a skill, with realistic prompts, expected outputs and verifiable assertions, run once with the skill and once without it. The second arm is the whole point, because without a baseline a passing eval only proves the model can do the task, not that the skill helped. See the guidance.
There is nothing to build. agent-skills-eval implements the format, runs both arms, grades assertions, aggregates pass rate against tokens and time, and emits a static HTML report. It runs through npx against any OpenAI-compatible endpoint.
Scope
Two or three skills, not eighteen. Their own guidance says start with a handful of cases and expand after seeing the first results, and each case costs two model runs.
Worth picking a spread rather than three favourites:
- one that should clearly win, where the skill encodes something a model has no way to guess,
git-commit for instance, since a project's commit convention is not derivable
- one that might not,
clean-code being the obvious candidate, because a capable model already writes reasonably clean code without being told to
- one in between,
jpa-patterns perhaps, where the knowledge is real but widely published
Not a requirement for new skills
This has to stay a measurement tool rather than a tax on contributors. Writing assertions is real work, running them costs money, and a fork does not get repository secrets. Nobody should need an API key to add a skill. The routing check already covers the part that must not regress, and it is a text check for the contributor and one cheap call for the maintainer.
The uncomfortable part
The baseline arm is a capable model with no skill loaded. Some of these skills may show little or no gain. That would be a genuine result and the right response would be to cut or rewrite the skill, not to stop measuring. Worth deciding that now, before the numbers exist.
The README says these skills give "consistent, high-quality AI assistance". The project has claimed fewer iterations and better token efficiency since the beginning. None of it has ever been measured.
The specification defines how:
evals/evals.jsoninside a skill, with realistic prompts, expected outputs and verifiable assertions, run once with the skill and once without it. The second arm is the whole point, because without a baseline a passing eval only proves the model can do the task, not that the skill helped. See the guidance.There is nothing to build.
agent-skills-evalimplements the format, runs both arms, grades assertions, aggregates pass rate against tokens and time, and emits a static HTML report. It runs throughnpxagainst any OpenAI-compatible endpoint.Scope
Two or three skills, not eighteen. Their own guidance says start with a handful of cases and expand after seeing the first results, and each case costs two model runs.
Worth picking a spread rather than three favourites:
git-commitfor instance, since a project's commit convention is not derivableclean-codebeing the obvious candidate, because a capable model already writes reasonably clean code without being told tojpa-patternsperhaps, where the knowledge is real but widely publishedNot a requirement for new skills
This has to stay a measurement tool rather than a tax on contributors. Writing assertions is real work, running them costs money, and a fork does not get repository secrets. Nobody should need an API key to add a skill. The routing check already covers the part that must not regress, and it is a text check for the contributor and one cheap call for the maintainer.
The uncomfortable part
The baseline arm is a capable model with no skill loaded. Some of these skills may show little or no gain. That would be a genuine result and the right response would be to cut or rewrite the skill, not to stop measuring. Worth deciding that now, before the numbers exist.