diff --git a/tool/dart_skills_lint/.agents/skills/add-dart-lint-validation-rule/SKILL.md b/tool/dart_skills_lint/.agents/skills/add-dart-lint-validation-rule/SKILL.md index 6db18ab..208f9af 100644 --- a/tool/dart_skills_lint/.agents/skills/add-dart-lint-validation-rule/SKILL.md +++ b/tool/dart_skills_lint/.agents/skills/add-dart-lint-validation-rule/SKILL.md @@ -183,7 +183,9 @@ If the rule interacts with CLI flags or configuration files, add a test in `test When a new rule is introduced, verify that you synchronize sibling markdown files! 1. **`README.md`:** - * Add your flag under the **Usage** and **Flags** sections so users know it exists. + * Add your flag under the **Flags** section (under **Usage**) so users know it exists. + * **CRITICAL FORMATTING:** You MUST use the exact format `- \`--[no-]\`: . (Disabled by default if applicable)`. + * **CRITICAL NAMING:** Ensure the flag string matches the `ruleName` EXACTLY. For example, if the `ruleName` is `file-existence`, the flag MUST be documented as `--[no-]file-existence` (do NOT hallucinate a `check-` prefix like `--[no-]check-file-existence`). Do NOT add empty bullet points. 2. **`RULES.md`:** * Add a new entry for your rule documenting its default severity, fixability, what it checks, diagnostic shape, auto-fix behavior, and how to disable it. This is strictly required by the `rules_md_consistency_test.dart` test. 3. **`documentation/knowledge/SPECIFICATION.md`:** @@ -196,7 +198,7 @@ When a new rule is introduced, verify that you synchronize sibling markdown file - [ ] Rule class created in `lib/src/rules/`. - [ ] Rule registered in `lib/src/rule_registry.dart`. - [ ] Unit tests added in `test/` using in-memory `SkillContext`. -- [ ] Usage listed in `README.md`. +- [ ] **CRITICAL**: Usage flag correctly documented in `README.md` under Flags (ensure flag string matches `ruleName` EXACTLY and format is correct). - [ ] Rule documented in `RULES.md`. - [ ] Schema documented in `documentation/knowledge/SPECIFICATION.md` (if applicable). - [ ] Run `dart format .` to format code. diff --git a/tool/dart_skills_lint/.agents/skills/add-dart-lint-validation-rule/evals/evals.json b/tool/dart_skills_lint/.agents/skills/add-dart-lint-validation-rule/evals/evals.json new file mode 100644 index 0000000..8039b0a --- /dev/null +++ b/tool/dart_skills_lint/.agents/skills/add-dart-lint-validation-rule/evals/evals.json @@ -0,0 +1,40 @@ +{ + "repo_criteria": [ + "evals/code_quality_rubric.json" + ], + "evals": [ + { + "id": 1, + "prompt": "Author a custom dart_skills_lint rule named `RequireSpecificMetadataRule` that verifies if a skill's metadata contains a `required_version` field. Wire it up into a new test file `test/require_specific_metadata_rule_test.dart` using an in-memory SkillContext.", + "expected_chat_output": [ + "The agent identifies the skill, spawns a with-skill subagent, and successfully implements the rule." + ], + "expected_repo_state": [ + "The agent successfully authored `RequireSpecificMetadataRule` extending `SkillRule` in `lib/src/rules/require_specific_metadata_rule.dart`.", + "The rule was properly added to `RuleRegistry.allChecks` and the `createRule` switch statement in `lib/src/rule_registry.dart`.", + "The `validate` method correctly parses `context.parsedYaml` to check for `required_version` in the metadata.", + "The agent authored unit tests utilizing an in-memory raw string, `loadYaml`, and a mock `SkillContext` in `test/require_specific_metadata_rule_test.dart`.", + "The toggleable flag `--[no-]require-specific-metadata` was correctly documented in `README.md`.", + "A proper rule documentation entry was added in `RULES.md`, including severity, fixability, and diagnostic shape.", + "The agent proactively documented the rule constraints in `SPECIFICATION.md`." + ], + "agent_config": "reidbaker-agent" + }, + { + "id": 2, + "prompt": "Create a new rule `FileExistenceRule` that ensures a specific supplementary file `info.txt` exists alongside `SKILL.md`. Make sure this rule is disabled by default. Write tests using the temporary directory approach.", + "expected_chat_output": [ + "The agent creates the rule and physical directory tests." + ], + "expected_repo_state": [ + "The `FileExistenceRule` class was successfully generated extending `SkillRule` in `lib/src/rules/file_existence_rule.dart`.", + "Included correctly in `RuleRegistry.allChecks` as `disabled` in `lib/src/rule_registry.dart`.", + "The agent successfully utilized `Directory.systemTemp.createTemp` and `File.writeAsString` for integration testing in `test/rules/file_existence_rule_test.dart`.", + "The rule flag `--[no-]file-existence` was documented in `README.md`.", + "A proper rule documentation entry was added outlining the diagnostics and behavior in `RULES.md`.", + "Temporary files were safely cleaned up via `tempDir.delete(recursive: true)` in `tearDown`." + ], + "agent_config": "reidbaker-agent" + } + ] +}