Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -183,7 +183,9 @@ If the rule interacts with CLI flags or configuration files, add a test in `test
When a new rule is introduced, verify that you synchronize sibling markdown files!

1. **`README.md`:**
* Add your flag under the **Usage** and **Flags** sections so users know it exists.
* Add your flag under the **Flags** section (under **Usage**) so users know it exists.
* **CRITICAL FORMATTING:** You MUST use the exact format `- \`--[no-]<rule-name>\`: <brief description>. (Disabled by default if applicable)`.
* **CRITICAL NAMING:** Ensure the flag string matches the `ruleName` EXACTLY. For example, if the `ruleName` is `file-existence`, the flag MUST be documented as `--[no-]file-existence` (do NOT hallucinate a `check-` prefix like `--[no-]check-file-existence`). Do NOT add empty bullet points.
2. **`RULES.md`:**
* Add a new entry for your rule documenting its default severity, fixability, what it checks, diagnostic shape, auto-fix behavior, and how to disable it. This is strictly required by the `rules_md_consistency_test.dart` test.
3. **`documentation/knowledge/SPECIFICATION.md`:**
Expand All @@ -196,7 +198,7 @@ When a new rule is introduced, verify that you synchronize sibling markdown file
- [ ] Rule class created in `lib/src/rules/`.
- [ ] Rule registered in `lib/src/rule_registry.dart`.
- [ ] Unit tests added in `test/` using in-memory `SkillContext`.
- [ ] Usage listed in `README.md`.
- [ ] **CRITICAL**: Usage flag correctly documented in `README.md` under Flags (ensure flag string matches `ruleName` EXACTLY and format is correct).
- [ ] Rule documented in `RULES.md`.
- [ ] Schema documented in `documentation/knowledge/SPECIFICATION.md` (if applicable).
- [ ] Run `dart format .` to format code.
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
{
"repo_criteria": [
"evals/code_quality_rubric.json"
],
"evals": [
{
"id": 1,
"prompt": "Author a custom dart_skills_lint rule named `RequireSpecificMetadataRule` that verifies if a skill's metadata contains a `required_version` field. Wire it up into a new test file `test/require_specific_metadata_rule_test.dart` using an in-memory SkillContext.",
"expected_chat_output": [
"The agent identifies the skill, spawns a with-skill subagent, and successfully implements the rule."
],
"expected_repo_state": [
"The agent successfully authored `RequireSpecificMetadataRule` extending `SkillRule` in `lib/src/rules/require_specific_metadata_rule.dart`.",
"The rule was properly added to `RuleRegistry.allChecks` and the `createRule` switch statement in `lib/src/rule_registry.dart`.",
"The `validate` method correctly parses `context.parsedYaml` to check for `required_version` in the metadata.",
"The agent authored unit tests utilizing an in-memory raw string, `loadYaml`, and a mock `SkillContext` in `test/require_specific_metadata_rule_test.dart`.",
"The toggleable flag `--[no-]require-specific-metadata` was correctly documented in `README.md`.",
"A proper rule documentation entry was added in `RULES.md`, including severity, fixability, and diagnostic shape.",
"The agent proactively documented the rule constraints in `SPECIFICATION.md`."
],
"agent_config": "reidbaker-agent"
},
{
"id": 2,
"prompt": "Create a new rule `FileExistenceRule` that ensures a specific supplementary file `info.txt` exists alongside `SKILL.md`. Make sure this rule is disabled by default. Write tests using the temporary directory approach.",
"expected_chat_output": [
"The agent creates the rule and physical directory tests."
],
"expected_repo_state": [
"The `FileExistenceRule` class was successfully generated extending `SkillRule` in `lib/src/rules/file_existence_rule.dart`.",
"Included correctly in `RuleRegistry.allChecks` as `disabled` in `lib/src/rule_registry.dart`.",
"The agent successfully utilized `Directory.systemTemp.createTemp` and `File.writeAsString` for integration testing in `test/rules/file_existence_rule_test.dart`.",
"The rule flag `--[no-]file-existence` was documented in `README.md`.",
"A proper rule documentation entry was added outlining the diagnostics and behavior in `RULES.md`.",
"Temporary files were safely cleaned up via `tempDir.delete(recursive: true)` in `tearDown`."
],
"agent_config": "reidbaker-agent"
}
]
}
Loading