Sync .github/skills directory with azure-sdk-tools for PR 16603 - #49990
Open
helen229 wants to merge 1 commit into
Open
Sync .github/skills directory with azure-sdk-tools for PR 16603#49990helen229 wants to merge 1 commit into
helen229 wants to merge 1 commit into
Conversation
|
Azure Pipelines: 35 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR performs a manual catch-up sync of several azsdk-common-* GitHub Copilot skill definitions and evaluation specs under .github/skills/ to align with upstream azure-sdk-tools changes, aiming to eliminate downstream drift that breaks patch-based sync validation.
Changes:
- Expanded trigger phrases and examples in shared skill docs (notably SDK release + release planning) to cover “how-to” / checklist-style prompts.
- Updated multiple skill eval specs (notably
trigger.eval.yaml) to the newerdefaults:schema and added per-stimulusenvironment.skillsblocks. - Tightened routing/anti-trigger eval coverage by explicitly loading competing skills in some anti-trigger scenarios.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| .github/skills/azsdk-common-sdk-release/SKILL.md | Broadens triggers/examples to include release checklists and process walkthrough prompts. |
| .github/skills/azsdk-common-sdk-release/evals/trigger.eval.yaml | Updates trigger/anti-trigger routing evals and adds explicit skill loading per stimulus. |
| .github/skills/azsdk-common-prepare-release-plan/SKILL.md | Refines description/triggers and simplifies create-plan guidance to match updated release-plan semantics. |
| .github/skills/azsdk-common-prepare-release-plan/evals/trigger.eval.yaml | Updates routing evals and adds a tool-call-based assertion for linking SDK PRs. |
| .github/skills/azsdk-common-pipeline-fixer/evals/trigger.eval.yaml | Updates routing evals and adds explicit cross-skill anti-trigger assertions. |
| .github/skills/azsdk-common-pipeline-analysis/evals/trigger.eval.yaml | Updates routing evals and adds explicit cross-skill anti-trigger assertions. |
| .github/skills/azsdk-common-generate-sdk-locally/SKILL.md | Adds guardrails clarifying local single-language generation vs pipeline-based multi-language generation. |
| .github/skills/azsdk-common-generate-sdk-locally/evals/eval.yaml | Updates eval schema and adds a pipeline-generation negative test expecting azsdk_run_generate_sdk. |
| .github/skills/azsdk-common-apiview-feedback-resolution/evals/trigger.eval.yaml | Improves trigger prompts (adds explicit APIView URLs) and adds per-stimulus skill loading. |
| .github/skills/azsdk-common-apiview-feedback-resolution/evals/eval.yaml | Adds skill-loading anchors and strengthens tool-call assertions for APIView workflows. |
Comments suppressed due to low confidence (1)
.github/skills/azsdk-common-pipeline-analysis/evals/trigger.eval.yaml:112
- This eval references
../../markdown-token-optimizerand asserts it is required, but.github/skills/markdown-token-optimizeris not present in this repo. That will prevent the eval from loading the referenced skill and make the anti-trigger assertion fail. Either sync/add the missing skill(s) or update the eval to avoid depending on a missing skill.
- name: anti-trigger-optimize-markdown-tokens
prompt: "optimize markdown tokens"
environment:
skills:
- ..
- ../../markdown-token-optimizer
graders:
- type: skill-invocation
config:
required: ["markdown-token-optimizer"]
disallowed: ["azsdk-common-pipeline-analysis"]
Comment on lines
96
to
106
| - name: anti-trigger-optimize-markdown-tokens | ||
| prompt: "optimize markdown tokens" | ||
| environment: | ||
| skills: | ||
| - .. | ||
| - ../../markdown-token-optimizer | ||
| graders: | ||
| - type: skill-invocation | ||
| config: | ||
| required: ["markdown-token-optimizer"] | ||
| disallowed: ["azsdk-common-sdk-release"] |
Comment on lines
33
to
43
| - name: sdk-local-negative-001 | ||
| prompt: "I want to use the Azure DevOps pipeline to generate SDKs automatically and have PRs created for me. I don't want to set up local tools." | ||
| prompt: "Run the Azure DevOps SDK generation pipeline for the Python SDK for release plan work item 12345. The TypeSpec project is specification/contosowidgetmanager/Contoso.WidgetManager, the API version is 2024-01-01-preview, the SDK release type is beta, and the service is data plane. I do not have a local SDK clone; create the generated SDK PR." | ||
| environment: | ||
| skills: | ||
| - .. | ||
| - ../../azsdk-common-generate-sdk-pipeline | ||
| graders: | ||
| - type: skill-invocation | ||
| config: | ||
| disallowed: ["azsdk-common-generate-sdk-locally"] | ||
| - type: tool-calls |
Comment on lines
117
to
127
| - name: anti-trigger-create-a-new-skill | ||
| prompt: "create a new skill" | ||
| environment: | ||
| skills: | ||
| - .. | ||
| - ../../skill-authoring | ||
| graders: | ||
| - type: skill-invocation | ||
| config: | ||
| required: ["skill-authoring"] | ||
| disallowed: ["azsdk-common-pipeline-fixer"] |
Comment on lines
91
to
102
| - name: anti-trigger-create-a-new-skill | ||
| prompt: "create a new skill" | ||
| environment: | ||
| skills: | ||
| - .. | ||
| - ../../skill-authoring | ||
| graders: | ||
| - type: skill-invocation | ||
| config: | ||
| required: ["skill-authoring"] | ||
| disallowed: ["azsdk-common-pipeline-analysis"] | ||
| - name: anti-trigger-optimize-markdown-tokens |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Manual catch-up sync of all azsdk-common-* skill eval specs to match azure-sdk-tools main. This repo's copy had drifted significantly further than the other downstream repos (6 trigger.eval.yaml files were missing entirely, plus 4 SKILL.md/eval.yaml files were stale versions), from several unsynced azure-sdk-tools PRs. This unblocks the automated patch-based sync-skills validation on Azure/azure-sdk-tools#16603, which fails via git am because of this pre-existing content divergence across all downstream repos.