Skip to content

Sync .github/skills directory with azure-sdk-tools for PR 16603 - #49990

Open
helen229 wants to merge 1 commit into
mainfrom
sync-skills-pr16603-manual
Open

Sync .github/skills directory with azure-sdk-tools for PR 16603#49990
helen229 wants to merge 1 commit into
mainfrom
sync-skills-pr16603-manual

Conversation

@helen229

Copy link
Copy Markdown
Member

Manual catch-up sync of all azsdk-common-* skill eval specs to match azure-sdk-tools main. This repo's copy had drifted significantly further than the other downstream repos (6 trigger.eval.yaml files were missing entirely, plus 4 SKILL.md/eval.yaml files were stale versions), from several unsynced azure-sdk-tools PRs. This unblocks the automated patch-based sync-skills validation on Azure/azure-sdk-tools#16603, which fails via git am because of this pre-existing content divergence across all downstream repos.

Copilot AI review requested due to automatic review settings July 29, 2026 20:33
@helen229
helen229 requested a review from a team as a code owner July 29, 2026 20:33
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
35 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR performs a manual catch-up sync of several azsdk-common-* GitHub Copilot skill definitions and evaluation specs under .github/skills/ to align with upstream azure-sdk-tools changes, aiming to eliminate downstream drift that breaks patch-based sync validation.

Changes:

  • Expanded trigger phrases and examples in shared skill docs (notably SDK release + release planning) to cover “how-to” / checklist-style prompts.
  • Updated multiple skill eval specs (notably trigger.eval.yaml) to the newer defaults: schema and added per-stimulus environment.skills blocks.
  • Tightened routing/anti-trigger eval coverage by explicitly loading competing skills in some anti-trigger scenarios.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
.github/skills/azsdk-common-sdk-release/SKILL.md Broadens triggers/examples to include release checklists and process walkthrough prompts.
.github/skills/azsdk-common-sdk-release/evals/trigger.eval.yaml Updates trigger/anti-trigger routing evals and adds explicit skill loading per stimulus.
.github/skills/azsdk-common-prepare-release-plan/SKILL.md Refines description/triggers and simplifies create-plan guidance to match updated release-plan semantics.
.github/skills/azsdk-common-prepare-release-plan/evals/trigger.eval.yaml Updates routing evals and adds a tool-call-based assertion for linking SDK PRs.
.github/skills/azsdk-common-pipeline-fixer/evals/trigger.eval.yaml Updates routing evals and adds explicit cross-skill anti-trigger assertions.
.github/skills/azsdk-common-pipeline-analysis/evals/trigger.eval.yaml Updates routing evals and adds explicit cross-skill anti-trigger assertions.
.github/skills/azsdk-common-generate-sdk-locally/SKILL.md Adds guardrails clarifying local single-language generation vs pipeline-based multi-language generation.
.github/skills/azsdk-common-generate-sdk-locally/evals/eval.yaml Updates eval schema and adds a pipeline-generation negative test expecting azsdk_run_generate_sdk.
.github/skills/azsdk-common-apiview-feedback-resolution/evals/trigger.eval.yaml Improves trigger prompts (adds explicit APIView URLs) and adds per-stimulus skill loading.
.github/skills/azsdk-common-apiview-feedback-resolution/evals/eval.yaml Adds skill-loading anchors and strengthens tool-call assertions for APIView workflows.
Comments suppressed due to low confidence (1)

.github/skills/azsdk-common-pipeline-analysis/evals/trigger.eval.yaml:112

  • This eval references ../../markdown-token-optimizer and asserts it is required, but .github/skills/markdown-token-optimizer is not present in this repo. That will prevent the eval from loading the referenced skill and make the anti-trigger assertion fail. Either sync/add the missing skill(s) or update the eval to avoid depending on a missing skill.
  - name: anti-trigger-optimize-markdown-tokens
    prompt: "optimize markdown tokens"
    environment:
      skills:
        - ..
        - ../../markdown-token-optimizer
    graders:
      - type: skill-invocation
        config:
          required: ["markdown-token-optimizer"]
          disallowed: ["azsdk-common-pipeline-analysis"]

Comment on lines 96 to 106
- name: anti-trigger-optimize-markdown-tokens
prompt: "optimize markdown tokens"
environment:
skills:
- ..
- ../../markdown-token-optimizer
graders:
- type: skill-invocation
config:
required: ["markdown-token-optimizer"]
disallowed: ["azsdk-common-sdk-release"]
Comment on lines 33 to 43
- name: sdk-local-negative-001
prompt: "I want to use the Azure DevOps pipeline to generate SDKs automatically and have PRs created for me. I don't want to set up local tools."
prompt: "Run the Azure DevOps SDK generation pipeline for the Python SDK for release plan work item 12345. The TypeSpec project is specification/contosowidgetmanager/Contoso.WidgetManager, the API version is 2024-01-01-preview, the SDK release type is beta, and the service is data plane. I do not have a local SDK clone; create the generated SDK PR."
environment:
skills:
- ..
- ../../azsdk-common-generate-sdk-pipeline
graders:
- type: skill-invocation
config:
disallowed: ["azsdk-common-generate-sdk-locally"]
- type: tool-calls
Comment on lines 117 to 127
- name: anti-trigger-create-a-new-skill
prompt: "create a new skill"
environment:
skills:
- ..
- ../../skill-authoring
graders:
- type: skill-invocation
config:
required: ["skill-authoring"]
disallowed: ["azsdk-common-pipeline-fixer"]
Comment on lines 91 to 102
- name: anti-trigger-create-a-new-skill
prompt: "create a new skill"
environment:
skills:
- ..
- ../../skill-authoring
graders:
- type: skill-invocation
config:
required: ["skill-authoring"]
disallowed: ["azsdk-common-pipeline-analysis"]
- name: anti-trigger-optimize-markdown-tokens
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants