diff --git a/.claude/skills/qe-on-duty-triage/SKILL.md b/.claude/skills/qe-on-duty-triage/SKILL.md new file mode 100644 index 0000000..10f0392 --- /dev/null +++ b/.claude/skills/qe-on-duty-triage/SKILL.md @@ -0,0 +1,290 @@ +--- +name: qe-on-duty-triage +description: >- + Interactively triage a QE on-duty report generated by qe-on-duty-reporter — + walk through failed CI runs, PRs needing QE review, CVEs, and open bugs, and + dispatch the right skill (investigate-gh-run, issue-requirements, + create-github-issue, etc.) for whichever item the user picks. Use when the + user wants to work through a QE on-duty report, triage failed runs, or + decide what to do next after running the qe-on-duty-reporter. Requires that + a report has already been generated and that the delegate skills are + imported — this skill does not investigate anything itself, it routes. +--- + +# QE On-Duty Triage + +Router/dispatcher skill. It does not investigate CI failures, summarize +issues, or draft anything itself — it reads the QE on-duty snapshot, lets the +user pick an actionable item, then hands off to the specialized skill that +actually does the work. If a needed skill isn't available, say so explicitly +instead of improvising the work yourself. + +The user may pass `$ARGUMENTS`: a date (`2026-08-11`), a specific snapshot +path, `weekly`, or nothing (defaults to the latest daily snapshot). + +## Step 0: Confirm the delegate skills are actually available + +This skill only ever *offers* the following skills — never reimplement their +work inline: + +| Item type | Delegate skill | +| ----------------------- | ---------------------- | +| Failed workflow run | `investigate-gh-run` | +| Open bug / issue | `issue-requirements` | +| CVE needing a tracking issue | `create-github-issue` | +| Test bug found by an investigation | `playwright-testing` (cross-repo follow-up, see Step 7) | + +Check the current list of available skills (shown to you in the system +reminder / skills listing for this session). Note which of the three above +are present. If one is missing, you may still list items of that type, but +when the user picks an action that needs the missing skill, tell them plainly +it isn't imported and cannot be dispatched — do not fake the investigation +yourself and do not silently skip the item. + +## Step 1: Locate the report and snapshot + +The reporter lives at `qe-on-duty-reporter/`. Reports and snapshots are +paired by date/time: + +- Snapshot (structured data): `qe-on-duty-reporter/output/snapshots/YYYY-MM-DD/HHMM.json` +- Report (human-readable): `qe-on-duty-reporter/output/reports/daily/YYYY-MM-DD/HHMM.md` + +Resolve which snapshot to use: + +- No argument → latest snapshot overall: + ```bash + find qe-on-duty-reporter/output/snapshots -name '*.json' -not -name 'latest.json' | sort | tail -1 + ``` +- A date like `2026-08-11` → latest snapshot for that date: + ```bash + find qe-on-duty-reporter/output/snapshots/2026-08-11 -name '*.json' -not -name 'latest.json' | sort | tail -1 + ``` +- A path → use it directly. +- `weekly` → look under `qe-on-duty-reporter/output/reports/weekly/` instead; that report is prose-only (no JSON snapshot), so read the markdown directly and skip Step 2's structured parsing. + +**If no snapshot/report exists**, stop and tell the user to run the reporter +first (`cd qe-on-duty-reporter && python main.py`), per `README.md`. Don't +fabricate data. + +## Step 2: Load and categorize (daily reports) + +Use the project's own dataclasses to parse the snapshot — don't hand-roll +JSON parsing, the model already normalizes fields and drops stale keys: + +```bash +cd qe-on-duty-reporter && python3 -c " +from qe_on_duty.models.snapshot import DailySnapshot +s = DailySnapshot.load('') + +failed = [w for w in s.workflows if w.conclusion == 'failure'] +prs = s.prs +cves = [c for c in s.cves if c.state == 'open' and c.severity in ('critical', 'high')] +bugs = s.issues + +print(f'Failed workflow runs: {len(failed)}') +print(f'PRs needing QE review: {len(prs)}') +print(f'Critical/High CVEs: {len(cves)}') +print(f'Open bugs: {len(bugs)}') +" +``` + +This gives you the counts. Keep the loaded categories in mind for Steps 3-4 +(you'll re-run similar snippets to print the actual items once a category is +chosen). + +## Step 3: Ask which category to triage + +Use `AskUserQuestion` with one option per **non-empty** category (skip +categories with zero items; if only one category is non-empty, skip the +question and go straight to Step 4 for it). Example options: + +- Failed CI runs (N) — investigate failing overnight workflow runs +- PRs needing QE review (N) — pull requests awaiting review/testing +- CVEs (N) — open critical/high security alerts +- Open bugs (N) — issues to gather context on + +## Step 4: List items in the chosen category + +Print a numbered list grouped by repository (mirror the report's grouping). +Keep each line to the essentials: number, repo, title/name, and link. + +```bash +python3 -c " +from qe_on_duty.models.snapshot import DailySnapshot +s = DailySnapshot.load('') +failed = [w for w in s.workflows if w.conclusion == 'failure'] +for i, w in enumerate(failed, 1): + print(f'{i}. [{w.repository}] {w.workflow_name} (#{w.run_number}) - {w.url}') +" +``` + +Since these lists can be long (20+ items), do **not** use `AskUserQuestion` +for item selection — it caps at 4 options. Ask in plain text: "Which item(s) +would you like to act on? (number, comma-separated numbers, 'all', or +'skip')". Multiple items are supported — see Step 5a for how they're +dispatched. + +## Step 5: Offer an action for the selected item(s) + +Match the item type to its action menu (≤4 options, use `AskUserQuestion`). +The menu is the same whether one or several items were picked — the fan-out +logic (Step 5a) only applies once the user picks an investigate/gather-style +action. + +**Failed workflow run:** +- Investigate root cause → first check whether a tracking issue already exists for this failure (`gh issue list --repo --search "" --state all`); if one is found, surface it up front so the investigation can confirm/extend it rather than duplicate it. Then delegate to `investigate-gh-run` with the run URL as `$ARGUMENTS` +- Open the run link only (no investigation) +- Skip / already known (e.g. known flaky or infra issue) + +**PR needing QE review:** +- Open the PR for manual review (print the URL / `gh pr view --repo --web`) +- Investigate failing checks on this PR → delegate to `investigate-gh-run` (find the relevant run: `gh pr checks --repo `, take the failing run's URL) +- Skip + +**CVE:** +- View the alert (print the URL / package / versions) +- File a tracking issue → delegate to `create-github-issue`, but first check there isn't already one open for this CVE/package in that repo (`gh issue list --repo --search ""`) +- Skip + +**Open bug:** +- Gather full requirements/context → delegate to `issue-requirements` with the issue reference +- Skip + +If the chosen action needs a delegate skill that Step 0 found missing, stop +and tell the user instead of proceeding. + +## Step 5a: Multiple items — dedupe, cap, and parallel dispatch + +This only applies when the user selected **more than one item** and picked an +**investigate/gather-style action**. It fans out to parallel subagents so N +runs/issues get investigated concurrently instead of one at a time. + +**Only two delegate skills are safe to fan out this way:** + +- `investigate-gh-run` (failed workflow runs) +- `issue-requirements` (open bugs) + +**Never fan out `create-github-issue`.** That skill has a hard rule to stop +and wait for the user to confirm a draft before creating anything. A +background subagent can't get that confirmation — it would either stall +forever or (worse) someone "fixes" it by skipping the confirmation gate. If +the user picks "file a tracking issue" for multiple CVEs, handle them +**sequentially in the main conversation**, one draft-and-confirm cycle at a +time — do not spawn agents for this. + +For the two safe cases: + +1. **Dedupe first.** Multiple failed runs often share one root cause — the + same test failing across a platform matrix (Win10/Win11/ARM), or the same + workflow re-triggered a few times. Group the selected runs by + `workflow_name` (and note if `run_number`s are close together / same + branch). Tell the user which runs you're treating as duplicates and that + you'll investigate one representative per group, applying the finding to + the rest. Only dispatch one agent per distinct group. + +2. **Cap and confirm before dispatching.** If more than ~5 distinct + groups/items remain after deduping, tell the user how many agents that + implies and ask them to confirm or narrow the selection first — each + agent makes its own round of `gh`/GitHub API calls, and that many + concurrent investigations can trip secondary rate limits and burns real + token/cost budget. + +3. **Dispatch in parallel.** Launch one `Agent` call per distinct item, all + in a single message so they run concurrently. Each prompt should be + self-contained (the agent starts with no context): + + ``` + Agent({ + description: "Investigate failed run #", + prompt: "Use the investigate-gh-run skill to investigate this CI failure: . Report back: summary, root cause, classification (App bug | Test bug | Likely flaky | Environment/infra), and recommended action with file:line if applicable." + }) + ``` + + (Swap the skill name/URL for `issue-requirements` + issue reference when + triaging bugs instead.) + +4. **Aggregate on completion.** Each agent's report is not shown to the user + automatically — when all finish, collect their findings yourself and + present one combined summary (a table: item → root cause → classification + → recommended action), noting which entries were deduped together. + +## Step 6: Execute and loop + +For a **single** selected item, invoke the delegate skill directly via the +`Skill` tool with the item's URL/reference as `args` — no need to spawn a +subagent for one item. For simple non-delegated actions (open link, print +alert details, `gh pr view`), just run the read-only `gh`/print command +directly — no skill needed for those. For **multiple** items with an +investigate/gather action, follow Step 5a instead. + +After finishing, ask whether the user wants to triage another item (same or +different category) or stop. If an `investigate-gh-run` result comes back +classified as a **Test bug** with a known file:line, offer Step 7 before +looping. Otherwise loop back to Step 3/4 as needed rather than re-reading the +snapshot from scratch each time. + +## Step 7: Follow-up — fixing a test bug lives in a different repo + +`playwright-testing` covers the Podman Desktop Playwright framework, but the +failing test almost never lives in *this* repo (`podman-desktop-e2e`, which +just hosts the reporter) — it lives in whichever repo the failed workflow +belongs to (`podman-desktop/podman-desktop`, `podman-desktop/e2e`, an +extension repo, etc.). That's a different git repo, with its own history and +its own blast radius for edits/commits, so this step always confirms with the +user before touching it. + +1. **Resolve a local clone.** Repos in this environment are cloned as + siblings under one parent directory, named after the short repo name + (e.g. `podman-desktop/extension-minikube` → a directory named + `extension-minikube`). Find the parent and search it: + ```bash + PARENT=$(dirname "$(git rev-parse --show-toplevel)") + SHORT_NAME=$(basename "") + find "$PARENT" -maxdepth 1 -type d -iname "$SHORT_NAME" + ``` + +2. **If a local clone is found**, ask the user (`AskUserQuestion`) how to + proceed — do not switch or edit anything without this confirmation: + - **Fix it here now** — work directly in that repo for the rest of this + step (absolute paths / `git -C ` so nothing in the current repo's + context gets confused), then invoke the `playwright-testing` skill with + the failure details (file:line, error, run URL) as context. Stop after + proposing/applying the fix — do not commit unless the user explicitly + asks, same as any other change. + - **Just give me the command** — print a copy-paste block for the user to + run in their own terminal/session instead, e.g.: + ``` + cd && claude + ``` + followed by a ready-made prompt summarizing the failure (test name, + file:line, error, run URL) for them to paste in that session. + - **Skip** — move on. + +3. **If no local clone is found**, say so and offer (don't run without + confirmation) a clone command as a sibling directory: + ```bash + git clone git@github.com:/.git "$PARENT/" + ``` + Once cloned (or if the user declines), fall back to the "just give me the + command" option above. + +## Hard rules + +1. Never perform the actual investigation/analysis yourself when a delegate + skill exists for it — dispatch to it. This skill's job is routing, not + doing the work twice. +2. Never invent report data. If the snapshot is missing a field or a category + is empty, say so. +3. Never silently skip an action because its delegate skill is unavailable — + tell the user explicitly. +4. Don't page through hundreds of items eagerly — list one category at a + time, per Step 3. +5. Never fan `create-github-issue` out to background subagents — it requires + per-item user confirmation before creating anything. Multiple CVEs picked + for issue-filing are handled one at a time in the main conversation. +6. Never dispatch parallel agents without deduping near-identical failures + first and confirming the count with the user above ~5 groups. +7. Never switch into or edit another repo (Step 7) without first confirming + the path with the user, and never commit there unless explicitly asked — + same rule as this repo, just easier to forget when you've changed + directory.