Skip to content

feat(ai-eval): add E2E failure evaluation skill and Slack formatter - #625

Draft
amisskii wants to merge 1 commit into
podman-desktop:mainfrom
amisskii:feat/ai-eval-skill
Draft

feat(ai-eval): add E2E failure evaluation skill and Slack formatter#625
amisskii wants to merge 1 commit into
podman-desktop:mainfrom
amisskii:feat/ai-eval-skill

Conversation

@amisskii

@amisskii amisskii commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Adds Claude Code skill (ai-eval/skills/evaluate-e2e-results/SKILL.md) for automated classification of Playwright E2E test failures
  • Adds Slack Block Kit formatter (scripts/slack_formatter.py) as optional post-processing step

Evaluation skill

  • Parses json-results.json, error-context.md, and Playwright trace archives
  • 3-layer analysis: error-context.md → json-results.json errors → trace analysis
  • Classifies failures as regression | test_bug | flaky | infra | unknown with confidence levels
  • Evidence-grounded root causes — escalates to trace analysis when Layer 1 shows WHAT failed but not WHY
  • Skipped test analysis — distinguishes by-design skips from cascade skips caused by failures
  • Handles missing json-results.json gracefully (falls back to JUnit XML)
  • Designed for --print mode with read-only tools in CI pipelines

Slack formatter

  • Transforms evaluation JSON into Slack Block Kit message
  • Shows classification emoji, root cause, recommended action per failure
  • Cascade skip summary when failures cause additional test skips
  • Links to artifacts and CI run

Test plan

  • Run skill locally against sample test artifacts with claude --print --append-system-prompt-file
  • Verify JSON output is valid and parseable
  • Verify Slack formatter produces valid Block Kit JSON
  • Test with all-pass, single-failure, and multi-failure test runs

Related: containers/podman-desktop-internal#1030

🤖 Generated with Claude Code

Adds Claude Code skill for automated classification of Playwright E2E
test failures in CI pipelines, with a companion Slack Block Kit formatter.

The skill analyzes json-results.json, error-context.md, and trace files
to classify failures as regression/test_bug/flaky/infra with evidence-
grounded root causes. Designed for --print mode in Tekton pipelines.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant