feat: add an ai-failure-notifier workflow and script

.github/workflows/ci.yaml

@@ -0,0 +1,45 @@
+name: Tests
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+permissions: {}
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        # One entry per tool. Add a directory here when you add a package.
+        package: [ai-failure-notifier]
+        python-version: ['3.10', '3.12', '3.14']
+    defaults:
+      run:
+        working-directory: ${{ matrix.package }}
+    steps:
+      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0  # v7.0.0
+        with:
+          persist-credentials: false
+      - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39  # v8.2.0
+        with:
+          python-version: ${{ matrix.python-version }}
+      # --locked, so that a lockfile which no longer matches its pyproject.toml
+      # fails here rather than silently resolving to something else.
+      - run: uv sync --locked --group unit
+      - run: uv run pytest
+
+  lint:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0  # v7.0.0
+        with:
+          persist-credentials: false
+      - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39  # v8.2.0
+      # Ruff is pinned in the root dependency group, and its configuration
+      # lives there too, so both run once across every package rather than
+      # per matrix entry.
+      - run: uv run --group lint ruff check .
+      - run: uv run --group lint ruff format --check .

.gitignore

@@ -216,3 +216,6 @@ __marimo__/
 
 # Streamlit
 .streamlit/secrets.toml
+
+# uv
+.venv/

README.md

@@ -0,0 +1,40 @@
+# charm-tech-code
+
+Shared tooling for the Charm Tech repositories (`operator`, `charmlibs`, `jubilant`, `pebble`, `concierge`, and the rest of the estate).
+
+Each tool is its own package in its own top-level directory, with its own `pyproject.toml`, `src/`, `tests/` and lockfile - the same shape `canonical/charmlibs` uses. Adding a tool means adding a directory, not adding to an existing package, so a tool's dependencies are paid only by the workflows that run that tool.
+
+| directory | what it does |
+|---|---|
+| [`ai-failure-notifier`](ai-failure-notifier) | Triages and enriches the issue opened when a scheduled workflow fails. |
+
+Code here is consumed by workflow YAML in the repository that runs it, pinned by commit SHA:
+
+```yaml
+run: uvx --from "git+https://github.com/canonical/charm-tech-code@<40-char-sha>#subdirectory=ai-failure-notifier" ai-failure-notifier
+```
+
+The point is that the code lives in one place. A tool used by eleven repositories should be fixed once, not eleven times, and the workflow YAML that differs per repository stays in that repository.
+
+There is no release process and nothing is published. The SHA in the `uvx` line is the version, which is the same trust decision every pinned `uses: actions/checkout@<sha>` line in those repositories already makes.
+
+## Configuration
+
+Ruff's configuration lives in the root `pyproject.toml` and is copied from `canonical/operator`, so that a file can move between the two repositories without being reformatted. Packages deliberately do not carry their own `[tool.ruff]` block: ruff uses the closest configuration it finds rather than merging, so a local one would silently override the shared one.
+
+`preview` is set in configuration rather than passed as `--preview` on the command line, which is how operator's `tox.ini` does it. That way an editor, a hook and CI agree without anyone having to remember the flag. It is load-bearing rather than cosmetic - the preview style hugs brackets inside calls, and without it a good deal of existing code reformats.
+
+## Developing
+
+```shell
+cd <package>
+uv sync --group unit
+uv run pytest
+```
+
+Lint and format run from the root, across every package at once:
+
+```shell
+uv run --group lint ruff check .
+uv run --group lint ruff format --check .
+```

ai-failure-notifier/README.md

@@ -0,0 +1,22 @@
+# ai-failure-notifier
+
+Enriches the placeholder issue that a scheduled workflow's failure notifier opens.
+
+When a scheduled workflow fails, the notifier in the repository opens an issue with a generic title and a link to the failing job. This tool picks that placeholder up, reads the failing run's job logs, reduces them to a deterministic failure signature, searches for issues that look like the same failure, and then either comments on the existing one or rewrites the placeholder with a real title, body and labels.
+
+Without an API key it falls back to a plain notification. That is deliberate: the notifier has to work when everything else is broken, so nothing here is allowed to be a hard dependency of it.
+
+## Running it
+
+```shell
+uvx --from "git+https://github.com/canonical/charm-tech-code@<40-char-sha>#subdirectory=ai-failure-notifier" ai-failure-notifier
+```
+
+It reads its inputs from the environment: `GH_TOKEN`, `REPO`, `RUN_ID`, `WORKFLOW_NAME`, `RUN_URL`, and optionally `OPENROUTER_API_KEY` and `OPENROUTER_MODEL`. See `canonical/operator`'s `.github/workflows/ai-failure-enrich.yaml` for the calling side, including the environment mechanics the key depends on.
+
+## Developing
+
+```shell
+uv sync --group unit
+uv run pytest
+```

ai-failure-notifier/pyproject.toml

@@ -0,0 +1,36 @@
+[project]
+name = "charm-tech-code-ai-failure-notifier"
+version = "0.1.0"
+description = "Triage and enrich the issues opened when a scheduled workflow fails."
+readme = "README.md"
+requires-python = ">=3.10"
+authors = [
+    {name = "The Charm Tech team at Canonical Ltd."},
+]
+license = "Apache-2.0"
+# Deliberately none. The tool is invoked by `uvx --from git+...` on a GitHub
+# runner, so every dependency added here is paid on every failed scheduled
+# run, in a workflow whose whole point is to still work when things are
+# broken. It talks to GitHub through `gh`, which the runner already has, and
+# to OpenRouter through urllib.
+dependencies = []
+
+[project.scripts]
+ai-failure-notifier = "charm_tech_code.ai_failure_notifier:main"
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/charm_tech_code"]
+
+[dependency-groups]
+unit = ["pytest"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+
+# Ruff configuration is at the root of the monorepo, deliberately not repeated
+# here: ruff uses the closest config it finds rather than merging, so a
+# [tool.ruff] block in this file would silently override the shared one.

ai-failure-notifier/src/charm_tech_code/ai_failure_notifier/__init__.py

@@ -0,0 +1,20 @@
+# Copyright 2026 Canonical Ltd.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+"""Triage and enrich the issue opened when a scheduled workflow fails."""
+
+from charm_tech_code.ai_failure_notifier.cli import main
+
+__all__ = ['main']

ai-failure-notifier/src/charm_tech_code/ai_failure_notifier/apply.py

@@ -0,0 +1,79 @@
+# Copyright 2026 Canonical Ltd.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+"""Writing the decision back to GitHub, and the no-LLM fallback."""
+
+from __future__ import annotations
+
+from typing import Any
+
+from charm_tech_code.ai_failure_notifier import github, summary
+
+
+def plain_fallback_body(workflow_name: str, run_url: str) -> str:
+    """The plain, generic body text used whenever enrichment is unavailable."""
+    return f"Scheduled workflow '{workflow_name}' failed: {run_url}"
+
+
+def render_body(body: str, workflow_name: str, marker: str) -> str:
+    """Assemble an issue or comment body, footer and marker included.
+
+    The `Workflow: <name>` footer is what keeps the notifier's coarse search
+    working after enrichment has rewritten the title and body: the search
+    matches on the workflow name, and without the footer it would depend on
+    the model happening to leave the name in the title.
+    """
+    return f'{body.rstrip()}\n\nWorkflow: {workflow_name}\n\n{marker}'
+
+
+def apply_entry(
+    repo: str,
+    entry: dict[str, Any],
+    marker: str,
+    workflow_name: str,
+    *,
+    default_target: int | None = None,
+) -> str:
+    """Create or comment on an issue per one envelope entry, stamping `marker`."""
+    body = render_body(entry['body'], workflow_name, marker)
+    if entry['action'] == 'new':
+        # The repo's label set is centrally managed, so anything the model
+        # asked for that doesn't exist is dropped rather than created.
+        labels = github.filter_labels(entry.get('labels') or [], github.existing_labels(repo))
+        dropped = set(entry.get('labels') or []) - set(labels)
+        if dropped:
+            summary.write_step_summary(
+                f'Dropped labels that do not exist in this repo: {", ".join(sorted(dropped))}.'
+            )
+        args = ['issue', 'create', '--repo', repo, '--title', entry['title'], '--body', body]
+        for label in labels:
+            args += ['--label', label]
+        issue_type = entry.get('issue_type')
+        result = None
+        if issue_type:
+            result = github.gh(*args, '--type', issue_type, check=False)
+            if result.returncode != 0:
+                summary.write_step_summary(
+                    f'`gh issue create --type {issue_type}` failed ({result.stderr.strip()}); '
+                    'retrying without --type.'
+                )
+                result = None
+        if result is None:
+            result = github.gh(*args)
+        return result.stdout.strip()
+    else:
+        target = entry.get('target_issue', default_target)
+        github.gh('issue', 'comment', str(target), '--repo', repo, '--body', body)
+        return f'commented on #{target}'

ai-failure-notifier/src/charm_tech_code/ai_failure_notifier/candidates.py

@@ -0,0 +1,69 @@
+# Copyright 2026 Canonical Ltd.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+"""Building the pool of issues a failure might already have."""
+
+from __future__ import annotations
+
+import datetime
+
+from charm_tech_code.ai_failure_notifier.constants import (
+    CLOSED_CANDIDATE_WINDOW_DAYS,
+    MAX_CANDIDATES,
+)
+from charm_tech_code.ai_failure_notifier.models import CandidateIssue
+
+
+def within_window(iso_timestamp: str, now: datetime.datetime, days: int) -> bool:
+    """Return whether `iso_timestamp` falls within `days` of `now`."""
+    ts = datetime.datetime.fromisoformat(iso_timestamp.replace('Z', '+00:00'))
+    return now - ts <= datetime.timedelta(days=days)
+
+
+def build_candidates_block(
+    open_issues: list[CandidateIssue],
+    closed_issues: list[CandidateIssue],
+    now: datetime.datetime,
+) -> str:
+    """Render the {{CANDIDATES_BLOCK}} the prompt expects.
+
+    Up to MAX_CANDIDATES entries: open issues first, then recently-closed
+    issues (<=14 days) filling any remaining slots, explicitly labelled as
+    closed so the LLM never auto-treats one as a strong match. Calibration on
+    past scheduled failures found a closed issue can corroborate a match but
+    should never be enough to dedupe against on its own.
+    """
+    entries: list[str] = []
+    for issue in open_issues:
+        if len(entries) >= MAX_CANDIDATES:
+            break
+        entries.append(f'- **#{issue.number} — {issue.title}** (open)\n  > {issue.excerpt()}')
+
+    recent_closed = [
+        i
+        for i in closed_issues
+        if i.closed_at and within_window(i.closed_at, now, CLOSED_CANDIDATE_WINDOW_DAYS)
+    ]
+    for issue in recent_closed:
+        if len(entries) >= MAX_CANDIDATES:
+            break
+        entries.append(
+            f'- **#{issue.number} — {issue.title}** (closed {issue.closed_at} -- '
+            f'recently closed; treat as at most a medium-confidence match)\n  > {issue.excerpt()}'
+        )
+
+    if not entries:
+        return '(no open issues found for this workflow)'
+    return '\n'.join(entries)