Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 27 additions & 3 deletions .github/scripts/probe_issue.py
Original file line number Diff line number Diff line change
Expand Up @@ -308,6 +308,17 @@ def test_deploy(charm, juju: jubilant.Juju):
`https://raw.githubusercontent.com/canonical/jubilant/v1.12.0/jubilant/_juju.py`). \
For release notes or PRs, use `github.com` URLs. The `fetch_url` tool is \
allowlisted to these domains and returns up to 64KB of text per request.

**Prioritize charm-dev tools over standard tools.** The claim is usually \
about how a charm-dev tool (Jubilant, ops, pytest-jubilant) behaves, not \
about how pytest or Python logging works in general. Understand the \
charm-dev tool first — read its release notes, source, and recent PRs to \
see what changed — then dig into standard tools (pytest, logging) only if \
the mechanism still isn't clear. For example, if a claim is about Jubilant \
log output, start by checking what changed in the relevant Jubilant \
version's release notes and `wait()` method, not by reading pytest's \
logging plugin source. The charm-dev tool is the thing that changed; \
pytest's behaviour is the stable backdrop.
"""


Expand Down Expand Up @@ -357,6 +368,16 @@ def test_deploy(charm, juju: jubilant.Juju):
version and test against it. Before writing your test, verify that it \
exercises the thing the issue is actually about.

**Understand the charm-dev tool before the standard tool.** If the claim \
involves a charm-dev tool (Jubilant, ops, pytest-jubilant), understand what \
that tool does and what changed recently before diving into standard tools \
like pytest or Python logging. The charm-dev tool is the thing that likely \
changed; the standard tool is the stable backdrop. For example, if a claim \
is about Jubilant log output and the issue references a version, check the \
Jubilant release notes and `wait()` source first — that's where the \
behavioural change lives. Only then, if the mechanism still isn't clear, \
read pytest's logging plugin to understand how the config interacts.

**Do not be shy about integration tests.** `run_tox` runs `format,lint,unit` \
only — not integration tests. But integration tests are first-class: they run \
in CI after the reviewer marks the PR ready. Write them when the claim is \
Expand Down Expand Up @@ -472,6 +493,11 @@ def test_deploy(charm, juju: jubilant.Juju):
is true, what the PR tests, and what green (or red) CI means for each claim. \
Use proper markdown: headers (`##`), bullet points, code blocks (fenced \
with triple backticks), and paragraphs separated by blank lines.
- **Do not hard-wrap lines.** Write each paragraph as a single long line — \
the markdown renderer handles wrapping. Hard-wrapped lines break rendering \
and make editing harder.
- **Do not include a "Changes" or "Files changed" section.** The reviewer \
can see the diff in GitHub. Focus on reasoning, not a file listing.

Example `.PR.md`:

Expand All @@ -483,9 +509,7 @@ def test_deploy(charm, juju: jubilant.Juju):
- **A**: log_level=INFO retains INFO logs in the captured section.
- **B**: without it, DEBUG logs appear from log_file_level.

I believe the doc is correct. I added a test asserting that no DEBUG records \
appear in caplog when log_level=INFO is set. If CI passes, the doc is \
validated. If CI fails, the doc is refuted.
I believe the doc is correct. I added a test asserting that no DEBUG records appear in caplog when log_level=INFO is set. If CI passes, the doc is validated. If CI fails, the doc is refuted.
```

The reasoning is a core part of the adversarial approach: the reviewer needs \
Expand Down