From 37ae6ebdaf6f72812bbd7962ea0e6e082f81c09e Mon Sep 17 00:00:00 2001 From: Dave Wilding Date: Tue, 25 Aug 2026 21:41:19 +0800 Subject: [PATCH] Tweak .PR.md format and prioritize charm-dev tools MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three changes to the prompt: 1. No hard-wrapping: instruct the agent to write each paragraph as a single long line. The markdown renderer handles wrapping. Hard-wrapped lines break rendering and make editing harder. 2. No Changes section: instruct the agent not to include a 'Changes' or 'Files changed' section. The reviewer can see the diff in GitHub. Focus on reasoning, not a file listing. 3. Prioritize charm-dev tools over standard tools: the claim is usually about how a charm-dev tool (Jubilant, ops, pytest-jubilant) behaves, not about how pytest or Python logging works in general. The agent should understand the charm-dev tool first (release notes, source, recent PRs) then dig into standard tools only if the mechanism still isn't clear. This would have helped the agent discover that Jubilant 1.12.0 moved verbose gron diffs to DEBUG and status changes to INFO (PR #351) — which is exactly what makes log_level=INFO meaningful. --- .github/scripts/probe_issue.py | 30 +++++++++++++++++++++++++++--- 1 file changed, 27 insertions(+), 3 deletions(-) diff --git a/.github/scripts/probe_issue.py b/.github/scripts/probe_issue.py index cb1662a..ebfdfac 100644 --- a/.github/scripts/probe_issue.py +++ b/.github/scripts/probe_issue.py @@ -308,6 +308,17 @@ def test_deploy(charm, juju: jubilant.Juju): `https://raw.githubusercontent.com/canonical/jubilant/v1.12.0/jubilant/_juju.py`). \ For release notes or PRs, use `github.com` URLs. The `fetch_url` tool is \ allowlisted to these domains and returns up to 64KB of text per request. + +**Prioritize charm-dev tools over standard tools.** The claim is usually \ +about how a charm-dev tool (Jubilant, ops, pytest-jubilant) behaves, not \ +about how pytest or Python logging works in general. Understand the \ +charm-dev tool first — read its release notes, source, and recent PRs to \ +see what changed — then dig into standard tools (pytest, logging) only if \ +the mechanism still isn't clear. For example, if a claim is about Jubilant \ +log output, start by checking what changed in the relevant Jubilant \ +version's release notes and `wait()` method, not by reading pytest's \ +logging plugin source. The charm-dev tool is the thing that changed; \ +pytest's behaviour is the stable backdrop. """ @@ -357,6 +368,16 @@ def test_deploy(charm, juju: jubilant.Juju): version and test against it. Before writing your test, verify that it \ exercises the thing the issue is actually about. +**Understand the charm-dev tool before the standard tool.** If the claim \ +involves a charm-dev tool (Jubilant, ops, pytest-jubilant), understand what \ +that tool does and what changed recently before diving into standard tools \ +like pytest or Python logging. The charm-dev tool is the thing that likely \ +changed; the standard tool is the stable backdrop. For example, if a claim \ +is about Jubilant log output and the issue references a version, check the \ +Jubilant release notes and `wait()` source first — that's where the \ +behavioural change lives. Only then, if the mechanism still isn't clear, \ +read pytest's logging plugin to understand how the config interacts. + **Do not be shy about integration tests.** `run_tox` runs `format,lint,unit` \ only — not integration tests. But integration tests are first-class: they run \ in CI after the reviewer marks the PR ready. Write them when the claim is \ @@ -472,6 +493,11 @@ def test_deploy(charm, juju: jubilant.Juju): is true, what the PR tests, and what green (or red) CI means for each claim. \ Use proper markdown: headers (`##`), bullet points, code blocks (fenced \ with triple backticks), and paragraphs separated by blank lines. +- **Do not hard-wrap lines.** Write each paragraph as a single long line — \ +the markdown renderer handles wrapping. Hard-wrapped lines break rendering \ +and make editing harder. +- **Do not include a "Changes" or "Files changed" section.** The reviewer \ +can see the diff in GitHub. Focus on reasoning, not a file listing. Example `.PR.md`: @@ -483,9 +509,7 @@ def test_deploy(charm, juju: jubilant.Juju): - **A**: log_level=INFO retains INFO logs in the captured section. - **B**: without it, DEBUG logs appear from log_file_level. -I believe the doc is correct. I added a test asserting that no DEBUG records \ -appear in caplog when log_level=INFO is set. If CI passes, the doc is \ -validated. If CI fails, the doc is refuted. +I believe the doc is correct. I added a test asserting that no DEBUG records appear in caplog when log_level=INFO is set. If CI passes, the doc is validated. If CI fails, the doc is refuted. ``` The reasoning is a core part of the adversarial approach: the reviewer needs \