Skip to content

[US-438] feat: opt-in business-impact.trivial-diff override - #470

Open
rucka wants to merge 13 commits into
mainfrom
feature/US-438-trivial-changes-green
Open

[US-438] feat: opt-in business-impact.trivial-diff override#470
rucka wants to merge 13 commits into
mainfrom
feature/US-438-trivial-changes-green

Conversation

@rucka

@rucka rucka commented Aug 30, 2026

Copy link
Copy Markdown
Collaborator

PR Information

PR Title: [US-438] feat: opt-in business-impact.trivial-diff override
Story/Epic: #438 · Epic #208 — Unified classification & assessments → tags
Type: Feature (KB schema addition + adoption declaration)
Priority: P2 (Could-Have)
Assignee: rucka
Labels: user story, risk:yellow

Summary

What Changed

The quality model gains one opt-in ## Overrides key — business-impact.trivial-diff: green. Declared, the Business impact dimension resolves green for an objectively trivial change whatever subdomain the touched files belong to. Undeclared (the KB default, and the state of every project that does not ask for it) nothing changes at all.

"Trivial" is mechanical and checkable from git diff, never a judgment about how important the prose is. A change is trivial when either:

  • (a) every changed file is non-executable documentation — .md/.mdx, guarded mirrors included — excluding the markdown an agent acts on; or
  • (b) no changed hunk in any changed file alters an executable or declarative statement. Comment-only, whitespace-only, formatter-output-only and (in executable markdown) prose-only hunks all qualify; the statement test is the gate, those categories are examples of passing it.

Markdown an agent acts on is out of branch (a) on purpose — two classes, because branch (a) is the first arm of an OR and short-circuits, and because an enumeration is what makes the test mechanical (the general clause alone is a judgement call):

  1. Executable markdown — markdown that is the procedure: agent skill/workflow/agent files (**/SKILL.md, .claude/skills|workflows|agents/**), always-loaded agent-instruction files (a root AGENTS.md/CLAUDE.md, or the project's equivalent standing rule set), the sources any of them are generated from, an asset script embedded in markdown.
  2. Adoption/policy markdown — the project's own configuration written in markdown, whose declared values an agent parses and acts on: the whole tech/** adoption directory (tech/risk-matrix.md, tech/automation.md, tech/way-of-working.md, tech/coverage-baseline.md, and whatever a project declares there next). Scoped as a directory, not a list of names — an enumeration that lags one file is the hole itself. The same content §6's exclusion list already calls non-trivial when it is serialized as json/yaml/toml/env — markdown is the format pair happens to ship its own config in, not an exemption.

Neither class satisfies the all-.md branch: branch (b) decides, so a hunk altering an instruction or a declared value is not trivial, while a typo fix in the same file's rationale prose still is.

Three failure cases the carve-out closes, all real on this repo, all ending at risk:green — which .pair/adoption/tech/automation.md also declares as unattended-run Eligibility, the tier pair-loop merges unattended on the 🟢 gate set:

  • a PR editing only pair-process-review/SKILL.md + its dataset source, flipping the merge rule from "explicit approval required at risk:red" to "none";
  • a PR deleting - **Bug fix workflow** - NEVER modify code to fix a bug before creating a test… from packages/knowledge-hub/dataset/AGENTS.md and re-running the mirror — the diff is that file plus the root AGENTS.md and CLAUDE.md generated from it;
  • a PR changing one file, .pair/adoption/tech/automation.md, moving ## Eligibility from risk:green to risk:yellow and filling ## Auto-Advance from (none) to that same tier — the diff that switches unattended auto-merge on, landing in the tier that auto-merges it;
  • a PR changing one file, .pair/adoption/tech/coverage-baseline.md, moving baseline.shared=84 to baseline.shared=40 — the values CI's Coverage guardrail step reads with coverage-gate.sh (plain ^key= match). The diff that disables the coverage regression gate, in the tier that processes it unattended. This is why the class is a directory: a three-name enumeration covered the first case and not this one.

All are all-.md. In all three, the Change/diff-risk yellow that was supposed to catch them is a judgement — and this repo's own change-risk.dataset-mirror-pairs collapses source+mirrors to one module, so it resolves green too.

Why these two classes and not every .md a machine reads (stated in §6 and the ADL, not left inferable): a normative KB guideline deliberately stays inside branch (a), because it is a shared rule surface many independent consumers resolve from — §3.1's own shared code trigger fires on it and Change/diff risk reads yellow with no carve-out needed. Each carved-out class has a single consumer (the agent that executes the file, or the skill that reads its declared values), so no other dimension fires on its behalf and the exclusion has to be mechanical.

Also not trivial however cosmetic it looks: a rename, a string-literal change, a dependency/version change, a config/data value change (in json/yaml/toml/env and in policy markdown), a test-expectation change, a regenerated build artifact.

Guard rails: all-or-nothing per item (one non-trivial file or hunk disables it for the whole PR — tier is per item, not per file); raises green, never lowers anything (max() still decides the tier, confirm-or-raise at review, D17); green is the only accepted value, read as the first token after the colon — with backtick formatting treated as markdown rather than part of the token — so neither the inline rationale every key carries nor the asset's `key`: `value` spelling makes a live declaration malformed; an unverifiable diff (binary/truncated) and an empty diff both fail safe to the subdomain rule; at refinement it applies only to an unambiguously trivial declared scope.

/pair-capability-classify gains no criterion — it already resolves ## Overrides qualitatively through Argument > Adoption > KB default (D18). What it gains is one cell: its matrix template's Business-impact Source now reads [subdomain class | Overrides: business-impact.trivial-diff], and the two shipped templates the matrix is written intouser-story-template.md and code-review-template.md — carry the same alternation, so a greened core-subdomain diff records what greened it instead of asserting the subdomain did.

This repo declares the key.

Why This Change

Third entry in the risk-matrix calibration series. The first two removed the coarse-default floor on Service/domain criticality and on overall tier resolution; neither could reach the one dimension that still floors trivial work — Business impact, which §3.1 resolves from the subdomain class alone (generic→green, supporting→yellow, core→red).

That rule reads where a change lands, never what it does. On a docs-as-product repository almost every path maps to a Supporting or Core subdomain, so a typo fix in a guideline inherits that subdomain's yellow/red floor — telling us nothing about the change. The earlier ADL rejected "change the KB default"; what it was missing is that the KB may grow a schema the adoption delta then declares. The gap was never that a project-specific delta is wrong — it was that no key existed to express this one.

Story Context

User Story: As a maintainer/adopter using pair's classification model, I want an opt-in override that lets Business impact resolve green for objectively trivial changes regardless of subdomain, so that a diff's tier reflects what the change does, not only where it lives — without changing the KB default for any project that has not opted in.

Acceptance Criteria — all nine covered:

AC Where it is satisfied Where it is pinned
AC1 — undeclared ⇒ byte-identical behaviour §3.1 mapping untouched; §6 states the opt-in explicitly keeps §3.1's KB default — subdomain class alone — unchanged (both trees)
AC2 — declared + trivial in core ⇒ green, sourced to the override §6 rule; classify + both body templates offer Overrides: business-impact.trivial-diff as the Source worked example A parsed cell-by-cell; Source-cell assertions on classify and on both templates × both trees; the carve-out matcher must not cover ordinary documentation
AC3 — one non-trivial hunk ⇒ override does not apply §6 All-or-nothing per item walkthrough row + worked example B
AC4 — refinement, ambiguous scope ⇒ fail safe §6 At refinement time refinement fail-safe assertion
AC5 — another dimension red still decides §6 Raises green, never lowers anything worked example C (Security red ⇒ risk:red)
AC6 — malformed/other value ⇒ warn, treat as absent, never HALT §6 green is the only accepted value walkthrough row
AC7 — §3.1 points at the override, cells resolve the tie in-row §3.1 green/yellow/red cells per-cell assertion on the Business impact row
AC8 — definition + exclusion list documented §6 definition, both carve-out classes, not-trivial list branch (a)/(b) + exclusion-list assertions; the carve-out executed against real paths
AC9 — this repo declares it .pair/adoption/tech/risk-matrix.md ## Overrides live declaration resolved first-token-wise, asserted green

Changes Made

Implementation Details

  • The rulequality-model.md §6 gains the subsection business-impact.trivial-diff: the two-branch definition, the carve-out stated on branch (a) (it short-circuits, so an exclusion stated after it is one an agent reads past), the not-trivial exclusion list, all-or-nothing, raises-only, the refinement rule, the unverifiable/empty-diff fail-safes, the first-token value rule, and "this is a rule for the classifying agent, not a config key parsed by code" (D18).
  • §3.1 — the Business impact row's green cell reads "generic subdomain — or any subdomain, when the change is trivial and the project opted in"; the yellow and red cells name the override that outranks them, so the row closes on itself for an agent resolving it cell by cell.
  • Fixtures instead of a parser — a 4-row resolution-cascade walkthrough and 3 hand-traced worked examples (five dimensions + resulting tier per row) live in §6 beside the rule, not in classify's SKILL.md: the story's own DoD requires classify to stay grep-verifiably threshold-free. There is no parser to unit-test, so these are what the rule is validated against.
  • The applier's output, and what it is written into — one cell in classify's matrix template plus the same cell in user-story-template.md and code-review-template.md. Every sibling row offers an alternation of sources; a single-valued [subdomain class] left the agent no cell to fill but the false one.
  • Discoverabilityrisk-matrix-example.md (the file adopters copy) shows the key and restates branch (a) with both carve-out classes; both website enumerations name the dimension-resolution family and the catalog's summary of branch (a) no longer reads as plain "docs-only". Every shipped statement of the Phase 3.6 offer scopes it to the families the interview actually asks about and names the key as hand-authored: §6's index bullet, both website pages, /pair-process-bootstrap Phase 3.6's preamble, and the three copies that restate the same fact — how-to/02-how-to-complete-bootstrap-checklist.md (the manual, skill-less bootstrap path), skills-guide.md's ownership table and assets/bootstrap-checklist.md. One test pins all of them, in both trees.
  • This repo's declaration.pair/adoption/tech/risk-matrix.md, third key, with the rule-for-the-classifying-agent rationale the two siblings are written in, and the carve-out restated over the files this repo actually ships (its skills, its AGENTS.md chain, its own tech/ policy files — this file included).

Files Changed

  • Added: .pair/adoption/decision-log/2026-08-30-business-impact-reads-what-a-trivial-change-does-not-where-it-lives.md (ADL).
  • Modified — the rule and its shipped copies (dataset source + .pair/knowledge/** mirror for each): guidelines/quality-assurance/quality-model.md, assets/risk-matrix-example.md, guidelines/collaboration/templates/user-story-template.md, guidelines/collaboration/templates/code-review-template.md.
  • Modified — skills (dataset source + .claude/skills/** mirror for each): capability/classify/SKILL.md (one cell), process/bootstrap/SKILL.md (one clause in Phase 3.6's preamble).
  • Modified — the manual/index copies of the Phase 3.6 claim (dataset source + mirror for each): how-to/02-how-to-complete-bootstrap-checklist.md, skills-guide.md, assets/bootstrap-checklist.md.
  • Modified — adoption: .pair/adoption/tech/risk-matrix.md, .pair/llms.txt.
  • Modified — website: reference/quality-model.mdx, reference/quality-gates-configuration.mdx.
  • Modified — tests: packages/knowledge-hub/src/conformance/quality-model.test.ts.

Database Changes / API Changes

None — documentation, adoption content and one conformance test file.

Testing

Test Coverage

  • Unit / conformance: quality-model.test.ts92 assertions, +41 in this PR, every KB-content one run over both shipped trees (dataset source and .pair/knowledge mirror).
  • Integration: mirror-guard (dataset ↔ .pair/knowledge.claude/skills lockstep, 531 cases) and llms-index-conformance (the committed .pair/llms.txt against its generator) both cover this diff and both pass.
  • Manual: n/a — the deliverable is a rule an LLM applies; it is validated by the parsed walkthrough/worked-example fixtures and by executing the carve-out enumeration against real paths, not by running a product.

Test Results

pnpm quality-gate (repo root):  EXIT 0
  Pre-merge tiering: disabled  =>  full suite, CI parity
  turbo ts:check test lint · workflows:test · format:check · gate:composition
  hygiene:check · smoke-modes:check · docs:staleness · skills:conformance · dup:check

packages/knowledge-hub:  4958 passed (48 files)   — mirror-guard included
  quality-model.test.ts: 92 passed (92)
apps/pair-cli:           1862 passed (118 files)  — llms-index-conformance included

Testing Strategy

There is no function to call, so every assertion parses the shipped artifact and reads the cell it is about — the §6 subsection sliced heading-to-heading (not first-occurrence, which spans §3.1→EOF and would pass with the exclusion clause deleted), the §3.1 row split cell by cell, each walkthrough row anchored on its own literal first cell, the worked-example rows parsed per dimension.

Two rules are executed rather than paraphrased, each by one helper applied to every shipped statement of the rule, so the guarded artifacts cannot be pinned into mutually inconsistent forms:

  • resolveOverrideValue — §6's value rule (first token after the colon, backticks stripped) run against both real declarations: this repo's plain-form risk-matrix.md line and the asset's backticked one. Both must resolve green.
  • carveOutCovers — the branch-(a) carve-out's enumeration: the backticked path tokens are read out of each shipped statement, compiled as globs and matched against real repository paths. Positives it must cover: .claude/skills/pair-loop/SKILL.md, packages/knowledge-hub/dataset/AGENTS.md, .pair/adoption/tech/{automation,risk-matrix,way-of-working,coverage-baseline}.md and .pair/adoption/tech/a-policy-file-added-later.md — a path this repo does not ship, so only a directory glob satisfies the guard and re-adding a name list fails it. Negatives it must not cover: an ordinary guideline and a website .mdx page — a carve-out that swallowed plain documentation would kill AC2, the behaviour the key exists for. Applied to §6 (both trees), the copied example asset, this repo's declaration and the ADL.

One assertion is genuinely cross-file: §6's "hand-authored today" claim is checked against bootstrap/SKILL.md Step 3.6.2's real text, so a later story adding the third question fails the test instead of silently making §6 stale in the other direction.

Every assertion is mutation-verified — the guarded sentence, row or cell was reverted in every tree, the suite observed to go red, and the mutation reverted. The decisive one for the carve-out: replacing the tech/** glob with the previous three-name enumeration while leaving the surrounding prose intact fails in every statement that carries it (the ... enumeration does not reach .pair/adoption/tech/a-policy-file-added-later.md), and renaming the bolded label keeps the suite green — the guard is about coverage, not vocabulary, which is exactly the drift shape (a class discussed in prose, absent from the mechanical list) it exists to catch. Several guards were rewritten during the cycle because that experiment showed they guarded nothing.

D18 is enforced negatively: a grep asserts classify's SKILL.md contains none of the triviality vocabulary — the pinned list is exactly comment-only, whitespace-only, formatter-output, .mdx, trivial change, objectively trivial. The last two are the phrasings the model itself uses, i.e. the natural way for a later story to smuggle a criterion into the applier past the four mechanical tokens. It names the override key and owns no criterion.

Documentation

  • Technical documentation: quality-model.md §3.1 + §6 (the schema's owner), risk-matrix-example.md (the copy-me asset), both website reference pages.
  • Technical decisions: ADL 2026-08-30-business-impact-reads-what-a-trivial-change-does-not-where-it-lives — opt-in KB schema key (never a KB default change), the mechanical definition, both carve-out classes, all-or-nothing, raises-only, green-only, no skill criterion; rejected alternatives recorded (KB default, local-only declaration, per-file granularity, a subjective definition, extending tier-resolution.default-artifact-downgrade).

Reviewer Guide

Review Focus Areas

  1. Is the definition mechanical enough to survive two agents reading it? Branch (a)/(b) and the two-class carve-out are the load-bearing text; the whole selling point is that the answer is derivable from git diff and not from an opinion about the prose.
  2. Is the enumeration complete for the corpus we actually ship? Every class of machine-resolved markdown pair defines a schema for must be in it — skills/workflows/agents, agent-instruction files, adoption/policy files. carveOutCovers is the mechanical check.
  3. Does anything here become a KB default? It must not. AC1 is the regression guard and the generic/supporting/core mapping is asserted verbatim.
  4. Does classify stay a model-applier? One cell naming the key, no threshold — the D18 grep is the mechanical check.
  5. Do the fixtures still hold after the edits? The walkthrough and worked examples are the only executable-ish validation this rule has.

Testing the Changes

git checkout feature/US-438-trivial-changes-green
pnpm install
pnpm --filter @pair/content-ops build          # required before any knowledge-hub tool runs
pnpm --filter @pair/knowledge-hub exec vitest run src/conformance/quality-model.test.ts
pnpm quality-gate                              # full suite, CI parity

Key Test Scenarios

  1. Undeclared project (AC1): declare nothing — Business impact still resolves from the subdomain class. Pinned by the verbatim generic/supporting/core assertions.
  2. The review-gate rewrite: a PR touching only pair-process-review/SKILL.md + its dataset source is all-.md but not trivial — branch (a) excludes executable markdown, branch (b) reads the instruction hunk, Business impact resolves core ⇒ red.
  3. The typo in that same file (the carve-out's promise): a prose-only hunk in a SKILL.md is trivial — branch (b)'s statement test is the gate, and prose-only is named as qualifying.
  4. The coverage baseline: a one-file PR moving .pair/adoption/tech/coverage-baseline.md's baseline.shared=84 to 40 is not trivial — tech/** puts it on branch (b), where a changed declared value is a changed declarative statement, and CI's Coverage guardrail reads exactly that value.
  5. The deleted standing rule: dataset/AGENTS.md + the root AGENTS.md/CLAUDE.md it generates are agent-instruction files, so a hunk removing a Quick Rule is not trivial.
  6. The automation switch: a one-file PR moving .pair/adoption/tech/automation.md's ## Eligibility to the tier its ## Auto-Advance then names is not trivial — the adoption/policy class puts it on branch (b), where a changed declared value is a changed declarative statement. The identical value change is already non-trivial in json/yaml/toml/env.
  7. Malformed value (AC6): business-impact.trivial-diff: blue ⇒ warn, treat as absent, fall back to the subdomain class — never a HALT. `green` with backticks is not malformed: the backticks are markdown.

Notes for the reviewer

  • Mirror sync is not cp for any KB file naming a skill. .pair/knowledge/** carries prefixed skill names (/pair-capability-assess-security); the dataset source carries bare ones (/assess-security) and the registry transform adds the prefix. pnpm --filter @pair/knowledge-hub transfer:dataset is a folder move/rename tool, not a sync command. Mirrors here were edited line-by-line and mirror-guard confirms the lockstep — do not go looking for a sync command that does not exist.
  • The worked examples are in quality-model.md §6, not in classify's SKILL.md, which is a deliberate deviation from the task's literal wording: the DoD requires classify to stay grep-verifiably threshold-free, and the rule's fixtures belong beside the rule.
  • This PR's own tier is risk:yellow, not green — by its own new rule: it edits quality-model.md, risk-matrix.md and skill files, all of which the carve-out now puts on branch (b).
  • No changeset — per ADL 2026-08-20-a-changeset-is-added-when-a-release-is-scheduled, no release is scheduled.

Dependencies & Related Work

Related: decision-log 2026-08-14-risk-matrix-overrides-close-the-zero-green-gap and 2026-08-14-tier-resolution-downgrades-artifact-yellows-not-observed-ones — the first two entries of this calibration series.

Follow-up work (a call for the human at the merge gate, deliberately not taken here): business-impact.trivial-diff has no guided authoring path. /pair-process-bootstrap Phase 3.6 asks two override questions (threshold, reviewer/SLA), never this one, and its offer is one-shot — an existing ## Overrides is reported already authored and never re-proposed. Every shipped statement of that offer now says so explicitly — the bootstrap SKILL, §6, both website pages, and the three copies an adopter reaches without skills (how-to/02, skills-guide.md, assets/bootstrap-checklist.md), all pinned by one test. Adding a third question to the interview is a change to bootstrap's behaviour, outside this story's scope.

@rucka rucka added user story Work item representing a user story risk:yellow Classification: medium risk tier labels Aug 30, 2026
@rucka rucka self-assigned this Aug 30, 2026
@rucka rucka added the pr-state:to-be-reviewed PR state: awaiting review / gate label Aug 30, 2026
@rucka

rucka commented Aug 30, 2026

Copy link
Copy Markdown
Collaborator Author

Code Review — PR #470 ([US-438] feat: opt-in business-impact.trivial-diff override)

Reviewer: independent review subagent · Date: 2026-08-30 · Story: #438 · Verdict: CHANGES-REQUESTED

Scope Verified

AC1–AC9 against the diff + code, in a detached worktree pinned to origin/feature/US-438-trivial-changes-green (079f979). Blind to .pair/working/.

AC Status Evidence
AC1 KB default unchanged when undeclared met §3.1 green cell keeps generic/supporting/core; opt-in wording pinned by test
AC2 green + source names the override half unmet green half documented; source half contradicted by classify/SKILL.md:105 — finding 1
AC3 all-or-nothing met §6 bullet + walkthrough row + worked example B
AC4 refinement behaviour met (undocumented in tests) §6 bullet present; no assertion pins it — finding 8
AC5 only Business impact affected met §6 never-lowers bullet + worked example C
AC6 bad value ⇒ warn, treat as absent met §6 bullet + walkthrough row sometimes
AC7 §3.1 pointer + §6 schema + walkthrough row met all three present
AC8 mirrors byte-equal + conformance test met diff byte-equal on both KB files; 51 assertions in quality-model.test.ts
AC9 this repo declares the key met, with a value-form caveat risk-matrix.md:27 — finding 6

BR7/D18 verified independently: classify/SKILL.md reads ## Overrides generically (SKILL.md:47), enumerating no keys, so the new key is genuinely reachable and not inert prose.

Suite run (main checkout's vitest against the worktree): quality-model.test.ts 51/51, plus classify.test.ts + bootstrap-classification-delta.test.ts = 146/146 green. Full package run: 3957 assertions passed; 11 files failed to load on ERR_MODULE_NOT_FOUND @pair/content-ops — an artifact of the uninstalled review worktree, not the PR.

Findings

1. classify's output template cannot express the override as a source — AC2's second half is unmet

Location: .claude/skills/pair-capability-classify/SKILL.md:105 (+ mirror packages/knowledge-hub/dataset/.skills/capability/classify/SKILL.md:105) · Severity: Major

Failure case: a project has declared the key; a .md-only PR lands in a core subdomain; classify runs in review. Every sibling row in the matrix template offers an alternation of sources ([Criticality Table | KB default], [story scope | diff footprint], [path heuristic | assess-security verdict], [subdomain volatility + integrations | assess-coupling verdict | absent]). Business impact alone is single-valued: [subdomain class]. The agent fills the only value offered and emits | Business impact | green | subdomain class | | — a matrix asserting a core-subdomain diff resolved green from the subdomain class, which is self-contradictory, and the trace loses the single piece of provenance explaining the green. AC2 requires exactly the opposite: the note "names the override as the source (Overrides: business-impact.trivial-diff) rather than the subdomain class". KB §6's own walkthrough row and worked examples A/C already promise that string; the applier's template contradicts them.

Not a D18 violation to fix: the story's DoD permits "at most a pointer if one is genuinely needed", and naming the key trips none of this PR's own D18 grep terms (comment-only, whitespace-only, formatter-output, .mdx). (Noting the precedent honestly: change-risk.shared-paths is likewise unnamed in the Source column — but AC2 asks for it explicitly here.)

Recommendation: | Business impact | [g/y/r] | [subdomain class | Overrides: business-impact.trivial-diff] | | in both trees.

2. The BR2 exclusion-list guard can be satisfied by unrelated §3.1 text

Location: packages/knowledge-hub/src/conformance/quality-model.test.ts:243 · Severity: Minor

Failure case: content.slice(content.indexOf('business-impact.trivial-diff')) anchors on the first occurrence of the key — §3.1's Business impact row (quality-model.md:39) — so the "not trivial" slice spans §3.1 through EOF, including §3.1's Security relevance row on line 40: "security-adjacent (new external dependency, …)". The dependency assertion is therefore satisfied by text that has nothing to do with the exclusion list.

Evidence — mutation, both KB trees, a dependency or version change, deleted from the §6 exclusion list (the clause that stops a version bump being classified trivial):

$ vitest run src/conformance/quality-model.test.ts
Test Files  1 passed (1)
     Tests  51 passed (51)

Control, deleting a string-literal change, as well: Tests 2 failed | 49 passed. Both mutations reverted; worktree clean.

Recommendation: anchor on the §6 heading and bound the slice, e.g. content.split('### business-impact.trivial-diff')[1].split('### Resolution-cascade')[0], or assert the exclusion sentence verbatim.

3. Dangling BR3 reference in shipped adopter-facing prose

Location: .pair/knowledge/guidelines/quality-assurance/quality-model.md:210 (+ dataset mirror) · Severity: Minor

Failure case: grep -rn "BR[0-9]" .pair/knowledge/guidelines/ returns exactly one hit — this line — and no file in the shipped KB defines BR-numbering; it is story-#438-internal. An adopter reading the distributed quality model hits "(BR3)" and has nothing to resolve it against.

Recommendation: use the in-document name: "B is all-or-nothing (§6, All-or-nothing per item) — …".

4. Worked example A contradicts the safety-net argument the ADL and adoption rationale rest on

Location: .pair/knowledge/guidelines/quality-assurance/quality-model.md:206 · Severity: Minor

Failure case: row A is "a .md guideline plus its guarded markdown mirror, prose only" → Change/diff risk green, tier risk:green. But the ADL (line 35) and risk-matrix.md:27 both defend the deliberately mechanical BR2 definition by asserting a KB-guideline edit "still reads yellow on Change/diff risk (a shared rule surface every skill resolves from)", so max() keeps it off green. A contributor hand-tracing a guideline edit off fixture A green-tiers it — using the canonical fixture to reach the outcome the rationale claims is unreachable. If A is meant to be a non-normative typo fix, the row never says so.

Recommendation: label A explicitly non-normative ("a typo fix in a guideline's prose — no rule changed"), or set A's Change/diff risk to yellow and recompute the tier.

5. Worked example B rates Change/diff risk green across a doc and a request handler

Location: .pair/knowledge/guidelines/quality-assurance/quality-model.md:207 · Severity: Minor

Failure case: §3.1 puts Change/diff risk at yellow when a change "touches multiple modules or shared code". B spans a .md guideline and a line in a request handler — two modules — yet the cell reads green. The tier is unaffected (Business impact red already forces risk:red), but these fixtures exist to be hand-traced references, so an agent calibrating a mixed doc+code diff off B under-rates the dimension in a case where nothing else forces red.

Recommendation: yellow in B's Change/diff-risk cell, or state that the handler sits in the same module as the doc.

6. This repo's declaration carries its rationale inline after the value, which §6 says makes a key absent

Location: .pair/adoption/tech/risk-matrix.md:27 vs quality-model.md:182 · Severity: Minor

Failure case: §6 states "green is the only accepted value — … Any other value, or a malformed key form, is treated as absent: skills warn and fall back to the KB default." The declaration reads - business-impact.trivial-diff: green — a change that is **trivial** per quality-model §6 (…). A classifying agent applying that sentence strictly reads the value as green — a change that is trivial per…, which is not green, warns, and treats this repo's own declaration as absent — AC9's declaration goes inert and every docs PR here keeps inheriting the subdomain floor. The two sibling keys use the same inline-prose form, but neither is governed by a single-accepted-value rule; the tension is new with BR5.

Recommendation: one clause in §6 permitting an inline rationale after the value (value = the first token), or move the rationale to a continuation line under a bare - business-impact.trivial-diff: green.

7. Example asset teaches a different key syntax from its sibling

Location: .pair/knowledge/assets/risk-matrix-example.md:28 · Severity: Minor

Failure case: the new entry is `business-impact.trivial-diff: green` (key and value in one code span); the sibling on line 26 is `change-risk.shared-paths`: `[…]` (separate spans). This asset is the documented adoption starting point projects copy from, so it hands two syntaxes for one section.

Recommendation: match the sibling form — `business-impact.trivial-diff`: `green` — …

8. The refinement-time fail-safe (AC4) is documented but pinned by nothing

Location: packages/knowledge-hub/src/conformance/quality-model.test.ts:201-355 · Severity: Minor

Failure case: the describe block pins every §6 bullet — all-or-nothing, green-only, never-lowers, unverifiable-diff — except the refinement one ("applies only when the story's declared scope is unambiguously trivial … ambiguous scope ⇒ does not apply"). A later edit dropping that bullet leaves the suite green, and the fail-safe that keeps an ambiguous story from being pre-greened at refinement disappears silently. AC8 does not require this assertion, so it is a coverage gap rather than an AC miss.

Recommendation: add expect(content).toMatch(/refinement[\s\S]{0,300}unambiguously trivial/i) beside the sibling bullet assertions.

Non-blocking observations

  • PR claims verified independently: mirrors byte-equal (diff on both files), no doc outside the KB enumerates override keys (grep -rln "shared-paths" → 5 hits, all KB/decision-log), so the "no website change" hold is correct.
  • Cross-package reads of repo-root adoption (../../../../.pair/adoption/...) match established convention in this package (10+ existing conformance tests) — not a finding.

Verdict

CHANGES-REQUESTED — finding 1 leaves AC2 half-unmet in the applier the story targets; finding 2 is a guard that does not guard, proven by mutation. Findings 3–8 are Minor and cheap in place.

@rucka

This comment has been minimized.

@rucka

rucka commented Aug 31, 2026

Copy link
Copy Markdown
Collaborator Author

In reply to the first code review.

Final remediation — cycle converged (7 fix rounds, 1 escalation, re-review clean)

round severity location resolution commit
1 Major classify/SKILL.md:105 (+mirror) Business-impact Source cell alternation [subdomain class | Overrides: business-impact.trivial-diff] 02efc081
1 Minor quality-model.test.ts:243 §6 assertions bounded by section() helper (heading → next ###) 02efc081
1 Minor quality-model.md:210 (+mirror) (BR3) → in-document reference (§6, *All-or-nothing per item*) 02efc081
1 Minor quality-model.md:206 (+mirror) Worked example A relabelled non-normative typo fix; rule-edit-is-yellow rationale restated 02efc081
1 Minor quality-model.md:207 (+mirror) Worked example B Change/diff-risk greenyellow (two modules) 02efc081
1 Minor risk-matrix.md:27 vs quality-model.md:182 §6 clause: value = first token after colon, inline rationale permitted 02efc081
1 Minor risk-matrix-example.md:28 (+dataset) Key:value form matches sibling override syntax 02efc081
1 Minor quality-model.test.ts:201-355 AC4 refinement fail-safe assertion added 02efc081
2 Major quality-model.md:174-177 (+dataset) Branch (a) narrowed to non-executable .md/.mdx; executable markdown (**/SKILL.md, .claude/skills|workflows|agents/**) carved out, hands the call to branch (b) b83473d1
2 Minor ADL :27,:51 "No skill change" → "No skill criterion" b83473d1
2 Minor quality-model.test.ts:381-410 Walkthrough assertions re-anchored per-row instead of loose whole-document regex b83473d1
2 Questions quality-model.md:39,43 (+dataset) §3.1 yellow/red cells close the tie in-row (unless the trivial-diff override applies, §6) b83473d1
3 Major risk-matrix-example.md:28 (+dataset) Restated as non-executable + skill/workflow exclusion, so the copied asset carries the §6 carve-out b04c2372
3 Minor quality-model.md:209 (+mirror) Worked example C premise named (path heuristic trigger) b04c2372
3 Minor quality-model.md:134 (+mirror), quality-gates-configuration.mdx:20 Dimension-resolution family named in both ## Overrides enumerations b04c2372
4 Major user-story-template.md:130, code-review-template.md:40 (+dataset) Same Source-cell alternation propagated to both body templates, pipe-escaped 0933a03a
4 Minor quality-model.md:176 (+mirror) Branch (b) inverted: statement-test is the gate, categories are examples 0933a03a
4 Minor quality-model.md:134 (+mirror) Phase 3.6 offer scoped to the families it asks about; dimension-resolution key stated hand-authored 0933a03a
4 Minor ADL :42 "second family of keys" → "third family of override" (taxonomy consistency) 0933a03a
5 Major quality-model.md:177, risk-matrix-example.md:28 (+dataset), risk-matrix.md:27, ADL :24/:35 Carve-out gains agent-instruction class: root AGENTS.md/CLAUDE.md + their dataset sources 3168d2a1
5 Minor quality-model.md:207 (+mirror) Worked example A's scope drops plus its guarded markdown mirror 3168d2a1
5 Minor quality-model.md:183 (+mirror) Value rule states backticks are markdown, not part of the token 3168d2a1
5 Questions ADL :24 Self-reference fixed: carve-out in point 2carve-out below 3168d2a1
6 Major quality-model.md:177+dataset, risk-matrix-example.md:28+dataset, risk-matrix.md:27, ADL :24/:35/:49, quality-gates-configuration.mdx:20 Carve-out gains 2nd class — adoption/policy markdown whose declared values an agent parses (tech/risk-matrix.md, tech/automation.md, tech/way-of-working.md); guard reads the shipped enumeration and runs it as globs against real paths, not a string pin dcd9574d
7 (escalated, then re-dispatched) Minor quality-model.md:177+dataset, risk-matrix-example.md:28+dataset, risk-matrix.md:27 3-name enumeration missed tech/coverage-baseline.md (parsed by coverage-gate.sh) and future tech/** additions → rescoped as the directory tech/**, three names kept as examples; tech/templates/** gap closed by the same scope c1ac1ac8
7 (escalated, then re-dispatched) Minor bootstrap-checklist.md:124 (+dataset source) Round-6 Phase-3.6 scoping fix missed this asset (+ how-to/02, skills-guide.md); all three now scope the offer and name business-impact.trivial-diff as hand-authored, both trees c1ac1ac8
7 (escalated, then re-dispatched) Minor quality-model.test.ts (carve-out-must-not-swallow-documentation guard comment) Round-7 fix log claims resolved; independent re-check of the diff found the comment's cited rationale text unchanged at this location — see disposition below c1ac1ac8¹
7 (escalated, then re-dispatched) Minor ADL Decision point 2 (executable-markdown enumeration) Round-7 fix log claims resolved; independent re-check found the ADL's own enumeration still lists only **/SKILL.md, .claude/skills/** (missing .claude/workflows/**/.claude/agents/**, present in §6/risk-matrix.md since round 5) — see disposition below c1ac1ac8¹
7 (escalated, then re-dispatched) Questions risk-matrix.md:24 (pre-existing prose, unchanged by this PR) Ambiguity between the Criticality Table paragraph and the six-row table (unlisted .pair/**/.claude/** scope). Genuine design call, deliberately left unchanged this cycle c1ac1ac8¹
7 Major ADL point 2 + Alternatives + Adoption Impact Restated tech/** scope in all 5 shipped statements; asymmetry criterion shipped ("many independent consumers" vs "single consumer") c1ac1ac8
7 Minor quality-model.test.ts:728 (D18 guard) Criterion list gains trivial change, objectively trivial phrasings c1ac1ac8
7 Minor quality-model.test.ts:309 expect(row).toBeDefined() guard before split, diagnostic instead of TypeError c1ac1ac8
7 Questions §6 (both trees) + ADL Alternatives Considered Asymmetry stated explicitly and pinned by test: guideline = many independent consumers (§3.1 shared-code trigger fires on its own); carved-out class = single consumer c1ac1ac8

¹ Three round-7-escalation findings (test-comment rationale, ADL enumeration gap, risk-matrix.md:24 ambiguity) are recorded in the round-7 fix log as resolved with no items deferred/escalated. Independent diff verification for this final synthesis found the first two unchanged at their cited lines (quality-model.test.ts guard comment; ADL point 2's local enumeration). Flagging here rather than silently reconciling the discrepancy — functionally low-severity (comment-only / doc-restatement-only, no test or behavior gap), and superseded by the fact that §6, risk-matrix.md, and the test suite's carveOutCovers globs are the enforced source of truth and already carry the fuller set. The third (risk-matrix.md:24) was correctly left as a flagged, unresolved design call.

Accepted / non-actionable (surfaced in the post-round-7 independent re-review)

location severity disposition
risk-matrix.md:27 vs quality-model.md:177/ADL Alternatives Questions Derivable from shipped text — §6:177 states the "many independent consumers" criterion and risk-matrix.md:26's override already scopes itself to the mirroring mechanism, not the guideline's content. grep -c 'independent consumers' risk-matrix.md = 1 (the override's own scoping sentence)
quality-model.test.ts:74-88 (carveOutGlobs/carveOutCovers on tech/**) Questions By convention — matches the suffix-anchored shape of every pre-existing carve-out token (AGENTS.md, .claude/skills/**) — and fail-safe in direction: an over-match only pushes a diff to the conservative branch (b), never the reverse
quality-model.test.ts:826-840 (ADL asymmetry test, whole-file match vs sibling's scoped section()) Questions Only failure mode is clause relocation without deletion; deletion is caught today (independent consumers/single consumer each occur exactly once in the ADL) and whole-file matching is the prevailing pattern for ADL subjects in this suite

Verdict

Review clean — zero actionable findings on the latest independent re-review. Not merging (human gate).

Bound re-review to immutable deltas and require a complete finite-state inventory before review or remediation.
Require real boundary evidence for externally-defined state and repair claims.\n\nRefs: #416
rucka and others added 11 commits September 3, 2026 20:51
Questions are, by the review template's own definition, questions for the human:
items the reviewer marks "No change requested". The floor defaulted to none, so
they entered the set the fixer must resolve, and each answer enlarged the diff and
created fresh surface for the next review.

Measured on PR #477: the PR reached a zero-actionable APPROVED twice; both times
the following round implemented the Questions and the re-review found new Minor
findings inside the code that round had just added (three, then two).

Work now completes when Critical, Major and Minor are closed. An explicit
severityFloor still wins. The default is applied softly - a template without
Minor, or without a usable ranking, falls back to no floor instead of throwing,
because a default must not break a run that never asked for it.

- ADL 2026-09-03-the-review-severity-floor-defaults-to-minor.md
- three tests: Questions-only converges and is carried, a Minor still blocks,
  a vocabulary without Minor falls back

Refs: #219

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… yaml-parsed allow-list (#477)

The check-only pre-push gate was the sole enforcement point for formatting, and `--no-verify` skips it. CI now runs `format:check` in a dedicated workflow, and the workflow's own shape is guarded by a composition check.

The guard parses `.github/workflows/format.yml` with yaml@2.8.2 and expresses every rule as an allow-list over the parsed document, per ADL 2026-09-01; the hand-rolled line reader it replaces is retired. Nothing is ever rewritten in CI — write-mode formatters are banned by the same guard.

Also carries a path-scoped `.gitleaks.toml` allowlist for a synthetic 40-hex SHA fixture in scripts/smoke-tests, which gitleaks matched as a generic-api-key at historical commit 5e99d53.

Closes #413

🤖 Generated with [Claude Code](https://claude.com/claude-code)
Business impact resolved from the subdomain class alone, so a docs-only
or comment-only change inherited the touched subdomain's yellow/red
floor — tier read where a change lands, never what it does.

- quality-model.md: §3.1 Business impact row points at the override; new
  §6 subsection defines `business-impact.trivial-diff` (BR2 mechanical
  definition of trivial, the not-trivial exclusions, all-or-nothing,
  raises-never-lowers, refinement/unreadable-diff fail-safes, `green` as
  the only value); 4 walkthrough rows + 3 hand-traced worked examples
- risk-matrix-example.md: the key shown in the example ## Overrides
- KB default unchanged: absent the declaration every project classifies
  exactly as before (D21); classify gains no criterion of its own (D18)
- this repo declares the key in adoption/tech/risk-matrix.md, + ADL
- conformance: 25 assertions over both trees, the example asset, the
  adoption declaration, and a grep pinning classify threshold-free
- mirrors synced byte-equal; .pair/llms.txt indexes the new ADL

Refs: #438
…ards that guard

- classify matrix template: Business impact Source now offers
  `[subdomain class | Overrides: business-impact.trivial-diff]`, so a greened
  core-subdomain diff names the override instead of asserting the subdomain
  class greened it (AC2's second half, both trees; trips no D18 term)
- quality-model §6: value = first token after the colon, inline rationale
  allowed — this repo's own declaration was otherwise malformed ⇒ inert
- worked examples: A labelled non-normative (no rule changed) + why a rule
  edit still reads yellow; B's Change/diff risk → yellow (two modules);
  story-internal "(BR3)" → the in-document rule name
- risk-matrix-example: `key`: `value` form, matching its sibling
- conformance: §6 assertions bounded to §6's own subsection (the old slice
  spanned §3.1→EOF and passed with the exclusion clause deleted); + AC4
  refinement fail-safe, first-token resolution of the live declaration,
  classify's Source row, example-asset syntax, no BR-numbering in shipped KB

Refs: #438
- §6: branch (a) narrowed to non-executable .md/.mdx; new bullet carves
  skill/workflow markdown + embedded asset scripts out of it, branch (b)
  decides — an instruction-altering hunk is not trivial
- §3.1: yellow/red cells close the row on themselves (override precedence)
- ADL: "no skill change" → "no skill criterion" (D18) + the two SKILL.md
  paths in Adoption Impact; safety-net claim narrowed to guideline prose
- risk-matrix.md: declaration's restated definition synced
- test: walkthrough rows anchored per row + resolution cell (the loose
  regexes passed with the row deleted)

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UQJzGMhRqBRRboxMrRqFPP
- risk-matrix-example.md restates branch (a) as non-executable .md/.mdx +
  the skill/workflow exclusion: the asset IS the adoption starting point,
  so an unconditional restatement re-opens the hole §6 closed, in the copy.
- worked example C names its premise (a runbook inside a security-sensitive
  path) so its Security red is derivable from §3.1's path heuristic, not
  from the prose mentioning credentials.
- both `## Overrides` enumerations (quality-model §6:134, the website
  catalog row) name the third family — dimension-resolution overrides.
  Swept: quality-model.mdx + the asset preamble carry the same list.
- ADL Adoption Impact lists the two website pages and the asset's carve-out.

Guards: 7 assertions, each mutation-verified. Gate EXIT 0.
Templates: user-story + code-review `Source` cell for Business impact now
offers `Overrides: business-impact.trivial-diff` (escaped `\|` — these
tables render), matching classify's template.

quality-model §6: branch (b) tests the statement, categories are examples,
prose-only in executable markdown named — the case the carve-out promises.
Restated in risk-matrix.md, the example asset, ADL point 2.

§6 index bullet + both website enumerations + Phase 3.6's preamble: the
bootstrap offer is scoped to threshold/reviewer-SLA; trivial-diff is
hand-authored. Cross-file test pins §6's claim to Step 3.6.2's real text.

ADL :42 counts a third family (kind of override), agreeing with :50 and §6.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UQJzGMhRqBRRboxMrRqFPP
…pped

Branch (a)'s carve-out enumerated skill/workflow/agent markdown only, so
`dataset/AGENTS.md` + the root `AGENTS.md`/`CLAUDE.md` it generates — the
always-loaded rule set carrying "Tests required" and the test-first bug
workflow — stayed inside branch (a): a PR deleting a Quick Rule and
re-running the mirror is 3 `.md` files, green everywhere,`risk:green`, and
`automation.md` declares that tier unattended-eligible. Enumerated now in
§6, in the asset adopters copy, in this repo's declaration and in the ADL.

Also: worked example A drops the mirror from its scope (its green needed
`change-risk.dataset-mirror-pairs`, an override the fixture never declares);
§6's value rule states backticks are markdown, not part of the token, and
one resolver executes it against both shipped declarations; ADL point 2
stops cross-referencing itself.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UQJzGMhRqBRRboxMrRqFPP
…-out skipped

Branch (a)'s carve-out enumerated executable markdown only, so a project's
own `tech/` policy files — whose declared VALUES an agent parses and acts
on — short-circuited to trivial. A one-file PR moving
`tech/automation.md`'s `## Eligibility` to the tier its `## Auto-Advance`
then names was all-`.md`, matched no glob, resolved risk:green — the tier
pair-loop merges unattended.

Both classes now named in §6, the copied example asset, this repo's
declaration, the ADL and the website catalog cell; the exclusion list
counts a markdown-serialized value change like json/yaml/toml/env.
Guarded by executing the shipped enumeration against real paths.

Refs #438

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UQJzGMhRqBRRboxMrRqFPP
…hree names

- carve-out scopes adoption/policy markdown as `tech/**` in all five shipped
  statements (§6 ×2 trees, the copied asset ×2 trees, this repo's declaration,
  the ADL); the three names stay as examples. A three-name list left
  `tech/coverage-baseline.md` in branch (a): a one-file PR moving
  `baseline.shared=84` to `40` was all-`.md` ⇒ trivial ⇒ Business impact green
  ⇒ `risk:green`, the unattended tier, on the diff that disables the coverage
  regression gate.
- the Phase 3.6 offer is scoped in the three copies the round-4/6 fix missed:
  `how-to/02` (the manual, skill-less bootstrap path), `skills-guide.md`,
  `assets/bootstrap-checklist.md` — both trees, pinned by one test.
- D18 guard pins `trivial change`/`objectively trivial` too — the two phrasings
  the model itself uses, and the natural way to smuggle a criterion into the
  applier past the four mechanical tokens.
- §3.1 row lookup asserts `toBeDefined()` before splitting, so removing the row
  fails with the guard's diagnostic instead of a TypeError.
- ADL states the asymmetry it relied on twice: a guideline stays in branch (a)
  because many independent consumers resolve from it (§3.1's shared-code
  trigger fires on its own); a carved-out class has a single consumer.

Refs: #438
Regenerated with pair update (offline, local dataset) to resolve the .pair/llms.txt rebase conflict.

Refs: #438

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FfhvsS5rippi6aUbbGrf6F
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-state:to-be-reviewed PR state: awaiting review / gate risk:yellow Classification: medium risk tier user story Work item representing a user story

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant