Skip to content

knowledge: a generated closed-vocabulary reference needs coverage assertions, not just --check (1 ingested, 2 dropped as in-flight dups of #47) #69

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
choiyounggi wants to merge 1 commit into
base: main
Choose a base branch
from
Open

knowledge: a generated closed-vocabulary reference needs coverage assertions, not just --check (1 ingested, 2 dropped as in-flight dups of #47) #69

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
167 changes: 124 additions & 43 deletions .dev-loop/INGEST_REPORT.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,53 +1,134 @@
# Knowledge consolidation — 15 open PRs (#17–#40) → one reconciled state
# Knowledge flush — 3 insight(s): 1 ingested, 2 dropped as in-flight duplicates

The 15 open `knowledge/*` PRs (created 2026-08-04 → 2026-08-05, before the
harvest processed-store dedupe fix in #41) contained 123 file-versions of ~75
unique pages, with the same insight landing at up to 3 different paths across
up to 8 PRs. Per-PR review would re-import those duplicates, so — as with the
#6–#13 consolidation — this branch carries the reconciled end-state and the 15
PRs are closed in its favor.
Queue drained: 3 pending candidates from 2 sessions (`linkly`, `groundwork`).

## Verified best-practice

Every adopted page's sources were carried from its originating PR's flush, where
they were live-verified at flush time; no new URLs were introduced during
consolidation (checked mechanically: every `http(s)` URL in every merged page
appears in a source PR's diff; every added body line in amended pages traces to
a source PR hunk — orphan-line verification). Confidence fields were kept as the
originating flushes set them, except client-side-rate-limiting where the union
of provider-doc citations (Okta, Auth0, GitHub, OpenAI, RFC 6585) supports
`verified` for the load-bearing claims. One subagent's fabricated content (12
files matching neither main nor any PR, with invented source URLs) was detected
by the same verification and replaced with true PR content.
### 1. Generate a closed-vocabulary reference from the owning constant, and gate it — INGESTED

**Claim.** When a reference document enumerates a closed vocabulary that a model will
emit tokens from (DSL verbs, config keys, diagnostic codes, enum members), generate the
enumerated part from the compiler/runtime constant, gate it with a `--check` mode called
from the test suite, and put a machine-readable generated banner on line 1. A green
`--check` is not sufficient on its own: it compares the committed file against the
generator's own output, so a generator that hardcodes a column regenerates happily and
stays green. Per-member coverage assertions against the constant, plus a
not-a-single-repeated-value negative control, are a separate layer.

**Sources checked (all fetched this session, quotes verified):**

| Source | What it establishes |
|--------|---------------------|
| https://pkg.go.dev/cmd/go | "To convey to humans and machine tools that code is generated, generated source should have a line that matches the following regular expression (in Go syntax)": `^// Code generated .* DO NOT EDIT\.$`, before the first non-comment text — the banner is machine-readable by convention, not a polite comment |
| https://prettier.io/docs/en/cli | `--check` "will output a human-friendly message and a list of unformatted files, if any"; returns "exit code `1` in the second case, which is helpful inside the CI pipelines" — the check-mode-as-gate shape |
| https://google.github.io/styleguide/docguide/best_practices.html | "Change your documentation in the same CL as the code change"; and for a fact owned elsewhere, "Do not write your own guide … Link to it instead" |
| https://platform.claude.com/docs/en/agents-and-tools/tool-use/define-tools | "Provide extremely detailed descriptions. This is by far the most important factor in tool performance" — what the model is given determines what it emits |
| https://go.dev/blog/generate | Shows the convention in practice (`// Code generated by stringer -type Pill pill.go; DO NOT EDIT.`) but states no regex; the normative text is in `cmd/go` above |

**Local verification (re-run this session, not taken from the harvested block):**

- `python3 scripts/gen_plugin_references.py --check` in `linkly` → `rc=0` against the
committed references.
- `impl/tests/test_plugin_references.py` runs it inside the suite:
`test_no_drift_between_source_and_committed_files` asserts `returncode == 0`;
`test_check_mode_detects_a_hand_edit` and `test_check_mode_reports_a_missing_file`
both assert `returncode == 1`;
`test_generated_files_carry_the_do_not_edit_banner` asserts the line-1 banner
(confirmed present in `plugins/lnpl/skills/lnpl-authoring/references/declarations.md`).
- The `--check`-is-insufficient nuance is documented in that same suite, in the
docstring of `test_every_diagnostic_code_reaches_the_document_with_its_grade`:
"`--check` alone cannot see this: it compares the committed file against the
generator's own output, so a generator that hardcoded every grade to `warning` would
regenerate happily and stay green" — with
`test_the_grade_column_is_not_a_single_repeated_value` as the negative control. This
refinement was **not** in the harvested candidate; it was found while verifying it and
is the strongest part of the page.
- Counter-example, also asserted in the suite:
`impl/tests/test_cli_diagnostics.py::test_compile_reports_all_six_and_still_succeeds`
pins the hand-written golden `examples/login.lnpl` at `rc == 0` with
`err.count("unknown-verb") == 3` — three steps compile to nothing while the compile
reports success. The harvested block claimed "3 of 6 steps"; the verified form is
"3 unknown-verb diagnostics on a golden example that still exits 0", and the page says
only that.

**Confidence: `verified`.** The generation/`--check`/banner mechanics are backed by
official docs (Go, Prettier, Google docguide) and by a reproduction run this session;
the agent-consumer framing is backed by Anthropic's own tool-definition guidance.

### 2. Usage-limit worker pause vs. crash — DROPPED (duplicate, see Open-PR check)

Claim (workers go quiet simultaneously, every liveness check passes, find the
`You've hit your session limit · resets HH:MM` marker, resume after the reset with a
state-recheck → remaining-DoD → completion-signal prompt) was **not** independently
re-verified, because it is already carried in open PR #47 with the same field evidence.
No new page or edit was made for it.

### 3. Dispatch immediately after `worker_done` → `runtime_unavailable` — DROPPED (duplicate)

Same disposition: already carried in open PR #47, including the task-consumption
asymmetry (a failed `worker-start` consumes the Task, so recreate from the spec rather
than retrying it) and the `tui-idle` precondition.

## Existing-layer check

- Merged-main near-dup scan before consolidation: pairwise Jaccard over
title + "When this applies" across all 141 merged pages → **0 flagged pairs**;
previously merged content carries no duplication.
- Cross-PR dedup during consolidation: 10 duplicate clusters collapsed to one
canonical page each (rate limiting 8→1, call-site enumeration 7→folded into
the canonical merged in #20, stderr/exit-0 diagnostics 4→1, sysroot 2→1,
env-off-switch 2→1, completion predicates 2→1, robots.txt 2→1,
harness-mediated results 2→1, leaked artifacts 2→1, orchestration category
naming unified). Three near-pairs kept distinct after trigger comparison,
with mutual `related:` links (differential setup vs interpretation; expansion
semantics vs off-switch design; import-time tactics vs level choice).
- 24 existing pages received union-merged amendments; additions already present
in main (from #16/#20) were skipped, and all non-canonical `related:` ids
were remapped to canonical page ids (post-merge broken-link scan: 0).
Routed via `INDEX.md`. The harvested `domain: platforms` hint was checked and rejected
for candidate 1 — `platforms` is scoped to OS-level differences (shells, BSD-vs-GNU,
filesystems, toolchains), and this is a document-verification case. `qa` owns
"automated verification of document deliverables (spec/RFC gates)".

Pages read: qa-document-verification-spec-document-gates, qa-document-verification-editing-a-gated-document, qa-deliverables-generated-artifacts-as-deliverable-source, qa-exploratory-lowered-declaration-survival, backend-common-api-design-unenforced-declarations, infrastructure-agent-orchestration-control-signals-vs-primary-artifacts, infrastructure-agent-orchestration-pane-delivery-confirmation, testing-quality-tests-that-cannot-fail

Overlaps found and how each was resolved:

| Existing page | Overlap | Resolution |
|---------------|---------|------------|
| `qa-deliverables-generated-artifacts-as-deliverable-source` | Both say "use the generator's output as the body". That page's trigger is a **hand-off deliverable** (ERD, schema reference for a partner/review) and it has no gate: its question is "re-run the generator or hand-write it?" | Kept separate. New page's trigger is a **shipped reference an agent reads back**, and its content is the gating layer (`--check` in the suite, banner assertion, coverage assertions). Cross-linked both directions; step 1 of the new page defers to it for the generated-body-vs-hand-written-narrative split |
| `qa-document-verification-spec-document-gates` | Nearest neighbour. Its "External agreement" axis (added on open PR #66, unmerged) says a gate must resolve the owning constant for a copied cell | Kept separate, and the distinction is stated: that page gates a **hand-written** document with grep/parse checks; the new page covers a **generated** document, where the generator itself becomes the drift source and `--check` is self-consistent. Cross-linked both directions |
| `backend-common-api-design-unenforced-declarations` | Declared-but-not-enforced behavior | Cited from the new page's edge case for "the runtime ignores unknown members" — the reason this gate matters more, not less, in that case |
| `testing-quality-tests-that-cannot-fail` | Red-run proof for gates | `related:` link only; no content duplicated |
| `infrastructure-agent-orchestration-control-signals-vs-primary-artifacts` | Read for candidates 2 and 3 | It already owns the stalled-vs-dead-vs-alive case; PR #47 extends it with exactly these two candidates. No edit made |
| `infrastructure-agent-orchestration-pane-delivery-confirmation` | Read for candidate 2's delivery mechanics | Already covers send/confirm; nothing to add |

Conflicts flagged: none. No existing directive contradicts the new page.

Link-integrity note: the new page originally cited
`backend-common-change-impact-widening-a-closed-value-table`, which exists **only** on
open PR #51's branch and not on `main`. Every `related:` id was resolved against this
checkout's `wiki/` and that one was replaced with
`backend-common-api-design-unenforced-declarations` so the page carries no dangling id
if #51 is rejected.

Edits to existing files: `wiki/qa/index.md` (+1 row), plus `related:` back-links and a
`last_verified` bump on `spec-document-gates.md` and
`generated-artifacts-as-deliverable-source.md`. No existing body text was rewritten.

## Open-PR check

Listed with `gh pr list --repo choiyounggi/dev-loop --state open --search "head:knowledge/"`
— 14 open heads: #68, #66, #64, #62, #61, #58, #57, #56, #55, #52, #51, #50, #49, #47.
Six with plausibly overlapping triggers were fetched and diffed against `origin/main`
under `wiki/`: #68, #66, #64, #51, #50, #47.

| Candidate | Overlapping open head | Verdict |
|-----------|----------------------|---------|
| 1 — generated closed-vocabulary reference + `--check` gate | #66 (`knowledge/choiyounggi-20260808-013406`) touches `qa/document-verification/spec-document-gates.md` and adds `qa/deliverables/command-transcripts-in-a-document.md`; #51 adds `backend/common/change-impact/widening-a-closed-value-table.md` | **new** — #66 gates a hand-written table by resolving the owning constant; it does not cover generating the document, `--check` in the suite, the generated banner, or the fact that `--check` cannot detect a hardcoding generator. #51 is about widening a closed value table in code, not its reference document. No open head adds a page under a generated-reference trigger |
| 2 — usage-limit pause vs. crash | #47 (`knowledge/dch0202-20260806-130040`) | **drop** — #47 already adds this verbatim as an edge-case row on `control-signals-vs-primary-artifacts.md` ("search each worker's pane/terminal tail for the CLI's usage-limit marker (e.g. `You've hit your session limit · resets HH:MM`) … re-verify state → remaining definition-of-done → completion signal"), plus the matching `infrastructure/index.md` "load when" extension and the same 2026-08-06 three-worker/`resets 01:10` field-evidence line. The only delta in the queued row was naming the send mechanism (`orca terminal send --enter`, tmux `send-prompt.sh send`), which `pane-delivery-confirmation` already owns — not worth a push to a stale branch |
| 3 — dispatch after `worker_done` | #47 (same head) | **drop** — #47 carries it as the adjacent edge-case row, including `orca terminal wait --for tui-idle` as the precondition, "the done message is the worker's report time, not the substrate's release time", and the consumed-Task rule ("create a new task from the same spec — the consumed one cannot be retried"). Nothing in the queued row is absent |

No sibling duplicate PR was opened, and nothing was pushed to #47.

## Routing decision

- New categories: `infrastructure/agent-orchestration` (5 pages; unified the
competing `orchestration`/`agent-orchestration` names), `databases/data-survey`
(1), `qa/deliverables` (1). All other pages route into existing categories.
- Canonical-path decisions: rate limiting → `backend/common/reliability/`
(sits beside timeouts-and-retries; 6 of 8 variants chose it); stderr
diagnostics → `platforms/processes/` (concern spans beyond shells); leaked
artifacts → `testing/data/artifact-leakage-from-a-suite`; call-site
enumeration → the existing `backend/common/change-impact/` page.
- All 38 new pages listed in their domain indexes (nearest-index rule; backend
routes via its python sub-index for bytecode-cache-staleness); INDEX.md domain
summaries updated for infrastructure/qa/databases. Full-wiki lint: frontmatter,
ids, related-links, index coverage, size, qualifiers, staleness → 0 findings.
| Insight | Target | New category? |
|---------|--------|---------------|
| 1 | `qa` / `document-verification` / `generated-reference-drift-gates.md` (id `qa-document-verification-generated-reference-drift-gates`) — **new page** | No. `document-verification` already exists and owns automated checks that decide whether a document meets its requirements; this is that, for a generated document. `qa/deliverables` was considered and rejected: its trigger is producing a hand-off deliverable, not gating a shipped one |
| 2 | none — retired as a pending duplicate of #47 | — |
| 3 | none — retired as a pending duplicate of #47 | — |

The harvested `domain: platforms` hint on insight 1 was overridden: `platforms` covers
OS-level differences across macOS/Linux/Windows, and nothing about this case is
OS-dependent. Insights 2 and 3 also carried `platforms` hints; #47 had already routed the
same content to `infrastructure/agent-orchestration`, which is where it belongs.

All 3 queued rows are retired from the active queue (1 ingested, 2 dropped) so neither
dropped row re-crosses the auto-flush threshold.
Loading
Loading