Inline link discovery: 91 high-confidence proposals + completed learned-engine evaluation by elimelt · Pull Request #138 · elimelt/notes

elimelt · 2026-08-01T08:41:00Z

Summary

Intent: everything from the inline-link end-to-end run in one PR — the reviewable high-confidence link list, the windowed-Qwen learned engine completion, the full evaluation, and the corpus note update. No notes are modified by the engine; proposals only.

What's in here

The production list (`linkdiscovery/reports/inline-link-proposals-2026-08-01.{md,jsonl}`)

91 anchored inline links across 67 notes from the audited deterministic baseline engine at threshold 0.65, with per-note budgets, MMR, and hard constraints. Top picks: "the physical layer" → physical-layer note, "load balancing" (HTTP) → load-balancing, "flow control" → transport flow-control, "memory-level parallelism" → the MLP benchmark.

The learned engine, finished end to end

Decision: windowed Qwen token-state encoding (512/384 overlap, deepest-inside-window state selection per the spec's causal-pooling caveat), token-state caching, --token-encoder hashing|qwen CLI wiring (+1,327 lines incl. 18 new tests; suite now 1,105).
Learned: the encoder swap fixed retrieval exactly as SPEC §9 predicted — held-out recall@1 went 0.0 → 0.964 (tying the anchor dictionary). The binding constraint moved to the naturalness head (AUC 0.664 on 9 test negatives) — label-limited, not encoder-limited.
Decision (spec thresholds): plateau verdict — the baseline stays the production engine; the learned stream (69 accepted @0.40, inline-link-proposals-learned-2026-08-01.md) ships as a supplementary review stream, almost disjoint from the baseline set (1 shared span). Full tables: inline-eval-summary-2026-08-01.md + raw JSONs.

The audit behind it (full disclosure)

300 stratified links judged by two independent Claude annotator agents (delegated by the repo owner), one guideline-refinement round after κ(anchor_natural)=0.41 on title-shaped anchors. Final: κ(target)=1.00, κ(anchor)=0.98, κ(placement)=0.88 → GO (153 clean positives; 124 A / 29 B / 147 C / 0 D).
Learned: zero wrong-target links in 300 — corpus link noise is entirely anchor phrasing and duplicate Related-notes placements.

Note update

ml/nlp/missing-link-discovery's v2 section now carries the final results (audit kappas, the encoder natural experiment, verdict, this PR's pointer).

Review notes

Constraint: next lever per the spec is Wikipedia-keyphraseness pretraining for the naturalness head + more negative anchor labels — not LoRA.
Some notes propose the same target at multiple spans (e.g. "connected components" ×3 in one proof note) — pick one when applying.
Gates: 1,105 tests, mypy strict (inline package clean), ruff lint+format green; Quartz build passes.

🤖 Generated with Claude Code

…1 links) intent(reports): reviewable list of anchored inline links the corpus is missing, from the inline subsystem's audited baseline engine at threshold 0.65 — proposals only, no notes modified decision(engine): deterministic baseline per the SPEC §12 fallback; the learned path ran but its retrieval head barely beat uniform on the test-grade hashing token encoder, so it does not gate this list decision(audit): 300-link audit run by two independent Claude annotator agents at the owner's request, with one guideline-refinement round; final kappas 0.88-1.00, GO verdict (153 clean positives) learned(corpus): zero wrong-target links in 300 audited; all label noise is anchor phrasing and duplicate Related-notes placements Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

intent(inline): replace the test-grade hashing token encoder as the learned path's production encoder so heads train on real representations decision(windowing): overlapping 512/384 windows with per-token global char offsets; multi-window tokens take the state where the token sits deepest inside its window (most left context, per the spec's causal last-token-pooling caveat); windowing params are fingerprinted decision(caching): token states and catalog vectors cached in the artifact store keyed by encoder fingerprint + content hash, mirroring the embedding-cache discipline — corpus re-encoding drops to zero on re-runs decision(cli): --token-encoder hashing|qwen on inline train/propose, defaulting to hashing so tests stay model-free learned(eval): the encoder swap fixed retrieval exactly as the spec predicted (held-out recall@1 0.0 -> 0.964); the binding constraint moved to the naturalness head (AUC 0.664 on 9 test negatives), so the audited deterministic baseline remains the review-tool engine

…note intent(reports): publish the end-to-end evaluation of the learned inline engine against the deterministic baseline, plus the learned engine's supplementary proposal stream, alongside the production list decision(verdict): plateau per the spec's thresholds — baseline stays the review-tool engine; windowed Qwen token states fixed retrieval (held-out recall@1 0.0 -> 0.964) but the naturalness head (AUC 0.664, 9 test negatives) is now the label-limited binding constraint decision(ml/nlp-note): v2 section updated from "gated on the audit" to final results — audit kappas, tier distribution, encoder natural experiment, and the PR pointer

… directions intent(reports): consolidate the quantitative eval, a rank-depth qualitative sample with observed failure modes, and prioritized tuning directions into one reviewable report learned(failures): four concrete failure modes documented — naturalness label scarcity, cross-domain homonym anchors, same-target duplication next to existing links, and generic single-noun tail anchors — the first two bind the learned engine, the last two are mechanical selection fixes decision(priorities): P0 is negative-label acquisition (review-decision harvesting, rule-defined synthetic negatives, Wikipedia keyphraseness ablation) and two selection rules; LoRA explicitly gated behind that data

elimelt · 2026-08-01T18:46:46Z

Added reports/inline-engine-performance-report-2026-08-01.md — full performance assessment (quantitative tables, rank-depth qualitative sample including the honest failures at ranks 31/61/89, four documented failure modes) and prioritized tuning directions (P0: negative-label harvesting from review decisions + two mechanical selection rules; P1: calibration on real review outcomes, frozen expert benchmark, reranker bake-off; P2: LoRA explicitly gated behind the new data).

…ne scoring intent(inline): a 160-item case-by-case review of the production proposal list measured 37.4% precision — dominated by four mechanical failure classes, not model quality; fix them outside any learned score decision(select): Rule A rejects drafts whose target is already linked within 600 chars (reason near_existing_same_target, gap recorded); Rule C raises the naturalness floor to 0.5 for lowercase single-word anchors (kills generic tail like "resistance"); Rule D caps accepted proposals per (note, target) at 1 in MMR pick order — the existing same-target redundancy penalty only reordered, it never rejected, so one note accepted the same target six times decision(baseline): Rule B multiplies target correctness by 1 - 0.35*(1 - same_family) for cross-family pairs, exempting TitleCase/acronym anchors — "Paxos" names one global concept while lowercase "memory management" is family-polysemous; fixes the observed OS-note -> LLM-serving wrong-domain class constraint(select): rejected drafts stay in the proposal set as abstained records with per-rule reason flags, keeping every selection decision auditable learned(select): measured against the 160 review verdicts the rules lift precision 37.4% -> 66.7% while retaining 88.2% of review-accepted links, with zero unreviewed newcomers entering the list

… runner intent(inline): close the improvement loop the performance report specified — review decisions become training negatives and calibration data, and a frozen benchmark makes every future tuning claim falsifiable decision(train): review decisions route per-head instead of through audit-tier semantics — anchor_ok is direct naturalness ground truth (always labeled), accept->A, reject+target_ok->B, wrong target->D with target_index at the recorded wrong target for the reranker's negative BCE decision(calibrate): temperature fits on review-outcome logits per engine; application unified into one shared post-draft pass (_calibrate_drafts) used by both engines, so double-application is structurally impossible decision(benchmark): runner locates nullable-span cases by first verbatim anchor occurrence; draft-level kinds (natural/acceptable span, correct target) read head scores, selection-level kinds (no_link, placement, incorrect_target, reverse) read the accepted set rejected(train): mapping review verdicts onto audit tiers wholesale — it would discard anchor judgments on Tier-B items, the exact signal the naturalness head is starved of learned(train): the reranker's Tier-D BCE had a latent shift-invariance hole — listwise CE pins only relative order, and with no positive BCE anchor the optimizer shifts all logits down, collapsing every absolute probability to 0 while training loss stays ~0; audit data had zero Tier-D rows so it never fired until the review harvest. Fixed by anchoring the BCE step with true pairs at 1.0; regression test asserts positives clear an absolute floor constraint(workflow): existing-link suppression exempts Related-notes zones per the audit guideline duplication rule — prose is the preferred home; the two review-accepted items Rule A initially removed were both blocked solely by navigation entries

…ration intent(inline): make the measurement basis durable — the benchmark is the falsifiability instrument, the review decisions are the training and calibration provenance decision(benchmark): 53 hand-authored cases across the seven SPEC §7 judgment kinds, 24 hard-case-flagged (cross-family homonyms, index notes, a zero-link personal note, unit tokens, duplicate-placement traps); built exclusively from unlinked spans so it shares nothing with audit labels or review decisions — the artifact stays valid as training data grows constraint(benchmark): never train on it; anchors verified verbatim against the corpus at authoring time learned(reviews): 160 decisions (91 baseline, 69 learned) measure honest precision 37.4% / 4.3% at the shipped operating points; fitted temperature is 20.0 for both engines (scores carry rank information but almost no absolute-probability information), baseline ECE 0.35 -> 0.14

…retrained eval intent(reports): every intervention this round was measured against the 160-item review ground truth; this records the numbers and the two discoveries that reframe the tuning priorities decision(reports): the production list is now the 45-link rules list (66.7% measured precision, 88.2% good-link retention); the 244-item retrained learned list ships as an uncalibrated discovery stream only learned(benchmark): both engines score an identical 0.547 on the frozen benchmark because span candidate generation cannot propose never-before-linked lowercase phrases — the binding constraint is upstream of every head, and no amount of head tuning moves it learned(eval): review-harvested negatives raised the naturalness head's held-out AUC 0.664 -> 0.767 and widened the natural/not-natural gap 0.05 -> 0.25; retrieval stays saturated at 0.964 through the reranker fix

elimelt · 2026-08-01T22:45:26Z

Tuning round 1 — all three follow-ups executed, each measured

Executed the report's P0/P1 directions end to end: case-by-case review of all 160 shipped proposals, four mechanical selection rules, review-harvested retraining + calibration, and the frozen expert benchmark. Full write-up: reports/inline-tuning-round-1-2026-08-01.md.

Measured, in order

Honest precision of the shipped lists (160-item review, per-head verdicts, preserved in reviews/): baseline 37.4%, learned 4.3%. The earlier qualitative estimate ("good through rank ~60") was wrong — precision is 75% in ranks 1–20 and ~25–35% after.
Four selection rules (existing-target proximity w/ Related-notes exemption, cross-family penalty with TitleCase exemption, single-word floor, per-note same-target cap — the review found MMR's redundancy penalty reorders but never rejects): baseline precision 37.4% → 66.7% at 88.2% good-link retention. New production list: inline-link-proposals-rules-2026-08-01.md — 45 links, every one review-verdicted.
Review-harvested retraining: naturalness held-out AUC 0.664 → 0.767 (gap between natural/not-natural anchors widened 5×). The retrain exposed and fixed a latent reranker bug — shift-invariant listwise loss + negative-only BCE collapsed all absolute probabilities to 0 the first time real wrong-target negatives existed; fixed with positive BCE anchors + regression test.
Calibration (fit on the 160 outcomes): baseline ECE 0.35 → 0.14; the calibrated learned engine now correctly accepts zero proposals at review thresholds — at 4.3% measured precision, that's the system being honest.
Frozen benchmark (53 cases, 7 judgment kinds, zero overlap with any training data): both engines score an identical 0.547, all failures are misses, zero false links — because the bottleneck is upstream of both engines: span candidate generation never proposes never-before-linked lowercase phrases ("congestion collapse", "sampled softmax"). That's the new P0.

New in the PR

linkdiscovery/benchmarks/expert-benchmark-v1.json (frozen, never trained on), linkdiscovery/reviews/ (decisions + calibration), 4 selection rules + review-training + calibration + benchmark runner (inline calibrate, inline benchmark, inline train --reviews, inline propose --calibration), the tuning report, v2 proposal lists, retrained eval JSON. Suite: 1223 passed, mypy/ruff clean.

The improvement loop is now fully wired: reviewing the 45-link list produces the next round's negatives and the retrained engine's first valid calibration automatically.

🤖 Generated with Claude Code

elimelt and others added 3 commits August 1, 2026 01:40

elimelt changed the title ~~High-confidence inline link proposals: 91 anchored links across 67 notes~~ Inline link discovery: 91 high-confidence proposals + completed learned-engine evaluation Aug 1, 2026

elimelt added 4 commits August 1, 2026 15:08

Navigation Menu

Inline link discovery: 91 high-confidence proposals + completed learned-engine evaluation - #138

Inline link discovery: 91 high-confidence proposals + completed learned-engine evaluation#138
elimelt wants to merge 8 commits into
mainfrom
inline-link-proposals

elimelt commented Aug 1, 2026 •

edited

Loading

Uh oh!

elimelt commented Aug 1, 2026

Uh oh!

elimelt commented Aug 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

elimelt commented Aug 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What's in here

The production list (linkdiscovery/reports/inline-link-proposals-2026-08-01.{md,jsonl})

The learned engine, finished end to end

The audit behind it (full disclosure)

Note update

Review notes

Uh oh!

elimelt commented Aug 1, 2026

Uh oh!

elimelt commented Aug 1, 2026

Tuning round 1 — all three follow-ups executed, each measured

Measured, in order

New in the PR

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

elimelt commented Aug 1, 2026 •

edited

Loading

The production list (`linkdiscovery/reports/inline-link-proposals-2026-08-01.{md,jsonl}`)