Skip to content

fix(cues): session-2 precision pass — memory-test rule, word-boundary clips, 0.35 dedup backstop - #86

Merged
xerhab merged 1 commit into
mainfrom
fix/cue-session2-fixes
Jul 27, 2026
Merged

fix(cues): session-2 precision pass — memory-test rule, word-boundary clips, 0.35 dedup backstop#86
xerhab merged 1 commit into
mainfrom
fix/cue-session2-fixes

Conversation

@xerhab

@xerhab xerhab commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

What

Fixes the five cue-quality failure classes surfaced by the second real recording (21 min of entity-dense YouTube tech audio, conv 7ca38157, now a harness fixture alongside the family session and the frozen 12):

  1. Cross-generation confabulation — a model number newer than the weights got a sibling's specs pasted under its name ("Galaxy Z Fold 8 … launched August 2023"). Three prompt formulations iterated (v10→v12); the shipped v12 reframes the rule as a specific-memory test the model can actually run ("family resemblance is not knowledge — no specific memory, no cue"), extended to vaguely-recognized names/acronyms, plus an evidence-side rule (an article about the predecessor covers nothing about the successor) and a stale-date rule (never correct anniversaries/ages from the internal clock).
  2. False corrections — the correction trigger now requires positively knowing the truth, and never cues that a name the speakers used "does not exist" (production had inverted truth on "Jet Grind Radio", a real US regional title).
  3. Paraphrase duplicates — avoid-list now bans re-approaching a surfaced subject from a different angle; session substance-dedup backstop lowered 0.5 → 0.35, calibrated on 90 real production cues (retitled rewords measure 0.30–0.38; closest genuinely-distinct pairs 0.26–0.27 — 0.35 is the safe floor).
  4. Mid-word truncation — 8 of 51 production bodies were chopped by the hard body[:240] slice; replaced with a word-boundary clip + ellipsis and an "under 200 characters" nudge (final replay: 2/401 clipped, cleanly).
  5. Stale-topic cues — newest-turns recency rule (fast topic-switching audio kept getting cues for topics left 60+s earlier). Also: lifestyle-advice ban, bilingual mishear rule (a stray foreign-looking word is the speakers' other language, not Dutch).

Verification

Replayed the full 14-conversation export (985 attempts) through the deployed gpt-oss-120b (medium effort) per iteration, hand-bucketed both real sessions each round, self-judged comparatively — full record in scripts/cue_eval/RESULTS-2026-07.md:

run cues novelty relevance accuracy judge dups garbled ctrl question conv
v9 baseline 461 1.87 1.93 1.92 28 10 1
v12 (this PR) 401 1.88 1.90 1.94 17 5 5

Hand-bucketed on session 2: wrong/confabulated ~15 → ~5±2 per run, paraphrase-dup pairs 6+ → ~1-2, stale/dictionary near zero, and the direct-question conversation went from near-mute (1 cue) to properly answered (5). Volume lands at 87% of baseline with the cuts concentrated in the failure classes.

Known residual (documented): first-party sibling-spec transfer (the Z Fold 8 case) survives every prompt formulation — the model genuinely believes it knows the product. The fix is the already-planned retrieval cross-check of entity cues, not more prompt text.

Tests: full API suite 306 passed, coverage 96.04% (gate 85%), ruff clean. New tests pin every new prompt rule, three truncation cases, and a session-level regression test using the measured 0.38 paraphrase pair.

….35 dedup

Five failure classes from the second real recording (21 min of entity-dense
YouTube tech audio, conv 7ca38157, now a harness fixture), fixed and verified
by replaying the full 14-conversation set through the deployed gpt-oss-120b:

- Cross-generation confabulation: prompt v10-v12 iterations landed on a
  specific-memory test ("family resemblance is not knowledge; no specific
  memory, no cue") plus a stale-date rule (never correct anniversaries/ages
  from the internal clock) and evidence-side generation-mismatch coverage.
  Invented-detail cues roughly halved per run; the first-party sibling-spec
  case (Z Fold 8) survives and is documented for the retrieval cross-check.
- False corrections: correction trigger now requires positively knowing the
  truth; never cue that a name the speakers used "does not exist" (the Jet
  Grind Radio inversion).
- Paraphrase duplicates: avoid-list bans same-subject-different-angle cues;
  substance-dedup backstop 0.5 -> 0.35, calibrated on 90 real cues (rewords
  0.30-0.38 vs closest distinct pairs 0.26-0.27).
- Mid-word truncation: word-boundary clip + ellipsis and a 200-char body
  nudge (8/51 production bodies were chopped mid-word; now 2/401 clipped,
  cleanly).
- Stale-topic cues: newest-turns recency rule; also a lifestyle-advice ban
  and a bilingual mishear rule (stray foreign word != third language).

Final replay: 401 cues / 985 attempts (87% of baseline volume) with judged
accuracy 1.92 -> 1.94, judge-flagged dups 28 -> 17, garbled control halved,
question-only conversation 1 -> 5 cues. Full record in
scripts/cue_eval/RESULTS-2026-07.md.
@xerhab
xerhab merged commit 6ce0f5e into main Jul 27, 2026
3 checks passed
@xerhab
xerhab deleted the fix/cue-session2-fixes branch July 27, 2026 12:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant