fix(cues): session-2 precision pass — memory-test rule, word-boundary clips, 0.35 dedup backstop - #86
Merged
Merged
Conversation
….35 dedup
Five failure classes from the second real recording (21 min of entity-dense
YouTube tech audio, conv 7ca38157, now a harness fixture), fixed and verified
by replaying the full 14-conversation set through the deployed gpt-oss-120b:
- Cross-generation confabulation: prompt v10-v12 iterations landed on a
specific-memory test ("family resemblance is not knowledge; no specific
memory, no cue") plus a stale-date rule (never correct anniversaries/ages
from the internal clock) and evidence-side generation-mismatch coverage.
Invented-detail cues roughly halved per run; the first-party sibling-spec
case (Z Fold 8) survives and is documented for the retrieval cross-check.
- False corrections: correction trigger now requires positively knowing the
truth; never cue that a name the speakers used "does not exist" (the Jet
Grind Radio inversion).
- Paraphrase duplicates: avoid-list bans same-subject-different-angle cues;
substance-dedup backstop 0.5 -> 0.35, calibrated on 90 real cues (rewords
0.30-0.38 vs closest distinct pairs 0.26-0.27).
- Mid-word truncation: word-boundary clip + ellipsis and a 200-char body
nudge (8/51 production bodies were chopped mid-word; now 2/401 clipped,
cleanly).
- Stale-topic cues: newest-turns recency rule; also a lifestyle-advice ban
and a bilingual mishear rule (stray foreign word != third language).
Final replay: 401 cues / 985 attempts (87% of baseline volume) with judged
accuracy 1.92 -> 1.94, judge-flagged dups 28 -> 17, garbled control halved,
question-only conversation 1 -> 5 cues. Full record in
scripts/cue_eval/RESULTS-2026-07.md.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Fixes the five cue-quality failure classes surfaced by the second real recording (21 min of entity-dense YouTube tech audio, conv
7ca38157, now a harness fixture alongside the family session and the frozen 12):body[:240]slice; replaced with a word-boundary clip + ellipsis and an "under 200 characters" nudge (final replay: 2/401 clipped, cleanly).Verification
Replayed the full 14-conversation export (985 attempts) through the deployed gpt-oss-120b (medium effort) per iteration, hand-bucketed both real sessions each round, self-judged comparatively — full record in
scripts/cue_eval/RESULTS-2026-07.md:Hand-bucketed on session 2: wrong/confabulated ~15 → ~5±2 per run, paraphrase-dup pairs 6+ → ~1-2, stale/dictionary near zero, and the direct-question conversation went from near-mute (1 cue) to properly answered (5). Volume lands at 87% of baseline with the cuts concentrated in the failure classes.
Known residual (documented): first-party sibling-spec transfer (the Z Fold 8 case) survives every prompt formulation — the model genuinely believes it knows the product. The fix is the already-planned retrieval cross-check of entity cues, not more prompt text.
Tests: full API suite 306 passed, coverage 96.04% (gate 85%), ruff clean. New tests pin every new prompt rule, three truncation cases, and a session-level regression test using the measured 0.38 paraphrase pair.