docs(skills): cue-eval-review — recorded-session review & replay-eval methodology - #87
Merged
Conversation
…y-eval methodology Captures the workflow behind the 2026-07 cue precision cycles (PRs #80/#81/#86) as a reusable Claude skill: DB export commands, the hand-review failure-class taxonomy, the replay/judge harness rules that keep comparisons honest (frozen set, re-baseline per export, subset-then-full, stochastic-effect handling), threshold calibration on real data, and the ship/document steps.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a Claude skill at
.claude/skills/cue-eval-review/SKILL.mddocumenting how cue quality is reviewed and evaluated, so future sessions follow the same methodology that produced the 2026-07 precision cycles (#80, #81, #86):docker exec Tenir-Postgres psqlqueries for segments and cues (host has no local psql).at_ms, bucket bad cues into named failure classes (cross-generation confabulation, false corrections, paraphrase duplicates with the measured similarity bands, stale-topic, mishears), review-only until asked to fix. Includes the NFKC-normalization gotcha for regex probes.scripts/cue_eval/honestly: freeze the conversation set with controls, re-baseline on the current export, iterate subset→full, hand-read every iteration, match production decoding, treat stochastic effects as halved-not-fixed, and use the self-judge only comparatively.Privacy note baked into the skill: deployment transcripts stay in analysis for the user and never get pasted into shipped code, prompts, tests, or docs.
Verification
Docs-only (no code paths touched). CLI flags and commands cited in the skill were checked against the current
replay.py/judge.pyargument parsers and thecues/segmentstable schemas in the deployment DB.