Add a corpus-eval skill for evaluating parsing changes against real data - #320
Merged
Conversation
The agent parses two undocumented formats owned by someone else — Claude Code's transcript JSONL and its tmux pane — and this host happens to hold every transcript the fleet has written plus a live Claude to produce real dialogs. The recent chat-fidelity work (#317, #318, #319) leaned on that corpus rather than on hand-written fixtures, and found things unit tests structurally cannot: shapes occurring 5 times in 61k entries, and a py-vs-js mirror divergence that turned out to predate the change. Writes that method down so the next parsing change doesn't rediscover it: inventory the shapes before designing, replay old-vs-new over one snapshot classifying diffs into identical/intended/regression, measure the py/js mirror parity against its PRE-EXISTING baseline, count what the change actually surfaces, and capture pane fixtures from a live TUI instead of inventing the glyphs. Records the traps that each cost real time, including the two worth the most: comparing JSON.stringify output across the two languages reports ~30k mismatches where there are 13 (key insertion order differs — deep-equal the parsed structures), and those 13 are a codepoint-vs-UTF-16 clip-length artifact on emoji-bearing entries, not a mirror bug. Both recipes were re-run as written while drafting this, and reproduce the documented figures (60,615 entries identical old-vs-new on a no-op branch; 13/38,425 mirror mismatches, each confirmed non-BMP).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Documents the evaluation method the recent chat-fidelity work (#317, #318, #319) used, so the next parsing change doesn't rediscover it.
Why
agent/parses two undocumented formats owned by someone else — Claude Code's transcript JSONL and its tmux pane. Unit tests can only assert what you already believed the format was, which is exactly the wrong instrument. This host happens to hold every transcript the fleet has ever written (~580, ~61k entries, spanning many Claude Code versions) and can produce real TUI dialogs on demand.Leaning on that found things tests structurally cannot:
ExitPlanMode) — real, and invisible to sampling;What the skill covers
hub-agent.pyvstunnel-agent.jsover the corpus, measured against the divergence that already exists.Plus the traps that each cost real time. The two most valuable:
JSON.stringifyoutput across the two languages reported 29,761 mismatches where there were 13 — key insertion order differs between the languages, so deep-equal the parsed structures instead._clipcuts at 2000 codepoints in Python andclipat 2000 UTF-16 units in JS, so a block whose first 2000 characters contain a non-BMP character (emoji) truncates one character apart. Confirmed by inspection —nonBMP=true,pyCodepoints=2000vsjsCodepoints=1999, nothing but the cut point differing.Verification
Both recipes were re-run as written while drafting, and reproduce the documented figures: 60,615 entries identical old-vs-new on a no-op branch, and 13 / 38,425 mirror mismatches with every one confirmed non-BMP.
Every count quoted was re-measured rather than copied from the earlier session — which surfaced something worth recording in the skill itself: the corpus is live and grows under you (577 transcripts where an earlier pass saw 559), so the counts are illustrative rather than fixtures, and a differential must read one snapshot for both sides or entries appearing mid-run show up as phantom diffs.
Notes
No code changes; docs only.
CLAUDE.mdis deliberately untouched — it doesn't reference the existingverifyskill either (skills are auto-discovered from.claude/skills/), and it is currently over its own 150k budget.