fix(narration): match atWord anchors outside ASCII - #45
Conversation
`normalizeWord` strips a token with `[^\w']`, and JavaScript's `\w` is
`[A-Za-z0-9_]` without the `u` flag. Every Cyrillic, Han, Devanagari, Greek,
Hebrew and Arabic token therefore normalises to the empty string and they all
compare equal, so `atWord` returns whichever word comes first in the scene for
any anchor. It returns a plausible number rather than null, so nothing reports
it. Accented Latin is mangled rather than erased: "Anträge" only matches a
caller who writes "antrge".
`\p{L}\p{M}\p{N}` with the `u` flag. `\p{M}` matters as much as the rest:
combining marks carry the sound in Devanagari, Thai and Arabic, so without them
"काल" still collapses onto "कल" and the anchor still lands on the wrong word.
Keeping marks means a decomposed spelling stops equalling a precomposed one, so
normalise to NFC first, otherwise an NFD "Anträge" never matches Whisper's NFC
"Anträge" and the effect silently never fires.
Also refuse an anchor that strips to nothing rather than matching the first
token that also strips to nothing, which returned a confident 1000ms for
punctuation.
10 of the 12 new cases fail against the previous expression. The one caller in
demos/showcase.demo.ts reads `atWord(...) ?? fb`, so a stricter miss falls back
to its fixed timeout.
|
Thank you @Joilence good catch |
|
Thanks @Joilence — this one is worse than it looks from the diff, which is exactly why it's a good fix.
Verified:
One edge case for a follow-up, not a change request here. Turkish/Azerbaijani dotted capital I still misses: Worth recording that the obvious fix doesn't work — I checked before suggesting it:
Filing that separately. Merging this. Thanks again — both of these were well-scoped and well-tested. |
Why
atWordnormalises the anchor and every transcript token throughnormalizeWord:\wwithout theuflag is[A-Za-z0-9_], so any token with no ASCII letters strips to''and they all compare equal:The match loop returns on the first equal token, so a Cyrillic, Han, Devanagari, Greek, Hebrew or Arabic anchor does not miss. It matches word one and returns a confident wrong time. A three-word Russian scene gives 1000ms for its last word, and 1000ms for a word it does not contain at all. Accented Latin is mangled rather than erased, so
Anträgeis reachable only by a caller who also drops the umlaut.NarrationTimelineis public API (index.ts:29).What
\p{L}\p{M}\p{N}under theuflag, which Unicode property escapes require.\p{M}included: combining marks carry sound in Devanagari, Thai and Arabic, so dropping them collapsesकालontoकलand the anchor still lands wrong.ASCII behaviour is unchanged: case and trailing punctuation still fold away.
Test
12 cases in
tests/narration-atword.test.tscovering ru, zh, hi, de, en, two combining-mark pairs, NFC/NFD and punctuation. Revertingsrc/narration.tsto 040d8b5, 10 of the 12 fail:The 2 passers are
deanden, the cases the bug does not reach: diacritic Latin strips identically on both sides, ASCII is untouched.npm teston the branch: 752 passed.npm run buildclean. The test has no capability gate and the repo has no vitest config, sonpm testin CI picks it up under vitest's default include.Notes
atWord(...) ?? fb, so a stricter null falls through to its fixed timeout.atWordis not in the README, so no docs change.