Update language data to langtags.json API 1.4 (BL-15916) - #149
Merged
Conversation
https://issues.bloomlibrary.org/youtrack/issue/BL-15916 Update the source data files to the latest releases: - langtags.json: API 1.3.1 (2025-06-16) -> API 1.4 (2026-06-09) - iso-639-3.tab: latest ISO 639-3 code table (renames + new codes) - iso-639-3-macrolanguages.tab: re-downloaded (content unchanged) As of langtags.json API 1.4, most region-specific entries (e.g. en-GB, de-AT, the ar-Arab-* set) no longer carry the iso639_3 field. Without it, langtagProcessing.ts was skipping these entries entirely, which dropped the bulk of each language's regions and alternative tags (e.g. English regions 99 -> 4, French 28 -> 1). langtagProcessing.ts now derives the iso639_3 code from the language subtag when the field is absent, so those entries consolidate into their language as before; the rest of the pipeline (including macrolanguage handling) then treats them like any normal entry. Regenerated languageData.json and equivalentTags.json from the new data. Also updated two data-sensitive assertions in languageSearch.spec.ts that reflected values changed upstream in API 1.4: - "Kamarakotos" (aoc's former autonym) was removed from langtags.json, so the find-by-autonym test now uses Korean (kor). - caa (Ch'orti')'s autonym now begins with "cho", so it legitimately ranks above xco (Chorasmian) for the query "cho"; the ranking assertion was updated accordingly. Ran the full find-language test suite (npx vitest run): 172/172 pass. The langtag-processing pipeline's remaining console warnings are all the documented known macrolanguage anomalies (nor, san, hbs, zap). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ndling https://issues.bloomlibrary.org/youtrack/issue/BL-15916 Per Noel's note on the card, add a test for our handling of the wsg/gon and arb/ar language pairs. arb/ar already had substantial coverage, but the Gondi macrolanguage (gon) and its individual members (wsg, esg, gno) had none. The new test asserts, for both a simple representative pair (arb/ara) and a multi-member macrolanguage (gon): - the macrolanguage and its individual members are all findable by name; - the representative individual language is surfaced by its own ISO 639-3 code (arb, gno) with the individual language subtag, not the macrolanguage subtag, and with parentMacrolanguage set; - searching the macrolanguage code (gon) surfaces the individual members; - wsg (Adilabad Gondi) absorbs the retired code "ggo", so searching "ggo" still finds it. Ran the full find-language test suite (npx vitest run): all pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
https://issues.bloomlibrary.org/youtrack/issue/BL-15916 The card notes that the raw source files and the files our pipeline generates from them lived together in one folder and should be separated. Move the raw source files into a new language-data/source/ subfolder, leaving the generated files (languageData.json, equivalentTags.json), which the runtime imports, in language-data/: - language-data/source/langtags.json - language-data/source/iso-639-3.tab - language-data/source/iso-639-3-macrolanguages.tab Updated the source-file import/read paths in langtagProcessing.ts and langtagProcessingHelpers.ts, and the Data sources section of the README. Verified: re-running the langtag-processing pipeline from the new paths produces byte-identical languageData.json and equivalentTags.json; tsc typecheck passes; the full find-language test suite passes (174/174). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
Author
|
Claude (Opus 4.8): Consulted Devin on 2026-07-22 15:58 USMST up to commit Devin's review is clean: 0 Bugs, 0 Investigate flags. It raised 3 Informational observations (not mirrored as threads, per our low-signal policy) — all confirming the change looks correct: (1) the |
…t cards https://issues.bloomlibrary.org/youtrack/issue/BL-15916 The "Search with typos gets fuzzy matches" e2e expected the Russian card to be on screen immediately after searching "rusian". The result list lazy-renders its cards (react-lazyload): a card far enough down the list isn't in the DOM until the list is scrolled near it, so scrollIntoViewIfNeeded has nothing to scroll to. Russian is a fuzzy match that isn't near the top ("Belarusian" is an exact substring match and outranks it), so it can sit below the initially rendered window — and the langtags.json API 1.4 data nudged it from list position 9 to 10, just past that window, which is why the test started timing out. This is not a search regression (Russian is still surfaced, just not at the top). Add a scrollListToLanguageCard helper that scrolls the results list a screenful at a time until the requested card mounts, then brings it into view, and use it in the typos test. This makes the assertion robust to where a card lands in the list rather than depending on it being in the initial lazy-render window. Note: could not run the e2e locally to confirm (an unrelated local-environment issue makes the beforeEach clearSearch step time out here); relying on CI, where the test actually fails, to validate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
Author
|
Claude (Opus 4.8): Consulted Devin on 2026-07-22 16:48 USMST up to commit |
andrew-polk
marked this pull request as ready for review
July 23, 2026 00:05
Contributor
Author
|
@nabalone Will you please look at at least the new macrolanguage test and the loop to add the iso639 code when it is missing? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Updates the language database to the latest source data and handles a breaking change in the new langtags.json format.
What changed
Update language data to langtags.json API 1.4 (
fix)langtags.json: API 1.3.1 (2025-06-16) → API 1.4 (2026-06-09)iso-639-3.tab: latest ISO 639-3 code table (renames + new codes)iso-639-3-macrolanguages.tab: re-downloaded (content unchanged)en-GB,de-AT, thear-Arab-*set) no longer carry theiso639_3field. The old pipeline skipped those entries entirely, which dropped the bulk of each language's regions and alternative tags (English regions 99 → 4, French 28 → 1, Arabic 27 → 3).langtagProcessing.tsnow derivesiso639_3from the language subtag when the field is absent, so those entries consolidate into their language as before and the rest of the pipeline (including macrolanguage handling) treats them normally.languageData.jsonandequivalentTags.json.Add coverage for wsg/gon and arb/ara macrolanguage handling (
chore, per Noel's note on the card)wsgabsorbs the retiredggocode.Separate raw source data from processed data files (
chore, per the card)language-data/source/; the runtime-imported generated files stay inlanguage-data/. Updated script paths and the README.Verification
tsctypecheck: clean.vitest run): 174/174 pass.Ref: https://issues.bloomlibrary.org/youtrack/issue/BL-15916
This change is