Skip to content

Normalize Mistral tokenizer regexes for Laguna - #61

Merged
Qubitium merged 1 commit into
mainfrom
agent/laguna-mistral-regex-fix
Jul 22, 2026
Merged

Normalize Mistral tokenizer regexes for Laguna#61
Qubitium merged 1 commit into
mainfrom
agent/laguna-mistral-regex-fix

Conversation

@Qubitium

Copy link
Copy Markdown
Contributor

Summary

  • enable the Transformers fix_mistral_regex correction by default for Tokenicer string loads
  • preserve explicit caller overrides and carry normalized kwargs through fallback loaders
  • retry tokenizer-only loading for the newer Hugging Face strict dataclass base exception
  • add regression tests and bump Tokenicer to 0.0.14

Root cause

Laguna S 2.1 ships a Mistral-family pre-tokenizer regex that Transformers identifies as incorrect. GPTQModel and other downstream loaders should receive the correction from Tokenicer instead of maintaining per-model tokenizer patches.

Validation

  • python -m pytest -q tests/test_mistral_regex.py — 3 passed
  • python -m pytest -n 8 -q tests — 467 passed, 4 environment/checkpoint skips
  • real /monster/data/model/Laguna-S-2.1 comparison — corrected token boundaries for eBay, mL, NaOH, and pH
  • python -m ruff check tokenicer/tokenicer.py tests/test_mistral_regex.py tests/test_loop_models.py
  • source distribution build for 0.0.14 and twine check

@Qubitium
Qubitium marked this pull request as ready for review July 22, 2026 05:09
@Qubitium
Qubitium merged commit 3718e2d into main Jul 22, 2026
26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant