A Python/Cython implementation of Pruned BPE, a Byte Pair Encoding tokenizer that improves visible vocabulary efficiency by pruning low-exposure tokens while preserving internal merge structure.
Byte-Pair Encoding (BPE) (subword-based tokenization) algorithm implementaions from scratch with python
An educational Python project for learning tokenization step by step by building character-level, byte-level, and BPE tokenizers from scratch.
Controlled comparison of subword tokenizer vocabularies (BPE vs Unigram-LM) over chemistry SMILES
Pure Rust runtime for loading SentencePiece models and encoding/decoding text 🦀.
LLM-inspired BiLSTM pipeline for real-time, multi-label toxicity inference across adversarial discourse modalities.
Open byte-level BPE tokenizer for Uzbek (16k vocab) — 1.85 tokens/word vs GPT-4's 2.62, at ~12× smaller vocab. Built-in apostrophe/okina normalization; loads with Hugging Face Transformers.
Implementation of Deep Averaging Networks (DAN) for sentiment classification with experiments on GloVe embeddings and subword tokenization using Byte Pair Encoding (BPE).
A clean, educational implementation of the Byte Pair Encoding algorithm used in modern language models like GPT.
Custom BPE tokenizer built from scratch on WikiText-2 (30k vocab). Covers data cleaning, deduplication, HuggingFace tokenizers training, evaluation (compression ratio, UNK-free coverage, consistency), and save/reload as PreTrainedTokenizerFast.
Paper: A Comparison of Different Tokenization Methods for the Georgian Language
Build, analyze, and visualize a custom Byte Pair Encoding (BPE) tokenizer trained on the WikiText-2 dataset with vocabulary analysis, compression evaluation, and an interactive GitHub Pages demo.
From-scratch Byte Pair Encoding (BPE) tokenizer with a multilingual compression-ratio study (English, German, Spanish, French) benchmarked against GPT-2/3.5/4. Includes full research report, reproduction script, and notebook.
This repository hosts our comprehensive study on text tokenization methods, covering word-, character-, and subword-level algorithms such as BPE, WordPiece, and Unigram, and extends to discussions on multilingual, mathematical, and code tokenization. It examines their efficiency, consistency, semantic preservation, and influence on LLMs.
A minimal Python implementation of Byte Pair Encoding (BPE) with step-by-step visualization of merge operations and vocabulary updates.
BPE & Unigram Vocab Training library
NeuralPiece: An adaptive byte-level neural tokenizer designed to surpass traditional BPE and Unigram via deep learning chunking.
Merge Barriers in BPE Tokenization: How Tokenizer Design Causally Determines Attention Head Specialization. Paper, 23 eval scripts, 86 result files, tokenizer definitions, 4 model checkpoints.
Add a description, image, and links to the subword-tokenization topic page so that developers can more easily learn about it.
To associate your repository with the subword-tokenization topic, visit your repo's landing page and select "manage topics."