Minimal from-scratch implementations of the components inside a vision-language model (VLM), anchored on Qwen-VL.
Two tracks. llm is the language backbone — how a modern decoder-only LLM differs
from the original Transformer. cv is the vision tower — pixels to patch tokens.
They are independent bodies of work; the point of this repo is the seam where they
join: cv/vl splices vision tokens into the backbone and cv/mrope gives them
multimodal positions. That join is the VLM. Each directory builds one mechanism in
isolation.
How a current decoder-only LLM differs from the original Transformer.
| dir | mechanism | paper |
|---|---|---|
rope |
rotary position embedding | 2104.09864 |
rmsnorm |
RMS normalization | 1910.07467 |
swiglu |
gated feed-forward network | 2002.05202 |
gqa |
grouped-query attention | 2305.13245 |
nano |
the above assembled into a small Llama | 2302.13971 |
kvcache |
KV-cached decoding and sampling | 1911.02150 |
moe |
sparse mixture-of-experts | 1701.06538 |
bpe |
byte-pair encoding tokenizer | 1508.07909 |
mla |
multi-head latent attention | 2405.04434 |
localattn |
sliding-window + local/global attention | 2310.06825 |
specdec |
speculative decoding | 2211.17192 |
From pixels to tokens, then the 2024–25 question: how to spend the visual-token budget.
| dir | mechanism | paper |
|---|---|---|
vit |
image as a sequence of patches | 2010.11929 |
mae |
masked-autoencoder pretraining | 2111.06377 |
videomae |
masked autoencoding for video | 2203.12602 |
clip |
contrastive image-text pretraining | 2103.00020 |
vl |
vision tokens spliced into an LLM | 2304.08485 |
tokbudget |
the visual-token cost ledger | 2403.06764 |
framesample |
temporal budget before the encoder | 2502.21271 |
tokmerge |
spatial token merging | 2404.16821 |
mrope |
multimodal rotary position ids | 2409.12191 |
siglip |
sigmoid contrastive loss | 2303.15343 |
Each directory documents one mechanism (what it is, how it differs from the
baseline it replaces, its tensor-shape contract) as the spec for a single-file
PyTorch implementation. No pretrained weights, no large datasets. docs/ holds a
generated mastery map of these components — run
.venv/bin/python docs/build.py and open docs/index.html.
Part of the glass* line of from-scratch reproductions:
glassbox (a VLM's internals) · glassalign (post-training) · glassloop (the agent loop).