Build an LLM from Scratch

A from-scratch implementation of a small language model, built part by part, from a text file to a system capable of generation, classification, instruction following, parameter-efficient adaptation, and reasoning-style preference training.

This repository is the companion codebase for an eight-part series. The capstone essay — docs/article.md — is the reflective piece on what building the system teaches that calling an API never will.

Repository layout

Build_LLM_from_Scratch/
├── README.md                  # this file
├── demo.py                    # end-to-end demo: runs all 8 parts in sequence
├── docs/
│   └── article.md            # capstone essay (1,538 words)
├── llm_from_scratch/
│   ├── __init__.py           # public API
│   ├── torch_impl.py         # all implementations (tokenizer, attention, transformer, LoRA)
│   ├── part1_tokenizer.py    # BPE tokenization
│   ├── part2_embeddings.py  # token + positional embeddings
│   ├── part3_attention.py    # (numpy reference — preserved for study)
│   ├── part4_transformer.py  # (numpy reference — preserved for study)
│   ├── part5_pretrain.py    # pretraining: next-token CE, sliding-window loader, generation
│   ├── part6_sft.py         # instruction fine-tuning
│   ├── part7_lora.py         # LoRA adaptation
│   └── part8_dpo_grpo.py    # DPO and GRPO preference training
└── tests/
    └── test_components.py   # unit tests for all components

Blog series

#	Blog Post	Link
1	Build Tokenizer First	Build an LLM From Scratch [1]
2	Learn Self-Attention like a 5-Year-Old	Build an LLM From Scratch [2]
3	Built a GPT Architecture From Scratch	Build an LLM From Scratch [3]
4	Pretraining an LLM on a Laptop	Build an LLM From Scratch [4]
5	Fine-Tuning LLM Into a Spam Classifier	Build an LLM From Scratch [5]
6	From Next-Token Prediction to ChatGPT	Build an LLM From Scratch [6]
7	I Fine-Tuned an LLM using LoRA From Scratch	Build an LLM From Scratch [7]
8	I Built a Reasoning Model From Scratch	Build an LLM From Scratch [8]

The eight-part journey

Part	Topic	Output
1	Tokenization	A text file becomes integers (BPE)
2	Embeddings	Token + positional lookup tables
3	Attention	Learned weighted sum over the past (causal, multi-head, GQA, KV cache)
4	Transformer block	Residual stream + LayerNorm + MLP, stacked
5	Pretraining	Next-token cross-entropy on a tiny corpus
6	Instruction fine-tuning (SFT)	A model that follows directives
7	LoRA	Parameter-efficient adapter training
8	DPO / GRPO	Reasoning-style preference training

The abstraction stack, top to bottom

The capstone essay walks the reader backward from an API to the integers a model actually sees:

generate()
 ↓
sampling
 ↓
logits
 ↓
transformer blocks
 ↓
attention
 ↓
embeddings
 ↓
token IDs
 ↓
text

Each arrow is a place where an API hides a decision. Building the stack by hand makes every decision visible.

The five deeper realizations

The essay condenses the eight parts into five mental-model shifts:

1. Tokenization is part of model design. BPE vocabulary size, byte coverage, and special tokens all leak into behavior.
2. Attention is routing, not magic. A learned weighted average over past hidden states, nothing more.
3. Generation is a systems problem as well as an ML problem. Memory bandwidth and autoregressive serial dependency dominate inference time.
4. Fine-tuning changes behavior through objectives and data, not "knowledge injection." LoRA, DPO, and GRPO are all just gradient descent on different loss surfaces.
5. Modern techniques become easier to understand once the base mechanism is visible. GQA, RoPE, LoRA, DPO, GRPO — each solves a specific, visible bottleneck.

Running

# Run all 8 parts end-to-end
python3 demo.py

# Run individual parts
python3 -m llm_from_scratch.part5_pretrain --epochs 5 --batch-size 8
python3 -m llm_from_scratch.part6_sft --epochs 3
python3 -m llm_from_scratch.part7_lora --rank 4 --epochs 3
python3 -m llm_from_scratch.part8_dpo_grpo --method grpo --epochs 2

# Run tests
python3 tests/test_components.py

Requirements

• Python 3.9+
• PyTorch 2.8+
• NumPy 1.26+

What this project is — and is not

It is:

• A teaching artifact. Every layer is small enough to read.
• A faithful reimplementation of the core mechanisms in modern LLMs using PyTorch.
• A reference for the decisions an engineer makes when there is no API to hide behind.

It is not:

• A production model. Outputs are plausible on a narrow corpus and are intended for learning, not deployment.
• A substitute for industrial-scale training infrastructure. Frontier models require distributed data parallelism, tensor parallelism, mixed-precision scheduling, and curated data mixtures at a scale this project does not attempt.

Reading order

Read the eight parts in order. The capstone essay is meant to be read after the eight parts, not as a substitute for them. It tells one story — from a text file to a trainable, adaptable, instruction-following, reasoning-ready model — that the individual articles do not.