gpt-from-scratch

This repository contains the collection of explorative notebooks pure in python and in the language that we, humans can read. Have tried to compile all lectures from the Andrej Karpathy's 💎 playlist on Neural Networks - which we will end up with building GPT.

Inspired by Andrej Karpathy’s "Let’s Build GPT", this project guides you step‑by‑step to build a GPT from scratch, demystifying its architecture through clear, hands‑on code.

This project implements GPT-1 using PyTorch, focusing on foundational transformer architectures for natural language processing tasks.

Comprehensive fresh start for how LLMs work

MicroGPT is a clean, educational implementation of the GPT (Generative Pre-trained Transformer) architecture built from first principles with detailed explanations and comprehensive testing.

HackingGPT part 7 where we learn the foundations to create a custom GPT from absolute scratch.

HackingGPT part 1 where we learn the foundations to create a custom GPT from absolute scratch.

Build and train a GPT-style language model from scratch with Python and PyTorch, covering neural network foundations, BPE tokenization, attention, Transformers, and text generation.

This notebook builds a complete GPT (Generative Pre-trained Transformer) model from scratch using PyTorch. It covers tokenization, self-attention, multi-head attention, transformer blocks, and text generation and all explained step-by-step with a simple nursery rhyme corpus.

SeedGPT is a minimal decoder-only LLM from scratch, inspired by andrej karpathy's tutorials, generating text token-by-token based on input word prompts.

HackingGPT part 2 where we learn the foundations to create a custom GPT from absolute scratch.

Complete GPT implementation from scratch with BPE tokenizer, transformer architecture, and trained models on Shakespeare and WikiText. Achieves perplexity 7.84 on WikiText-2 with 12.6M parameters.

Annotated study fork of Karpathy nanoGPT — GPT-2 training from scratch with extended notes on causal self-attention, positional encoding, layer norm placement, and efficient fine-tuning on custom datasets.

HackingGPT part 3 where we learn the foundations to create a custom GPT from absolute scratch.

Ask the Bhagavad Gita a question in your own words and get the shlokas that answer it — all 18 chapters, 700 verses, verbatim in Sanskrit, Hindi and English. Speaker-aware dense retrieval (bge-m3), crisis triage before the model, and a char-GPT built from scratch.

HackingGPT part 4 where we learn the foundations to create a custom GPT from absolute scratch.

A hands-on learning project for building and training a small GPT-style language model from scratch in PyTorch. The goal is to understand transformers from the inside out: attention, residual blocks, training loops, and what the model is actually doing at each layer.

HackingGPT part 5 where we learn the foundations to create a custom GPT from absolute scratch.

HackingGPT part 6 where we learn the foundations to create a custom GPT from absolute scratch.