This notebook accompanys the YouTube video The world's smallest transformer
In here, we train a small transformer with a 4-2-4 architecture (no bias and no activation functions, although with tanh, and even with ReLU, it would give very similar results. The dataset consists of two sentences and four words, but we have experiments with slightly larger datasets as well. The plotting function plots the embedding that results from the middle layer of the neural network, and the projections given by the output layer. The training run is animated through the epochs.
| tiny_llm.ipynb | Tiny LLM | 