This project implements a fully connected feedforward neural network from first principles in Python, without using machine learning frameworks for the network itself. It uses the MNIST dataset to recognise handwritten digits.
I built this as a learning tool to understand matrix-based gradient descent from scratch. The first version was written in 2023 completely blind, using only 3Blue1Brown's maths tutorial videos on neural networks and gradient descent as guidance, especially Gradient descent, how neural networks learn.
- Matrix-based forward propagation and backpropagation
- Gradient descent weight updates with training support
- Custom sigmoid activation and cost functions
- Configurable network architecture and hyperparameters
- Model saving/loading with JSON
The network supports NumPy for CPU execution and CuPy for CUDA GPU execution. Install the GPU requirements only when using the CuPy backend.
Install a CuPy wheel matching the installed CUDA runtime. For current CUDA 12 systems:
pip install -r requirements-gpu.txtRun an individual GPU training or test job with --backend cupy:
python src/main.py --train --backend cupy --model models/fresh.json --epochs 5 --batch-size 256Benchmark CPU and GPU backends with the same model and settings:
python src/main.py --benchmark-batches --backends numpy,cupy --epochs 5 --learning-rate 0.1 --batch-sizes 256 --benchmark-output outputs/numpy_vs_cupy_full_summary.csv --benchmark-history-output outputs/numpy_vs_cupy_full_history.csv --benchmark-plot docs/assets/numpy_vs_cupy_full_stats.pngCreate a configurable, seeded model:
python src/randomiser.py models/medium.json --structure 784,512,512,10 --seed 42
python src/main.py --train --model models/medium.json --structure 784,512,512,10 --backend cupy --epochs 5 --batch-size 256src/main.py- Command-line entry point for training, testing, and benchmarkingsrc/dependencies.py- Core neural network layers and matrix operationssrc/data.py- Model validation, model JSON loading/saving, and MNIST loadingsrc/training.py- Shared training, evaluation, batching, and metric plotting helperssrc/benchmark.py- Batch-size benchmark recording and graph generationsrc/randomiser.py- Random weight initialisation moduledata.json- Example saved weightsrequirements.txt- Python package dependencies
Copy and paste this into PowerShell from the project folder:
# Create and activate a local virtual environment
python -m venv .venv
.venv\Scripts\activate
# Install Python dependencies
pip install -r requirements.txt
# Create, train, and test a fresh random model
python src/randomiser.py models/fresh.json
python src/main.py --train --model models/fresh.json --epochs 1 --learning-rate 0.1
python src/main.py --test --model models/fresh.jsonTo test the included saved model only:
python src/main.py --testTrain or test a saved model file. By default, this uses data.json.
Training saves updated weights and biases back to the selected model file.
python src/main.py --test
python src/main.py --trainTraining defaults to 30 epochs and a learning rate of 0.1:
python src/main.py --train --epochs 10 --learning-rate 0.05Useful flags:
--test- Evaluate a model against the MNIST test set--train- Train a model and save updated weights/biases--benchmark-batches- Compare training cost, timing, throughput, and test accuracy across batch sizes--backend BACKEND- Usenumpy(CPU, default) orcupy(CUDA GPU) for training or testing--backends LIST- Comma-separated backends for--benchmark-batches, for examplenumpy,cupy--structure LAYERS- Comma-separated MNIST layer sizes, for example784,512,512,10--model PATH- Load/save a specific model file--epochs N- Number of training epochs--learning-rate VALUE- Training learning rate--batch-size N- Number of examples per training update--batch-sizes LIST- Comma-separated batch sizes for benchmarking, for example1,8,32,128--cpu-batch-sizes LIST- Optional NumPy-only batch-size sweep for best-tested CPU throughput--gpu-batch-sizes LIST- Optional CuPy-only batch-size sweep for best-tested GPU throughput--benchmark-train-limit N- Limit benchmark training examples for quicker comparisons--benchmark-test-limit N- Limit benchmark test examples for quicker comparisons--benchmark-runs N- Recorded runs per backend and batch size; defaults to 3--benchmark-warmup-batches N- Unrecorded CuPy warm-up batches before each run; defaults to 1--benchmark-output PATH- Save benchmark summary stats as CSV--benchmark-history-output PATH- Save per-epoch benchmark stats as CSV--benchmark-plot PATH- Save benchmark comparison graphs--verbose- Print model details and every test prediction--show-tf-logs- Show TensorFlow startup logs
Train with mini-batches:
python src/main.py --train --model models/fresh.json --epochs 5 --learning-rate 0.1 --batch-size 32Benchmark different batch sizes from the same starting model:
python src/main.py --benchmark-batches --model models/fresh.json --epochs 1 --batch-sizes 1,8,16,32,64,128By default, benchmark results are saved to:
outputs/batch_benchmark_summary.csvoutputs/batch_benchmark_history.csvoutputs/batch_benchmark_stats.png
For a quick benchmark while experimenting, limit the dataset:
python src/main.py --benchmark-batches --model models/fresh.json --epochs 1 --batch-sizes 1,16,64,256 --benchmark-train-limit 5000 --benchmark-test-limit 1000Create a new random model file:
python src/randomiser.py models/fresh.jsonExisting model files are protected by default. To overwrite one, pass --force and confirm the prompt:
python src/randomiser.py models/fresh.json --forceThe scripts download MNIST through TensorFlow/Keras if needed. TensorFlow startup logs are hidden by default during main.py runs.
Run the regression suite, including CPU/GPU output parity when CuPy is installed:
python -m unittest discover -s tests -v