Audio Classification with Mel Spectrograms & CNN

Classifies audio from the MUSAN dataset into music / noise / speech using a CNN trained on Mel spectrograms — reaching 98 % validation accuracy.

---

What this project does

Three steps from raw audio to a working classifier:

Step 1 — Feature extraction  .wav → Mel spectrogram
Each audio file is sliced into 3-second chunks. An STFT computes a 2D time × frequency representation, a Mel filterbank compresses it to 128 perceptually-spaced bands, and power-to-dB scaling gives a compact (128 × 94) float32 image per chunk.

Step 2 — Training a CNN
A 4-block CNN (MelCNN) treats each Mel spectrogram as a single-channel image and learns to classify it. Weighted cross-entropy handles class imbalance; ReduceLROnPlateau adapts the learning rate automatically.

Step 3 — Evaluation (three modes)

Script	What it does
evaluation/eval_model.py	Held-out evaluation → confusion matrix + per-class confidence analysis
evaluation/inspect_model.py	Interactive layer-by-layer viewer: watch each ConvBlock transform the input
evaluation/live_classify.py	Real-time microphone classification — new prediction every second

---

Pipeline

musan.tar.gz
     │
     ▼
training/prepare_musan.py   →   musan/  +  musan_index.json
     │
     ▼
training/build_dataset.py   →   training_data/
     │
     ▼
training/train.py           →   best_model.pt  +  plots/training_curves.png
     │
     ▼
evaluation/eval_model.py    →   plots/confusion_matrix.png
evaluation/inspect_model.py →   interactive layer viewer
evaluation/live_classify.py →   real-time mic classification

---

How to get MUSAN

1. Download the archive (~10 GB) from the official mirror:

   https://www.openslr.org/17/
   

2. Place musan.tar.gz in the project root (next to training/prepare_musan.py).
3. Run training/prepare_musan.py once — it extracts the archive and creates musan_index.json.

The dataset folder and the archive are excluded from version control via .gitignore.

---

Scripts

1. training/prepare_musan.py

Run once. Extracts the MUSAN archive and builds a JSON index of all .wav files.

python training/prepare_musan.py

Prerequisite: musan.tar.gz must be present in the project folder.
Output:

• musan/ — extracted dataset directory
• musan_index.json — list of all .wav file paths

---

2. training/build_dataset.py

Converts the entire MUSAN dataset into Mel spectrograms and saves them as NumPy arrays.

python training/build_dataset.py

What it does:

• Each audio file is split into 3-second chunks (no overlap between chunks; any remainder shorter than 3 s is discarded)
• For each chunk a Mel spectrogram is computed (STFT with 50 % window overlap, Mel filterbank, dB scaling)
• Each spectrogram is saved as a float32 array of shape (128 × 94)
• The class label is derived automatically from the file path

Parameters:

Parameter	Value	Description
SAMPLE_RATE	16 000 Hz	Audio sample rate
CHUNK_SEC	3.0 s	Length of one training example
N_MELS	128	Number of Mel frequency bands
N_FFT	1024	FFT window length
HOP_LENGTH	512	Step size (= 50 % overlap)
FMAX	8 000 Hz	Upper frequency cutoff

Output — training_data/:

training_data/
  music/     music_0000000.npy  …
  noise/     noise_0000000.npy  …
  speech/    speech_0000000.npy …
  metadata.csv

metadata.csv contains for each .npy file: path, class name, label ID.

Class	Label ID
music	0
noise	1
speech	2

Note on train/test split: Chunks from the same source file should always be assigned to the same partition (train or test) to avoid data leakage.

---

3. training/train.py

Trains a CNN classifier (MelCNN) on the generated .npy dataset.

python training/train.py

Architecture — MelCNN:

Input  (1 × 128 × 94)
  │
  ├─ ConvBlock 1: Conv2d → BatchNorm → ReLU → MaxPool  →  (32 × 64 × 47)
  ├─ ConvBlock 2:                                        →  (64 × 32 × 23)
  ├─ ConvBlock 3:                                        →  (128 × 16 × 11)
  ├─ ConvBlock 4:                                        →  (256 × 8 × 5)
  │
  ├─ AdaptiveAvgPool2d(1,1)                             →  (256 × 1 × 1)
  └─ FC: 256 → 128 → 3 (with Dropout 0.5)

Training parameters:

Parameter	Value	Description
EPOCHS	50	Number of full passes through the training data
BATCH_SIZE	64	Samples per gradient update
LR	1e-3	Initial learning rate (Adam)
VAL_SPLIT	0.15	Fraction of data held out for validation

Class imbalance is handled via weighted cross-entropy loss (inverse-frequency weights). The learning rate is halved automatically if validation loss does not improve for 3 epochs (ReduceLROnPlateau).

Output:

• best_model.pt — model weights with lowest validation loss
• training_curves.png — Loss and Accuracy plots over all epochs

---

4. evaluation/eval_model.py

Evaluates the trained model on files that were held out during training.

python evaluation/eval_model.py

What it does:

• Loads source files not seen during training (reads train_val_split.json)
• For each file: splits into 3-second segments, runs inference, takes the majority-vote prediction
• Prints per-class accuracy and average softmax confidence (correct vs. wrong predictions)

Output — plots/confusion_matrix.png:

---

5. evaluation/inspect_model.py

Interactive layer-by-layer visualisation of the trained CNN.

python evaluation/inspect_model.py

What it does:
Loads a set of validation samples, runs a forward pass with activation hooks, and opens an interactive matplotlib window. Slide through layers and samples to watch each ConvBlock transform the input spectrogram into progressively abstract feature maps.

Controls:

Key	Action
← / →	Switch layer (Input Mel → ConvBlock 1 → … → ConvBlock 4)
↑ / ↓	Switch sample
Sliders	Same as arrow keys, mouse-controlled

---

6. evaluation/live_classify.py

Classifies microphone input in real time.

python evaluation/live_classify.py

Stop with Ctrl+C.

How it works:

• A ring buffer continuously captures audio from the default microphone at 16 kHz
• Every second, the most recent 3-second window is classified by MelCNN
• Softmax probabilities are smoothed over the last 3 predictions and rendered as ASCII bars:

Live Classification  (Ctrl+C to quit)
──────────────────────────────────────────
  music    [████████████░░░░░░░░] 62% ◄
  noise    [░░░░░░░░░░░░░░░░░░░░]  3%
  speech   [██████░░░░░░░░░░░░░░] 35%

---

Background: Key Concepts

STFT — Short-Time Fourier Transform

A regular Fourier Transform analyses the entire signal at once and tells you which frequencies are present — but not when. The STFT solves this by cutting the signal into short overlapping windows and computing a separate FFT for each one, producing a 2D time × frequency representation.

50 % Window Overlap

Each window is N_FFT = 1024 samples long, but the next window starts only HOP_LENGTH = 512 samples later — so consecutive windows overlap by half. This is necessary because a windowing function (e.g. Hann window) tapers the signal to zero at the edges; without overlap, information at those edges would be lost.

Mel Filterbank

The FFT produces frequency bins on a linear scale. Human hearing, however, is logarithmic — we perceive fine differences at low frequencies but not at high ones. The Mel scale models this:

$$m = 2595 \cdot \log_{10}!\left(1 + \frac{f}{700}\right)$$

A Mel filterbank is a set of triangular bandpass filters spaced evenly on this scale. With N_MELS = 128, the ~513 FFT bins are compressed to 128 values in a perceptually meaningful way — more compact and more relevant for the CNN than raw FFT bins.

Processing chain

Signal  →  [Windowing + STFT]  →  linear spectrogram
                                          │
                                   [Mel filterbank]
                                          │
                                   Mel spectrogram  (128 bands)
                                          │
                                   [power_to_db]
                                          │
                                   dB Mel spectrogram  →  .npy

---

Dependencies

pip install librosa numpy tqdm matplotlib
pip install torch  # CPU-only
# On a CUDA cluster, match the PyTorch build to the driver version:
# conda install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia

---

Results

The model reaches 97 % validation accuracy after 50 epochs on the MUSAN dataset.

Training curves