Project 7 — Static Malware Behaviour Triage with Evasive-Sample Evaluation

Course: Applied Machine Learning · Team ML_SE_ALL_12 Members: Tarik Bilgin Demirci (lead), Berk Kahraman, Yasar Yavuz Akbas, Umut Türklay

What this project is

Build static-feature malware triage models (malicious vs. benign, plus malware family) from the EMBER2024 dataset, and — crucially — evaluate them the way a security team would:

• Leakage-free temporal split (train on past weeks, test on newer weeks).
• Class imbalance handling for rare families.
• Calibration of scores (reliability, ECE, Brier) so thresholds are meaningful.
• Evasive-sample evaluation on the EMBER2024 challenge set (6,315 files that evaded ~70 AV engines) — the heart of the project.

We do not run any malware. EMBER2024 ships pre-extracted static features only (no executable binaries).

Dataset — EMBER2024

• Source: https://github.com/FutureComputing4AI/EMBER2024 (package: thrember)
• 3.2M files across PE (Win32/Win64/.NET), APK, PDF, ELF.
• Labels: benign/malicious + family + behaviour (+ packer/exploit/group).
• Built-in temporal split: 52 train weeks → 12 newer test weeks; plus a challenge (evasive) split.
• Scope (M4 Max, local): develop on the small .NET PE subset, scale to Win32 for final numbers, always evaluate on the challenge set.

The dataset is multi-GB and is never committed — each teammate downloads it locally into data/ (see Setup).

Setup

Requires Python ≥ 3.10 (we use 3.12).

The ML code and notebook are identical on every OS (pure Python + relative paths). Only the environment setup differs.

macOS (Apple Silicon)

LightGBM needs the OpenMP runtime:

brew install libomp
python3.12 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Windows (PowerShell)

Install Python 3.12 first (python.org installer, or winget install Python.Python.3.12):

py -3.12 -m venv .venv
.venv\Scripts\Activate.ps1
pip install -r requirements.txt

No libomp needed — LightGBM's Windows wheel bundles the OpenMP runtime.

Note: signify is pinned to 0.8.1 — version 0.9.x renames a class (SignedPEFile) that thrember still imports, which breaks installation.

Get the data

Easiest — just open the notebook and Run All. The first cell calls data.ensure_dataset_ready(), which auto-downloads the EMBER2024 dev bundle (~5GB, from HuggingFace) and vectorises it. On later runs it returns instantly.

Or do it manually from the CLI (into the git-ignored data/ folder):

# Small dev bundle: .NET (Dot_Net) train+test + challenge set
python -m scripts.download_data --dev

# Final-scale Win32 data (large) when ready
python -m scripts.download_data --file-type Win32 --split train
python -m scripts.download_data --file-type Win32 --split test

Pipeline (reproduce from a clean clone)

Open notebooks/malware_triage.ipynb and Run All — it bootstraps the data (download + vectorize) and trains every model on first run. Equivalent CLI steps:

python -m scripts.download_data --dev          # .NET train+test + challenge (~5GB)
python -m scripts.vectorize --label-type label # -> X_*.dat / y_*.dat (dim 2568)
python -m scripts.train_baseline               # LightGBM + LogReg, temporal split
python -m scripts.calibrate                    # Platt/Isotonic, operating threshold
python -m scripts.train_family                 # top-20 family multiclass
python -m scripts.evaluate_evasive             # challenge-set detection + error analysis

Repo layout

.
├── data/         # EMBER2024 data — git-ignored, downloaded locally
├── models/       # trained models — git-ignored
├── results/      # figures (results/figures/) & metric JSONs (committed)
├── notebooks/    # malware_triage.ipynb — main presentation notebook
├── scripts/      # download / vectorize / train / calibrate / evaluate
├── src/          # config + reusable code (data, modeling, calibration, family, eda)
└── requirements.txt

Results summary (.NET subset, temporally-newer test)

Task	Metric	Result
Binary triage	ROC-AUC	0.998
Binary triage	TPR @ FPR=1%	96.3%
Calibration	ECE (isotonic)	0.003
Evasive (challenge)	TPR @ FPR=0.1%	47.6% (vs 87.5% normal)
Family (top-20)	macro-F1	0.56

Headline: strong binary triage, but at a strict 0.1% FPR the detection rate halves on evasive samples — quantifying exactly where a static triage model breaks.

Reproducibility

• All paths are relative to the repo root (src/config.py) — works on any machine after clone + data download. No absolute/hard-coded paths.
• Random seed fixed in src/config.py (RANDOM_SEED = 42).