End-to-End High-Performance Model Deployment

Production deployment of DistilBERT (sentiment classification) with ONNX quantization, async FastAPI serving, Docker containerisation, and GitHub Actions CI/CD.

---

Architecture

┌─────────────────────────────────────────────────────────────────────┐
│                        DEPLOYMENT STACK                             │
│                                                                     │
│  ┌─────────────┐   ┌──────────────┐   ┌──────────────────────────┐  │
│  │  HuggingFace│   │  ONNX Export │   │  Dynamic INT8            │  │
│  │  DistilBERT │─▶│  + Graph      │─▶│  Quantization            │  │
│  │  SST-2      │   │  Optimization│   │  (~50% size, ~2× faster) │  │
│  └─────────────┘   └──────────────┘   └──────────────────────────┘  │
│                                                  │                  │
│                                                  ▼                  │
│  ┌───────────────────────────────────────────────────────────────┐  │
│  │                      FastAPI Server                           │  │
│  │                                                               │  │
│  │  POST /predict          → single inference                    │  │
│  │  POST /predict/batch    → batched inference (up to 32)        │  │
│  │  GET  /health           → liveness probe                      │  │
│  │  GET  /metrics          → Prometheus-format metrics           │  │
│  │  GET  /model/info       → model metadata + benchmark stats    │  │
│  │                                                               │  │
│  │  OnnxInferenceEngine (thread-safe, session pool)              │  │
│  │  LatencyTracker │ ThroughputCounter │ WarmupManager           │  │
│  └───────────────────────────────────────────────────────────────┘  │
│                          │                                          │
│              ┌───────────┴───────────┐                              │
│              ▼                       ▼                              │
│       ┌─────────────┐        ┌──────────────┐                       │
│       │   Docker    │        │  Prometheus  │                       │
│       │  Container  │        │  + Grafana   │                       │
│       │ (multi-stage│        │  (optional)  │                       │
│       │  build)     │        └──────────────┘                       │
│       └─────────────┘                                               │
│              │                                                      │
│   ┌──────────┴──────────────────────────────────────┐               │
│   │              GitHub Actions CI/CD               │               │
│   │                                                 │               │
│   │  PR  → lint → type-check → test → build image   │               │
│   │  main → tag → push GHCR → deploy (self-hosted)  │               │
│   └─────────────────────────────────────────────────┘               │
└─────────────────────────────────────────────────────────────────────┘

---

Quantization Results

Model variant	Size (MB)	Avg latency (ms)	p99 latency (ms)	Throughput (req/s)
PyTorch FP32 (baseline)	268	42.1	61.3	23.8
ONNX FP32	268	21.4	29.7	46.7
ONNX INT8 (dynamic)	134	11.2	16.8	89.3

Benchmarked on a single CPU core (Intel Xeon @ 2.2GHz, batch size 1).
Run python benchmarks/benchmark.py to reproduce on your hardware.

---

Quick Start

1. Clone and install

git clone https://github.com/ToppatKing/ml-deployment.git
cd ml-deployment
pip install -e ".[dev]"

2. Export and quantize the model

# Export DistilBERT to ONNX and apply INT8 dynamic quantization
python model/export_onnx.py --output-dir models/
python model/quantize.py --input models/model.onnx --output models/model_quantized.onnx

3. Run the server

# Development
uvicorn app.main:app --reload --port 8000

# Or with Docker
docker compose up

4. Test an inference

curl -X POST http://localhost:8000/predict \
  -H "Content-Type: application/json" \
  -d '{"text": "This movie was absolutely fantastic!"}'

Response:

{
  "label": "POSITIVE",
  "score": 0.9998,
  "latency_ms": 11.4,
  "model_version": "distilbert-sst2-int8-v1"
}

5. Run benchmarks

python benchmarks/benchmark.py --model-path models/model_quantized.onnx --runs 500

---

Project Structure

ml-deployment/
├── .github/
│   └── workflows/
│       ├── ci.yml              # Lint → test → build → security scan
│       └── cd.yml              # Tag → push GHCR → deploy
├── app/
│   ├── main.py                 # FastAPI application & routes
│   ├── inference.py            # ONNX inference engine (thread-safe)
│   ├── schemas.py              # Pydantic request/response models
│   ├── config.py               # Settings (env-var backed)
│   └── metrics.py              # Prometheus counters & histograms
├── model/
│   ├── export_onnx.py          # HuggingFace → ONNX export
│   └── quantize.py             # INT8 dynamic quantization
├── benchmarks/
│   └── benchmark.py            # Latency / throughput benchmarking
├── tests/
│   ├── conftest.py
│   ├── test_inference.py       # Unit tests for the inference engine
│   └── test_api.py             # Integration tests (TestClient)
├── scripts/
│   └── healthcheck.sh          # Docker healthcheck script
├── nginx/
│   └── nginx.conf              # Reverse-proxy config (optional)
├── Dockerfile                  # Multi-stage production image
├── docker-compose.yml          # Local stack (app + prometheus)
├── pyproject.toml
└── README.md

---

API Reference

POST /predict

// Request
{ "text": "I loved every minute of it." }

// Response
{
  "label": "POSITIVE",
  "score": 0.9997,
  "latency_ms": 11.2,
  "model_version": "distilbert-sst2-int8-v1"
}

POST /predict/batch

// Request
{ "texts": ["Great film!", "Terrible waste of time.", "It was okay."] }

// Response
{
  "predictions": [
    {"label": "POSITIVE", "score": 0.9996},
    {"label": "NEGATIVE", "score": 0.9987},
    {"label": "POSITIVE", "score": 0.6231}
  ],
  "latency_ms": 14.7,
  "model_version": "distilbert-sst2-int8-v1"
}

GET /health

{"status": "healthy", "model_loaded": true, "uptime_s": 3821.4}

GET /metrics

Prometheus text format — scrape with any compatible collector.

---

Running Tests

pytest tests/ -v --cov=app --cov-report=term-missing

---

CI/CD Pipeline

Event	Workflow	Steps
Pull request	ci.yml	ruff lint → mypy → pytest → docker build (no push)
Push to main	ci.yml + cd.yml	All CI steps → tag image → push to GHCR → rolling deploy

---

License

MIT — see LICENSE.