gpu-bench: Cross-Stack GPU Inference Benchmark (CUDA/TensorRT vs ROCm)

Most TensorRT benchmarks conflate two separate effects — graph compilation and lower precision — into one before/after number. I wanted to isolate them: a three-rung ladder (stock PyTorch → TensorRT FP32 → TensorRT FP16) that holds precision constant across the first two rungs and compilation constant across the last two, so each rung's gain traces back to one specific cause. This is Phase 1 on an RTX 3080; Phase 2 repeats the same harness on AMD's MI300X to see how much of it holds off NVIDIA.

Phase 1 — CUDA/TensorRT ladder (complete)

Hardware: RTX 3080 (10GB), WSL2 Ubuntu, PyTorch 2.13 + CUDA 12.6, TensorRT 11.1 Model: YOLO11n (pretrained COCO), 640x640 input

Three-rung ladder isolating compilation gains from precision gains:

Stack	Batch 1	Batch 4	Batch 8
PyTorch CUDA	8.6 ms / ~117 fps	10.1 ms / 398 fps	13.0 ms / 616 fps
TensorRT FP32	~4.7 ms / ~220 fps	~6.6 ms / ~600 fps	~10.8 ms / ~740 fps
TensorRT FP16	~4.4 ms / ~230 fps	~5.6 ms / ~715 fps	~8.0 ms / ~1000 fps

Key findings:

• Compilation (TensorRT graph optimization, precision held at FP32) gains most at batch 1 (~45%), shrinking to ~19% at batch 8 — small batches are launch-overhead-bound, and fusion attacks exactly that fixed cost.

• FP16 gains ~0 at batch 1 (inside measured noise floor) but ~25% at batch 8 — precision only pays where bytes are the bottleneck (bandwidth-bound regime).

• The "FP16" engine is actually mixed-precision: 94.3% of nodes converted; detection-head coordinate/grid math (values up to 8400, exceeding the autocast safe range of ±512) retained in FP32.

• Measured run-to-run noise floor of ~±0.5 ms at batch 1 on shared desktop hardware; p95 tail dominated by desktop GPU contention (spikes scattered through runs, not startup-clustered).

• Reported memory (pytorch_mem_mb in results.csv) comes from torch.cuda.max_memory_allocated(), which only tracks PyTorch's own allocator — it's blind to TensorRT's internal allocations, so the tensorrt-fp32/fp16 rows undercount actual GPU footprint. Only comparable rung-to-rung within TensorRT, not against the PyTorch baseline.

Methodology

• 50 warmup runs discarded (lazy init, cuDNN autotuning, allocator warmup, clock ramp)
• 100 timed runs per config; torch.cuda.synchronize() inside the timing loop (GPU calls are async — without it you time the handoff, not the work)
• Median + p95 reported; raw pre-sort latencies logged (arrival order distinguishes warmup residue from ambient contention)
• Replicate runs per rung; deltas judged against measured noise floor
• Dynamic-shape TensorRT engine (batch 1-8) accepted a small specialization penalty to keep the config matrix consistent across rungs

Phase 2 — ROCm on MI300X (in progress)

Porting the identical harness to PyTorch-ROCm on AMD Developer Cloud / MI300X.

The AMD-side ladder runs PyTorch-ROCm baseline → ONNX Runtime (MIGraphX EP) → raw MIGraphX. MIGraphX is the headline comparison: it's AMD's first-party graph-level optimizer, the closest architectural match to what TensorRT does on the CUDA side. The ONNX Runtime rung comes first for two reasons: it reuses the exact .onnx artifact from the CUDA export (same model file on both stacks — the cleanest possible comparison), and it banks a complete cross-stack table before the higher-risk raw-MIGraphX work begins. The EP-vs-raw gap is itself a measurement: how much does the convenience wrapper cost?

Files

• bench.py — the harness (swap MODEL_NAME / RUN_LABEL per rung)
• results.csv — all runs, appended
• setup_rocm.sh — Phase 2 environment bootstrap for the MI300X box (ROCm sanity check, venv, PyTorch-ROCm wheel, ultralytics + onnxruntime-rocm)