Title
[Issue]: KFD runlist oversubscription at exactly 8 CP queue slots causes sclk pin + preemption hangs on RDNA2 (gfx1030) under 3+ concurrent ROCm processes — root cause + fix (amdgpu.num_kcq)
Environment
- GPU: AMD Radeon PRO V620 (Navi 21 / gfx1030)
- OS: Ubuntu 24.04.4 LTS
- Kernel: 6.8.0-137-generic
- ROCm: 7.2.4
- Workload: 2-3 concurrent llama.cpp server processes (HIP backend), each an independent long-lived process holding its own HSA queues — not a single multi-stream process.
Symptom
Under sustained 3-process concurrency on one card:
- GPU core clock (sclk) pins at maximum continuously, even when all three processes are otherwise idle between requests (~42 W idle-but-pinned vs a true idle floor well under that).
- Reported GPU busy% reads ~99% even when no process is actively decoding — the busy metric does not reflect real work.
- Under load, threads intermittently hang stuck in
ioctl on /dev/kfd inside libhsa-runtime64, requiring a process kill to recover.
Root cause (traced against kernel 6.8 amdgpu/amdkfd source, verified against sysfs on this box)
Navi 21 exposes 16 HQD compute-queue slots on MEC0. The amdgpu kernel driver reserves amdgpu.num_kcq of those as kernel compute rings — default 8 — leaving exactly 8 CP queue slots for all ROCm userspace processes on the card combined.
Each concurrent HIP process consumes multiple CP queues (in our testing, ~3 per process: 2 HIP-visible + 1 ROCr-internal queue that isn't covered by GPU_MAX_HW_QUEUES). At 3 concurrent processes this reaches or exceeds the 8-queue budget. KFD's runlist-size calculation (pm_calc_rlib_size()) detects the oversubscription and emits a chained runlist instead of a flat one; the MEC hardware scheduler then round-robins the queue sets on a fixed timer quantum indefinitely. This means the CP front-end never truly idles (the sclk pin, and the misleading busy% reading), and the constant evict/preempt cycling this produces exercises gfx10's known-fragile preemption path — which is where we see the hangs.
This budget is considerably tighter than AMD's own Instinct-class oversubscription guidance implies (that guidance describes 24 queues; gfx9 has 8 queues/pipe where RDNA2 has 4).
Fix, measured
- Partial mitigation:
GPU_MAX_HW_QUEUES=2 per process. Caps the HIP-visible queue count per process (AMD's own workload-tuning guidance already recommends =2 for single-stream llama.cpp-style workloads), no throughput cost measured for a single compute stream. This raises the ceiling from ~daily failures to only failing under full 3-process co-residency, because the ROCr-internal queue isn't covered by this cap (still ~3 queues/process floor).
- Full fix:
amdgpu.num_kcq=2 kernel module parameter, raising the userspace CP queue budget from 8 to 14. With this + the GPU_MAX_HW_QUEUES=2 cap, 3 fully concurrent processes measured at 9/14 queues used, zero oversubscription lines in dmesg, and — notably — the card reached a genuine 3-process idle state for the first time (sclk 0, ~7 W, 0% busy), confirming the busy%/clock-pin symptom above was entirely the oversubscription artifact, not real GPU load.
Question for maintainers
Is the num_kcq=8 default (reserving half of Navi 21's 16 HQD slots for kernel rings) intended to leave RDNA2 users this little multi-process headroom, or is this closer to a bringup-era default that hasn't been revisited for the "several independent long-lived ROCm processes sharing one consumer/workstation card" use case? Related reports we found describe the same symptom class on RDNA3/RDNA4 (pinned clocks, hangs that don't reproduce on Vulkan) but none isolated this specific mechanism:
Happy to provide the full sysfs dump, dmesg oversubscription lines, or test additional configurations if useful.
Title
[Issue]: KFD runlist oversubscription at exactly 8 CP queue slots causes sclk pin + preemption hangs on RDNA2 (gfx1030) under 3+ concurrent ROCm processes — root cause + fix (
amdgpu.num_kcq)Environment
Symptom
Under sustained 3-process concurrency on one card:
ioctlon/dev/kfdinsidelibhsa-runtime64, requiring a process kill to recover.Root cause (traced against kernel 6.8 amdgpu/amdkfd source, verified against sysfs on this box)
Navi 21 exposes 16 HQD compute-queue slots on MEC0. The amdgpu kernel driver reserves
amdgpu.num_kcqof those as kernel compute rings — default 8 — leaving exactly 8 CP queue slots for all ROCm userspace processes on the card combined.Each concurrent HIP process consumes multiple CP queues (in our testing, ~3 per process: 2 HIP-visible + 1 ROCr-internal queue that isn't covered by
GPU_MAX_HW_QUEUES). At 3 concurrent processes this reaches or exceeds the 8-queue budget. KFD's runlist-size calculation (pm_calc_rlib_size()) detects the oversubscription and emits a chained runlist instead of a flat one; the MEC hardware scheduler then round-robins the queue sets on a fixed timer quantum indefinitely. This means the CP front-end never truly idles (the sclk pin, and the misleading busy% reading), and the constant evict/preempt cycling this produces exercises gfx10's known-fragile preemption path — which is where we see the hangs.This budget is considerably tighter than AMD's own Instinct-class oversubscription guidance implies (that guidance describes 24 queues; gfx9 has 8 queues/pipe where RDNA2 has 4).
Fix, measured
GPU_MAX_HW_QUEUES=2per process. Caps the HIP-visible queue count per process (AMD's own workload-tuning guidance already recommends=2for single-stream llama.cpp-style workloads), no throughput cost measured for a single compute stream. This raises the ceiling from ~daily failures to only failing under full 3-process co-residency, because the ROCr-internal queue isn't covered by this cap (still ~3 queues/process floor).amdgpu.num_kcq=2kernel module parameter, raising the userspace CP queue budget from 8 to 14. With this + theGPU_MAX_HW_QUEUES=2cap, 3 fully concurrent processes measured at 9/14 queues used, zero oversubscription lines indmesg, and — notably — the card reached a genuine 3-process idle state for the first time (sclk 0, ~7 W, 0% busy), confirming the busy%/clock-pin symptom above was entirely the oversubscription artifact, not real GPU load.Question for maintainers
Is the
num_kcq=8default (reserving half of Navi 21's 16 HQD slots for kernel rings) intended to leave RDNA2 users this little multi-process headroom, or is this closer to a bringup-era default that hasn't been revisited for the "several independent long-lived ROCm processes sharing one consumer/workstation card" use case? Related reports we found describe the same symptom class on RDNA3/RDNA4 (pinned clocks, hangs that don't reproduce on Vulkan) but none isolated this specific mechanism:sched_policy=1rather than the queue budget)Happy to provide the full sysfs dump,
dmesgoversubscription lines, or test additional configurations if useful.