llama.cpp CUDA hangs non-deterministically on RTX 5070 Ti (Blackwell/SM120) — same command, works once, hangs on identical rerun

[t-timms]

Summary

llama.cpp (any recent build, any model — not specific to one architecture) hangs non-deterministically under CUDA on WSL2 with an RTX 5070 Ti (Blackwell, SM120). The hang is immune to SIGTERM and requires SIGKILL. The decisive evidence: the same binary run with the identical command, back to back, produced one clean exit and one hard hang. That rules out anything deterministic — a specific commit, CUDA toolkit version, or flag.

Environment

• Windows: 10.0.26200.9168
• WSL: 2.7.11.0 (kernel 6.18.33.2-2, WSLg 1.0.73.2)
• GPU: NVIDIA GeForce RTX 5070 Ti, compute capability 12.0 (Blackwell, SM120)
• Driver: 610.88 (confirmed latest GeForce release as of this report)
• Distro: Ubuntu (WSL2), CUDA toolkit tested at both 13.2 and 12.8
• llama.cpp: reproduced on current master and on pre-regression tag b7376

Symptom

llama-cli (built with -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=120) sometimes prints partial startup diagnostics (GPU detection, memory breakdown) before stalling, sometimes stalls almost immediately after CUDA device detection. Once stalled:

• CPU and GPU utilization both sit near 0%
• The process stays alive indefinitely (observed up to 26+ minutes before being killed)
• timeout's default SIGTERM does not terminate it — kill -9 is required
• The rest of the system remains responsive throughout (nvidia-smi continues to work normally, other CUDA workloads on the same GPU are unaffected)

Reproduction

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=120 -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j 12 --target llama-cli

# any GGUF model reproduces this — confirmed with a completely mainstream one:
# bartowski/Llama-3.2-1B-Instruct-GGUF, Q4_K_M

./build/bin/llama-cli -m Llama-3.2-1B-Instruct-Q4_K_M.gguf \
  -p "The capital of France is" -n 32 -ngl 99 -c 2048 --temp 0

Run this several times in a row. In our testing across ~7 attempts with this and one other model, only one completed cleanly; the rest hung. Sample size is small, but the failure rate is high, not occasional.

What we ruled out (8 independently tested variables)

1. Model-specific cause — reproduced with a completely mainstream, non-experimental model (Llama-3.2-1B-Instruct), not just an exotic architecture. Rules out anything model-specific.
2. CUDA toolkit version — reproduced identically on CUDA 13.2 and a from-scratch CUDA 12.8 rebuild.
3. llama.cpp commit/version — reproduced identically on current master and on tag b7376 (the last confirmed-good SM120 tag referenced in ggml-org/llama.cpp#18090, predating a documented Blackwell regression window).
4. GGML_CUDA_DISABLE_GRAPHS=1 — no change in behavior.
5. GPU driver version — confirmed already on the latest GeForce release (610.88).
6. WSL platform version — confirmed already on 2.7.11.0, past the 2.7.0 release that fixed a separate Blackwell CUDA-graph-capture issue.
7. Fresh WSL boot — reproduced on a freshly booted WSL instance (~4 minutes uptime), ruling out state accumulated from prior runs.
8. Boot-time race / settle delay — reproduced again after waiting 60+ seconds post-boot before the first CUDA call, ruling out a simple GPU-PV initialization race.

Related reports

This may be connected to NVIDIA/open-gpu-kernel-modules#1111 (silent GSP firmware hang on SM120 under sustained llama.cpp inference). That report's list of unsuccessful mitigations (BIOS updates, llama.cpp rebuild from HEAD, driver adjustments, CUDA 12.8) closely matches what we independently tried and also failed to fix. There's an important distinction worth flagging rather than conflating, though: that report describes a full host freeze after ~45 minutes of sustained load requiring a hard power-cycle, with the rest of the system going unresponsive. Ours is a single-process hang that can happen almost immediately, with the rest of the system (including nvidia-smi and other GPU workloads) staying fully responsive throughout. This could be the same underlying issue at an earlier or lesser-severity stage, or a related but distinct bug. Noting the connection without asserting they're identical.

Also possibly related: microsoft/WSL#13734 (dxgkio_query_adapter_info ioctl failures blocking CUDA init on WSL2), though that report did not reach a confirmed resolution either.

Questions

• Is there a way to get any diagnostic signal (Xid code, dxgkrnl log, GSP state) during the stall rather than silence?
• Is this failure mode tracked internally as related to sshd server cause any ssh client unresponsive #1111, or is it a distinct issue?
• Any known parameter to force a GSP/GPU-PV reset instead of a full WSL shutdown when this happens?

Happy to run additional diagnostics or a debug build if that would help narrow this down further.

[github-actions]

Logs are required for review from WSL team

If this a feature request, please reply with '/feature'. If this is a question, reply with '/question'.
Otherwise, please attach logs by following the instructions below, your issue will not be reviewed unless they are added. These logs will help us understand what is going on in your machine.

How to collect WSL logs

Download and execute collect-wsl-logs.ps1 in an administrative powershell prompt:

Invoke-WebRequest -UseBasicParsing "https://raw.githubusercontent.com/microsoft/WSL/master/diagnostics/collect-wsl-logs.ps1" -OutFile collect-wsl-logs.ps1
Set-ExecutionPolicy Bypass -Scope Process -Force
.\collect-wsl-logs.ps1

The script will output the path of the log file once done.

If this is a networking issue, please use .\collect-wsl-logs.ps1 -LogProfile networking instead, following the instructions in Collect WSL logs for networking issues

Once completed please upload the output files to this GitHub issue.

See Collect WSL logs (recommended method).

If you choose to email these logs instead of attaching them to the bug, please send them to wsl-gh-logs@microsoft.com with the GitHub issue number in the subject, and include a link to your GitHub issue comment in the message body, and reply with '/emailed-logs'.

[t-timms]

WslLogs-2026-08-14_16-20-33.tar.gz

[t-timms]

Logs above were captured live during a reproduction of the hang — collect-wsl-logs.ps1 was run while llama-cli was actively stalled, and saved while the process was still hung rather than after killing it, in case that state mattered for diagnosis.

[github-actions]

Diagnostic information

.wslconfig found
Detected appx version: 2.7.11.0

[chemwolf6922]

Hi Tremayne Timms (@t-timms) .
Please help collect the call stack when llama.cpp hangs to help diagnose this issue. To do that:

1. Build llama.cpp with the -DCMAKE_BUILD_TYPE=RelWithDebInfo option.
2. When the hang happens, execute the following script in wsl:

pid=$(pgrep -n llama-cli)
{
  date -Ins
  ps -L -p "$pid" -o pid,tid,stat,wchan:40,comm
  echo "===== PROCESS MAPS ====="
  sudo cat "/proc/$pid/maps"
  for task in /proc/$pid/task/*; do
    echo "===== TID ${task##*/} ====="
    echo "--- syscall ---"
    sudo cat "$task/syscall"
    echo "--- kernel stack ---"
    sudo cat "$task/stack"
  done
} |& tee llama-hang-linux-waits.txt

sudo gdb -q -p "$pid" -batch \
  -ex 'set pagination off' \
  -ex 'set print address on' \
  -ex 'info threads' \
  -ex 'thread apply all bt 30' \
  -ex 'info sharedlibrary' \
  -ex 'info proc mappings' |& tee llama-hang-gdb.txt

dmesg -T | grep -Ei 'dxg|xid|nvrm|gpu|cuda' | tail -200

[chemwolf6922]

Please also provide the exact llama.cpp commit id when providing the info above. Thanks.