distro matrix: stage H could only ever fail, in three more ways - #65
Merged
Conversation
The teardown check reported `VLLM_GONE:False` on ubuntu:26.04 across two releases and was
carried in the 0.8.7 release notes as a possible VRAM leak. It was not one. The same run's
own log said "stopping vLLM — the GPU is free again" and its nvidia-smi reported no compute
apps on the card; only the process bookkeeping disagreed, and it was wrong.
Three defects, all in the stage rather than the product:
1. `pgrep -f 'vllm|VLLM'` matched **PID 1**. In a container PID 1 is the `bash -c <script>`
running this test, and the script text contains "vllm" — `--components core,vllm`, and
the pgrep pattern itself. PID 1 never exits while the script runs, so the "are they all
gone?" loop could never succeed. VLLM_GONE:False was a foregone conclusion.
2. The UI was curled **once**, immediately after the readiness loop — but that loop waits on
vLLM's `/v1/models`, and glq-chat starts gradio only after the server answers. A single
curl wins or loses that race by luck: it passed on 2026-08-17 and failed on 2026-08-18
with identical code. Now polled, with the same die-early check the readiness loop uses.
3. Nothing asserted on `GPU_APPS_AFTER`, which is the one signal that cannot be fooled by
process bookkeeping. An empty nvidia-smi compute-apps list is the actual claim — the card
is free — so it is now an assertion rather than a line in the output.
Verified on an RTX PRO 6000 (sm_120, no CUDA Toolkit), glq 0.8.7 installed from source:
EXT_OK:True GEN_TEXT:Paris... CHAT_READY:True UI_OK:True
CHAT_TEXT:Paris... VLLM_PIDS:4398 4485 VLLM_GONE:True GPU_APPS_AFTER:
1 passed in 417.62s
with PID 1 absent from VLLM_PIDS for the first time.
cnygaard
enabled auto-merge (squash)
August 18, 2026 18:47
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The teardown check reported
VLLM_GONE:Falseon ubuntu:26.04 across two releases, held up the 0.8.7 tag, and is written into the 0.8.7 release notes as a possible VRAM leak.There was no leak. The same run's own log said
stopping vLLM — the GPU is free again, and itsnvidia-smireported no compute apps on the card. Only the process bookkeeping disagreed, and the bookkeeping was broken.Three defects, all in the test
1.
pgrep -f 'vllm|VLLM'matched PID 1. In a container PID 1 is thebash -c <script>running this test, and the script's own text containsvllm—--components core,vllm, and the pgrep pattern itself. PID 1 never exits while the script runs, so the "are they all gone?" loop could never succeed:2. The UI was curled once, immediately after the readiness loop. But that loop waits on vLLM's
/v1/models, andglq-chatstarts gradio only after the server answers — so there is a window where the API is up and the UI is not. A single curl wins or loses by luck: it passed on 2026-08-17 and failed on 2026-08-18 with identical code. Now polled, with the same die-early check the readiness loop uses.3.
GPU_APPS_AFTERwas printed but never asserted — and it is the one signal process bookkeeping cannot fool. An empty compute-apps list is the claim being made, so it is now an assertion.Verified
RTX PRO 6000 (sm_120), no CUDA Toolkit, glq 0.8.7 installed from source:
Teardown is now confirmed three independent ways: every vLLM PID gone, nvidia-smi clean, and the supervisor's own stop message.
Follow-up
The 0.8.7 release notes carry the retracted claim under "Scope and gaps" — it says the container leg reported vLLM surviving the stop signal. That is now known to be a test artifact and the notes should be corrected; not done in this PR since it edits a published release rather than the tree.
This run also re-confirms, on a second Blackwell box, that 0.8.7 works end to end on sm_120 with no CUDA Toolkit: install, kernels, FULL CUDA-graph capture, a served model, coherent output via both the API and the UI, and a clean shutdown.