Skip to content

distro matrix: stage H could only ever fail, in three more ways - #65

Merged
cnygaard merged 1 commit into
mainfrom
fix/distro-stage-h-false-negatives
Aug 18, 2026
Merged

distro matrix: stage H could only ever fail, in three more ways#65
cnygaard merged 1 commit into
mainfrom
fix/distro-stage-h-false-negatives

Conversation

@cnygaard

Copy link
Copy Markdown
Owner

The teardown check reported VLLM_GONE:False on ubuntu:26.04 across two releases, held up the 0.8.7 tag, and is written into the 0.8.7 release notes as a possible VRAM leak.

There was no leak. The same run's own log said stopping vLLM — the GPU is free again, and its nvidia-smi reported no compute apps on the card. Only the process bookkeeping disagreed, and the bookkeeping was broken.

Three defects, all in the test

1. pgrep -f 'vllm|VLLM' matched PID 1. In a container PID 1 is the bash -c <script> running this test, and the script's own text contains vllm--components core,vllm, and the pgrep pattern itself. PID 1 never exits while the script runs, so the "are they all gone?" loop could never succeed:

VLLM_PIDS:1 4475 4562      ← PID 1 is the test's own shell
VLLM_GONE:False            ← foregone conclusion

2. The UI was curled once, immediately after the readiness loop. But that loop waits on vLLM's /v1/models, and glq-chat starts gradio only after the server answers — so there is a window where the API is up and the UI is not. A single curl wins or loses by luck: it passed on 2026-08-17 and failed on 2026-08-18 with identical code. Now polled, with the same die-early check the readiness loop uses.

3. GPU_APPS_AFTER was printed but never asserted — and it is the one signal process bookkeeping cannot fool. An empty compute-apps list is the claim being made, so it is now an assertion.

Verified

RTX PRO 6000 (sm_120), no CUDA Toolkit, glq 0.8.7 installed from source:

EXT_OK:True   GEN_TEXT:Paris. Paris is the most visited city
CHAT_READY:True   UI_OK:True   CHAT_TEXT:Paris. Paris is the most visited city
VLLM_PIDS:4398 4485      ← PID 1 absent for the first time
VLLM_GONE:True
GPU_APPS_AFTER:          ← card empty
stopping vLLM — the GPU is free again

1 passed in 417.62s

Teardown is now confirmed three independent ways: every vLLM PID gone, nvidia-smi clean, and the supervisor's own stop message.

Follow-up

The 0.8.7 release notes carry the retracted claim under "Scope and gaps" — it says the container leg reported vLLM surviving the stop signal. That is now known to be a test artifact and the notes should be corrected; not done in this PR since it edits a published release rather than the tree.

This run also re-confirms, on a second Blackwell box, that 0.8.7 works end to end on sm_120 with no CUDA Toolkit: install, kernels, FULL CUDA-graph capture, a served model, coherent output via both the API and the UI, and a clean shutdown.

The teardown check reported `VLLM_GONE:False` on ubuntu:26.04 across two releases and was
carried in the 0.8.7 release notes as a possible VRAM leak. It was not one. The same run's
own log said "stopping vLLM — the GPU is free again" and its nvidia-smi reported no compute
apps on the card; only the process bookkeeping disagreed, and it was wrong.

Three defects, all in the stage rather than the product:

1. `pgrep -f 'vllm|VLLM'` matched **PID 1**. In a container PID 1 is the `bash -c <script>`
   running this test, and the script text contains "vllm" — `--components core,vllm`, and
   the pgrep pattern itself. PID 1 never exits while the script runs, so the "are they all
   gone?" loop could never succeed. VLLM_GONE:False was a foregone conclusion.

2. The UI was curled **once**, immediately after the readiness loop — but that loop waits on
   vLLM's `/v1/models`, and glq-chat starts gradio only after the server answers. A single
   curl wins or loses that race by luck: it passed on 2026-08-17 and failed on 2026-08-18
   with identical code. Now polled, with the same die-early check the readiness loop uses.

3. Nothing asserted on `GPU_APPS_AFTER`, which is the one signal that cannot be fooled by
   process bookkeeping. An empty nvidia-smi compute-apps list is the actual claim — the card
   is free — so it is now an assertion rather than a line in the output.

Verified on an RTX PRO 6000 (sm_120, no CUDA Toolkit), glq 0.8.7 installed from source:

    EXT_OK:True   GEN_TEXT:Paris...   CHAT_READY:True   UI_OK:True
    CHAT_TEXT:Paris...   VLLM_PIDS:4398 4485   VLLM_GONE:True   GPU_APPS_AFTER:
    1 passed in 417.62s

with PID 1 absent from VLLM_PIDS for the first time.
@cnygaard
cnygaard enabled auto-merge (squash) August 18, 2026 18:47
@cnygaard
cnygaard merged commit 7177bac into main Aug 18, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant