chat: make the wait for vLLM configurable, and give the matrix 20 minutes - #69
Merged
Conversation
…utes The supervisor gave up after a hardcoded 900 s with no way to ask for more. That is generous for a 1.8 GiB checkpoint on an idle card and too short in two real cases: a large MoE loading from cold storage, and any situation where several servers start at once. Measured in the distro matrix on a 16-core box at 5-way concurrency: **12 of 44 legs failed with "glq-chat never got a server"** — not because anything was broken, but because five simultaneous weight loads and CUDA-graph captures outran the window. The same legs pass solo. `still has compute apps` stayed 0 throughout, so nothing was actually wrong with GLQ. `glq-chat --ready-timeout SECONDS` now sets it, defaulting to the same 900 s as before so nobody's behaviour changes silently. Zero and negative are refused at parse time: "give up before asking" surfaces as an instant, inexplicable startup failure rather than as the configuration error it is. The matrix raises both halves to 20 minutes — its own polling loop AND the value it passes to glq-chat. Raising either alone just moves which one fires first: the loop waited 10 minutes while the chat gave up at its own 15, so the chat died and the loop's liveness check broke out early. `GLQ_DISTRO_READY_TIMEOUT` overrides.
Wheel-mode legs install whatever is published, and --ready-timeout only exists from 0.8.8. Passing it unconditionally is an argparse error on older releases — glq-chat exits before starting anything and the leg fails for a feature that has not shipped. Exactly the trap that made stage H fail against 0.8.6's flag set. Probe `glq-chat --help` and pass the flag only when it is supported, saying so when it is not, so a leg running on the older default is visible rather than silent.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The supervisor gave up after a hardcoded 900 s with no way to ask for more.
That is generous for a 1.8 GiB checkpoint on an idle card, and too short in two real cases: a large MoE loading from cold storage, and any situation where several servers start at once.
The measurement
Distro matrix on a 16-core box at 5-way concurrency:
The same legs pass solo, and which ones fail moves between runs —
ubuntu2604passed both modes at-n 2and failed source at-n 5. That is startup outrunning the window, not a distro or mode property.Change
glq-chat --ready-timeout SECONDS, defaulting to the existing 900 s so nobody's behaviour changes silently.glq-chat, withGLQ_DISTRO_READY_TIMEOUTto override.That last point is the subtle one: raising either alone just moves which one fires first. The harness waited 10 minutes while the chat gave up at its own 15, so the chat died and the harness's liveness check broke out of the loop early. Both numbers had to move together.
Tests
Three new cases: the default is unchanged, the flag reaches the supervisor, and a non-positive value exits. 81 chat/supervisor tests pass; the generated container script still parses under
bash -n, withseq 1 240and--ready-timeout 1200both present.