Skip to content

chat: make the wait for vLLM configurable, and give the matrix 20 minutes - #69

Merged
cnygaard merged 2 commits into
mainfrom
feat/ready-timeout-flag
Aug 19, 2026
Merged

chat: make the wait for vLLM configurable, and give the matrix 20 minutes#69
cnygaard merged 2 commits into
mainfrom
feat/ready-timeout-flag

Conversation

@cnygaard

Copy link
Copy Markdown
Owner

The supervisor gave up after a hardcoded 900 s with no way to ask for more.

That is generous for a 1.8 GiB checkpoint on an idle card, and too short in two real cases: a large MoE loading from cold storage, and any situation where several servers start at once.

The measurement

Distro matrix on a 16-core box at 5-way concurrency:

never got a server           12     ← of 44 legs
Error in memory profiling     3
still has compute apps        0     ← nothing was actually wrong

The same legs pass solo, and which ones fail moves between runs — ubuntu2604 passed both modes at -n 2 and failed source at -n 5. That is startup outrunning the window, not a distro or mode property.

Change

  • glq-chat --ready-timeout SECONDS, defaulting to the existing 900 s so nobody's behaviour changes silently.
  • Zero and negative are refused at parse time. "Give up before asking" surfaces as an instant, inexplicable startup failure rather than as the configuration error it is.
  • The matrix raises both halves to 20 minutes — its own polling loop and the value it passes to glq-chat, with GLQ_DISTRO_READY_TIMEOUT to override.

That last point is the subtle one: raising either alone just moves which one fires first. The harness waited 10 minutes while the chat gave up at its own 15, so the chat died and the harness's liveness check broke out of the loop early. Both numbers had to move together.

Tests

Three new cases: the default is unchanged, the flag reaches the supervisor, and a non-positive value exits. 81 chat/supervisor tests pass; the generated container script still parses under bash -n, with seq 1 240 and --ready-timeout 1200 both present.

…utes

The supervisor gave up after a hardcoded 900 s with no way to ask for more. That is generous
for a 1.8 GiB checkpoint on an idle card and too short in two real cases: a large MoE loading
from cold storage, and any situation where several servers start at once.

Measured in the distro matrix on a 16-core box at 5-way concurrency: **12 of 44 legs failed
with "glq-chat never got a server"** — not because anything was broken, but because five
simultaneous weight loads and CUDA-graph captures outran the window. The same legs pass
solo. `still has compute apps` stayed 0 throughout, so nothing was actually wrong with GLQ.

`glq-chat --ready-timeout SECONDS` now sets it, defaulting to the same 900 s as before so
nobody's behaviour changes silently. Zero and negative are refused at parse time: "give up
before asking" surfaces as an instant, inexplicable startup failure rather than as the
configuration error it is.

The matrix raises both halves to 20 minutes — its own polling loop AND the value it passes
to glq-chat. Raising either alone just moves which one fires first: the loop waited 10
minutes while the chat gave up at its own 15, so the chat died and the loop's liveness check
broke out early. `GLQ_DISTRO_READY_TIMEOUT` overrides.
Wheel-mode legs install whatever is published, and --ready-timeout only exists from 0.8.8.
Passing it unconditionally is an argparse error on older releases — glq-chat exits before
starting anything and the leg fails for a feature that has not shipped. Exactly the trap
that made stage H fail against 0.8.6's flag set.

Probe `glq-chat --help` and pass the flag only when it is supported, saying so when it is
not, so a leg running on the older default is visible rather than silent.
@cnygaard
cnygaard merged commit f0ba296 into main Aug 19, 2026
3 checks passed
@cnygaard
cnygaard deleted the feat/ready-timeout-flag branch August 19, 2026 19:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant