Merged
Conversation
Follow-up to the first cut, which only warmed the Nemotron stream and so did nothing when the stream itself is the problem — which is the actual case here. The deployment runs the hybrid backend (stream partials + Parakeet finals), but the Nemotron streaming server is unreachable/misconfigured (the host port that should serve it answers as an unrelated HTTP LLM). In that state the old code opened the stream *lazily on the first audio chunk*, inline on the caption path, and — because the api reads audio frames serially — stalled every frame behind the connect. Measured against a hung handshake, `open()` blocks for the whole `open_timeout`, which shipped at 15s. So each session stalled for up to 15s with no captions while audio piled up, then flushed the whole backlog at once: exactly "a delay, then all the words since the convo started". Re-decode partials were also suppressed the entire time because the stream still counted as "active". Meanwhile the offline Parakeet path is fast and healthy (~110-140ms per decode), so it can carry the live captions on its own with no perceptible startup delay. Fix — decouple captions from the stream entirely: - Open the stream only OFF the caption path (a background task, kicked off by warmup() and re-armed by push), never inline. `push` never awaits it. - Until the stream is confirmed open, and forever if it never opens, partials come from the fast re-decode path. A dead/slow stream no longer delays or suppresses captions. - Switch a turn to stream partials only if the stream was ready at the turn's first chunk (latched per turn), so a stream that finishes opening mid-turn can't rewrite a caption already on screen; a mid-turn stream failure falls straight back to re-decode for the rest of the turn. - Add API_STT_STREAM_OPEN_TIMEOUT_MS (default 4s, was a hard-coded 15s) bounding how long we keep trying the stream before settling on Parakeet-only partials. Verified end to end against the real Parakeet server with the stream pointed at a hung endpoint (full 4s timeout): streaming the JFK clip in real time, the first caption lands at ~790ms with real words and the full transcript follows; the worst single push blocks ~660ms (a Parakeet decode), never the stream open. Full api suite green (296 passed), coverage 96%, ruff clean.
…stops wedging (XERK-128) The live cause of the caption stall: the Nemotron streaming server on the GPU box answers HTTP /health in ~1ms but NEVER completes the /v1/audio/stream WebSocket handshake — the 101 is never sent, so the api's stream open hangs for its whole timeout every session. Reproduced against the live server (raw upgrade: no bytes in 8s; health: 1ms) and confirmed consistent (3/3 opens time out). This is the `websockets` library biting the handshake a second time. The server already switched off uvicorn's legacy `websockets` impl (which 400s a valid upgrade against websockets>=14) to `websockets-sansio`; that workaround now hangs instead against the version frozen in the image (deps are unpinned `>=`). Both failures are the same root: a hard dependency on uvicorn's fast-moving `websockets` adapter. Fix: serve with **wsproto**, a self-contained sans-I/O WebSocket implementation independent of the `websockets` library, via `ws=WS_IMPL`. Verified locally to complete the handshake AND the full partial/reset protocol driven by the api's real client. `websockets` stays installed (uvicorn[standard] pulls it; the api's own client uses it) but no longer serves this endpoint. Also add a real-uvicorn regression test: it starts an actual server on the configured WS impl and does a real handshake + protocol round-trip. The existing tests use FastAPI's TestClient, whose in-memory WS transport bypasses uvicorn's adapter entirely — so they could never have caught this. A wedged handshake now fails that test (open_timeout) instead of shipping. And align the repo compose's published Nemotron port 9402 -> 9403 to match the GPU deployment (DockerOps tenir-gpu.yaml: 9401 Parakeet, 9402 cue LLM, 9403 Nemotron), so the two topologies stop disagreeing on which port is which. NOTE: final proof needs a rebuild on the GPU host and a re-test of the live WS — which can't run from here; local repro + the api-side graceful fallback (#84) cover the rest.
xerhab
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
XERK-128 — the full fix (two layers)
"When a session starts there is a delay before text starts to appear, but when it does the text has all the words since the convo started."
The delay is the api opening the Nemotron partials WebSocket inline on the caption path while that server's WS handshake hangs. Two independent fixes, so captions are fast and the stream actually works:
1. api — never block captions on the stream
The old code opened the Nemotron WS lazily on the first audio chunk, inline; because the WS handler reads frames serially, a hung handshake stalled every frame behind it for the whole
open_timeout(shipped at 15s), while re-decode partials were suppressed. Per session: up to 15s of silence, then the whole backlog flushed at once.warmup(), re-armed bypush);pushnever awaits it.API_STT_STREAM_OPEN_TIMEOUT_MS(default 4s, was a hard-coded 15s).2. nemotron-stt — fix the WS handshake wedge
Measured against the live GPU box: Nemotron on 9403 answers HTTP
/healthin ~1ms but never completes the/v1/audio/streamhandshake (raw upgrade → no bytes in 8s; 3/3 opens time out). That's thewebsocketslibrary biting the handshake a second time — the legacy uvicorn impl 400s a valid upgrade against websockets≥14, and thewebsockets-sansioworkaround now silently hangs against the version frozen in the image (deps are unpinned>=).ws=WS_IMPL) — a self-contained sans-I/O implementation independent of thewebsocketslibrary's churn. Verified locally to complete the handshake and the full partial/reset protocol via the api's real client.TestClient, whose in-memory transport bypasses uvicorn's WS adapter, so they could never have caught this; a wedged handshake now fails that test.docker-compose.yml's published Nemotron port 9402 → 9403 to match the GPU deployment (9401 Parakeet, 9402 cue LLM, 9403 Nemotron).Verification
pushblocks ~660ms (a decode), never the stream open.Deploy notes
API_STT_STREAM_OPEN_TIMEOUT_MS.