fix: fail fast on endpoint response stalls - #462
Conversation
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
github-actions
Bot
added
the
size/normal
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #462 +/- ##
=======================================
Coverage ? 81.12%
=======================================
Files ? 150
Lines ? 20308
Branches ? 0
=======================================
Hits ? 16474
Misses ? 3834
Partials ? 0 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
02eeec7 to
a5b80ac
Compare
|
|
||
| 1. Publish `SessionEventType.STARTED` | ||
| 2. Start receiver coroutine (`_receive_responses`) | ||
| 2. Start receiver coroutine (`_receive_responses`). When the no-progress timeout is set, each active in-flight cohort arms a liveness watchdog; it exits as soon as the cohort drains. The watchdog fails the session only when an in-flight request has no streamed chunk or final result for the configured interval. |
There was a problem hiding this comment.
Lets not use cohort here.
| help=( | ||
| "Fail a run when in-flight requests make no " | ||
| "response progress for this many seconds" | ||
| ), | ||
| ), | ||
| ] = Field( | ||
| None, | ||
| gt=0, | ||
| description=( | ||
| "Fail a run when requests are in flight but no " | ||
| "response chunk or completion arrives for this many seconds. Disabled " | ||
| "by default; set above the longest expected interval between response " | ||
| "progress (full request latency for non-streaming endpoints) and, for " | ||
| "TensorRT-LLM disaggregated serving, match hang_detection_timeout." | ||
| ), |
There was a problem hiding this comment.
Lets keep a single copy in the help/description so there isn't drift over time.
arekay-nv
left a comment
arekay-nv
left a comment
There was a problem hiding this comment.
The opt-in liveness guard is useful, but this implementation adds avoidable work and state to the load generator's hot paths. The most serious issue is retaining every completed UUID in a second phase-lifetime container, which is an O(total requests) memory regression at the repository's 50k+ QPS target. The disabled path also performs work for every response, and low-concurrency workloads can create/cancel a watchdog task per request. Please keep the disabled path inert and use bounded active-request state plus a single event/deadline-driven watchdog.
| ) | ||
|
|
||
| if phase_issuer is not None and query_id in phase_issuer.uuid_to_index: | ||
| phase_issuer.completed_uuids.add(query_id) |
There was a problem hiding this comment.
[P1] Avoid retaining every normally completed UUID here. completed_uuids lives for the entire phase alongside uuid_to_index, so this adds a second O(total requests) hash index. At the repository's 50k+ QPS target, long phases can accumulate millions of decoded UUID strings and multi-GB incremental memory pressure, eventually affecting cache locality or causing OOM. Keep tombstones only for exceptional/synthetic completions, or track active IDs and remove them on completion.
| # This is a session-level liveness guard: any response proves | ||
| # the endpoint/transport is making progress. In particular, | ||
| # warmup responses may arrive after the performance phase starts. | ||
| self._record_response_activity() |
There was a problem hiding this comment.
[P2] Please keep the default-disabled path inert. This unconditionally calls monotonic_ns() and writes _last_response_progress_ns for every chunk and final response even when no_progress_timeout_s is None; streaming frame rate can substantially exceed request QPS. Gate activity tracking on the feature being enabled and reuse the timestamp already taken while handling the response.
| return | ||
| if self._progress_watchdog_task and not self._progress_watchdog_task.done(): | ||
| self._progress_watchdog_task.cancel() | ||
| self._progress_watchdog_task = asyncio.create_task(self._watch_no_progress()) |
There was a problem hiding this comment.
[P2] This creates a new asyncio task and timer for every 0-to-1 in-flight transition, then cancels it on 1-to-0. Concurrency-one and sparse workloads therefore allocate/cancel a watchdog task per request, directly adding scheduler/timer-heap work to the guarded hot path. Prefer one phase/session-lifetime watchdog driven by an activity event and updated deadline.
| self._strategy_task.cancel() | ||
| break | ||
| self._handle_response(resp) | ||
| await asyncio.sleep(timeout_s) |
There was a problem hiding this comment.
[P2] Sleeping a full timeout on every iteration can detect silence almost 2x later than configured: if progress occurs shortly after arming, the first wake observes it and then sleeps another full interval. Compute the remaining duration from last_progress + timeout or use an event-driven deadline. That also avoids keeping stale timer work around longer than necessary.
3 participants
Summary
no_progress_timeout_sthrough YAML and direct CLI aliasesWhen this is useful
Use this for automated runs where a request can be accepted but the endpoint then becomes silent—for example, a TensorRT-LLM disaggregated executor or KV-transfer stall that never reaches the normal terminal-error path. It is also engine-agnostic: it catches the same client-visible silent failure through vLLM, a frontend, or transport. Without it, the benchmark can remain blocked until an outer wall-time limit.
The guard is disabled by default. Enable it only for deployments where that failure mode matters; it starts after work is issued, runs only while requests are in flight, and resets on an observed stream chunk or final result. It does not diagnose or restart the backend—it makes the benchmark fail with a clear error. For non-streaming endpoints, configure it above the full expected request latency. For TensorRT-LLM disaggregated serving, the documented starting value is 300 seconds, matching the executor
hang_detection_timeout.Validation
pytest tests/unit/config/test_schema.py tests/unit/commands/test_benchmark.py tests/unit/load_generator/test_async_session.py -q(334 passed)python scripts/regenerate_templates.py --checkgit diff --checkEndpoint made no response progress for 10.0s with 1 request(s) in flight. The 10-second deadline was intentional fault-injection coverage, not the deployment recommendation.