fix: bound history to 70% context + 512KB cap (503/timeout/empty-response/slowness) - #169
Open
Fahad090NP wants to merge 10 commits into
Open
fix: bound history to 70% context + 512KB cap (503/timeout/empty-response/slowness)#169Fahad090NP wants to merge 10 commits into
Fahad090NP wants to merge 10 commits into
Conversation
…ontext window Long multi-turn conversations (or many repeated turns without Compact Conversation) can grow past the model context limit, so the upstream rejects the oversized request (HTTP 400) or returns an empty stream — surfaced by VS Code as 'No response came'. Add a pure helper that drops the oldest messages until the payload fits the input budget, preserving the anchor and current prompt and never splitting a tool-call group. Add HISTORY_TRIM_SAFETY_MARGIN_TOKENS.
…hatResponse Call trimOldMessagesToFitContext after the image-history trim, before the prompt-token estimate, so the request payload is bounded to the model's input context window. Logs a [history-trim] diagnostic when messages are dropped.
Add unit tests for no-op when fitting, dropping oldest while keeping anchor + last, never splitting a tool-call group, and no-op when only anchor + last remain.
Add a Fixed entry to CHANGELOG and a concise issue doc (68-20260820) describing the conversation-history trimming that prevents oversized-payload 'No response came' errors.
…YTES Bound conversation-history trimming to a safe fraction of the context window (0.7, matching the ~70%-full failure threshold the reporter hit) and add a hard 512 KB payload ceiling as a backstop for inaccurate token estimates.
…p units
Rewrite trimOldMessagesToFitContext to bound the payload on both a token
budget and a hard byte ceiling, drop complete tool-call groups as one unit
(never orphaning a reference), run in O(n) (estimate once, subtract per dropped
unit) instead of the previous O(n^2) whole-payload re-estimation that caused
per-request session slowness, and return { removed, finalTokens, finalBytes }
so the caller skips a second re-estimation.
…stimate Use HISTORY_TRIM_TARGET_RATIO and MAX_REQUEST_PAYLOAD_BYTES for the history budget and consume the estimate returned by trimOldMessagesToFitContext instead of re-estimating the trimmed payload.
Add cases for the hard byte-cap backstop, dropping a complete tool-call group as one unit, returned token/byte estimates, and anchor/current-prompt preservation; update the signature for the new maxBytes argument.
Document the 503/timeout/empty-response symptoms, the 70% context ratio and 512 KB byte-cap bounds, the O(n) performance fix, and the extended tests.
A chat request could fail with a hard "Sorry, your request failed" dialog when fetch() threw a transient network error (e.g. ECONNRESET, EAI_AGAIN, or the undici socket-reuse race in nodejs/undici#5450) before any HTTP response arrived. The engine only retried HTTP outcomes (400 param-patch, 5xx backoff), so the thrown error propagated as a hard failure. Wire the existing isTransientFetchError classifier (used by the model-list fetch, issue ltmoerdani#78) into the chat path via a new fetchWithTransientRetry wrapper that retries transient fetch() throws with exponential backoff + jitter (TRANSIENT_FETCH_* constants, matching the model-list values). Refs ltmoerdani#69.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes the full symptom set from issue #165 — all rooted in an unbounded conversation-history payload:
503 Endpoint is unavailable(payloadBytes≈ 783 KB)request timed out after 10m 0s(upstream hangs on a huge payload)The reporter observed failures begin at ~70% context full and that Compact Conversation fixes them — confirming the payload (not the model) is the cause.
Root cause
provideLanguageModelChatResponsesent the entire history every turn. The only guard was image trimming. Two gaps remained:context − output − margin, which can still sit above the ~70% failure threshold, so trimming to "the window" alone didn't always drop enough.Changes
src/config.ts—HISTORY_TRIM_TARGET_RATIO = 0.7(stay safely below the observed failure threshold) andMAX_REQUEST_PAYLOAD_BYTES = 512 KB(hard byte backstop for inaccurate token estimates).src/provider/historyTrim.ts—trimOldMessagesToFitContextnow bounds on both a token budget and the byte ceiling, runs in O(n) (estimate once, subtract per dropped unit), drops complete tool-call groups as one unit (never orphaning a reference), preserves anchor + current prompt, and returns{ removed, finalTokens, finalBytes }.src/provider/OpenCodeProvider.ts— trims to the 70% ratio + byte cap and reuses the returned estimate (no second re-estimation).src/test/messages.test.ts— extended: byte-cap enforcement, tool-group dropping, returned estimates, anchor/prompt preservation.docs/issues/68-20260820-history-trim-context-overflow.md.Verification
npm test(all pass) andnpm run lint(all 7 gates) green.Closes #165
PEACE BE UPON YOU
Also includes: chat-request fetch() resilience (issue #69)
This PR now also contains the fix for transient
fetch()failures on the chatrequest path (the
Reason: fetch failed: TypeError: fetch failederror). Theengine now retries transient network throws via
fetchWithTransientRetry,mirroring the model-list fetch retry from issue #78. See
docs/issues/69-20260820-chat-request-fetch-resilience.md.