feat(inference): declare how a context window was determined - #129
Merged
Conversation
The agent already distinguishes a window read from the backend's max_model_len from one asserted in models.json, but sent only the number. A receiver could not tell them apart, and they are not equally trustworthy. A detection is self-verifying: vLLM and SGLang refuse to start when --max-model-len exceeds what the KV cache can hold, so a declared window is backed by memory that demonstrably exists. An override is an assertion nothing on the agent side can check. That distinction starts to matter once share of traffic is weighted by the declared window, because the override path is the one with an incentive attached. context_source now travels with the value on both the heartbeat metadata and the schema-1 declaration, carrying "detected" or "override". An absent field means unknown, deliberately. Agents that do not send it read exactly as they did before, so this needs no coordinated rollout, and "unknown" and "pending" are not sent explicitly — that is what absent already conveys, and emitting them would add a value every receiver has to special-case. A source is only ever attached to a window that was actually determined. The provider now returns the window and its provenance together rather than just an int, so the two cannot disagree.
This was referenced Sep 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the request in #128.
The agent already distinguishes a window read from the backend's
max_model_lenfrom one asserted inmodels.json, but sent only the number. A receiver could not tell them apart, and they are not equally trustworthy:--max-model-lenexceeds what the KV cache can hold, so the value is backed by memory that demonstrably exists;That matters once share of traffic is weighted by the declared window, because the override path is the one carrying the incentive.
What is sent
context_sourcenow travels with the value, on both the heartbeat metadata and the schema-1 declaration:{"model_id": "...", "context_length": 32768, "context_source": "detected"} {"model_id": "...", "context_length": 128000, "context_source": "override"} {"model_id": "...", "context_length": 32768}Absent means unknown, by construction
As requested — an agent that does not send the field reads exactly as it did before, so this ships without a coordinated rollout.
unknownandpendingare deliberately not sent explicitly: that is what an absent field already conveys, and emitting them would add a value every receiver has to special-case. A source is only ever attached to a window that was actually determined, so a length of 0 never carries one.The tests assert the serialised form, not just the helper, because an empty string is a present field and would defeat the point.
Also
The provider function now returns the window and its provenance together rather than a bare
int, so the two cannot drift apart at any of the three declaration sites.Verified
4 new tests: the mapping for each source, the JSON genuinely omitting the key when undetermined, heartbeat metadata carrying provenance while an undetermined model is not declared at all, and the same for the schema-1 declaration including its serialised form. Existing context tests updated to the richer type.
8/8 packages,
go vetandgofmtclean. Documented indocs/configuration.md.