Skip to content

feat(inference): declare how a context window was determined - #129

Merged
flyworker merged 1 commit into
mainfrom
feat/declare-context-provenance
Sep 5, 2026
Merged

feat(inference): declare how a context window was determined#129
flyworker merged 1 commit into
mainfrom
feat/declare-context-provenance

Conversation

@flyworker

Copy link
Copy Markdown
Member

Closes the request in #128.

The agent already distinguishes a window read from the backend's max_model_len from one asserted in models.json, but sent only the number. A receiver could not tell them apart, and they are not equally trustworthy:

  • a detection is self-verifying — vLLM and SGLang refuse to start when --max-model-len exceeds what the KV cache can hold, so the value is backed by memory that demonstrably exists;
  • an override is an assertion nothing on the agent side can check.

That matters once share of traffic is weighted by the declared window, because the override path is the one carrying the incentive.

What is sent

context_source now travels with the value, on both the heartbeat metadata and the schema-1 declaration:

{"model_id": "...", "context_length": 32768, "context_source": "detected"}
{"model_id": "...", "context_length": 128000, "context_source": "override"}
{"model_id": "...", "context_length": 32768}

Absent means unknown, by construction

As requested — an agent that does not send the field reads exactly as it did before, so this ships without a coordinated rollout.

unknown and pending are deliberately not sent explicitly: that is what an absent field already conveys, and emitting them would add a value every receiver has to special-case. A source is only ever attached to a window that was actually determined, so a length of 0 never carries one.

The tests assert the serialised form, not just the helper, because an empty string is a present field and would defeat the point.

Also

The provider function now returns the window and its provenance together rather than a bare int, so the two cannot drift apart at any of the three declaration sites.

Verified

4 new tests: the mapping for each source, the JSON genuinely omitting the key when undetermined, heartbeat metadata carrying provenance while an undetermined model is not declared at all, and the same for the schema-1 declaration including its serialised form. Existing context tests updated to the richer type.

8/8 packages, go vet and gofmt clean. Documented in docs/configuration.md.

The agent already distinguishes a window read from the backend's max_model_len
from one asserted in models.json, but sent only the number. A receiver could
not tell them apart, and they are not equally trustworthy.

A detection is self-verifying: vLLM and SGLang refuse to start when
--max-model-len exceeds what the KV cache can hold, so a declared window is
backed by memory that demonstrably exists. An override is an assertion nothing
on the agent side can check. That distinction starts to matter once share of
traffic is weighted by the declared window, because the override path is the
one with an incentive attached.

context_source now travels with the value on both the heartbeat metadata and
the schema-1 declaration, carrying "detected" or "override".

An absent field means unknown, deliberately. Agents that do not send it read
exactly as they did before, so this needs no coordinated rollout, and "unknown"
and "pending" are not sent explicitly — that is what absent already conveys,
and emitting them would add a value every receiver has to special-case. A
source is only ever attached to a window that was actually determined.

The provider now returns the window and its provenance together rather than
just an int, so the two cannot disagree.
@flyworker
flyworker merged commit e3321f0 into main Sep 5, 2026
1 check passed
@flyworker
flyworker deleted the feat/declare-context-provenance branch September 5, 2026 16:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant