Verify the 32768 context declarations against each backend's actual max_model_len

[flyworker]

The agent declares a per-model context window at register and heartbeat. That number is shown to users and decides which requests are routed to a node, so a wrong declaration costs the operator real traffic.

The question this issue opened with was whether 32768 is a detected value or a fallback the agent emits when detection fails. Several nodes declare it on models whose published window is larger, which is what raised the question.

Resolved for the one node that could be inspected directly. On Nova Provider, 32,768 is a correct measurement of a backend genuinely configured at 32,768, and the agent has no fallback constant. Evidence in the comments below, including the memory arithmetic that explains why it is 32,768 on that hardware.

What was checked on the agent

• backend is vLLM, launched --served-model-name TheDrummer/Cydonia-24B-v4.3 --max-model-len 32768
• /v1/models returns max_model_len: 32768
• the agent declares 32768 (detected) — a measurement, not a constant
• a long prompt sent to that backend returns HTTP 400, "maximum context length is 32768 tokens"

There is also no fallback constant in the agent and never has been: git log -S'32768' -- '*.go' returns three commits and every occurrence is a test fixture; no DefaultContextLength exists in the history. resolveModelContexts already omits any model whose window it cannot determine, which is the behaviour asked for below — 7 of this node's 8 models are absent from the declaration rather than guessed at.

32,768 is a very common --max-model-len for 24B-class models on constrained VRAM, so more than one operator landing on it is what a popular default looks like.

Still open: the same check on other nodes

Per node, this settles it:

docker inspect <container> --format '{{join .Config.Cmd " "}}' | grep -A1 max-model-len
curl -s localhost:<port>/v1/models        # compare max_model_len
computing-provider inference status       # compare what the agent declares

If the command line, /v1/models, and inference status all agree, the declaration is accurate and there is nothing to fix. A case where the declared value and the backend's own max_model_len disagree would be a real bug and worth reporting here.

Preferred agent behaviour (unchanged, and already implemented)

If the real window cannot be determined, send nothing rather than a guess. An absent declaration is treated as "unknown" and falls back to the model's published window; a present one is treated as a claim and routed on. Guessing low silently costs the operator every long-context request.

For vLLM and SGLang the value comes from max_model_len in /v1/models. For backends that expose nothing (Ollama, llama.cpp, OpenAI-compatible proxies), models.json supports a manual context_length override, and it wins over detection.

Follow-up: declaration provenance

The agent already distinguishes detected / override / unknown internally, and the two paths differ in how much they can be trusted:

• detected (max_model_len from vLLM/SGLang) is self-verifying — vLLM refuses to start when --max-model-len exceeds what the KV cache can hold.
• override (context_length in models.json) is forwarded as written, because the agent has no way to confirm it.

Sending that provenance alongside the value would let the receiving side tell a self-verifying declaration from an unverifiable one. Tracked separately; no agent change is required for it to be adopted, since an absent field means unknown and every agent in the field keeps working unchanged.

[flyworker]

I own Nova Provider, so I can answer the closing question directly, with the box in front of me.

Short version: on this node the declaration is a correct measurement, not a guess, and the 121k-prompt evidence does not hold up.

What the box reports

$ docker ps
vllm/vllm-openai:latest   vllm-cydonia   0.0.0.0:30000->8000/tcp

$ docker inspect vllm-cydonia --format '{{join .Config.Cmd " "}}'
... --served-model-name TheDrummer/Cydonia-24B-v4.3 --max-model-len 32768

$ curl -s localhost:30000/v1/models
{"id":"TheDrummer/Cydonia-24B-v4.3","owned_by":"vllm","max_model_len":32768}

So: vLLM, launched with --max-model-len 32768, reporting 32768. Container started 2026-08-30, RestartCount=0, so it has been 32768 for the whole period in question.

And the agent reports it as a measurement, not a constant:

$ computing-provider inference status
Reported context windows
  TheDrummer/Cydonia-24B-v4.3                 32768  (detected)
  Qwen/Qwen3.8-27B                                -  (not reported)
  ... 7 model(s) report no context window.

The 121k prompt could not have been served

I sent one to that backend just now:

HTTP 400 — This model's maximum context length is 32768 tokens. However, you
requested 1 output tokens and your prompt contains at least 32768 input tokens...

vLLM rejects, as the issue predicts it would. This node also keeps its own per-request log, and across 1010 Cydonia rows the largest tokens_in is 197, with none above 32768.

So 424 requests larger than 32768, up to 121,210 tokens, successfully is not something this backend can have done. That points at the server-side attribution — whether those requests were counted against the right (provider, model) pair, whether their prompt sizes are being measured as sent rather than as accepted, and whether rejected requests are being recorded as served.

On the fallback hypothesis

There is no fallback constant in the agent, and there never has been. git log -S'32768' -- '*.go' returns three commits and every occurrence is a test fixture; there is no DefaultContextLength in the history at all.

The current behaviour is already what the issue asks for — resolveModelContexts omits any model whose window it cannot determine:

Models with no known window are omitted — the server falls back to the catalog value for those.

This node demonstrates that: 7 of its 8 models are absent from the declaration rather than guessed at, because their backend exposes no max_model_len.

The two patterns, re-read

• "schema 1 reports either a detected value or 32768" — schema-1 agents omit unknowns entirely, so every value you see from them is a detection. That makes 32768 a reading, not a default.
• "the same agent reports a specific value when it identifies the engine and 32768 when it does not" — this node has no declared_engine for Cydonia and still reports a correct detected 32768, so engine identification and window detection are not coupled that way.

32768 is simply a very common --max-model-len for 24B-class models on constrained VRAM. Three providers landing on it for Cydonia and gemma is what a popular default looks like.

Worth checking on the other two

The check that settles it per node is the one above: --max-model-len on the vLLM/SGLang command line versus what /v1/models returns. If inference-07 and zm1 also show 32768 there, all six declarations are accurate and the discrepancy is entirely in the request-size accounting.

Happy to dig further on the agent side if a case turns up where the declared value and the backend's own max_model_len actually disagree — that would be a real bug and I would want it fixed. I have not been able to produce one here.

[flyworker]

Following up with the memory arithmetic, because it answers a question the first comment left open: not just whether 32,768 is real, but why it is 32,768.

The hardware sets the number

Nova Provider runs Cydonia on two RTX 3080s (10 GiB each), TP=2, AWQ, fp8 KV cache. From vLLM's own startup log:

Model loading took 6.66 GiB          (per GPU, of a 13.26 GiB checkpoint)
Available KV cache memory: 1.55 GiB  (per GPU)
GPU KV cache size: 40,592 tokens
Maximum concurrency for 32,768 tokens per request: 1.24x

That works out to a measured 80.1 KiB of KV per token across both GPUs, already at fp8. So the KV cache tops out at 40,592 tokens, and 32,768 is the sensible round window underneath it — it leaves 1.24× concurrency, meaning the engine can hold more than one full-length request. Declaring 40k would drop that to 1.0×: one long request at a time, everything else queued.

On 10 GiB cards the weights are what eat the budget. 6.66 GiB of the 8.8 GiB per-GPU allowance is model, leaving 1.55 GiB for KV. Reaching 64k needs 5.00 GiB total (2.50 GiB/GPU) and would require pushing gpu-memory-utilization to ~0.97, which leaves ~300 MiB of headroom on a 10 GiB card — not a margin to run production on, since vLLM refuses to start rather than degrade. 0.95 → ~56k is the honest two-card ceiling. TP=4 would drop weights to 3.33 GiB/GPU and put 131,072 comfortably in reach, at the cost of the second model currently on those cards.

So the declaration is not a fallback, not a placeholder, and not wrong. It is a correct reading of a genuinely constrained backend.

One consequence for the planned routing weight

If share of traffic is weighted by declared window relative to the catalogue value, this node is weighted at 32,768/131,072 for a limit it declared honestly and accurately. That is the intended behaviour — smaller capacity, smaller share — but it is worth being explicit that the mechanism is measuring hardware, not conduct.

The part that deserves a second look is that the two declaration paths have very different trustworthiness:

• Detected (max_model_len from vLLM/SGLang) is self-verifying. An operator cannot inflate it: vLLM refuses to start when --max-model-len exceeds what the KV cache can hold, so a declared window is backed by memory that actually exists.
• Override (context_length in models.json, for llama.cpp/Ollama/proxies that expose nothing) is an unchecked assertion. The agent forwards whatever is written there, because it has no way to confirm it.

Once traffic share depends on the declared number, the override path is the one with an incentive attached, and it is the path where nothing on the agent side can validate the claim. Worth deciding server-side how much weight an unverified declaration should carry relative to a detected one — the agent already distinguishes them internally (override / detected / unknown) and could send that provenance along with the value if it would help.

Happy to add the provenance field to the declaration if that is useful.

[flyworker]

The memory arithmetic is the most useful thing on this issue — thank you for working it out rather than just asserting the number. 80.1 KiB of KV per token at fp8, 1.55 GiB of KV budget per card, a 40,592-token cache, 1.24× concurrency at 32,768, and 0.95 → ~56k as the honest two-card ceiling. That is a window derived from hardware, and it settles what this issue opened with.

On the accounting question from your first comment

Checked on our side. It is a timing artefact, not an accounting one. The requests you were asked about predate your container's creation by several weeks, and nothing served since it was created exceeds the declared window. Your box and our records agree — they were describing two different backends.

So: nothing to fix on the agent, and nothing to fix in the accounting. Happy to walk through the detail with you directly.

The reconciliation that produced the question was the weak part. It compared a declaration read today against a long window of history, with nothing tying a declaration to the period it was actually in force. That does not detect a discrepancy; it detects an operator who narrowed a window, which is a normal thing to do. Being reworked.

"Measuring hardware, not conduct" is right, and worth saying plainly

A node that declares a smaller window honestly is not being penalised for anything — it is being matched to what it can actually serve, which is the outcome that makes the request succeed instead of failing at the backend. Declaring 40k to look bigger, at 1.0× concurrency, would be worse for you and worse for the consumer.

What must never happen is an accurate low declaration and an inflated high one landing in the same place. Which is why the next point is the important one.

Yes, please add the provenance field

detected / override / unknown is the right split, and you have put your finger on the asymmetry precisely:

• a detection is self-verifying — vLLM refuses to start when the window exceeds what the KV cache can hold, so the number is backed by memory that demonstrably exists;
• an override is an assertion nothing on the agent side can check, and it is the path that would carry the incentive.

Today a receiver cannot tell them apart, and the override path is in use on the network right now, so this is live rather than hypothetical.

One request: please make an absent field mean unknown, so every agent in the field keeps working unchanged and you can ship it whenever it suits you rather than coordinating a rollout.

How much a self-verifying declaration should count for relative to an unverifiable one is being tracked separately on our side. The direction is settled — a detection is never worth less than an override — the size is not, and there is a real question about how a new node with an honest override and no history should be treated before it has served anything. Not something to decide in a hurry.

[flyworker]

Thanks for chasing the accounting question down — a timing artefact between a declaration read today and history from a backend that no longer exists is a much better answer than either of the ones we started with, and it explains the whole picture without anything being broken.

Provenance is implemented: #129.

context_source travels with the value on both the heartbeat metadata and the schema-1 declaration:

{"model_id": "...", "context_length": 32768,  "context_source": "detected"}
{"model_id": "...", "context_length": 128000, "context_source": "override"}
{"model_id": "...", "context_length": 32768}

Absent means unknown, as asked. An agent that does not send the field reads exactly as it did before, so it needs no coordinated rollout — deploy the receiver whenever it suits you and the values start appearing as agents update.

Two decisions worth stating, since they affect what a receiver has to handle:

• unknown and pending are not sent explicitly. That is what an absent field already conveys, and emitting them would add a value every receiver has to special-case for no gain. So the field is either missing, detected, or override — three states, not five.
• A source is only ever attached to a window that was actually determined. A context_length of 0 never carries one, so there is no combination where the provenance describes nothing.

The tests assert the serialised JSON rather than just the helper, because an empty string is a present field and would defeat the point.

On the open question of what a new node with an honest override and no history should be worth: agreed that is not one to rush. Worth noting the agent can tell you why an override exists — it is only reachable for backends that expose no max_model_len at all, so an override on a llama.cpp or Ollama endpoint is the operator supplying information the engine simply does not publish, not choosing to bypass detection. If it would help to distinguish "override because detection is impossible here" from "override on a backend that could have been detected", the agent knows which is which and I can send that too.

[flyworker]

Verifying the provenance change on a live node turned up a real agent bug, and it is the answer to a question this issue asked directly:

models.json supports a manual context_length override. Is it applied, and does it win over the fallback?

It was not applied. Fixed in #130.

loadModelMappings read models.json correctly, then the registry's added/updated callbacks rebuilt each entry from a hand-written field list that omitted ContextLength — and those callbacks fire after the initial load, and again on every hot reload. The override survived only until the registry reported the model back, which is immediately.

On this node:

"Qwen/Qwen3.8-27B": { "endpoint": "http://localhost:30001", "context_length": 65536 }

before:  Qwen/Qwen3.8-27B      -      (not reported)
after:   Qwen/Qwen3.8-27B      65536  (override)

Qwen runs on llama.cpp, which publishes no max_model_len, so the override was the only source of truth and the agent was declaring nothing — the server fell back to the catalogue value for a window the operator had stated explicitly.

This is the same concern as the one this issue opened with, arriving from the opposite direction: not a guess sent as a measurement, but a measurement silently discarded. And it is the more damaging of the two under the planned weighting, because it under-declares an operator who did everything right.

local_model was being dropped by the same code path. It does not currently bite — every forwarding path asks the registry first and only falls back to the mapping — but it was one refactor away from mattering. Both callbacks now go through a single constructor that copies every field the registry holds, so a field added later cannot be lost in the mirror again.

Both changes are deployed on Nova Provider (0.5.4+git.b74da56), which now declares:

Qwen/Qwen3.8-27B              65536  (override)
TheDrummer/Cydonia-24B-v4.3   32768  (detected)

Worth knowing server-side: any provider using a context_length override on a backend that exposes no max_model_len has been declaring nothing at all, not a wrong value. So an offering that appears to have "no declaration" in provider_models may be an operator whose override was being erased rather than one who never configured it — which changes how the absent case should be read in the reconstruction.

[flyworker]

Both agent-side changes are now in a published release: v0.5.5. Previously they existed only on Nova Provider, which was no use to anyone else in the table above.

• fix: stop the registry callbacks erasing context_length and local_model #130 — the models.json context_length override is no longer erased by the registry callbacks
• feat(inference): declare how a context window was determined #129 — declarations carry context_source: detected, override, or absent for unknown

That matters for inference-07 and zm1 specifically. If either is using a context_length override on a backend that publishes no max_model_len, it has been declaring nothing rather than a wrong value, and upgrading is what makes their real window visible. Operators can take it with:

computing-provider update --check
sudo computing-provider update

update verifies the download against the release checksums; it does not restart a running agent, so the new declaration goes out on the next restart.

Where this leaves the issue as I understand it:

thread	state
agent emits a 32768 fallback where detection fails	refuted — no such constant, and none in the history
Nova Provider served 121k on a 32768 declaration	timing artefact; the requests predate that container
is the models.json override applied?	was a real bug, fixed and released
receiver cannot distinguish detected from asserted	context_source added and released
inference-07 and zm1 declarations	unverified — needs their operators, and is worth re-checking after they upgrade
weighting for verified vs unverified declarations	tracked separately server-side

Happy for this to stay open on the last two rows, or to close it and open something narrower for the re-check once those two nodes are on v0.5.5 — whichever is more useful to you.

[flyworker]

Closing — everything actionable on the agent has been fixed and released.

What this issue produced:

• fix: stop the registry callbacks erasing context_length and local_model #130 — a real bug it uncovered: models.json context_length was read at startup and then erased by the registry callbacks, so an override on a backend publishing no max_model_len declared nothing at all. This was the more damaging of the two failure modes considered here, because it under-declares an operator who configured things correctly.
• feat(inference): declare how a context window was determined #129 — declarations now carry context_source (detected / override / absent for unknown), so a receiver can tell a self-verifying measurement from an unverifiable assertion.

Both shipped in v0.5.5.

What it ruled out: there is no 32768 fallback in the agent and never has been — every occurrence in the history is a test fixture — and the >32k request counts were a timing artefact against a container that no longer exists.

Two threads outlive this issue and are not tracked by closing it: the inference-07 and zm1 declarations are still unverified, and is worth re-checking once those nodes are on v0.5.5 rather than before; and how much a verified declaration should be worth relative to an unverified one is being decided server-side.