feat(llm): add rolling cache breakpoints to anthropic message history - #933
Conversation
toAnthropicWireMessages emitted the message array with no cache markers, so every turn paid full input rates for the whole transcript. The API allows four breakpoints and only two were used (system head, last tool). Place up to two message-level markers deterministically on completed turn boundaries: the newest on the last message of the most recent completed turn — making everything before the incoming turn one cacheable prefix — and an older marker one completed turn further back so a warm entry survives when the newest is invalidated. The trailing in-flight group (incoming user message plus any partial tool exchanges) never gets a marker, so placement is stable across tool-loop iterations and never splits a tool_use/tool_result pairing; a defensive guard also refuses to mark any tool_use-bearing content. The message budget is what remains of the provider-wide limit after the system and tool markers, so total breakpoints stay within four for every shape of request. Third-party Anthropic-compatible gateways (e.g. minimax) receive the same standard field; if one rejects it the request errors through the normal error path by design — no special-case retries. Closes #921
|
Code review — rolling cache breakpointsReviewed at high effort (8 finder angles). Direction is right; a few things should land before merge. Ranked by impact. 1. The tool loop is exactly where this stops helping — and the comment's rationale doesn't hold here
The doc comment justifies excluding the trailing in-flight group like this:
That isn't true of this codebase's loop. So with a 20-iteration loop over exec/file-read output — usually the bulk of the request — every iteration re-bills the entire growing tool transcript at full input rate. That's the cost this PR is meant to eliminate, and it's the part left on the table. Moving the newest marker to the last completed 2. The tools-absent final call sits in its own cache lineage
Every turn ends with a To be fair to the design: that write isn't necessarily wasted. Turn N's final call can hit turn N-1's final-call entry, since both share the tools-absent shape. But it's a separate lineage that only pays off when consecutive turns land inside the 5-minute TTL — and before this PR that call billed at a flat 1.0x with no markers at all. This is the concrete thing the unchecked test-plan box should measure. Recording 3. The budget arithmetic can only ever return 2
All four input combinations collapse to the same answer:
The accounting also hard-codes an assumption that 4. Two identical branches, and a variable that exists only to pick between them
if prevGroupStart >= 0 {
ends = append(ends, i-1)
} else if i > 0 {
ends = append(ends, i-1)
}
5. A skipped marker forfeits its slot instead of falling back
Older, perfectly markable turn ends sit at 6. Comment says two turns back; the code marks adjacent turns
Line 206 keeps 7.
|
…#936) Follow-up review fixes for #933 (#921). - Fill breakpoint slots newest-first and consume a slot only when a marker actually lands, so a turn that cannot carry one (empty content, trailing tool_use) falls back to an older completed turn instead of being dropped. Previously the budget was truncated before markability was known, so a long history could ship one marker or none. - Replace the unreachable budget arithmetic (hasSystemBlocks/hasTools bools could only ever yield 2) with a reserved-slot count plus a separate anthropicMessageCacheBudget helper, pinned per reservation level by test. - Collapse the two identical branches in anthropicCompletedTurnEndIndexes and drop the prevGroupStart variable that only chose between them. - Route all four cache_control literals through anthropicEphemeralCacheControl. - Correct the doc comment: the fallback marker sits one completed turn back, not two, and the in-flight exclusion is a conservative choice rather than a free one — agent.Loop re-sends the same slice, so a marker there would be read by the next iteration. - Add the CHANGELOG entries #933 never wrote.
Makes the measurement #933's test plan deferred actually readable. Anthropic builds one prefix-matched cache key from tools -> system -> messages, so a request that omits tools cannot hit an entry written by a tool-bearing request even inside the same turn. agent.Loop ends every turn with a tools-absent call (ToolChoice none), so each turn emits both shapes and their cache reads/writes were being summed together — which is why the question "does the tools-absent final call pay for a cache entry nothing reads back" could not be answered from recorded usage. - Record ToolCount on usage.Entry from len(opts.Tools). - Add a "shape" group-by that keys on tool presence, so GET /v1/usage/summary?period=today&group_by=shape returns with-tools and no-tools rows. The handler passes group_by straight through, so no API change was needed. - Document how to read the two rows, what a regression looks like, and the three things that produce a false negative (pre-upgrade entries defaulting to no-tools, the ~1024-token minimum cacheable prefix, and the 5-minute ephemeral TTL) in docs/usage-signals.md. This is instrumentation only: no placement or request-building behavior changes, so existing cache behavior is untouched.
Closes the measurement question #933's test plan deferred, by removing the condition that made it ambiguous rather than by measuring it. The finalization call — the one that extracts an answer after the loop exhausts its iterations — went out with Tools: nil. Providers render tools -> system -> messages into one prefix-matched cache key, so a tools-absent request cannot read the prefix the tool-bearing iterations just wrote, and pays its own cache write at the 1.25x premium for an entry no following call in that turn can read back. Keeping the tool list and suppressing it with tool_choice=none puts the call back in the turn's lineage. This is also the first time ToolChoiceNone actually reaches the wire: every provider emits tool_choice only inside `if len(tools) > 0` (anthropic.go, openai_compat_client.go, gemini_native_convert.go all share the shape), so with Tools: nil the "none" was silently dropped and suppression relied entirely on there being no tools to call. Dropping the tools is what made a text answer structural rather than a request the provider might ignore, so that guarantee is kept as a fallback: if the finalization response comes back with tool calls and no content, retry once the old way. Without it that case degrades to the max-iterations error instead of an answer. Costs an extra round trip only when a provider misbehaves. TestLoop_Run_FinalizesWithoutToolsWhenMaxIterationsReached asserted the mechanism (tool count 0); it now asserts the outcome it was protecting, with the ignore-none path covered by its own test. Verified: make test green, make lint-diff 0 issues, diff coverage 100% (28/28). CLI providers are unaffected — claude-code-cli and antigravity-cli never read ChatOptions.Tools.
1 participant
Summary
cache_controlmarkers on the Anthropic message array at completed-turn boundaries: newest on the last message of the most recent completed turn, older one turn back as a rolling fallback.tool_use/tool_resultpairing; defensive guard also skips any tool_use-bearing content.kind: anthropic, e.g. minimax) receive the same standard field; rejection degrades through the normal error path by design.Closes #921 · Part of #919 (LP-002)
Test plan
tool_useand matchingtool_result