Skip to content

fix(agent): keep the turn's final call in the same cache lineage - #940

Merged
devlikebear merged 1 commit into
mainfrom
fix/final-call-cache-lineage
Aug 25, 2026
Merged

fix(agent): keep the turn's final call in the same cache lineage#940
devlikebear merged 1 commit into
mainfrom
fix/final-call-cache-lineage

Conversation

@devlikebear

Copy link
Copy Markdown
Owner

Summary

Resolves the open question from the #933 review — does the tools-absent final call pay a cache-write premium for an entry nothing reads back? — by removing the condition that made it ambiguous, rather than by measuring it.

The finalization call (the one that extracts an answer after the loop exhausts its iterations, internal/agent/loop.go:428) went out with Tools: nil. Providers render toolssystemmessages into one prefix-matched cache key, so that request:

  • cannot read the prefix the tool-bearing iterations of the same turn just wrote, and
  • pays its own cache write at the 1.25x premium for an entry no following call in that turn can read back.

Keeping the tool list and suppressing it with tool_choice: none puts the call back in the turn's lineage.

A side effect worth stating plainly

This is the first time ToolChoiceNone() actually reaches the wire. Every provider emits tool_choice only inside if len(tools) > 0:

  • internal/llm/anthropic.go:171
  • internal/llm/openai_compat_client.go:108
  • internal/llm/gemini_native_convert.go (same shape)

So with Tools: nil, the none was silently dropped on every provider and suppression rested entirely on there being no tools to call.

The guarantee that had to be preserved

Dropping the tools is what made a text answer structural rather than a request the provider might ignore. TestLoop_Run_FinalizesWithoutToolsWhenMaxIterationsReached pinned that, though it asserted the mechanism (toolCount == 0) rather than the outcome.

So the old behavior is kept as a fallback: if the finalization response comes back with tool calls and no content, retry once with the tools removed. Without it, a provider that ignores none degrades that turn into the exceeded max iterations error instead of an answer. The extra round trip happens only when a provider misbehaves.

The existing test now asserts the outcome it was protecting; the ignore-none path has its own test.

Provider impact

This is internal/agent/loop.go, so it affects every provider — not just Anthropic:

Provider ToolChoiceModeNone maps to Effect
anthropic {"type":"none"} joins the turn's cache lineage
openai / openai-compatible "none" same, plus OpenAI's automatic prefix caching
gemini-native Mode: "NONE" same
claude-code-cli, antigravity-cli unaffected; neither reads ChatOptions.Tools

The cost tradeoff: the tools block is now sent on the final call too. On a caching provider it is read back cheaply; on a non-caching one it is a modest input increase, since the same block already went out on every loop iteration.

Test plan

  • New test: final call keeps the tool list and sends tool_choice=none
  • New test: provider ignoring none triggers the tools-removed retry and still answers
  • Both verified red before the change (got [1 0])
  • make test — full suite green
  • make vet, gofmt clean
  • make lint-diff — 0 issues
  • make test-cover-diff — 100.0% (28/28 changed lines)

🤖 Generated with Claude Code

Closes the measurement question #933's test plan deferred, by removing the
condition that made it ambiguous rather than by measuring it.

The finalization call — the one that extracts an answer after the loop
exhausts its iterations — went out with Tools: nil. Providers render
tools -> system -> messages into one prefix-matched cache key, so a
tools-absent request cannot read the prefix the tool-bearing iterations just
wrote, and pays its own cache write at the 1.25x premium for an entry no
following call in that turn can read back.

Keeping the tool list and suppressing it with tool_choice=none puts the call
back in the turn's lineage. This is also the first time ToolChoiceNone
actually reaches the wire: every provider emits tool_choice only inside
`if len(tools) > 0` (anthropic.go, openai_compat_client.go,
gemini_native_convert.go all share the shape), so with Tools: nil the "none"
was silently dropped and suppression relied entirely on there being no tools
to call.

Dropping the tools is what made a text answer structural rather than a
request the provider might ignore, so that guarantee is kept as a fallback:
if the finalization response comes back with tool calls and no content, retry
once the old way. Without it that case degrades to the max-iterations error
instead of an answer. Costs an extra round trip only when a provider
misbehaves.

TestLoop_Run_FinalizesWithoutToolsWhenMaxIterationsReached asserted the
mechanism (tool count 0); it now asserts the outcome it was protecting, with
the ignore-none path covered by its own test.

Verified: make test green, make lint-diff 0 issues, diff coverage 100%
(28/28). CLI providers are unaffected — claude-code-cli and antigravity-cli
never read ChatOptions.Tools.
@sonarqubecloud

Copy link
Copy Markdown

@devlikebear
devlikebear merged commit 6642341 into main Aug 25, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant