Skip to content

fix(llm): stop rolling cache breakpoints from forfeiting marker slots - #936

Merged
devlikebear merged 1 commit into
mainfrom
fix/anthropic-cache-breakpoint-cleanup
Aug 24, 2026
Merged

fix(llm): stop rolling cache breakpoints from forfeiting marker slots#936
devlikebear merged 1 commit into
mainfrom
fix/anthropic-cache-breakpoint-cleanup

Conversation

@devlikebear

Copy link
Copy Markdown
Owner

Summary

Review follow-up to #933 (#921). Cleanup and one real logic gap; placement rules are unchanged.

  • Marker slots were silently forfeited. The budget was truncated to the newest two completed turns before markability was known, so a newest turn that cannot carry a marker (empty assistant content, or a block array ending in tool_use) cost a slot outright and the request shipped with one breakpoint or none. Slots are now filled newest-first and consumed only when a marker actually lands, so an unmarkable turn falls through to an older completed turn.
  • The budget arithmetic was unreachable. hasSystemBlocks/hasTools could only ever yield 2 — all four combinations collapse through the cap — and the budget <= 0 guard was dead. Replaced by a single reservedBreakpoints count plus anthropicMessageCacheBudget, whose branches are now reachable and pinned per reservation level.
  • Two identical branches in anthropicCompletedTurnEndIndexes, plus the prevGroupStart variable that existed only to choose between them.
  • Four copies of the cache_control literal, now behind anthropicEphemeralCacheControl() (returns a fresh map per call — callers store it into blocks they do not own).
  • Two wrong doc comments. The fallback marker sits one completed turn back, not two. And the in-flight exclusion was justified by "would write a fresh entry per loop iteration without ever being read", which is not true here: agent.Loop appends each tool exchange to the same slice and re-sends it, so a marker on a completed in-flight tool_result would be read by the next iteration. The comment now states this is a deliberately conservative choice and names the in-flight marker as the next step.
  • CHANGELOG entries feat(llm): add rolling cache breakpoints to anthropic message history #933 never wrote.

Deliberately out of scope

Actually placing a marker inside the in-flight group, and the Tools: nil final call at internal/agent/loop.go:431 (tools precede messages in the cached prefix, so that call sits in its own cache lineage and pays a 1.25x write). Both change token spend rather than clean up code, and #933's test plan left the cache_read_input_tokens measurement unchecked — they belong in a PR that measures first.

Test plan

  • 6 new tests: rolling window advancing across turns, fallback to an older turn when the newest is unmarkable, per-reservation budget table, length-mismatch guard
  • Fallback test verified to fail against the old logic ([3] vs [1,3])
  • make test — full suite green
  • make vet, gofmt clean
  • make lint-diff — 0 issues
  • make test-cover-diff — 95.7% (45/47 changed lines), threshold 80%

🤖 Generated with Claude Code

Follow-up review fixes for #933 (#921).

- Fill breakpoint slots newest-first and consume a slot only when a marker
  actually lands, so a turn that cannot carry one (empty content, trailing
  tool_use) falls back to an older completed turn instead of being dropped.
  Previously the budget was truncated before markability was known, so a
  long history could ship one marker or none.
- Replace the unreachable budget arithmetic (hasSystemBlocks/hasTools bools
  could only ever yield 2) with a reserved-slot count plus a separate
  anthropicMessageCacheBudget helper, pinned per reservation level by test.
- Collapse the two identical branches in anthropicCompletedTurnEndIndexes and
  drop the prevGroupStart variable that only chose between them.
- Route all four cache_control literals through anthropicEphemeralCacheControl.
- Correct the doc comment: the fallback marker sits one completed turn back,
  not two, and the in-flight exclusion is a conservative choice rather than a
  free one — agent.Loop re-sends the same slice, so a marker there would be
  read by the next iteration.
- Add the CHANGELOG entries #933 never wrote.
@sonarqubecloud

Copy link
Copy Markdown

@devlikebear
devlikebear merged commit 745423d into main Aug 24, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant