Skip to content

fix(prompt): stop invalidating provider prompt caches on every turn - #932

Merged
devlikebear merged 1 commit into
mainfrom
fix/prompt-cache-prefix
Aug 23, 2026
Merged

fix(prompt): stop invalidating provider prompt caches on every turn#932
devlikebear merged 1 commit into
mainfrom
fix/prompt-cache-prefix

Conversation

@devlikebear

Copy link
Copy Markdown
Owner

Closes #920. Phase 0 / LP-001 of #919.

What was wrong

internal/prompt/builder.go:70 wrote a second-resolution timestamp as the first line of every system prompt. Prompt caching is prefix-matched, so nothing ever matched:

  • the cache_control marker on the Anthropic system block wrote a fresh entry every turn and never read one,
  • OpenAI and Gemini automatic prefix caching never engaged,
  • the whole static body (identity, planning, formatting, long-running command rules, workspace bootstrap) was re-charged at write rates on every turn.

Two things the issue did not name turned up during the audit and are fixed here as well:

  1. The builder's tail was not the assembled prompt's tail. buildContextFromResult and handler_chat_context.go append the memory rule, skills, session override, style, goal, and critic after the builder's output — so the per-turn ## Prior Context recall sat mid-prompt with static text behind it. Moving only the clock would have passed a prefix unit test while delivering almost none of the win.
  2. The chat memory cache stored the fully assembled prompt. The prefetch goroutine (handler_chat_prefetch.go) builds without the session's WorkDirs/CurrentDir, so once its entry landed in the cache the next turn's prompt lost ## Working Directories entirely — a correctness bug, not just a caching one.

What changed

Area Change
internal/prompt/builder.go Clock removed from the head; emitted as a trailing ## Current Time block, truncated to the minute. BuildResult exposes StaticPrompt / DynamicTail (Prompt is still the concatenation). Ordering invariant documented on BuildResultFor. PresetRelevantMemory lets a caller replay cached recall without re-searching.
internal/tarsserver/handler_chat*.go The dynamic tail is carried separately and emitted last, after skills / override / style / goal / critic. buildLLMMessagesWithTail sends it as a second, adjacent system message; an empty tail collapses to one message as before.
internal/llm/anthropic.go System messages are emitted as one text block each, with cache_control on the first (stable) block instead of after the joined text. A single system message behaves exactly as before.
internal/tarsserver/memory_cache.go Stores only the recall payload. The prompt is always rebuilt from live options, so cache hit ≡ cache miss by construction — immune to the next BuildOptions field somebody forgets to plumb.

Decisions the issue asked to document

  • System-prompt tail, not the first user message. Decorating the outgoing user turn would either pollute persisted history or leave the sent message differing from the stored one — which breaks the message-prefix matching feat(llm): add rolling cache breakpoints to Anthropic message history #921 (LP-002) is about to rely on. The system tail has neither problem.
  • Minute resolution. Second resolution is more precision than any assistant answer needs and re-breaks the prefix on every retry, tool-loop restart, and rapid follow-up — exactly the burst where a hit is worth most. Hour resolution would buy little more, since recall churn dominates in memory-bearing sessions, and it makes the time visibly stale.
  • Anthropic needs the block split, not just reordering. One breakpoint on a joined system block is all-or-nothing: a volatile tail anywhere inside it means zero reads. Reordering alone fixes OpenAI/Gemini (automatic longest-prefix) but not Anthropic. Breakpoint count stays at 2 of the allowed 4 (system + last tool), leaving room for feat(llm): add rolling cache breakpoints to Anthropic message history #921.

Observed, deliberately not changed

  • helpers_agent.go:496 — the cron/minimal agent prompt still embeds a second-resolution timestamp. One-shot prompts with no cross-turn reuse; nothing to cache.
  • Single-system-message paths (telegram, sub-agents, previews) stay all-or-nothing on Anthropic, exactly as before. They get the reordering benefit on OpenAI/Gemini.
  • BuildPriorContextPreview never included the prompt head, so it was unaffected.

Acceptance criteria

Criterion Status
Leading region byte-identical across two consecutive turns TestBuildResult_StaticPrefixSurvivesClockChange
Model still knows the current time TestBuildResult_KeepsCurrentTimeInTail, ..._SubAgentPromptStillCarriesTime
Prefetched and live prompts remain identical TestPrepareChatContext_CacheHitRebuildsLiveStaticRegion
Second Anthropic turn reports non-zero cache_read_input_tokens needs a live two-turn run against a real Anthropic tier — not runnable in CI, see below

Validation

  • bash scripts/windows_test.sh — green (the CI Windows job's exact selection).
  • golangci-lint run --enable=errcheck --enable=staticcheck --new-from-rev on the changed packages — 0 issues.
  • scripts/check_diff_coverage.sh95.0% (96/101 changed coverable lines), threshold 80%.
  • Note that windows_test.sh excludes internal/llm and internal/tarsserver wholesale, so locally those two packages were verified by targeted runs (all chat/memory/prompt/context/anthropic tests pass) plus a base-commit comparison confirming every remaining failure is a pre-existing Windows environment issue (POSIX paths, owner-only file modes, symlink privilege, macOS/Linux notifiers). The full-package proof lands in the Linux CI jobs.
  • Still outstanding: the manual two-turn session against a real Anthropic tier, recording cache_read_input_tokens before and after. Worth recording the before number too — the existing last-tool breakpoint may already produce a non-zero read, so the delta is what attributes the win to the system prefix.

Out of scope

Cache breakpoints on the message history (#921) and any change to what the static sections contain.

🤖 Generated with Claude Code

The system prompt led with a second-resolution wall-clock timestamp, so
its first token block differed on every request. Prompt caching is
prefix-matched, so nothing ever matched: the cache_control marker on the
Anthropic system block wrote a fresh entry each turn and never read one,
and OpenAI/Gemini automatic prefix caching never engaged either. The
whole static body — identity, planning, formatting, long-running command
rules, workspace bootstrap — was re-charged at write rates every turn.

Move the clock to a "## Current Time" block at the very end of the
prompt and truncate it to the minute, so a burst of turns shares one
prefix. Keep it in the system prompt rather than the first user message:
decorating the outgoing user turn would either pollute persisted history
or break message-prefix matching that LP-002 is about to rely on.

The builder's tail was not the assembled prompt's tail — the chat
assembler appends the memory rule, skills, session override, style,
goal, and critic *after* the builder's output, so per-turn recall sat
mid-prompt with static text behind it. BuildResult now exposes
StaticPrompt and DynamicTail separately and the assembler emits the tail
last, with the ordering invariant documented on BuildResultFor.

Anthropic gets one cache breakpoint on the system block, so a volatile
tail anywhere inside it is still all-or-nothing. Emit system messages as
one text block each and mark only the first — callers order them
stable-first, so the tail now falls outside the cached prefix. A single
system message behaves exactly as before.

Also fix a prompt-identity bug the audit turned up: the chat memory
cache stored the fully assembled prompt, and the prefetch goroutine
builds without the session's work dirs or current dir. Once its entry
landed in the cache, the next turn's prompt lost "## Working
Directories" entirely. The cache now holds only the recall payload and
the prompt is always rebuilt from live options, so a cache hit and a
cache miss are byte-identical by construction.

Closes #920

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@sonarqubecloud

Copy link
Copy Markdown

@devlikebear
devlikebear merged commit 4998a2b into main Aug 23, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(prompt): stop invalidating provider prompt caches on every turn

1 participant