fix(prompt): stop invalidating provider prompt caches on every turn - #932
Merged
Conversation
The system prompt led with a second-resolution wall-clock timestamp, so its first token block differed on every request. Prompt caching is prefix-matched, so nothing ever matched: the cache_control marker on the Anthropic system block wrote a fresh entry each turn and never read one, and OpenAI/Gemini automatic prefix caching never engaged either. The whole static body — identity, planning, formatting, long-running command rules, workspace bootstrap — was re-charged at write rates every turn. Move the clock to a "## Current Time" block at the very end of the prompt and truncate it to the minute, so a burst of turns shares one prefix. Keep it in the system prompt rather than the first user message: decorating the outgoing user turn would either pollute persisted history or break message-prefix matching that LP-002 is about to rely on. The builder's tail was not the assembled prompt's tail — the chat assembler appends the memory rule, skills, session override, style, goal, and critic *after* the builder's output, so per-turn recall sat mid-prompt with static text behind it. BuildResult now exposes StaticPrompt and DynamicTail separately and the assembler emits the tail last, with the ordering invariant documented on BuildResultFor. Anthropic gets one cache breakpoint on the system block, so a volatile tail anywhere inside it is still all-or-nothing. Emit system messages as one text block each and mark only the first — callers order them stable-first, so the tail now falls outside the cached prefix. A single system message behaves exactly as before. Also fix a prompt-identity bug the audit turned up: the chat memory cache stored the fully assembled prompt, and the prefetch goroutine builds without the session's work dirs or current dir. Once its entry landed in the cache, the next turn's prompt lost "## Working Directories" entirely. The cache now holds only the recall payload and the prompt is always rebuilt from live options, so a cache hit and a cache miss are byte-identical by construction. Closes #920 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Closes #920. Phase 0 / LP-001 of #919.
What was wrong
internal/prompt/builder.go:70wrote a second-resolution timestamp as the first line of every system prompt. Prompt caching is prefix-matched, so nothing ever matched:cache_controlmarker on the Anthropic system block wrote a fresh entry every turn and never read one,Two things the issue did not name turned up during the audit and are fixed here as well:
buildContextFromResultandhandler_chat_context.goappend the memory rule, skills, session override, style, goal, and critic after the builder's output — so the per-turn## Prior Contextrecall sat mid-prompt with static text behind it. Moving only the clock would have passed a prefix unit test while delivering almost none of the win.handler_chat_prefetch.go) builds without the session'sWorkDirs/CurrentDir, so once its entry landed in the cache the next turn's prompt lost## Working Directoriesentirely — a correctness bug, not just a caching one.What changed
internal/prompt/builder.go## Current Timeblock, truncated to the minute.BuildResultexposesStaticPrompt/DynamicTail(Promptis still the concatenation). Ordering invariant documented onBuildResultFor.PresetRelevantMemorylets a caller replay cached recall without re-searching.internal/tarsserver/handler_chat*.gobuildLLMMessagesWithTailsends it as a second, adjacent system message; an empty tail collapses to one message as before.internal/llm/anthropic.gocache_controlon the first (stable) block instead of after the joined text. A single system message behaves exactly as before.internal/tarsserver/memory_cache.goBuildOptionsfield somebody forgets to plumb.Decisions the issue asked to document
Observed, deliberately not changed
helpers_agent.go:496— the cron/minimal agent prompt still embeds a second-resolution timestamp. One-shot prompts with no cross-turn reuse; nothing to cache.BuildPriorContextPreviewnever included the prompt head, so it was unaffected.Acceptance criteria
TestBuildResult_StaticPrefixSurvivesClockChangeTestBuildResult_KeepsCurrentTimeInTail,..._SubAgentPromptStillCarriesTimeTestPrepareChatContext_CacheHitRebuildsLiveStaticRegioncache_read_input_tokensValidation
bash scripts/windows_test.sh— green (the CI Windows job's exact selection).golangci-lint run --enable=errcheck --enable=staticcheck --new-from-revon the changed packages — 0 issues.scripts/check_diff_coverage.sh— 95.0% (96/101 changed coverable lines), threshold 80%.windows_test.shexcludesinternal/llmandinternal/tarsserverwholesale, so locally those two packages were verified by targeted runs (all chat/memory/prompt/context/anthropic tests pass) plus a base-commit comparison confirming every remaining failure is a pre-existing Windows environment issue (POSIX paths, owner-only file modes, symlink privilege, macOS/Linux notifiers). The full-package proof lands in the Linux CI jobs.cache_read_input_tokensbefore and after. Worth recording the before number too — the existing last-tool breakpoint may already produce a non-zero read, so the delta is what attributes the win to the system prefix.Out of scope
Cache breakpoints on the message history (#921) and any change to what the static sections contain.
🤖 Generated with Claude Code