From 9d0576d42cc3610417b7fe9ef0c440bf9f144777 Mon Sep 17 00:00:00 2001 From: devlikebear Date: Mon, 24 Aug 2026 16:12:53 +0900 Subject: [PATCH] feat(usage): split recorded cache usage by request shape MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Makes the measurement #933's test plan deferred actually readable. Anthropic builds one prefix-matched cache key from tools -> system -> messages, so a request that omits tools cannot hit an entry written by a tool-bearing request even inside the same turn. agent.Loop ends every turn with a tools-absent call (ToolChoice none), so each turn emits both shapes and their cache reads/writes were being summed together — which is why the question "does the tools-absent final call pay for a cache entry nothing reads back" could not be answered from recorded usage. - Record ToolCount on usage.Entry from len(opts.Tools). - Add a "shape" group-by that keys on tool presence, so GET /v1/usage/summary?period=today&group_by=shape returns with-tools and no-tools rows. The handler passes group_by straight through, so no API change was needed. - Document how to read the two rows, what a regression looks like, and the three things that produce a false negative (pre-upgrade entries defaulting to no-tools, the ~1024-token minimum cacheable prefix, and the 5-minute ephemeral TTL) in docs/usage-signals.md. This is instrumentation only: no placement or request-building behavior changes, so existing cache behavior is untouched. --- CHANGELOG.md | 2 ++ docs/usage-signals.md | 33 +++++++++++++++++ internal/usage/client.go | 1 + internal/usage/tracker.go | 6 ++++ internal/usage/tracker_summary.go | 10 +++++- internal/usage/tracker_test.go | 60 +++++++++++++++++++++++++++++++ 6 files changed, 111 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index fc82d692..cfa4d336 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,8 @@ The format is based on Keep a Changelog and the project follows Semantic Version ### Added +- **요청 형태별 캐시 사용량 측정 (#921)** — `usage.Entry`에 `tool_count`를 기록하고 `GET /v1/usage/summary?group_by=shape`로 호출을 `with-tools` / `no-tools` 두 행으로 나눈다. Anthropic은 `tools` → `system` → `messages` 순으로 prefix를 만들기 때문에 tools 없는 요청은 같은 턴 안이라도 tool을 실은 요청이 쓴 캐시 엔트리에 걸릴 수 없다. `agent.Loop`가 매 턴을 tools 없는 호출로 끝내므로 한 턴이 두 형태를 모두 만들고, 이 분리 없이는 둘의 캐시 read/write가 뭉쳐 보여 어느 쪽이 이득이고 손해인지 판별할 수 없었다. #933이 남긴 미검증 측정 항목을 실제로 읽을 수 있게 하는 계측이며, 판독 방법과 함정은 `docs/usage-signals.md`에 정리했다. `tool_count`는 이 변경 이후 기록분에만 있으므로 그 이전 항목은 전부 `no-tools`로 잡힌다. + - **Anthropic 메시지 배열 rolling cache breakpoint (#921)** — 완료된 턴 경계에 최대 2개의 `cache_control` 마커를 찍어 대화가 길어져도 이전 transcript를 캐시 prefix로 재사용한다. 가장 새 마커는 직전 완료 턴의 마지막 메시지에, 두 번째는 그 앞 턴에 놓여 이전 요청이 이미 데운 위치를 fallback으로 유지한다. 진행 중인 턴(수신 user 메시지 + in-flight tool 교환)에는 찍지 않으므로 tool 루프 반복 사이에 배치가 흔들리지 않고 `tool_use`/`tool_result` 쌍도 갈라지지 않는다. system/tool 마커를 포함한 요청 전체 breakpoint는 프로바이더 상한 4개를 넘지 않는다. - **`antigravity-cli` provider** — Reuses a locally installed `agy` CLI the way `claude-code-cli` reuses `claude`, via `agy --print --output-format stream-json`. The CLI owns its Google login in the system keyring, so the kind takes no `api_key` and no `base_url` and no credential passes through TARS. Multi-turn resumes through the stream's `conversation_id` (`--conversation`), `ReasoningEffort` maps to `--effort`, and a `json_schema` response format maps to `--json-schema`. Tools remain the CLI's own and are reported on `ChatResponse.ProviderExecutedTools` for audit only, never re-dispatched through TARS' registry. `AGY_CLI_MODE` accepts only `accept-edits` or `plan`; there is deliberately no path to `--dangerously-skip-permissions`. Requires Antigravity CLI 1.1.12 or newer. diff --git a/docs/usage-signals.md b/docs/usage-signals.md index eb989056..a5e78873 100644 --- a/docs/usage-signals.md +++ b/docs/usage-signals.md @@ -65,3 +65,36 @@ The latest available workspace signal files were re-read for [the agent-control- The deterministic Agent Harness Evaluation Pack independently exercises parallel fan-out, dependency handoff, and partial-child failure without requiring a default-visible planner or consensus surface. Session Tasks remain the high-signal planning surface and should evolve into the Durable Work Ledger UI. The signal stream available for this review ends on 2026-05-20. These counts are a current review of the latest available local telemetry, not a claim of data collection through August. + +## Anthropic cache-breakpoint measurement (#921) + +`GET /v1/usage/summary?period=today&group_by=shape` splits recorded calls into +`with-tools` and `no-tools` rows. + +The split exists because Anthropic renders `tools` → `system` → `messages` into +one prefix-matched cache key. A request that omits `tools` therefore cannot hit +an entry written by a tool-bearing request, even inside the same agent turn. +`agent.Loop` ends every turn with one tools-absent call (`ToolChoice: none`), +so each turn produces both shapes. + +**What the rows mean** + +| Row | Expected if placement is working | Suspected regression looks like | +| --- | --- | --- | +| `with-tools` | `cache_read_tokens` grows with conversation length; `cache_write_tokens` stays near one prefix per turn | reads flat at 0 — a silent invalidator upstream | +| `no-tools` | reads roughly match the previous turn's writes | writes each turn with reads at 0 — paying the 1.25x write premium for an entry nothing reads back | + +If `no-tools` shows sustained writes with near-zero reads, the tools-absent +final call is a net loss and the fix is to keep `tools` on that call and rely +on `tool_choice: none` (already supported by `toAnthropicToolChoice`) so it +shares the turn's cache lineage. + +**Caveats** + +- `tool_count` is recorded per call from this change onward. Entries written + before it default to 0 and land in `no-tools`, so restrict any comparison to + a window that starts after the upgrade. +- Anthropic's minimum cacheable prefix is ~1024 tokens; below that nothing + caches and both rows read 0 regardless of placement. +- Ephemeral entries expire after 5 minutes, so a turn that takes longer than + that between calls will miss for reasons unrelated to placement. diff --git a/internal/usage/client.go b/internal/usage/client.go index 81beaed6..d3c13d88 100644 --- a/internal/usage/client.go +++ b/internal/usage/client.go @@ -75,6 +75,7 @@ func (c *TrackedClient) Chat(ctx context.Context, messages []llm.ChatMessage, op CacheReadTokens: resp.Usage.CacheReadTokens, CacheWriteTokens: resp.Usage.CacheWriteTokens, EstimatedCostUSD: estimatedCost, + ToolCount: len(opts.Tools), Source: meta.Source, SessionID: meta.SessionID, RunID: meta.RunID, diff --git a/internal/usage/tracker.go b/internal/usage/tracker.go index 594a8666..9185e876 100644 --- a/internal/usage/tracker.go +++ b/internal/usage/tracker.go @@ -38,6 +38,12 @@ type Entry struct { SessionID string `json:"session_id,omitempty"` RunID string `json:"run_id,omitempty"` PricingKnown bool `json:"pricing_known"` + // ToolCount is how many tool definitions the request carried. Anthropic + // renders tools ahead of messages in the cached prefix, so a tools-absent + // call cannot hit an entry written by a tool-bearing one even within the + // same turn. Recording it makes the two cache lineages separable — see + // the "shape" group-by in Summary. + ToolCount int `json:"tool_count,omitempty"` } type Summary struct { diff --git a/internal/usage/tracker_summary.go b/internal/usage/tracker_summary.go index 91068ab1..d6458cbf 100644 --- a/internal/usage/tracker_summary.go +++ b/internal/usage/tracker_summary.go @@ -134,7 +134,7 @@ func periodRange(raw string, now time.Time) (time.Time, string, error) { func normalizeGroupBy(raw string) string { v := strings.TrimSpace(strings.ToLower(raw)) switch v { - case "provider", "model", "source", "project", "run": + case "provider", "model", "source", "project", "run", "shape": return v default: return "provider" @@ -151,6 +151,14 @@ func summaryKey(entry Entry, groupBy string) string { return firstNonEmptyTrimmed(entry.Source, "(none)") case "run": return firstNonEmptyTrimmed(entry.RunID, "(none)") + case "shape": + // Tools are rendered ahead of messages in the provider's cached + // prefix, so these two groups never share a cache entry — not even + // within a single agent turn. + if entry.ToolCount > 0 { + return "with-tools" + } + return "no-tools" default: return firstNonEmptyTrimmed(entry.Provider, "(none)") } diff --git a/internal/usage/tracker_test.go b/internal/usage/tracker_test.go index c159b32d..40b97b31 100644 --- a/internal/usage/tracker_test.go +++ b/internal/usage/tracker_test.go @@ -298,3 +298,63 @@ func requireUsageFileMode(t *testing.T, path string, want os.FileMode) { t.Fatalf("expected %s mode %04o, got %04o", path, want, got) } } + +// The tools-absent final call of an agent turn sits in a different cache +// lineage than the tool-bearing loop iterations before it, because tools are +// rendered ahead of messages in the cached prefix. Grouping by request shape +// is what makes that split readable from recorded usage. +func TestTracker_SummaryGroupByShape(t *testing.T) { + now := time.Date(2026, 2, 22, 12, 0, 0, 0, time.UTC) + tracker, err := NewTracker(t.TempDir(), TrackerOptions{ + Now: func() time.Time { return now }, + }) + if err != nil { + t.Fatalf("new tracker: %v", err) + } + + entries := []Entry{ + {Timestamp: now, Provider: "anthropic", Model: "claude-opus-5", ToolCount: 12, + InputTokens: 100, CacheReadTokens: 900, PricingKnown: true}, + {Timestamp: now, Provider: "anthropic", Model: "claude-opus-5", ToolCount: 12, + InputTokens: 100, CacheReadTokens: 900, PricingKnown: true}, + {Timestamp: now, Provider: "anthropic", Model: "claude-opus-5", ToolCount: 0, + InputTokens: 100, CacheWriteTokens: 1200, PricingKnown: true}, + } + for _, e := range entries { + if err := tracker.Record(e); err != nil { + t.Fatalf("record: %v", err) + } + } + + summary, err := tracker.Summary("today", "shape") + if err != nil { + t.Fatalf("summary: %v", err) + } + if summary.GroupBy != "shape" { + t.Fatalf("group_by should survive normalization, got %q", summary.GroupBy) + } + + rows := map[string]SummaryRow{} + for _, row := range summary.Rows { + rows[row.Key] = row + } + withTools, ok := rows["with-tools"] + if !ok { + t.Fatalf("expected a with-tools row, got %+v", summary.Rows) + } + noTools, ok := rows["no-tools"] + if !ok { + t.Fatalf("expected a no-tools row, got %+v", summary.Rows) + } + if withTools.Calls != 2 || noTools.Calls != 1 { + t.Fatalf("expected 2 tool-bearing and 1 tool-free call, got %d and %d", withTools.Calls, noTools.Calls) + } + // This is the shape of the suspected regression: the tool-free call pays a + // cache write and reads nothing back. + if noTools.CacheWriteTokens == 0 || noTools.CacheReadTokens != 0 { + t.Fatalf("no-tools row must carry the write/no-read split, got %+v", noTools) + } + if withTools.CacheReadTokens == 0 { + t.Fatalf("with-tools row should show cache reads, got %+v", withTools) + } +}