Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,9 @@ The format is based on Keep a Changelog and the project follows Semantic Version
### Fixed

- **rolling cache breakpoint가 마커를 조용히 흘리던 문제 (#921)** — 예산은 마킹 가능 여부를 따지기 전에 최신 2개 턴으로 잘려 있어서, 가장 새 턴이 마커를 못 받는 형태(내용이 빈 assistant 메시지, 끝이 `tool_use`인 블록)면 그 슬롯을 더 오래된 턴으로 넘기지 않고 그냥 버렸다. 이제 최신 턴부터 역순으로 훑으며 **실제로 마커가 찍힌 경우에만** 예산을 소모하므로, 마킹 불가능한 턴은 건너뛰고 그 앞 턴이 fallback 자리를 채운다. 아울러 breakpoint 예산 계산에서 실행되지 않던 분기를 걷어내고(`hasSystemBlocks`/`hasTools` bool 두 개 → 예약 슬롯 수 `int` 하나), `anthropicMessageCacheBudget`으로 분리해 예약 수준별로 테스트한다. `cache_control` 리터럴 4곳은 `anthropicEphemeralCacheControl()` 한 곳으로 모았다.
- **`anthropic/*` 가격 오버라이드가 내장 per-model 요금에 가려지던 문제 (#924)** — per-model 요금표가 들어오기 전에는 `anthropic/*`가 유일한 Anthropic 키였고, 운영자가 이 하나로 리셀러·정액제 요금을 표현했다. per-model 항목과 family prefix 표가 추가되면서 둘 다 오버라이드보다 먼저 조회돼, `claude-`로 시작하는 모델은 오버라이드가 전혀 닿지 않고 유령 지출이 쌓여 일/주 USD 한도를 잘못 트립시켰다. 이제 오버라이드는 내장 항목보다 **항상** 먼저 해석되며(오버라이드 안에서는 exact → provider wildcard → model wildcard 순), 내장 표는 오버라이드가 없을 때의 기본값 역할만 한다.
- **Fable 5 / Mythos 5가 Sonnet 요금으로 기록되던 문제 (#924)** — 두 모델은 `claude-opus-`/`sonnet-`/`haiku-` prefix 어디에도 걸리지 않아 Sonnet 요율 wildcard로 떨어졌고, 라인업에서 가장 비싼 모델이 3.3배 저평가됐다. $10/$50(캐시 read $1.00, write $12.50) 명시 항목을 추가했다.
- **wildcard 요금 진단이 의도된 catch-all에서도 울리던 문제 (#924)** — `gemini/*`·`gemini-native/*`는 처음부터 per-model 요금을 두지 않은 provider인데 정상 트래픽마다 "no per-model usage pricing" 경고가 찍혔다. 이제 해당 provider에 per-model 항목이 실제로 존재할 때만(= wildcard 적중이 진짜 공백일 때만) 경고하고, `*/model` 키는 provider-agnostic한 per-model 요금이므로 경고 대상에서 뺐다. 진단 로그는 tracker 뮤텍스를 놓은 뒤 방출한다.

- **프롬프트 캐시가 매 턴 무효화되던 문제 (#920)** — 시스템 프롬프트 첫 줄이 초 단위 wall-clock 타임스탬프라 어떤 프로바이더의 prefix 캐시에도 걸리지 않았고, 정적 본문 전체가 매 턴 write 요금으로 재과금됐다. 시각은 이제 프롬프트 맨 뒤 `## Current Time` 블록으로 내려가 분 단위로 truncate되고, `## Prior Context` 회상도 skills/style/goal/critic 등 세션 고정 섹션 **뒤**로 재배치된다. Anthropic 요청은 system을 블록 배열로 보내 `cache_control` breakpoint를 안정 블록에만 찍으므로 동적 tail이 캐시 prefix를 깨지 않는다. 정렬 규칙은 `prompt.BuildResultFor`에 불변식으로 문서화했다.
- **메모리 prefetch가 세션 작업 디렉터리를 지우던 문제 (#920)** — 채팅 메모리 캐시가 조립된 프롬프트를 통째로 저장하는 바람에, work dir을 모르는 prefetch 고루틴의 결과가 캐시에 올라가면 그 다음 턴 프롬프트에서 `## Working Directories` 섹션이 통째로 사라졌다. 이제 캐시는 회상 payload만 보관하고 프롬프트는 항상 live 옵션으로 다시 조립한다 — 캐시 hit과 miss가 구조적으로 동일한 프롬프트를 낸다.
Expand Down
6 changes: 6 additions & 0 deletions internal/usage/tracker.go
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,10 @@ type Tracker struct {
nowFn func() time.Time
limits Limits
priceByKey map[string]ModelPrice
// overrideByKey holds only operator-supplied rates. They are resolved
// ahead of every built-in entry, so a single `anthropic/*` override still
// covers models that ship with a per-model price.
overrideByKey map[string]ModelPrice
// warnedFallbackModels guards the wildcard-pricing diagnostic so each
// model warns once per tracker lifetime, not per call. Guarded by mu.
warnedFallbackModels map[string]struct{}
Expand Down Expand Up @@ -123,13 +127,15 @@ func NewTracker(workspaceDir string, opts TrackerOptions) (*Tracker, error) {
nowFn: nowFn,
limits: normalizeLimits(opts.InitialLimits),
priceByKey: defaultPriceTable(),
overrideByKey: make(map[string]ModelPrice, len(opts.PriceOverrides)),
warnedFallbackModels: make(map[string]struct{}),
}
for key, price := range opts.PriceOverrides {
k := strings.TrimSpace(strings.ToLower(key))
if k == "" {
continue
}
t.overrideByKey[k] = sanitizePrice(price)
t.priceByKey[k] = sanitizePrice(price)
}
_ = t.loadPersistedLimits()
Expand Down
76 changes: 63 additions & 13 deletions internal/usage/tracker_cost.go
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,11 @@ func defaultPriceTable() map[string]ModelPrice {
"anthropic/claude-sonnet-4-6": {InputPer1MUSD: 3.00, OutputPer1MUSD: 15.00, CacheReadPer1MUSD: 0.30, CacheWritePer1MUSD: 3.75},
"anthropic/claude-haiku-4-5": {InputPer1MUSD: 1.00, OutputPer1MUSD: 5.00, CacheReadPer1MUSD: 0.10, CacheWritePer1MUSD: 1.25},
"anthropic/claude-haiku-4-5-20251001": {InputPer1MUSD: 1.00, OutputPer1MUSD: 5.00, CacheReadPer1MUSD: 0.10, CacheWritePer1MUSD: 1.25},
// Fable/Mythos match none of the opus/sonnet/haiku family prefixes, so
// without explicit entries they fall to the Sonnet-rate wildcard — a
// 3.3x undercount on the priciest models in the lineup.
"anthropic/claude-fable-5": {InputPer1MUSD: 10.00, OutputPer1MUSD: 50.00, CacheReadPer1MUSD: 1.00, CacheWritePer1MUSD: 12.50},
"anthropic/claude-mythos-5": {InputPer1MUSD: 10.00, OutputPer1MUSD: 50.00, CacheReadPer1MUSD: 1.00, CacheWritePer1MUSD: 12.50},
// Fallback for gateway-hosted or unrecognized Anthropic-kind models
// (e.g. config/default.yaml routes MiniMax through kind: anthropic);
// Sonnet-class mid rates so unknown traffic still gets an estimate.
Expand Down Expand Up @@ -101,30 +106,69 @@ func (t *Tracker) EstimateCost(provider, model string, u llm.Usage) (float64, bo
return cost, true
}

// resolvePrice picks the rate for one (provider, model) pair.
//
// Operator overrides are consulted in full — exact, then provider wildcard,
// then model wildcard — before any built-in entry. Overrides are a deliberate
// statement about what this deployment is billed, so a reseller or flat-rate
// operator can still express it with a single `anthropic/*` entry even though
// the table now ships per-model Anthropic rates. Within each layer the more
// specific key wins.
//
// The fallback diagnostic is emitted after the lock is released, and only when
// falling back for a provider that does have per-model rates — for a provider
// priced by a single wildcard on purpose, a wildcard hit is the intended
// answer, not a gap.
func (t *Tracker) resolvePrice(provider, model string) (ModelPrice, bool) {
price, ok, warnProvider, warnModel := t.lookupPrice(provider, model)
if warnProvider != "" {
warnPriceFallback(warnProvider, warnModel)
}
return price, ok
}

func (t *Tracker) lookupPrice(provider, model string) (price ModelPrice, ok bool, warnProvider string, warnModel string) {
t.mu.Lock()
defer t.mu.Unlock()

p := strings.TrimSpace(strings.ToLower(provider))
m := strings.TrimSpace(strings.ToLower(model))
if p == "" || m == "" {
return ModelPrice{}, false
return ModelPrice{}, false, "", ""
}
for _, key := range []string{p + "/" + m, p + "/*", "*/" + m} {
if price, ok := t.overrideByKey[key]; ok {
return price, true, "", ""
}
}
if price, ok := t.priceByKey[p+"/"+m]; ok {
return price, true
return price, true, "", ""
}
if price, ok := matchFamilyPrice(p, m); ok {
return price, true
return price, true, "", ""
}
if price, ok := t.priceByKey[p+"/*"]; ok {
t.noteWildcardFallback(p, m)
return price, true
wp, wm := t.noteWildcardFallback(p, m)
return price, true, wp, wm
}
if price, ok := t.priceByKey["*/"+m]; ok {
t.noteWildcardFallback(p, m)
return price, true
// A `*/model` key is per-model pricing, just provider-agnostic —
// nothing is missing, so this is not a fallback worth reporting.
return price, true, "", ""
}
return ModelPrice{}, false
return ModelPrice{}, false, "", ""
}

// providerHasPerModelPrices reports whether the table carries any per-model
// entry for this provider. Callers must hold t.mu.
func (t *Tracker) providerHasPerModelPrices(provider string) bool {
prefix := provider + "/"
for key := range t.priceByKey {
if strings.HasPrefix(key, prefix) && key != prefix+"*" {
return true
}
}
return false
}

func matchFamilyPrice(provider, model string) (ModelPrice, bool) {
Expand All @@ -139,15 +183,21 @@ func matchFamilyPrice(provider, model string) (ModelPrice, bool) {
return ModelPrice{}, false
}

// noteWildcardFallback emits the fallback diagnostic at most once per model.
// Callers must hold t.mu.
func (t *Tracker) noteWildcardFallback(provider, model string) {
// noteWildcardFallback claims the once-per-model diagnostic slot and reports
// which (provider, model) the caller should warn about, or two empty strings
// when no warning is due. It never logs itself: callers hold t.mu, and the
// tracker mutex also guards Limits()/UpdateLimits, so the emission happens
// after the lock is released. Callers must hold t.mu.
func (t *Tracker) noteWildcardFallback(provider, model string) (string, string) {
if !t.providerHasPerModelPrices(provider) {
return "", ""
}
key := provider + "/" + model
if _, ok := t.warnedFallbackModels[key]; ok {
return
return "", ""
}
t.warnedFallbackModels[key] = struct{}{}
warnPriceFallback(provider, model)
return provider, model
}

func clampUsageTokens(u llm.Usage) (input int, output int, cached int, cacheRead int, cacheWrite int) {
Expand Down
114 changes: 114 additions & 0 deletions internal/usage/tracker_cost_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -205,6 +205,120 @@ func TestEstimateCost_PriceOverridesTakePrecedenceOverBuiltIns(t *testing.T) {
}
}

// An operator on a flat-rate or reseller plan expresses it with a single
// `anthropic/*` override. Before per-model pricing existed that was the only
// Anthropic key, so it covered every model; built-in per-model rates must not
// silently take that away.
func TestEstimateCost_WildcardOverrideCoversModelsWithBuiltInPrices(t *testing.T) {
tracker, err := NewTracker(t.TempDir(), TrackerOptions{
PriceOverrides: map[string]ModelPrice{
"anthropic/*": {InputPer1MUSD: 0, OutputPer1MUSD: 0, CacheReadPer1MUSD: 0, CacheWritePer1MUSD: 0},
},
})
if err != nil {
t.Fatalf("new tracker: %v", err)
}

oneMillion := llm.Usage{InputTokens: 1_000_000, OutputTokens: 1_000_000}
cases := []struct {
name string
model string
}{
{"built-in exact entry", "claude-opus-4-7"},
{"family prefix match", "claude-opus-4-9"},
{"built-in wildcard", "unknown-gateway-model"},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
got, ok := tracker.EstimateCost("anthropic", tc.model, oneMillion)
if !ok {
t.Fatalf("expected an estimate for %s", tc.model)
}
if got != 0 {
t.Fatalf("operator wildcard override must cover %s, got %v", tc.model, got)
}
})
}
}

// Claude Fable 5 and Mythos 5 match none of the opus/sonnet/haiku prefixes, so
// they land on the Sonnet-rate wildcard — a 3.3x undercount on the priciest
// models in the lineup.
func TestEstimateCost_FableAndMythosPricedAtTheirOwnRates(t *testing.T) {
tracker := newCostTestTracker(t)
for _, model := range []string{"claude-fable-5", "claude-mythos-5"} {
got, ok := tracker.EstimateCost("anthropic", model, llm.Usage{InputTokens: 1_000_000})
if !ok {
t.Fatalf("expected pricing for %s", model)
}
if got != 10.00 {
t.Fatalf("%s input should price at $10.00/1M, got %v", model, got)
}
got, ok = tracker.EstimateCost("anthropic", model, llm.Usage{OutputTokens: 1_000_000})
if !ok || got != 50.00 {
t.Fatalf("%s output should price at $50.00/1M, got %v ok=%v", model, got, ok)
}
}
}

// A provider whose table entry is only a wildcard has no per-model pricing to
// be missing, so the diagnostic is noise there.
func TestEstimateCost_WildcardOnlyProviderDoesNotWarn(t *testing.T) {
var mu sync.Mutex
warned := map[string]int{}
original := warnPriceFallback
warnPriceFallback = func(provider, model string) {
mu.Lock()
defer mu.Unlock()
warned[provider+"/"+model]++
}
defer func() { warnPriceFallback = original }()

tracker := newCostTestTracker(t)
if _, ok := tracker.EstimateCost("gemini-native", "gemini-3-pro", llm.Usage{InputTokens: 1_000}); !ok {
t.Fatalf("expected gemini wildcard estimate")
}

mu.Lock()
defer mu.Unlock()
if n := warned["gemini-native/gemini-3-pro"]; n != 0 {
t.Fatalf("wildcard-only provider must not warn, got %d warnings", n)
}
}

// The dedupe map is written under the tracker mutex; the goroutines must be the
// first callers for this model or the write path is never exercised.
func TestEstimateCost_FallbackDiagnosticDedupesUnderConcurrency(t *testing.T) {
var mu sync.Mutex
warnCount := 0
original := warnPriceFallback
warnPriceFallback = func(string, string) {
mu.Lock()
defer mu.Unlock()
warnCount++
}
defer func() { warnPriceFallback = original }()

tracker := newCostTestTracker(t)
usage := llm.Usage{InputTokens: 1_000}

var wg sync.WaitGroup
for i := 0; i < 8; i++ {
wg.Add(1)
go func() {
defer wg.Done()
_, _ = tracker.EstimateCost("anthropic", "never-seen-gateway-model", usage)
}()
}
wg.Wait()

mu.Lock()
defer mu.Unlock()
if warnCount != 1 {
t.Fatalf("expected exactly one warning across concurrent first calls, got %d", warnCount)
}
}

func TestEstimateCost_OpenAIEntriesReflectDocumentedPricing(t *testing.T) {
tracker := newCostTestTracker(t)

Expand Down
Loading