Skip to content

fix(accounting): distinguish observed and estimated model cost #732

Description

@drewstone

Failures reproduced

  1. Pi through cli-bridge returned 7,579 fresh input tokens, 8,832 cached input tokens, 2 output tokens, and usage.cost.total: 0 because its local model catalog has zero rates. cli-bridge treated that numeric zero as complete, and Runtime treated it as known cost.
  2. Runtime direct Router clients derive dollars from the local agent-eval price table and currently expose that estimate as known cost. A stale table or provider-specific cache price can therefore enforce a dollar cap against the wrong amount.
  3. Several unmetered executor paths deliberately leave $0 looking known.

Required behavior

  • Carry cost provenance from every model adapter: provider/billing receipt, local estimate, or uncaptured.
  • A local catalog or price-table estimate may remain visible as an estimate but may not set Runtime usdKnown true.
  • Missing or untrusted cost is never represented as measured zero.
  • Preserve fresh input, cache-read, cache-write, output tokens, and their known/unknown state through cli-bridge and Runtime.
  • Use a trusted complete provider receipt, including cache pricing, when available.
  • A caller dollar cap refuses work whose complete dollar amount cannot be proven; an uncapped run records unknown without inventing failure or zero.
  • Apply the rule to every Runtime executor and driver, not one model or provider.

Completion proof

  • The reproduced Pi zero-catalog case reports tokens/cache but usdKnown:false.
  • A nonzero local estimate remains labeled estimated and still has usdKnown:false.
  • A complete provider receipt reports observed cost and usdKnown:true.
  • Missing usage remains unknown.
  • Dollar-capped tests refuse unknown/estimated cost.
  • Static audit covers every Spend construction and no unmetered provider path silently emits known zero.
  • Full Runtime and cli-bridge suites pass.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions