Reproduction
agent-eval 0.144.0 using llmJudge through a CLI Bridge ChatClient and a capped CostLedger.
The judge requested maxTokens 320 on model pi/tangle-router/glm-5.2. maximumChargeForLlmRequest reserved $0.0046334. The completed provider receipt reported 17,721 input tokens, 1,205 output tokens, and $0.0132836. CostLedger then raised CostReservationExceededError after the paid call.
The runner adds substantial prompt overhead that is absent from the direct request estimate. CLI Bridge also currently accepts but does not enforce max_tokens for Pi: drewstone/cli-bridge#117
Expected
A CLI-backed ChatClient needs a truthful way to provide a transport-specific per-call maximum or prompt-overhead bound to llmJudge and CostLedger. The total run budget should remain fail-closed; callers should not need to lie about retry count, inflate maxTokens, or remove the total budget.
Impact
Paid semantic evaluations through CLI Bridge cannot complete under a correct cost cap even when their actual total spend remains below that cap.
Reproduction
agent-eval 0.144.0 using llmJudge through a CLI Bridge ChatClient and a capped CostLedger.
The judge requested maxTokens 320 on model pi/tangle-router/glm-5.2. maximumChargeForLlmRequest reserved $0.0046334. The completed provider receipt reported 17,721 input tokens, 1,205 output tokens, and $0.0132836. CostLedger then raised CostReservationExceededError after the paid call.
The runner adds substantial prompt overhead that is absent from the direct request estimate. CLI Bridge also currently accepts but does not enforce max_tokens for Pi: drewstone/cli-bridge#117
Expected
A CLI-backed ChatClient needs a truthful way to provide a transport-specific per-call maximum or prompt-overhead bound to llmJudge and CostLedger. The total run budget should remain fail-closed; callers should not need to lie about retry count, inflate maxTokens, or remove the total budget.
Impact
Paid semantic evaluations through CLI Bridge cannot complete under a correct cost cap even when their actual total spend remains below that cap.