feat(budget): cap what an agent may spend in money, tokens, or working time#3702
Open
dwin-gharibi wants to merge 20 commits into
Open
feat(budget): cap what an agent may spend in money, tokens, or working time#3702dwin-gharibi wants to merge 20 commits into
dwin-gharibi wants to merge 20 commits into
Conversation
… to define amazing budgets
…tor for docker agent
…level config keys
…ent core into runtime of docker aagent
…re and wiring into agent loop and session
…er agent core components like agent loops and runtime and client
…events and exceptions
… into tui sidebar to announcing them realtime
…zing tests for budget lines in tui
…into docker agent cmd runner
…nnouncing new config
…ple into docker agent
…new configuration
Contributor
Author
Contributor
Author
|
@Sayt-0 Deps fixed. |
Member
|
#3714 already cover a part of this pr, but I like the idea of budget limit |
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds budgets that stop an agent once it crosses a configured ceiling, and tracks every budget live in the TUI, broken down per agent.
Every limit is optional, unset means unlimited, and declaring nothing leaves runs unbudgeted. Existing configs are unaffected - the feature is inert unless opted into.
The named form follows the existing top-level
mcps/rag/toolsetsreference-by-name convention. An agent may list several budgets; all apply, and the first exhausted stops the run.Closes #3701.
Why this needs runtime code
The runtime already prices every response (
computeMessageCost) and hands that exact figure toafter_llm_callhooks, which the docs suggest using for "a sidecar cost ledger".But a ledger can only record.
executeAfterLLMCallHooks(hooks.go) discards the hookResult- both callsites invoke it as a bare statement, andafter_llm_call_test.golocks that in.Continue: falseis only honored inexecutor.go, which the event never reaches. So spend can be watched but not acted on; enforcement has to live in the loop.This PR adds no new pricing math. It reuses the value the session already bills, so the ceiling can never disagree with the invoice.
Design decisions
Each of these is a place a naive implementation is silently wrong, so they're called out for review rather than buried.
A budget name is one shared pot.
rootandresearcherabove both referenceshell-work, so together they cannot exceed$0.03. Per-agent copies would let a run spendmax_cost× N by fanning out to N sub-agents. Distinct names give independent pots. Sub-sessions likewise share the root's wallets - they run on the sameLocalRuntime(loop.goalready computesrootStream := !sess.IsSubSession()). This intentionally differs frommax_iterations, where per-child caps (agent_delegation.go) are correct: money doesn't behave like iterations.A budget spans the session, not a message.
RunStreamruns once per user message, so building the trackers there would silently redefinemax_costas "per message" - a session could re-spend the whole ceiling on every turn.ensureBudgetbuilds once and leaves them alone. AbudgetStartedflag distinguishes "not built yet" from "built, nothing to budget", so unbudgeted runs don't rebuild every message.max_timemeasures working time, not wall-clock. Follows directly from the above: a session sits idle while the user reads and types, so wall-clock would let a budget expire during a coffee break - leave the TUI open ten minutes and the next message instantly tripsmax_time: 2m. It accumulates turn durations instead. This also made the tracker simpler: no start timestamp, no clock.max_tokensuses its own accumulator, notsession.Usage().ApplyCompactionresets the session counters because they measure context length, not spend. Reading them would makemax_tokenssilently stop working on precisely the long runs it exists for.Unpriced models can't be enforced against, and say so.
computeMessageCostreturnsnilfor a model with no pricing table. There's no honest number to add, so it can't count towardsmax_cost. Failing closed would break local/custom-endpoint users; counting$0would read reassuringly low precisely because it's incomplete. Instead: a one-shot warning, an(unpriced spend)marker in the TUI, and docs pointing at the model-levelcost:block.Terminal, not resumable. No "continue?" prompt, unlike
max_iterations. A budget is a ceiling set on purpose; offering to raise it defeats it. This also keepsresumeChanand the headless consumers (chatserver,embeddedchat) untouched.Boundary-checked. Checked between turns beside
enforceMaxIterations, so a run overshoots by at most the turn in flight andmax_timewon't interrupt an in-flight call. Documented explicitly rather than left for users to discover.Changes
Config
BudgetConfig,Config.Budget(run-wide),Config.Budgets(named map),AgentConfig.Budgets(name list).agent-schema.json:budgetvia$ref, abudgetsmap, and the agent-levelbudgetsarray. Plus an HCL block rule sobudgets "tight" { ... }maps correctly.max_timereuses the existinglatest.Durationtype, so10m/30s/1h30mall work.CloneThroughJSON), so additive fields arrive as zero values.Versionstays"12".Runtime
pkg/runtime/budget.go:budgetTracker(one per budget name) andbudgetSet(the run-wide budget plus each agent's named budgets). Nil-safe throughout, so the loop needs no nil checks.enforceMaxIterations; newturnEndReasonBudgetExceededso a budget-killed run is distinguishable from a completed one in telemetry.budget_usage(oneBudgetStatusper active budget, each with a per-agent breakdown) andbudget_exceeded(carryingbudget,limit,used,max,config_path), registered inclient.gofor wire decoding.notificationhook rather than addingon_budget_exceeded-budget_exceededalready carries the structured payload. Easy to add later if asked.TUI
budget_exceededsurfaces as a warning, not a dialog (nothing to ask).Docs / example
docs/configuration/budget/index.md(weight 75, no collision), covering shared pots, the unpriced-model caveat, boundary granularity, and that budgetmax_tokens≠ modelmax_tokens.examples/budget.yaml- OpenRouter viabase_urlwith an explicitcost:block (somax_costis enforceable on a custom endpoint), two agents sharing one named budget, and deliberately low limits so the stop is easy to see.Verification
The Go CI jobs don't appear to run on this branch, so I ran the repo's own linters and tests locally. The
go 1.26.5toolchain is unfetchable here (403 fromproxy.golang.org), so tests ran against cachedgo1.26.4via a local-only-modfile;go.modis untouched.golangci-lint run(v2.12.2, repo's own.golangci.yml) over every touched package → 0 issuesgo test ./pkg/runtime/ ./pkg/config/...→ passing, including the schema drift guards (TestSchemaMatchesGoTypes,TestHCLBlockRulesCoverSchemaMaps,TestJsonSchemaWorksForExamples,TestDocYAMLSnippetsAreValid)go test -raceon the shared trackers → clean (sub-sessions and background agents record concurrently)markdownlint-cli2@0.22.1fromdocs/→ 0 errors, 111 filesagents.root: budgets: unknown budget "nope"), and a negative limit is rejected (budgets.shell-work: max_cost must not be negative, got -3).Things I verified are load-bearing rather than assuming:
"budget"/"budgets"totopLevelConfigKeys(doc_yaml_test.go) - without it, doc snippets containing them are silently skipped rather than validated. Proved by breaking a snippet:schema errors: [budget.max_cost: Invalid type. Expected: number, given: string]; restored → passes.$reffor thebudgetproperty rather than inlining.TestHCLBlockRulesCoverSchemaMapsdoesn't resolve refs and typesAdditionalPropertiesasany, so an inlined"additionalProperties": falsewould demand a nonexistent HCL rule.budgetsmap genuinely needed an HCL rule - the drift guard caught its absence and failed until one was added.Pre-existing failures unrelated to this PR:
pkg/runtime'sTestListGatewayModels_*/TestAvailableModels_Gateway*fail identically on a cleanorigin/mainworktree with none of this code present (verified, not assumed).pkg/tui/components/sidebartests cannot build under the-modfileworkaround (it resolves acellbufincompatible with the pinnedansi); the pre-existing sidebar tests fail the same way, so it's environmental - the package compiles and those tests will run in CI on go 1.26.5.Try it
Watch the sidebar track each budget until the run stops.