feat(ai-gateway): add project usage governance

[dviejokfs]

Summary

• attribute deployment-token AI usage to project, environment, deployment, and token IDs
• expose project/environment/token filters across AI usage APIs
• add operator-only instance, project, environment, and token governance policies for model allowlists, RPM, and monthly cost
• enforce limits across console nodes with PostgreSQL advisory locks, shared rate events, and atomic cost reservations
• convert unresolved streamed requests into durable conservative debits while releasing explicit upstream failures
• keep multimodal, streaming, and month-boundary accounting fail-closed, including conflicting OpenAI output-cap aliases
• frontend: a Governance tab on the AI Gateway pages to view spend and configure per-scope (instance/project/environment) model allowlists, RPM limits, and monthly budgets
• CLI: bunx @temps-sdk/cli ai governance list/set/unset for the same, hand-written against the plugin route per this repo's CLI-parity conventions (governance is served by the temps-ai-gateway plugin, so it's intentionally excluded from the CLI's committed OpenAPI spec)
• docs: ai governance documented in the temps-cli skill's command reference and routing table, with a pointer from temps-best-practices

Policies apply only from trusted deployment-token attribution. BYOK requests still obey allowlists and RPM limits but do not consume operator-funded monthly budgets. Internal control-plane AI calls remain outside this deployment-token HTTP boundary. Governance policies are operator-only: setting/viewing limits for any scope (including a project you own) requires an instance administrator, since a policy affects every request in that scope, not just the caller's own usage.

Closes #376

Note on this branch's history

This branch was originally built against an older commit and has since been reapplied against current main — temps-ai-gateway's gateway.rs, pricing.rs, providers.rs, and usage.rs were substantially refactored in the interim, so the governance logic was hand-reintegrated rather than merged, with a fresh migration timestamped after main's newer migrations. The original 2-commit history is preserved for reference on backup/ai-gateway-project-governance-orig in the working tree that produced this push.

Review pass and fixes applied

A full review-pr pass (Rust standards, security-auditor, migration safety, frontend/SDK, test coverage, evidence) found two blocking issues, since fixed:

• Migration/runtime bug: ai_gateway_rate_events was missing the request_id column that check_rates_and_record inserts into — RPM enforcement would 500 on every fresh deployment as soon as any scope had an RPM limit configured. No prior test exercised that INSERT, so it went uncaught. Fixed the migration and added an integration test (rpm_limit_rejects_after_threshold) that exercises the exact path.
• Security: all three governance handlers gated only on AiGatewayWrite, which Role::User also holds for their own projects' day-to-day AI usage — meaning a regular user could tamper with instance-wide limits or another project's governance policy by naming its scope. Added an explicit operator-only check (admin/platform-admin) to list/upsert/delete, matching the "operator-only" design intent stated above.

Also fixed: a negative RPM/budget value returned 500 instead of the documented 400 (status-code mapping bug), a stale rate-event cleanup window (comment said 1 minute, query used 1 hour), and closed three test-coverage gaps that had zero assertions on the actual enforcement behavior (RPM rejection, budget rejection, expired-reservation-to-conservative-debit conversion) plus CLI helper unit tests and a project_id/environment_id filter-binding test.

Out of scope, flagged as a separate follow-up: the security pass also found BYOK's x-provider-base-url header has no SSRF validation (private/internal IP ranges aren't rejected) — this predates this PR (present on main before the reapply) and isn't part of this diff; worth a dedicated fix.

Evidence

AI governance lifecycle and concurrency (real PostgreSQL/TimescaleDB)

cargo test -p temps-ai-gateway --test governance_integration -- --nocapture

cargo test: 12 passed

(includes the 3 new tests for RPM rejection, budget rejection, and expired-reservation conversion)

Gateway behavior and regression coverage

cargo test --lib -p temps-ai-gateway

cargo test: 234 passed

Migration applied / rollback (real TimescaleDB)

cargo test -p temps-migrations --test migration_tests test_migration_down -- --exact --nocapture

cargo test: 1 passed

Repository gates

cargo check --lib
$(command -v cargo) clippy -p temps-ai-gateway --lib --tests -- -D warnings
$(command -v cargo) clippy -p temps-migrations --lib --tests -- -D warnings

cargo check: 0 errors (60 crates)
cargo clippy: 0 warnings (both crates)

Frontend

cd web && bunx tsc --noEmit && bunx eslint <changed files>

tsc: clean
eslint: 0 errors, 0 warnings

Live-verified in-browser against a local dev instance: logged in, opened /ai-gateway/governance, saved RPM=42 + $12.50 monthly budget for the instance scope (PUT /api/ai/governance/instance -> 200), reloaded and confirmed persistence, removed limits (DELETE -> 204) and confirmed revert to unlimited, exercised the project/environment scope selector and the model-allowlist tri-state multi-select. No console errors. Scoped spend display verified against the project_id/environment_id filters newly wired into /ai/usage/summary.

CLI

cd apps/temps-cli && bun run typecheck && bun test src/commands/ai/governance.test.ts

tsc: clean
bun test: 24 passed

Live-verified against a local dev instance: ai governance set instance --rpm 100 --monthly-budget 50.00 -> ai governance list (table shows RPM 100 / $50.00) -> ai governance unset instance --yes -> ai governance list (empty again). Also verified --models <list>, --models none (block-all), and --json output.

Rust review, security re-review, migration/test review: APPROVE (post-fix, this session).

Second review-pr pass and fixes (this session)

A fresh /review-pr pass (Rust standards, security-auditor, migration safety, frontend/SDK, test coverage, evidence — all run independently, with findings verified against the actual diff before being trusted) found 3 blocking, 4 major, and 3 minor issues that survived the first review-pr pass above. All are fixed in c17c43722 and 76158dd44:

Blocking:

• usage_service.rs's reservation-cleanup DELETE cast its bound param to ::uuid against a varchar(64) column — Postgres has no implicit varchar↔uuid comparison, so this raised a runtime error on every successful governed request, rolling back the accompanying usage-log insert. Cast removed.
• release_cost_reservation deleted a reservation row unconditionally by request_id, with no guard against rows already promoted to a durable "conservative debit" by the expired-reservation cleanup job — a slow request that later failed upstream could silently erase spend the mechanism was built to make durable. Added an AND is_conservative_debit = FALSE guard.
• Embeddings requests always passed max_output_tokens = None into the budget-projection check, which unconditionally requires a Some value once any budget scope applies — the instant an operator configured any budget covering embedding traffic, every non-BYOK embedding request 400'd. Now passes Some(0) (embeddings have no output tokens by definition), letting projection proceed on input cost alone.

Major:

• MonthlyBudgetExceeded's HTTP-facing message included exact spent_microcents/limit_microcents, leaking the operator's configured budget and running spend to any deployment-token caller probing near the ceiling. Amounts stripped from the response; moved to a structured warn! log instead.
• ai_gateway_cost_reservations had a single-column PK on request_id, but the code inserts one row per applicable budget-limited scope sharing the same request_id — any deployment with two simultaneously active budget scopes (e.g. instance + project) hit a PK violation on the second insert. Migration changed to a composite (request_id, scope) primary key.
• No index existed on ai_usage_logs's new project_id/environment_id/deployment_token_id columns despite them being filtered in the budget roll-up query — added (billing_period, *) composite indexes matching the query shape.
• The operator-only gate (require_operator) and the AiGatewayWrite permission gate had no HTTP-layer test proving they actually reject a non-admin caller — added handler-level tests for all three governance endpoints, plus a frontend fix so the UI's "insufficient permissions" onboarding state recognizes require_operator's distinct problem title (previously only matched permission_guard!'s title; both now matched via the shared RFC 7807 problem type URI).

Minor: added .if_not_exists() to the migration's index creations, added a CHECK (reserved_microcents >= 0) constraint, and added a regression test locking in the stream_options.include_usage overwrite-caller-value behavior (intentional, to prevent billing bypass).

Re-verified evidence after the fix (this session, real Docker/Postgres/TimescaleDB, not asserted):

cargo check --lib -p temps-ai-gateway -p temps-migrations

cargo build: 0 errors, 2 warnings (28 crates) — warnings are pre-existing/unrelated
                                                  (workspace profile config, transitive dep)

cargo test --lib -p temps-ai-gateway

cargo test: 242 passed

$(command -v cargo) clippy -p temps-ai-gateway -p temps-migrations --lib --tests -- -D warnings

0 warnings

cargo test -p temps-ai-gateway --test governance_integration -- --nocapture

cargo test: 12 passed

cargo test -p temps-migrations --test migration_tests test_migration_down -- --exact --nocapture

cargo test: 1 passed

Known follow-ups, not blocking: no test exercises the PostgreSQL advisory-lock claim under real concurrency (two simultaneous check_request calls against the same scope) — the cross-node correctness invariant is currently proven only by code inspection. Separately, lock_scopes always includes the "instance" scope in its lock set, so any instance-wide RPM/budget policy serializes all governed traffic platform-wide on one advisory lock for the full check-and-insert transaction — worth a load test before recommending instance-wide governance as a default. Both flagged for a fast-follow, not this PR.

Rebase onto main + SDK regen (this session, commit 4632af563)

main had moved 11 commits since this branch's base, causing real merge conflicts (not stale-status noise): a migration registration conflict in crates/temps-migrations/src/migration/mod.rs (main added m20260816_000001_add_image_retention_hours, sharing this PR's date/sequence stamp — different table, no schema collision, just an ordering conflict) and generated-SDK conflicts in web/src/api/client/{sdk.gen.ts,index.ts,@tanstack/react-query.gen.ts} (main added unrelated endpoints — image retention, deployment media, project health summary — that regenerated the same files).

Resolved per this repo's rule for generated-client conflicts (never hand-merge): rebased, registered the governance migration last in Migrator::migrations() with a comment explaining the shared-stamp ordering, took one side to clear the generated-file conflicts, then regenerated the SDK in full against a temps serve instance built off the merged source (minted a temporary local admin API key for the auth-gated /api/api-docs/openapi.json endpoint, regenerated, immediately revoked the key).

Re-verified after the rebase (real Docker/Postgres/TimescaleDB, this session):

cargo check --lib   # workspace-wide, not just the two changed crates

cargo build: 0 errors, 71 crates

cargo test --lib -p temps-ai-gateway -p temps-migrations

cargo test: 249 passed

$(command -v cargo) clippy -p temps-ai-gateway -p temps-migrations --lib --tests -- -D warnings

0 warnings

cargo test -p temps-ai-gateway --test governance_integration -- --nocapture

cargo test: 12 passed

cargo test -p temps-migrations --test migration_tests test_migration_down -- --exact --nocapture

cargo test: 1 passed

cd web && bunx tsc --noEmit

clean, 0 errors

cd web && bunx eslint src/pages/AiGateway.tsx

0 errors, 0 warnings

PR mergeability confirmed clean post-rebase (gh pr view 543 --json mergeable → MERGEABLE, was CONFLICTING before this rebase).

Dedicated security-auditor pass + fixes (this session)

A dedicated, independent security review (beyond the standard review-pr gate) found two more disclosure issues in the same class as the already-fixed budget-amount leak, both fixed in f0daf9dc9:

• RateLimitExceeded — the 429 response body included the exact operator-configured limit_per_minute, letting any deployment-token caller learn the RPM cap by deliberately triggering it. Stripped from the response; the value is now only logged server-side. The already-computed retry_after_seconds (previously silently dropped) is now surfaced properly as a Retry-After header.
• InvalidGovernanceConfig — its Display embeds the offending field name and value from a corrupted/invalid stored governance config row (reachable via DB corruption or a direct manual write, since the write path itself validates non-negativity), and was forwarded verbatim into a 500 response body. Replaced with a generic message; details logged server-side.

Both fixes have new regression tests (test_rate_limit_exceeded_response_does_not_leak_configured_limit, test_invalid_governance_config_response_does_not_leak_field_or_value) asserting the response body does not contain the sensitive values.

Also confirmed still correctly fixed (re-verified independently, not just re-asserted): the operator-only gate on all three governance handlers, the embeddings budget-DoS fix, and that the BYOK x-provider-base-url SSRF gap remains genuinely untouched by this diff.

Filed as a separate tracked issue, not fixed in this PR: gotempsh/temps#705 — /ai/usage/summary and /ai/usage/by-provider have no ownership scoping at all, predating this PR (any AiGatewayRead holder — the default for Role::User — can already read any project's usage; this PR only added a project_id/environment_id filter on top of an already-unscoped query, making it easier to target but not introducing the underlying gap). Triaged the same way this repo already handles the analogous proxy-logs IDOR (#403): tracked separately rather than scope-creeping this PR.

Re-verified after these fixes (real Docker/Postgres, this session):

cargo test --lib -p temps-ai-gateway

cargo test: 244 passed

$(command -v cargo) clippy -p temps-ai-gateway --lib --tests -- -D warnings

0 warnings

cargo test -p temps-ai-gateway --test governance_integration -- --nocapture

cargo test: 12 passed

CI fix: skill docs generation drift (this session)

The initial push after the rebase failed CI's Scan skills/temps job — not a malicious-content flag, but a generated-file drift check: skills/temps/references/commands/ai.md is generated from skills/temps-cli/references/COMMANDS.md via skills/temps/scripts/generate_command_references.py, and this PR's earlier docs commit updated the source but never regenerated the derived file. Also caught by the same job: the new command's example used an unpinned @temps-sdk/cli reference, failing validate_skill.py's pinned-version check. Both fixed in 1ba0fdec8 — pinned the example to @temps-sdk/cli@0.1.33 and regenerated ai.md; local drift check and the skill's own unit tests now pass clean.

[github-actions]

📓 Changelog preview

This is what your commits will add to the generated CHANGELOG.md at release time (via git-cliff). Do not edit CHANGELOG.md by hand — it is generated from your Conventional Commit messages.

## [Unreleased]

### Added

- **ai-gateway:** Reapply project usage governance on current main
- **cli:** Add AI gateway governance commands
- **web:** Add AI gateway governance tab for cost limits

### Documentation

- **skills:** Document AI gateway governance CLI commands
- **skills:** Pin CLI version and regenerate command references

### Fixed

- **ai-gateway:** Wire project/environment filters into usage summary endpoints
- **web:** Scope AI gateway governance spend display to the selected scope
- **ai-gateway:** Fix review findings in governance policy enforcement
- **web:** Match governance permission errors by problem type, not title
- **ai-gateway:** Correct 10 governance bugs found in PR #543 review
- **ai-gateway:** Stop leaking governance config in error responses

### Miscellaneous

- **web:** Regenerate SDK client for AI gateway governance routes
- **web:** Regenerate SDK client for scoped usage summary params
- **web:** Regenerate SDK client after rebase onto main