Skip to content

feat: add trustworthy model-level token controls - #1

Merged
LeemanCheung merged 30 commits into
mainfrom
feat/token-usage-high-value-insights
Aug 28, 2026
Merged

feat: add trustworthy model-level token controls#1
LeemanCheung merged 30 commits into
mainfrom
feat/token-usage-high-value-insights

Conversation

@LeemanCheung

Copy link
Copy Markdown
Owner

Summary

  • add exact date × provider/model projection, reliable model-filtered trends, and export-v3 coverage metadata
  • add persistent per-route 30-day Token budgets with 80% warning, actual overage, and seven-day run-rate forecast states
  • gate route analytics and CSV export on session/model/day conservation; keep zero-token count-only routes valid
  • harden catalog timeout recovery, analysis stream termination, RPC validation, cancellation, and CSV formula neutralization
  • refresh committed GitHub-installable bundles, documentation, and package version to 0.2.0

Verification

  • ..\deepseek-harness\node_modules\.bin\vitest.cmd run — 19 files, 116 tests passed
  • host and client tsc --noEmit — passed
  • tsdown --config tsdown.config.ts — host/client bundles built
  • git diff --check and git diff --cached --check — passed
  • npm pack --dry-run --json — 0.2.0 payload validated

Notes

  • JSON export advances to dsh-token-usage/export-v3.
  • Route budgets remain passive local observability; they never block or reroute model requests.
  • USD budgets remain deferred until fee catalog provenance and user overrides are versioned.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 09593568a5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/projection.ts
total = addBuckets(total, usage, 1)
adjustModel(models, route, usage, 1, 'assistant')
adjustDay(days, day, usage, 1)
adjustModelDay(modelDays, route, day, usage, 1)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Reattribute identical samples when their UTC day changes

When an assistant/chunk usage sample arrives just before UTC midnight and the final assistant/message arrives afterward with the same route and bucket values, the existing equality fast path returns before this new adjustModelDay call. The newly exported date-by-model record therefore remains on the earlier day, and the conservation checks still accept it because the ordinary daily record is stale in exactly the same way. Include the day in the equality check so the final event moves both aggregates to its authoritative reporting date.

Useful? React with 👍 / 👎.

Comment on lines +739 to +741
const configurableModels = useMemo(
() => models.filter(model => !isUnattributed(model) && totalTokens(model.usage) > 0),
[models],

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Exclude fallback identities from exact-route controls

For assistant usage recorded before any request route is available, assistantRoute stores the fallback identity unknown/unknown; isUnattributed only recognizes the empty-string legacy fallback, so this filter exposes unknown/unknown as a configurable exact provider/model budget (and the analogous trend filter exposes it as an exact model). Such a budget pools unattributed calls rather than controlling a real route and can report a misleading healthy or exceeded state. Treat the projection's fallback identity as unattributed or otherwise require an authoritative route before offering these controls.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Owner Author

@codex address the two unresolved review threads and update this PR branch. Please make the fixes narrowly and add regression coverage:

  1. In src/projection.ts, the same turn/step sample may cross UTC midnight between provisional usage and the authoritative final message. The equality fast path must include previous.day === day; otherwise remove and re-add the sample so both days and modelDays move to the authoritative date. Add a projection test covering identical buckets/route across midnight.
  2. Treat both the legacy empty route and the projection fallback unknown/unknown as unattributed. Do not expose fallback identities in exact-route trend filters or route-budget configuration. Add component/aggregation regression coverage.
  3. Add a standalone GitHub Actions CI workflow for pull requests and main using Node 22.19.0. It must install dependencies, run host/client typechecks, all Vitest tests, rebuild host/client bundles, verify committed lib/ output is clean, and run npm pack --dry-run --json. Add only the development dependencies required to make those scripts reproducible outside the sibling DeepSeek Harness checkout; keep the existing DSH peer ranges and runtime behavior unchanged.
  4. Rebuild and commit lib/index.js, lib/client.js, and source map changes, update the PR verification summary/counts, and leave a concise note mapping each fix to its regression test.

@chatgpt-codex-connector

Copy link
Copy Markdown

To use Codex here, create an environment for this repo.

Copy link
Copy Markdown
Owner Author

@codex address that feedback

请同时补充两类回归测试:

  1. 同一 turn/step、相同 route 和相同 usage 的 chunk 在 UTC 零点前、final message 在零点后时,最终 days/modelDays 必须移动到最终消息日期;
  2. 缺少权威路由的 usage 不得进入精确路由趋势或路由预算候选项,unknown/unknown 必须按未归因处理。
    完成后请运行现有测试、host/client typecheck 与 build,并把结果写入 PR。

@chatgpt-codex-connector

Copy link
Copy Markdown

To use Codex here, create an environment for this repo.

@LeemanCheung
LeemanCheung merged commit 28f5ffc into main Aug 28, 2026
0 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant