fix(dashboard): normalize model label inside sum so token timeseries stop erroring - #7
Merged
Merged
Conversation
…stop erroring "Tokens by Model x Type Over Time" and "Token Volume Over Time by Model" applied label_replace AFTER `sum by (model[, type])`. The regex strips model suffixes, so `claude-opus-4-8[1m]` collapsed onto the already-present `claude-opus-4-8`; since label_replace rewrites rather than re-aggregates, two series ended up with the same labelset and PromQL aborted the panel with `vector cannot contain metrics with the same labelset` (rendered as the red error triangle + "No data"). Latent until both a plain and a suffixed variant of the same base model coexist in the query window. Move normalization inside the aggregation using the same three-pass form the Model Comparison panel already uses (strip -YYYYMMDD, [..], leading ~), so the sum merges the variants instead of colliding. Verified against the live stack through Grafana's datasource proxy: old expr errors, new expr returns success (3 model series / 12 model x type series with real datapoints). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TrMEKd8TZniXL9v3PvVHP2
![baz-reviewer[bot]](https://avatars.githubusercontent.com/in/933528?s=60&v=4){kind=small}
Comment on lines
+541
to
544
| "expr": "sum by (model, type) (label_replace(label_replace(label_replace(rate(claude_code_token_usage_tokens_total{session_id=~\"$session.*\"}[$__rate_interval]),\"model\",\"$1\",\"model\",\"^(.*)-[0-9]{8}$\"),\"model\",\"$1\",\"model\",\"^([^[]+)[[].*$\"),\"model\",\"$1\",\"model\",\"^~(.*)$\")) * $__interval_ms / 1000", | ||
| "refId": "A", | ||
| "legendFormat": "{{model}} - {{type}}" | ||
| } |
There was a problem hiding this comment.
Normalization logic duplicated
Lines 541-544 inline the same three-pass label_replace normalization as the Model Comparison table and lines 585-587, so any regex tweak has to be updated in several places and can drift — should we factor it into a shared Grafana template/variable or reuse the existing expression?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
User description
Problem
The Tokens by Model x Type Over Time and Token Volume Over Time by Model panels showed the red error triangle + No data. The real error was:
Both queries applied
label_replaceaftersum by (model[, type]). The regex strips model-name suffixes, soclaude-opus-4-8[1m]collapsed onto the already-presentclaude-opus-4-8. Becauselabel_replacerewrites labels rather than re-aggregating, two series ended up with an identical labelset and PromQL aborted the whole panel. Latent until both a plain and a suffixed variant of the same base model ([1m],-YYYYMMDD, leading~) coexist in the query window.Fix
Move normalization inside the aggregation, using the same three-pass form the Model Comparison panel already uses (strip
-YYYYMMDD, strip[..], strip leading~).sumnow merges the variants instead of colliding — and correctly combines their token counts.Verification
Ran both old and new queries through Grafana's datasource proxy against the live stack (Grafana 13.0.1):
status: error … same labelsetstatus: success— 3 model series (panel 25) and 12 model×type series (panel 10) with real datapoints.Confirmed rendering in the running Grafana after a provisioning reload.
🤖 Generated with Claude Code
Generated description
Below is a concise technical summary of the changes proposed in this PR:
Move label normalization into the
sum byaggregation so Prometheus merges suffixed model variants instead of aborting. Apply the same three-pass label stripping used by Model Comparison to both tokens panels’ Prometheus queries.Latest Contributors(0)