Skip to content

Expand cost pricing regression coverage - #249

Merged
tsouth89 merged 3 commits into
mainfrom
test/sbs-557-cost-pricing
Aug 10, 2026
Merged

Expand cost pricing regression coverage#249
tsouth89 merged 3 commits into
mainfrom
test/sbs-557-cost-pricing

Conversation

@tsouth89

@tsouth89 tsouth89 commented Aug 10, 2026

Copy link
Copy Markdown
Owner

Summary

  • validate every Codex and Claude pricing-table entry resolves with structurally valid rates
  • pin separate input, output, cache-read, and cache-write arithmetic
  • cover cache clamping, Claude tier boundaries, dated Sonnet 5 rates, zero/negative usage, and unknown models
  • keep the discovered gpt-5.1-codex-mini overcharge fix separate in SBS-710

Closes SBS-557.

Validation

  • cargo test --manifest-path rust/Cargo.toml core::cost_pricing::tests (29 passed)
  • cargo test --manifest-path rust/Cargo.toml cli::cost::tests (2 passed)
  • cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
  • cargo fmt --all --manifest-path rust/Cargo.toml

Summary by CodeRabbit

  • Tests
    • Added comprehensive coverage for pricing calculations across Codex and Claude models.
    • Verified model normalization, token channels, cache limits, threshold handling, date-based rates, and edge cases such as zero, negative, and unknown usage.

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
ceiling 68fb46a Commit Preview URL

Branch Preview URL
Aug 10 2026, 12:19 AM

@coderabbitai

coderabbitai Bot commented Aug 10, 2026

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in: 44 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: c21fb030-b926-41d2-b449-7c2d06680ebb

📥 Commits

Reviewing files that changed from the base of the PR and between ae85cf3 and 68fb46a.

📒 Files selected for processing (1)
  • rust/src/core/cost_pricing_tests.rs
📝 Walkthrough

Walkthrough

Changes

Pricing Test Coverage

Layer / File(s) Summary
Pricing helpers and table validation
rust/src/core/cost_pricing_tests.rs
Added shared rate comparison helpers and validation for Codex and Claude pricing entries, aliases, rates, and model reachability.
Token channel and cache pricing
rust/src/core/cost_pricing_tests.rs
Tested independent token pricing for Codex and Claude. Added cached-input clamping coverage.
Rate boundaries and edge cases
rust/src/core/cost_pricing_tests.rs
Tested Claude thresholds, Sonnet 5 date-based rates, zero and negative usage, and unknown models.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

  • tsouth89/ceiling#96: Introduced Codex/Claude token normalization and cache-clamping behavior covered by these tests.
  • tsouth89/ceiling#158: Introduced pricing behavior covered by the new rate, threshold, date, and cache tests.
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the primary change: expanded regression coverage for cost pricing.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch test/sbs-557-cost-pricing

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@rust/src/core/cost_pricing_tests.rs`:
- Around line 56-61: In rust/src/core/cost_pricing_tests.rs:56-61, extend the
Codex validation around the existing rate loop to validate every rate in
pricing.long_context when present, requiring finite and nonnegative values. In
rust/src/core/cost_pricing_tests.rs:76-85, before comparing Claude aliases,
validate every base and above-threshold rate for finiteness and nonnegativity,
and require threshold_tokens to be nonnegative; cover optional above-threshold
rates when present.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: bb196281-04fd-45e6-b84f-eb1584bd0726

📥 Commits

Reviewing files that changed from the base of the PR and between c200369 and ae85cf3.

📒 Files selected for processing (1)
  • rust/src/core/cost_pricing_tests.rs

Comment thread rust/src/core/cost_pricing_tests.rs Outdated
@github-actions

Copy link
Copy Markdown

Automated review

Found 7 issues:

  1. Test contradicts its own comment: known gpt-5.1-codex-mini normalization bug is asserted to resolve

    rust/src/core/cost_pricing_tests.rs:63 · severity: high · quick win

    every_codex_table_entry_has_valid_rates_and_resolves_to_a_price iterates CODEX_PRICING and asserts both contains_key(normalize_codex_model(model)) and codex_cost_usd(model, 0, 0, 0).is_some(), while the comment directly above admits gpt-5.1-codex-mini's normalization bug (SBS-710) is unfixed. When the loop reaches that entry, either normalize returns a missing key and the test panics on the base branch (red CI), or normalize returns a different existing key and the all-zero token call short-circuits to Some(0.0) before any price resolution, so the assertion passes without verifying the model resolves to a price. Either way the test cannot both pass and validate the title's claim for the one model the comment says is broken.

    Prompt for AI agents

    In rust/src/core/cost_pricing_tests.rs around line 63: Replace the zero-token check with codex_cost_usd(model, 1, 1, 1) and assert the cost equals 1input + 1cache_read + 1*output from the table entry (or at least is Some with a nonzero difference when rates are nonzero); until SBS-710 is fixed, explicitly skip gpt-5.1-codex-mini with a documented panic-free assertion, and add a test pinning the current buggy normalization so the bug has a failing test to drive the fix. Verify against the current code first; if no longer valid, skip with a brief reason. Keep the change minimal.

  2. Claude 'reachable' check runs with four zero channels and a hardcoded 2026-08-01 date

    rust/src/core/cost_pricing_tests.rs:83 · severity: medium · quick win

    every_claude_table_entry_is_reachable_and_aliases_match calls claude_cost_usd_on_date(model, 0, 0, 0, 0, NaiveDate::from_ymd_opt(2026, 8, 1)). With all token channels zero the routine can return Some(0.0) without resolving the model, so this assertion adds no reachability evidence beyond the normalize/get on lines 78-80; and because the date is a fixed literal, the test fails as soon as the table gains an entry whose rates become effective after 2026-08-01. The test therefore does not deliver the reachability guarantee its name promises and is fragile to dated table entries.

    Prompt for AI agents

    In rust/src/core/cost_pricing_tests.rs around line 83: Pass nonzero token counts (e.g., 1, 1, 1, 1) so the model lookup is actually exercised, and derive the date from the earliest effective date present in CLAUDE_PRICING (or loop over each entry's effective-date range) instead of hardcoding 2026-08-01. Verify against the current code first; if no longer valid, skip with a brief reason. Keep the change minimal.

  3. SBS-710 known bug makes every_codex_table_entry test fail on current code

    rust/src/core/cost_pricing_tests.rs:65 · severity: medium · quick win

    The comment on lines 63-64 acknowledges gpt-5.1-codex-mini normalizes to a missing key, but line 65 still asserts codex_cost_usd(model,0,0,0).is_some() for every entry in CODEX_PRICING. If the table contains that model and normalize_codex_model is buggy, the lookup returns None and the test fails, contradicting the claim this is a tests-only PR with no behavior fix.

    Prompt for AI agents

    In rust/src/core/cost_pricing_tests.rs around line 65: Skip or expect the known-bad model explicitly, e.g. if model == "gpt-5.1-codex-mini" { continue; } with a TODO referencing SBS-710, and add a test that the fix will flip to is_some. Verify against the current code first; if no longer valid, skip with a brief reason. Keep the change minimal.

  4. assert_claude_rates_equal uses exact equality for Option premium rates

    rust/src/core/cost_pricing_tests.rs:26 · severity: medium · quick win

    Lines 26-42 compare input_cost_per_token_above_threshold and the other *_above_threshold Option fields with assert_eq!. Bit-identical equality is brittle for f64; a future pricing literal or computed rate that differs by 1 ULP will spuriously fail, while the non-threshold rates correctly use assert_close. Trigger: any pricing update where the premium rate is not exactly representable.

    Prompt for AI agents

    In rust/src/core/cost_pricing_tests.rs around line 26: Compare the Option premium rates with assert_close when Some, or add a helper assert_option_close, instead of assert_eq!. Verify against the current code first; if no longer valid, skip with a brief reason. Keep the change minimal.

  5. codex_cached_input_is_clamped test does not verify split pricing

    rust/src/core/cost_pricing_tests.rs:104 · severity: medium · quick win

    Line 106 calls codex_cost_usd("gpt-5",10,100,0) and expects 10*cache_rate. This only covers the cached > input clamp. It does not check the partial-cache case cached < input where cost should be cached*cache_rate + (input-cached)*input_rate. A buggy impl that prices all input at cache rate when any cache is present, or that drops the non-cached remainder, would still pass.

    Prompt for AI agents

    In rust/src/core/cost_pricing_tests.rs around line 104: Add a case with cached < input, e.g. codex_cost_usd("gpt-5",100,30,0) and assert 30*cache_rate + 70*input_rate. Verify against the current code first; if no longer valid, skip with a brief reason. Keep the change minimal.

Also noted:

  • claude_tier_boundary only exercises input channelrust/src/core/cost_pricing_tests.rs:133 · medium
  • zero_negative_and_unknown leaves codex negative path untestedrust/src/core/cost_pricing_tests.rs:175 · medium

For coding agents: fix high findings and quick-win mediums now; batch involved mediums into at most one fix round; low notes are informational, do not fix them unless already editing that file; never exceed one CodeRev fix round per PR, decline remaining findings with a reason instead.

Advisory. Findings generated by deepseek-v4-flash and muse-spark-1.2-contributor, each filtered through a 3-vote refutation panel with the changed code in evidence.

Comment thread rust/src/core/cost_pricing_tests.rs
Comment thread rust/src/core/cost_pricing_tests.rs
Comment thread rust/src/core/cost_pricing_tests.rs
Comment thread rust/src/core/cost_pricing_tests.rs
Comment thread rust/src/core/cost_pricing_tests.rs
Comment thread rust/src/core/cost_pricing_tests.rs
Comment thread rust/src/core/cost_pricing_tests.rs
@tsouth89
tsouth89 merged commit 3480461 into main Aug 10, 2026
10 checks passed
@tsouth89
tsouth89 deleted the test/sbs-557-cost-pricing branch August 10, 2026 00:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant