Skip to content

feat: rate limit utilization extraction, scheduling, and passthrough - #17

Open
AmazingAng wants to merge 2 commits into
mainfrom
feat/ratelimit-passthrough
Open

feat: rate limit utilization extraction, scheduling, and passthrough#17
AmazingAng wants to merge 2 commits into
mainfrom
feat/ratelimit-passthrough

Conversation

@AmazingAng

Copy link
Copy Markdown
Owner

Summary

  • Extract upstream anthropic-ratelimit-* headers from every Claude API response (success and failure) in proxyWithRetry()
  • Store parsed RateLimitInfo per account in AccountManager, with console warnings at 80%/95%/throttled thresholds
  • Passthrough all anthropic-ratelimit-* headers to downstream clients (prefix-match, not whitelist), plus x-auth2api-account for account identification
  • Schedule utilization-aware account selection: sticky accounts release at >= 0.8 utilization, prefer lowest-utilization account, fall back to round-robin on cold start. Stale utilization data (> 30 min) decays to 0 to avoid permanently deprioritizing accounts after window resets
  • Enrich /admin/accounts snapshots with rateLimit and anonymousId (SHA-256 based, stable across restarts)

Compatibility with auth2api-router

Fully supports header_filter.lua which reads:

  • anthropic-ratelimit-unified-5h-utilization / 7d-utilization (routing decisions)
  • anthropic-ratelimit-requests-remaining / tokens-remaining (capacity checks)
  • anthropic-ratelimit-requests-limit / tokens-limit (admin display)
  • anthropic-ratelimit-unified-reset (admin display)

/admin/accounts enrichment is backward-compatible (additive fields only).

Changed files

File Change
src/upstream/ratelimit.ts NewextractRateLimitHeaders(), buildDownstreamRateLimitHeaders(), getEffectiveUtilization()
src/accounts/manager.ts RateLimitInfo type, recordRateLimit(), getAnonymousId(), utilization-aware getNextAccount(), enriched getSnapshots()
src/utils/http.ts ProxyOptions.success signature extended, centralized extraction/injection in proxyWithRetry()
src/handlers/anthropic.ts Inject rateLimitHeaders in success callback
src/handlers/openai.ts Same

Test plan

  • Verify rate limit headers appear in non-streaming responses
  • Verify rate limit headers appear in streaming responses (before SSE body)
  • Verify rate limit headers appear on final failure (429/5xx) responses
  • Verify /admin/accounts includes rateLimit and anonymousId fields
  • Verify account rotation prefers lower-utilization accounts
  • Verify stale utilization (> 30 min) decays to 0 and account re-enters rotation

Made with Cursor

- Add src/upstream/ratelimit.ts with extractRateLimitHeaders(),
  buildDownstreamRateLimitHeaders() (prefix-match all anthropic-ratelimit-*),
  and getEffectiveUtilization() (with 30-min staleness TTL)
- AccountManager: add RateLimitInfo tracking per account, recordRateLimit()
  with 0.8/0.95/throttled warnings, getAnonymousId() via SHA-256,
  utilization-aware scheduling in getNextAccount() (cooldown > util > round-robin),
  and enriched getSnapshots() with rateLimit + anonymousId
- proxyWithRetry(): centralized extraction from every upstream response,
  pass rateLimitHeaders to success callback, inject on final failure
- All handlers (anthropic + openai): inject rate limit headers before
  resp.json() / handleStreamingResponse()

Made-with: Cursor
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant