PyTokenCalc: Unified Token Counting for Multi-Provider LLMs

---

Why Star This?

• One library, 20+ LLM providers, 99%+ accuracy — Replace OpenAI, Anthropic, Google, Ollama, and 15+ more with a single validated API
• Quality validation built-in — Detect counting drift, validate model baselines, and catch silent errors before cost reconciliation
• No vendor lock-in — Switch providers by changing one string; same code for all
• Self-healing — Automatic fallback to pattern-based counting when providers go down

---

Release Status

v0.9.0 ✅ RELEASED (July 18, 2026)

Universal Token Counting for ANY LLM

Providers Supported:

• 8 cloud providers (OpenAI, Anthropic, Google, Cohere, Azure, RunPod, Together AI, Replicate)
• 7 local inference engines (Ollama, LM Studio, LocalAI, Llama.cpp, GPT4All, Text Gen WebUI, Jan)
• Custom provider registration (BYOM - Bring Your Own Model)
• Docker/Kubernetes deployment ready

Key Features:

• Pattern-based forward compatibility (claude-, gemini-, command-*)
• Model discovery system (automatic provider suggestion)
• Platform-aware tracking (same model, different platforms)
• Temporal variation monitoring (timestamp + session tracking)
• 99%+ accuracy vs official counters
• 104 tests passing, zero breaking changes

Documentation:

• CUSTOM_PROVIDERS.md (200+ lines, 10+ provider examples including Docker/Kubernetes)
• CHANGELOG.md (complete version history)
• RELEASE_NOTES_v0.9.0.md (comprehensive release documentation)

---

The Problem

You're building an LLM application. You need to count tokens accurately, predict costs, validate model efficiency, and know which providers are degrading. But right now:

• You don't trust your token counts. Are they accurate? Off by how much? No idea.
• Cost predictions are guesses. You budget $X, spend $Y. No visibility into why.
• Model baselines are stale. When did you last verify they're still valid?
• Provider health is invisible. You find out when your app breaks, not before.
• 10+ libraries to integrate. Each with different APIs and update cycles.

The nightmare:

• ❌ No validation layer between your code and provider APIs
• ❌ Silent errors: wrong counts go undetected until cost reconciliation
• ❌ Vendor lock-in: switching providers requires rewriting integrations
• ❌ Zero audit trail: no way to trace which decision used which counts
• ❌ Dead time: managing integrations instead of building features

You need a trusted, validated token oracle. Not just a counter.

---

The Solution: PyTokenCalc

Self-validating token intelligence for LLM applications.

One library. One API. All providers. With quality guarantees.

from pytokencalc.tokenizers import TokenCounterRegistry

registry = TokenCounterRegistry()

# Same code for every provider
tokens = registry.count_tokens("gpt-4o", "Your prompt")
tokens = registry.count_tokens("claude-3-5-sonnet", "Your prompt")
tokens = registry.count_tokens("llama-70b", "Your prompt")

print(f"Tokens: {tokens.input_tokens}")

That's it. No provider-specific code. No multiple libraries. No headaches.

---

Why PyTokenCalc?

Feature	PyTokenCalc	OpenAI library	Anthropic SDK	Others (Ollama, etc.)
Providers supported	20+	OpenAI only	Anthropic only	1 each
Unified API	✅ One call	❌ Different per provider	❌ Different	❌ All different
Accuracy	99%+ validated	Reference	Reference	Varies
Quality validation	✅ Built-in drift detection	❌ None	❌ None	❌ None
Cost prediction	✅ Yes	⚠️ Basic	⚠️ Basic	❌ No
Fallback counting	✅ Pattern-based	❌ None	❌ None	❌ None
Model discovery	✅ Auto-detect	❌ Manual	❌ Manual	❌ Manual
Custom providers	✅ Easy registration	❌ Not supported	❌ Not supported	❌ No
Audit trail	✅ Full history	❌ None	❌ None	❌ None

---

⚡ Quick Start: Terminal One-Liners

Don't want to write Python? Use the pycount command directly:

# Count tokens instantly
pycount "Hello world"
# Output: 2 tokens (input: 2)

# Pipe from stdin
echo "Your text here" | pycount

# Use a different model
pycount -m claude-3-sonnet "Your text"

# JSON output for scripting
pycount -j "Your text"
# {"model": "gpt-4", "tokens": 2, "input": 2, ...}

→ Full CLI Guide

---

What You Get

1. Validated Token Counts

Every token count is validated against historical baselines, provider reliability, and model efficiency standards. Know your data is accurate before it affects your decisions.

2. Self-Validating API

Built-in quality gates catch errors before they cascade:

• Token accuracy checks (±5% deviation detection)
• Model efficiency baselines (50+ sample statistical validation)
• Cost prediction confidence (±15% accuracy validation)
• Provider health monitoring (uptime, response time, consistency)

3. True Unified Interface

One function for 20+ providers (cloud and open-source).

# Works for everything
models = [
    "gpt-4o",
    "claude-3-5-sonnet",
    "gemini-2-flash",
    "llama-70b",
    "mistral-large",
]

for model in models:
    result = registry.count_tokens(model, "Your text")
    print(f"{model}: {result.input_tokens} tokens")

2. Smart Routing: Fast When You Can, Accurate When You Need To

• Local tokenizers for OpenAI GPT and open-source models (5-10ms)
• Cached API calls for proprietary models like Claude and Gemini (0ms if cached, 200ms first time)
• Automatic — you don't think about it

# Automatically uses tiktoken (fast, local, free)
gpt_tokens = registry.count_tokens("gpt-4o", text)  # 5ms

# Automatically caches and reuses
gpt_tokens = registry.count_tokens("gpt-4o", text)  # 0ms (cached)

3. 99%+ Accuracy vs Official Counts

Local tokenizers are 100% accurate for their models. API-backed counts match official numbers exactly. No guessing.

4. 70-80% Fewer API Calls

Intelligent caching means you don't hammer provider APIs just to count tokens.

5. Batch Operations

Count tokens for multiple prompts at once.

results = registry.count_batch([
    {"model": "gpt-4o", "text": "Prompt 1"},
    {"model": "llama-70b", "text": "Prompt 2"},
    {"model": "claude-3-5-sonnet", "text": "Prompt 3"},
])

for result in results:
    print(f"{result.input_tokens} tokens")

---

Installation

For Users

Basic (Local Tokenizers Only)

pip install pytokencalc

This gives you token counting for OpenAI GPT and Llama/Mistral models (covers 70% of use cases).

Full (Recommended)

pip install "pytokencalc[tokenizers]"

Installs optional dependencies for broader model support:

• tiktoken — OpenAI GPT models
• transformers — Llama, Mistral, and 1000+ HuggingFace models
• sentencepiece — Additional model support

For Developers

# Clone the repository
git clone https://github.com/Mullassery/pytokencalc.git
cd pytokencalc

# Create and activate virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install in development mode with dev dependencies
pip install -e ".[dev,tokenizers]"

# Run tests
pytest tests/
make test    # or use Makefile

Requirements:

• Python 3.9+
• pip or uv

---

Quick Example

from pytokencalc.tokenizers import TokenCounterRegistry

# Initialize once
registry = TokenCounterRegistry()

# Use anywhere
prompt = "Write me a Python function that..."
result = registry.count_tokens("gpt-4o", prompt)

print(f"Input tokens: {result.input_tokens}")
print(f"Latency: {result.latency_ms}ms")
print(f"Was cached: {result.cached}")
print(f"Source: {result.source}")  # "local", "api", or "formula"

Output:

Input tokens: 42
Latency: 5.2ms
Was cached: False
Source: local

Run it again with the same text:

result = registry.count_tokens("gpt-4o", prompt)
print(f"Latency: {result.latency_ms}ms (cached!)")

Output:

Latency: 0.3ms (cached!)

---

Why This Matters

Before PyTokenCalc

# Your code without PyTokenCalc
import tiktoken
from transformers import AutoTokenizer
import anthropic

# Different library, different API for each
enc = tiktoken.encoding_for_model("gpt-4o")
gpt_tokens = len(enc.encode(text))

llama_tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-70b")
llama_tokens = len(llama_tokenizer.encode(text))

client = anthropic.Anthropic()
claude_response = client.messages.count_tokens(
    model="claude-3-5-sonnet",
    messages=[{"role": "user", "content": text}]
)
claude_tokens = claude_response.input_tokens

# Three different patterns, three different libraries

With PyTokenCalc

from pytokencalc.tokenizers import TokenCounterRegistry

registry = TokenCounterRegistry()

gpt_tokens = registry.count_tokens("gpt-4o", text).input_tokens
llama_tokens = registry.count_tokens("llama-70b", text).input_tokens
claude_tokens = registry.count_tokens("claude-3-5-sonnet", text).input_tokens

# One pattern. One library. Done.

---

Supported Providers

🌐 Cloud LLM APIs (20+)

Explicit Support:

• ✅ OpenAI (GPT-4, GPT-3.5-turbo)
• ✅ Anthropic (Claude 3 + pattern: claude-*)
• ✅ Google (Gemini, pattern: gemini-*)
• ✅ Cohere (Command, pattern: command-*)
• ✅ Azure OpenAI (same as OpenAI)
• Plus: RunPod, Together AI, Replicate, HuggingFace Inference, and more

Dynamic Support: Any cloud provider with an API endpoint (RunPod, Llama Labs, custom backends)

💻 Local Inference Engines (7+)

Auto-Detected:

• LM Studio (localhost:1234)
• LocalAI (localhost:8080)
• Llama.cpp (localhost:8000)
• GPT4All (localhost:4891)
• Text Generation WebUI (localhost:5000)
• Jan (localhost:1337)
• Vllm (localhost:8000)

🦙 Open-Source Models

Built-in Ollama Support:

• Any model you pull into Ollama
• Automatic model discovery
• No model list required

HuggingFace Direct:

• 14+ known models supported
• Any HuggingFace model works

🚀 Bring Your Own Model (BYOM)

Completely Custom:

• Your fine-tuned models
• Your proprietary models
• Your locally-trained models
• Any model running on any GPU/CPU

Just expose an API and register it with PyTokenCalc.

🔍 Model Discovery

Don't know which provider? PyTokenCalc discovers it:

from pytokencalc.model_discovery import ModelDiscovery

# Automatic discovery
result = ModelDiscovery.lookup_model("llama-2-7b")
# Returns: ["ollama", "huggingface", "runpod", "together-ai"]

# Get setup instructions
report = ModelDiscovery.get_discovery_report("mistral-7b")
# Prints: Install, setup steps for each provider

---

Advanced Features

Custom Provider Registration

Use any LLM provider, including proprietary or self-hosted:

from pytokencalc.tokenizers.custom_provider_counter import (
    CustomProviderCounter,
    register_custom_provider,
)

# Register your RunPod endpoint
runpod = CustomProviderCounter(
    provider_name="runpod",
    base_url="https://api.runpod.io/v2/YOUR_ID",
    api_key="your-key"
)
runpod.register_models(["llama-2-7b", "mistral-7b"])
register_custom_provider(runpod)

# Now use it like any other provider
registry.count_tokens("llama-2-7b", text, provider="runpod")

See CUSTOM_PROVIDERS.md for 10+ examples.

Platform-Aware Token Counting

⚠️ Important: Same model on different platforms may have different token counts

# Same text, different platforms = different results
result_local = registry.count_tokens("llama-2-7b", text, provider="ollama")
result_cloud = registry.count_tokens("llama-2-7b", text, provider="runpod")

print(result_local.platform)      # "ollama"
print(result_cloud.platform)      # "runpod"
print(result_local.timestamp)     # When it was counted

Results are kept separate by platform and timestamp. Never aggregate or mix.

Temporal Variation Tracking

Cloud infrastructure changes over time. PyTokenCalc tracks this:

# First session
result1 = registry.count_tokens("gpt-4o", text)
print(f"Tokens: {result1.input_tokens}, Latency: {result1.latency_ms}ms")
# Output: Tokens: 100, Latency: 50ms

# Later session (after infrastructure update)
result2 = registry.count_tokens("gpt-4o", text)
print(f"Tokens: {result2.input_tokens}, Latency: {result2.latency_ms}ms")
# Output: Tokens: 101, Latency: 120ms (different!)

# Both are tracked with timestamps
print(result1.timestamp, result2.timestamp)  # Different times

---

API Reference

Count Tokens

from pytokencalc.tokenizers import TokenCounterRegistry

registry = TokenCounterRegistry()

# Single count
result = registry.count_tokens(
    model="gpt-4o",
    text="Your prompt here"
)

print(result.input_tokens)    # int - token count
print(result.latency_ms)      # float - milliseconds taken
print(result.cached)          # bool - came from cache?
print(result.source)          # str - "local", "api", or "formula"

Batch Count

# Multiple counts in one call
results = registry.count_batch([
    {"model": "gpt-4o", "text": "Text 1"},
    {"model": "llama-70b", "text": "Text 2"},
])

for result in results:
    print(f"{result.input_tokens} tokens")

Cache Stats

cache = registry.tokenizers[0].cache
stats = cache.get_stats()

print(stats.hit_rate)    # Cache hit rate (0-1)
print(stats.size)        # Number of cached entries
print(stats.total_calls) # Total counts requested

---

Real-World Use Cases

Use Case 1: Monitor Token Usage

def log_request(model, prompt, response):
    tokens = registry.count_tokens(model, prompt)
    print(f"Request used {tokens.input_tokens} tokens")

# Call it automatically on every LLM invocation
log_request("gpt-4o", "Hello", response)

Use Case 2: Token Budget Management

MAX_TOKENS_PER_HOUR = 1_000_000
used_tokens = 0

for prompt in user_prompts:
    token_count = registry.count_tokens("claude-3-5-sonnet", prompt)
    if used_tokens + token_count.input_tokens > MAX_TOKENS_PER_HOUR:
        print("Budget exceeded!")
        break
    
    used_tokens += token_count.input_tokens
    response = call_llm(prompt)

Use Case 3: Model Comparison

text = "Your 1000-word essay..."

for model in ["gpt-4o", "claude-3-5-sonnet", "gemini-2-flash"]:
    tokens = registry.count_tokens(model, text).input_tokens
    print(f"{model}: {tokens} tokens")

# See which model would be cheapest

---

Performance

Scenario	Latency	Notes
First local count (GPT)	5ms	tiktoken initialization
Cached local count	0.3ms	Memory lookup
First API count (Claude)	200-300ms	Network call
Cached API count	0.2ms	Memory lookup
Batch of 10 (mixed)	~10ms	Parallel processing

TL;DR: Local is instant. API counts are cached aggressively. You rarely wait.

---

Database & Storage

PyTokenCalc maintains a database to store token counts and enable reconciliation across API calls:

PyTokenCalc Standalone: Manages its own database for token accounting. OpenAnchor is optional.

PyTokenCalc + OpenAnchor: When you install OpenAnchor, it automatically includes PyTokenCalc. Both use the same database:

• PyTokenCalc stores: Raw token counts (token_events)
• OpenAnchor stores: Analysis & recommendations (attribution, patterns, etc)
• Single instance: Both projects share one database
• OpenAnchor does NOT create its own database—it enriches PyTokenCalc's database with intelligence

Important: OpenAnchor requires PyTokenCalc to be installed. You cannot use OpenAnchor without PyTokenCalc. However, you can use PyTokenCalc alone without OpenAnchor.

---

Token Counting Scope

PyTokenCalc counts tokens for all operations, including:

Inference (Text Generation)

# Count tokens for chat/completion endpoints
result = registry.count_tokens("gpt-4o", "Write a story about...")
# Tokens used for generating responses

Embeddings (Vector Representation)

# Count tokens for embedding API calls
result = registry.count_tokens("text-embedding-3-large", "Hello world")
# Tokens used for creating embeddings

Any Token Consumption

• Chat completions
• Text embeddings
• Image descriptions
• Code generation
• Summarization
• Classification
• Any model operation that consumes tokens

One function. All operations.

---

What's NOT Included

PyTokenCalc counts tokens accurately. That's what it does.

For advanced use cases like token intelligence, optimization, or analytics, use OpenAnchor (which bundles PyTokenCalc) or build on PyTokenCalc's token counting API:

• 🔍 Token intelligence & attribution (use OpenAnchor)
• 📊 Pattern detection & anomaly alerts (use OpenAnchor)
• 🎯 Optimization recommendations (use OpenAnchor)
• 📈 Plan capacity and model comparison (build on PyTokenCalc)

---

Contributing

Want to add support for a new provider? See ADDING_PROVIDERS.md for step-by-step guide.

---

License

MIT License. See LICENSE for details.

---

About

PyTokenCalc solves one problem perfectly: unified token counting across all LLM providers.

Author: Georgi Mammen Mullassery (@Mullassery)
Repository: https://github.com/Mullassery/PyTokenCalc

---

Stop integrating tokenizers. Start counting tokens.