A benchmark that tests every model on Ollama Cloud for inference speed and token cost.
| Metric | Description |
|---|---|
| TTFT | Time to first token (ms). Lower is better. |
| Total time | Full response time (ms). Lower is better. |
| Tokens/sec | Output tokens per second. Higher is better. |
| Output tokens | Total tokens generated across all prompts. |
| OpenAI-equiv cost | What this would cost at $5/1M input, $15/1M output. |
| Success rate | Fraction of prompts that returned a response. |
See the leaderboard:
- leaderboard.html — interactive HTML page
- leaderboard.md — markdown table
- results.json — raw per-prompt data
- Get an Ollama Cloud API key from https://ollama.com/settings/keys
- Copy
.env.exampleto.envand add your key:cp .env.example .env # edit .env and replace the placeholder with your key - Install Python 3 and requests:
pip install requests - Run the benchmark:
python3 benchmark.py
The script:
- Fetches all available models from Ollama Cloud
- Sends 5 prompts to each model via streaming API
- Measures TTFT, total time, token count, and tokens/sec
- Computes the OpenAI-equivalent cost (what you would pay at GPT-4o-class rates)
- Saves results incrementally after each model
- Generates markdown and HTML leaderboards
- Explain quantum entanglement in one paragraph for a high school student.
- Write a Python function that checks if a string is a valid palindrome.
- What are the main differences between TCP and UDP? List 5 key differences.
- Write a haiku about the ocean, then explain the imagery in two sentences.
- Give 3 tips for improving sleep quality, with a one-sentence explanation for each.
- Temperature 0, seed 42 for reproducibility
- Streaming API for TTFT measurement
- Token counts estimated at ~4 chars per token
- 60s timeout per request, 1 retry on failure
- OpenAI-equivalent pricing: $5/1M input + $15/1M output (GPT-4o-class)
- Ollama Cloud is free during beta — the cost column shows what you would pay elsewhere
- Python 3.10+
requestslibrary- An Ollama Cloud API key