relay

A self-hostable LLM gateway. One OpenAI-compatible API in front of Claude and OpenAI — smart routing, refusal-aware fallback, semantic response caching, virtual keys with budgets, and cost/latency observability.

npm install
cp .env.example .env    # add ANTHROPIC_API_KEY (and optionally OPENAI_API_KEY)
npm run dev              # → http://localhost:8787

import OpenAI from "openai";

// Point any existing OpenAI SDK client at relay — no other code changes.
const client = new OpenAI({ baseURL: "http://localhost:8787/v1", apiKey: "unused" });

const res = await client.chat.completions.create({
  model: "claude-opus-4-8",
  messages: [{ role: "user", content: "Hello!" }],
});

Why

Anyone shipping AI features ends up juggling multiple providers with different SDKs and failure modes, no unified cost/latency view, and money burned re-answering near-identical prompts. relay is a small, readable gateway that does the 20% that matters — not a reimplementation of LiteLLM/Portkey, but a focused tool built around three differentiators:

Features

• OpenAI-compatible API — POST /v1/chat/completions, streaming included. Point any OpenAI SDK client at it by changing baseURL.
• Refusal-aware fallback — a policy/safety decline (stop_reason: "refusal", OpenAI content_filter) fails over to the next provider in the chain immediately — never retried on the same one.
• Smart routing — route by explicit model, cheapest-first, fastest-first (measured rolling latency), or by required capability, across named model tiers.
• Circuit breaker + retry with backoff — a provider that's clearly down is skipped, not hammered.
• Response caching — exact-match + a dependency-free semantic cache (lexical embedding, cosine similarity) that catches near-duplicate phrasing without an embeddings API.
• Virtual keys — issue bearer tokens with a USD spend cap and a rate limit each.
• Cost/latency metrics — every attempt emits a request event with cost, latency, and outcome.

The benchmark (the point)

Synthetic, reproducible, zero live-API-cost — measures what the gateway's own mechanisms buy you, isolated from any real provider's actual reliability (see docs/DECISIONS.md for why):

$ npm run bench

relay benchmark — 2000 simulated requests per run, concurrency 40
primary: 10% refusal, 5% transient-error rate, 260ms latency
fallback: 1% refusal, 1% error rate, 400ms latency

  scenario             success rate   p50 latency   p99 latency   cache hit
  baseline                  86.0%         267ms         271ms
  + fallback               100.0%         268ms         694ms
  + fallback + cache       100.0%           0ms         671ms     84.3%

Reading it: the fallback chain recovers the ~14% of requests a single provider drops (86% → 100% success), at the cost of a heavier p99 (the tail where the fallback actually triggers). Adding the cache on top of realistic repeat-traffic (70% common questions) serves 84% of requests instantly from cache — p50 latency drops to 0ms — while the uncached 16% still pay for fallback when needed. Reproduce: npm run bench -- 2000 40 (n, concurrency).

Try it

npm run dev          # the gateway server → http://localhost:8787
npm run dashboard     # live demo dashboard (simulated traffic) → http://localhost:8080
npm run bench         # the benchmark above
npm test              # 42-test suite, mocked providers, no network calls

Routing examples

// Explicit — pin one model
{ model: "claude-opus-4-8", messages: [...] }

// Cost-optimized — cheapest model in the "fast" tier
{ model: "fast", messages: [...], relay: { policy: "cost" } }

// Explicit fallback chain — try Opus, then GPT-4o if Opus refuses/fails
{ model: "claude-opus-4-8", messages: [...], relay: { fallback: ["openai/gpt-4o"] } }

Virtual keys

curl -X POST localhost:8787/v1/keys -d '{"budgetUsd": 5, "rateLimit": {"limit": 60, "intervalMs": 60000}}'
# → { "id": "relay_...", "budgetUsd": 5, "spentUsd": 0, ... }

curl localhost:8787/v1/chat/completions \
  -H "Authorization: Bearer relay_..." \
  -d '{"model": "claude-opus-4-8", "messages": [{"role":"user","content":"hi"}]}'

Architecture

Full write-up — the Provider contract, routing tiers, resilience pipeline, caching, virtual keys — in docs/ARCHITECTURE.md. Design trade-offs in docs/DECISIONS.md.

Stack

TypeScript · Hono · official @anthropic-ai/sdk (adaptive thinking, refusal detection) · openai SDK as an optional peer dependency (injected client, no hard dependency) · Vitest · tsup.

MIT © Lavya Tanotra