Semantic Cache

A semantic cache middleware: embeds prompts, serves cached responses for semantically similar prompts, forwards misses to a mock model. Node/TS + Redis (vector search) + React.

Runs in 3 modes:

• Local — Redis in Docker, services run directly via tsx.
• Docker — every service in its own container on a shared Docker network.
• Kubernetes — deployed to a cluster from k8s/ manifests.

A miss ("how do I sort an array in javascript?") costs a ~2500ms model call; a reworded prompt ("js array sorting") is served from cache in single-digit ms, matched by embedding similarity.

Initial setup (all modes)

Requires Node 20+ and Docker Desktop.

Install dependencies:

npm install
cd dashboard & npm install & cd ..

Build the image (Docker and Kubernetes modes only):

docker build -t semantic-cache:v1 .

Local mode (Redis in Docker, services via tsx)

Start Redis:

docker run -d --name redis-local -p 6379:6379 -p 8001:8001 redis/redis-stack:latest

RedisInsight: http://localhost:8001/redis-stack/browser

Start services (separate terminals):

npx tsx ai-model-server.ts
npx tsx cache-server.ts
cd dashboard & npm run dev

Dashboard: http://localhost:5173

Test from dashboard, or use Powershell:

Invoke-RestMethod -Uri http://localhost:3000/ask -Method Post -ContentType "application/json" -Body '{"prompt":"how do I sort an array in javascript?"}'
Invoke-RestMethod -Uri http://localhost:3000/stats

Docker mode

docker network create scdemo

docker run -d --name redis-local --network scdemo -p 8001:8001 redis/redis-stack:latest
docker run -d --name mock-model --network scdemo semantic-cache:v1 node dist/ai-model-server.js
docker run -d --name cache --network scdemo -p 3000:3000 -e REDIS_URL=redis://redis-local:6379 -e MODEL_URL=http://mock-model:4000 semantic-cache:v1

Dashboard: http://localhost:3000

Kubernetes mode

Apply:

kubectl apply -f k8s/

Check:

kubectl get pods -w
kubectl logs deploy/semantic-cache

Expose service locally:

kubectl port-forward svc/semantic-cache 3000:80

Dashboard: http://localhost:3000

Teardown:

kubectl delete -f k8s/