feat(k8s): LiteLLM + Ollama deployment, validated on dev (incl. #70 /api fix)#73
Merged
Conversation
PR #72 routes chat through a LiteLLM proxy with per-user virtual keys/budgets. Extend the kustomize deployment to match: - base/litellm.yaml: LiteLLM Deployment+Service (:4000), config mounted from a litellm-config-file ConfigMap (base/litellm_config.yaml), secrets from litellm-secrets - base/ollama.yaml: Ollama StatefulSet+Service (:11434, 10Gi PVC) serving the local-model fallback; pulls phi4-mini on start (CPU-friendly) - backend gets LITELLM_URL (config) + LITELLM_MASTER_KEY (backend-secrets) to provision per-user virtual keys; still uses Azure for KB embeddings - config/<env>/litellm.env: LOCAL_LLM_MODEL=ollama/phi4-mini, LOCAL_LLM_API_BASE, LANGFUSE_HOST - new sealed secret litellm-secrets (master key, OpenAI key, Langfuse, Supabase DATABASE_URL) via scripts/sealed_secret_litellm.sh; secrets.sh + templates updated across overlays - bump overlay image tags to 0.1.7 (must be tagged/built before deploy) - fix .gitignore: anchor the docker-compose `/litellm.env` so it no longer ignores the committed kustomize/config/*/litellm.env Validated: kubectl kustomize renders all three overlays (20 objects each); scripts pass bash -n. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XpNzobVi83p3YKL9sqtGwZ
…penAI The project uses Azure OpenAI (already used for embeddings) and has no separate hosted-OpenAI key, so point LiteLLM's openai-model alias at Azure. The alias name stays "openai-model" (hardcoded in the backend LLMModel Literal + frontend). - litellm_config.yaml (kustomize base + repo root, kept in sync): openai-model litellm_params -> model=azure/<deployment> via os.environ/AZURE_MODEL, plus api_base/api_version/api_key from env - config/<env>/litellm.env: add AZURE_MODEL, AZURE_API_BASE, AZURE_API_VERSION - litellm-secrets: openai-api-key -> azure-api-key; base/litellm.yaml env and sealed_secret_litellm.sh updated; secrets.sh passes the existing AZURE_API_KEY - secrets.dat.template: drop OPENAI_API_KEY (Azure key reused); fix DATABASE_URL example to the Supabase SESSION POOLER (IPv4, 5432) — the direct host is IPv6-only - litellm.env.example: OpenAI -> Azure vars Validated: kubectl kustomize renders all three overlays (20 objects); the litellm ConfigMap carries the AZURE_* vars and the pod reads azure-api-key. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XpNzobVi83p3YKL9sqtGwZ
Make the "openai-model" alias a group of two interchangeable deployments (Azure OpenAI + hosted OpenAI). Provide a key for either (or both) in secrets.dat; LiteLLM routes to whichever authenticates. Gives deploy-time flexibility without committing to a provider now. - litellm_config.yaml (base + root): second openai-model deployment (openai/gpt-4o) - litellm-secrets regains openai-api-key alongside azure-api-key; base/litellm.yaml, sealed_secret_litellm.sh (7 args), secrets.sh and secrets.dat.template updated - unify the endpoint var: LiteLLM now reads AZURE_ENDPOINT (the value the backend already uses for embeddings) instead of a separate AZURE_API_BASE, and the backend/litellm placeholders are aligned so the shared value can't drift - note in configs/README that KB embeddings still require the Azure key specifically, and root litellm.env.example local model -> phi4-mini Validated: kubectl kustomize renders all overlays (20 objects); the mounted config has both openai-model deployments and the pod reads both api keys. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XpNzobVi83p3YKL9sqtGwZ
…d OpenAI - litellm_config.yaml (base + root): openai-model deployment model is now os.environ/OPENAI_MODEL (default openai/gpt-4o-mini) instead of hardcoded gpt-4o - config/<env>/litellm.env + litellm.env.example: add OPENAI_MODEL=openai/gpt-4o-mini - config/vcell-ai-rke-dev/backend.env: switch dev to hosted OpenAI — PROVIDER=openai (singleton uses a plain OpenAI client with base_url=AZURE_ENDPOINT on the non-azure path), AZURE_ENDPOINT=https://api.openai.com/v1, gpt-4o-mini / text-embedding-3-small Dev now runs chat + embeddings on hosted OpenAI (verified the key against /v1/models: valid, both models accessible), so it no longer blocks on the Azure endpoint. The Azure deployment in the openai-model group stays dormant in dev (placeholder endpoint) and LiteLLM falls back to the OpenAI deployment. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XpNzobVi83p3YKL9sqtGwZ
The litellm pod was CrashLoopBackOff / OOMKilled (exit 137) on the vxrails cluster: Prisma migrate + proxy startup spikes past 1Gi. Bump requests to 512Mi/250m and limits to 2Gi/1cpu. Verified: litellm reaches 1/1 Ready and completes its DB migrations. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XpNzobVi83p3YKL9sqtGwZ
DNS for vcell-ai-dev.cam.uchc.edu now resolves to the ingress, so switch the dev overlay from letsencrypt-staging (browser-untrusted) to letsencrypt-prod for a trusted cert. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XpNzobVi83p3YKL9sqtGwZ
Real dev values for vcell-ai-rke-dev: AUTH0_DOMAIN/CLIENT_ID/AUDIENCE and SUPABASE_URL (derived from the DB project ref). These are public OIDC/Supabase identifiers, not secrets. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XpNzobVi83p3YKL9sqtGwZ
The frontend image hardcoded NEXT_PUBLIC_API_URL=http://k8s-wn-01...:30001 at build time, so the browser's data/chat calls never hit the environment's own ingress. Since the ingress serves the frontend at / and the backend at /api on the same host, bake a RELATIVE "/api" instead: the browser calls the same origin it's served from, the ingress routes /api -> backend, and one image works in every environment. - frontend/Dockerfile: ARG/ENV NEXT_PUBLIC_API_URL defaulting to /api; drop the COPY .env step - build_containers.yml: pass --build-arg NEXT_PUBLIC_API_URL=/api instead of writing a hardcoded NodePort .env Fixes #70. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XpNzobVi83p3YKL9sqtGwZ
Point all overlays at 0.1.8 (frontend built with the relative /api base from #74) and set NEXT_PUBLIC_API_URL=/api in the frontend config so browser data calls hit the environment's own ingress instead of a baked NodePort. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XpNzobVi83p3YKL9sqtGwZ
Login returned 502 Bad Gateway at /auth/callback: on a successful callback the
@auth0/nextjs-auth0 SDK sets a large encrypted session cookie, and nginx's
default 4k proxy buffer overflows on that Set-Cookie response header ("upstream
sent too big header"). Add nginx.ingress.kubernetes.io/proxy-buffer-size: 16k to
the frontend ingress in all overlays. Applied live to dev.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XpNzobVi83p3YKL9sqtGwZ
…' into feat/kustomize-litellm
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XpNzobVi83p3YKL9sqtGwZ
jcschaff
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Completes the Kubernetes deployment for VCell-AI (builds on #69), extended for the LiteLLM architecture (#72) and folding in the #70 frontend fix (#74). Deploys qdrant, backend, frontend, litellm, ollama via kustomize + sealed secrets across three overlays (
vcell-ai-rke,-rke-dev,-local).What's in it
:4000) + in-cluster Ollama (phi4-mini) as thelocal-modelfallback; per-user virtual keys/budgets persisted to Supabase Postgres.openai-modelalias runs on Azure or hosted OpenAI (dev runs hosted OpenAIgpt-4o-mini/text-embedding-3-small).litellm-secrets;LITELLM_URL/LITELLM_MASTER_KEYwired into the backend.NEXT_PUBLIC_API_URL=/api, so browser data calls hit the environment's own ingress (one image, every env)./→ frontend,/api/*→ backend (prefix stripped);proxy-buffer-size: 16kso the Auth0 callback's large session cookie doesn't 502; cert-managerletsencrypt-prod.main(public biomodel browsing, BNGL viz, removed placeholder UI) — no new required env vars; integrates cleanly.Validated live on dev
https://vcell-ai-dev.cam.uchc.eduon a trusted Let's Encrypt cert; backend via the/apirewrite; Auth0 Authorization-Code + PKCE login (502-on-callback fixed via the proxy-buffer bump).0.1.9(built from this merged branch).Bring-up fixes (all included)
Valid ghcr
GH_PAT, LiteLLMOOMKilled1Gi→2Gi, dev on hosted OpenAI, staging→prod cert, Auth0proxy-buffer-size.Fixes #70. Supersedes #74 (its commits are merged here).
🤖 Generated with Claude Code
https://claude.ai/code/session_01XpNzobVi83p3YKL9sqtGwZ