Vulcan

Vulcan is a production-shaped multi-backend model-serving and training platform — sibling project to Argus. One serving contract, many runtimes (BentoML, Ray Serve, Triton, vLLM, KServe); distributed training / LoRA; MLflow + W&B tracking; DVC; and a tool-grounded LangGraph advisor — with GPU infra validated in CI and applied only out-of-band.

Docs: make docs-serve (MkDocs) · DEMO · Case study · Known gaps · ADRs · CHANGELOG

Release state: tagged v1.0.0 (phase-15) and v1.2.0 (phases 16–22: advanced GPU packaging, cost-per-token, training backends, LoRA/PEFT, DVC, experiment tracking, LangGraph advisor). No v1.1.0 tag (phase-17 track shipped on main without a separate minor tag).

ADR	Decision
ADR-001	Unified model serving contract (not per-backend APIs)
ADR-002	GPU cost-safety policy — no real GPUs in CI
ADR-003	MIG partitioning strategy (many-small vs large-batch)
ADR-004	Multi-tenant GPU scheduling with Kueue
ADR-005	Spot GPU strategy (cost, checkpoint contract, workload fit)
ADR-006	Routing policy (benchmark-driven selection + fallback)
ADR-007	Advanced GPU serving scope — GPTQ/AWQ/FP8 packs, speculative decoding docs, TensorRT-LLM templates; validate-only in CI (no invented tokens/s)
ADR-008	Self-hosted cost-per-token — $/GPU-hour assumptions for phase-7 instance types × benchmark throughput; labeled assumptions, not invoices
ADR-009	GPU cost-safety extends to training — CI uses CPU gloo world_size=2 only; no invented GPU throughput
ADR-010	Unified training job contract — TrainingJobSpec / TrainingJobResult for Ray Train, FSDP/DDP, DeepSpeed
ADR-011	LoRA / PEFT — fine-tune job type + BentoML base+adapter via unchanged /v1/infer; structural verify, no adapter hash pins
ADR-012	DVC for deterministic model exports — local remote in CI; MANIFEST cross-check; never track training/adapters by hash
ADR-013	Pluggable experiment tracking — MLflow self-hosted (:9014) + W&B offline-only (moto-style; no wandb.ai in CI)
ADR-014	LangGraph advisor — tool-grounded recommendations only; non-fabrication extends ADR-007; pinned local LLM, no paid API in CI

Managed training/hosting comparison: pipelines/sagemaker/ (moto in CI; manual runbook).

Bedrock as a selectable LLM backend: bedrock-gateway/ (thin adapter; moto in CI; optional :9006).

Training→serving loop: pipelines/kubeflow/ (KFP + Training Operator composing Kueue/Karpenter/checkpointing → KServe; runbook).

Routing gateway: gateway/ on :9007 (ADR-006; recorded benchmarks + Bedrock pricing; explainable fallback).

Observability: observability/ — Prometheus :9008, Grafana :9009, Tempo :9010 (make up-observability). Phase-17 adds cost-per-token panels (Bedrock pricing-reference + ADR-008 $/GPU-hour math) and GPU utilization via real DCGM Helm under observability/gpu-metrics/ for phase-7 pools, with a LIVE-SYNTHETIC DCGM-shaped exporter for compose/CI. Phase-18 adds $/training-step from training/results/ × the same assumptions file.

Training (phase-18/19/21): training/ — Ray Train, FSDP/DDP, and DeepSpeed behind contracts/training-job-contract/ (ADR-010). LoRA/PEFT fine-tune under training/fsdp-ddp/lora/ with BentoML reference-tiny-llm-lora-demo serving (ADR-011). Experiment tracking via training/common/tracking.py (ADR-013; VULCAN_TRACKER_BACKEND=none|mlflow|wandb). CI runs CPU only (ADR-009); optional status HTTP on :9011–:9013 and MLflow on :9014 (docker compose --profile training up).

Advisor (phase-22 / v1.2.0 close-out): advisor/ — LangGraph tool-grounded routing/cost advisor (ADR-014). Tools query live Prometheus, benchmark/results/*.json, and gateway routing; CI asserts every number in the answer appears in that run’s tool evidence. No hosted LLM in CI.

Advanced GPU serving (phase-16): serving/vllm/gpu-variants/ (GPTQ/AWQ/FP8 resource manifests) and serving/triton/tensorrt-llm/ (TensorRT-LLM config.pbtxt + Dockerfile + runbook) — schema/config.pbtxt lint in CI only; no GPU build or invented throughput (ADR-007).

---

Architecture (north star)

flowchart LR
  Client["clients / console"] --> GW["gateway"]
  Client --> Adv["advisor/\nLangGraph"]
  GW --> Contract["model-contract\n/health /metrics /v1/infer"]
  Contract --> Bento["serving/bentoml"]
  Contract --> Ray["serving/ray-serve"]
  Contract --> Triton["serving/triton\n(+ tensorrt-llm)"]
  Contract --> VLLM["serving/vllm\n(+ gpu-variants)"]
  Contract --> KServe["serving/kserve"]
  TrainContract["training-job-contract"] --> Train["training/\nray-train · fsdp-ddp · deepspeed"]
  Train --> Track["tracking.py\nMLflow / W&B offline"]
  GPU["gpu-infra + autoscaling"] -.->|"schedule / scale"| Contract
  Obs["observability\n(+ gpu-metrics / cost-exporter)"] --> GW
  Obs --> Contract
  Bench["benchmark/results"] --> GW
  Bench --> Adv
  Obs --> Adv
  GW --> Adv

Loading

Full design: ARCHITECTURE.md

---

Quick start

git clone https://github.com/hamidmatiny/Vulcan.git && cd Vulcan
cp .env.example .env
make models-export      # once — pin-identical GPT-2 + ResNet-18 weights
make up                 # :9000 bentoml · :9002 ray-serve · :9003 triton · :9004 vllm
make test
curl -s localhost:9004/health
VULCAN_BACKEND_URL=http://127.0.0.1:9004 VULCAN_CONFORMANCE_MODALITIES=llm make test-serving-common
make benchmark-vllm       # → benchmark/results/vllm-cpu.json
make down

Ports: Vulcan owns 9000–9099 on the host (avoids Argus and other stacks).
Pinned models: models/MANIFEST.md · BentoML · Ray Serve · Triton · vLLM (LLM-only) · Conformance: serving/common/

GPU policy: CI and make up never provision real GPUs. Manual GPU benchmarks live in docs/benchmarks/ (ADR-002).

---

Repository layout

contracts/model-contract/     OpenAPI + JSON Schema (platform contract)
serving/{common,bentoml,ray-serve,triton,vllm,kserve}/
  vllm/gpu-variants/          GPTQ / AWQ / FP8 resource manifests (phase-16)
  triton/tensorrt-llm/        TensorRT-LLM template + Dockerfile (phase-16)
gateway/                      Benchmark-driven routing (:9007)
advisor/                      LangGraph tool-grounded advisor (ADR-014)
benchmark/                    Harnesses (CPU local; GPU manual)
gpu-infra/{gpu-operator,mig,kueue}/
autoscaling/{karpenter,checkpointing}/
pipelines/{kubeflow,sagemaker}/
bedrock-gateway/
infra/{terraform,helm,argocd}/
observability/                Prometheus / Grafana / Tempo / OTel
  cost-exporter/              Routing cost + cost-per-token (ADR-006/008)
  gpu-metrics/                Real DCGM Helm/scrape + synthetic-dcgm
console/  models/
docs/{adr,benchmarks,runbooks}/  tests/e2e/

---

Engineering bar

• Commits: phase-N: <summary> or fix(<component>): <summary> only
• ADRs for architectural decisions under docs/adr/
• README per component
• CI from day one — lint, tests, ADR gate for contracts/, gpu-infra/, advanced GPU paths, and cost/GPU-metrics assumptions
• ≥ 65% coverage on gated packages
• Cursor rules in .cursor/rules/ enforce contract-first + CPU-fallback automatically