Generative AI Stack Playground

This repository builds a single-node GenAI serving lab on an AWS GPU EC2 instance. Terraform provisions the host, and Ansible installs Kubernetes with KServe, Envoy AI Gateway, llm-d routing, monitoring, and benchmarking tools.

The end state is a ready-to-test OpenAI-compatible endpoint for a KServe RawDeployment vLLM workload.

terraform apply is the only automated deployment entry point:

Prerequisite: The machine running this Terraform project must have both Terraform and Ansible installed. As a recommendation, install Ansible in a Python virtual environment to keep its dependencies isolated from the system Python installation.

Create the VPC, public subnet, internet gateway, route table, and security group.
Create the GPU EC2 instance defined in variables.auto.tfvars.
Copy config/ and scenarios/ to /home/ubuntu/ on the instance.
Render ansible_hosts.
Run ansible/general_master.yaml to install the Kubernetes and GenAI stack.

1. Stack components and versions

Versions are pinned mostly in ansible/general_master.yaml.

Component	Version / image	Notes
Kubernetes	`v1.34.2` install target	kubeadm single-node cluster.
Cilium	`1.19.5`	CNI, installed with Cilium CLI `v0.19.5`.
NVIDIA GPU Operator	`v26.3.3`	Installed from the NVIDIA Helm repo with CDI enabled.
KServe	`v0.17.0`	Standard mode. Workloads run as RawDeployment.
vLLM runtime	`vllm/vllm-openai:v0.24.0`	`config/vllm_server_runtime.yaml`.
Gateway API Inference Extension	`v1.5.0`	Upstream release manifest.
Envoy Gateway	`v1.7.0`	Helm install.
Envoy AI Gateway	`v0.6.0`	CRDs and controller.
llm-d EPP	`ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0`	Endpoint picker for the Llama scenario.
vLLM render sidecar	`vllm/vllm-openai-cpu:v0.24.0`	Tokenization sidecar for EPP.
Redis	`redis:6.0.6`	Envoy AI Gateway rate-limit backend.
Prometheus	chart `25.24.0`	NodePort `30090`.
Grafana	chart `8.5.2`	NodePort `30091`, includes a preconfigured dashboard.
aiperf	latest from PyPI	Installed in `/root/aiperf-venv`.

2. Project layout

.
├── *.tf                         # Terraform root module
├── variables.tf                 # Terraform input variables
├── variables.auto.tfvars        # Default region, AZ, AMI, instance type, disk
├── ansible/
│   ├── general_master.yaml      # Main playbook called by Terraform
│   ├── single_node_k8s.yaml     # kubeadm, Cilium, Helm, NVIDIA GPU Operator
│   ├── install-kserve-gw-api.yaml
│   ├── grafana_prometheus_monitoring.yaml
│   ├── benchmarking.yaml
│   └── print_stack_info.yaml
├── config/
│   ├── vllm_server_runtime.yaml
│   ├── redis_rate_limiting.yaml
│   └── monitoring.yaml
└── scenarios/kserve/
   ├── hf_token.yaml
   ├── inference_service_vllm_llmd.yaml
   ├── gaie_epp_llmd.yaml
   ├── envoy_gw.yaml
   ├── ai_gw_route.yaml
   └── dashboards/

3. Terraform variables

Edit variables.auto.tfvars for normal lab changes.

Variable	Purpose	Current value
`project_name`	Name prefix for AWS resources.	`GenAI-Training`
`region`	AWS Region.	`us-east-2`
`availability_zone`	AZ suffix. Combined with region.	`b` -> `us-east-2b`
`vpc_cidr`	VPC CIDR.	`192.168.0.0/16`
`public_subnet_cidr`	Public subnet CIDR.	`192.168.2.0/24`
`private_subnet_cidr`	Declared variable reserved for future private-subnet layouts.	`192.168.1.0/24`
`local_ranges`	CIDRs allowed to SSH and access NodePorts.	Defined in `variables.tf`. Update for your IP.
`key_name`	Existing EC2 key pair name.	`ec2-key-us-east-2`
`ec2_instances`	GPU EC2 instance definitions.	One `g6e.12xlarge` KServe node.

Current instance definition:

ec2_instances = [
  {
    id                 = "k8s-single-node-kserve"
    instance_type      = "g6e.12xlarge"
    ami                = "ami-07679d5b5f2a90395"
    disk_size          = 512
    configuration_type = "k8s_single"
  }
]

configuration_type = "k8s_single" selects the KServe + Envoy AI Gateway installation path.

4. Deploy the stack

Create the SSH key pair first

Terraform expects the EC2 key pair to already exist in AWS. It also expects the matching private key in the repository root.

Example:

aws ec2 create-key-pair \
  --region us-east-2 \
  --key-name ec2-key-us-east-2 \
  --query 'KeyMaterial' \
  --output text > ec2-key-us-east-2.pem

chmod 400 ec2-key-us-east-2.pem

If you use another key name, update key_name in variables.auto.tfvars.

Run Terraform

terraform init
terraform apply
terraform output ec2_hostnames

Example output:

ec2_hostnames = [
  {
    "hostname" = "ec2-18-217-123-45.us-east-2.compute.amazonaws.com"
    "id" = "GenAI-Training-ec2-ml-k8s-single-node-kserve"
    "type" = "k8s_single"
  },
]

GPU availability

GPU instances may not be available in every AZ. If AWS returns an insufficient-capacity or unsupported-instance error:

Change availability_zone.
Try another region.
Update ami because AMI IDs are Region-specific.
If needed, choose another GPU instance_type.

The configured AMI should be a GPU-ready Ubuntu/CUDA image compatible with the NVIDIA GPU Operator.

5. Validate the running cluster

After Terraform finishes, SSH to the EC2 instance and validate the base stack there:

ssh -i ./ec2-key-us-east-2.pem ubuntu@<ec2-public-dns>

ubuntu@k8s-single-node-kserve:~$ kubectl get pods -A
cert-manager              cert-manager-9cdb9969-xc42j                                       1/1     Running     0             12m
cert-manager              cert-manager-cainjector-5fb67577f8-fjx22                          1/1     Running     0             12m
cert-manager              cert-manager-webhook-8f49bfccc-zjw4h                              1/1     Running     0             12m
envoy-ai-gateway-system   ai-gateway-controller-64585dddcd-4j955                            1/1     Running     0             10m
envoy-gateway-system      envoy-gateway-5c8b5fd5fb-l6pqm                                    1/1     Running     0             10m
envoy-gateway-system      envoy-ratelimit-6bf9d7d85c-2zlzm                                  1/1     Running     2 (10m ago)   10m
gpu-operator              gpu-feature-discovery-54mn8                                       1/1     Running     0             12m
gpu-operator              gpu-operator-1787022915-node-feature-discovery-gc-7c4dcfbf52688   1/1     Running     0             12m
gpu-operator              gpu-operator-1787022915-node-feature-discovery-master-6b96c5s67   1/1     Running     0             12m
gpu-operator              gpu-operator-1787022915-node-feature-discovery-worker-2xbpj       1/1     Running     0             12m
gpu-operator              gpu-operator-5d9df66687-n8k5t                                     1/1     Running     0             12m
gpu-operator              nvidia-container-toolkit-daemonset-dwvkp                          1/1     Running     0             12m
gpu-operator              nvidia-cuda-validator-l6ch8                                       0/1     Completed   0             11m
gpu-operator              nvidia-dcgm-exporter-7qxc9                                        1/1     Running     0             12m
gpu-operator              nvidia-device-plugin-daemonset-fpqrz                              1/1     Running     0             12m
gpu-operator              nvidia-operator-validator-6cb44                                   1/1     Running     0             12m
kserve                    kserve-controller-manager-cc4d68d75-8z4dz                         2/2     Running     0             11m
kube-system               cilium-envoy-dd8gf                                                1/1     Running     0             13m
kube-system               cilium-operator-8745984f8-mhlp2                                   1/1     Running     0             13m
kube-system               cilium-zcr5l                                                      1/1     Running     0             13m
kube-system               coredns-66bc5c9577-26tdm                                          1/1     Running     0             13m
kube-system               coredns-66bc5c9577-csjwq                                          1/1     Running     0             13m
kube-system               etcd-k8s-single-node-kserve                                       1/1     Running     0             13m
kube-system               kube-apiserver-k8s-single-node-kserve                             1/1     Running     0             13m
kube-system               kube-controller-manager-k8s-single-node-kserve                    1/1     Running     0             13m
kube-system               kube-proxy-p8wvw                                                  1/1     Running     0             13m
kube-system               kube-scheduler-k8s-single-node-kserve                             1/1     Running     0             13m
monitoring                grafana-6978df4648-rx2mt                                          1/1     Running     0             9m26s
monitoring                prometheus-server-7c6d6fc957-bqk2g                                2/2     Running     0             9m43s
redis-system              redis-5d6c7d5b87-pwczm                                            1/1     Running     0             9m59s

Expected checks:

gpu-operator pods are running, with validator pods completed.
kserve-controller-manager is running in the kserve namespace.
Envoy Gateway and Envoy AI Gateway controller pods are running.
Prometheus and Grafana are running in monitoring.

6. Configure inference workloads

The scenario manifests are copied to /home/ubuntu/scenarios/kserve during terraform apply.

You must get a Hugging Face token from Hugging Face. The model meta-llama/Llama-3.2-1B-Instruct requires authenticated access.

Edit hf_token.yaml on the EC2 instance and set your token in the hf-secret Secret before applying it.

6.1 Create namespace and HF secret

Run these commands on the EC2 instance:

kubectl apply -f scenarios/kserve/hf_token.yaml
kubectl get ns kserve-test
kubectl -n kserve-test get secret hf-secret

Example output:

namespace/kserve-test created
secret/hf-secret created

NAME          STATUS   AGE
kserve-test   Active   6s

NAME        TYPE     DATA   AGE
hf-secret   Opaque   1      6s

Creates:

Namespace kserve-test
Secret hf-secret

6.2 Deploy the KServe InferenceService

kubectl apply -f scenarios/kserve/inference_service_vllm_llmd.yaml
kubectl -n kserve-test get inferenceservice llama3-1b
kubectl -n kserve-test get pods -l serving.kserve.io/inferenceservice=llama3-1b -w

Creates:

InferenceService/llama3-1b
3 vLLM predictor pods
1 GPU request per predictor
KV-cache event publishing on port 5556

Example result:

$ kubectl -n kserve-test get inferenceservice llama3-1b
NAME       URL                                      READY   PREV   LATEST   PREVROLLEDOUTREVISION   LATESTREADYREVISION   AGE
llama3-1b  http://llama3-1b-kserve-test.example.com True           100                              llama3-1b-predictor   8m

$ kubectl -n kserve-test get pods -l serving.kserve.io/inferenceservice=llama3-1b
NAME                                    READY   STATUS    RESTARTS   AGE
llama3-1b-predictor-6d6dfc7f47-7m92p    2/2     Running   0          8m
llama3-1b-predictor-6d6dfc7f47-h9p7x    2/2     Running   0          8m
llama3-1b-predictor-6d6dfc7f47-vx8nd    2/2     Running   0          8m

Logs:

kubectl -n kserve-test logs deploy/llama3-1b-predictor  -f

6.3 Deploy the InferencePool and endpoint picker

Run these commands on the EC2 instance from /home/ubuntu:

kubectl apply -f scenarios/kserve/gaie_epp_llmd.yaml
kubectl -n kserve-test get inferencepool llama3-1b
kubectl -n kserve-test get deploy,svc llama3-1b-epp

Creates:

InferencePool/llama3-1b
Deployment/llama3-1b-epp
Service/llama3-1b-epp
llm-d prefix-cache, KV-cache-utilization, queue, and max-score picker plugins
RBAC for pod discovery and metrics

Example result:

$ kubectl -n kserve-test get inferencepool llama3-1b
NAME       AGE
llama3-1b  52s

$ kubectl -n kserve-test get deploy,svc llama3-1b-epp
NAME                             READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/llama3-1b-epp    1/1     1            1           52s

NAME                    TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                     AGE
service/llama3-1b-epp   ClusterIP   10.96.121.184   <none>        9002/TCP,5557/TCP,9090/TCP   52s

Logs:

kubectl -n kserve-test logs deploy/llama3-1b-epp -c epp --tail=100
kubectl -n kserve-test logs deploy/llama3-1b-epp -c vllm-render --tail=100

6.4 Deploy the Gateway

Run these commands on the EC2 instance from /home/ubuntu:

kubectl apply -f scenarios/kserve/envoy_gw.yaml
kubectl get gatewayclass envoy-ai-gateway-basic
kubectl -n default get gateway envoy-ai-gateway-basic
kubectl -n envoy-gateway-system get svc

Example output:

envoyproxy.gateway.envoyproxy.io/envoy-ai-gateway-basic created
gatewayclass.gateway.networking.k8s.io/envoy-ai-gateway-basic created
gateway.gateway.networking.k8s.io/envoy-ai-gateway-basic created
clienttrafficpolicy.gateway.envoyproxy.io/client-buffer-limit created

NAME                     CONTROLLER                                      ACCEPTED   AGE
envoy-ai-gateway-basic   gateway.envoyproxy.io/gatewayclass-controller   True       35s

NAME                     CLASS                    ADDRESS   PROGRAMMED   AGE
envoy-ai-gateway-basic   envoy-ai-gateway-basic             True         35s

NAME                                               TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
envoy-default-envoy-ai-gateway-basic-196ae44c      LoadBalancer   10.96.80.112    <pending>     80:31994/TCP   23s

Creates:

EnvoyProxy/envoy-ai-gateway-basic
GatewayClass/envoy-ai-gateway-basic
Gateway/envoy-ai-gateway-basic
ClientTrafficPolicy/client-buffer-limit

6.5 Deploy the AI Gateway route

Run these commands on the EC2 instance from /home/ubuntu:

kubectl apply -f scenarios/kserve/ai_gw_route.yaml
kubectl -n kserve-test get aigatewayroute envoy-ai-gateway-basic

Creates:

AIGatewayRoute/envoy-ai-gateway-basic
Header match: x-ai-eg-model: llama3-1b
Backend: InferencePool/llama3-1b
Token accounting metadata

Expected route output:

NAME                     STATUS     AGE
envoy-ai-gateway-basic   Accepted   18s

At this point the Envoy Gateway data plane service exists:

kubectl -n envoy-gateway-system get svc

Example output:

NAME                                               TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
envoy-default-envoy-ai-gateway-basic-196ae44c      LoadBalancer   10.96.80.112    <pending>     80:31994/TCP   2m

The cluster has no LoadBalancer implementation, so EXTERNAL-IP stays <pending>. Use the implicit NodePort instead. From the EC2 instance use localhost:<nodePort>; from your laptop use <ec2-public-ip>:<nodePort> if your source IP is allowed in local_ranges.

7. Test with a basic OpenAI request

Find the Envoy Gateway NodePort from the EC2 instance:

kubectl -n envoy-gateway-system get svc

Look for the 80:<nodePort>/TCP mapping in the Envoy Gateway service. In the example below, the NodePort is 31994:

NAME                                               TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
envoy-default-envoy-ai-gateway-basic-196ae44c      LoadBalancer   10.96.80.112    <pending>     80:31994/TCP   2m

From the EC2 instance:

curl -s http://localhost:<nodePort>/v1/chat/completions \
  -H 'content-type: application/json' \
  -H 'x-ai-eg-model: llama3-1b' \
  -d '{
    "model": "llama3-1b",
    "messages": [
      {"role": "user", "content": "Write one sentence about Kubernetes and GPUs."}
    ],
    "max_tokens": 64,
    "temperature": 0.2
  }' | jq

Example output:

{
  "id": "chatcmpl-7f8d0f37b8b34b6d9d7f8c7a3f4f21d4",
  "object": "chat.completion",
  "created": 1763421000,
  "model": "llama3-1b",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Kubernetes can schedule GPU-enabled workloads so AI models can run efficiently across accelerated infrastructure."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 20,
    "completion_tokens": 16,
    "total_tokens": 36
  }
}

From your laptop, if your IP is allowed in local_ranges:

curl -s http://<ec2-public-ip>:<nodePort>/v1/chat/completions \
  -H 'content-type: application/json' \
  -H 'x-ai-eg-model: llama3-1b' \
  -d '{
    "model": "llama3-1b",
    "messages": [
      {"role": "user", "content": "Write one sentence about Kubernetes and GPUs."}
    ],
    "max_tokens": 64,
    "temperature": 0.2
  }' | jq

Optional: apply token rate limiting

You can apply rate_limit.yaml to enforce an Envoy Gateway token-aware rate limit for the llama3-1b route.

Run these commands on the EC2 instance from /home/ubuntu:

kubectl apply -f scenarios/kserve/rate_limit.yaml
kubectl -n default get backendtrafficpolicy model-specific-token-limit-policy

Example output:

backendtrafficpolicy.gateway.envoyproxy.io/model-specific-token-limit-policy created

NAME                                AGE
model-specific-token-limit-policy   12s

The policy attaches to Gateway/envoy-ai-gateway-basic and limits requests matching x-ai-eg-model: llama3-1b. It also uses x-user-id as the distinct client selector, so include that header when testing:

curl -kv http://localhost:<nodePort>/v1/chat/completions \
  -H 'content-type: application/json' \
  -H 'x-ai-eg-model: llama3-1b' \
  -H 'x-user-id: test-user-1' \
  -d '{
    "model": "llama3-1b",
    "messages": [
      {"role": "user", "content": "Write a longer answer that consumes output tokens."}
    ],
    "max_tokens": 256,
    "temperature": 0.2
  }' | jq

Repeat requests with the same x-user-id to validate the limit behavior. Once the policy is enforced, repeated requests for the same user can return 429 HTTP response:

* Connected to localhost (127.0.0.1) port 31186
> POST /v1/chat/completions HTTP/1.1
> Host: localhost:31186
> User-Agent: curl/8.5.0
> Accept: */*
> content-type: application/json
> x-ai-eg-model: llama3-1b
> x-user-id: test-user-1
> Content-Length: 184
> 
} [184 bytes data]
< HTTP/1.1 429 Too Many Requests
< x-went-into-resp-headers: true
< x-ratelimit-limit: 1000, 1000;w=60
< x-ratelimit-remaining: 0
< x-ratelimit-reset: 21
< date: Tue, 18 Aug 2026 03:43:39 GMT
< content-length: 0}
}

Remove the rate-limiting policy to continue

kubectl delete -f scenarios/kserve/rate_limit.yaml

8. Test with aiperf

Ansible installs aiperf in a virtual environment at:

/root/aiperf-venv

Activate it on the EC2 instance:

sudo -i
source /root/aiperf-venv/bin/activate

aiperf downloads the Hugging Face tokenizer, so authenticate Hugging Face inside this environment first:

hf auth login --token <token>

Use the Envoy Gateway NodePort as the --url target, for example localhost:<nodePort>.

Synthetic profile

aiperf profile \
  --model llama3-1b \
  --tokenizer meta-llama/Llama-3.2-1B-Instruct \
  --endpoint-type chat \
  --streaming \
  --url localhost:<nodePort> \
  --request-count 10000 \
  --concurrency 300 \
  --isl 1000 \
  --osl 100

Example summary output:

                                             NVIDIA AIPerf | LLM Metrics: Effective                                              
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃                                    Metric ┃        avg ┃  min ┃        max ┃        p99 ┃        p90 ┃        p50 ┃       std ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━┩
│          Effective Concurrency (requests) │     297.21 │ 0.00 │     300.00 │     300.00 │     300.00 │     300.00 │     22.71 │
│  Effective Decode Throughput (tokens/sec) │  11,465.43 │ 0.00 │  45,044.63 │  27,502.59 │  20,163.18 │   8,726.55 │  5,733.95 │
│ Effective Prefill Throughput (tokens/sec) │ 127,715.46 │ 0.00 │ 614,770.97 │ 282,309.25 │ 197,395.45 │ 124,179.60 │ 59,008.88 │
│   Effective Decode Concurrency (requests) │     241.80 │ 0.00 │     300.00 │     297.00 │     285.00 │     253.00 │     44.53 │
│  Effective Prefill Concurrency (requests) │      55.41 │ 0.00 │     300.00 │     184.00 │     112.00 │      45.00 │     41.50 │
│   Effective Total Throughput (tokens/sec) │ 139,180.89 │ 0.00 │ 614,770.97 │ 289,936.07 │ 206,948.48 │ 135,908.53 │ 57,423.50 │
│      Effective Decode Throughput Per User │      47.41 │ 0.00 │     385.41 │     114.73 │      78.15 │      36.89 │     22.41 │
│                         (tokens/sec/user) │            │      │            │            │            │            │           │
│     Effective Prefill Throughput Per User │   3,032.32 │ 0.00 │  13,246.96 │   8,002.73 │   5,043.77 │   2,789.88 │  1,537.58 │
│                         (tokens/sec/user) │            │      │            │            │            │            │           │
│                 Tokens In Flight (tokens) │ 308,490.13 │ 0.00 │ 321,045.14 │ 317,515.89 │ 315,453.99 │ 310,853.11 │ 23,577.16 │
│         Effective Latency (CO-aware) (ms) │   2,327.70 │ 1.30 │   4,322.09 │   3,688.50 │   3,003.04 │   2,454.59 │    685.08 │
└───────────────────────────────────────────┴────────────┴──────┴────────────┴────────────┴────────────┴────────────┴───────────┘

Mooncake trace profile

curl -o mooncake_trace.jsonl \
  https://raw.githubusercontent.com/kvcache-ai/Mooncake/refs/heads/main/FAST25-release/arxiv-trace/mooncake_trace.jsonl

aiperf profile \
  --model llama3-1b \
  --tokenizer meta-llama/Llama-3.2-1B-Instruct \
  --endpoint-type chat \
  --streaming \
  --url localhost:<nodePort> \
  --input-file mooncake_trace.jsonl \
  --custom-dataset-type mooncake_trace \
  --fixed-schedule \
  --concurrency 25

After running the Mooncake trace profile, open the preconfigured Grafana dashboard: (Credentials admin/admin)

http://<ec2-public-ip>:30091

Name		Name	Last commit message	Last commit date
ansible		ansible
config		config
img		img
scenarios/kserve		scenarios/kserve
.gitignore		.gitignore
.pre-commit-config.yaml		.pre-commit-config.yaml
CLAUDE.md		CLAUDE.md
README.md		README.md
ansible.cfg		ansible.cfg
compute.tf		compute.tf
main.tf		main.tf
networking.tf		networking.tf
outputs.tf		outputs.tf
variables.auto.tfvars		variables.auto.tfvars
variables.tf		variables.tf

Navigation Menu

Repository files navigation

Generative AI Stack Playground

1. Stack components and versions

2. Project layout

3. Terraform variables

4. Deploy the stack

Create the SSH key pair first

Run Terraform

GPU availability

5. Validate the running cluster

6. Configure inference workloads

6.1 Create namespace and HF secret

6.2 Deploy the KServe InferenceService

6.3 Deploy the InferencePool and endpoint picker

6.4 Deploy the Gateway

6.5 Deploy the AI Gateway route

7. Test with a basic OpenAI request

Optional: apply token rate limiting

8. Test with aiperf

Synthetic profile

Mooncake trace profile

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

Generative AI Stack Playground

1. Stack components and versions

2. Project layout

3. Terraform variables

4. Deploy the stack

Create the SSH key pair first

Run Terraform

GPU availability

5. Validate the running cluster

6. Configure inference workloads

6.1 Create namespace and HF secret

6.2 Deploy the KServe InferenceService

6.3 Deploy the InferencePool and endpoint picker

6.4 Deploy the Gateway

6.5 Deploy the AI Gateway route

7. Test with a basic OpenAI request

Optional: apply token rate limiting

8. Test with aiperf

Synthetic profile

Mooncake trace profile

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages