eks-inference

An EKS cluster for inference workloads, provisioned from a local kind cluster through AWS Controllers for Kubernetes (ACK) and managed as data in ConfigHub.

Karpenter provisions GPU nodes on demand: a quantized-LLM pool on L4/A10G spot, and an H200 pool for when you have the capacity reservation to use it.

Everything is driven by config. Scaling a model up is a change to a Unit and a release, not a kubectl command.

Install

cub plugin install confighub/eks-inference
cub eksinf --help

The plugin is the admin tool for this stack. You do not need this repo checked out to use it. Needs cub v0.4.0 or newer — pre-1.0 the second version number is the API version, and v0.4 is the API this plugin speaks.

First, the component bases

cub eksinf install

This pulls the eight component bundles from their OCI registry and creates a <component>-base Space for each. It needs nothing but cub and a ConfigHub login — no cluster, no Docker, no AWS account, no cost — and it is once per ConfigHub organization, not once per stack. Bases are the shared upstream that every variant clones from, so a sandbox and a real deployment use the same ones.

Everything below starts from here. The two paths are independent: do either first, or both, in any order.

Path A: just the configuration, for free

cub eksinf sandbox up

About thirty seconds, no AWS account, no Docker, nothing to bill. You get the real configuration tree: the downstream variants for both planes, the platform-profile links that carry shared values across them, the vet-placeholders gate, and a published release per variant in ConfigHub's OCI registry. What is missing is a consumer — no Argo CD pulls those releases and no ACK controller acts on them.

It uses variant sandbox rather than dev, so it can sit alongside a real stack in the same organization; you can leave it running and build the real thing later without undoing anything. cub eksinf sandbox down --yes removes it.

Path B: the real stack

This is where clusters and cloud resources appear. Note that cub cluster up is needed before deploy — it is what produces the Target to deploy against — and not before install, so arriving here from Path A costs you nothing.

# 1. A local management cluster, wired to ConfigHub via Argo CD.
cub cluster up --name inference-mgmt

# 2. The parameter surface. Its Space has no Target and is never deployed.
#    Skip if you already have platform-profile-dev.
cub variant create dev platform-profile-base

# 3. Give the ACK controllers AWS credentials. The one out-of-band step —
#    the Secret is never a ConfigHub Unit. Do this BEFORE deploying: the
#    controllers read credentials once at startup, and a bad identity fails
#    here in a second rather than as a condition on a VPC later.
#
#    Add --profile NAME unless your default AWS identity is the right one.
#    Add --cluster inference-mgmt if you have other cub-managed clusters:
#    creds normally finds the management cluster by its ack-system namespace,
#    which does not exist yet, and it asks rather than guessing between them.
cub eksinf creds create-user --yes        # or: creds use-existing

# 4. Deploy the management plane. This creates AWS infrastructure.
cub eksinf deploy --plane mgmt --target inference-mgmt/target

# 5. Watch it converge. Measured: ~5 min for the network, ~10 for the EKS
#    control plane, ~3 more for the system nodegroup.
cub eksinf status

When the EKS cluster is ACTIVE, bring it under management and deploy the workload plane onto it:

# 6. Enroll EKS: install Argo CD, register a worker and OCI target, bootstrap
#    the root app-of-apps, and install argobot. Never creates or destroys a
#    cluster.
#
#    argobot force-syncs the matching Argo Application the moment a deploy
#    happens, so step 7 and every later release land in seconds rather than
#    waiting out Argo's reconcile interval. `cub cluster up` installs it for the
#    mgmt cluster; without it here the two planes would behave differently under
#    the same command. Pass --no-argobot to skip it.
#
#    --grant-access is needed after step 3's create-user path: ACK built the
#    cluster as its own identity, so EKS trusts THAT principal and not yours,
#    and the API server rejects you outright. It adds an access entry for your
#    identity. Omit it if you already have cluster-admin on the cluster.
cub eksinf enroll cluster --name inference-demo \
  --eks-cluster inference-demo --region us-west-2 --grant-access

# 7. Deploy Karpenter, the GPU runtime, and the workloads.
cub eksinf deploy --plane workload --target inference-demo/target

deploy links each plane to the platform-profile itself, between creating the variants and publishing them — it has to, since publishing is gated on there being no unfilled placeholders left. cub eksinf link-profile is still there to inspect or rework the links after the fact:

cub eksinf link-profile --list      # show the bindings and existing links
cub eksinf link-profile --unlink    # remove them

No GPU is running yet: every workload that wants one ships at replicas: 0, so installing costs nothing. To actually provision a GPU:

cub function do --space inference-workloads-dev --where "Slug = 'smoke-gpu'" set-replicas 1
cub release publish inference-workloads-dev

Karpenter launches a g6.xlarge in about 90 seconds. Scale back to 0 and it is released. Do not use kubectl scale — the Argo Application syncs with selfHeal: true, so a manual scale is reverted within a minute, having reported success.

Talking to the model

Scale up the model itself, wait for it to serve, then chat with it:

cub function do --space inference-workloads-dev --where "Slug = 'vllm-qwen'" set-replicas 1
cub release publish inference-workloads-dev

# ~10 min: Karpenter launches a node (~45s), it pulls a multi-GB image, then
# vLLM downloads the model. Watch it with:
kubectl --kubeconfig ~/.confighub/clusters/inference-demo.kubeconfig \
  get pods -n inference -w

# Then, from inside the cluster — no port-forward, nothing exposed:
kubectl --kubeconfig ~/.confighub/clusters/inference-demo.kubeconfig \
  exec -it -n inference deploy/chat -- chat

-it matters; without it you get no prompt. /reset clears the conversation, /quit exits. The client is the one thing here that ships at replicas: 1 — it is a 10m-CPU pod on the system nodegroup, so it costs nothing and is always there to talk to. Before the model is up it reports a refused connection rather than hanging.

The endpoint is OpenAI-compatible, so if you would rather see the wire format:

kubectl --kubeconfig ~/.confighub/clusters/inference-demo.kubeconfig \
  exec -n inference deploy/chat -- \
  curl -sS http://vllm-qwen.inference.svc.cluster.local:8000/v1/models

Scale vllm-qwen back to 0 when you are done — it holds the g6.xlarge.

Taking it down

Path A needs none of this — it created no infrastructure. cub eksinf sandbox down --yes and you are done.

Path B bills ~$130/month idle, so taking it down is part of the walkthrough rather than an appendix:

# Destroys the AWS resources, in the only safe order: workload plane, then mgmt.
# Roughly 20-35 minutes, most of it waiting for EKS and the NAT gateway.
cub eksinf teardown --yes --profile <aws-profile>

# The two local pieces teardown deliberately leaves to you.
cub eksinf creds delete-user --yes        # the IAM user, if you created one
cub cluster down --name inference-mgmt    # the kind cluster

teardown goes THROUGH config rather than around it — deleting Kubernetes objects does not work here, because Argo restores them and the ACK controllers run with deletionPolicy: retain, so even a delete that survives leaves the AWS resource running. It verifies against EC2 at the end rather than trusting the Kubernetes view.

Add --delete-config to remove the ConfigHub Spaces too; without it the config survives and cub eksinf deploy can rebuild onto a fresh cluster. See docs/teardown.md.

Hardening this for production

This stack ships with a full config-driven lifecycle: Argo prunes, and the ACK controllers run deletionPolicy: delete. Config decides whether a resource exists, so removing a Unit removes the AWS resource it describes. Deploy creates, un-deploy destroys, and nothing is asymmetric.

That is right for a demo you stand up and tear down repeatedly. It is probably wrong for infrastructure you would be sad to lose, where you want destruction to be hard and deliberate.

cub eksinf has no flag for any of this, and does not need one. It wrote the configuration into ConfigHub and stopped; the configuration is yours. Every knob below is a cub command against config the plugin does not own.

Replace inference-demo and dev with your cluster name and variant.

# 1. Stop Argo deleting resources when they leave the config.
cub function do --space inference-demo-argo-apps --where "Slug LIKE '%-dev'" \
  set-bool-path argoproj.io/v1alpha1/Application spec.syncPolicy.automated.prune false
cub release publish inference-demo-argo-apps

# 2. Stop ACK deleting the AWS resource when its object goes away.
#    SET THESE TWO TOGETHER. Prune with retain is the worst combination: Argo
#    deletes the Kubernetes object and the AWS resource silently survives,
#    orphaned and billing, with nothing left that knows it exists.
cub function do --space ack-controllers-dev --where "Slug LIKE '%-controller'" \
  set-env-var controller DELETION_POLICY retain
cub release publish ack-controllers-dev

# 3. Refuse a sync that would empty an Application entirely — the blast radius
#    of a bad bundle, rather than of a single bad Unit.
cub function do --space inference-demo-argo-apps --where "Slug LIKE '%-dev'" \
  set-bool-path argoproj.io/v1alpha1/Application spec.syncPolicy.automated.allowEmpty false
cub release publish inference-demo-argo-apps

# 4. Protect the expensive, slow-to-rebuild things individually. A delete gate
#    refuses deletion of the Unit itself, so it survives a --recursive space
#    delete and anything else that sweeps broadly.
cub unit update --space eks-cluster-dev cluster  --delete-gate protected=true
cub unit update --space aws-network-dev network  --delete-gate protected=true

Verify what you changed, rather than trusting that you did:

cub unit data eks-cluster-dev --space inference-demo-argo-apps | grep -A4 syncPolicy
cub unit data ec2-controller --space ack-controllers-dev | grep -A1 DELETION_POLICY

Note that cub eksinf deploy re-enables prune on every run, because it is the behaviour this stack's teardown assumes. If you have hardened a deployment, run step 1 again after any deploy — or stop using deploy for that variant and drive it with cub variant create and cub release publish directly, which is all deploy was doing on your behalf.

The two apply planes

The single fact that shapes this repo: kind and EKS are different apply targets. Components are separated by which cluster applies them before they are separated by anything else.

  kind (cub cluster up)              AWS                  EKS (cub eksinf enroll)
  ─────────────────────              ───                  ──────────────────────
  ack-controllers  ──────────────▶   VPC, subnets, NAT
  aws-network      ──────────────▶   IAM roles            karpenter
  eks-cluster      ──────────────▶   EKS control plane    gpu-runtime
  karpenter-aws    ──────────────▶   Karpenter IAM        inference-workloads

Ownership is split, never migrated. kind keeps the provisioning plane permanently; the workload plane belongs to EKS from the day it is written. Since nothing moves between planes there is no adoption step, and no way for the cluster to delete itself.

Karpenter is the case that proves the point: its IAM role and Pod Identity association are ACK resources only kind can create, while its controller and NodePools run on EKS. One component in each plane.

cub eksinf components lists them.

How values cross component boundaries

Three mechanisms, three different times. Choosing the wrong one is the main way this goes wrong.

Mechanism	Carries	When
ConfigHub links	names, CIDRs, tags, AMI aliases	config time, in the hub
ACK *Ref fields	actual AWS IDs (vpc-0a1b…)	runtime, in-cluster
Argo sync waves	apply ordering	apply time

AWS IDs do not exist at config time, so ConfigHub cannot propagate them — ACK resolves object names to IDs itself. What ConfigHub is for is the values that must agree across components, of which karpenter.sh/discovery is the sharpest example: it spans two planes, and a mismatch produces no error at all, just a Karpenter that never launches a node.

Links span the plane boundary; sync waves cannot. A single edit to the platform-profile Unit reaches components applied by two different clusters. Ordering between planes is not expressible in config — deploy mgmt, let it converge, then deploy workload.

See docs/dependencies.md.

What this costs

Idle, in us-west-2, with nothing scheduled:

Resource	Approx. monthly
EKS control plane	$73
NAT gateway (1)	$33 + data processing
2 × t4g.medium	$24
Total	~$130/month

GPU nodes are on top of that and only exist while a workload asks for one: roughly $0.80/hr for a g6.xlarge, and tens of dollars an hour for H200.

cub eksinf status reports what is running from EC2, not from Kubernetes — a Node object can outlive its instance, and an unreachable cluster reports zero nodes, so kubectl is wrong in both directions.

The ACK controllers run with deletionPolicy: retain, so deleting Units or letting Argo prune them does not delete AWS resources. Teardown is deliberate — see docs/teardown.md.

Developing this repo

Only needed to change the config itself or cut a release.

make render    # helm charts + handwritten CRs -> configs/
make verify    # fail if configs/ drifts from sources (CI gate)
make bundles   # configs/ -> dist/<component>.tar.gz
make push      # -> ghcr.io/confighub/configs/eks-inference/<component>:latest
make plugin    # build ./eksinf locally
make check     # go vet + go test + gofmt, as CI runs them

CI runs exactly these targets; there is no build logic in the workflow files.

The rendered output in configs/ is committed on purpose: it makes a chart version bump reviewable as a diff, and it is what the OCI bundles contain. See docs/flattening.md for what is lost when a Helm chart is flattened into literal YAML, and how the build guards against it.

Two release cadences, deliberately independent:

• config bundles float at :latest, republished on every push to main
• the plugin is cut from a v* tag as a GitHub release

Layout

components.yaml           the component set: name, plane, order. Embedded in the plugin.
versions.env              pinned chart versions and render inputs
Makefile                  build entry points
main.go, embed.go, cmd/   the eksinf plugin
scripts/render.sh         helm template + copy -> configs/
scripts/guard.sh          rejects Helm constructs that do not survive flattening
scripts/bundle.sh         reproducible tarballs + oras push
src/                      sources: chart values and handwritten ACK resources
configs/                  rendered output (committed; one file per ConfigHub Unit)
iam/                      the IAM policy `creds create-user` attaches

File names in configs/ are an interface: bundles install with --granularity per-file, so each file becomes one Unit and renaming a file renames a Unit.

Taking a newer bundle

cub eksinf install

Re-running install takes the current bundles. A re-upload 3-way merges the new bundle against the last one, so Unit IDs, target bindings and links survive, and so do changes made in ConfigHub afterwards — which matters here, because link-profile and the set-env-var setters mutate Units after upload. A bundle that has not moved is a no-op.

Bases are upstreams, so taking a newer one does not move anything that is deployed. Promote per variant when you want it:

cub variant promote <component>-dev
cub release publish <component>-dev

--prune additionally empties Units the bundle no longer produces; --recreate deletes and rebuilds, which is only needed to change granularity. See docs/install.md.

Versioning the plugin

0.MINOR.PATCH, and the distinction is not decorative — the number is the only thing a reader has to go on when deciding whether an upgrade needs attention.

• PATCH — fixes. Something did not work and now does. This is the default, and a release of nothing but fixes is a patch release however many commits it contains.
• MINOR — new commands or flags, or a change that can make a previously working invocation fail: a new gate, a renamed flag, a different default.

Pre-1.0 lets us change anything at any time; it is not a licence to make the version meaningless. Bumping MINOR by reflex is how that happens, and v0.7.0 is an example — eight commits, all fixes, tagged as though it added something.

Releases are cut from a tag:

git tag v0.7.1 && git push origin v0.7.1

If GitHub is not processing push events (it happens), dispatch the workflow against the TAG rather than main — the version is derived from the ref, so dispatching against main builds a plugin that calls itself main:

gh workflow run release-plugin.yml --ref v0.7.1

Docs

• dependencies.md — how values cross component boundaries, and the path-escaping trap
• flattening.md — why Helm is rendered to literal YAML, what breaks, how the guard works
• aws-credentials.md — credential modes, the IAM policy, SSO sessions
• karpenter.md — the node pools, why the GPU AMI is pinned, the interruption queue
• install.md — what the install and deploy commands actually do
• teardown.md — deleting AWS resources, given deletionPolicy: retain