This guide deploys vtafarm on Hetzner Cloud. You start with an empty Hetzner project and finish with a running application. It builds five layers, in this order:
- a 3-node high-availability k3s cluster
- Rancher, running on that k3s cluster
- an RKE2 cluster that Rancher creates for a farm
- the platform inside that cluster: cert-manager, Longhorn and HashiCorp Vault
- the vtafarm frontend and API
Each layer is one OpenTofu stack. You build layers 1 and 2 once. Layers 3 to 5 are one farm, so repeat steps 6 to 8 for every farm you need. Vault keeps the master seed of every VTA encrypted and separated per user, which is what makes a farm safe enough to run vtafarm on. See docs/vault.md.
internet
│
┌─────────────▼──────────────┐
│ Hetzner Load Balancer │
│ :6443 Kubernetes API │ → servers
│ :80 HTTP ─┐ │ → all nodes
│ :443 HTTPS ─┘ Traefik │
└─────────────┬──────────────┘
10.0.1.10
┌───────────────────────────┴────────────────────────────┐
│ Hetzner private network 10.0.1.0/24 │
│ │
│ server-1 server-2 server-3 │
│ 10.0.1.101 10.0.1.102 10.0.1.103 │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ k3s │ │ k3s │ │ k3s │ │
│ │ + etcd │◄────►│ + etcd │◄────►│ + etcd │ │
│ │ Traefik │ │ Traefik │ │ Traefik │ │
│ │ Rancher │ │ Rancher │ │ Rancher │ │
│ └──────────┘ └──────────┘ └──────────┘ │
│ placement group = spread (separate hosts) │
└────────────────────────────┬───────────────────────────┘
│
▼ etcd snapshot every 6h
Hetzner Object Storage (S3)
The farm Vault stores the master seed of every VTA. It cannot unseal itself after a restart, so a second Vault does that for it. This transit Vault runs on one node and holds one key:
internet
│
┌─────────────▼───────────────┐
│ Hetzner Load Balancer │
│ :6443 Kubernetes API │ → servers
│ :9345 RKE2 registration │ → servers
│ :80 :443 Traefik │ → all nodes
└─────────────┬───────────────┘
┌───────────────────────────┴─────────────────────────────┐
│ Hetzner private network 10.10.1.0/24 │
│ │
│ server-1 server-2 server-3 │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ RKE2 │ │ RKE2 │ │ RKE2 │ │
│ │ + etcd │◄────►│ + etcd │◄────►│ + etcd │ │
│ │ Longhorn │◄────►│ Longhorn │◄────►│ Longhorn │ │
│ │ vault-0 │◄────►│ vault-1 │◄────►│ vault-2 │ │
│ └──────────┘ └──────────┘ └──────────┘ │
│ namespace vault — Raft, HA, auto-unsealed │
│ │ │
│ │ seal "transit" │
│ ▼ │
│ ┌───────────────────────────────────────────┐ │
│ │ namespace vault-transit │ │
│ │ vault-transit-0 holds `autounseal` │ │
│ │ Shamir-sealed, unsealed by hand │ │
│ │ NetworkPolicy: reachable from ns/vault │ │
│ └───────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────┘
A stack cannot run until the stack before it exists. You therefore apply them in order:
| Stack | Waits for |
|---|---|
01-infra |
nothing |
02-rancher |
stack 01's kubeconfig |
03-rke2-clusters |
Rancher to be reachable |
04-vtafarm-platform |
stack 03's cluster to be Active |
05-vtafarm-app |
stack 04's Vault to be bootstrapped |
Stacks 03, 04 and 05 keep one directory per cluster, under clusters/<name>. Each directory
has its own state file, and each state file is locked separately, so creating or destroying one
cluster does not affect the others. All of them live in the bucket, not on your laptop; see
docs/remote-state.md.
The full directory tree and the reasons behind this layout are in docs/design-decisions.md.
Collect all of this before you start.
- Tools —
brew install opentofu kubectl helm jq. OpenTofu must be 1.12 or newer - Hetzner API token, Read & Write — Console → Security → API tokens. Shown once
- SSH key pair — the one you already use, or
ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519 - A DNS zone you can edit — at your registrar. The farm domain in step 8 must be on Cloudflare, see below
- A private Object Storage bucket in Nuremberg (
nbg1) — Console → Object Storage → Create Bucket. The name must be globally unique. Object Lock disabled, Visibility private - S3 credential pair — Console → Security → S3 credentials. The secret is shown once
vtafarm itself, in step 8, also needs:
- Cloudflare API token and zone ID — from the Cloudflare dashboard, for the zone of your domain. vtafarm creates tenant domains automatically, and Cloudflare is the only DNS provider it supports today
did:keykeypair —make gen-keypairin the vtafarm-api repo
State lives in the bucket, not on your laptop, so the first file to fill in is .env. It names
the bucket, the folder to keep state and tfvars under, and the S3 credentials. Every make
target reads it:
cp .env.example .env
code .envPrepare the bucket once. This enables versioning and a retention rule, so a bad apply can be rolled back and the etcd snapshots sharing the bucket do not pile up:
make state-bucket-setupThen configure the stack itself:
cd stacks/01-infra
cp terraform.tfvars.example terraform.tfvars
code terraform.tfvarsGo through every value in the file and read the comments above them. They say what each value does and which ones you have to fill in. Continue with step 2 when the file is complete.
Once it is, make tfvars-push uploads it to the bucket, where a colleague picks it up with
make tfvars-pull. Do that for every stack you configure from here on.
docs/remote-state.md covers the whole arrangement.
cd ../..
make init
make applyThis takes about 5–8 minutes. OpenTofu looks stuck at null_resource.kubeconfig. It is
waiting for the first server to finish its bootstrap, so this is normal.
OpenTofu writes the kubeconfig to stacks/01-infra/kubeconfig.yaml:
export KUBECONFIG=$PWD/stacks/01-infra/kubeconfig.yaml
kubectl get nodes -o wideYou can also merge it into ~/.kube/config and use it as one more context:
make kubeconfig-merge
kubectl config use-context k3s-rancherIn both cases you should see three Ready nodes with the roles
control-plane,etcd,master:
NAME STATUS ROLES AGE VERSION
k3s-rancher-server-1 Ready control-plane,etcd,master 4m v1.35.7+k3s1
k3s-rancher-server-2 Ready control-plane,etcd,master 3m v1.35.7+k3s1
k3s-rancher-server-3 Ready control-plane,etcd,master 2m v1.35.7+k3s1
Save the join token:
make tokenStore it in your password manager now. The token lets new nodes join, and it also encrypts the secret data inside etcd, so you cannot restore an etcd snapshot without it.
make outputsTake the load_balancer_ipv4 value and create this record:
rancher.yourdomain.com. A <load_balancer_ipv4>
Wait until DNS has propagated before you continue. If you do not, the Let's Encrypt challenge fails:
dig +short rancher.yourdomain.comcd stacks/02-rancher
cp terraform.tfvars.example terraform.tfvars
code terraform.tfvarsGo through every value in the file and read the comments above them. Reuse the bucket and the S3 credentials from step 1.
cd ../..
make apply-rancherThis takes 5–10 minutes: cert-manager first, then Rancher, then the certificate. READY
must be True:
kubectl -n cattle-system get certificatemake rancher-password
open https://rancher.yourdomain.comLog in as admin with that password. Rancher asks you to change it at the first login.
Create a Rancher API key in the user menu, under Account & API Keys. Then generate an OpenTofu directory for the new cluster. The name you pass is both the cluster name in Rancher and the directory name:
make new-rke2-cluster CLUSTER=rke2-vtafarm-production
code stacks/03-rke2-clusters/clusters/rke2-vtafarm-production/terraform.tfvarsGo through every value in the file and read the comments above them. The scaffold already
wrote terraform.tfvars for you, so there is nothing to copy.
For a disposable single-node development cluster, set the following values. Dev mode creates
one all-in-one node using server_type, without a Hetzner private network or load balancer.
It exposes ingress directly on the node. ssh_allowed_cidrs controls both SSH and Kubernetes
API access and may be left open for a disposable cluster:
dev = true
server_type = "cx33"
ssh_allowed_cidrs = ["0.0.0.0/0"]Omit dev or leave it false for the normal highly available topology. A dev cluster has no
control-plane redundancy and is not suitable for production.
make init-rke2 CLUSTER=rke2-vtafarm-production
make apply-rke2 CLUSTER=rke2-vtafarm-productionWait until the cluster is Active in Rancher, under Cluster Management. The first apply
stores an empty, unusable kube_config, because Rancher is still creating the cluster while
OpenTofu finishes. Refresh the state once the cluster is Active:
make refresh-rke2 CLUSTER=rke2-vtafarm-productionThen write the kubeconfig and use it directly:
make kubeconfig-rke2 CLUSTER=rke2-vtafarm-production
export KUBECONFIG=$PWD/stacks/03-rke2-clusters/clusters/rke2-vtafarm-production/kubeconfig.yaml
kubectl get nodesOr merge it into ~/.kube/config and use it as one more context:
make kubeconfig-merge-rke2 CLUSTER=rke2-vtafarm-production
kubectl config use-context rke2-vtafarm-productionEither way the file lands in the cluster's directory, where stacks 04 and 05 read it.
To add a second cluster, repeat this step with a different name, for example
make new-rke2-cluster CLUSTER=rke2-vtafarm-staging. The two directories are independent.
This step installs the platform inside the RKE2 cluster: cert-manager, Longhorn and the two Vaults.
Use the same cluster name as in step 6. This stack reads the kubeconfig from that cluster's directory, so the two names must match.
make new-vtafarm-platform CLUSTER=rke2-vtafarm-production
code stacks/04-vtafarm-platform/clusters/rke2-vtafarm-production/terraform.tfvarsFill in the Longhorn backup bucket and S3 credential pair. Reuse the private bucket from step 1;
the default longhorn/<cluster-name> prefix keeps each downstream cluster separate. The other
defaults install Vault and schedule Longhorn volume backups for 00:00 UTC every day.
make init-vtafarm-platform CLUSTER=rke2-vtafarm-production
make apply-vtafarm-platform CLUSTER=rke2-vtafarm-productionThis takes 5–10 minutes.
make vault-status CLUSTER=rke2-vtafarm-productionBoth Vaults start sealed, and a sealed Vault reports itself as not ready. The transit pod
shows 0/1. The farm pods stay in CreateContainerConfigError, because they wait for a secret
that does not exist yet. This is the expected state after this apply. It is not a failure.
Initializing and unsealing a Vault produces recovery keys and root tokens. These must never be written into OpenTofu state, so you run those steps by hand. Point kubectl at the cluster first:
export KUBECONFIG=$PWD/stacks/03-rke2-clusters/clusters/rke2-vtafarm-production/kubeconfig.yaml-
Init the transit Vault. This writes the Shamir keys and its root token to
vault-init-transit.json:kubectl exec -n vault-transit vault-transit-0 -- \ vault operator init -key-shares=5 -key-threshold=3 -format=json > vault-init-transit.json
-
Unseal it. Run this three times, with a different Shamir key each time:
kubectl exec -it -n vault-transit vault-transit-0 -- vault operator unseal -
Create the unseal token for the farm Vault. Paste the root token from
vault-init-transit.jsonafter the=:export VAULT_TOKEN= make vault-bootstrap CLUSTER=rke2-vtafarm-production TARGET=transit -
Restart the farm pods so they read the new token:
kubectl -n vault rollout restart statefulset/vault
-
Init the farm Vault. Its peers unseal themselves from here on:
kubectl exec -n vault vault-0 -- vault operator init -format=json > vault-init-farm.json
-
Configure it for vtafarm-api. Paste the root token from
vault-init-farm.json:export VAULT_TOKEN= make vault-bootstrap CLUSTER=rke2-vtafarm-production TARGET=farm
⚠️ Move bothvault-init-*.jsonfiles into your password manager now, then delete the local copies. They hold the recovery keys and the root tokens. Git ignores them, which is not the same as safe. Every cluster has its own keys, and a Vault snapshot is worthless without them.
The full procedure, the tenant isolation model and the day-2 operations are in docs/vault.md.
This step installs the vtafarm applications: the frontend and the API.
make new-vtafarm-app CLUSTER=rke2-vtafarm-production
code stacks/05-vtafarm-app/clusters/rke2-vtafarm-production/terraform.tfvarsGo through every value in the file and read the comments above them. domain drives every
hostname the farm serves.
Create the two A records next. Both point at the load balancer of the cluster. cert-manager cannot issue the certificate until these names resolve:
vtafarm.yourdomain.com. A <cluster_ingress_ip>
vtafarm-api.yourdomain.com. A <cluster_ingress_ip>
make init-vtafarm-app CLUSTER=rke2-vtafarm-production
make apply-vtafarm-app CLUSTER=rke2-vtafarm-productionThis stack generates the JWT secret and the database password instead of taking them from you. They exist only in this stack's state file, so back that file up.
make outputs-vtafarm-app CLUSTER=rke2-vtafarm-productionCreate the first admin:
export KUBECONFIG=$PWD/stacks/03-rke2-clusters/clusters/rke2-vtafarm-production/kubeconfig.yaml
kubectl exec -it deployment/vtafarm-api -- ./enrollOr, if you merged the kubeconfig in step 6, switch to that context instead:
kubectl config use-context rke2-vtafarm-production
kubectl exec -it deployment/vtafarm-api -- ./enrollThe farm is now running. Everything after this point is a runbook.
| Document | Covers |
|---|---|
| docs/testing.md | health checks, detailed verification, the HA failover test |
| docs/operations.md | day-2 make targets, kubeconfig contexts, adding nodes, what is backed up |
| docs/remote-state.md | where state and tfvars live, locking, and how a second operator joins |
| docs/backup-restore.md | disaster recovery, four failure scenarios, and a drill |
| docs/cluster-migration.md | moving a live farm — database, Vault and tenants — onto a new cluster |
| docs/upgrade.md | how to upgrade k3s, Rancher, cert-manager, the OS, a vtafarm release and the providers |
| docs/vault.md | Vault init, unseal, bootstrap, the isolation model and day-2 tasks |
| docs/vault-upgrade.md | how to upgrade both Vaults and drain a node |
| docs/troubleshooting.md | read this first when something is stuck |
| docs/design-decisions.md | the repository layout, and why the topology and the version pins are what they are |
| docs/teardown.md | how to destroy one layer, one cluster, or everything |
| docs/opentofu-primer.md | a short introduction to OpenTofu, if it is new to you |
| docs/cost.md | the monthly cost, and what each scaling step adds |
VTA Farm is licensed under the Apache License 2.0. See NOTICE for attribution. Contributions are welcome under the terms in CONTRIBUTING.md.