██████╗███████╗██████╗ ██╗ ██╗ ██╗ █████╗ ██████╗
██╔════╝██╔════╝██╔══██╗██║ ██║ ██║ ██╔══██╗██╔══██╗
██║ █████╗ ██████╔╝███████║ ██║ ███████║██████╔╝
██║ ██╔══╝ ██╔═══╝ ██╔══██║ ██║ ██╔══██║██╔══██╗
╚██████╗███████╗██║ ██║ ██║ ███████╗██║ ██║██████╔╝
╚═════╝╚══════╝╚═╝ ╚═╝ ╚═╝ ╚══════╝╚═╝ ╚═╝╚═════╝
A fully-gitopsed Rook/Ceph playground on k3s - a sandbox to play around with ceph, kubernetes, Cilium, et cetera. Ceph traffic prettily lit up in Hubble.
| Layer | Technology | Notes |
|---|---|---|
| VMs | Lima + Apple Virtualization.framework | Ubuntu 24.04, 4 nodes, vmType: vz |
| Kubernetes | k3s v1.33 | Single binary, SQLite, ~500 MB RAM |
| CNI + LB | Cilium 1.19.3 | eBPF, kube-proxy free, Gateway API, L2 LB, Hubble |
| Storage | Rook v1.19.1 + Ceph Squid v19.2.2 | RBD, CephFS, RGW (S3) |
| GitOps | ArgoCD stable | Kustomize-Helm, sync waves, insecure (TLS at gateway) |
| TLS | cert-manager v1.14.5 | Self-signed CA ceph-lab-ca, wildcard *.ceph.lab |
| Observability | Prometheus, Grafana, Sloth, Loki, Tempo, OTel | Standalone charts, SLO burn-rates, Logs, S3 traces, Hubble L7 flows |
┌─────────────────────────────────────────────────┐
│ Mac host 192.168.56.1 │
│ *.ceph.lab → 192.168.56.200 (via dnsmasq) │
└────────────┬────────────────────────────────────┘
│ host-only network (192.168.56.0/24)
│ (socket_vmnet — Lima ceph-lab network)
┌─────────▼──────────┐
│ ceph-control :50 │ k3s server 2 vCPU / 6 GiB
└─────────┬──────────┘
│
┌─────────▼──────────────────────────────┐
│ ceph-node-1/2/3 :61/:62/:63 │
│ k3s agents + Ceph OSDs │
│ 3 vCPU / 8 GiB each │
│ /dev/vdc + /dev/vdd (5 GiB, raw) │
└────────────────────────────────────────┘
│
┌─────────▼──────────────────────────────┐
│ Cilium Gateway 192.168.56.200 │
│ LB pool 192.168.56.192/27 │
└────────────────────────────────────────┘
- macOS 13+ (Apple Silicon or Intel)
- Lima —
brew install lima - socket_vmnet —
brew install socket_vmnetthenmake setuponce brew install helm kubectl jq yq
# 1. Clone this repo
git clone https://github.com/YOUR_USERNAME/ceph-lab.git
cd ceph-lab
# 2. One-time host setup (Lima networks, socket_vmnet)
make setup
# 2.5 (Optional) Configure passwordless sudo for local DNS and network management
# This prevents password prompts when starting/stopping the environment
make setup-sudoers
# 3. Configure your environment
cp .env.example .env
# Edit .env — at minimum set:
# GITOPS_REPO_URL=git@github.com:YOUR_USERNAME/ceph-lab.git
# (plus SSH deploy key or HTTPS token)
# 4. Boot the cluster (~10–20 min first run)
make up
# This creates 4 Lima VMs, provisions k3s + Cilium, merges kubeconfig,
# configures dnsmasq, and bootstraps ArgoCD — all in one shot.
# 5. Watch ArgoCD sync the world
kubectl get applications -n argocd -w --context ceph-lab
# 6. Check Ceph health (once rook waves settle, ~5–10 min)
kubectl exec -it -n rook-ceph deploy/rook-ceph-tools -- ceph status| Service | URL | Credentials |
|---|---|---|
| ArgoCD | https://argocd.ceph.lab | admin / see .env (ARGOCD_BCRYPT_PASSWORD) |
| Ceph Dashboard | https://dashboard.ceph.lab | admin / kubectl -n rook-ceph get secret rook-ceph-dashboard-password -o jsonpath='{.data.password}' | base64 -d |
| Grafana | https://grafana.ceph.lab | admin / prom-operator |
| Hubble UI | https://hubble.ceph.lab | — |
| Prometheus | https://prometheus.ceph.lab | — |
| Alertmanager | https://alertmanager.ceph.lab | — |
| S3 (objectstore) | https://s3.ceph.lab | — |
| S3 (shared objectstore) | https://s3-shared.ceph.lab | — |
Run bash provisioning/scripts/open_urls.sh for a live summary including credentials.
Everything is deployed in dependency order — no manual sequencing needed:
| Wave | Component | Why |
|---|---|---|
| -15 | gateway-api CRDs | Must exist before Cilium starts |
| -10 | cilium | Gateway CRDs must precede; creates GatewayClass |
| -6 | prometheus-operator-crds | CRDs before the stack |
| -5 | grafana, prometheus, tempo | Observability backbone (metrics, tracing) |
| 0 | otel-collector | OpenTelemetry pipeline for traces |
| 1 | l7-policies | CiliumNetworkPolicies (Cilium must exist) |
| 5 | topology-catalog, loki | Static catalog-info.yaml ConfigMap, Log aggregation |
| 6 | promtail | Log shipping (depends on Loki) |
| 10 | argocd-ingress | HTTPRoutes + GRPCRoute for ArgoCD UI |
| 20 | rook operator | Helm chart, CRDs |
| 25 | rook cluster | CephCluster CR — PostSync gate blocks until HEALTH_OK |
| 30 | rook storage, ceph-latency-bridge | BlockPool, CephFS, ObjectStores, toolbox; SLO metrics exporter |
| 31 | rook dashboards | Grafana ConfigMaps for Ceph Cluster/OSD/Pool views |
| 33 | sloth | PrometheusServiceLevel CRs + SLO burn-rate rules |
| 35 | rook gateway | HTTPRoutes for Ceph Dashboard, S3 endpoints |
| 40 | s3-traffic-generator | Optional load generator for S3 |
These aliases are pre-configured inside every VM (fish and bash):
ceph-status # ceph status via toolbox pod
ceph-df # ceph df detail
ceph-osd-tree # ceph osd tree
ceph-health # ceph health detail
ceph-pools # ceph osd pool ls detail
ceph-pg-stat # ceph pg stat
ceph-log # ceph log last 50
rook-tools # exec into toolbox bash session
rook-status # kubectl get cephcluster -n rook-ceph
watch-pods # watch kubectl get pods -n rook-ceph
argo-apps # kubectl get applications -n argocd
argo-sync # argocd app sync --all
hubble-rook # hubble observe --namespace rook-ceph
hubble-drops # hubble observe --verdict DROPPED
get-pass # extract dashboard password into clipboard
open-urls # bash /ceph-lab/provisioning/scripts/open_urls.sh# From Mac — destroys all Ceph data, zeros OSD disks
bash provisioning/scripts/wipe_ceph_disks.sh
# Then from inside ceph-control:
make ssh
bash /ceph-lab/provisioning/scripts/install_argocd.sh
# ArgoCD will re-sync and redeploy Rook from scratchpython3 provisioning/scripts/manage_k8s_config.py remove
make destroyAll tuneable via .env (see .env.example for descriptions):
| Variable | Default | Purpose |
|---|---|---|
GITOPS_REPO_URL |
(required) | Your fork's clone URL |
SANDBOX_NUM_CEPH_NODES |
3 | Worker count (min 3 for HA) |
SANDBOX_OSD_DISKS_PER_NODE |
2 | Raw OSD disks per worker |
SANDBOX_INSTALL_ARGOCD |
1 | Auto-run bootstrap after make up |
SANDBOX_CONFIGURE_DNSMASQ |
1 | Write macOS dnsmasq entry for *.ceph.lab |
ROOK_VERSION |
v1.19.1 | Rook Helm chart version |
CILIUM_VERSION |
1.19.3 | Cilium Helm chart version |
GATEWAY_API_VERSION |
v1.5.1 | Gateway API CRD version |
ARGOCD_VERSION |
v3.5.1 | ArgoCD install tag (pinned deliberately — bump by hand, don't float stable) |
SANDBOX_CACHE_ENABLED |
1 | Local apt + OCI pull-through cache (task cache-up/cache-down); falls through silently on miss |
- Mastering Ceph, 2nd Ed. — Nick Fisk
- Red Hat Ceph Storage 9 Documentation
- Ceph Upstream Docs (Squid)
- Rook Ceph Documentation
- Cilium Documentation
- ArgoCD Documentation
See also:
- docs/ceph-cheatsheet.md — quick Ceph command reference
- docs/gitops-argocd-lessons.md — hard-won GitOps/ArgoCD lessons and the OutOfSync debugging playbook
- docs/observability-tour.md — guided walkthrough of the Prometheus/Grafana + Hubble observability stack
- docs/operational-posture.md — maintenance vs normal cluster posture
- docs/rgw-s3-runbook.md — RGW / S3 objectstore operations and bucket management
- k3s not kubeadm — k3s uses SQLite instead of etcd, saving ~600 MB on the control node. This is important because
ceph-controlis only 6 GiB and also runs Rook's operator + CSI pods. - Lima vmType: vz — uses Apple's Virtualization.framework directly; native performance on Apple Silicon and Intel Macs running macOS 13+. No QEMU overhead.
- OSD disks must stay raw — Rook auto-discovers
/dev/vdcand/dev/vddon each worker (Lima virtio-blk). Pre-formatting them will break OSD creation. /dev/vdbis the k3s data disk — not an OSD. ThedeviceFilter: "^vd[cd]"incephcluster.yamltargets onlyvdc/vdd.preserve*OnDelete: trueis set on CephFilesystem and CephObjectStore as a safety net against accidental sync prunes.- Cilium replaces kube-proxy — do not install kube-proxy; Cilium handles all service routing via eBPF.
- Gateway API CRDs must precede Cilium —
install_cilium.shinstalls them first so Cilium discovers the CRDs on startup and auto-creates theciliumGatewayClass. - CephFilesystemSubVolumeGroup is required (Rook v1.17+) — included in
rook/storage/filesystem.yaml. Without it dynamic CephFS provisioning silently fails. - PostSync health gate —
rook-clusterhas a PostSync Job that pollsCephClusteruntilstate=Connectedandhealth=HEALTH_OKbefore ArgoCD advances to wave 30. - Hubble metrics are disabled at bootstrap and should be enabled once Prometheus is synced. They provide L7 flow data and blast-radius metrics.
- Standalone Prometheus, no Operator — We run the community
prometheusandgrafanacharts, notkube-prometheus-stack. Since there is no operator,PrometheusRuleandServiceMonitorCRs are not consumed by the cluster. - Sloth SLOs at build time — Because there's no operator, Sloth is run as a build-time step (
task gen-slos) to render rules directly into PrometheusserverFiles, rather than running as a live controller. - Topology Catalog — A static
catalog-info.yamlis deployed as a ConfigMap to map out Ceph and infrastructure dependencies.