Skip to content

dev-environment.md

Latest commit

 

History

History
129 lines (96 loc) · 6.51 KB

dev-environment.md

File metadata and controls

129 lines (96 loc) · 6.51 KB
Raw

The local dev environment

A k3d cluster and a Tiltfile substrate that stand up everything deploy/chart/thump assumes already exists, so the whole five-beat loop runs on a laptop with no rig repo, no IAP tunnel, and no Ceph. The other cluster profile, thump-test, is provisioned by a separate repo under ~/projects/ceph/; dev is provisioned by this one, via deploy/dev/.

Prerequisites

  • k3d, helm, tilt, kubectl — brew install k3d helm tilt-dev/tap/tilt kubectl
  • Docker (or OrbStack) with at least 12 GB RAM / 6 CPU allocated to the VM. The substrate (Cilium, cert-manager, Prometheus, Alertmanager, Loki, Tempo, otel-collector, Grafana, S3Mock) runs about 5 GB; the trimmed OTel demo adds roughly another 4 GB.
  • An Anthropic API key. clank has no fake Model implementation in this repo — internal/replay only replays one recorded transcript, it can't serve a live loop — so ANTHROPIC_API_KEY in .env is the one thing this environment can't stand up for you. A five-beat cycle on Haiku costs fractions of a cent; nothing about running this environment burns meaningful spend.

Bringing it up

task dev:up

This is task dev:cluster (idempotent k3d cluster create --config deploy/dev/k3d.yaml) followed by tilt up -- --cluster=dev. Cluster creation can't live inside the Tiltfile itself — allow_k8s_contexts() and helm() both evaluate at Tiltfile load, so the k3d context has to exist before Tilt starts at all.

Once Tilt is running, its dev-substrate resource runs deploy/dev/bootstrap.sh: ten pinned Helm releases plus a handful of manifests, in an order that isn't arbitrary — Cilium's own chart refuses to render a ServiceMonitor without prometheus-operator-crds already installed, and cluster-issuer.yaml needs cert-manager's CRDs Established before it can apply. Expect 10–15 minutes on first boot; every step after is helm upgrade --install --wait, so a re-run converges rather than reinstalling.

Every beat runs with TRIGGER_MODE_MANUAL in Tilt, same as every other profile — nothing redeploys until you tell it to.

Reaching the UI

No Gateway API or Ingress in this environment (deploy/dev/values/cilium.yaml turns gatewayAPI off — see that file for why). Tilt runs each of the following as its own supervised kubectl port-forward:

Service Local port What it's for
Grafana localhost:3000 Dashboards over the Prometheus + Tempo datasources
Prometheus localhost:9090 Raw PromQL, including slo:current_burn_rate:ratio
Hubble UI localhost:12000 Cilium's flow visibility
OTel demo frontend localhost:8080 The Astronomy Shop storefront itself

Grafana's admin password is admin (deploy/dev/values/grafana.yaml) — this cluster never leaves your laptop, so it isn't worth managing as a secret.

Breaking something

task chaos:cart-failure    # flip cartFailure on in otel-demo/flagd-config

This is chaos/flagd-cart-failure.sh inject, unchanged from every other profile — it patches a ConfigMap, and deploy/dev/values/otel-demo.yaml carries the same mountedConfigMaps/null-field rewiring the rig uses so flagd hot-reloads the change in place instead of needing a restart.

Watch it land:

kubectl logs -n thump -l app.kubernetes.io/component=rattle -f

rattle's burn-rate detector scores a trailing window, so expect the first "detection" log line something like 5–6 minutes after injection, not immediately. clank reasons over it next ("reasoned"), hiss rules on the proposal ("decision"), and thump executes the chosen action (patching otel-demo/flagd-config). Restore the flag when you're done:

task chaos:cart-restore

Live mode and forge binding

deploy/tilt-values-dev.yaml sets thump.executor: live with killSwitch.armed: true.

Unlike the production rig (thump-test), dev requires no GitOps target (FORGE_REPO). actuate.New (internal/actuate/kube.go) only requires a forge when the loaded catalog authors a maintenanceRelease action. The dev profile catalog (config/dev/actions/catalog.yaml) authors in-cluster mutations — patching otel-demo/flagd-config — and leaves release contracts to thump-test. bind validates every contract at startup, finds no release actions, and starts clean with FORGE_REPO unset.

When task chaos:cart-failure fires, the loop runs end-to-end through detection, evidence gathering, governance, and actual cluster mutation. Confirmed live 2026-08-15: slo_burn:cart cleared corroboration (3 distinct tools — metrics, loki, kube) at 0.95 self-reported confidence against the 0.75 floor, hiss approved, thump applied disable-cart-failure, and the flag was back to off within ~5m20s of injection with no human involved — see thump-running-notes.md, 2026-08-15 entry.

Operator surface

When a detection requires manual intervention or governance holds an action, the operator CLI (calipers) can interact with the live cluster over the port-forwarded NATS server:

task dev:certs                          # extract NATS TLS certificates to bin/certs/
task dev:incidents                      # list active incidents over NATS
task dev:approve FP=<fingerprint>       # approve a held incident by fingerprint

To read what a run actually reasoned — every tool call, citation, and confidence term, not a log-line reconstruction — export and unseal its sealed transcript:

task dev:transcript RUN=<run_id>        # one run: bin/transcripts/<run_id>/run.jsonl + run.set.json
task dev:transcripts                    # every run under transcripts/, same layout per run

<run_id> comes from a "reasoned" log line's run_id field. Both targets port-forward svc/s3mock to local 9091 (Tilt already owns 9090 for Prometheus) and tear it down on exit. THUMP_SEAL_KEY and THUMP_NATS_JS_KEY are persisted stably in .env (seeded automatically on first tilt up if absent, D-31), so transcript extraction and WAL decryption work across routine tilt down/up cycles without key drift.

acme, the third domain

domains.acme.enabled is true in this profile — a second, orthogonal domain alongside the OTel demo (this profile runs no Ceph; that's thump-test alone), standing in for someone who has never onboarded a domain and following docs/onboarding.md's own fixture. Its catalog entry is acme-shed-load (config/dev/actions/catalog.yaml), scoped to dependency_saturation. Fault injection is direct, not a task target:

chaos/acme-fault.sh inject
chaos/acme-fault.sh restore