diff --git a/README.md b/README.md index c8e901e..561628b 100644 --- a/README.md +++ b/README.md @@ -11,6 +11,36 @@ Hexol is both: top, and - a **CLI** — to act on an inventory: `render` it, `apply` it, `inspect` what it produced. +## The shape: an engine, plus a thin layer you write + +Hexol ships an **engine**, not a catalog of targets. Four layers, each thinner than it sounds: + +``` +kernel / engine the fold-of-ops core — domain-agnostic + ↓ +generic renderers -o sexp / json / yaml — work on any resolved state + ↓ +YOUR thin domain layer a renderer (renders-with) + a few constructs (define-construct) + ↓ +your inventory content the actual resources / config you describe +``` + +The shipped Kubernetes, Terraform, and Ansible libraries *are* that thin layer — already written, for those +domains. For a domain hexol doesn't ship — OpenStack oslo.config, a house INI format, an in-house CRD — **writing +the thin layer is the intended first step, not a workaround.** It's a renderer registered with `renders-with` +([`hexol/kernel.scm:484`](hexol/kernel.scm#L484)) plus a few `define-construct` constructs — typically a few dozen +lines. In return the engine hands you ordering, introspection (`tree`/`explain`), and multi-target rendering for free. + +Three minimal, in-tree worked examples of exactly this bootstrap: +- [`hexol/sql.scm`](hexol/sql.scm) — SQL-DDL vocabulary + a `-o sql` renderer (library form). +- [`hexol/ledger.scm`](hexol/ledger.scm) — a ledger-cli journal + `-o ledger` (library form). +- [`examples/oslo-config.scm`](examples/oslo-config.scm) — the whole bootstrap in one file: an oslo.config INI + renderer plus a `config-section` / `keystone-authtoken` service vocabulary, ~40 lines. + +So "hexol has no oslo.config renderer and no OpenStack vocabulary" is true the way "a compiler ships no program" is +true: you write the thin layer for your domain. [`docs/extending.md`](docs/extending.md) walks the bootstrap step by +step. + ## Example An inventory is a program that builds your config. Here's a Kubernetes one diff --git a/docs/authoring.md b/docs/authoring.md index d2618af..ecb9e4d 100644 --- a/docs/authoring.md +++ b/docs/authoring.md @@ -290,7 +290,8 @@ hexol/ bin/hexol # the CLI: render / tree / ops / explain / secret examples/ # one self-contained file each inventory.scm # region table + per-region body + hx-each (the engine itself) - regions.scm # the region table as an importable module (CMDB sync source) + regions.scm # the region table as an importable module (shared by inventory.scm) + oslo-config.scm # bootstrap sketch: a from-scratch INI/oslo.config target in one file kubernetes.scm # consumer of (hexol k8s): namespaced apps + compliance demo helm-kube-prometheus-stack.scm # the Helm chart converted to (hexol k8s) ops terraform.scm # consumer of (hexol terraform): AWS + OpenStack, one combined config @@ -303,17 +304,4 @@ docs/ model.md # the fold-of-ops engine model authoring.md # this guide extending.md # building target libraries + worked examples - cmdb.md # the event-sourced CMDB (as built) -cmdb/ # the event-sourced CMDB built on the same kernel (see docs/cmdb.md) - store.scm # fact log, library lookup, refold - server.scm # HTTP front-end - json.scm # sexp -> JSON - region-render.scm # resolve the per-region body against a fact's attrs - region-body.scm # the per-region hexol inventory - apps.scm # Helm releases per region - libraries/ # versioned op vocabularies (v1, v2 — the library-bump demo) -bin/cmdb-server # boot the CMDB HTTP server -bin/sync-inventory # push a region table as facts -bin/promote # waved image/chart rollouts -test/cmdb-store.scm test/cmdb-server.scm # CMDB tests ``` diff --git a/docs/extending.md b/docs/extending.md index 377b88e..53ad65d 100644 --- a/docs/extending.md +++ b/docs/extending.md @@ -5,6 +5,104 @@ extensibility model, the rule for what belongs in a library, two worked target libraries (Terraform and a Helm-chart conversion), and the introspection you get for free. +## hexol is a substrate — you bring the target + +The engine is deliberately domain-agnostic: it folds ops over an alist and +knows nothing about Kubernetes, Terraform, or your domain. The shipped +libraries (`(hexol k8s)`, `(hexol terraform)`, `(hexol ansible)`) are not +"hexol's features" — they are **thin layers written on top of the engine**, +each the same shape you write for a domain hexol doesn't ship. + +So if hexol has no renderer and no vocabulary for *your* target — an +OpenStack oslo.config file, a bespoke INI format, an in-house CRD — +**bootstrapping that thin layer is the normal first step, not a gap you're +working around.** The whole layer is two moving parts: + +1. **A renderer** — a `(state -> writes text)` procedure registered with + `renders-with` ([`hexol/kernel.scm:484`](../hexol/kernel.scm#L484)), exposed + as `hexol render -o `. (You often don't even need this: the built-in + `-o sexp | json | yaml` render *any* resolved state generically.) +2. **A few constructs** — domain nouns built with `define-construct` (see the + section below), each returning an op that appends to some accumulator in + state. + +That's it. In return the engine hands you ordering, `tree`/`explain` +introspection, and multi-target rendering — for free, because your constructs +bottom out in the same op record as everything else. + +Two in-tree libraries are minimal worked examples of exactly this pattern: +[`hexol/sql.scm`](../hexol/sql.scm) (a SQL-DDL vocabulary + `-o sql`) and +[`hexol/ledger.scm`](../hexol/ledger.scm) (a ledger-cli journal + `-o ledger`). +Each is well under 300 lines and touches no kernel code. + +### Worked example: standing up an oslo.config target + +[`examples/oslo-config.scm`](../examples/oslo-config.scm) keeps the *entire* +bootstrap in one file — renderer + constructs + a small inventory — so you can +see the shape and the effort at a glance. It's the OpenStack case the layering +above describes, made concrete: ~40 lines to teach hexol to emit +oslo.config-style INI with a service vocabulary. + +The renderer walks an `(ini_sections)` accumulator and prints each section: + +```scheme +(define (render-ini state) + (for-each + (lambda (section) ; section = (name . ((key . value) …)) + (format #t "[~a]\n" (car section)) + (for-each (lambda (kv) (format #t "~a = ~a\n" (car kv) (ini-value (cdr kv)))) + (cdr section)) + (newline)) + (or (state-get state '(ini_sections)) '()))) + +(renders-with "ini" render-ini) ; now: hexol render -o ini +``` + +Then the domain nouns. `config-section` is the generic one — `#:open? #t` +lets any `(key value)` through, because an INI section has no fixed key set: + +```scheme +(define-construct config-section + #:head name + #:open? #t + #:build (op:append '(ini_sections) (cons name extra) (list 'config-section name))) +``` + +`keystone-authtoken` is the *vocabulary* half — the `[keystone_authtoken]` +block every OpenStack service carries, with the boilerplate defaulted so a +caller only states what differs (this is the "service vocabulary" a thin layer +adds; `config-section` alone would make you respell it in every service): + +```scheme +(define-construct keystone-authtoken + #:head () + #:fields ((auth-url #:required) (password #:required) + (username #:default "nova") (project-name #:default "service")) + #:build (op:append '(ini_sections) + `(keystone_authtoken (auth_url . ,auth-url) (auth_type . password) + (username . ,username) (password . ,password) + (project_name . ,project-name)) + (list 'keystone-authtoken))) +``` + +An inventory then reads like real config, and every CLI view works unchanged: + +```scheme +(hx-ops + (config-section 'DEFAULT (transport_url "rabbit://…") (debug #f)) + (keystone-authtoken (auth-url "https://controller:5000/v3") + (password "nova-service-password"))) +``` +``` +./bin/hexol render -o ini -i examples/oslo-config.scm # the INI file +./bin/hexol tree -i examples/oslo-config.scm # the op tree +``` + +That is the whole substrate story: the engine is general; the ~40 lines above +are the layer *you* own. The rest of this document is the mechanics — +`define-construct` in depth, the library/example boundary, and two larger +worked targets. + ## The contract The kernel exposes a small set of op constructors — `op:merge`, `op:set`, diff --git a/examples/inventory.scm b/examples/inventory.scm index 7bcdbed..10a2d50 100644 --- a/examples/inventory.scm +++ b/examples/inventory.scm @@ -13,20 +13,14 @@ ;;; `$` computes derived ids. The three regions exercise the branches ;;; (gpu/advanced/prod, sovereign/strict, standard/basic/dev). -(use-modules (hexol)) +(use-modules (hexol) (examples regions)) ;; ---------- the region table (data) ---------- ;; ;; Dispatch axes as data: dc, geo, hw-profile, network-profile, tier, -;; sovereignty. Each cdr is the attribute seed for the body. - -(define regions - '((alpha5 (region . alpha5) (dc . alpha) (geo . eu) (hw-profile . gpu-dense) - (network-profile . advanced) (tier . prod) (sovereignty . none)) - (bravo1 (region . bravo1) (dc . bravo) (geo . eu) (hw-profile . standard) - (network-profile . sovereign) (tier . prod) (sovereignty . strict)) - (charlie6 (region . charlie6) (dc . charlie) (geo . na) (hw-profile . standard) - (network-profile . basic) (tier . dev) (sovereignty . none)))) +;; sovereignty. Each cdr is the attribute seed for the body. The table itself +;; lives in examples/regions.scm as an importable module, so other consumers +;; can share it — this file just folds a body over it. (hx-each regions #:into regions ;; Per-region defaults. diff --git a/examples/oslo-config.scm b/examples/oslo-config.scm new file mode 100644 index 0000000..26420ac --- /dev/null +++ b/examples/oslo-config.scm @@ -0,0 +1,103 @@ +;;; examples/oslo-config.scm — bootstrapping a NEW domain target in one file. +;;; +;;; This is the thin layer every hexol user writes for their own domain: a +;;; renderer (registered with `renders-with`) plus a couple of domain +;;; constructs (`define-construct`). Nothing here is in the library — it is +;;; the ~40 lines you stand up on top of the engine to teach it a new output +;;; format and vocabulary. Here that format is OpenStack-style oslo.config +;;; INI, and the vocabulary is a service's config sections. +;;; +;;; hexol/sql.scm and hexol/ledger.scm are exactly this pattern promoted to +;;; library modules; this file keeps it inline so the whole bootstrap is +;;; visible at once. Every CLI view still works, for free: +;;; +;;; ./bin/hexol render -o ini -i examples/oslo-config.scm # the INI file +;;; ./bin/hexol render -i examples/oslo-config.scm # resolved state +;;; ./bin/hexol tree -i examples/oslo-config.scm # the op tree +;;; +;;; Numbers and endpoints are made up. + +(use-modules (hexol) (hexol construct) + (srfi srfi-1) (ice-9 format)) + +;; ---------- the renderer (state -> INI text) ---------- +;; +;; Each construct below appends a `(section-name . ((key . value) …))` pair to +;; the `(ini_sections)` accumulator, so resolving the inventory *is* the build. +;; This walks that list and prints each section. Registered as `-o ini`. + +(define (ini-value v) + (cond ((eq? v #t) "true") + ((eq? v #f) "false") + ((string? v) v) + ((symbol? v) (symbol->string v)) + ((number? v) (number->string v)) + ((list? v) (string-join (map ini-value v) ", ")) ; multi-opt values + (else (format #f "~a" v)))) + +(define (render-ini state) + (for-each + (lambda (section) + (format #t "[~a]\n" (car section)) + (for-each (lambda (kv) (format #t "~a = ~a\n" (car kv) (ini-value (cdr kv)))) + (cdr section)) + (newline)) + (or (state-get state '(ini_sections)) '()))) + +(renders-with "ini" render-ini) + +;; ---------- domain constructs (define-construct) ---------- +;; +;; `config-section`: the generic INI section. `#:open? #t` lets any `(key +;; value)` through into `extra` — an INI section has no fixed key set. Values +;; are evaluated Scheme (typed-constructor rule), so booleans and refs are +;; natural. +(define-construct config-section + #:head name + #:open? #t + #:build (op:append '(ini_sections) (cons name extra) (list 'config-section name))) + +;; `keystone-authtoken`: domain *vocabulary* — the `[keystone_authtoken]` +;; block every OpenStack service carries, with the boilerplate defaulted so a +;; caller only states what differs. This is the "service vocabulary" a thin +;; layer adds; `config-section` alone would make you respell it every time. +(define-construct keystone-authtoken + #:head () + #:fields ((auth-url #:required) + (username #:default "nova") + (password #:required) + (project-name #:default "service") + (user-domain #:default "Default") + (project-domain #:default "Default")) + #:build + (op:append '(ini_sections) + `(keystone_authtoken + (auth_url . ,auth-url) + (auth_type . password) + (username . ,username) + (password . ,password) + (project_name . ,project-name) + (user_domain_name . ,user-domain) + (project_domain_name . ,project-domain)) + (list 'keystone-authtoken))) + +;; ---------- the inventory: one service's nova.conf ---------- + +(hx-ops + (config-section 'DEFAULT + (transport_url "rabbit://openstack:secret@controller:5672/") + (my_ip "10.0.0.31") + (debug #f) + (enabled_apis (list 'osapi_compute 'metadata))) + + (config-section 'api + (auth_strategy "keystone")) + + (keystone-authtoken + (auth-url "https://controller:5000/v3") + (password "nova-service-password")) + + (config-section 'vnc + (enabled #t) + (server_listen "$my_ip") + (server_proxyclient_address "$my_ip"))) diff --git a/examples/regions.scm b/examples/regions.scm new file mode 100644 index 0000000..afa6415 --- /dev/null +++ b/examples/regions.scm @@ -0,0 +1,20 @@ +;;; examples/regions.scm — the region table as an importable module. +;;; +;;; The dispatch axes as plain data, split out of examples/inventory.scm so +;;; more than one consumer can share the same table: the inventory folds a +;;; per-region body over it (see inventory.scm), and any other program can +;;; `(use-modules (examples regions))` to read the same source of truth. +;;; +;;; Each entry is `(name . attribute-seed)` — the cdr is exactly the query +;;; attributes the body resolves against for that region. + +(define-module (examples regions) + #:export (regions)) + +(define regions + '((alpha5 (region . alpha5) (dc . alpha) (geo . eu) (hw-profile . gpu-dense) + (network-profile . advanced) (tier . prod) (sovereignty . none)) + (bravo1 (region . bravo1) (dc . bravo) (geo . eu) (hw-profile . standard) + (network-profile . sovereign) (tier . prod) (sovereignty . strict)) + (charlie6 (region . charlie6) (dc . charlie) (geo . na) (hw-profile . standard) + (network-profile . basic) (tier . dev) (sovereignty . none))))