Transfer of major fixes

TODO.md

@@ -10,69 +10,53 @@ Near-term, concrete items. See `ROADMAP.md` for the bigger picture.
 
 ## upac-lib
 
-Test-coverage pass in progress, going file by file through the non-command core first
-(`errors.rs`/`lock.rs`/`search.rs`/`fs.rs`/`orchestrator/*`/`database/*` done), commands
-(`mutated`/`unmutated`) last. Remaining core files not yet visited: `deploy/{error,retention,mod}.rs`
-(`esp.rs` skipped — real mount), `scripts/{error,file,load,pipeline,primitive}.rs`,
-`plugin/decoder/{error,unpack,mod}.rs`, `plugin/boot/{error,manifest,mod}.rs`,
-`composefs/{diff,error,mod}.rs`, `config/mod.rs`, `boot/{error,mod}.rs`.
-
-`boot/mod.rs`'s UKI staging only ever writes the single fixed `upac-to` slot
-(`layout::boot::UPAC_TO_SLOT`) — doc chapter 3's disk-layout map still describes a two-slot
-`upac-from.efi`/`upac-to.efi` A/B scheme, but no code anywhere writes/reads an `upac-from` slot.
-Real unfinished A/B swap, not just a stale doc — needs a decision (implement the second slot, or
-formally drop it and fix the doc to match the single-slot design `lib.toml`'s own comment already
-argues for).
-
-Genesis (`up-sp`) now installs the actual bootloader binary onto a fresh ESP for systemd-boot and
-rEFInd (`StageBootStage::run`, `lib/setup/lib.toml`'s `[genesis]` source paths) — confirmed working
-via a live VM test for systemd-boot; rEFInd wired the same way but not yet VM-verified. grub is NOT
-handled — a real `grub-install`-equivalent (target-specific generated `grubx64.efi`, not a plain
-file copy) is out of scope for now; either shell out to `grub-install` against the mounted ESP, or
-explicitly document grub as unsupported for genesis whole-disk mode.
-
-`StageBootStage` picks which ESP loader binary to copy via a `match input.boot_plugin.as_deref()`
-against literal `"systemd-boot"`/`"refind"` strings — this bypasses the actual dynamic boot-plugin
-system (`resolve_boot_plugin`/`BootPluginManifest`/`static_plugins`), which is supposed to be the
-one place plugin names are known. Adding a 5th booter plugin would require editing this match by
-hand instead of just dropping in a new plugin. The correct fix is extending the `Booter` ABI itself
-with a 4th function (e.g. `esp_loader_source() -> CSlice`, empty for uki/grub) so genesis asks the
-already-resolved plugin for its own install-time source path instead of hardcoding names — but that
-means bumping `BOOT_ABI_VERSION` and touching all 4 `booters/*` crates, so deliberately deferred;
-the hardcoded match stays as a known, scoped limitation until then.
-
-**Genesis-produced disks don't actually boot into the installed system yet** — found via a live
-QEMU/OVMF test (systemd-boot now starts, finds the BLS entry, loads kernel+initramfs — that part
-works after the bootloader-binary fix above). Two separate gaps, both required:
-1. `partition.rs`'s `LINUX_PARTITION_TYPE_GUID` (`0fc63daf-8483-4772-8e79-3d69d8477de4`, generic
-   "Linux filesystem data") should be the discoverable-root GUID
-   (`4f68bce3-e8cd-4db1-96e7-fbcaf984b709`, "Linux root x86-64") so `systemd-gpt-auto-generator`
-   can find the deploy partition at all instead of hanging on `/dev/gpt-auto-root`.
-2. Even with (1) fixed, a plain partition mount isn't how composefs systems boot — nothing in this
-   project resolves `composefs.digest=<hash>` (the kernel cmdline param `write_boot_entry` already
-   writes) against the on-disk repository, mounts the erofs image with fs-verity, and overlays
-   `state/deploy/<digest>/etc/`. **Found a real, existing upstream tool for exactly this**:
-   `composefs-setup-root` (crates.io, same `composefs-rs` project/version as our `composefs`/
-   `composefs-boot` deps) — a Rust binary, not something we'd write ourselves. Our on-disk layout
-   already matches its hardcoded expectations (`composefs/`, `state/deploy/<digest>/`) after
-   renaming `etc-upper` → `etc` (done, `lib.toml`'s `config_dir_name`). What's still missing: the
-   actual boot-time integration — the live VM's initramfs is systemd-based (mkinitcpio's `systemd`
-   hook, not classic busybox-style hooks), so this needs a systemd unit ordered between
-   `sysroot.mount` and `initrd-switch-root.target` (same role as ostree's
-   `ostree-prepare-root.service`), not a classic mkinitcpio hook script. Also unresolved: whether
-   upac needs to ship/package this integration itself, or whether it's expected to already exist
-   on the source distro (same assumption as the systemd-boot/rEFInd binary copy above) — needs
-   checking whether Arch/AUR already has a package for this.
-
-**Genesis tracks the entire bootstrapped system as a single synthetic "rootfs" package**, not
-per-package (`ReadMetaStage` reads one `meta.toml`, `ImportTreesStage` imports all of source's
-`usr`/`etc` wholesale). Found while reasoning about the `composefs-setup-root` hook: if it needs to
-already be installed on the source system (via pacman) for genesis to pick it up, its files still
-end up attributed to the one fake "rootfs" package in our database — no real per-package
-provenance for anything baked into the source image, unlike a `pacstrap`-then-`up install` flow
-would give. Decision made: genesis should eventually be rewritten to install real, individually
-decoded packages through the same pipeline `up install` uses, instead of importing a pre-built
-directory wholesale — no special-casing even for the kernel package. This is a genesis rewrite, not
-a patch; deliberately deferred until after a dedicated code-cleanup/macro-consolidation pass
-(reduce duplicated lines, extract shared macros) elsewhere in the codebase first.
-
+Test-coverage pass in progress. The entire non-command core is covered (`errors.rs`/`lock.rs`/
+`search.rs`/`fs.rs`/`orchestrator/*`/`database/*`/`deploy/*`/`scripts/*`/`composefs/*`/`config/*`/
+`boot/*`/`plugin/decoder/{error,manifest,triggers}.rs`/`plugin/boot/{error,manifest}.rs`), except
+`plugin/decoder/unpack.rs`/`plugin/decoder/mod.rs`/`plugin/boot/mod.rs` (need a real dlopen'd/
+`builtin-*` plugin) and `deploy/esp.rs` (real mount table) — both explicit, justified skips. Every
+`mutated`/`unmutated` command's own `<Command>Error` enum is also now covered (inline tests next to
+each `error.rs`, since `mutated`/`unmutated` aren't `pub`) — only each variant's own logic, not the
+macro-generated `Common(...)` delegation shared with `errors.rs`'s already-tested `CommonError`.
+Remaining: the `Stage::run()` bodies themselves — each needs a real composefs `Repository`/`Deploy`/
+database in context, likely out of scope for unit tests unless a pure-logic helper turns out to be
+extractable.
+
+**`genesis`'s `system/` mechanism is done**: `ImportSystemStage` requires `<source>/system/` (a
+literal 1:1 mirror of the target's real `/usr`, sibling to the package archives —
+`EnumeratePackagesStage` already skips it, it only looks at files) to contain
+`lib/systemd/system/composefs-setup-root.service` (hard error, `SetupError::
+ComposefsSetupRootUnitNotFound`, if missing), imports the whole tree into `PrefixTree`, and creates
+the unit's `*.target.wants/` enablement symlink itself. This is also how a built `up`/`upac-lib`/
+booters gets onto a genesis'd disk at all — genesis never installs itself automatically, whoever
+assembles `--source` has to place it under `system/` too, same assumption already made for the
+systemd-boot/rEFInd binaries. Confirmed `composefs-setup-root`'s own hardcoded expectations already
+match upac's on-disk layout exactly (repo at `composefs/`, per-deploy state at `state/deploy/<hex>/`,
+`composefs=<hex>` cmdline karg) — no restructuring was needed, only the unit + the `system/` plumbing.
+Still unresolved: whether upac ships/packages the `composefs-setup-root` binary itself or expects it
+to already exist on the source distro (same open question as the systemd-boot/rEFInd binaries).
+
+**Booter ABI redesign — decided this session, execution in progress file-by-file under direct
+supervision (no batch edits).** Four canonical plugin responsibilities:
+
+1. Plugin sets itself for one-time boot (`set_one_shot`) — done.
+2. Plugin sets itself for persistent boot (`confirm_boot`) — done, including UKI's `to.efi`↔
+   `from.efi` file swap (needed a new `esp_mount_point` parameter on `confirm_boot`, added this
+   session; the swap only fires when the confirmed `entry_name` is the `to` slot specifically).
+3. Plugin installs itself onto the ESP (`install`) — done for grub (real `grub-install
+   --removable --no-nvram` + a minimal `blscfg` `grub.cfg`), no-op for the other 3.
+4. Plugin declares where its own pre-built loader binary lives in the source package tree
+   (`esp_loader_source`) — done, stays a separate passive query (only genesis can reach the
+   composefs tree to copy the bytes out, plugins can't do this step themselves).
+
+**`write_boot_entry` must search only for the resource type the selected plugin needs, not
+autonomously scan everything and guess.** Right now (`lib/lib/src/boot/mod.rs`) it calls
+`get_boot_resources` unconditionally, takes whichever single boot resource exists in the tree
+(Type1/Type2/`UsrLibModulesVmLinuz`), and only errors if more than one is found total — completely
+independent of which plugin was actually selected. This is the case in all 6 call sites:
+`lib/lib/src/boot/mod.rs` itself, `mutated/{files,installer,uninstaller,update,rollback}/checkout.rs`,
+and `lib/setup/src/genesis/entry.rs`. Needs to take the (now always-explicit) plugin name and require
+specifically: `uki` → `Type2` only; `grub`/`systemd-boot`/`rEFInd` → `Type1`/`UsrLibModulesVmLinuz`
+only — hard error if that type isn't present, even if a different type is. This also means
+`resolve_boot_plugin` must run before `write_boot_entry` everywhere — today the 5 ordinary
+`checkout.rs` stages call it after (only genesis already has the order right).

booters/booter.toml

@@ -8,42 +8,74 @@
 # own build.rs reads this same file and generates its own private layout::boot module; nothing
 # here is a Cargo dependency between plugins, just a shared data file.
 #
-# efi_sysfs_path is the fixed Linux kernel sysfs mount point for UEFI firmware (never
-# distro/deployment-configurable, unlike the ESP mount point). efivarfs_path is the fixed
-# subdirectory where the kernel exposes individual EFI variables as files (`efivar`'s own crate
-# never surfaces this path, so plugins that need to clear the immutable attribute on an existing
-# variable file — kernel efivarfs sets it after first write, and `efivar` never clears it back off
-# before overwriting — have to reconstruct the exact same path themselves). sd_boot_loader_guid is
-# systemd's vendor GUID for its own boot-loader-interface EFI variables (LoaderInfo,
-# LoaderEntryOneShot, LoaderEntryDefault), per systemd's BOOT_LOADER_INTERFACE.md. boot_next_var/
-# boot_order_var are UEFI global-namespace variables, no vendor GUID needed — boot_order_var is
-# only needed to clear its immutable attribute before `VarManager::set_boot_order` touches it
-# (that method writes the variable internally, so the uki plugin can't intercept the write() call
-# itself the way it can for boot_next_var).
+# efivarfs_path is the fixed subdirectory where the kernel exposes individual EFI variables as
+# files (`efivar`'s own crate never surfaces this path, so plugins that need to clear the
+# immutable attribute on an existing variable file — kernel efivarfs sets it after first write,
+# and `efivar` never clears it back off before overwriting — have to reconstruct the exact same
+# path themselves). sd_boot_loader_guid is systemd's vendor GUID for its own boot-loader-interface
+# EFI variables (LoaderEntryOneShot, LoaderEntryDefault), per systemd's BOOT_LOADER_INTERFACE.md.
+# boot_next_var/boot_order_var are UEFI global-namespace variables, no vendor GUID needed —
+# boot_order_var is only needed to clear its immutable attribute before
+# `VarManager::set_boot_order` touches it (that method writes the variable internally, so the uki
+# plugin can't intercept the write() call itself the way it can for boot_next_var).
 [boot]
-efi_sysfs_path = "/sys/firmware/efi"
 efivarfs_path = "/sys/firmware/efi/efivars"
 sd_boot_loader_guid = "4a67b082-0a4c-41cf-b6c7-440b29bb8c4f"
 boot_next_var = "BootNext"
 boot_order_var = "BootOrder"
-loader_info_var = "LoaderInfo"
 loader_entry_one_shot_var = "LoaderEntryOneShot"
 loader_entry_default_var = "LoaderEntryDefault"
 
+# efi_linux_dir is the fixed UEFI-style (backslash) directory composefs-boot's own
+# `write_t2_simple` always writes UKI images under (`EFI/Linux/`, hardcoded upstream, not
+# configurable) — used to build the file-path device-path segment when genesis registers a
+# UEFI Boot#### entry for a UKI slot, so the entry's path always matches where the image
+# actually lands. efi_linux_real_path is the same directory spelled as a real, mounted-
+# filesystem-relative path (forward slashes, joined via `Path::join`) — used by
+# `Uki::confirm_boot` to find `to.efi`/`from.efi` on the real, mounted ESP; the two can't share
+# a definition (see `lib/setup/lib.toml`'s own `genesis.efi_linux_dir` for the same split,
+# there for the same reason: different crate, different path flavor needed).
+#
+# to_slot/from_slot must stay in sync with `upac-lib`'s own `lib.toml`
+# (`boot.upac_uki_to_slot`/`upac_uki_from_slot`) — two separate crates, can't share a single
+# Rust constant, so each carries its own literal copy. `Uki::confirm_boot` only performs the
+# to/from file swap when the confirmed `entry_name` is `to_slot` specifically (confirming a
+# fallback boot into `from_slot` means `to_slot` may be broken — nothing should be promoted).
+[uki]
+efi_linux_dir = "\\EFI\\Linux\\"
+efi_linux_real_path = "EFI/Linux"
+to_slot = "upac-to"
+from_slot = "upac-from"
+
 # grub has no EFI-variable-based one-shot mechanism — it's file-based (grubenv), driven through
 # grub's own grub-reboot/grub-set-default tools rather than a hand-rolled binary-format writer.
 # Both the grubenv location and the tool names differ across distro packaging: Debian/Ubuntu/Arch
 # ship grub-reboot/grub-set-default writing /boot/grub/grubenv, while Fedora/RHEL keep the
 # "grub-*" names reserved for legacy GRUB Legacy and ship grub2-reboot/grub2-set-default writing
 # /boot/grub2/grubenv instead. *_primary/*_fallback are tried in that order.
+#
+# install_bin_primary/install_bin_fallback: same Debian/Arch-vs-Fedora naming split as the
+# reboot/set-default tools above, for the one-time `grub-install` genesis runs against a freshly
+# partitioned ESP. install_target is grub's own `--target` platform name (this project is
+# UEFI/x86_64-only throughout, matching systemd_boot/refind's own *_x64 binary names below).
+# install_bootloader_id is grub's `--bootloader-id`, used only to name the install's own
+# NVRAM-independent EFI/BOOT/BOOTX64.EFI fallback copy (see backend.rs's `--removable --no-nvram`
+# — genesis relies on the same UEFI firmware fallback path uki/systemd-boot/refind's binaries
+# already occupy, not a Boot#### entry, so a fresh disk boots without any NVRAM setup).
 [grub]
-grubenv_primary = "/boot/grub/grubenv"
-grubenv_fallback = "/boot/grub2/grubenv"
 reboot_bin_primary = "grub-reboot"
 reboot_bin_fallback = "grub2-reboot"
+
 set_default_bin_primary = "grub-set-default"
 set_default_bin_fallback = "grub2-set-default"
 
+install_bin_primary = "grub-install"
+install_bin_fallback = "grub2-install"
+
+install_target = "x86_64-efi"
+
+install_bootloader_id = "upac"
+
 # rEFInd has no separate one-shot/persistent pair of variables like systemd-boot's
 # LoaderEntryOneShot/LoaderEntryDefault — it has exactly one, PreviousBoot, which it writes itself
 # just before launching any OS and reads back as the default on its next launch. Selecting an

booters/grub/Cargo.toml

@@ -28,7 +28,8 @@ name = "upac_boot_grub"
 crate-type = ["cdylib", "rlib"]
 
 [dependencies]
-upac-abi = { workspace = true }
+upac-abi   = { workspace = true }
+upac-types = { workspace = true }
 
 [build-dependencies]
 toml = { workspace = true }

booters/grub/build.rs

@@ -19,26 +19,25 @@ fn main() -> Result<(), Box<dyn Error>> {
     let raw = read_to_string(&source)?;
     let config: Value = from_str(&raw)?;
 
-    let mut generated = String::new();
-
-    let sections = config.as_table().ok_or("booter.toml: root must be a table")?;
-    for (section, entries) in sections {
-        generated.push_str(&format!("pub mod {section} {{\n"));
+    let section = "grub";
+    let entries = config
+        .get(section)
+        .and_then(Value::as_table)
+        .ok_or_else(|| format!("booter.toml: [{section}] must be a table"))?;
 
-        let entries = entries
-            .as_table()
-            .ok_or_else(|| format!("booter.toml: [{section}] must be a table"))?;
-        for (key, value) in entries {
-            let value = value
-                .as_str()
-                .ok_or_else(|| format!("booter.toml: {section}.{key} must be a string"))?;
+    let mut generated = String::new();
+    generated.push_str(&format!("pub mod {section} {{\n"));
 
-            generated.push_str(&format!("    pub const {}: &str = {value:?};\n", key.to_uppercase()));
-        }
+    for (key, value) in entries {
+        let value = value
+            .as_str()
+            .ok_or_else(|| format!("booter.toml: {section}.{key} must be a string"))?;
 
-        generated.push_str("}\n");
+        generated.push_str(&format!("    pub const {}: &str = {value:?};\n", key.to_uppercase()));
     }
 
+    generated.push_str("}\n");
+
     let out = Path::new(&var("OUT_DIR")?).join("layout.rs");
     write(out, generated)?;