Add Btrfs fallback for other Linux distros by locnnil · Pull Request #956 · canonical/workshop

locnnil · 2026-07-21T22:29:50Z

Description

Enables Btrfs fallback for other Linux distributions.

ZFS is the preferred storage driver, but it isn't available everywhere: WSL, SUSE, Arch, Fedora, and other distributions may ship without it. Keying the fallback off the distribution name is too coarse; it assumes every non-Ubuntu host lacks ZFS and every Ubuntu-derived one has it, neither of which holds for Debian and its derivatives.

Instead, detect whether the host provides a usable ZFS kernel module, already loaded (present at /dev/zfs AND /sys/module/zfs ) or loadable for the running kernel per modules.dep(5) (/lib/modules/$(uname -r)/modules.dep) or modules.builtin (/lib/modules/$(uname -r)/modules.builtin) and keep ZFS only then, falling back to btrfs otherwise. Only the kernel module is checked: LXD ships as a snap that bundles the zpool/zfs userspace tools, so a loaded or loadable module is what actually determines usability from the host side, tested on Arch Linux and openSUSE tumbleweed.

Closes: workshop does not work on non-Ubuntu systems #742

Self-review quick check

Make decisions that cost a lot to reverse explicit in the PR description.
Avoid nested conditions.
Delete dead code and redundant comments. TODO: check if it is okay to delete isWSL()
Normalise symmetries by sticking to doing identical things identically.

// one way to handle errors
if err := f(); err != nil {
   ...
}

// one way to handle multiple returns
val, err := f()
if err != nil {
   ...
}
...

Check that coupled code elements, files, and directories are adjacent. For example, test data is stored as close as possible to a test.
Put variable declaration and initialisation together.
Divide large expressions into digestable and self-explanatory ones. Use multiple variables if required.
Put a blank line between two logically different chunks of code.
Follow the style guide for new error messages.

Docs

Procedure:

I have checked and added or updated relevant documentation.
I have checked and added or updated relevant release notes.
I have included the technical author in the review.

Content:

Headings and titles accurately describe the content.
New and updated pages include correct metadata.
Documentation tests are added or updated where applicable (for tutorial/ and how-to/ sections).
Documentation follows the style guide.
If needed, docs/.coverage.yaml updated, coverage tags added (.. artefact).

Or:

I confirm the PR has no implications for documentation.

ZFS is the preferred storage driver, but it isn't available everywhere: WSL, SUSE, Arch, Fedora and other distributions may ship without it. Keying the fallback off the distribution name is too coarse -- it assumes every non-Ubuntu host lacks ZFS and every Ubuntu-derived one has it, neither of which holds for Debian and its derivatives. Instead, detect whether the host provides a usable ZFS kernel module (already loaded, or loadable for the running kernel per modules.dep / modules.builtin) and keep ZFS only then, falling back to btrfs otherwise. Only the kernel module is checked: LXD ships as a snap that bundles the zpool/zfs userspace tools, so a loaded or loadable module is what actually determines usability from the host side. Signed-off-by: Lincoln Wallace <lincoln.wallace@canonical.com>

The storage docs described the Btrfs fallback as WSL-specific. Reword the three affected passages to reflect that Workshop uses ZFS wherever it detects ZFS is available and otherwise falls back to Btrfs, keeping WSL as an example rather than the sole case. Signed-off-by: Lincoln Wallace <lincoln.wallace@canonical.com>

jonathan-conder · 2026-07-21T22:35:16Z

It looks to me like the way LXD does it is just trying to load the zfs module; maybe we should follow their lead on that

jonathan-conder · 2026-07-21T22:37:40Z

I think we also need to be more careful about where we use the storagePoolDriver variable. Currently we can assume it's accurate, since whether the host is WSL or not is unlikely to change. The availability of the ZFS module can easily change, though; we need a separate mechanism to indicate "what's the current driver of the workshop pool."

locnnil · 2026-07-22T00:23:10Z

I think we also need to be more careful about where we use the storagePoolDriver variable. Currently we can assume it's accurate, since whether the host is WSL or not is unlikely to change. The availability of the ZFS module can easily change, though; we need a separate mechanism to indicate "what's the current driver of the workshop pool."

So, considering the case where ZFS was available when the pool was created but later stops being present, most likely because the user installed a different kernel that doesn't ship the ZFS module. What should we do with the now-unusable ZFS pool?

I can think of those options:

Put the daemon into a degraded state and add an actionable error message like "restore the ZFS-capable kernel, or remove the pool to recreate it as Btrfs"
Warn the user with an actionable message like "Old ZFS pool detected on system without support for ZFS on path <path-to-zfs-pool>" but create a new Btrfs pool under a different name.

I believe that the first option is better, right?

jonathan-conder · 2026-07-22T01:37:09Z

I believe that the first option is better, right?

I agree, but we should not suggest removing the pool, it will probably break the workshops using it (if it's even possible to remove it). Instead maybe it's best to reinstall the Workshop snap.

We just need to test what actually happens in that scenario, maybe LXD will fail to start and we probably handle that already.

ZFS is the preferred storage driver, but it isn't available everywhere (Debian, SUSE, Fedora and WSL, among others, may ship without it). Use ZFS only when LXD reports it as a supported storage driver; which LXD determines by trying to load the driver's kernel module, and fall back to Btrfs otherwise. Keying the choice off actual availability rather than the host distribution makes the fallback apply to any host lacking ZFS, while still using ZFS wherever it works. Derive the pool's driver from LXD and the pool itself rather than from a fixed value: pick it from LXD's supported drivers when creating the pool, and read it back from the pool wherever behaviour depends on it. A pool's driver is fixed at creation and ZFS availability can change, so a single cached value can't be trusted. When the workshop pool exists but its storage driver is no longer usable (for example a ZFS pool after booting a kernel without the ZFS module), put the daemon into degraded mode with an actionable message instead of failing obscurely. Closes: canonical#742 Signed-off-by: Lincoln Wallace <lincoln.wallace@canonical.com>

locnnil · 2026-08-14T01:50:14Z

@jonathan-conder I've implemented the requested change in 54bf6dd, which stops assuming a fixed state for ZFS availability. I then tested the degraded-state behavior for the scenario where ZFS is present at first but later vanishes.

I ran this on an Ubuntu Resolute LXD VM, reproducing a workshop pool created on ZFS whose kernel module later becomes unavailable after a kernel change.

Reproduction

Started with ZFS available, so the workshop pool was created on the zfs driver.
Built and installed a .deb for a newer kernel (7.2.0-rc7) that doesn't ship the ZFS module.
Removed the previous kernel and its ZFS module package (linux-main-modules-zfs-7.0.0-29-generic).
Rebooted into the new kernel and ran workshop list.

Result

Instead of surfacing a raw LXD error, the daemon goes into degraded mode with an actionable message:

$ workshop list
error: system is not healthy: cannot use the "workshop" storage pool, its storage driver may be unavailable (for example a ZFS pool after booting a kernel without the ZFS module): Error loading "zfs" module: Failed running: modprobe -b zfs
: exit status 1 (modprobe: FATAL: Module zfs not found in directory /lib/modules/7.2.0-rc7)
Boot into a kernel that provides the pool's storage driver, or reinstall Workshop to recreate the pool on an available driver

Recovery

The ZFS pool can't be deleted through LXD while the module is missing (lxc storage delete loads the driver, which fails), so the recovery is to reinstall Workshop: this recreates the workshop pool on an available driver (Btrfs) for a host that no longer provides ZFS.

On adding a Spread test

Let me know if you think we should add a Spread test covering this case. Two caveats:

For my manual test I had to build and install an upstream kernel, since simply uninstalling linux-main-modules-zfs-7.0.0-xx-generic isn't enough to reproduce a realistic scenario. For an automated test we could instead force the broken state, making it reproducible by just forcefully removing a package.
The bigger drawback is that this would require Spread to run on LXD VMs instead of containers. That would be a significant change to the current testing infrastructure.

jonathan-conder · 2026-08-16T22:11:54Z

+	// ZFS is workshop's preferred storage driver; Btrfs is the fallback used
+	// on hosts where LXD reports ZFS as unavailable.
+	storageDriverZFS   = "zfs"
+	storageDriverBtrfs = "btrfs"
+


I don't see what we gain from these constants

jonathan-conder · 2026-08-16T22:14:15Z

+		return fmt.Errorf(`cannot use the %q storage pool, its storage driver may be unavailable (for example a ZFS pool after booting a kernel without the ZFS module): %w
+Boot into a kernel that provides the pool's storage driver, or reinstall Workshop to recreate the pool on an available driver`, storagePool, err)


is the error returned from GetStoragePool related to this explanation?

Yes, from my tests, this is what you get:

$ workshop list error: system is not healthy: cannot use the "workshop" storage pool, its storage driver may be unavailable (for example a ZFS pool after booting a kernel without the ZFS module): Error loading "zfs" module: Failed running: modprobe -b zfs : exit status 1 (modprobe: FATAL: Module zfs not found in directory /lib/modules/7.2.0-rc7) Boot into a kernel that provides the pool's storage driver, or reinstall Workshop to recreate the pool on an available driver

So the error returned by GetStoragePool is:

Error loading "zfs" module: Failed running: modprobe -b zfs : exit status 1 (modprobe: FATAL: Module zfs not found in directory /lib/modules/7.2.0-rc7)

Perhaps the explanation should be less verbose? Since the error is already quite informative.

That's interesting and a bit unexpected. I'd rather not rely on specific LXD error messages if possible, it's likely not something they have tests for. How about checking info.Environment.StorageSupportedDrivers like you did below?

The error message is useful though, does it appear in the LXD logs? Maybe we can direct users there

jonathan-conder · 2026-08-16T22:14:35Z

+func checkStoragePool(conn lxd.InstanceServer, supported []api.ServerStorageDriverInfo) error {
+	_, _, err := conn.GetStoragePool(storagePool)
+	if err == nil {
 		return nil


do we need to check if the storage pool driver is supported here?

jonathan-conder · 2026-08-16T22:15:36Z

+	// loading each pool's driver, so an existing but currently-unloadable pool
+	// (e.g. a ZFS pool whose module is gone) is reported by checkStoragePool
+	// rather than failing here.
+	names, err := conn.GetStoragePoolNames()


do we need to list all the storage pools here? what stops us querying workshop directly?

jonathan-conder · 2026-08-16T22:17:22Z

 		inst.Devices = map[string]map[string]string{}
 	}
-	if err := mergeDevices(inst.Devices, snapshot.Sdks, name); err != nil {
+	usesZFS, err := poolUsesZFS(conn)


this adds an additional API call to these operations, do we have any info on the impact it has on latency? we could alternatively cache the storage pool driver at launch

jonathan-conder · 2026-08-16T23:15:41Z

The ZFS pool can't be deleted through LXD while the module is missing (lxc storage delete loads the driver, which fails), so the recovery is to reinstall Workshop: this recreates the workshop pool on an available driver (Btrfs) for a host that no longer provides ZFS.

I don't see how reinstalling Workshop will help. If LXD is unable to delete the pool then it will still be there after the reinstall.

For my manual test I had to build and install an upstream kernel, since simply uninstalling linux-main-modules-zfs-7.0.0-xx-generic isn't enough to reproduce a realistic scenario. For an automated test we could instead force the broken state, making it reproducible by just forcefully removing a package.

It's not reasonable to do that in a test. It's a rare enough case that I think some manual recovery steps are OK.

The bigger drawback is that this would require Spread to run on LXD VMs instead of containers. That would be a significant change to the current testing infrastructure.

we already do this, it's not a change at all

locnnil added 2 commits July 21, 2026 17:34

jonathan-conder reviewed Aug 16, 2026

View reviewed changes

Merge branch 'main' into add-btrfs-fallback

9ce2502

Navigation Menu

Uh oh!

Add Btrfs fallback for other Linux distros - #956

Add Btrfs fallback for other Linux distros#956
locnnil wants to merge 4 commits into
canonical:mainfrom
locnnil:add-btrfs-fallback

locnnil commented Jul 21, 2026

Uh oh!

jonathan-conder commented Jul 21, 2026

Uh oh!

jonathan-conder commented Jul 21, 2026

Uh oh!

locnnil commented Jul 22, 2026

Uh oh!

jonathan-conder commented Jul 22, 2026

Uh oh!

locnnil commented Aug 14, 2026

Uh oh!

jonathan-conder Aug 16, 2026

Uh oh!

jonathan-conder Aug 16, 2026

Uh oh!

locnnil Aug 19, 2026

Uh oh!

jonathan-conder Aug 20, 2026

Uh oh!

jonathan-conder Aug 16, 2026

Uh oh!

jonathan-conder Aug 16, 2026

Uh oh!

jonathan-conder Aug 16, 2026

Uh oh!

jonathan-conder commented Aug 16, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		return fmt.Errorf(`cannot use the %q storage pool, its storage driver may be unavailable (for example a ZFS pool after booting a kernel without the ZFS module): %w
		Boot into a kernel that provides the pool's storage driver, or reinstall Workshop to recreate the pool on an available driver`, storagePool, err)

Uh oh!

Conversation

locnnil commented Jul 21, 2026

Description

Self-review quick check

Docs

Uh oh!

jonathan-conder commented Jul 21, 2026

Uh oh!

jonathan-conder commented Jul 21, 2026

Uh oh!

locnnil commented Jul 22, 2026

Uh oh!

jonathan-conder commented Jul 22, 2026

Uh oh!

locnnil commented Aug 14, 2026

Uh oh!

jonathan-conder Aug 16, 2026

Choose a reason for hiding this comment

Uh oh!

jonathan-conder Aug 16, 2026

Choose a reason for hiding this comment

Uh oh!

locnnil Aug 19, 2026

Choose a reason for hiding this comment

Uh oh!

jonathan-conder Aug 20, 2026

Choose a reason for hiding this comment

Uh oh!

jonathan-conder Aug 16, 2026

Choose a reason for hiding this comment

Uh oh!

jonathan-conder Aug 16, 2026

Choose a reason for hiding this comment

Uh oh!

jonathan-conder Aug 16, 2026

Choose a reason for hiding this comment

Uh oh!

jonathan-conder commented Aug 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jonathan-conder commented Aug 16, 2026 •

edited

Loading