The Impossible Triangle

I have been using ZFS for a long time for my home servers and my NAS. Until recently, I have been sticking to OSes with built-in ZFS support such as Ubuntu and TrueNAS when I need ZFS. It has been working great, highly reliable and is my go-to choice for data storage.

However, since last year, I have been slowly converting my home servers from Ubuntu to NixOS and I have started to experience some friction:

  • ROCm is unstable under 6.12 and 6.18. The lowest kernel version that is stable is 6.19.
  • Historically, ROCm is somewhat sensitive to kernel driver versions.
  • NixOS only provides ZFS as DKMS.
  • ZFS Linux kernel support is usually 1 minor version behind.
  • NixOS doesn’t provide any kernel that has reached EOL.

This means:

  1. If I want to use NixOS + ROCm stably, I have to use the latest kernel.
  2. If I want to use NixOS + ZFS, I have to use an LTS kernel.

Hence, if I want to use NixOS + ZFS + ROCm, I have to at least wait until an LTS kernel that solves the unstable issue is released.

At the same time, a similar, yet different combination has surfaced for Debian + ZFS + Wi-Fi:

  1. Debian 13 ships kernel version 6.12 by default.
  2. The Intel BE200 Wi-Fi card is unstable under 6.12 and requires backported kernel / firmware.
  3. ZFS doesn’t always support kernel versions in Debian backports and can be easily broken by apt full-upgrade.

This means:

  1. If I want to use my Wi-Fi card, I have to use the kernel and firmware from Debian backports.
  2. If I want to guarantee ZFS compatibility, I have to use the default supplied kernel.

Therefore, this combination is impossible until Debian 14 comes out and ships with a kernel / firmware that works with ZFS.

This is partially due to Linux’s design. When two kernel version requirements surface, it becomes very difficult to choose a combination that satisfies all requirements. I have never been a fan of out-of-tree kernel modules in Linux, since they have always been fragile and sensitive to kernel versions.

After some consideration, I have decided to drop ZFS from most of my home servers and move to BTRFS. Since BTRFS is supported by the Linux kernel natively and doesn’t have to rely on DKMS, it won’t have kernel version issues like ZFS.

Feature Comparison

This is a comparison of features I need for the filesystem of my home servers, sorted by importance:

Feature ZFS BTRFS + LUKS XFS + MDADM + LUKS
Data-aware RAID* Yes Yes No
RAID 0/1/10 Yes Yes Yes
RAID with TRIM Yes Yes Yes? (Requires Deterministic Zero After TRIM for RAID Levels other than RAID 0)
Data Integrity** Yes Yes No
Encryption Yes (Native) Yes (via LUKS) Yes (via LUKS)
Snapshots Yes Yes No
Quota Yes Yes? (No refquota, questionable when used with snapshots) Yes
Parity RAID Yes Yes? (Unstable!) No
Alternative Mount Points Yes (via Datasets) Yes (via Subvolumes) No
Data-aware RAID*: This refers to the RAID being aware of free space and actually used space and being able to skip unused space during a scrub / repair.

Data Integrity**: This refers to the ability to detect bitrot and data corruption and the ability to understand which copy is correct and recover the correct data when enough parity is available. Traditional RAID can detect mismatches in data/parity but does not know which copy is correct.

Features I get from BTRFS

These are the reasons why I choose BTRFS to replace ZFS.

Data-aware RAID

This is the most important feature I need from ZFS. Compared to a traditional RAID, it has the following features:

  1. It is aware of space used. If only 10% of data is used, only 10% of data is scrubbed or repaired. This greatly speeds up the rebuilding process when a RAID is mostly empty.
  2. The data is not strictly mapped by sector ID. Hence, it can TRIM sectors like a normal disk and not confuse the RAID driver if disks return inconsistent data when a sector was not being used.
  3. Data integrity features (See next section for details).

At this moment, BTRFS is the only alternative to ZFS RAID.

Data Integrity

Traditionally, when a disk experiences bitrot, the data that resides on it becomes silently corrupted. When this is used with a RAID, the array can detect the differences between different disks, but it cannot tell which copy is the correct one. A filesystem with data integrity features can tell when the data has been corrupted. When a RAID is used, it can check between different copies and determine the correct copy by calculating the checksum of the corrupted file against available copies. When this is combined with regular scrubbing, it can correct bitrot without damaging the data.

Alternative Mount Points

Unlike traditional filesystems, both ZFS and BTRFS support mounting a part of the filesystem to a different mount point. This allows me to mount specific directories under a filesystem and share the free space and underlying RAID array without having to mount the parent directory.

ZFS Example

data-pool -> /mnt/data-pool
data-pool/docker -> /var/lib/docker
data-pool/home/futursolo -> /home/futursolo

BTRFS Example

/mnt/data-vol -> /mnt/data-vol
/mnt/data-vol/@docker -> /var/lib/docker
/mnt/data-vol/@home/@futursolo -> /home/futursolo

Features I will be missing

For BTRFS + LUKS, the only two features that I will be missing are Parity RAID and Disk Quota.

Parity RAID

I don’t have an array larger than four disks, so RAID 10 will do for the moment. There is an RFC to add a stable implementation of RAID 5/6 via RAID Stripe Tree to BTRFS. Hopefully, this feature will land later this year or early next year. I will switch to RAID 6 once it is stable.

File Quota

The second feature that is kind of questionable is Disk Quota. I do not need Disk Quota directly, however, Incus (a Linux Hypervisor Platform) needs a specific kind of disk quota that only tracks the space used by the file itself. However, BTRFS’s disk quota will include snapshots and can cause the free space to be smaller than declared space. The Incus documentation covers this topic in this Q&A.

Create a BTRFS Filesystem

In this example, a BTRFS filesystem is created with the following layout:

First, install the necessary tools:

sudo nala install cryptsetup btrfs-progs

Then determine which disks to use in the new array:

$ lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
... (Other Disks)
sdb      8:16   0   10G  0 disk
sdc      8:32   0   10G  0 disk
sdd      8:48   0   10G  0 disk
sde      8:64   0   10G  0 disk

In this case, sdb, sdc, sdd and sde are used.

Before creating the encrypted volumes, wipe all filesystems from these disks.

$ sudo wipefs --all /dev/sdb
$ sudo wipefs --all /dev/sdc
$ sudo wipefs --all /dev/sdd
$ sudo wipefs --all /dev/sde

After the existing filesystems are wiped, each drive is formatted. A passphrase is required during this process. It serves as a fallback if the keyfile is ever lost and the data needs to be recovered.

$ sudo cryptsetup luksFormat /dev/sdb

WARNING!
========
This will overwrite data on /dev/sdb irrevocably.

Are you sure? (Type 'yes' in capital letters): YES
Enter passphrase for /dev/sdb:
Verify passphrase:

After sdb is finished, repeat this step for sdc, sdd and sde.

In order to unlock at boot, a keyfile is created and stored at /root/.config/luks/data-vol.key.

$ sudo mkdir -p /root/.config/luks
$ sudo dd bs=4096 count=1 if=/dev/random of=/root/.config/luks/data-vol.key iflag=fullblock
$ sudo chmod 400 /root/.config/luks/data-vol.key

Once the key is created, it is added to a key slot in each drive.

$ sudo cryptsetup luksAddKey /dev/sdb /root/.config/luks/data-vol.key
Enter any existing passphrase:

After sdb is finished, repeat this step for sdc, sdd and sde.

Before configuring auto-unlock at boot, determine the UUIDs of each drive with the following command:

ls -l /dev/disk/by-uuid
total 0
lrwxrwxrwx 1 root root  9 Sep  6 04:08 0d037c9b-9ac3-4e96-9449-c828a1bce0c8 -> ../../sdc
lrwxrwxrwx 1 root root  9 Sep  6 04:08 1ef44241-6f2c-4dd5-b9b7-3fa7cc4dc7e1 -> ../../sde
lrwxrwxrwx 1 root root  9 Sep  6 04:08 82b79ea7-8d27-4042-b568-6ba0b04d04d3 -> ../../sdd
lrwxrwxrwx 1 root root  9 Sep  6 04:08 f32b1697-a14c-4df2-ad04-93d02e8278c4 -> ../../sdb

After confirming the UUIDs, add the drives to /etc/crypttab. For an SSD, enable discard in the last field, otherwise it can be left blank.

There are some security implications when enabling discard for an encrypted drive. Since I try to mirror the behaviour of TrueNAS native encryption, these implications apply to both setups. More about the security aspects in this documentation.

data-vol-dev0 UUID=f32b1697-a14c-4df2-ad04-93d02e8278c4 /root/.config/luks/data-vol.key discard
data-vol-dev1 UUID=0d037c9b-9ac3-4e96-9449-c828a1bce0c8 /root/.config/luks/data-vol.key discard
data-vol-dev2 UUID=82b79ea7-8d27-4042-b568-6ba0b04d04d3 /root/.config/luks/data-vol.key discard
data-vol-dev3 UUID=1ef44241-6f2c-4dd5-b9b7-3fa7cc4dc7e1 /root/.config/luks/data-vol.key discard

UUID is preferred here since device enumeration can change the order of sdb, sdc, sdd and sde upon boot.

Once saved, reboot the machine and verify with the following command:

$ lsblk
NAME            MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINTS
... (Other Disks)
sdb               8:16   0   10G  0 disk
└─data-vol-dev0 252:1    0   10G  0 crypt
sdc               8:32   0   10G  0 disk
└─data-vol-dev1 252:2    0   10G  0 crypt
sdd               8:48   0   10G  0 disk
└─data-vol-dev2 252:0    0   10G  0 crypt
sde               8:64   0   10G  0 disk
└─data-vol-dev3 252:3    0   10G  0 crypt

If all disks are unlocked as a data-vol-devX device, this means the setup is successful.

A BTRFS volume can be created with the following command:

$ sudo mkfs.btrfs -L data-vol \
    --metadata raid10 --data raid10 \
    --checksum sha256 \
    /dev/mapper/data-vol-dev0 \
    /dev/mapper/data-vol-dev1 \
    /dev/mapper/data-vol-dev2 \
    /dev/mapper/data-vol-dev3
btrfs-progs v6.17.1
See https://btrfs.readthedocs.io for more information.

Performing full device TRIM /dev/mapper/data-vol-dev0 (9.98GiB) ...
Performing full device TRIM /dev/mapper/data-vol-dev1 (9.98GiB) ...
Performing full device TRIM /dev/mapper/data-vol-dev3 (9.98GiB) ...
Performing full device TRIM /dev/mapper/data-vol-dev2 (9.98GiB) ...
Label:              data-vol
UUID:               8360888a-2581-4cb7-a738-dbc1ea4d5fe9
Node size:          16384
Sector size:        4096        (CPU page size: 4096)
Filesystem size:    39.94GiB
Block group profiles:
  Data:             RAID10            2.00GiB
  Metadata:         RAID10          128.00MiB
  System:           RAID10           16.00MiB
SSD detected:       no
Zoned device:       no
Features:           extref, skinny-metadata, no-holes, free-space-tree
Checksum:           sha256
Number of devices:  4
Devices:
   ID        SIZE  PATH
    1     9.98GiB  /dev/mapper/data-vol-dev0
    2     9.98GiB  /dev/mapper/data-vol-dev1
    3     9.98GiB  /dev/mapper/data-vol-dev2
    4     9.98GiB  /dev/mapper/data-vol-dev3

Note the UUID (8360888a-2581-4cb7-a738-dbc1ea4d5fe9). It will be needed later. Before mounting, create a mount point.

$ sudo mkdir -p /mnt/data-vol

Add the following line to /etc/fstab:

UUID=8360888a-2581-4cb7-a738-dbc1ea4d5fe9 /mnt/data-vol btrfs defaults,compress=zstd,noatime,autodefrag 0 0

The following options are specified here:

  • compress=zstd: Enable zstd compression.
  • noatime: Disable access time.
  • autodefrag: Enable automatic defragmentation.

The discard option is not specified here, as the fstrim.timer systemd timer will trim all free space weekly.

Mount the filesystem with the following commands:

$ sudo systemctl daemon-reload
$ sudo mount -a

If it mounts successfully, the mounted filesystem appears in df -h.

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
... (Other Filesystems)
/dev/dm-2        20G  5.7M   20G   1% /mnt/data-vol

Create a Subvolume

With the root filesystem mounted, a subvolume can be created with the following command:

$ sudo btrfs subvolume create /mnt/data-vol/@docker

Before replacing the docker subvolume, stop docker.

$ sudo systemctl stop docker

The mount point directory also needs to be cleaned. All existing docker data will be lost!

$ sudo rm -rf /var/lib/docker/*

Then add the following line to /etc/fstab:

UUID=8360888a-2581-4cb7-a738-dbc1ea4d5fe9 /var/lib/docker btrfs defaults,compress=zstd,noatime,autodefrag,subvol=@docker 0 0

Mount the subvolume with the following command:

$ sudo systemctl daemon-reload
$ sudo mount -a

The subvolume will show up in the mounted filesystems:

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
... (Other Filesystems)
/dev/dm-2        20G  5.7M   20G   1% /mnt/data-vol
/dev/dm-2        20G  5.7M   20G   1% /var/lib/docker

Conclusion

The reason I switched comes down to one thing: ZFS is out-of-tree on Linux. Every kernel update means DKMS has to recompile the module, and that process breaks whenever the kernel version ZFS supports doesn’t match what the distribution ships. On the other hand, BTRFS moves with the kernel and introduces no separate compatibility constraint. That single difference is what lets me escape the impossible triangle.

ZFS is still a fantastic filesystem and I will continue to use it on systems that ship with built-in support, such as TrueNAS. For other home servers, BTRFS gives me most features I need from ZFS without having to worry about kernel versions.

There are some trade-offs such as Parity RAID, but they are in progress to be solved by upcoming Linux kernels.

BTRFS has been maturing steadily, and for most of my home servers, it is starting to become the right answer.