BTRFS: Default Mount Options are Suboptimal

[the-burrito-triangle]

Currently, CachyOS uses unnecessary/suboptimal BTRFS mount options space_cache=v2 and compress=zstd in /etc/fstab.

1. Mount option space_cache=v2is not needed for a clean install and has been a default mkfs.btrfs option (free-space-tree) since btrfs-progs 5.15 and supported since kernel 4.5. It is redundant and unnecessary to add this mount option.

2. Mount option compress=zstd defaults to compression level 3, for NVMe drives, this greatly slows down throughput (zstd compression on CPU is slower than NVMe write speed). Compression level 1 (compress=zstd:1) has been the default on Fedora for some time now and is the better choice for modern storage media that uses PCIe vs. legacy interfaces like SATA. (Even for SATA, level 3 is questionable for SSDs with weak CPU compression performance, but nowhere near as bad when used with an NVMe drive.) Compression, can, however, improve write performance on slow interfaces (e.g. SATA with a HDD, USB 2.0 and possibly USB 3.0 storage devices) when using higher compression levels (e.g., >= 3). To achieve the best performance, the CPU's compression rate (for a given level) must be faster than the interface's write bandwidth. Otherwise compression stymies write performance as the CPU becomes the bottleneck and can actually slow the whole system down and kill battery life.

Space Cache V2 Notes:

https://btrfs.readthedocs.io/en/latest/mkfs.btrfs.html

(default since btrfs-progs 5.15, kernel support since 4.5)
Online conversion can be done by mounting with space_cache=v2, this is sufficient to be done one time.

https://wiki.tnonline.net/w/Btrfs/Space_Cache

With btrfs-progs 5.15 and newer, the Free Space Tree is default. There is no need to specify any special options to take advantage of the new cache.

[1Naim]

2. Mount option compress=zstd defaults to compression level 3, for NVMe drives, this greatly slows down throughput (zstd compression on CPU is slower than NVMe write speed). Compression level 1 (compress=zstd:1) has been the default on Fedora for some time now and is the better choice for modern storage media that uses PCIe vs. legacy interfaces like SATA. (Even for SATA, level 3 is questionable for SSDs with weak CPU compression performance, but nowhere near as bad when used with an NVMe drive.) Compression, can, however, improve write performance on slow interfaces (e.g. SATA with a HDD, USB 2.0 and possibly USB 3.0 storage devices) when using higher compression levels (e.g., >= 3). To achieve the best performance, the CPU's compression rate (for a given level) must be faster than the interface's write bandwidth. Otherwise compression stymies write performance as the CPU becomes the bottleneck and can actually slow the whole system down and kill battery life.

Unless you have data to back this up, I don't agree. See https://lore.kernel.org/all/20250416100634.GB13877@suse.cz/

[ptr1337]

space_cache=v2 can be indeed removed, but lets keep the compression to the default one. Fedora's testing is questionable as you see in the above link.

[the-burrito-triangle]

Thanks for replying guys, I appreciate it.

I understand everyone's conservative stance regarding using the default compression level (3), but here are some benchmarks to show what I'm talking about.

1. Download the "Silesia-compression-corpus", which is what Zstd uses for their benchmarks.
   NOTE: The silesia.zip from the github repo is what you want.

https://github.com/yewq/Silesia-compression-corpus

https://github.com/facebook/zstd

For reference, several fast compression algorithms were tested and compared on a desktop featuring a Core i7-9700K CPU @ 4.9GHz and running Ubuntu 24.04 (Linux 6.8.0-53-generic), using lzbench, an open-source in-memory benchmark by @inikep compiled with gcc 14.2.0, on the Silesia compression corpus.

2. Extract the zip, tar without compression, and run the built-in benchmark for zstd for compression levels 1 through 5 (can be set to 1..15 if you want):

#!/bin/bash
#BENCHMARK ZSTD COMPRESSION WITH SILESIA COMPRESSION CORPUS
ZIP=$(ls |grep -wc silesia.zip)
TAR=$(ls |grep -wc silesia.tar)
#download silesia.zip compression corpus, if needed
#can fail without internet access, so check zip was downloaded
if [[ $ZIP == 0 && $TAR == 0 ]]; then
  curl -LO https://github.com/yewq/Silesia-compression-corpus/raw/refs/heads/main/silesia.zip
  if $(ls |grep -wq silesia.zip); then
    ZIP=1
  fi
fi
#convert silesia zip to uncompressed tar, if needed
if [[ $ZIP == 1 && $TAR == 0 ]]; then
  mkdir silesia
  unzip ./silesia.zip -d ./silesia
  rm silesia.zip
  tar -cvf silesia.tar ./silesia
  rm -r silesia
  TAR=1
fi
#run zstd benchmark
#can fail without internet access, so check that tar was made / is present
if [ $TAR == 1 ]; then
  echo -e "$(zstd -V)" "\nCompressing the Silesia Corpus..."
  for i in {1..5}; do zstd -b$i -i3 ./silesia.tar; done
fi

With the silesia.tar on a tmpfs directory and using an i9-11900T (limited to 4.5GHz) set to "performance" power mode I get:

$ ./zstd_silesia_test.sh
*** Zstandard CLI (64-bit) v1.5.6, by Yann Collet *** 
Compressing the Silesia Corpus...
 1#silesia.tar       : 211957760 ->  73420606 (x2.887),  561.7 MB/s, 1773.4 MB/s
 2#silesia.tar       : 211957760 ->  69502622 (x3.050),  462.3 MB/s, 1635.7 MB/s
 3#silesia.tar       : 211957760 ->  66522722 (x3.186),  327.7 MB/s, 1598.0 MB/s
 4#silesia.tar       : 211957760 ->  65327227 (x3.245),  302.4 MB/s, 1586.7 MB/s
 5#silesia.tar       : 211957760 ->  63040850 (x3.362),  168.9 MB/s, 1581.8 MB/s

This closely matches what zstd shows for an i9-9700K at 4.9GHz
https://github.com/facebook/zstd

Compressor name  Ratio  Compression  Decompress.
zstd 1.5.7 -1    2.896  510 MB/s     1550 MB/s

And here is lzbench results for an EPIC 9554 at 3.1GHz
https://github.com/inikep/lzbench

Compressor name  Compression  Decompress.  Compr. size  Ratio
zstd 1.5.6 -1    422 MB/s     1347 MB/s    73421914     34.64
zstd 1.5.6 -2    344 MB/s     1246 MB/s    69503444     32.79
zstd 1.5.6 -5    125 MB/s     1197 MB/s    63040310     29.74

Obviously, the kernel's Zstd implementation is slightly different than user space Zstd. I'll need to make a ZRAM mount with BTRFS to get more accurate results, but the obvious take away from the above benchmarks is that compression level 3 has a stark 42% drop in write speed relative to level 1 for an approximate 3% difference in compressed file size relative to the original file [(size_lvl1 - size_lvl3) / size]*100, or a 10% further reduction in size relative to level 1 (size_lvl3 / size_lvl1)*100. So not much gained, space wise, for the performance penalty. Zstd with level 1 is a decent all-round choice for good compression. LZO I have yet to test, but may actually be the better choice for fast NVMe drives.

When I get time, I'll try some benchmarks with LZO and Zstd on a ZRAM mount, which should use the kernel's internal compression algorithms. If anyone has suggestions on how to accurately test compression speeds with the Silesia Corpus on a ZRAM mount or another method, let me know. I'm not here to spread FUD or misinformation. I just want reasonable defaults that make sense for a given interface's write bandwidth and CPU compression speed.

Finally, an important point to bring up is that uncompressible files are still partially compressed (to see if the file is compressible) so writing is always hit with a small penalty when compression is enabled--which is expected. However, uncompressible files can show reasonably fast "compression" with Zstd.

CachyOS iso compressed with the same benchmark parameters and tmpfs mount:

for i in {1..5}; do zstd -b$i -i3 ./cachyos-desktop-linux-250422.iso; done
Not enough memory; testing 2709 MB only...
 1#-linux-250422.iso :2840941909 ->2836396000 (x1.002), 4817.3 MB/s  17537.6 MB/s
Not enough memory; testing 2709 MB only...
 2#-linux-250422.iso :2840941909 ->2833709837 (x1.003), 4747.7 MB/s  17548.2 MB/s
Not enough memory; testing 2709 MB only...
 3#-linux-250422.iso :2840941909 ->2832032364 (x1.003), 3482.8 MB/s, 17537.7 MB/s
Not enough memory; testing 2709 MB only...
 4#-linux-250422.iso :2840941909 ->2832001239 (x1.003), 3749.3 MB/s, 17503.1 MB/s
Not enough memory; testing 2709 MB only...
 5#-linux-250422.iso :2840941909 ->2831772266 (x1.003), 2386.3 MB/s, 17373.2 MB/s

Note the much faster write and read speeds than with the Silesia Corpus. While fast, the BTRFS compressibility check will slow down as compression level is increased. Obviously, the compressibility check is only done on a small part of the file, but the perf hit remains (effectively a delayed write). This is why choosing the right algorithm and compression level is important for fast storage devices to reduce write latency. Slower storage devices, on the other hand, will likely not see a meaningful drop in perf, as either the interface is too slow (e.g., USB) or the storage media itself is too slow (HDD).

[ventureoo]

Thanks for replying guys, I appreciate it.

I understand everyone's conservative stance regarding using the default compression level (3), but here are some benchmarks to show what I'm talking about.

1. Download the "Silesia-compression-corpus", which is what Zstd uses for their benchmarks.
   NOTE: The `silesia.zip` from the github repo is what you want.

https://github.com/yewq/Silesia-compression-corpus

https://github.com/facebook/zstd

For reference, several fast compression algorithms were tested and compared on a desktop featuring a Core i7-9700K CPU @ 4.9GHz and running Ubuntu 24.04 (Linux 6.8.0-53-generic), using lzbench, an open-source in-memory benchmark by @inikep compiled with gcc 14.2.0, on the Silesia compression corpus.

2. Extract the zip, tar without compression, and run the built-in benchmark for zstd for compression levels `1` through `5` (can be set to `1..15` if you want):

#!/bin/bash
#BENCHMARK ZSTD COMPRESSION WITH SILESIA COMPRESSION CORPUS
ZIP=$(ls |grep -wc silesia.zip)
TAR=$(ls |grep -wc silesia.tar)
#download silesia.zip compression corpus, if needed
#can fail without internet access, so check zip was downloaded
if [[ $ZIP == 0 && $TAR == 0 ]]; then
  curl -LO https://github.com/yewq/Silesia-compression-corpus/raw/refs/heads/main/silesia.zip
  if $(ls |grep -wq silesia.zip); then
    ZIP=1
  fi
fi
#convert silesia zip to uncompressed tar, if needed
if [[ $ZIP == 1 && $TAR == 0 ]]; then
  mkdir silesia
  unzip ./silesia.zip -d ./silesia
  rm silesia.zip
  tar -cvf silesia.tar ./silesia
  rm -r silesia
  TAR=1
fi
#run zstd benchmark
#can fail without internet access, so check that tar was made / is present
if [ $TAR == 1 ]; then
  echo -e "$(zstd -V)" "\nCompressing the Silesia Corpus..."
  for i in {1..5}; do zstd -b$i -i3 ./silesia.tar; done
fi

With the silesia.tar on a tmpfs directory and using an i9-11900T (limited to 4.5GHz) set to "performance" power mode I get:

$ ./zstd_silesia_test.sh
*** Zstandard CLI (64-bit) v1.5.6, by Yann Collet *** 
Compressing the Silesia Corpus...
 1#silesia.tar       : 211957760 ->  73420606 (x2.887),  561.7 MB/s, 1773.4 MB/s
 2#silesia.tar       : 211957760 ->  69502622 (x3.050),  462.3 MB/s, 1635.7 MB/s
 3#silesia.tar       : 211957760 ->  66522722 (x3.186),  327.7 MB/s, 1598.0 MB/s
 4#silesia.tar       : 211957760 ->  65327227 (x3.245),  302.4 MB/s, 1586.7 MB/s
 5#silesia.tar       : 211957760 ->  63040850 (x3.362),  168.9 MB/s, 1581.8 MB/s

This closely matches what zstd shows for an i9-9700K at 4.9GHz https://github.com/facebook/zstd

Compressor name  Ratio  Compression  Decompress.
zstd 1.5.7 -1    2.896  510 MB/s     1550 MB/s

And here is lzbench results for an EPIC 9554 at 3.1GHz https://github.com/inikep/lzbench

Compressor name  Compression  Decompress.  Compr. size  Ratio
zstd 1.5.6 -1    422 MB/s     1347 MB/s    73421914     34.64
zstd 1.5.6 -2    344 MB/s     1246 MB/s    69503444     32.79
zstd 1.5.6 -5    125 MB/s     1197 MB/s    63040310     29.74

Obviously, the kernel's Zstd implementation is slightly different than user space Zstd. I'll need to make a ZRAM mount with BTRFS to get more accurate results, but the obvious take away from the above benchmarks is that compression level 3 has a stark 50% roll-off in write speed relative to level 1 for an approximate 3% difference in compressed file size relative to the original file [(size_lvl1 - size_lvl3) / size]*100, or a 10% further reduction in size relative to level 1 (size_lvl3 / size_lvl1)*100. So not much gained, space wise, for the performance penalty. Zstd with level 1 is a decent all-round choice for good compression. LZO, I have yet to test, but may actually be the better choice for fast NVMe drives.

When I get time, I'll try some benchmarks with LZO and Zstd on a ZRAM mount, which should use the kernel's internal compression algorithms. If anyone has suggestions on how to accurately test compression speeds with the Silesia Corpus on a ZRAM mount or another method, let me know. I'm not here to spread FUD or misinformation. I just want reasonable defaults that make sense for a given interface's write bandwidth and CPU compression speed.

Finally, an important point to bring up is that uncompressible files are still partially compressed (to see if the file is compressible) so writing is always hit with a small penalty when compression is enabled--which is expected. However, uncompressible files can show reasonably fast "compression" with Zstd.

CachyOS iso compressed with the same benchmark parameters and tmpfs mount:

for i in {1..5}; do zstd -b$i -i3 ./cachyos-desktop-linux-250422.iso; done
Not enough memory; testing 2709 MB only...
 1#-linux-250422.iso :2840941909 ->2836396000 (x1.002), 4817.3 MB/s  17537.6 MB/s
Not enough memory; testing 2709 MB only...
 2#-linux-250422.iso :2840941909 ->2833709837 (x1.003), 4747.7 MB/s  17548.2 MB/s
Not enough memory; testing 2709 MB only...
 3#-linux-250422.iso :2840941909 ->2832032364 (x1.003), 3482.8 MB/s, 17537.7 MB/s
Not enough memory; testing 2709 MB only...
 4#-linux-250422.iso :2840941909 ->2832001239 (x1.003), 3749.3 MB/s, 17503.1 MB/s
Not enough memory; testing 2709 MB only...
 5#-linux-250422.iso :2840941909 ->2831772266 (x1.003), 2386.3 MB/s, 17373.2 MB/s

Note the much faster write and read speeds than with the Silesia Corpus. While fast, the BTRFS compressibility check will slow down as compression level is increased. Obviously, the compressibility check is only done on a small part of the file, but the perf hit remains (effectively a delayed write). This is why choosing the right algorithm and compression level is important for fast storage devices to reduce write latency. Slower storage devices, on the other hand, will likely not see a meaningful drop in perf, as either the interface is too slow (e.g., USB) or the storage media itself is too slow (HDD).

I don't think that such benchmarks are representative in case of filesystem compression. Btrfs does not compress files in a single stream, but breaks it into 128 KB chunks. Besides, you don't take into account that Btrfs does it in parallel, when the script you presented runs zstd in a single thread.

[the-burrito-triangle]

I don't think that such benchmarks are representative in case of filesystem compression. Btrfs does not compress files in a single stream, but breaks it into 128 KB chunks. Besides, you don't take into account that Btrfs does it in parallel, when the script you presented runs zstd in a single thread.

That's fair, and you bring up a good point about parallelism of Btrfs.

Setting option -T, and reruning the previous compression benchmark:

$ ./zstd_silesia_test.sh
# default
Compressing the Silesia Corpus with one thread...
 1#silesia.tar       : 211957760 ->  73420606 (x2.887),  561.0 MB/s, 1771.4 MB/s
 2#silesia.tar       : 211957760 ->  69502622 (x3.050),  460.1 MB/s, 1627.0 MB/s
 3#silesia.tar       : 211957760 ->  66522722 (x3.186),  327.8 MB/s, 1598.7 MB/s
 4#silesia.tar       : 211957760 ->  65327227 (x3.245),  290.5 MB/s, 1580.4 MB/s
 5#silesia.tar       : 211957760 ->  63040850 (x3.362),  172.9 MB/s, 1584.7 MB/s
# option -T0
Compressing the Silesia Corpus with threads = cores...
 1#silesia.tar       : 211957760 ->  73492861 (x2.884), 3029.6 MB/s, 1771.3 MB/s
 2#silesia.tar       : 211957760 ->  69589385 (x3.046), 2413.6 MB/s, 1632.5 MB/s
 3#silesia.tar       : 211957760 ->  66554373 (x3.185), 1459.1 MB/s, 1599.8 MB/s
 4#silesia.tar       : 211957760 ->  65399874 (x3.241),  741.6 MB/s, 1581.4 MB/s
 5#silesia.tar       : 211957760 ->  63103791 (x3.359),  641.1 MB/s, 1576.1 MB/s
# option -T$(nproc)
Compressing the Silesia Corpus with all available threads...
 1#silesia.tar       : 211957760 ->  73492861 (x2.884), 3710.5 MB/s, 1770.7 MB/s
 2#silesia.tar       : 211957760 ->  69589385 (x3.046), 2673.4 MB/s, 1638.8 MB/s
 3#silesia.tar       : 211957760 ->  66554373 (x3.185), 1262.9 MB/s, 1599.0 MB/s
 4#silesia.tar       : 211957760 ->  65399874 (x3.241),  647.8 MB/s, 1590.7 MB/s
 5#silesia.tar       : 211957760 ->  63103791 (x3.359),  589.0 MB/s, 1572.9 MB/s

This test actually highlights why compression levels higher than 1 are potentially suboptimal for NVMe drives (assuming similar behavior between compression levels for the in-kernel zstd algorithm, ignoring chunk size), as compression speed doesn't necessarily scale linearly with number of threads. Using the level 1 compression results from earlier, one might naively assume 8*561.7 MiB/s = 4493.6 MiB/s CPU compression speed (using number of physical cores times the single-threaded performance).

I'll try more concrete tests with a ZRAM mount when I have time. Worst case, I cure myself of ignorance and leave well-enough alone; best case, I can tune my system for a better compromise between compression and write performance.

[sahfd4eiugf]

I just ran into this exact issue installing CachyOS for the first time. ZSTD-3 is defaulted and AFAIK its not possible to change to LZO until after the installation using btrfs set property on all @ subvolumes

[ventureoo]

I just ran into this exact issue installing CachyOS for the first time. ZSTD-3 is defaulted and AFAIK its not possible to change to LZO until after the installation using btrfs set property on all @ subvolumes

I reduced compression level in #64 to 1, but I'm not sure if really want to use LZO for NVMe, since LZO compresses data much worse than Zstd even with level 1.

[ptr1337]

Yeah, we will not change to lzo.

[the-burrito-triangle]

How hard would it be to add another drop-down menu to select compression level for btrfs during a fresh install? This way users who want to change the default value can set it before installation. One thing I like about openSUSE's installer is how much the user can configure before installation (can change kernel boot-time arguments and manually select any arbitrary packages to install).

In general, zstd:1 is fine for older NVMes, TB3/4, USB4, USB3.x 10Gbs SSDs. It is, however, questionable for PCIe 5.0 and 4.0 NVMe drives though, in this case lzo is likely the better choice. zstd:2 is best for USB 3.0 5Gbs / SATA SSDs and zstd:3 is fine for anything slower (HDDs or really old SSDs.). A storage read benchmark could probably auto select the right compression algorithm / level at install time. Or maybe the storage benchmark test results are printed and the user can decide which compression level to pick?

Regardless, I think setting zstd:1 as default makes sense for most systems. It's a fair compromise of compression vs. speed and is the best fit for both older and modern HW.