1# Test setup 2 3Processor: x86_64, Intel(R) Xeon(R) Platinum 8369B CPU @ 2.70GHz * 2 VCores 4 5Storage: Cloud disk, 3000 IOPS upper limit 6 7OS Kernel: Linux 6.2 8 9Software: LZ4 1.9.3, erofs-utils 1.6, squashfs-tools 4.5.1 10 11Disclaimer: Test results could be varied from different hardware and/or data patterns. Therefore, the following results are **ONLY for reference**. 12 13# Benchmark on multiple files 14 15[Rootfs of Debian docker image](https://github.com/debuerreotype/docker-debian-artifacts/blob/dist-amd64/bullseye/rootfs.tar.xz?raw=true) is used as the dataset, which contains 7000+ files and directories. 16Note that that dataset can be replaced regularly, and the SHA1 of the snapshot "rootfs.tar.xz" used here is "aee9b01a530078dbef8f08521bfcabe65b244955". 17 18## Image size 19 20| Size | Filesystem | Cluster size | Build options | 21|-----------|------------|--------------|----------------------------------------------------------------| 22| 124669952 | erofs | uncompressed | -T0 [^1] | 23| 124522496 | squashfs | uncompressed | -noD -noI -noX -noF -no-xattrs -all-time 0 -no-duplicates [^2] | 24| 73601024 | squashfs | 4096 | -b 4096 -comp lz4 -Xhc -no-xattrs -all-time 0 | 25| 73121792 | erofs | 4096 | -zlz4hc,12 [^3] -C4096 -Efragments -T0 | 26| 67162112 | squashfs | 16384 | -b 16384 -comp lz4 -Xhc -no-xattrs -all-time 0 | 27| 65478656 | erofs | 16384 | -zlz4hc,12 -C16384 -Efragments -T0 | 28| 61456384 | squashfs | 65536 | -b 65536 -comp lz4 -Xhc -no-xattrs -all-time 0 | 29| 59834368 | erofs | 65536 | -zlz4hc,12 -C65536 -Efragments -T0 | 30| 59150336 | squashfs | 131072 | -b 131072 -comp lz4 -Xhc -no-xattrs -all-time 0 | 31| 58515456 | erofs | 131072 | -zlz4hc,12 -C131072 -Efragments -T0 | 32 33[^1]: Forcely reset all timestamps to match squashfs on-disk basic inodes for now. 34[^2]: Currently erofs-utils doesn't actively de-duplicate identical files although the on-disk format supports this. 35[^3]: Because squashfs uses level 12 for LZ4HC by default. 36 37## Sequential data access 38 39```bash 40hyperfine -p "echo 3 > /proc/sys/vm/drop_caches; sleep 1" "tar cf - . | cat > /dev/null" 41``` 42 43| Filesystem | Cluster size | Time | 44|------------|--------------|---------------------------------| 45| squashfs | 4096 | 10.257 s ± 0.031 s | 46| erofs | uncompressed | 1.111 s ± 0.022 s | 47| squashfs | uncompressed | 1.034 s ± 0.020 s | 48| squashfs | 131072 | 941.3 ms ± 7.5 ms | 49| erofs | 4096 | 848.1 ms ± 17.8 ms | 50| erofs | 131072 | 724.2 ms ± 11.0 ms | 51 52## Sequential metadata access 53 54```bash 55hyperfine -p "echo 3 > /proc/sys/vm/drop_caches; sleep 1" "tar cf /dev/null ." 56``` 57 58| Filesystem | Cluster size | Time | 59|------------|--------------|---------------------------------| 60| erofs | uncompressed | 419.6 ms ± 8.2 ms | 61| squashfs | 4096 | 142.5 ms ± 5.4 ms | 62| squashfs | uncompressed | 129.2 ms ± 3.9 ms | 63| squashfs | 131072 | 125.4 ms ± 4.0 ms | 64| erofs | 4096 | 75.5 ms ± 3.5 ms | 65| erofs | 131072 | 65.8 ms ± 3.6 ms | 66 67[ Note that erofs-utils currently doesn't perform quite well for such cases due to metadata arrangement when building. It will be fixed in the later versions. ] 68 69## Small random data access (~7%) 70 71```bash 72find mnt -type f -printf "%p\n" | sort -R | head -n 500 > list.txt 73hyperfine -p "echo 3 > /proc/sys/vm/drop_caches; sleep 1" "cat list.txt | xargs cat > /dev/null" 74``` 75 76| Filesystem | Cluster size | Time | 77|------------|--------------|---------------------------------| 78| squashfs | 4096 | 1.386 s ± 0.032 s | 79| squashfs | uncompressed | 1.083 s ± 0.044 s | 80| squashfs | 131072 | 1.067 s ± 0.046 s | 81| erofs | 4096 | 249.6 ms ± 6.5 ms | 82| erofs | uncompressed | 237.8 ms ± 6.3 ms | 83| erofs | 131072 | 189.6 ms ± 7.8 ms | 84 85 86## Small random metadata access (~7%) 87 88```bash 89find mnt -type f -printf "%p\n" | sort -R | head -n 500 > list.txt 90hyperfine -p "echo 3 > /proc/sys/vm/drop_caches; sleep 1" "cat list.txt | xargs stat" 91``` 92 93| Filesystem | Cluster size | Time | 94|------------|--------------|---------------------------------| 95| squashfs | 4096 | 817.0 ms ± 34.5 ms | 96| squashfs | 131072 | 801.0 ms ± 40.1 ms | 97| squashfs | uncompressed | 741.3 ms ± 18.2 ms | 98| erofs | uncompressed | 197.8 ms ± 4.1 ms | 99| erofs | 4096 | 63.1 ms ± 2.0 ms | 100| erofs | 131072 | 60.7 ms ± 3.6 ms | 101 102## Full random data access (~100%) 103 104```bash 105find mnt -type f -printf "%p\n" | sort -R > list.txt 106hyperfine -p "echo 3 > /proc/sys/vm/drop_caches; sleep 1" "cat list.txt | xargs cat > /dev/null" 107``` 108 109| Filesystem | Cluster size | Time | 110|------------|--------------|---------------------------------| 111| squashfs | 4096 | 20.668 s ± 0.040 s | 112| squashfs | uncompressed | 12.543 s ± 0.041 s | 113| squashfs | 131072 | 11.753 s ± 0.412 s | 114| erofs | uncompressed | 1.493 s ± 0.023 s | 115| erofs | 4096 | 1.223 s ± 0.013 s | 116| erofs | 131072 | 598.2 ms ± 6.6 ms | 117 118## Full random metadata access (~100%) 119 120```bash 121find mnt -type f -printf "%p\n" | sort -R > list.txt 122hyperfine -p "echo 3 > /proc/sys/vm/drop_caches; sleep 1" "cat list.txt | xargs stat" 123``` 124 125| Filesystem | Cluster size | Time | 126|------------|--------------|---------------------------------| 127| squashfs | 131072 | 9.212 s ± 0.467 s | 128| squashfs | 4096 | 8.905 s ± 0.147 s | 129| squashfs | uncompressed | 7.961 s ± 0.045 s | 130| erofs | 4096 | 661.2 ms ± 14.9 ms | 131| erofs | uncompressed | 125.8 ms ± 6.6 ms | 132| erofs | 131072 | 119.6 ms ± 5.5 ms | 133 134 135# FIO benchmark on a single large file 136 137`silesia.tar` (203M) is used to benchmark, which could be generated from unzipping [silesia.zip](http://mattmahoney.net/dc/silesia.zip) and tar. 138 139## Image size 140 141| Size | Filesystem | Cluster size | Build options | 142|-----------|------------|--------------|-----------------------------------------------------------| 143| 114339840 | squashfs | 4096 | -b 4096 -comp lz4 -Xhc -no-xattrs | 144| 104972288 | erofs | 4096 | -zlz4hc,12 -C4096 | 145| 98033664 | squashfs | 16384 | -b 16384 -comp lz4 -Xhc -no-xattrs | 146| 89571328 | erofs | 16384 | -zlz4hc,12 -C16384 | 147| 85143552 | squashfs | 65536 | -b 65536 -comp lz4 -Xhc -no-xattrs | 148| 81211392 | squashfs | 131072 | -b 131072 -comp lz4 -Xhc -no-xattrs | 149| 80519168 | erofs | 65536 | -zlz4hc,12 -C65536 | 150| 78888960 | erofs | 131072 | -zlz4hc,12 -C131072 | 151 152## Sequential I/Os 153 154```bash 155fio -filename=silesia.tar -bs=4k -rw=read -name=job1 156``` 157 158| Filesystem | Cluster size | Bandwidth | 159|------------|--------------|-----------| 160| erofs | 65536 | 624 MiB/s | 161| erofs | 16384 | 600 MiB/s | 162| erofs | 4096 | 569 MiB/s | 163| erofs | 131072 | 535 MiB/s | 164| squashfs | 131072 | 236 MiB/s | 165| squashfs | 65536 | 157 MiB/s | 166| squashfs | 16384 | 55.2MiB/s | 167| squashfs | 4096 | 12.5MiB/s | 168 169## Full Random I/Os 170 171```bash 172fio -filename=silesia.tar -bs=4k -rw=randread -name=job1 173``` 174 175| Filesystem | Cluster size | Bandwidth | 176|------------|--------------|-----------| 177| erofs | 131072 | 242 MiB/s | 178| squashfs | 131072 | 232 MiB/s | 179| erofs | 65536 | 198 MiB/s | 180| squashfs | 65536 | 150 MiB/s | 181| erofs | 16384 | 96.4MiB/s | 182| squashfs | 16384 | 49.5MiB/s | 183| erofs | 4096 | 33.7MiB/s | 184| squashfs | 4096 | 6817KiB/s | 185 186## Small Random I/Os (~5%) 187 188```bash 189fio -filename=silesia.tar -bs=4k -rw=randread --io_size=10m -name=job1 190``` 191 192| Filesystem | Cluster size | Bandwidth | 193|------------|--------------|-----------| 194| erofs | 131072 | 19.2MiB/s | 195| erofs | 65536 | 16.9MiB/s | 196| squashfs | 131072 | 15.1MiB/s | 197| erofs | 16384 | 14.7MiB/s | 198| squashfs | 65536 | 13.8MiB/s | 199| erofs | 4096 | 13.0MiB/s | 200| squashfs | 16384 | 11.7MiB/s | 201| squashfs | 4096 | 4376KiB/s | 202