Extending the archive directory across multiple disks with MergerFS
We recommend planning the archive storage space in advance and, for large volumes of data, using dedicated technologies such as ZFS, which evenly balance data across disks, provide fault tolerance, and can seamlessly scale data arrays to dozens of disks and hundreds of terabytes of data.
The problem
You launched Yucca in a basic configuration with the archive on the system disk. Over time, the number of cameras and the amount of archive grew, and the current disk is no longer enough. You could add one or more disks to the system and evenly distribute the archive storage across them – this would require stopping Yucca, creating a disk array, and migrating all the data. In this case the service may be unavailable for several hours, which is unacceptable.
The solution
If you want to gradually extend the directory used for archive storage by adding additional disks, but without using RAID technologies such as ZFS or mdadm, and without rebuilding your entire infrastructure, you can use a union filesystem.
There are several solutions in this space, but we'll focus on mergerfs, and everything below is in its context. A comparison of mergerfs with other solutions is available here.
mergerfs is a union filesystem designed to simplify storing and managing files across multiple regular storage devices; it's similar to mhddfs, UnionFS, and Aufs. You can find more details in the project's official repository – https://github.com/trapexit/mergerfs.
Roughly speaking, mergerfs creates a virtual directory that can span several real directories on different disks with different filesystems, such as ext4 or XFS. mergerfs does not split files into parts – they are stored in full, with all metadata, in one of the subdirectories:
A + B = C
/disk1 /disk2 /merged
| | |
+-- /dir1 +-- /dir1 +-- /dir1
| | | | | |
| +-- file1 | +-- file2 | +-- file1
| | +-- file3 | +-- file2
+-- /dir2 | | +-- file3
| | +-- /dir3 |
| +-- file4 | +-- /dir2
| +-- file5 | |
+-- file6 | +-- file4
|
+-- /dir3
| |
| +-- file5
|
+-- file6
Pros
- Easy to extend space by adding new disks;
- Easy to set up – mergerfs is a userspace filesystem, so you can install it as a regular program on any Linux;
- No special requirements for the directories being merged – they can be disks of different sizes, filesystems, or even network shares;
- Doesn't require rebuilding the whole infrastructure.
Cons
- No load balancing – the load on some disks may be higher than on others, and as a result the archive from some cameras may be served slower than from others;
- No redundancy – a disk failure will result in complete data loss.
Installation
Installation instructions are available in the repository.
If your distribution's package manager includes mergerfs, check whether its version is up to date. If the version is outdated, we recommend using the latest release found on the page. Details for common distributions are provided below.
The steps described below were performed on Ubuntu 22.04, and mergerfs was installed via the deb package from the repository.
For openSUSE there's an OBS repository with the latest versions.
Configuration
In this example, a virtual machine with a 20 gigabyte disk and two additional 10 gigabyte disks:
# df -h | grep -v -E "(tmpfs|overlay)"
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 19G 4.5G 14G 26% /
/dev/sda15 253M 142K 252M 1% /boot/efi
/dev/sdb 9.8G 28K 9.3G 1% /mnt/HC_Volume_100511911
/dev/sdc 9.8G 28K 9.3G 1% /mnt/HC_Volume_100511912
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 0 19.1G 0 disk
├─sda1 8:1 0 18.8G 0 part /
├─sda14 8:14 0 1M 0 part
└─sda15 8:15 0 256M 0 part /boot/efi
sdb 8:16 0 10G 0 disk /mnt/HC_Volume_100511911
sdc 8:32 0 10G 0 disk /mnt/HC_Volume_100511912
Yucca is running on the server, with 6 cameras and an archive added:
cat /opt/yucca/yucca.toml | grep "alloc_dir ="
alloc_dir = "/opt/yucca/data/alloc"
Data is being successfully written to the archive:
# tree /opt/yucca/data/alloc/ | head
/opt/yucca/data/alloc/
├── 2
│ ├── archive
│ │ └── 2024
│ │ └── 03
│ │ └── 27
│ │ ├── 15.dvr
│ │ └── 15.ranges
│ ├── logs
│ │ └── ffmpeg.log
│ └── segment
...
Now, inside the mount points of our additional disks, let's create a directory that we'll mount and assign permissions so that Yucca can write to it:
mkdir -p /mnt/HC_Volume_100511911/yucca
mkdir -p /mnt/HC_Volume_100511912/yucca
chown yucca:yucca /mnt/HC_Volume_100511911/yucca
chown yucca:yucca /mnt/HC_Volume_100511912/yucca
Create the directory where we'll mount the union volume:
mkdir -p /mnt/yucca_mergerfs
Mount the union volume:
mergerfs -o fsname=mergerfs,cache.files=partial,dropcacheonclose=true,category.create=mfs,moveonenospc=true,minfreespace=20G /mnt/HC_Volume_100511911/yucca:/mnt/HC_Volume_100511912/yucca:/opt/yucca/data/alloc /mnt/yucca_mergerfs
You can read about the meaning of all the parameters in the official documentation. Let's check that the volume is mounted:
The resulting volume has a size equal to the sum of all the merged volumes.
# df -h | grep -v -E "(tmpfs|overlay)"
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 19G 5.2G 13G 30% /
/dev/sda15 253M 142K 252M 1% /boot/efi
/dev/sdb 9.8G 28K 9.3G 1% /mnt/HC_Volume_100511911
/dev/sdc 9.8G 28K 9.3G 1% /mnt/HC_Volume_100511912
mnt/HC_Volume_100511911/yucca:mnt/HC_Volume_100511912/yucca:opt/yucca/data/alloc 38G 5.2G 31G 15% /mnt/yucca_mergerfs
Let's check that the /mnt/yucca_mergerfs and /opt/yucca/data/alloc directories have the same set of files:
ls /mnt/yucca_mergerfs/
2 3 4 5 6 8
ls /opt/yucca/data/alloc/
2 3 4 5 6 8
Now, using any text editor, change the alloc_dir parameter:
# sed -i 's|/opt/yucca/data/alloc|/mnt/yucca_mergerfs|' /opt/yucca/yucca.toml
# cat /opt/yucca/yucca.toml | grep "alloc_dir ="
alloc_dir = "/mnt/yucca_mergerfs"
Restart the Yucca server and check that everything is fine:
systemctl restart yucca
systemctl status yucca
Yucca writes segments in hour-long chunks, so data on the new disks /mnt/HC_Volume_100511911/yucca and /mnt/HC_Volume_100511912/yucca will only appear once the next hour begins.
root@ubuntu-2gb-fsn1-1:~# tree /mnt/HC_Volume_100511911/yucca/ | head
/mnt/HC_Volume_100511911/yucca/
├── 2
│ ├── archive
│ └── logs
│ └── ffmpeg.log
├── 3
│ ├── archive
│ │ └── 2024
│ │ └── 03
│ │ └── 27
root@ubuntu-2gb-fsn1-1:~# tree /mnt/HC_Volume_100511912/yucca/ | head
/mnt/HC_Volume_100511912/yucca/
├── 2
│ ├── archive
│ │ └── 2024
│ │ └── 03
│ │ └── 27
│ │ ├── 16.dvr
│ │ └── 16.ranges
│ ├── logs
│ └── segment
To have the union directory mounted on system reboot, add an entry to /etc/fstab:
# yucca mergerfs
/mnt/HC_Volume_100511911/yucca:/mnt/HC_Volume_100511912/yucca:/opt/yucca/data/alloc /mnt/yucca_mergerfs mergerfs cache.files=partial,dropcacheonclose=true,category.create=mfs,moveonenospc=true,minfreespace=20G 0 0
Check that there are no errors:
mount -a
Reboot the system to verify.
Conclusion
We haven't tested mergerfs under heavy load or in other exceptional situations, but we can say that, in general, everything works stably and no issues are observed. If you do run into problems, reach out to the project's tracker, where you can also financially support the developers.