Setting Up LUKS and Mergerfs + Snapraid on Proxmox

Wanted to use a combine storage but zfs pool cannot add mismatched drives so had to use another option: Snapraid

Snapraid and mergerfs diagram

Setting up LUKS on my disks

After setting up Proxmox on Debian with LUKS encryption on the disk, its now time for me to setup LUKS for all my hard drives. This was fairly straightforward.

lsblk -f 

Next, I shred all the disk contents since I am starting off clean:

shred -v -n 1 /dev/sdX

Then I encrypted the disks

cryptsetup luksFormat /dev/sdX

Following the instructions and providing the passphrases for all of my disks ensured that they become encrypted. Now that they are encrypted, they are no longer available to be mounted and used from the proxmox interface. So we must first unencrypt the disk and map it.

cryptsetup open /dev/sdX sdX_decrypt

Then you can change your file system format of the unecrypted disk if you want

mkfs.ext4 /dev/mapper/sdX_decrypt

Next, I setup autodecrypt on startup, this won't be much of a security issue as the root directory of the proxmox and swap is already protected with LUKS encryption too. So there's no way for anyone to get the key file. First write your keyfile and make sure the permissions are strict:

echo 'your-passphrase' > /root/luks-keyfile
chmod 600 /root/luks-keyfile

Next, you setup auto-decrypt by going to the /etc/crypttab

nano /etc/crypttab
sdX_decrypt UUID=[your disk UUID] /root/luks-keyfile luks

When you save a keyfile like this, there's usually a hidden trailing character behind it (e.g: \n). You can use this command to take out any hidden trailing characters:

perl -pi -e 'chomp if eof' /root/luks-keyfile

Test the keyfile and make sure it unlocks then close it again

cryptsetup luksClose sdX_decrypt
cryptsetup luksOpen /dev/sdX test_decrypt --key-file /root/luks-keyfile
cryptsetup luksClose sdX_decrypt

Then, go to /etc/fstab to automatically mount your decrypted volumes on startup

/dev/mapper/sda_decrypt /mnt/volume1 btrfs defaults 0 2

After you are sure that everything's working update your grub and init-ramfs

update-initramfs -u
update-grub

This way it would be decrypted, mounted and ready to go when you reboot!

Note: If it still doesn't work, you can simply add your keyfile key to the passwords that would decrypt the disk

cryptsetup luksAddKey /dev/sdX /root/luks-keyfile

Setting up Snapraid and Mergerfs

Setting up Mergerfs

Install mergerfs if your system does not have it

sudo apt-get install snapraid

First, we have to create a fused storage containing all the disks. Add this in the /etc/fstab:

/mnt/sda:/mnt/sdb:/mnt/sdc /mnt/storage fuse.mergerfs defaults,allow_other,use_ino,category.create=mfs,nonempty 0 0

/mnt/sda:/mnt/sdb:/mnt/sdc:

• These are the source directories or mount points. It indicates that mergerfs will pool together the contents of these three directories (/mnt/sda, /mnt/sdb, /mnt/sdc) into a single merged filesystem. /mnt/storage:

• This is the target directory where the merged view of the three source directories will be mounted. It will appear as if all the files in /mnt/sda, /mnt/sdb, and /mnt/sdc are in /mnt/storage. fuse.mergerfs:

• This specifies that the filesystem type is fuse.mergerfs, which is a user-space filesystem that merges the contents of the source directories.

These are mount options for mergerfs. Here's what each means:

• defaults: This refers to the default mount options provided by the system.

• allow_other: This allows users other than the one who mounted the filesystem to access the merged filesystem.

• use_ino: This option ensures that the filesystem uses inode numbers rom the underlying filesystems, which can be useful for certain backup and NFS operations.

• category.create=mfs: This dictates the policy for creating new files. mfs stands for "Most Free Space," meaning new files will be written to the underlying drive that has the most free space available.

• nonempty: This allows the target directory (/mnt/storage) to be non-empty at the time of mounting. Without this option, the directory must be empty.

• 0 0: These are dump and pass options, used for filesystem checks. The first 0 indicates that the filesystem should not be dumped (backed up) by the dump command, and the second 0 indicates that the filesystem should not be checked during boot by fsck.

These are the primary options that are needed but feel free to check out all the options in mergerfs here

After finishing first, test your configuration using the command:

mount -a

If there's an error the system would output an error. Otherwise, you are fine!

Setting up Snapraid

Install snapraid if your system does not have snapraid

sudo apt-get install mergerfs

Next, you have to create at /etc/snapraid.conf. This is the snapraid configuration file where the contents, disks and parity files are written. If you have disks with different storage sizes, snapraid want the largest one to be a parity file. These are the files used by SnapRAID to store the parity data used in recovering in case of a data lost. Here's a sample snapraid.conf:

# Defines the file to use as parity storage
# It must NOT be in a data disk
# Format: "parity FILE_PATH"
parity /mnt/parity/snapraid.parity
 
# Defines the files to use as content list
# You can use multiple specification to store more copies
# You must have least one copy for each parity file plus one. Some more don't
# hurt
# They can be in the disks used for data, parity or boot,
# but each file must be in a different disk
# Format: "content FILE_PATH"
content /mnt/storage/snapraid.content #Fused mounted of sda, sdb and sdc
content /mnt/sda/.snapraid.content
content /mnt/sdb/.snapraid.content
content /mnt/sdc/.snapraid.content
 
# Defines the data disks to use
# The order is relevant for parity, do not change it
# Format: "disk DISK_NAME DISK_MOUNT_POINT"
disk d0 /mnt/sda
disk d1 /mnt/sdb
disk d2 /mnt/sdc
 
# Excludes hidden files and directories (uncomment to enable).
#nohidden
 
# Defines files and directories to exclude
# Remember that all the paths are relative at the mount points
# Format: "exclude FILE"
# Format: "exclude DIR/"
# Format: "exclude /PATH/FILE"
# Format: "exclude /PATH/DIR/"

After modifying the config save the file. Try to store a sample file in one of the disks and then run the first sync

snapraid sync

Next, setup a scheduled sync for snapraid using crontab

crontab -e

Choose you favority editor and then add this in crontab:

0 1 * * * /usr/bin/snapraid sync

This will sync 1 am everyday by running the executable at /usr/bin/snapraid. Note that you can find where the snapraid executable is stored using:

whereis snapraid

Save and exit crontab. Now you have set up mergerfs and snapraid on your system.