Orchestrating encrypted mounts with systemd
I’ve been slowly self-hosting more and more services on my home server. I have a non-boot partition that stores all the persistent data, mostly used by various docker containers. I’d never bothered encrypting this partition, mostly out of laziness, but when I recently had to upgrade to a larger disk, I thought it was a good opportunity to encrypt it with LUKS.
The setup process was straightforward. Once you have an encrypted partition, the usage is simple. I start by unlocking the disk:
sudo cryptsetup luksOpen /dev/nvme0n1p1 encrypted_data
This prompts for a password and, after successful authentication, creates a decrypted mapping at /dev/mapper/encrypted_data. I can then mount this on the filesystem, for instance at /data.
The problem: headless reboots
This worked fine, but I ran into issues when rebooting my server. It’s a headless machine and I don’t have any way to enter the password interactively at boot time. This meant that when the server booted up, the /data mount point was empty, which made a lot of my containers unhappy. They expected to find their data in /data and, in its absence, started creating fresh data.
I wasn’t looking for a fully automated solution. I was happy to SSH into the machine and manually unlock the disk every time I rebooted the server, but I needed a bit of orchestration to ensure that certain services (e.g. docker, NFS server, etc.) didn’t start until the /data mount point was available.
The solution: systemd mount units
After some research, I discovered that systemd can handle this through its integration with /etc/fstab.
I started by adding an entry to /etc/fstab:
/dev/mapper/encrypted_data /data ext4 defaults,noauto 0 0
The noauto option is key here: it prevents a filesystem from being automatically mounted at boot time or when I run mount -a.
But if it doesn’t mount automatically, what’s the point of having an entry in /etc/fstab anyway? It serves two purposes:
1. Shorthand mounting
Once I’ve unlocked the LUKS partition and /dev/mapper/encrypted_data is available, I can simply run:
sudo mount /data
This reads and uses all the options defined in /etc/fstab.
2. systemd mount units
More importantly, systemd automatically generates mount units from /etc/fstab entries at boot time. By having an entry in fstab, I get a systemd mount unit that I can reference in other units. In my case, systemd automatically generated data.mount.
When I first boot the server, the /data unit is inactive and unmounted. Once I manually mount /data, systemd considers that unit “active”. This state change is what other services can wait for.
To configure docker to wait for /data, I created a systemd override:
sudo systemctl edit docker.service
and added:
[Unit]
After=
After=data.mount
BindsTo=data.mount
[Install]
WantedBy=
WantedBy=data.mount
After=ensures the service starts only afterdata.mounthas mounted,BindsTo=makes the service stop ifdata.mountunmounts or fails,WantedBy=makes the service start withdata.mountrather than at the normalmulti-user.target.
I repeated the same process for docker.socket and nfs-server.service.
Alternatives
Another option I discovered was to generate a key, store it somewhere on the root filesystem, add it to the LUKS volume, and then configure initramfs to use it at boot time to decrypt the partition. This allows fully automatic booting but didn’t make sense to me. If the server is physically compromised, encryption adds little protection if someone can access the root filesystem and read the stored key.
The other option is to keep the password prompt but allow remote unlocking with something like dropbear, which integrates an SSH server into the initramfs to allow early boot SSH access for unlocking the encrypted disk or partition. I think I’d go this route if my boot partition were encrypted, but it seemed too complicated for a separate partition.