Systemd is a powerful init-system for Unix; most of the Distros have moved away from sysv-based (System V, init calls scripts based on /etc/inittab and /etc/rc*.d). Notable forks of “traditional” (back-wards oriented?) distributions are Devuan (Debian without Systemd), funtoo (forked from Gentoo).
Now systemd has many nice features such as fine-grained dependency-tracking and therefore concurrent, parallel startup of independent resources. Dependency may be one service depending on the other (feature-wise or time-wise) or even a provided resource (a device, a mount-point, a file within a mount-point).
With all this flexibility comes power, and complexity.
At HS-Esslingen, I maintain a computer pool with lots of Raspberry PIs: this I want to boot over the network, namely NFS, so that I don’t have to juggle with SD-cards being overwritten, being in a poor state and always out-of-sync. Students should be able to do anything on their respective RPI (like programming and loading a kernel module), yet be able to go back to reboot the well-tested and full-featured Raspbian provided to them centrally.
Read-only NFS works very well (except for corner-cases such as Chromium requiring loads of dynamic libraries that need to be fetched single-file over NFS).
However, with every Raspbian update, there are changes that require adaptations: With Raspbian 10.2 being based on Debian Buster, there are several ones, one particular I want to highlight here:
The Service systemd_timesyncd requires access to the file /var/lib/systemd/timesync/clock –this directory therefore has to be read-writable. To do so, we need to mount an overlay on top of RAM-backed tmpfs.
This is to be only, if we have the common case of student work on the RPIs, but should not be done, when we *want* to access and update files in /var/lib. Another directory made accessible locally using RAM-backed storage is /var/cache, e.g. to allow work with aptitude apt search.
Previously, I had done this check, the creation of mount-point directories and the mounts themselves in one extra systemd service and hooking that into the systemd local-fs.target. This is not very elegant.
A better way is to specify two mount-targets: var-lib.mount and var-cache.mount, which are very similar:
[Unit]
Description=Overlay FS for /var/lib
Documentation=https://www.kernel.org/doc/Documentation/filesystems/overlayfs.txt
Documentation=https://www.freedesktop.org/wiki/Software/systemd/APIFileSystems
Wants=create_tmp_dirs.service
After=create_tmp_dirs.service
ConditionPathIsMountPoint=/var/tmp
[Mount]
What=overlay
Where=/var/lib
Options=lowerdir=/var/lib,upperdir=/var/tmp/lib_upper,workdir=/var/tmp/lib_work
Type=overlay
Then, one may adapt systemd-timesyncd.service by replacing it into /etc/systemd/system, adding just one line:
RequiresMountsFor=/var/lib/systemd/timesync/clock.