Gentoo, btrfs arrays and systemd: a public service announcement

This week, I upgraded my media center/filer and after a reboot (new kernel), systemd was blocking on my btrfs mount. It’s a 3-partition RAID1 array (until upstream says RAID5 is safe). Systemd was somehow waiting on it, with the infamous red spinner. Adding noauto to fstab did allow the machine to boot properly, but the mount itself silently failed: mount /my/mount/point would return 0 but nothing would show up in /proc/mounts nor in the mount point itself.

It turns out that the latest version of systemd reaches the local-fs target faster than earlier releases (at least that’s my theory) and the kernel has not yet fully figured out what partitions belong in which array. So what I needed was to tell systemd to run btrfs dev scan before attempting to mount local filesystems.

While searching for clues, I came across this stack exchange question which has the correct answer (though I did make a few changes). I’ll reproduce here the correct version for Gentoo, in case anyone runs into this:

$ cat /etc/systemd/system/local-fs-pre.target.wants/btrfs-dev-scan.service
[Unit]
Description=Btrfs scan devices
Before=local-fs-pre.target
DefaultDependencies=false

[Service]
Type=oneshot
ExecStart=/sbin/btrfs device scan

[Install]
WantedBy=local-fs-pre.target

I’m not exactly sure why “local-fs-pre.target” needs to be specified three times (twice inside the file, once in the path), but it does the trick: systemd waits for btrfs’s device scan to return before mounting file systems. Maybe btrfs-progs should ship such a file…

As a side note, while digging for information, I found out that systemd actually reads the fstab and translates it into unit files at boot time. The generated files are located in /run/systemd/generator/.

One final piece of information: if I had taken the time to read journalctl -b carefully, I would have saved hours. If you have any issues with systemd, read the damn journal.

I’ll take the opportunity to thank the kinds folks of #btrfs on FreeNode who promptly helped me.

That’s it for tonight, thanks for reading.

8 thoughts on “Gentoo, btrfs arrays and systemd: a public service announcement”

  1. > I’m not exactly sure why “local-fs-pre.target” needs to be specified three times (twice inside the file, once in the path), but it does the trick: systemd waits for btrfs’s device scan to return before mounting file systems.

    Each time does something different:

    Normally, the service file itself should be placed in /etc/systemd/system (if it’s a custom service, distro-installed services will be in /usr/lib/systemd/system, which the /etc location overrides).

    The [install] section “WantedBy” entry then tells systemd where to put the symlink to the service when the unit is enabled (systemctl enable ). Here, btrfs-dev-scan.service is to be a dependency of local-fs-pre.target (which in turn is a dependency of local-fs.target, see the files in the /usr location), so that’s where we tell systemd to install the symlink, so that the -pre target “wants” the scan.

    The “before” entry in the [unit] section, meanwhile, is similar in idea to openrc’s “before”. It tells systemd the unit it appears in should be fully started /before/ the unit named in the line.

    While you placed the file directly in the appropriate wants dir instead of in the general systemd and that works for now, if you should ever systemctl disable that service, you’ll probably find the service file itself gone, since that’s supposed to be a /symlink/ to the service file, not the service file itself, which should be placed in the general systemd/system dir, and disabling a service removes the symlink as listed in the [install] section wanted entry.

    With the file located in system where it should be, the install entry will tell enable where to place the symlink to /enable/ the unit, and where to remove it from to /disable/ the unit. Then local-fs-pre.target.wants isn’t in the path any more, as that’s where the symlink will go to enable the unit. Meanwhile, the before entry says to make sure it’s fully up /before/ local-fs-pre.target. And all three locations are now explained. =:^)

    This is mostly covered in the systemd.units manpage.

    Meanwhile, I strongly recommend reading the systemd for admins series. Just as it’s possible to run gentoo without reading the handbook (except perhaps for the install section), but trying to do so as a gentoo newbie will almost certainly result in a much harder and more difficult experience than it should be or has to be, it’s certainly possible to run systemd without reading the systemd for admins series, but by /not/ reading that series and yet trying to actually /do/ anything non-default with systemd, you’re simply making things harder on yourself. So really, just go read it. You’ll find systemd management /so/ much simpler if you do. =:^)

    The systemd for admins series, all 21 entries, can be found under that heading, here, or at LWN, where they were published in order more or less as they were written:

    http://www.freedesktop.org/wiki/Software/systemd/

    While you’re there, you might wish to take a look at some of the other documentation and/or watch some of the videos. I read pretty much the whole for-admins series as well as the tips/tricks, faqs, and some of the other user/admin links, before I ever attempted to switch to systemd (tho I turned on the USE flag and had everything rebuilt as I was reading, and it was done well before I was).

    I actually watched most of the videos, trying to really grok what it was all about, just as I read the entire handbook back in the day and was actually helping other gentooers (who evidentally didn’t) before I had it installed myself. That’s my normal way of researching something I’m about to jump into, and while I wouldn’t expect everyone to go to the depth that I did, really, do read at least the first several of those systemd for admins posts, because they really /do/ make understanding how systemd works, and they really /will/ save you a lot of grief in the longer term, exactly as reading the gentoo handbook tends to do and for exactly the same reasons.

    Meanwhile, yes, a general btrfs-dev-scan is recommended before attempting to mount multi-device btrfs. Since I have just that, a pair of devices configured as btrfs raid1 for both data and metadata, as my rootfs, I must run an initramfs (dracut-based, with the btrfs module of course) to do the scan and mount root, and of course that does the scan for the rest of my btrfs as well, that original initramfs-based pre-root-mount scan does the trick for me here, but if you’re either booting to a non-btrfs root or your btrfs root is single-device, you won’t have to do that pre-root-mount, and will thus need to do it later. And a service configured exactly as this one is, is a great way to do just that on a systemd-based system. =:^)

    Duncan

  2. That fix with /etc/systemd/system/local-fs-pre.target.wants/btrfs-dev-scan.service is very much obvious. Gives one more reason to love systemd.

  3. Neat trick. I myself solved this by creating a mount file for my btrfs array, which depended on a .service very similar to the one you posted. I didn’t know about the fs-pre target.
    Why it needs the path three times? Because from what I know about systemd, this unit file is a bit wrong. The [Install] section specifies which target pulls it in, which is redundant with the WantedBy line, wich is redundant with the path. IMO, the correct unit file would be placed in /etc/systemd/system;
    wouldn’t have the “Before=” line;
    and, once ready, would be enabled using systemctl enable btrfs-dev-scan.

    what systemctl enable does is put a symlink into the folder /etc/systemd/system/${WantedBy} to the unit being enabled. I hope this is at least a bit clear.

    Serafean.

  4. Argh, typo :
    “The [Install] section specifies which target pulls it in, which is redundant with the Before line, wich is redundant with the path”

  5. The file should actually be installed as /etc/systemd/system/btrfs-dev-scan.service, then enabled with systemctl enable btrfs-dev-scan.service which will create a symlink as requested in the [Install] section. This reduces the cases where local-fs-pre.target is explictly mentioned to two — once to tell systemd that this needs to finish before the target is run, once to tell systemd that it needs to be run at all for the target to be considered run.

  6. I’m having this exact same problem except on LVM (not BTRFS, the file system is ext4).

    I adapted your systemd unit to change the command from:
    /sbin/btrfs device scan
    to:
    /sbin/lvscan –cache
    as I hoped that would work… but I still get the same (imho puzzling) result. Do you have any ideas on how I can fix this problem for LVM2?

    Also, have you reported this in any issue trackers?

    Thanks!

  7. A quick explanation about why that target appears 3 times. They mean different things.

    First, for anybody who isn’t systemd-saavy – a target is just a virtual service – something that can have dependencies and which can be depended on, but which doesn’t actually spawn any processes. You could do the same in openrc with a shell script that starts and stops but doesn’t launch anything. Runlevels are one use, and milestones are another – if you know you need the local filesystem up for your service you don’t have to guess what services are needed to get the filesystem up.

    The presence of a unit in /etc/systemd/system/local-fs-pre.target.wants means that YOU PERSONALLY want this unit to load anytime the local-fs-pre target is loaded. Normally this should be a symlink to the actual service file (typically one directory lower), but being unix you could stick the unit in there.

    The presence of the WantedBy=local-fs-pre.target line in the unit means that the unit author recommends that if you use this you should load it anytime the local-fs-pre target is loaded. If you enable the unit with the “systemctl enable btrfs-dev-scan” command line then systemctl will create the symlink I talked about in the previous paragraph. This line has no effect whatsoever at startup and you could remove this line with no ill effects. It is just used to allow the sysadmin to easily install the symlink.

    The presence of Before=local-fs-pre.target tells systemd that this unit is a reverse dependency of that target. Obviously you told it to run before that target anyway, but if you told it to run by some later point in the boot cycle systemd would still load it sooner since it is a reverse dependency. This is analogous to sticking something in the default runlevel in openrc but it gets run earlier/later since something depends on it. This is the only line that ensures it gets loaded at the right time as far as I understand it, but I could be wrong on that (I don’t know if sticking a unit in the .wants directory actually forces ordering). Also, before is a weak dependency – systemd will try to load this unit first, but failure to load this will not prevent the other from loading. Again, I’m not 100% certain on this so feedback is welcome.

    Compared to openrc the only additional item here is the WantedBy line. OpenRC would of course have the dependency line in the init script, and it would have the symlink in the runlevel. The only thing OpenRC doesn’t have is a line in the script that recommends what runlevel to run the script in. Of course, OpenRC just uses boot vs default typically so this isn’t as much of a problem compared to systemd which breaks the boot cycle in to many more stages, but having it recommend putting the right scripts in the right places would be a value-add IMHO. Actually, having more stages might or might not also be something OpenRC should consider.

  8. WantedBy won’t affect systemd behavior at all, it only affects systemctl enable and it just tells that the symlink will be created at the path you already use in your article. I can only guess that you’re creating the symlink yourself or that you are putting the file directly in .wants. Also note that in systemd dependecies and ordering are independent, i.e Wants doesn’t affect ordering and Before doesn’t affect dependencies.

Comments are closed.