Some ideas regarding / and /usr

What is the / ?

On unix-like system we have the wonderful abstration that let us keep on a single hierarchy all the file systems, it is called mount: we take the logic tree contained in a filesystem and we graft it over another. The initial file system holding all those grafts is the root file system and it lives on the mount point /. Let’s call it rootfs, it usually contains everything needed to start the system.

What (the hell) is /usr ?

Back in the old times where space was constrained the rootfs (/) started to get too big, so, as quick workaround some stuff deemed not fundamental moved on a separate fs mounted as /usr. The quick workaround proved itself useful so started to remain permanent with few exceptions (Hi Hurd, had you been rewritten from scratch this month already again?).

Split-/usr mount, what is the deal?

Linux had since long a quite useful volume-manager, it let you do a number of increasingly complex transformation over storage remaining nearly-agnostic regarding the file system. Being able to extend your storage by adding disks and merging them in a single logical storage seems useful, so is having software-mediated RAID setups.

Gentoo had since long a tutorial how to split all the rootfs mountpoints in different volumes. The idea makes sense or not depending on your tastes. Many people liked it and used it.

The rootfs contained a statically linked set of binaries useful to mount /usr and that was all of it. We can call this set of tools nowadays early-boot tools and the partition holding them early-bootfs. The two usually are the same.

Once /usr is mounted the boot process can start anything else.

Your problem now is figure out what is needed to boot /usr:

  • A volume can be a logical one created by a volume-manager
  • The file system could be a module and the module could be compressed by something the kernel doesn’t understand by itself
  • The file system can be userspace (thanks to fuse) and it could be implemented by an interpreted language such as python
  • Volume and filesystem can be networked, thus you need to bring your network up and the network could require additional components.
  • If you are into crypto, again it could be something mediated by the userspace at volume and file system level.

The possibilities are many and if you want to support them all no matter how unlikely it looks a complex problem.

Everything is broken let’s break it some more!

Some loud mouthed people decided to go against one of the key design item in writing a component such as an init system, that is keeping the single point of failure as simple as possible to reduce the chance it fails.

They kept adding compulsory dependencies to it that use to live on /usr. I think that as a way to get a problem so his arguably half baked solutions can be sell as only future and to make the most annoying situation in the previous paragraph the default.

The rational reaction would had been to just tell them to keep playing with their broken toys in a corner.

Initramfs

So we have this problem, a arbitrary long list of components that would make the rootfs large and some genius actively tries to have the first program started require most of it and shovel the concept down everybody else.

One solution would be just merge rootfs, early-init-bootfs and the whole /usr together somehow, welcome back to the the early 1900! (incidentally you will need also /var mounted but that’s a digression)

Obviously the problems causing the initial split-/usr are still there.

Linux has a neat feature called initramfs, the successor of initrd, it is great to keep modules and all the stuff you might need to mount your rootfs in a place the kernel could always reach no matter what.

So a solution would be keeping all that’s needed to mount the rootfs-now-merged-with-/usr in the initramfs since by definition is always reachable.

It is not exactly the most elegant solution but arguably works as long you get the list of required component right.

The elephant in the /boot

Some rhetoric questions:

  • “The initramfs is somehow related to its kernel, what happens if you keep more than 1 kernel around?”
  • “Which is the sane size limit for it?”
  • Initramfs can get stale easily, how much time takes to create it and keep it up to date?”

The answers might vary. The short is that you need good tools and lots of space.

Alternatives

You need good tools and good knowledge about what you need for your early boot, you have to put it somewhere and keep it up to date easily. Possibly it shouldn’t depend on your kernel yet be easy to access it.

/boot as early-boot partition

That is one of the simpler ideas, we just keep a separate copy of what is needed /boot, historically most concerned people kept a recovery there so makes sense for them use it as early-boot.

Static and restrict rootfs

If you know what you are doing as long you can keep in your rootfs your tools by linking them statically (so the whole deal about compressed modules is taken care of) and you aren’t using strange stuff (so just lvm and normal fs), you do not care about this whole deal. AS LONG YOUR DISTRIBUTION DOESN’T PLAY GAMES. Nor you drink the kool-aid and use stuff that breaks by design static linking or makes as hard as possible keeping a minimal amount of stuff in the rootfs.

Summary

We always need your help and feedback to make so Gentoo keeps giving you good options and currently working systems keep working in the next future. Thanks for reading.

5 thoughts on “Some ideas regarding / and /usr”

  1. Your insight is much valued, and it is good for one’s state of mind and mental health to finally find people who have the same views (ergo, one is not led into thinking one is crazy).

    As an addendum (I don’t know if you’ll be posting on this later on), the initramfs is simply a cpio archive (which can be compressed in gz, bz, whateverz the kernel supports) which has a basic filesystem structure that the kernel understands (much like /). In it you only need what’s need to mount / and call your /’s init. Actually you just need the binaries (statically-linked, of course) and init itself. All the magic is done in init, and it can be as small as 48 lines (including whitespace and comments), complete with a rescue shell (which drops you to busybox) in case something goes wrong.

    As a best practice (I think I read about this either on the new wiki, the old one, or the clone), I have /usr/src/initramfs where I put all that basic FS structure, plus THE init. Using the following oneliner I can recreate my initramfs, gzip it, and put it on /boot:

    find . -print0 | cpio –null -ov –format=newc | gzip -9 > /boot/gentoo-initramfs.gz

    I started doing this on the machines I controlled that had LVM+LUKS, for which an initram is needed if your root filesystem resides within it.

    I believe this process can even be automated almost in its entirety, by configuring proper USE (static) for the needed packages, and I would love to see Gentoo come up with a dracut alternative, so we can have initram’s the Gentoo way, and not the bloated way 🙂

    1. And it is all nice and dandy as long you can statically link and you track all you need properly =) See the new blogpost about it.

  2. You mean your previous one? I already read it. Of course, you have to keep track of the static binaries (which in my case, are busybox, cryptsetup and lvm.static, in their respective bin/sbin directories). It’s not much, and so far they have survived system updates without needing to rebuild the initram. I know it’s not the best, but right now it’s the minimum, cleanest and easiest thing to do with a root filesystem on an encrypted partition/PV (which is becoming more and more common on certain employers nowadays).

    I note that the ONLY thing my initramfs does, is mount my root filesystem. Nothing else. And it should NOT do anything else.

  3. Well, it ought to do a *hair* more than start LVM and mount the rootfs: it needs to pivot to the real root and then start /sbin/init. I still don’t have a nice writeup yet for the wiki, but I do have an outline of what I did to get an early-boot setup in /boot and without an initramfs:
    http://forums.gentoo.org/viewtopic-p-7430062-highlight-.html#7430062

    The script, as you can see, is a bit longer (55 versus 48 lines), but I like error recovery. I’m a programmer, after all. Note that I avoid grub2, which has become a bit, well, grubby in its generality.

  4. Even though I originally designed it to solve a different problem, I suggest you take a look at my initramfs generator, Tinytium (https://github.com/raffaellod/tinytium).
    It uses Portage to keep the contents of the initramfs up to date, and has automatic dependency pull-in from the host system (e.g. it picks the ld-linux.so from your /lib).

    At the moment it does not support lvm, but it shouldn’t be difficult at all to make lvm part of the packages it emerges.
    Eventually, I want to make it a valid replacement for genkernel.

    Please let me know if you take a look at it – I’d love to hear comments.

Leave a Reply to Mike Thompson Cancel reply

Your email address will not be published.