Early boot fun

Just few notes spurted from a discussion with a friend regarding why he feels we suck badly.

Early boot

Let’s make a quite rough description on how booting could work:

  • Imagine you are the kernel, you just found your rootfs, managed to run your init and you are happy. That’s probably the earliest we care.
  • Init got called and starts running some scripts, maybe checking the rootfs consistency before remounting r/w, maybe checking the other essential mount points before mounting them or maybe start the device manager first and then checking what is going to mount, assuming what is essential still requires some modules loaded and that the device manager will do.
  • Move further and set up the network
  • Maybe now mount the mount points that require the network (nfs?)
  • Now get the other daemons up and running, maybe in parallel, maybe bring up some graphical login.

Now let’s see who are the actors: rootfs, init, device-manager and maybe incidentally volume-manager and networking.

Ideally your rootfs should contain

  • anything required by init to run, easy, init should be as small as possible to make sure this single point of failure really hard to fail.
  • anything required by the device-manager to load modules, should be a no brainer, well, maybe if you want your modules compressed with some new or exotic compressor because it is “faster” that way, you have to fit it in the rootfs.
  • If your essential mountpoints require a volume-manager the same applies, lvm can require something weird depending on the setup so either you link it statically or you have to put it again where it is reachable, same could be said for any kind of advanced crypto at volume leve.
  • We discussed about mount and again we could have fuse-fs using a scripting language or other stuff that make issuing mount a little more complex that we would expect (and again fs-level crypto happening in the userspace)
  • The network would just need some modules loaded right? Wrong, it might need some special daemons doing any kind of bizantine authentication, and if you are really looking for pain you could be willing to netmount those mentioned file systems or even do volume management over bizantine network (ok there is a limit in this kind of perversion and we are just halfway).
  • Once everything is mounted the rest of the system can be brought up w/out much qualms.

So in the end your rootfs can be quite fat contain full copies of python so you can mount that funny file system, have lots of lovely brittle deps because you thought NetworkManager is the only way to get the network up and meanwhile that having some important stuff (e.g. /var) netmounted is all the rage.

Fun (aka pain)

So in short your rootfs could be as big as a compact live distribution and have as many moving parts as one (or more), well it could be just your distribution if you do not keep everything in a separated mount point.

Some years ago that was one of the suggested ways, you keep essential stuff in / and then every other root mountpoint would have its partition, maybe using some advanced stuff just because.

Then you get told that the right place in which you have to fit all discussed above has to be something called initramfs and obviously tell the kernel about it.

Probably nobody would be that crazy to end up with the far corner case, so the initramfs would have to copy just few (20+??) libraries and some (30+?!) binaries in the normal case and you have to keep it synced up properly (joy).

Most people could live happy with just a statically linked lvm and udev living in a small partition easy to mount and that would be the start and the end for them, but certain wise guys will tell you that static linking is harmful, the whole concept is broken anyway since our bluetooth subsystem requires lots of userspace and then you’d be w/out a keyboard in case something goes wrong (so you should shove bluez and happy deps in your initramfs/rootfs?).

Summing up

There are easy, simple and working solution for just some realistic scenarios, but not covering everything that’s possible.

There are more complex, brittle and error prone ones that might cover everything and more (and maybe still fail in some basic situations).

The fact lots of lemmings flocks over the complex/brittle because the guy with the largest mouth is the best speaker is sad, overly sad.

That said if you were happily using since 10 years ago a lvm setup as described by our guides of the time and now you are afraid that your next userspace update your system will break horribly if you don’t go through the hoops of making an initramfs, that won’t work for you out of box and will force you to modify your bootloader or do some other time consuming work:

I’m sorry.

(Luckily somebody prepared a portage hook to prevent some breakages https://gist.github.com/mansr/7289969 not all of them but the most glaring are covered)

Incidentally, you can still pester us, help us getting better programs (e.g. contribute to eudev, kmod, lvm and everything else you use) and take an active part in the community and hopefully protect your simple and working solutions.

8 thoughts on “Early boot fun

  1. Mike Thompson

    Luca, hurray for you!

    I’ve never been happy at the idea of an initramfs. Not only do you have to remember to update it when parts of the system change, you have to go through a time-consuming reconstruction of that blob any time you want to tweak something. Add to all that the fact that at every boot there is an overhead for unpacking the initramfs and then doing a bunch of modprobing.

    As I described in the forums, I’m in the small-is-beautiful boot-partition camp that you mention. That partition holds extlinux (which is much cleaner than grub2) and a busybox-based early userspace. I avoid module insertion by compiling drivers in the kernel. I also answer that any kind of service relying on a lot of handholding (such as bluetooth keyboard support) is flat-out unsuited for use during early userspace. (If I had to use a serial port via GPIO pins to do that debugging, I’d just accept that and carry on.) I’m not sure what is supposed to be broken about using statically linked executables, but if I had to, I’d put ulibc onto my boot partition.

    Judging from dmsg, I spend less than a tenth of a second in my early userspace. Most of that is taken by the vgchange that gets my LVM started.

    I sure agree with your complaints about the well-funded persuasive loudmouth and his over-complicated schemes. The way around it is not only to find solutions and workarounds such as you mention but also for those of us who turn away from the enchantment to keep the faith and encourage one another.

    Reply
  2. lu_zero Post author

    Mike, if you could also document it in the wiki I’d link on my blog.

    The best way to prevent the problems we are seeing is to provide alternatives.

    The whole rhetoric about systemd is that it is inevitable because…

    because they break everything they could (e.g. cgroups) and tried to exaust people writing alternatives while not being paid as most systemd contributors are.

    Probably I’ll set up a bountysource for this purpose later…

    Reply
  3. Rich

    Seems like the simple solution is to just stick all that stuff in /usr and have the initramfs mount it, and that is how most distros handle it. There is no reason to stick all that stuff in an initramfs.

    Sure, you could stick everything and the kitchen sink in root, but how is that any better than sticking it all in /usr? Either way you have a big partition that is critical for boot.

    Reply
    1. lu_zero Post author

      Rich0 said:
      Seems like the simple solution is to just stick all that stuff in /usr and have the initramfs mount it, and that is how most distros handle it. There is no reason to stick all that stuff in an initramfs.

      Sure, you could stick everything and the kitchen sink in root, but how is that any better than sticking it all in /usr? Either way you have a big partition that is critical for boot.

      You seem to miss the point that you have to fit the kitchen sink and everything in the initramfs, then make sure the elephant fits the boot partition and stays up to date properly.

      You ignore the problem completely (you need to know what is needed and make it as self contained as possible) and state that the solution is fitting something arbitrarily large into the initramfs.

      It is not just you doing this, apparently most people trusted the loudmouthed guy words w/out even thinking what’s involved.

      Reply
  4. Rich

    I tend to agree that figuring out the dependencies is potentially a challenge. However, that challenge exists even when trying to decide what needs to be installed in root. If you just stuck the entire contents of a root filesystem in an initramfs in theory you are no worse off other than in disk requirements for the boot partition, and perhaps load time since initramfs isn’t optimized for random access from disk.

    Reply
  5. lu_zero Post author

    Being a ramdisk should be even faster, just it is Nx the number of kernels you keep around.

    As said, people is missing the point that we do not have tools good enough: no matter where you place what is going your early-boot you need to know what it is.

    Secondarily no matter how much space you have, you want to have as few moving parts as possible, so preventing static linking is one of the most stupid ideas ever. (and using non-C and non-bash is again an headache )

    Reply
  6. Rich

    Just a comment RE bash – you can get a fair speed boost (if you care) by using dash. Just be aware that it doesn’t handle bash-isms – which has caused a dracut bug or two that I’ve stumbled on.

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>