Again on shoveling stuff in other people mouth

Again we got a fun thread about having to do some extensive change on perfectly working systems because somebody has a *plan* and you must abide to it.

If before the plan was to have systemd as the true and only init system (on why systemd seems to me a bad idea by itself I’ll discuss on a later post, possibly after throughly study its latest iteration and comparing it), now the plan is to force people not to have a separate /usr or use an initramfs with an early boot system because… “because doing otherwise is broken and already had been in ages”.

That doesn’t tell you much and if you have lots of systems running perfectly on a separate /usr setup and you went that way because it was documented as a best practice, you might feel enraged.

Now, let’s make clear that there are operating systems that keep everything in /usr and have next to nothing in / (and system that do not have /usr at all and everything is in /), you can argue a lot about what’s the best and why. FreeBSD or Hurd approaches have both interesting perks.

The fact is that *now* you have lots of people with perfectly working system in a configuration somebody decided that is wrong and *unsupportable*.

If you try to dig down a bit more you’ll discover that the “brokeness” is mainly due:

  • Somebody keen in using a library that traditionally is in /usr for some fringe feature
  • Somebody hell bent to use glib everywhere
  • Somebody wanting to have d-bus running in the early boot phase
  • Some udev rules using some data that currently resides in /usr

All considered forcing people to spend lots of time because somebody might want to use a bluetooth keyboard on early boot (thus requiring bluez, thus requiring d-bus basically because you can’t use bluez without it) or other non widespread use case is not exactly nice.

Surely trying to get a cleaner layout so we have a bare mountpoint directory, a early boot system in initramfs and the rest of the system cleanly split isn’t bad by itself and probably it is something I would consider neat.

But you still need to have a good separation between what is early boot and what is not and you need to make sure the boot process doesn’t get too complex or too tightly coupled with systems that can and will break easily.

I’m quite happy that alternatives are already almost available for simple systems not needing the additional features requiring those extensive changes.

Hopefully somebody will have time to try to add rules marking in udev so complex rules won’t be triggered when the system isn’t ready for them and deploys using special layouts could stay supported in a way or another.

In the other news Gentoo had been accepted to participate to the Google Summer of Code and there are two projects proposed by me, one is about documenting and if needed extending openrc to be a complete viable alternative to systemd, the other about using containers and qemu-user to have better tools to do cross developement.

27 thoughts on “Again on shoveling stuff in other people mouth”

  1. We’re in bad shape when a great distro like Gentoo has to cater it’s init to some horrible “upstream” RedHat distro. Don’t we have people who can keep us from going down this insane systemd path?

    I’ll stop using Gentoo on my servers, at the very least, the day that systemd becomes the “required” init system.

    Kudos to Lennart Poettering for producing such incredibly bad stuff as pulseaudio, avahi, and systemd. The only “plummers” he needs to associate with are those giving the Linux community an enema by flushing him and Fedora down the drain.

    And as you mention … _every_ computer I own, have built, or sold runs very well now without any of that junk.

    I’m following your projects.

    1. Upstream isn’t horrible, just they have different views and they are unreasonable in accepting feedbacks. Who is delivering feedbacks should provide them in nice patches that won’t hinder anybody though.

  2. I’ve been watching this whole systemd thing with horror. The Poettering Dystopia would have all kinds of odd kitchen scraps go into the PID==0 process (listening on sockets, starting D-Bus and then listening on it for events, filesystem mounting, hostname setting). If anything goes awry and the systemd process crashes, say goodbye to your running system ’cause it’s going down. For someone who jokes about his PulseAudio being “what breaks your audio”, I can’t say I put much faith in his not breaking my system.

    I do *not* want to use an initrd, but systemd seems to want to force that. I’d sure rather not have to deal with the breakage if and when flies migrate from /bin, /sbin, and /lib{,32,64} to /usr. This move is not an absolute requirement for systemd (it is supposedly agnostic toward that particular move), but the Fedora people push it mighty hard along with systemd.

    I don’t want to follow the path of insanity. There are a lot of people who are fervent supporters of systemd as well as the move to /usr and the requirement to use /run (as in “why can’t you backward people get with the times?”). I’m glad that there are other people who sound the warning about being caught up in the enticing siren song.

    Back when I read your first message about shoving things in people’s mouths (or to use the idiom more common in American English, “shoving things down people’s throats”), I was a bit amused by what you had to say because I didn’t understand the issues. Now that I have found out about systemd and have had actual experience with Baselayout 2 and OpenRC, my attitude has turned to alarm.

    There are a lot of systems which do not run a full GUI desktop intended for point-and-play end users. Should these systems be forced to use cgroups, D-Bus, initrd, or even udev? (There are still people who do useful work with a static /dev and most people live quite happily without cgroups. If I didn’t use LXC on one machine, I’d skip cgroups myself.) We don’t need the fancy one-size-fits-all solution imposed on us.

    Thank you for raising the issue early on. I would hope we could interest the Debian people in OpenRC. It, after all, tries to achieve many of the same goals as systemd, but in a much more modest, reasonable, and sound way.

  3. One of the big issues is RedHat, by default they seem to think you should have X11 up and running on a server install, which means gnome2 and it’s then natural for them to have dependencies to things you normally wouldn’t need in a early boot.

    I see it a really bad option to break something which has been working fine since day one, when there are somewhat easy workarounds for those few who has a bluetooth keyboard, just use a USB one if you need one in early boot stage.
    I don’t want to see all of my things in /usr, I want to keep them as it is and have the feel of a multiuser system and not a single user system, which RedHat is working toward (there is microsoft for those who likes to run single user system).

    1. It isn’t as bad as it might look (having everything in /usr and keeping / as mount point fs), the problem is that they do not explain what is the long run plan and why it should be a good idea.
      As I wrote other posix systems have this kind of layout variation.

  4. I can understand why you would have separate /etc, /home, /var, even /boot but what’s the point of /usr? Binaries is what makes your OS, right? Trying to separate that is a bit futile and, imvho, just as /usr/X11 was a mistake so is /usr and everything should be merged back to / (and don’t even try to lie about disk space, it in facts decreases if you go from two to one partition, despite what some irrational systemd haters claim).
    lu_zero, I too would get hateful if I had to use initrd which I thankfully don’t plan to use and while I’m most certainly not impressed by the fact that systemd has worse networking support than OpenRC, overall sysvinit was long overdue on coup de grace and if we are making a new one we might as well go all out and create something shiny and cool. And while I’m staying with my wired stuff I still consider early Bluetooth support a good thing.
    About udev, dear idiots, if I’m not mistaken, a WRT54 router can handle the full udev, not using it is just elitist grade hosing of yourself. I don’t care much about you but, please, don’t spread this cancer of hating to Linux newbies.
    P.S. Mike, quite a few knowledgeable Gentoo people use KDE and calling them “point-and-play end users” is very insulting. And what has cgroups ever done to you? The only limitation is that you need to allocate hard bandwidth for non-root processes to have RT if you enable certain cgroup suboption UNLESS you use systemd but its not systemd’s fault and in fact Poettering’s PulseAudio is one of the few programs that actually use non-root RT so stop whining like a sperger.

    1. OpenRC supports cgroups just fine btw… (and even before systemd existed if you consider the work on lxc)

  5. I’d always understood that the traditional bonus for having /usr be a separate partition/slice/whatnot was largely that it could be mounted read-only so that users would be unable to modify it as it typically contained vendor-provided binaries that didn’t change very often.

    I don’t think I’ve ever seen this done on a Linux setup though — especially in Gentoo’s case since every time you built a new package you’d have to briefly unmount and remount as read-write which would be incredibly disruptive.

    Thinking about it like this, maybe /usr is past it’s prime? I dunno.

    1. The were and are few perks in getting /usr mount separated, the main one is having a simpler way to use lvm instead of having / in lvm (required initrd and some machinery).

  6. Okay, / as a mount place fs sounds absolutely terrific. Can I suggest another optimization – since it’s just a mount place it’s not going to contain any real data therefore it doesn’t need to have a physical partition – make it virtual! Go beyond POSIX! Have every drive and partition available individually. And to automate it, drives instead of a mount point could have a letter assigned to them. Obviously, we need a way to know which is which, having two removable media should be enough for most people to do their floppy workout and that means from now on separate /usr can be known as C: – this is gonna rock everyone’s socks off!

  7. This blog post is wrong in so many ways.

    Lets start with something simple: On your Gentoo box, please run
    for i in `find /bin /sbin /lib* -type f -executable`; do ldd ${i} | \
    grep usr > /dev/null && echo ${i}; done
    Then tell me again that a separate /usr really works or that it’s just a few udev rules.

    I agree that “doing otherwise is broken” is not a good reason, but there are a lot more, e.g. sharing different OS’es one one host or sharing /usr over the network. If you did that with a separate / partition, how would you update the stuff in /(s)bin and /lib(64) with a package manager? If there are only symlinks, there is no problem.

    Last but not least Fedora’s /usrmove has nothing to do with systemd, glibc or dbus. Lennart was not even involved in Fedora’s feature, but still you are referring to him as (the “someone”) several times. Basically your post comes down to “If Lennart supports it, it must be BS.” This is sad on the one hand and doesn’t even make a good rant on the other.

    1. In my case I have

      So some special pam modules (and that can be fun if you don’t have the early boot separation), libarchive/bsdtar that should live in /usr, udev and udisk and not much. So I can tell you that a separate /usr works for me really nicely and for even more people does.

      Someone is someone. If Lennart fits the description so be it. But isn’t just Lennart behaving more than often in that way. Lennart is quite vocal, agreed.

  8. Christoph:

    # for i in `find /bin /sbin /lib* -type f -executable`; do ldd ${i} | grep usr > /dev/null && echo ${i}; done


    So I can say that a separate /usr really works and it’s just a few udev rules.

  9. In response to the commenter “not this again”, let me point out that I am certainly not trying to sow seeds of hatred; indeed, I refrain from calling people names. Please avoid that. My main criticism of systemd is that it tries to do too much. The program which executes at PID 0 must do its minimal job highly reliably because it has no backstop. If that task segfaults, death is swift and certain. The author of systemd explicitly rejects the Unix wisdom of programs doing their single jobs well. GNU software often bends that rule, and most of the time I’m all right with that. This is one instance, however, where feature creep risks catastrophic failure. My beef with systemd is with the software and its architecture, not its maker.

    I certainly don’t have anything against using udev, D-Bus, or Cgroups; most of my systems use the first two and one uses the third. I can think of an important Linux distribution which lacks all three: Android. Actually, Android has two features that “not this again” talks up in his messages: no /usr and the root directory in RAM.

    In my original post, I wrote “There are a lot of systems which do not run a full GUI desktop intended for point-and-play end users.” I did not write “There are a lot of systems which do not run full GUI desktops, the mark of point-and-play end users.” There’s a world of difference between the two statements. If you read the sentence as my equating GUI’s with simpleminded users, I’m sorry to have given that impression. I think you’d be very hard pressed to find anyone who maintains a Gentoo system who’d fall into the point-and-play category. Also, as far as that goes, I make heavy use of KDE, including in an LXC guest–for which I need cgroups.

    Now, to Luca’s main point: not only would the adoption of systemd and the move of all executables to /usr cause a lot of disruption, I fail to see why they would be worth all the trouble. Red Hat would especially like the move to /usr in order to simplify the mounting of remote filesystems, but they already like using initrd’s. They wouldn’t sweat the actual move: they already do major fixed releases. For rolling-release distributions, though, there would be a world of hurt.

    Red Hat is also a big proponent of systemd. It would shave off seconds from their boot times and give the impression of being lean. It would start services only when there was a demand for them so that the distro would not have to work out dependency graphs for the startup of services. That’s nice, but that kind of thing makes me worry about race conditions. And, oh, there’s that little matter of asking that the writers of daemonized applications like Apache, sshd, MySQL, and D-Bus to rewrite their startup code so that they could let systemd listen on their assigned ports and somehow pass the open sockets to the starting de-daemonized daemons. That trick kind of works–until you have a daemon that would like to adjust some of the socket parameters before the listen() call. It’s all pretty elaborate, and all pretty new. Dependency-based startup in OpenRC works pretty nicely already–and it’s portable. I don’t know about everybody else, but for me reboot time is not an issue at all. I go months between reboots.

    Why on earth, by the way, is it necessary to be able to go into an interactive debugging mode early in the boot process with a Bluetooth keyboard? If you have to do that kind of debugging, what’s the matter with plugging in a wired keyboard or even a serial cable? Starting services too early leads to instability.

  10. This is so tragicomic. Here’s a an incomprehensive list of things that Mike and the other folks didn’t get right:

    a) there is no PID 0. Init is PID 1.

    b) systemd is a set of daemons, only a part of the code that is included in systemd is actually executed as PID 1

    c) Android makes use of cgroups and also includes D-Bus.

    d) systemd puts an emphasis on Unix heritage. i.e. exposes services in the file system via cgroups, Reintroduces proper multi-seat support. It brings Linux back closer to how Solaris/AIX worked since a long time, for example in regards to using a joined /usr and so on. It is implemented as a set of processes responsible for individual taks, but integrates them closely, reusing a lot of code. systemd makes Linux Unixy. You are very confused if you misunderstand that.

    e) it’s the non-event sysv system that is prone to race conditions, not systemd. systemd manages boot-up starting things when the resources they need become available. In sysvinit we wait random times before we fsck stuff, we wait random times before we mount file systems. We use hacks like “udev settle” to guess when hardware that might be available actually showed up. sysvinit is inherently racy, and thoses races are fixed with systemd.

    f) systemd does not enforce usage of initrds. It does not enforce merging /usr. It supports initrd-less systems just fine, and so does support split /usr systems. In fact, systemd made Fedora work without initrd for the first time in decades again.

    g) systemd exposes almost all available setsocktopt() options that might be useful before invoking listen() in the socket configuration file

    h) systemd is not and never has been about boot speed. It mostly achieves that as side effect. It’s about correctness, deduplication of code and maintainability.

    i) the main reason of merging /usr is to allow that the whole vendor-supplied OS bits can be mounted read-only. Merging /usr makes read-only /usr useful and complete. You guys seem to imply it would conflict with that. Au contraire, mon ami!

    You guys really have no clue what you are talking about.

    Good luck with mdev!

    1. a) typos happen

      b) ldd /var/tmp/portage/sys-apps/systemd-43/image/bin/systemd => (0x00007fff0c3ff000) => /usr/lib64/ (0x00007fbc94460000) => /lib64/ (0x00007fbc94243000) => /usr/lib64/ (0x00007fbc94035000) => /lib64/ (0x00007fbc93e2c000) => /lib64/ (0x00007fbc93c1e000) => /lib64/ (0x00007fbc93a18000) => /usr/lib64/ (0x00007fbc93804000) => /lib64/ (0x00007fbc935fb000) => /lib64/ (0x00007fbc93270000)
      /lib64/ (0x00007fbc9469d000) => /lib64/ (0x00007fbc9306c000) => /lib64/ (0x00007fbc92e67000) => /lib64/ (0x00007fbc92c51000)

      I guess you are a bit misguided. Why my system daemon should be non-functional if I’m missing dbus or kmod or zlib ?

      c) so what?

      d) cgroups is a “me too” after effect, check how the support got introduced and the whole debate with linus about it.

      e) You never used openrc. I’m quite sure somebody told me how nice upstart event system was and I shown him a nice race condition.

      f) Fedora is broken, this is Gentoo

      g) So what? I need to write my socket configuration in an ini file so my daemon might run properly ?

      h) the codebase speaks for that.

      i) I already stated that you have reasons to have /usr the bsd way, but you can have the same results with the unified / hurd way.

      Maybe some do know what they are talking about and you aren’t among them at all. Please at least look at the software you are discussing about.

  11. I can’t explain as well as fickmaus but I’ll do my best.
    b) You need IPC, for Linux we usually use D-Bus. Of course there are other IPC options but I don’t know details on why D-Bus was chosen.
    c) that was a reply to the BS that Mike Thompson was propagating, a sad person obsessed with hatred for things he either does not understand or acts as if he didn’t understand so that he could hate more
    e) even with OpenRC init scripts can race. For example, if you start iptables and some other program that uses that functionality, they are very likely to clash because there’s not generic way to say “start this script AFTER has INITIALIZED” note how initialization and startup are two different things you can’t test generically, you can, of course, use ugly tricks to know that but that’s horrible and abysmal. My solution is to explicitly sleep for guestimated seconds to be sure that the initial rules have been loaded which of course make startup slower and can sometimes still fail if iptables use even more time than usually and it doesn’t scale that well either. And let’s not get started on how many times I have had an OpenRC service start with [Ok] only to crash right after that and good luck in debugging it or even noticing until you encounter missing functionality. xdm is terribly prone to this but not even the worst example.
    f) pathetic (I wanted to say more but it’s not possible to say it without causing an uproar)
    g) no idea if it uses ini but that’s is one of the sanest formats out there, in fact, next time bash Poettering or whoever created systemd for not using XML or at least JSON, I mean, ini, pfft, that’s too easy and user friendly. Oh, if you are complaining about config files, does this mean you have never written an sysvinit script? They are essentially that and in shell language even and a job so horrible I think I’d prefer shovelling snow for an hour instead.

    Sidenote: I’m afraid he was either a professional troll or he was rather well aware of the software he was discussing.

    1. fickmaus is rude and making a fool of himself.
      b) I know a couple of details, dbus is the evolution of the experience people had with corba and dcop. It was/is geared towards desktop. Using it for something different might not be the best idea.
      e) you have a keyword for marking scripts that optionally might use something if it is about to run, the rest of the description. Services crashing aren’t a fault of the init system. OpenRC can tell you that they are crashed and supervise can do something about it, maybe this summer openrc might get less crude in managing the situation.
      f) I still prefer broken, but pathetic might fit as well.
      g) You are missing the point completely, what is wrong is not using a simple and quick to parse configuration file for running daemons (that is a good idea till you don’t discover you need to embed more logic into it), what is wrong is having to use a configuration file to do something your daemon wants to do by itself: socket configuration and management.

      ldd is a wonderful tool to show how people are wrong or misguided.

  12. e) if i understand OpenRC correctly, it executes the init.d script and then after a certain time checks if the process is still working or something like that. And it can follow forked processes, too. But not double forked. Nor can it know if the script actually finished, you only know that it didn’t outright fail (and you can’t safely know that either but let’s leave that for a bit later). In particular I have had countless occasions when my firewall, content filter and IPS clobber each other rules even though I have modified the scripts to have them depend on iptables, it just doesn’t work. The only clean solution is making the time OpenRC waits before checking status longer which affects each and every init script. Lastly, just few days ago I had miscompiled an xorg module and it segfaulted upon loading. So you do /etc/init.d/xdm status to read that it’s ‘started’, even though it’s clearly dead. This is okay if you notice it yourself but this also means that you can’t trust OpenRC to actually know what is going on.
    g) generally speaking, configuration files should be configuration, which is why we have init.d and conf.d folders and the less logic is put in the init script, the faster it will execute which is important (the only reason shell is used for configuration is because it’s trivial to “parse” from shell). Btw, most of the problems I have been encountering are caused by parallel startup, sadly I power on and off this box every day and having to wait possibly minutes until every time hogging init script has returned control is not an option.

  13. I freely admit that I don’t know everything and may sometimes get details confused, especially for aspects of the system which never give me much trouble. When I’m wrong, I do try to get back on the right track, and I avoid calling people dismissive names just because they disagree with me.

    The whole discussion about systemd, udev versions above 173, and the need to use initrd’s has taken on the tenor of the debate around health-care reform here in the United States. Those on either side of the question focus their main attention on aspects that the debate that people on the opposite side find to be rather unimportant. There is a lot of name-calling all the way around, and the whole question has now come before the Supreme Court. I certainly am not trying to bring that debate to this forum, so please refrain from taking up a position on it here! My whole point is that we have become that intensely vitriolic about these system-initialization issues, and we don’t have a Supreme Court to hear any appeals. We have to work it out for ourselves.

    Now to the discussion at hand. Yes, I misidentified the position of the init task; it is indeed PID 1, not 0. That said, it is *still* the process with no backstop. I’ve not been suffering from random crashes of the init task and I’m hoping that life continues like that.

    When I first read some of Lennart Poettering wrote about systemd, I was pretty intrigued by the idea of having the daemons quietly block during their initializations as a way to let them all start in parallel but with systemd mediating things so that they would all sort themselves out without needing any explicit declarations of what depends on what else. I even started to make a suggestion for patching the kernel to support the special handling of sockets. (Incidently, contrary to Fickmaus’ point (h), Mr. Poettering, from his very first blog post on systemd, often points to fast boot times as a motivation behind systemd.)

    It was when I looked more deeply into the details that I got concerned. There have to be odd tricks involved to pass sockets around between systemd and daemons. (Yes, the suggestion I started to make would fall in that category.) Because now systemd wants to be able to restart services that crash without data loss, it has to listen on the sockets all the time. So far as I can tell, for certain services it is necessary for systemd to create secondary sockets to pass to daemons and then copy data from one socket to another as the daemons handle requests. It handles mounting filesystems and setting user quotas. It wants to be your crond and do logging. Yes, I can follow the logic for adding each of these capabilities. It is when I look back on it and realize how large and all-encompassing the project becomes, things get very worrisome. Also, as systemd development runs up against things like shutdown problems, odd corner cases in daemons, or being confused by the absence of udev inside of LXC guests, it grows ever more complicated. I don’t know about you, but long experience has taught me that software systems that try to do too much are prone to failure.

    (One note: so far as I have been able to tell from the examples of process listings on Mr. Poettering’s site, there is only a single systemd process. There are lots of entries contained in CGroups in the /systemd-1/ tree, but those are for the spawned daemons, not actual systemd processes. So, One Task to crash them all, One Task to bind them.)

    The Linux kernel is also a very large and comprehensive project, but it has a careful development process, hundreds of developers, a great deal of oversight and testing, and very importantly, a long track record. Systemd hasn’t gone even two years since its first release and more than half of its work is the product of a single developer. This might not be such an issue except for the very large risk-exposure surface the software presents.

    What compounds the worry about systemd is that the /usr-move proposal is coming at the same time. Though neither of these initiatives depend on the other, it sure seems like the people leading the charge for the one are also very enthusiastic about the other.

    I’ve never had to worry much about system initialization, or crons, or logging, or the operation of the mounter, which is great because I have other fish to fry. What I can see, though, are the looming headaches I’d rather avoid. I keep trying to work out issues with things like LXC, KDE, PHP, and Apache; I *don’t* want also to have to figure out why fundamental systems have gone south. Look at KDE: finally, by version 4.7, things that were uprooted after the 3.5 series are settling down and working pretty well, and that’s after four years of sweat by dozens of developers (well, there are more issues, but those aren’t for this thread either). Now here comes a new system making many promises yet it takes over so much of the system and wants everyone to be assimilated. With desktop environments there are choices, as there have been for system services. We like our choices.

    As Luca points out, the push with systemd and the /usr move would take away our choices: move now or be swept behind.

    Here’s an odd suggestion about the /usr move. I don’t claim expertise about it, and I certainly don’t have any idea about how it would work out in practice. It might be completely unworkable, but here it goes: have Portage be able to override the build prefix on a per-package basis. For those special packages you need to have in /bin or /sbin to boot without an initrd and thus you don’t want moved into /usr (or indeed you would like to move out of /usr for the same reason), you could set the prefix for that package. This is not a shove-it-down-your-throat suggestion: the individual Gentoo administrator could choose to use this depending on his or her needs.

  14. Choice is good but it’s very easy to hose yourself. Gentoo has at least partially gotten rid of the ueber “optimisation” ricers, do we really now need prefix ricers so that upstreams again have valid reasons why refuse gentoo specific bug reports with prejudice? It’s not that you can’t find valid bugs by abusing build system with weird stuff but upstream righteously isn’t fond of fixing what isn’t broken for everyone but you (believe me, my Gentoo is compiled with mostly Clang). Personally I’m not using Gentoo on my laptop because mobile systems needs special care and despite years of Gentoo’ing I know I don’t know enough to deliver that as well as, say, Fedora can.
    There was idea to do just that and keep stuff as it is right now and the counter argument was that diverging from upstream is not what Gentoo is about and that it will likely become unmaintainable soonish.

  15. Point well taken! I know that some distributions, Debian for example, sometimes tweak the prefixes of packages (for example /usr/local/bin/php vs /usr/bin/php), but I myself am uneager to do that on boxes I run. My suggestion was only for those folks who might really want it. Yes, I imagine support would be an issue.

    I’m typing this on a Thinkpad T400 laptop on kernel 3.2.1 and KDE 4.7 with Gentoo stable and am delighted in how well it works.

  16. It seems awkward to me that you even *dare* to speak about “shoveling stuff in other people mouth” after I saw what you did to the Wikipedia OpenRC article.

Leave a Reply

Your email address will not be published. Required fields are marked *