The pointless art of subslots

The sub-slots feature of EAPI 5 was announced as if it was the ultimate solution to the problem of SONAME changes on library upgrades. However, the longer I see it, the more I believe that it is not really a good solution, and that it misses the actual issue targeting somewhere nearby.

The issue is likely well-known by most of the Gentoo users. Every time a library changes its ABI, it changes the SONAME (the filename programs link to) to avoid breaking existing programs. When the package is upgraded, the new version is installed under the new name, and the old one is removed. As a direct result, all applications linking to the old version become broken and need to be rebuilt.

The classic way of handling this is to run the revdep-rebuild tool. It takes a while to scan the system with it but it supposedly finds all broken executables and initiates a rebuild of them. Of course, the system is in broken state until all relevant packages are rebuilt, and sometimes they just fail to build…

As you can guess, this is far from being perfect. That’s why people tried to find a better solution, and a few solutions were actually implemented. I’d like to describe them in a quasi-chronological order.

Using slots with slot-operator deps

A perfect solution that has been long advocated by Exherbo developers. I’m not aware, though, if they ever used it themselves. I didn’t see an exact explanation of how they expect it to be done, therefore I am mostly explaining here how I think it could be done.

The idea is that every SONAME-version of the library uses a different slot. That is, every time the SONAME changes, you change slot as well. Using different slots for each SONAME means that the incompatible versions of the library can be installed in parallel until all applications are rebuilt. This has a few requirements though.

First of all, only the newest slot may install development files such as headers. This requires that every version bump is accompanied by a revision bump of the older version, dropping the development files. On each upgrade, user builds not only the new version but also rebuilds the older version.

To handle the upgrades without a small moment of breakage (and risk of longer breakage if a build fails), the package manager would need to build both packages before starting the merge process. I doubt that enforcing this is really possible right now.

Secondly, the ebuilds installing development files would need to block the older versions (in other slots) doing the same while keeping the versions lacking development files non-blocked.

To explain this better: let’s assume that we have: foo-1, foo-1-r1, foo-2, foo-2-r1, foo-3, … The -r0 versions have development files and -r1 versions don’t have them (they are just the upgrade compatibility ebuilds). Now, the blocker in foo-3 would need to block all the older -r0 versions and not -r1 ones.

In a real-life situation, there will likely be differing revision numbers as well. And I don’t know any way of handling this other than explicitly listing all blocked versions, one by one.

And in the end, reverse dependencies need to use a special slot-dependency operator which binds the dependency to the slot that was used during the package build. But it’s least of the problems, I believe.

The solution of preserved-libs

An another attempt of solving the issue was developed in portage-2.2. Although it is available in mainstream portage nowadays, it is still disabled by default due to a few bugs and the fact that some people believe it’s a hack.

The idea of preserved-libs is for the package manager to actually trace library linkage within installed programs and automatically preserve old versions of libraries as long as the relevant programs are not rebuilt to use the newer versions. As complex and as simple as that.

Preserving libraries this way doesn’t require any specific action from the package maintainer. Portage detects itself that a library with a new SONAME has been installed during an upgrade and preserves the old one. It also keeps track of all the consumers that link against the old version and remove it after the last one is rebuilt.

Of course it is not perfect. It can’t handle all kinds of incompatibilities, it won’t work outside the traditional executable-library linkage and the SONAME tracking is not perfect. But I believe this is the best solution we can have.

The nothing-new in sub-slots

Lately, a few developers who believed that preserved-libs is not supposed to go mainstream decided to implemented a different solution. After some discussion, the feature was quickly put into EAPI 5 and then started to be tested on the tree.

The problem is that it’s somehow a solution to the wrong problem. As far as I am concerned, the major issue with SONAMEs changing is that the system is broken between package rebuilds. Tangentially to this, sub-slots mostly address having to call tools like revdep-rebuild which is not a solution to the problem.

Basically all sub-slots do is forcing rebuild on a given set of reverse dependencies when the sub-slot of package changes. The rebuilds are pulled into the same dependency graph as the upgrade to be forced immediately after it.

I can agree that sub-slots have their uses. For example, xorg-server modules definitely benefit from them, and so may other cases which weren’t handled by preserved-libs already. For other cases the sub-slots are either not good enough (virtuals), redundant (regular libraries) or even broken (packages installing multiple libraries).

Aside from the xorg module benefit, I don’t see much use of sub-slots. On systems not having preserved-libs enabled, they may eventually remove the need for revdep-rebuild. On systems having preserved-libs, it can only result in needless or needlessly hurried rebuilds.

A short summary

So, we’re having two live solutions right now: one in preserved-libs, and other in sub-slots. The former addresses the issue of system being broken mid-upgrade, the latter removes (partially?) the need for calling an external tool. The former allows you to rebuild the affected packages at any convenient time, the latter forces you to do it right away.

What really worries me is that people are so opposed to preserved-libs, and at the same time accept a partial, mis-designed work called sub-slots that easily. Then advertise it without thoroughly explaining how and when to use it, and what are the problems with it. And, for example, unnecessarily rebuilding webkit-gtk regularly would be an important issue.

A particular result of that was visible when sub-slot support was introduced into app-text/poppler. That package installs a core library with quite an unstable ABI and a set of interface libraries with stable ABIs. External packages usually link with the latter.

When sub-slot support was enabled on poppler, all reverse dependencies were desired to use sub-slot matching. As a result, every poppler upgrade required needlessly rebuilding half of the system. The rev-deps were reverted but this only made people try to extend the sub-slots into a more complex and even less maintainable idea.

Is this really what we all want? Does it benefit us? And why the heck people reinvented library preservation in eclasses?!

16 thoughts on “The pointless art of subslots

  1. zmedico

    Sub-slots and preserve-libs are actually complementary, since sub-slots enable rebuilds to be triggered in advance, so that there’s no need to run @preserved-rebuild. Note that preserve-libs is enabled by default in portage-2.1.12, which should be marked stable within a few weeks.

    1. nfk

      As expected even Gentoo users can’t read just like most people – sub-slots are the thing for drivers that directly depend on something because they *have* to be rebuilt after every upgrade of their direct dependency and it’s known beforehand that they *will* have to be rebuilt or bad things will happen. On the other hand having LibreOffice in forced rebuild mode every other day due to poppler was not fun – it’s huge and so are some other poppler users. But that wouldn’t be that bad if it wasn’t needless – my preserve-libs would detect if LibreOffice got broken, kept it working and would allow me to rebuild it when I have the free CPU time and memory to rebuild LibreOffice. And I’m shocked that sub-slots can’t be disabled in FEATURES. I think we all know which desktop environment the people that came up with this use.

  2. mrueg

    Maybe you haven’t seen it, but preserve-libs will become sooner enabled than (maybe) expected.

    “portage-2.1.12
    ==================================

    * FEATURES=preserve-libs is enabled by default.”

    Anyway I like it, it works like a charm and doesn’t add more complexity to the ebuild space.

  3. marduk

    I don’t think I like either, but admittedly I haven’t used preserved libs in a while. I first started using it when it was first introduced there were issues with some packages. Libs were being kept around even when there was nothing linking to them anymore. It gave me an icky feeling. I never had any major problems with revdep-rebuild (that I couldn’t fix) and I don’t mind the wait, so I just made sure preserved-libs was disabled in FEATURES and went on with my life.

    When subslots came out they seemed promising but, as you’ve said, I’ll occasionally get a list of stuff that gets rebuilt even though it need not be. To me revdep-rebuild (almost) always got it right. The only downside is you might have to wait an hour for chromium to rebuild to use it, or occasional. Or it would not pick up something and I’d have to rebuild it manually.

    I can’t remember the last time revdep-rebuild has resulted in an unusable system (and I’ve been a Gentooer for at least 10 years).

  4. chithanh

    Note that the concept of sub-slots is not tied to immediate rebuilds. Just portage chooses to handle them that way. It would be perfectly possible to have them just add packages to some kind of “need-rebuild” set.

  5. Alexis

    There are also these subslots that are set to a way too much finegrained value causing completely needless rebuilds.

  6. Sergei Trofimovich

    1. Subslots work not only for shared libraries.
    perl, haskell, ocaml, python – all need an ‘-updater’ due to missing feature
    2. It’s nice to see what will break (and will be rebuilt) before merge, not after
    3. primary user is x11-drivers which silently break on every Xorg upgrade
    and you get nonworking input on reboot (usually a month later here)

    1. nfk

      You are not supposed to update xorg-server, not rebuild drivers and then keep working for a month. Not just that, portage does ewarn or einfo that the drivers have to be rebuilt after xorg update. In fact, there’s @x11-module-rebuild set available for those wise enough to use Portage 2.2 (or is it now available for everyone?). Not doing that is admin neglect.

      That being said, I have always wondered why emerge does not rebuild drivers automatically – it does know when an update or downgrade takes place and therefore drives must be rebuilt and simply rebuilding them after every xorg rebuild isn’t going to hurt anyone anyway and they aren’t that big.

  7. Jonas Lihnell

    I’d love to have the preserve library feature as long as there is enough useful tools to tell me what libraries I’m keep, how to get rid of them, what to upgrade etc.

    If subslotting could work with preserve libraries to maintain lists of this information and have portage remind me after every merge that I have old libraries that I could and/or should rebuild my packages against that would to me be the most comfortable situation.

    1. nfk

      Why are you commenting on things you clearly know nothing about? And why are you doing that in the know-it-all style of the average /. wiseguy?

      I have been using preserve-libs for many years and for as long as I can remember there’s been a list of preserved libs with full paths, binaries linking to the preserved lib also for each binary the package that owns it is shown as well. And preserve-libs was one of the first users of sets as far as I know so tools are available as well.

  8. Genone

    The core issue is that the problem isn’t uniform across packages. While working on preserve-libs I had to realize that lkinking is a much more complex issue than I first thought, and there are some (at least theoretical) cases it doesn’t handle properly and never will. However it is a system that will work in reality with minimum inconvenience/overhead for devs and users in I guess 80% of all cases.

    Sub-Slots are another step towards moving ebuilds from simple build scripts to a formalized package definition, like most EAPI extensions. This is good in theory, but in reality it can make things more problematic than the “hack” approach as more abstraction makes the whole system harder to understand as details are lost on the way. And if different problem classes are mapped to a common decision space it may result in decisions that are formally correct, but don’t match practical expectations.

  9. vinky

    I agree with you, preserve-libs together with good usage of subslots will solve most problems that I see. Currently I’m more annoyed than happy about the subslot feature since I have no idea how it works and I get the feeling its not tested enough. When I first came in contact with preserve-libs it made perfect sense directly (though at the time it was a bit buggy), a feeling I didnt get about subslots

  10. Mikachu

    One thing that annoys me with how portage handles subslots is you can’t see which package actually triggers what other rebuild, if there are several of those little r things listed, even with –tree.

    1. Akos

      Yes, that’s my problem exactly. For example, for a couple of days now, any time I try to upgrade something (even if it’s an absolutely trivial package), there is a forced rebuild of xcb-related stuff, starting from xcb-proto and libxcb through mesa to virtualbox…

  11. Pingback: Allan McRae » Blog Archive » Interesting Links – May 2013 - Still no witty tagline!

Comments are closed.