A distribution kernel for Gentoo

The traditional Gentoo way of getting a kernel is to install the sources, and then configure and build one yourself. For those who didn’t want to go through the tedious process of configuring it manually, an alternative route of using genkernel was provided. However, neither of those variants was able to really provide the equivalent of kernels provided by binary distributions.

I have manually configured the kernels for my private systems long time ago. Today, I wouldn’t really have bothered. In fact, I realized that for some time I’m really hesitant to even upgrade them because of the effort needed to update configuration. The worst part is, whenever a new kernel does not boot, I have to ask myself: is it a real bug, or is it my fault for configuring it wrong?

I’m not alone in this. Recently Михаил Коляда (zlogene) has talked to me about providing binary kernels for Gentoo. While I have not strictly implemented what he had in mind, he inspired me to start working on a distribution kernel. The goal was to create a kernel package that users can install to get a working kernel with minimal effort, and that would be upgraded automatically as part of regular @world upgrades.

Pros and cons of your own kernel

If I am to justify switching from the old tradition of custom kernels to a universal kernel package, I should start by discussing the reasons why you may want to configure a custom kernel in the first place.

In my opinion, the most important feature of a custom kernel is that you can fine-tune it to your hardware. You just have to build the drivers you need (or may need), and the features you care about. The modules for my last custom kernel have occupied 44 MiB. The modules for the distribution kernel occupy 294 MiB. Such a difference in size also comes with a proportional increase of build time. This can be an important argument for people with low-end hardware. On the other hand, the distribution kernel permits building reusable binary packages that can save more computing power.

The traditional Gentoo argument is performance. However, these days I would be very careful arguing about that. I suppose you are able to reap benefits if you know how to configure your kernel towards a specific workload. But then — a misconfiguration can have the exact opposite effect. We must not forget that binary distributions are important players in the field — and the kernel must also be able to achieve good performance when not using a dedicated configuration.

At some point I have worked on achieving a very fast startup. For this reason I’ve switched to using LILO as the bootloader, and a kernel suitable for booting my system without an initramfs. A universal kernel naturally needs an initramfs, and is slower to boot.

The main counterargument is the effort. As mentioned earlier, I’ve personally grown tired of having to manually deal with my kernel. Do the potential gains mentioned outweigh the loss of human time on configuring and maintaining a custom kernel?

Creating a truly universal kernel

A distribution kernel makes sense only if it works on a wide range of systems. Furthermore, I didn’t forget the original idea of binary kernel packages. I didn’t want to write an ebuild that can install a working kernel anywhere. I wanted to create an ebuild that can be used to build a binary package that’s going to work on a wide range of setups — including not only different hardware but also bootloaders and /boot layout. A package that would work fine both for my ‘traditional’ LILO setup and UEFI systemd-boot setup.

The first part of a distribution kernel is the right configuration. I wanted to use a well-tested configuration known to build kernels used on many systems, while at the same time minimizing the maintenance effort on our end. Reusing the configuration from a binary distro was the obvious solution. I went for using the config from Arch Linux’s kernel package with minimal changes (e.g. changing the default hostname to Gentoo).

The second part is an initramfs. Since we need to support a wide variety of setups, we can’t get away without it. To follow the configuration used, Dracut was the natural choice.

The third and hardest part is installing it all. Since I’ve already set a goal of reusing the same binary package on different filesystem layouts, the actual installation needed to be moved to postinst phase. Our distribution kernel package installs the kernel into an interim location which is entirely setup-independent, rendering the binary packages setup-agnostic as well. The initramfs is created and installed into the final location along with the kernel in pkg_postinst.

Support for different install layouts is provided by reusing the installkernel tool, originally installed by debianutils. As part of the effort, it was extended with initramfs support and moved into a separate sys-kernel/installkernel-gentoo package. Furthermore, an alternative sys-kernel/installkernel-systemd-boot package was created to provide an out-of-the-box support for systemd-boot layout. If neither of those two work for you, you can easily create your own /usr/local/bin/installkernel that follows your own layout.

Summary

The experimental versions of the distribution kernel are packaged as sys-kernel/vanilla-kernel (in distinction from sys-kernel/vanilla-sources that install the sources). Besides providing the default zero-effort setup, the package supports using your own configuration via savedconfig (but no easy way to update it at the moment). It also provides a forced flag that can be used by expert users to disable the initramfs.

The primary goal at the moment is to test the package and find bugs that could prevent our users from using it. In the future, we’re planning to extend it to other architectures, kernel variants (Gentoo patch set in particular) and LTS versions. We’re also considering providing prebuilt binary packages — however, this will probably be a part of a bigger effort into providing an official Gentoo binhost.

29 thoughts on “A distribution kernel for Gentoo”

    1. Fun fact Arch applies patches to the official kernel. So no vanilla there.

  1. This is an interesting development and will certainly help new users trying Gentoo. FYI I’ve been adapting Fedora kernel configurations for Gentoo for my own use:

    https://setphaserstostun.org/posts/gentoo-54x-generic-kernel-configuration/
    https://setphaserstostun.org/posts/gentoo-419x-generic-kernel-configuration/
    https://setphaserstostun.org/posts/gentoo-414x-generic-kernel-configuration/

    They’re not truly generic configurations as they’re setup for use with OpenRC and don’t use an initramfs so they’re going to work only on simpler setups.

    1. My solution is actually quite close to that. The missing part is a tool to help you configure it. The package utilises the de facto standard savedconfig.eclass, so it accepts external configuration.

      1. Nice!
        For configuration, IMO, genkernel is enough.
        For basic .config, I prefer the ubuntu one, because is more tested than Arch one.

  2. Thank you. I often wished that. It has the potential to lower the border when installing Gentoo significantly. In the past, I had the iteration cycle: “Boot into chroot. Compile kernel. Does the kernel run?” for multiple installations. Mainly, because the one home grown kernel is entirely machine dependent and does not run on another machine. Also sometimes, I haved changed the underlying hardware (mainboard/CPU/…) and have to do this cycle again.

    A binary generic kernel can be a base on which you can compile your kernel later on.

    With that, I also thought about not having a seperate Gentoo distribution kernel, but provide the ability to install other distribution kernels (for example Debian/Arch/Fedora/Ubuntu). Is this feasible? Maybe this even would lower maintainance burden.

    1. Yes, that’s entirely possible. Once this matures a bit more, I’m planning to split common logic into an eclass, and that should help adding new kernel variants.

  3. It certainly would be nice to have a fall back, I’ve had to boot to a rescue USB stick before to fix things

    My kernels are efi stubs with the kernel options and all drivers & firmware baked in – about 6MB each, once I changed the kernel boot string and stupidly broke all of them all at once

  4. I like this approach. I’ve been running Gentoo since 2003 or 2004, and while I generally love the DIY aspect of Gentoo, I’ve never really come to terms with kernel configuration. And as I grow older (I’m now 67), it’s getting more and more in the way of other things. For a long time I’ve been running genkernel with a script, basically “zcat /proc/config.gz > /usr/src/linux/.config; cd /usr/src/linux; make olddefconfig; cd; genkernel –oldconfig all”. I may be missing new important kernel settings this way, and it would surely be helpful to have others deal with those things.

  5. I like this idea. It’s always nice to have the option of running a prebuilt kernel that you know will run. Them guys over at funtoo have been doing this for awhile with the debian kernel binaries.

    Interesting that you mention using configs from arch too. I’ve been doing this for years…with genkernel even. Never failed me. Admitingly there is alot of stuff in those configs though…some of it that I change but not much.

  6. Strange thing. I never experienced problem with this. I just copy config from previous kernel and check if there are any new options.

    Only recently I had a problem on new PC and had not time to check what was wrong. But I simply generated initramfs and it works – I’ll check later what the problem is.

    I think that it would be better to have some tool that helps with generating config for compilation. Like something that makes almost everything a module and generate initramfs. But actually we can have more than one solution, right?

    1. @Marcinnnn there is allmodconfig which makes everything a module whenever possible. There is also localmodconfig which enableds only modules you currently have loaded I believe.

  7. When will this be combined with make localyesconfig?

    Don’t want to miss out on eliminating those 250MiB of useless modules you’re estimating.

  8. I begun to made my own kernel on Suse, long before using gentoo, and that in order to get better performances with multimedia applications. I switched to gentoo a few years later and done some testing. On the same machine, to recompile the suse kernel, after changing only its processor and latency options, gave me similar performances than with gentoo., the difference was so marginal than it could not be notified without benchmarks.

    Which imply the processor and latency options of a kernel are essential in order to get wanted system performances, and you cannot change them at run time.

  9. Configuring and streamlining my own kernel was fun in 2000. In 2019 it is a chore. I have seen a few pages left and right showing you how to use ubuntu kernel or other distro. And funtoo had those kernels to be compatible with redhat and suse.

    Ultimately I blame upstream for not providing some simple top level configuration tool:
    * VM host support
    * docker support
    * systemd
    * chrome
    and what not, and it does the rest for you.

    I don’t care so much about the building as to the configuring and possibly the proper installing. I used to do kernel without initramfs, nowadays genkernel does that for me.

    I’ll certainly follow your work closely.

  10. Why not as an OPTION or failsafe plan?
    Also why not a SCRIPT that would build or help to build a good enough kernel for the detected system as the default – 99% with no error – one as plan B?

    1. It’s an option. Existing kernel build methods will continue being supported.

      You are free to put your time and effort into creating such a script. I have neither time nor experience to create it.

  11. Hi Michał, thanks a lot for your work on this!

    I tried to run/install both vanilla-kernel and vanilla-kernel-bin. Sadly, my system refuses to boot since it can’t mount the root partition formatted with ext4.

    I noticed that the config you pass for the compilation process loads ext4 as a module instead of compiling it right into the kernel:

    CONFIG_EXT4_FS=m

    Could this be the issue? I tried both UEFI and CSM boot via GRUB.

    1. Are you running stable? There’s a similar report on Bugzilla, and the conclusion seems to be that it’s a buggy dracut version. Try 049-r3.

  12. Hi Michał,
    thanks for sharing this post. In order for this to reach a broader audience of new users – who might benefit more from a gentoo binary kernel, a section in the handbook should mention this .
    Don’t you think so?

  13. Hi. I think this is certainly a useful idea, as indeed, kernel configuration playtime is scarce nowadays.

    What are your thoughts on somehow integrating kernel configuration somehow into Gentoo profiles? I’m `amd64`, so clearly it’s the simple scenario, but wouldn’t it be interesting if final `.config` could be somehow be composed through standard base profiles and user overrides?

    I’d love to have some kind of an upstream-maintained, minimally solid, security-concious base, and add my device configurations and other extra wishes in a custom profile.

    Maybe still too complicated, but throwing it out here regardless.

  14. Hi Michał,

    Have you considered starting with providing usable defconfigs or alternatively config fragments that can be used together with the default arch defconfigs (using the merge_config.sh kernel script)?

    Especially config fragments seem nice in that they could do something similar to CONFIG_CHECK in ebuilds but instead of checking they could be easily included in the kernel config to ensure the required config options are enabled.

    Both defconfigs and combinations of config fragments could be used to build binary packages from as well of course.

    I’d love a gentoo provided binhost btw, would be great for those annoying repackaged webkit libraries. I’d be glad to help with those efforts if you want some help :)

  15. I would love to see support for zfs from those packages. I jumped ship from embedded some time ago and I agree with everybody – kernel config is a plague nowdays. Unfortunately using zfs as my rootfs makes me very unlikely of updating kernels.

  16. I’m gmt and I endorse this product/service.

    But, honestly I want more! Here’s what my bike-shed blueprint library contains about this topic.

    Portage/EAPI has sufficient semantic richness to do kernel configuration intrinsically. Using gmake as an interactive, skinnable, configuration GUI is–obviously–a freaking /great/ idea, don’t get me wrong. But cascading profiles and use-flags might work as well or better.

    Basically, I suspect, with enough cleverness, slurping all or most of the kconfig constraints and turning them into a {USE_,{B,P,R,}}DEPEND hierarchy could be mostly or completely automated. The clever bits might be a fun project and relatively stable, since kconfig is quite ossified.

    Obviously, there’s lots of “less-clever” bits like initramfs/bootloader/UEFI packaging. Not hard, exactly, just sprawling and painful since you’re breaking boot every time you make a mistake. Thankfully these mostly already exist but need “porting” and maintaining.

    That just leaves one big problem: the fifty screenfuls of use-flags. I think the only solution to that is to break it up into a hierarchy of “fake” kernel-feature packages that act as proxies for the upstream useflags. Or something like that.
    This part might require some minor EAPI innovation. Or a tool to automagically make it all work right… Not sure, exactly, [insert solution here].

    Next time a I have a spare man-year I plan to give it a try :)

  17. Just stumbled upon this while upgrading my Gentoo setup on an aging Dell Latitude E6440.

    I originally wanted to unmask the latest Gentoo-sources with the ~AMD64 keyword and by accident unmasked Gentoo-Kernel ~AMD64 instead 😇

    Anyway, lucky accident, I guess. I really think that the gentoo-kernel method is superior to building your own kernel or using GenKernel just because it seamlessly integrates with regular World updates.

    The included custom kernel config works just fine on my machine, everything works as expected.

Leave a Reply to Dominique Michel Cancel reply

Your email address will not be published.