Distribution Kernels: module rebuilds, better ZFS support and UEFI executables

The primary goal of the Distribution Kernel project is provide a seamless kernel upgrade experience to Gentoo users. Initially, this meant configuring, building and installing the kernel during the @world upgrade. However, you had to manually rebuild the installed kernel modules (and @module-rebuild is still broken), and sometimes additionally rebuild the initramfs after doing that.

To address this, we have introduced a new dist-kernel USE flag. This flag is automatically added to all ebuilds installing kernel modules. When it is enabled, the linux-mod eclass adds a dependency on virtual/dist-kernel package. This virtual, in turn, is bound to the newest version of dist-kernel installed. As a result, whenever you upgrade your dist-kernel all the module packages will also be rebuilt via slot rebuilds. The manual @module-rebuild should no longer be necessary!

ZFS users have pointed out that after rebuilding sys-fs/zfs-kmod package, they need to rebuild the initramfs for Dracut to include the new module. We have combined the dist-kernel rebuild feature with pkg_postinst() to rebuild the initramfs whenever zfs-kmod is being rebuilt (and the dist-kernel is used). As a result, ZFS should no longer require any manual attention — as long as rebuilds succeed, the new kernel and initramfs should be capable of running on ZFS root once the @world upgrade finishes.

Finally, we have been asked to provide support for uefi=yes Dracut option. When this option is enabled, Dracut combines the EFI stub, kernel and generated initramfs into a single UEFI executable that can be booted directly. The dist-kernels now detect this scenario, and install the generated executable in place of the kernel, so everything works as expected. Note that due to implementation limitations, we also install an empty initramfs as otherwise kernel-install.d scripts would insist on creating another initramfs. Also note that until Dracut is fixed to use correct EFI stub path, you have to set the path manually in /etc/dracut.conf:

uefi_stub=/usr/lib/systemd/boot/efi/linuxx64.efi.stub

OpenSSL, LibreSSL, LibreTLS and all the terminological irony

While we’re discussing the fate of LibreSSL, it’s worth noting how confusing the names of these packages became. I’d like to take this opportunity to provide a short note on what’s what.

First of all, SSL and its successor TLS are protocols used to implement network connection security. For historical reasons, many libraries carry ‘SSL’ in their name (OpenSSL, LibreSSL, PolarSSL) but nowadays they all support TLS.

OpenSSL is the ‘original’ crypto/SSL/TLS library. It is maintained independently of a specific operating system. It provides two main libraries: libcrypto and libssl (that also implements TLS).

LibreSSL is a fork of OpenSSL. It is maintained by OpenBSD as part of its base system. However, the upstream also maintains LibreSSL-portable repository that provides build system and portability glue for using it on other systems. LibreSSL provides partially compatible versions of libcrypto and libssl, and a new libtls library. Both libssl and libtls can be used for TLS support in your applications.

LibreTLS is a lightweight fork of libtls from LibreSSL that builds it against OpenSSL. This makes it possible to build programs written for libtls against OpenSSL+LibreTLS instead of LibreSSL.

So, to summarize. OpenSSL is the original, while LibreSSL is the OpenBSD fork. libtls is the LibreSSL original library, while LibreTLS is its fork for OpenSSL. Makes sense, right? And finally, despite the name, they all implement TLS.

How Debuggers Work: Getting and Setting x86 Registers, Part 2: XSAVE

In the previous part of this article, I have described the basic methods of getting and setting the baseline registers of 32-bit and 64-bit x86 CPUs. I have covered General Purpose Registers, baseline Floating-Point Registers and Debug Registers along with their ptrace(2) interface.

In the second part, I would like to discuss the XSAVE family of instructions. I will describe the different variants of this instruction as well as explain the differences between them and their limitations. Afterwards, I will compare the ptrace(2) API used to access its data on Linux, FreeBSD and NetBSD. Other systems such as OpenBSD or DragonFly BSD do not provide requests to retrieve or set extended registers, so the comparison may help them design their own APIs.

Continue reading

How Debuggers Work: Getting and Setting x86 Registers, Part 1

In this article, I would like to shortly describe the methods used to dump and restore the different kinds of registers on 32-bit and 64-bit x86 CPUs. The first part will focus on General Purpose Registers, Debug Registers and Floating-Point Registers up to the XMM registers provided by the SSE extension. I will explain how their values can be obtained via the ptrace(2) interface.

The ptrace(2) API is commonly used in all modern BSD systems and Linux, as all of them derive it from the original form designed and implemented in 4.3BSD. The primary focus in this article is on the FreeBSD and NetBSD systems. Nevertheless, the users of other Operating Systems such as OpenBSD, DragonFly BSD or Linux can still benefit from this article as the basic principles are the same and the code examples are intended to be easily adapted to other platforms.

A single CPU (in modern hardware: CPU core or CPU thread, if hyperthreading is available) can execute only one program thread at a time. In order to be able to run multiple processes and threads quasi-simultaneously, the Operating System must perform context switching — that is periodically suspend the currently running thread, save its state, restore the saved state of another thread and resume it. Saving and restoring the values of the processor’s registers play an important part in context switching. It is important that this process is fully transparent to the process being switched, and in a properly implemented kernel there should be no side effects that are perceptible to the program.

Continue reading

DISTUTILS_USE_SETUPTOOLS, QA spam and… more QA spam?

I suppose that most of the Gentoo developers have seen at least one of the ‘uses a probably incorrect DISTUTILS_USE_SETUPTOOLS value’ bugs by now. Over 350 have been filed so far, and new ones are filed practically daily. The truth is, I’ve never intended for this QA check to result in bugs being filed against packages, and certainly not that many bugs.

This is not an important problem to be fixed immediately. The vast majority of Python packages depend on setuptools at build time (this is why the build-time dependency is the eclass’ default), and being able to unmerge setuptools is not a likely scenario. The underlying idea was that the QA check would make it easier to update DISTUTILS_USE_SETUPTOOLS when bumping packages.

Nobody has asked me for my opinion, and now we have hundreds of bugs that are not very helpful. In fact, the effort involved in going through all the bugmail, updating packages and closing the bugs greatly exceeds the negligible gain. Nevertheless, some people actually did it. I have bad news for them: setuptools upstream has changed entry point mechanism, and most of the values will have to change again. Let me elaborate on that.
Continue reading “DISTUTILS_USE_SETUPTOOLS, QA spam and… more QA spam?”