Testing the safe time64 transition path

Recently I’ve been elaborating on the perils of transition to 64-bit time_t, following the debate within Gentoo. Within these deliberations, I have also envisioned potential solutions to ensure that production systems could be migrated safely.

My initial ideas involved treating time64 as a completely new ABI, with a new libdir and forced incompatibility between binaries. This ambitious plan faced two disadvantages. Firstly, it required major modification to various toolchains, and secondly, it raised compatibility concerns between Gentoo (and other distributions that followed this plan) and distributions that switched before or were going to switch without making similar changes. Effectively, it would not only require a lot of effort from us, but also a lot of convincing other people, many of whom probably don’t want to spend any more time on doing extra work for 32-bit architectures. This made me consider alternative ideas.

One of them was to limit the changes to the transition period — use a libt32 temporary library directory to prevent existing programs from breaking while rebuilds were performed, and then simply remove them, and be left with plain lib like other distributions that switched already. In this post, I’d like to elaborate how I went about testing the feasibility of this solution. Please note that this is not a migration guide — it includes steps that are meant to detect problems with the approach, and are not suitable for production systems.
Continue reading “Testing the safe time64 transition path”

The perils of transition to 64-bit time_t

(please note that there’s a correction at the bottom)

In the Overview of cross-architecture portability problems, I have dedicated a section to the problems resulting from use of 32-bit time_t type. This design decision, still affecting Gentoo systems using glibc, means that 32-bit applications will suddenly start failing in horrible ways in 2038: they will be getting -1 error instead of the current time, they won’t be able to stat() files. In one word: complete mayhem will emerge.

There is a general agreement that the way forward is to change time_t to a 64-bit type. Musl has already switched to that, glibc supports it as an option. A number of other distributions such as Debian have taken the leap and switched. Unfortunately, source-based distributions such as Gentoo don’t have it that easy. So we are still debating the issue and experimenting, trying to figure out a maximally safe upgrade path for our users.

Unfortunately, that’s nowhere near trivial. Above all, we are talking about a breaking ABI change. It’s all-or-nothing. If a library uses time_t in its API, everything linking to it needs to use the same type width. In this post, I’d like to explore the issue in detail — why is it so bad, and what we can do to make it safer.
Continue reading “The perils of transition to 64-bit time_t”

Overview of cross-architecture portability problems

Ideally, you’d want your program to work everywhere. Unfortunately, that’s not that simple, even if you’re using high-level “portable” languages such as Python. In this blog post, I’d like to focus on some aspects of cross-architecture problems I’ve seen or heard about during my time in Gentoo. Please note that I don’t mean this to be a comprehensive list of problems — instead, I’m aiming for an interesting read.
Continue reading “Overview of cross-architecture portability problems”

Gentoo: profiles and keywords rather than releases

Different distributions have different approaches to releases. For example, Debian simultaneously maintains multiple releases (branches). The “stable” branch is recommended for production use, “testing” for more recent software versions. Every two years or so, the branches “shift” (i.e. the previous “testing” becomes the new “stable”, and so on) and users are asked to upgrade to the next release.

Fedora releases aren’t really branched like Debian. Instead, they make a new release (with potentially major changes for an upgrade) every half a year, and maintain old releases for 13 months. You generally start with the newest release, and periodically upgrade.

Arch Linux follows a rolling release model instead. There is just one branch that all Arch users use, and releases are made periodically only for the purpose of installation media. Major upgrades are done in-place (and I have to say, they don’t always go well).

Now, Gentoo is something of a hybrid, as it combines the best of both worlds. It is a rolling release distribution with a single shared repository that is available to all users. However, within this repository we use a keywording system to provide a choice between stable and testing packages, to facilitate both production and development systems (with some extra flexibility), and versioned profiles to tackle major lock-step upgrades.

Continue reading “Gentoo: profiles and keywords rather than releases”

Optimizing distutils-r1.eclass via wheel reuse

Yesterday I’ve enabled a new distutils-r1.eclass optimization: wheel reuse. Without this optimization, the eclass would build a separate wheel for every Python implementation enabled, and then install every one of these wheels. In many cases, this meant repeatedly building the same thing. With the optimization enabled, under some circumstances the eclass will be able to build one (or two) wheels, and install them for all implementations.

This change brings the eclass behavior closer to the behavior of package managers such as pip. While this will cause no change for users who build packages for a single Python version only, it can bring some nice speedup when building for multiple interpreters. Particularly, pure Python packages using setuptools will no longer incur the penalty of having to start setuptools multiple times (which is quite slow), and packages using the stable ABI won’t have to build roughly identical extensions multiple times.

In this post, I’m going to shortly go over a few design considerations of the new feature.

Continue reading “Optimizing distutils-r1.eclass via wheel reuse”