The Ultimate Guide to EAPI 6

Now that EAPI 6 is Council-approved and pretty close to being deployed, I think it’s about time to write up a not-so-short guide to it. It’s especially important that EAPI 6 is a bit different from the previous EAPIs. It was not only seen as an opportunity to add new features but also to tidy things up a bit and improve strictness.

If you look into the PMS, you’d see that we’d not only added completely new stuff but also ported a few common eclass functions, in a little cleaner form, and new useful phase functions. Furthermore, we used EAPI 6 as an opportunity to finally increase strictness of Portage in regard to PMS, and ban some long-unwanted eclasses.

Therefore, I’d like to ask you — please don’t go hooray-enabling EAPI 6 support in your ebuilds and eclasses. Take a while to think, and do things right. Please think of EAPI 6 as a lifetime opportunity of improving your eclass APIs, as we improved the API provided by Package Managers.

Now, time for a little summary of changes, their implications and a bit of rationale.

Bash upgrade and compatibility setting

The first important change in EAPI 6 is the upgrade to bash-4.2 (the earlier EAPIs required bash-3.2). At a first glance, it just brings a few nice quirks like cleaner fd magic (used by multiprocessing.eclass), case conversion for variables and associative arrays. However, aside to that bash compatibility setting is forced — that could hopefully prevent existing code from breaking in future bash versions.

What does this solve, you ask? For example, it solves the tilde substitution issue:

PV="0_beta1"
MY_PV="${PV/_/~}"

The above snippet gives the expected result only when executed with bash-4.2 and lower. However, starting with bash-4.3 it performs tilde expansion on right-hand argument of the pattern substitution and puts your home directory in there. You can disable that via backslash-escaping the argument — but then it puts an extra backslash in older bash versions! How frustrating…

Now, if we have BASH_COMPAT fixed by the PM at 4.2, this discrepancy is no longer an issue. Because all supported versions of bash will force the old behavior, you can write the code without any conditionals, workarounds or worrying that it will suddenly break sometime in the future.

failglob in global scope

The second change to ebuild syntax is that filename expansion is now an explicit error in global scope. While this may sound scary at first, it wasn’t ever really supported. Note that this doesn’t affect phase functions, just the code executed when sourcing the ebuild. This is mostly targeted towards snippets like the following:

inherit python-r1

RDEPEND="$(python_gen_cond_dep 'dev-python/foo[${PYTHON_USEDEP}]' python2*)"

Do you see the problem here? python2* lacks quoting. This usually doesn’t cause any issues but if you happened to be in a directory with pythonFOOBAR file while sourcing the ebuild, it would suddenly pick it up and return pretty unexpected results. As a solution, an EAPI=6 ebuild would instantly abort here and require you to quote the pattern.

LC_COLLATE and LC_CTYPE settings

The description for locale settings may seem a little blurry but that’s for a reason. The original idea was that the Package Manager would export (pretty much like Paludis does):

LC_COLLATE=C
LC_CTYPE=C

to enforce stable sorting and case conversions. Without the former, some locales could cause results of globs such as patch files to apply in different order. Without the latter, some locales could give unexpected results of case changes. In particular, the Turkish tr_TR.UTF-8 locale is only too happy to map lowercase ‘i’ to uppercase ‘İ’ (yep, it has a dot above!)

So why is it worded this muddy? Because of our lovely Python and the lot of high quality scripts that are only too happy to interpret read files in the charset corresponding to the current locale, and therefore bail out hard when they don’t conform to the ASCII charset implied by the C locale.

So instead of C, we need a C.UTF-8 which hasn’t made its way into glibc for… how long exactly? Not to mention the portability. Therefore, we just request a seemingly sane locale, and hope we can work something out. In implementation, the Package Manager will most likely set LC_COLLATE=C directly and complain to the user if his LC_CTYPE does not conform.

Nevertheless, in EAPI 6 you should be able to safely remove your local LC_COLLATE and LC_CTYPE redefinitions, and hopefully also most of those excessive LC_ALL settings.

eapply, eapply_user and src_prepare()

This is probably the most controversial feature of the new EAPI. Those functions are intended as reasonably simple and predictable replacements for the commonly used eutils.eclass beasts, with new default src_prepare() to finish off the long-deprecated base.eclass.

eapply has a few substantial differences from epatch. However, I hope that people will nevertheless decide to abandon the latter, and use the new function, effectively also reducing the use of eutils.eclass. The differences are:

  • No -p* guessing. Instead, it defaults to -p1 and lets you override it. Now patches can apply predictably.
  • No magical configuration variables. All options are passed as parameters, followed by files. If options need to take separate arguments, -- is provided to separate options from files.
  • No compression support. There’s unpack that usually does that for you, you know, and patches in ${FILESDIR} are never compressed.
  • Simple directory handling. It just applies all *.diff and *.patch files from a directory. No custom suffixes, no exclusions.

While this may sound like a lot was lost, the function is still quite big and even twice as useful. Most importantly, all patches now apply (or fail) pretty predictably, and if they do fail, you get a clear output.

eapply_user provides user patch support on top of eapply. However, since user patches are a matter of configuration, the PMS really leaves the details to the Package Manager, and doesn’t even prohibit it from extending patching beyond eapply. eapply_user is considered obligatory, and can be safely called more than once (applying patches only the first time) to make eclass writing easier.

Both function are used in the default src_prepare() implementation which was inspired by the common implementations shared between multiple eclasses. In particular, it applies patches from the PATCHES variable (array strongly preferred due to ${FILESDIR}) and then user patches. Please note that in some cases, it would be advised to remove src_prepare() from your eclass altogether rather than calling the default from it.

einstalldocs and src_install()

This one’s much simpler and accepted in full agreement. It combines two categories of requests created against the documentation install code introduced in EAPI 4. Firstly, it splits the code off default_src_install() so that it can be reused without implicitly calling emake install. Secondly, it solves a number of problems with the original implementation.

So what’s the story here? EAPI 4 introduced a very nice src_install() phase which was able to install a few default documentation pieces automatically. That was quite liked, except that it was tightly bound with the default emake install call. As a result, a number of eclasses ended up (still) redefining it, with the most noticeable example of base.eclass having a split phase function for it. But since the eclass was long deprecated already, other eclasses had to copy the code rather than reuse it.

The idea of having a split function arose early during EAPI 6 work. While writing a new function, the issues and requests against the current default were collected and considered. Finally, the new function was given to early Council approval, and committed to eutils.eclass as a backport of the future EAPI 6 function. And as you can see now, it lands in EAPI 6 in exactly the same form, replacing the eutils implementation.

As a reminder, the changes from the original EAPI 4 implementation are:

  • empty DOCS variable disables installing documentation completely. Previously, it was undefined behavior (usually a dodoc usage error).
  • The DOCS variable can name directories as well as files.
  • HTML_DOCS variable has been added. It installs the listed files and/or directories (using dodoc, not deprecated dohtml!) into the html/ subdirectory of docdir.

As a result, the default src_install() implementation differs by the documentation install changes listed above. If you were overriding this phase just to get einstalldocs into it, you can remove it. If you were using einstalldocs from eutils.eclass, you do not have to inherit it anymore. And if you were inlining your own documentation install code, it is a good idea to replace it with standard one in EAPI 6.

get_libdir and econf changes

The next major target for EAPI 6 changes was the econf function, commonly used in src_configure(). Here, two goals were achieved. Firstly, a get_libdir function inspired by multilib.eclass was added. Secondly, --docdir and --htmldir parameters are now passed to configure scripts.

Why would get_libdir be related to econf at all? Well, except for the fact that it’s often used in configure script arguments, it pretty much shares the logic used to figure out libdir. So we’re not just moving a commonly used eclass function to the EAPI — we’re splitting the libdir determination logic off econf, and sharing it with ebuilds.

As for the two new parameters — --docdir and --htmldir — I don’t think they really need any comment here.

in_iuse

This one’s very controversial as well, and I’d personally prefer not having to introduce it at all. However, without it the developers would still resort to ugly hacks, so better to have a safe alternative. So, what’s the background?

Some developers found it quite convenient to do some conditionals in eclasses based on what ends up in IUSE. For example, if the ebuild has IUSE=static-libs, we magically enable code controlling static library build and install. Except for being ugly, this design had two major issues. Firstly, the code checking that was inconsistent and often buggy — some eclasses handled +static-libs, some didn’t. Secondly, the specification never mandated IUSE having any sane value throughout phase functions. So the results could have been quite surprising.

EAPI 6 solves both of those issues by providing a standard in_iuse function. All uses implementation-defined magic to check whether the specified flag is listed in the IUSE_EFFECTIVE conceptual variable. This includes not only flags set by ebuilds and eclasses but also implicitly added by profiles — in other words, any flag that is legal to the use function call.

I should note that in_iuse is legal only in phase functions — it is disallowed in global scope (i.e. while IUSE is still subject to changes). Nevertheless, I would heartily recommend avoiding it and using explicit conditional variables instead.

nonfatal die

Did you happen to dislike how some eclass-defined helpers could not be nonfatal properly? That’s no longer the case with EAPI 6. The die function has gained a simple -n option which makes it respect nonfatal, and therefore be a no-op when it is in effect.

In other words, you can write helpers like the following:

efoo() {
	#...
	foo || die -n "foo failed" || return ${?}
	#...
}

where die would simply return false and continue execution if nonfatal is in effect. Please remember that ‘nonfatal die’ does not return from the function implicitly. If you need to do so, you need to explicitly call return like in the above snippet.

unpack changes

Now we’re past the most interesting changes. For completeness, there were few small requests accepted for the unpack function:

  • Absolute and non-prefixed relative paths can be used now. The path handling is still a little creepy since pure filename (i.e. without a single slash) implicitly references ${DISTDIR} but anything with a slash is treated as a regular path.
  • .txz suffix is interpreted equivalently to .tar.xz. Previously, only the long form was accepted.
  • Suffix matching is case-insensitive now. Which means that .ZIP works as well as .zip. Hooray, we can now handle 8.3 filenames cleanly!

einstall banned, dohtml deprecated

Aside to new features, there are two important removals to be noted. The long-discouraged einstall function has been finally banned, and the dohtml function has been deprecated with the goal of removing it in EAPI 7.

The einstall command was intended as a cheap work-around for build systems that did not support DESTDIR= argument to emake install. Instead, it redefined a number of direct install paths (such as bindir=), passing the staging area path and target directory combined. Sadly, it has often been cause of confusion for new developers who have seen it as the standard install command (in line with econf, emake…).

Over a year ago, I’ve started tracking uses of einstall on behalf of QA, and noticed that most of the uses were completely unnecessary, either because the build system in question handled DESTDIR= correctly, or had a custom variable serving the same purpose. Furthermore, the function used parameter names matching autoconf while it was intended to deal with custom build systems which may require other variables. Therefore, it was decided that it is better to ban the command and inline the relevant make parameters whenever really unavoidable.

The dohtml function has been the official way of installing HTML documentation for quite a long time. No other command defined by PMS had as many options. Confusingly to some developers, it provided implicit filtering by file suffixes, with additional options to add additional suffixes, replace suffix list, filter by filename, exclude directories and directly alter install path. Moreover, there was yet another enhancement request asking for new suffixes in the default filter.

The Council has agreed with us that the function is overcomplex and confusing, and will be an object of never-ending requests and updates. Most of Gentoo developers have seen it as a way of installing files in the html/ subdirectory of docdir while it did so much more — and potentially confused them by excluding some files. It also resulted in absurd situations like people moving gtk-doc files out of the standard location used by gtk-doc just to match the ‘standard’ established by the function.

dohtml is officially deprecated since EAPI 6, and is expected to be banned in EAPI 7. The suggested replacement is dodoc which is much simpler, faster and in many cases will work the same (or even better, by restoring the files that were mistakenly missed by dohtml). It is possible that the current dohtml will eventually be moved to an eclass for the ebuilds that require filtering, yet the fate of this idea is unclear yet.

To help with the migration, Portage provides the following quirk (for make.conf or environment):

PORTAGE_DOHTML_WARN_ON_SKIPPED_FILES=yes

When run with this, each dohtml invocation will verbosely list all files that were skipped through filtering. A null output indicates that dohtml can be safely replaced by dodoc.

Global-scope helpers banned in Portage

EAPI 6 brings not only the spec changes but also new strictness to Portage. In particular, almost all helper functions are now banned in global scope, as they are banned by PMS and other package managers. What does this imply? Most importantly, toolchain’s use of USE=multislot is now officially banned and will no longer work in EAPI 6.

Eclasses banned in EAPI 6

Finally, the changes brought by EAPI 6 will likely result in a number of API changes in multiple eclasses. Moreover, some of the existing eclasses will become completely unnecessary and will be banned.

The most important example here is the base.eclass which was semi-officially deprecated for a long time. In EAPI 6, it is fully replaced by default phase functions and must not be used. Similarly, I’ve decided to discontinue and ban autotools-utils.eclass and autotools-multilib.eclass, the former serving a similar purpose as base.eclass did in the past and the latter being just a wrapper atop multilib-minimal.eclass.

Aside to that, I’ve banned a number of eclasses that are deprecated currently, to avoid their use being carried on to the next EAPI.

Summary

To summarize all the changes in EAPI 6, let me assemble a check list of tasks that need to be done when updating ebuilds to EAPI 6:

  1. You can update your bash code to use bash-4.2 features, and remove unnecessary conditionals for future-incompatible behavior. It is unlikely that you have any.
  2. Ensure that all patterns that are supposed to be passed as-is to functions are quoted.
  3. Remove unnecessary LC_COLLATE and LC_CTYPE if they only reset it to the C (or POSIX) locale.
  4. If you are using epatch, try hard to replace it with eapply. You may need to pass an explicit -p value.
  5. If you are using epatch_user, you have to replace it with eapply_user. If you are not and you define src_prepare(), you have to add it there.
  6. If you are left with src_prepare() that looks pretty much like the default one, drop it. It is also a good idea to remove unnecessary src_prepare() definitions from eclasses.
  7. If you are using einstalldocs, this function is now part of EAPI. Don’t count it as a reason to inherit eutils.eclass. If you are using a similar inline code, it’s about time you switch to einstalldocs.
  8. If you are using in_iuse, this function is now part of EAPI and even really legal. Don’t count it as a reason to inherit eutils.eclass. In fact, hurry to switch to EAPI 6 to get a predictable behavior!
  9. Check if you are still using anything from eutils.eclass. If you are not, remove the inherit and save a few cycles.
  10. If you are using get_libdir, this function is now part of EAPI. Don’t count it as a reason to inherit multilib.eclass. If you are not using anything else from multilib.eclass, remove the inherit.
  11. If you are passing standard --docdir and/or --htmldir to econf, you can remove them. EAPI 6 does that for you.
  12. If you are defining ebuild helpers that should respect nonfatal, you should update appropriate die to use die -n || return. However, please note that it’s usually a good idea to use regular die on usage errors, and respect nonfatal only for external errors.
  13. If you are using hacks to support .txz, uppercase archive suffixes like .ZIP or extracting files by absolute or relative paths, you can remove them and use new unpack features.
  14. If you are using einstall, remove it. I’m pretty sure you never really needed it. And if you really did, then you most likely missed some custom prefix option in upstream build system.
  15. If you are using dohtml, consider replacing it with dodoc -r. You can set PORTAGE_DOHTML_WARN_ON_SKIPPED_FILES=yes before doing that, to see if the two will be equivalent.
  16. If you are inheriting base.eclass, autotools-utils.eclass or any of the other banned eclasses, make sure to remove it from your eclasses and ebuilds.

1 thought on “The Ultimate Guide to EAPI 6”

  1. I thought autotools-utils was fairly useful. I guess using it for `static-libs` only was abusive. But it also provided out of source building out of the box and easy reconfiguration for the most part. Going back on these two fronts fells like a regression.

Comments are closed.