New @live-rebuild and @module-rebuild package sets

I’ve implemented new @live-rebuild and @module-rebuild package sets, and added them to the default sets.conf file for the next sys-apps/portage release (that will be 2.2_rc6). Here are the relevant sections from sets.conf:

# Installed ebuilds that inherit from known live eclasses.
[live-rebuild]
class = portage.sets.dbapi.InheritSet
world-candidate = False
inherits = cvs darcs git mercurial subversion

# Installed packages that own files inside /lib/modules.
[module-rebuild]
class = portage.sets.dbapi.OwnerSet
world-candidate = False
files = /lib/modules

Given the new InheritSet and OwnerSet classes, it’s easy to define similar sets based on inherited eclasses and installed files, respectively.

EAPI=”2_pre1″ with USE dependencies

Version 2.2_rc5 of sys-apps/portage adds support for EAPI=”2_pre1″ which has some changes in conditional USE dependency syntax. The previous experimental EAPI=”2_pre0″ value is no longer supported.

For more information about experimental EAPI features, refer to bug #233213 and also see the “Ebuild” chapter of the html documentation that is installed with portage when USE=doc is enabled. I’ve posted a copy of the EAPI=”2_pre1″ documentation here:

EAPI 2_pre1

  Helpers

    doman

      Language codes in file names are now used for path translation.

      Table 6.1. Man Page Path Translation

        Source        Destination
        foo.1         /usr/share/man/man1/foo.1
        foo.lang.1    /usr/share/man/lang/man1/foo.1

  USE Dependencies

    Unconditional USE Dependencies

      Table 6.2. Syntax Examples

        Example          Meaning
        foo[bar]         foo must have bar enabled
        foo[bar,baz]     foo must have both bar and baz enabled
        foo[-bar,baz]    foo must have bar disabled and baz enabled

    Conditional USE Dependencies

      Table 6.3. Syntax Examples

        Compact Form    Equivalent Expanded Form
        foo[bar?]       bar? ( foo[bar]  ) !bar? ( foo       )
        foo[!bar?]      bar? ( foo       ) !bar? ( foo[-bar] )
        foo[bar=]       bar? ( foo[bar]  ) !bar? ( foo[-bar] )
        foo[!bar=]      bar? ( foo[-bar] ) !bar? ( foo[bar]  )

Portage now supports building multiple packages in parallel

In >=portage-2.2_rc2 there are a few new emerge options that many Gentoo users will probably be interested in:

--jobs JOBS
	Specifies the number of packages to build simultaneously.
	Also see the related --load-average option.

--keep-going
	Continue as much as possible after an error. When an error
	occurs, dependencies are recalculated for remaining packages
	and any with unsatisfied dependencies are automatically
	dropped. Also see the related --skipfirst option.

--load-average LOAD
	Specifies that no new builds should be started if there are
	other builds running and the load average is at least LOAD (a
	floating-point number). This option is recommended for use in
	combination with --jobs in order to avoid excess load. See
	make(1) for information about analogous options that should
	be configured via MAKEOPTS in make.conf(5).

Here is some sample parallel build output from a catalyst stage2 build, with emerge’s new –jobs option enabled:

>>> Building (1 of 10) sys-devel/gettext-0.17 for /
>>> Building (2 of 10) sys-libs/zlib-1.2.3-r1 for /
>>> Building (3 of 10) virtual/libintl-0 for /
>>> Building (4 of 10) dev-util/unifdef-1.20 for /
>>> Installing virtual/libintl-0 to /
>>> Installing dev-util/unifdef-1.20 to /
>>> Building (5 of 10) sys-kernel/linux-headers-2.6.23-r3 for /
>>> Installing sys-libs/zlib-1.2.3-r1 to /
>>> Jobs: 3 of 10 complete, 2 running               Load avg: 3.44, 1.46, 0.69

Handling File Collisions Between Blocking Packages

Blocker Conflicts

If one package blocks another package, the two packages conflict such that they cannot be installed simultaneously. These conflicts are often due to file collisions. In some cases, packages that block each other can be temporarily installed simultaneously. In order to resolve file collisions that occur between two blocking packages that are installed simultaneously, the overlapping files must be removed from the contents list of the package which was installed first.

Some cases may exist such that temporary simultaneous installation of blocking packages will cause some sort of problem. However, this type of solution will only be chosen for blockers that can not be satisfied in any other way, such as by simple adjustment of merge order. In addition, this type of solution will not be chosen if a blocking package will overwrite files belonging to packages from the system set, or packages that are runtime dependencies of Portage itself. These constraints serve to limit the probability that a chosen solution will cause an unforeseen problem.

gtk-doc-am vs. gtk-doc-1.8-r2

These are the packages that would be merged, in reverse order:

Calculating dependencies... done!
[blocks b     ] dev-util/gtk-doc-am (is blocking dev-util/gtk-doc-1.8-r2)
[uninstall    ]  dev-util/gtk-doc-1.8-r2  USE="emacs -debug -doc"
[nomerge      ] x11-libs/gtk+-2.12.9-r2  USE="X cups jpeg tiff xinerama -debug -doc -vim-syntax"
[ebuild  N    ]  dev-util/gtk-doc-am-1.10-r1  0 kB

Total: 1 package (1 new, 1 uninstall), Size of downloads: 0 kB
Conflict: 1 block

coreutils vs. mktemp

These are the packages that would be merged, in reverse order:

Calculating dependencies... done!
[nomerge      ] sys-apps/coreutils-6.11 [6.9-r1] USE="acl nls xattr (-selinux) -static -vanilla%"
[blocks b     ]  sys-apps/mktemp (is blocking sys-apps/coreutils-6.11)
[blocks b     ] >=sys-apps/coreutils-6.10 (is blocking sys-apps/mktemp-1.5)
[uninstall    ]  sys-apps/mktemp-1.5
[ebuild     U ] sys-apps/coreutils-6.11 [6.9-r1] USE="acl nls xattr (-selinux) -static -vanilla%" 0 kB

Total: 1 package (1 upgrade, 1 uninstall), Size of downloads: 0 kB
Conflict: 2 blocks

Using Blockers to Automatically Uninstall Unwanted Packages

A while ago I wrote a blog entry about

using blockers to adjust merge order
. Now, in
portage-2.1.5,
blockers are also resolved automatically in cases when it makes
sense to uninstall a conflicting package (bug #172812).
This feature should allow automatic resolution of

blocker conflicts
in many more cases than previously possible, so Gentoo
users won’t be inconvenienced with the task of resolving them manually.

gtk-doc-am vs. gtk-doc-1.8-r2

These are the packages that would be merged, in reverse order:

Calculating dependencies... done!
[nomerge      ] x11-libs/gtk+-2.12.9-r2  USE="X cups jpeg tiff xinerama -debug -doc -vim-syntax"
[ebuild  N    ]  dev-util/gtk-doc-am-1.10-r1  0 kB
[blocks b     ] dev-util/gtk-doc-am (is blocking dev-util/gtk-doc-1.8-r2)
[uninstall    ]  dev-util/gtk-doc-1.8-r2  USE="emacs -debug -doc"

Total: 1 package (1 new, 1 uninstall), Size of downloads: 0 kB
Conflict: 1 block

coreutils vs. mktemp

These are the packages that would be merged, in reverse order:

Calculating dependencies... done!
[ebuild     U ] sys-apps/coreutils-6.11 [6.9-r1] USE="acl nls xattr (-selinux) -static -vanilla%" 0 kB
[blocks b     ]  sys-apps/mktemp (is blocking sys-apps/coreutils-6.11)
[blocks b     ] >=sys-apps/coreutils-6.10 (is blocking sys-apps/mktemp-1.5)
[uninstall    ]  sys-apps/mktemp-1.5

Total: 1 package (1 upgrade, 1 uninstall), Size of downloads: 0 kB
Conflict: 2 blocks

Portage Dependency Resolution – Decision Making

I’ve written “Part II. Dependency Resolution” for the sys-apps/portage documentation (installed with USE=doc enabled). The “Decision Making” chapter is posted here. There are also two more small chapters about package modeling and task scheduling.

Portage Dependency Resolution – Decision Making

Dependency Expression Evaluation

In terms of boolean logic, a dependency expression can
be expressed in disjunctive normal form (DNF), which is a
disjunction of conjunctive clauses. Each conjunctive clause
represents one possible alternative combination of dependency
atoms capable of satisfying the dependency expression.

Look-Ahead

When there are multiple combinations to choose from, a
look-ahead mechanism will choose an optimal combination to
satisfy constraints and minimize cost. The following package
states influence the cost calculation for a given combination:

  • installed
  • selected (for installation)
  • not selected (for installation)

In cost calculations, virtual packages by themselves are
considered to cost nothing since they do not directly install
anything. It is the dependencies of a virtual package that
contribute to it’s cost.

Constraint Propagation

Combinations that include packages from the "installed" or
"selected" categories are less costly than those that include
packages from the "not selected" category. When a package
is chosen for installation, it transitions to the "selected"
state. This state change propagates to the cost calculations of
later decisions, influencing later decisions to be consistent
with earlier decisions. This feedback mechanism serves to
propagate constraints and can influence the modeling process
to converge on a more optimal final state.

Expanded Search Space

When evaluating virtual atoms, an expanded search space is
considered which recursively traverses the dependencies of
virtual packages from all slots matching a given virtual
atom. All combinations in this expanded search space are
considered when choosing an optimal combination to satisfy
constraints with minimal cost.

Using Blockers to Adjust Merge Order and Avoid Conflicts

Gentoo users often think of blockers [1] negatively since it can be annoying to resolve them manually. Since portage-2.1.2, in some cases emerge can automatically adjust merge order to resolve blockers so that the user doesn’t have to know about them.

Often, the conflict is caused by 2 packages installing files in the same location. A fairly common case that can be resolved automatically is when a file moves from one package to another between versions. For example, udev versions between 111 and 113 install the file /etc/udev.d/rules.d/64-device-mapper.rules which is also installed by device-mapper-1.02.19-r1. If a user upgrades to device-mapper-1.02.19-r1 while a version of udev between 111 and 113 is still installed, a file collision will occur [2]. It’s best to avoid file collisions since the version that one package installs is not necessarily compatible with the version that another package installs. The collision can be avoided automatically by adding RDEPEND="!~sys-fs/udev-111 !~sys-fs/udev-112 !~sys-fs/udev-113" to the device-mapper-1.02.19-r1 ebuild. This causes the merge order to be adjusted so that udev is upgraded before device-mapper. The colliding file is uninstalled during the udev upgrade and then reinstalled during the device-mapper upgrade.

These are the packages that would be merged, in order:

Calculating dependencies… done!
[ebuild U ] sys-fs/udev-114 [113]
[ebuild U ] sys-fs/device-mapper-1.02.19-r1 [1.02.19]

File collisions aren’t the only situation where blockers can be useful to adjust merge order. For example, some stage tarballs contain a minimal version of perl than has the “build” USE flag enabled. Some packages such as Test-Harness that depend on perl can not be built until perl is rebuilt without that flag. This type of conflict can be represented in the perl ebuild with RDEPEND="build? ( !perl-core/Test-Harness )". The merge order will be automatically adjusted so that perl is rebuilt before Test-Harness is installed.

These are the packages that would be merged, in order:

Calculating dependencies… done!
[ebuild R ] dev-lang/perl-5.8.8-r2 USE=”-build*”
[ebuild N ] perl-core/Test-Harness-2.64

[1] A blocker is a type of ebuild dependency that represents a conflict between 2 packages.
[2] File collisions can be detected by adding “collision-protect” to the FEATURES variable in /etc/make.conf.