New overlay config files: metadata/layout.conf and /etc/portage/repos.conf
Since portage-2.2_rc29 there is support for a metadata/layout.conf file in an overlay which is used to specify names of repositories which satisfy dependencies on eclasses and/or ebuilds. See the QA Overlay Layout support thread on the gentoo-dev mailing list for information about why this is necessary, and refer to the portage(5) man page for details about the layout.conf format
If you want overlay eclasses to override eclasses from other repos then you'll want to refer to the portage(5) man page for information about the /etc/portage/repos.conf file which can be used to specify site-specific repository configuration information. Note that configuration settings which are specified in repos.conf do not apply to tools such as repoman(1) and egencache(1), since operations performed by these tools are inherently not site-specific.
Since portage-2.2_rc31, repos.conf can be used to create repository aliases, in case you want to substitute one repo for another one that is specified in layout.conf. This is especially useful for funtoo users, who may want something like this in /etc/portage/repos.conf:
[funtoo]
aliases = gentoo
Informative output for unresolved blockers
In portage-2.1.6.7, emerge output for unresolved blockers now includes information about how conflicting packages got pulled into the dependency graph. This information helps users understand the reason why a particular blocker hasn't been resolved automatically. Here is some sample output from bug 234907:
Calculating dependencies... done!
[ebuild N ] sys-libs/e2fsprogs-libs-1.41.3-r1 USE="nls"
[ebuild U ] sys-fs/e2fsprogs-1.41.3 [1.40.9]
[blocks b ] <sys-fs/e2fsprogs-1.41 ("<sys-fs/e2fsprogs-1.41" is blocking sys-libs/e2fsprogs-libs-1.41.3-r1)
[blocks B ] sys-libs/com_err ("sys-libs/com_err" is blocking sys-libs/e2fsprogs-libs-1.41.3-r1)
[blocks B ] sys-libs/ss ("sys-libs/ss" is blocking sys-libs/e2fsprogs-libs-1.41.3-r1)
[blocks B ] sys-libs/e2fsprogs-libs ("sys-libs/e2fsprogs-libs" is blocking sys-libs/com_err-1.40.9, sys-libs/ss-1.40.9)
* Error: The above package list contains packages which cannot be
* installed at the same time on the same system.
('installed', '/', 'sys-libs/com_err-1.40.9', 'nomerge') pulled in by
~sys-libs/com_err-1.40.9 required by ('installed', '/', 'sys-libs/ss-1.40.9', 'nomerge')
('installed', '/', 'sys-libs/ss-1.40.9', 'nomerge') pulled in by
sys-libs/ss required by world
('ebuild', '/', 'sys-libs/e2fsprogs-libs-1.41.3-r1', 'merge') pulled in by
>=sys-libs/e2fsprogs-libs-1.41 required by ('installed', '/', 'sys-apps/util-linux-2.14.1', 'nomerge')
~sys-libs/e2fsprogs-libs-1.41.3 required by ('ebuild', '/', 'sys-fs/e2fsprogs-1.41.3', 'merge')
The above output shows that the sys-libs/ss package got pulled into the dependency graph by the world set. This package is obsolete, so the user should remove it from /var/lib/portage/world. In the future, I plan to add support for emerge to ask the user if they'd like to have the obsolete package automatically removed from the world file (bug 257009).
New @live-rebuild and @module-rebuild package sets
I've implemented new @live-rebuild and @module-rebuild package sets, and added them to the default sets.conf file for the next sys-apps/portage release (that will be 2.2_rc6). Here are the relevant sections from sets.conf:
# Installed ebuilds that inherit from known live eclasses. [live-rebuild] class = portage.sets.dbapi.InheritSet world-candidate = False inherits = cvs darcs git mercurial subversion # Installed packages that own files inside /lib/modules. [module-rebuild] class = portage.sets.dbapi.OwnerSet world-candidate = False files = /lib/modules
Given the new InheritSet and OwnerSet classes, it's easy to define similar sets based on inherited eclasses and installed files, respectively.
EAPI="2_pre1" with USE dependencies
Version 2.2_rc5 of sys-apps/portage adds support for EAPI="2_pre1" which has some changes in conditional USE dependency syntax. The previous experimental EAPI="2_pre0" value is no longer supported.
For more information about experimental EAPI features, refer to bug #233213 and also see the "Ebuild" chapter of the html documentation that is installed with portage when USE=doc is enabled. I've posted a copy of the EAPI="2_pre1" documentation here:
EAPI 2_pre1
Helpers
doman
Language codes in file names are now used for path translation.
Table 6.1. Man Page Path Translation
Source Destination
foo.1 /usr/share/man/man1/foo.1
foo.lang.1 /usr/share/man/lang/man1/foo.1
USE Dependencies
Unconditional USE Dependencies
Table 6.2. Syntax Examples
Example Meaning
foo[bar] foo must have bar enabled
foo[bar,baz] foo must have both bar and baz enabled
foo[-bar,baz] foo must have bar disabled and baz enabled
Conditional USE Dependencies
Table 6.3. Syntax Examples
Compact Form Equivalent Expanded Form
foo[bar?] bar? ( foo[bar] ) !bar? ( foo )
foo[!bar?] bar? ( foo ) !bar? ( foo[-bar] )
foo[bar=] bar? ( foo[bar] ) !bar? ( foo[-bar] )
foo[!bar=] bar? ( foo[-bar] ) !bar? ( foo[bar] )
Portage now supports building multiple packages in parallel
In >=portage-2.2_rc2 there are a few new emerge options that many Gentoo users will probably be interested in:
--jobs JOBS Specifies the number of packages to build simultaneously. Also see the related --load-average option. --keep-going Continue as much as possible after an error. When an error occurs, dependencies are recalculated for remaining packages and any with unsatisfied dependencies are automatically dropped. Also see the related --skipfirst option. --load-average LOAD Specifies that no new builds should be started if there are other builds running and the load average is at least LOAD (a floating-point number). This option is recommended for use in combination with --jobs in order to avoid excess load. See make(1) for information about analogous options that should be configured via MAKEOPTS in make.conf(5).
Here is some sample parallel build output from a catalyst stage2 build, with emerge's new --jobs option enabled:
>>> Building (1 of 10) sys-devel/gettext-0.17 for / >>> Building (2 of 10) sys-libs/zlib-1.2.3-r1 for / >>> Building (3 of 10) virtual/libintl-0 for / >>> Building (4 of 10) dev-util/unifdef-1.20 for / >>> Installing virtual/libintl-0 to / >>> Installing dev-util/unifdef-1.20 to / >>> Building (5 of 10) sys-kernel/linux-headers-2.6.23-r3 for / >>> Installing sys-libs/zlib-1.2.3-r1 to / >>> Jobs: 3 of 10 complete, 2 running Load avg: 3.44, 1.46, 0.69
Handling File Collisions Between Blocking Packages
If one package blocks another package, the two packages conflict such that they cannot be installed simultaneously. These conflicts are often due to file collisions. In some cases, packages that block each other can be temporarily installed simultaneously. In order to resolve file collisions that occur between two blocking packages that are installed simultaneously, the overlapping files must be removed from the contents list of the package which was installed first.
Some cases may exist such that temporary simultaneous installation of blocking packages will cause some sort of problem. However, this type of solution will only be chosen for blockers that can not be satisfied in any other way, such as by simple adjustment of merge order. In addition, this type of solution will not be chosen if a blocking package will overwrite files belonging to packages from the system set, or packages that are runtime dependencies of Portage itself. These constraints serve to limit the probability that a chosen solution will cause an unforeseen problem.
gtk-doc-am vs. gtk-doc-1.8-r2
These are the packages that would be merged, in reverse order: Calculating dependencies... done! [blocks b ] dev-util/gtk-doc-am (is blocking dev-util/gtk-doc-1.8-r2) [uninstall ] dev-util/gtk-doc-1.8-r2 USE="emacs -debug -doc" [nomerge ] x11-libs/gtk+-2.12.9-r2 USE="X cups jpeg tiff xinerama -debug -doc -vim-syntax" [ebuild N ] dev-util/gtk-doc-am-1.10-r1 0 kB Total: 1 package (1 new, 1 uninstall), Size of downloads: 0 kB Conflict: 1 block
coreutils vs. mktemp
These are the packages that would be merged, in reverse order: Calculating dependencies... done! [nomerge ] sys-apps/coreutils-6.11 [6.9-r1] USE="acl nls xattr (-selinux) -static -vanilla%" [blocks b ] sys-apps/mktemp (is blocking sys-apps/coreutils-6.11) [blocks b ] >=sys-apps/coreutils-6.10 (is blocking sys-apps/mktemp-1.5) [uninstall ] sys-apps/mktemp-1.5 [ebuild U ] sys-apps/coreutils-6.11 [6.9-r1] USE="acl nls xattr (-selinux) -static -vanilla%" 0 kB Total: 1 package (1 upgrade, 1 uninstall), Size of downloads: 0 kB Conflict: 2 blocks
Using Blockers to Automatically Uninstall Unwanted Packages
A while ago I wrote a blog entry about
using blockers to adjust merge order. Now, in
portage-2.1.5,
blockers are also resolved automatically in cases when it makes
sense to uninstall a conflicting package (bug #172812).
This feature should allow automatic resolution of
blocker conflicts in many more cases than previously possible, so Gentoo
users won't be inconvenienced with the task of resolving them manually.
gtk-doc-am vs. gtk-doc-1.8-r2
These are the packages that would be merged, in reverse order: Calculating dependencies... done! [nomerge ] x11-libs/gtk+-2.12.9-r2 USE="X cups jpeg tiff xinerama -debug -doc -vim-syntax" [ebuild N ] dev-util/gtk-doc-am-1.10-r1 0 kB [blocks b ] dev-util/gtk-doc-am (is blocking dev-util/gtk-doc-1.8-r2) [uninstall ] dev-util/gtk-doc-1.8-r2 USE="emacs -debug -doc" Total: 1 package (1 new, 1 uninstall), Size of downloads: 0 kB Conflict: 1 block
coreutils vs. mktemp
These are the packages that would be merged, in reverse order: Calculating dependencies... done! [ebuild U ] sys-apps/coreutils-6.11 [6.9-r1] USE="acl nls xattr (-selinux) -static -vanilla%" 0 kB [blocks b ] sys-apps/mktemp (is blocking sys-apps/coreutils-6.11) [blocks b ] >=sys-apps/coreutils-6.10 (is blocking sys-apps/mktemp-1.5) [uninstall ] sys-apps/mktemp-1.5 Total: 1 package (1 upgrade, 1 uninstall), Size of downloads: 0 kB Conflict: 2 blocks
Portage Dependency Resolution - Decision Making
I've written "Part II. Dependency Resolution" for the sys-apps/portage documentation (installed with USE=doc enabled). The "Decision Making" chapter is posted here. There are also two more small chapters about package modeling and task scheduling.
Portage Dependency Resolution - Decision Making
Dependency Expression Evaluation
In terms of boolean logic, a dependency expression can
be expressed in disjunctive normal form (DNF), which is a
disjunction of conjunctive clauses. Each conjunctive clause
represents one possible alternative combination of dependency
atoms capable of satisfying the dependency expression.
Look-Ahead
When there are multiple combinations to choose from, a
look-ahead mechanism will choose an optimal combination to
satisfy constraints and minimize cost. The following package
states influence the cost calculation for a given combination:
- installed
- selected (for installation)
- not selected (for installation)
In cost calculations, virtual packages by themselves are
considered to cost nothing since they do not directly install
anything. It is the dependencies of a virtual package that
contribute to it's cost.
Constraint Propagation
Combinations that include packages from the "installed" or
"selected" categories are less costly than those that include
packages from the "not selected" category. When a package
is chosen for installation, it transitions to the "selected"
state. This state change propagates to the cost calculations of
later decisions, influencing later decisions to be consistent
with earlier decisions. This feedback mechanism serves to
propagate constraints and can influence the modeling process
to converge on a more optimal final state.
Expanded Search Space
When evaluating virtual atoms, an expanded search space is
considered which recursively traverses the dependencies of
virtual packages from all slots matching a given virtual
atom. All combinations in this expanded search space are
considered when choosing an optimal combination to satisfy
constraints with minimal cost.
Using Blockers to Adjust Merge Order and Avoid Conflicts
Gentoo users often think of blockers [1] negatively since it can be annoying to resolve them manually. Since portage-2.1.2, in some cases emerge can automatically adjust merge order to resolve blockers so that the user doesn't have to know about them.
Often, the conflict is caused by 2 packages installing files in the same location. A fairly common case that can be resolved automatically is when a file moves from one package to another between versions. For example, udev versions between 111 and 113 install the file /etc/udev.d/rules.d/64-device-mapper.rules which is also installed by device-mapper-1.02.19-r1. If a user upgrades to device-mapper-1.02.19-r1 while a version of udev between 111 and 113 is still installed, a file collision will occur [2]. It's best to avoid file collisions since the version that one package installs is not necessarily compatible with the version that another package installs. The collision can be avoided automatically by adding RDEPEND="!~sys-fs/udev-111 !~sys-fs/udev-112 !~sys-fs/udev-113" to the device-mapper-1.02.19-r1 ebuild. This causes the merge order to be adjusted so that udev is upgraded before device-mapper. The colliding file is uninstalled during the udev upgrade and then reinstalled during the device-mapper upgrade.
These are the packages that would be merged, in order:
Calculating dependencies... done!
[ebuild U ] sys-fs/udev-114 [113]
[ebuild U ] sys-fs/device-mapper-1.02.19-r1 [1.02.19]
File collisions aren't the only situation where blockers can be useful to adjust merge order. For example, some stage tarballs contain a minimal version of perl than has the "build" USE flag enabled. Some packages such as Test-Harness that depend on perl can not be built until perl is rebuilt without that flag. This type of conflict can be represented in the perl ebuild with RDEPEND="build? ( !perl-core/Test-Harness )". The merge order will be automatically adjusted so that perl is rebuilt before Test-Harness is installed.
These are the packages that would be merged, in order:
Calculating dependencies... done!
[ebuild R ] dev-lang/perl-5.8.8-r2 USE="-build*"
[ebuild N ] perl-core/Test-Harness-2.64
[1] A blocker is a type of ebuild dependency that represents a conflict between 2 packages.
[2] File collisions can be detected by adding "collision-protect" to the FEATURES variable in /etc/make.conf.