Optional runtime dependencies (or «suggested dependencies») are one of the late problems we’re facing in Gentoo. There’s definitely a need for some standard solution, and it’d be great to put it in the next EAPI. Sadly, there’s no consensus how to solve it.
The optional dependencies problem
Gentoo has a very neat solution for handling optional dependencies and optional features, probably ever since the beginning. It is called «USE flags» and they work very well with the «traditional» optional dependencies. By that, I mean optional dependencies which are both build- and run-time.
Such a dependencies have to be pulled in before the build process starts, and usually require passing specific options in the configure phase. What’s important, both enabling and disabling features requires rebuilding the program in question because of code branches being switched. Thus, it’s perfectly fine if changing USE flags implies rebuilding the package.
Sadly, when it comes to optional runtime dependencies, USE flags are not a perfect solution. «Switching» such a dependency doesn’t require rebuilding the program anymore. It’s usually not even switching — the program can determine in runtime whether a particular dependency is available, and either enable or disable respective features. Simple like that.
If one decides to use USE flags for that, they become partially meaningless. Unless flags start stripping off the code (which is a bad idea), feature availability is dependency- rather than flag-based. So, USE=-ssl is irrelevant if, say, pyopenssl is installed. What’s even worse, flag imply needless rebuilding of such packages just to pull in an additional dependency.
The simple hack — pkg_postinst() messages
The simplest solution right now is just listing the suggested dependencies in pkg_postinst() messages. Combined with has_version helper, those messages can give a pretty nice output, pointing out already installed packages — just take a look at sys-apps/systemd ebuild.
Of course, it’s not a real solution, rather relying on user doing the hard work. The biggest disadvantage is that the dependencies are often going to end up in @world. And then, if user decides to unmerge our package, portage is unable to find and unmerge them as well.
The SDEPEND solution
A pretty common idea is to establish a new variable called SDEPEND (for «suggested»). Such a variable would simply list relevant dependencies, and let portage handle the UI part somehow. It is a minimalistic solution, quite consistent with other parts of PMS. Sadly, it has a few big shortcomings.
First, using our current dependency syntax, you can’t specify that a particular feature requires more than one package; in other words, that two or more suggested dependencies are supposed to be pulled in together. Of course, solving this one would be pretty easy — e.g. by allowing grouping them with parantheses.
A much more important issue is describing what particular dependencies do. Although sometimes this could be guessed by package descriptions pretty well, usually a more friendly text would be great. So, we end up having to implement that somehow.
And that’s usually when Ciaran comes in with ugly exherbism DEPENDENCIES. Sure, it solves most of the issues pointed out here but, hell, do we really want such a thing? Isn’t dependency syntax obscure enough already?
And it’s all rather dependency-oriented. In other words, package comes first, then goes the feature description. «Pass dev-python/pyopenssl or dev-python/python-gnutls to enable secure connections support». I don’t think that’s the most user friendly solution.
The USE flag solution
Another solution is brining a new category of USE flags. It’s not important whether they would be specified using a special variable, common USE_EXPAND or another magical features. In fact, that could be a thing totally separate from USE flags. The point is that some of the package flags would be runtime-switchable.
Unlike traditional USE flags, such flags wouldn’t be stored in vdb. They would be evaluated in place instead, using package.use or similar files, and the dependency tree would use current state of such flags. Of course, they would be allowed for RDEPEND (PDEPEND) use only.
Why reuse USE flags for that? Because it’s the most user-friendly solution. User doesn’t have to learn anything new. He/she enables a flag, does emerge -vDtN @world and notices that new dependencies are pulled in but the package doesn’t have to be rebuilt for that.
If we just add some additional magic for regular USE flags, enabling run-time dependant SSL support could be exactly the same as enabling build-time one — even using the same USE=ssl.
And we could basically even give «backwards» support for older EAPIs. Package managers not supporting the new feature would simply treat runtime-switchable USE flags as regular USE flags, requiring rebuilds of the package.
USE-flags are cool for run-time optional dependencies, but they do not resolve other, but similar problem – non compilable built-time additions that use additional flags. Simpliest example: init.d scripts, cron jobs, additional scripts, doc files (even Doxygen compiled one – you still don’t need any rebuilding for that). It’s a shame when you need to reemerge something monstrous like libreoffice or webkit-gtk (or gcc) just for ‘doc’ USE-flag to be added (and just to discover, that this USE-flag adds just a few README.tar.bz2 files into /usr/share/doc and actual documentation was generated, but not installed due to long-playing bug in ebuild for distant past).
IMO, adding some new use-flag category for runtime deps only is not sufficient, portage rather need some way to tell what exactly needs to be done with USE-flag switch.
This could be done, IMO, in two generic approaches:
1) Add some monstrous variable, which would tell what function should be called to process this particular dependency, and portage should call functions for changed USE-flags only. But this will introduce some additional function-from-function dependencies (install needs compile), which also will require additional variable or even more monstrous variable syntax.
2) As we already introduced “changed USE-flag” term then… Why not just allow for ebuild to check what use-flags was changed? Like
Other functions – toggled_on, toggled_off, toggled. It will be also nice to introduce USE-flags groups in IUSE variable to allow batch-testing, like
. If any flag in group has changed full group has changed.
Talking about run-time dependencies – with such behaviour it will only require to add dynamic requirements generation in pkg_pretend function (or without anything at all, just group all flags-that-need-recompilation and protect any compilation with
– all run-time only flags will just do nothing. This will also work with current dependency calculations scheme just fine – you don’t need some additional special USE-flags) .
Of course – such proposition has some problems with “How we will determine what is installed by which flag and what to remove if flag switched off (or how not to clean executables when only new doc’s installed)?”. That’s a bigger problem, but it could be handled by just a little more sophisticated merge stage and, maybe, additional file markings in src_install by developer (which is bad by design).
With the current architecture, it’s practically impossible. And I don’t think anyone would volunteer to implement such a broad changes. Not to mention that with your proposal user may no longer be able to rebuild the whole package when he/she wants that done.
I’ve noticed that “it solves all the problems but Exherbo uses it so it’s ugly” comes up a lot. If it’s any help in your aesthetic judgement, though, I’m pretty sure the DEPENDENCIES syntax was originally designed for Gentoo cross-compiling, and not by anyone on your naughty list, and it predates Exherbo considerably.
I’d appreciate if you didn’t rephrase my words to spell random things I didn’t say. Thank you.
Um… very late friday evening but doesn’t
use? (foo bar)
do the trick with multiple deps on one feature?
You want to have a way to pull in optional components that do not require a rebuild and that could be split this way:
– Have a way to mark USE change that do not trigger rebuild
– Have a way to list such depedencies
So the simplest way is to add another dep group and add a use modifier.
Then you have to improve repoman a bit to avoid wrong usage.
Turning on and off abilities for an installed package is already something that Gentoo has.
It works well for slot dependencies, bash-completions, x11 backends etc. Can’t eselect be extended to pick and choose which (of the available) optional dependencies is used?
Ben Cordero: yes and no, the eselect modules which I understand, all are just fancy CLI for mv or ln -s and rm. For docs it would be enough although ideally you wouldn’t build them in the first place. More importantly though I doubt runtime dependency installation should be left to an eselect module since that sounds like something not within eselect problem domain.
Hello,
I’m glad that this is being discussed. Here I propose the “magic” that should be possible to implement within the current EAPI. If you like my idea, please do tell me where I should post it for the devs to see it. Perhaps I could try to implement it if you think it’s possible.
I. The dependencies would be set in RDEPEND conditionally with USE flags as with anything else.
II. Introduce a function check_runtime_deps in module runtime-deps to be inherited by ebuilds which could be triggered in src_prepare, pkg_setup, pkg_postinst or where applicable and would take a list of the non-build-time flags as arguments.
What it would do:
1) check if the package is already installed AND if yes, scan the list of flags passed as args AND if their state has changed, call the body of the runtime_deps_setup (sort of an AND pipeline)
2) merge the files created by runtime_deps_setup in a way that would NOT remove files from previous builds (“add-or-modify-only”-like merge), only “update”
3) call terminate_checking which would scan other build-time USE flags and if they have changed, continue ebuild processing, exit ebuild processing otherwise
III. The body of runtime_deps_setup would be declared in the ebuild and would mainly consist of
if use foo; then ...constructsI believe this would provide quite a flexible framework for documentation building or anything with the benefit of being able to specify where to perform this action (as not all packages are built in the same way, may need to prepare the source first, etc.) and also takes away the need to make some special USE flags. This would also make it again possible to have a lot more systemd, bash-completion, LANG for locales, etc. flags that IIRC some devs have obsoleted for the nuisance of needing to rebuild the whole package and advised the use of INSTALL_MASK.
The only things that I’m not exactly sure how to implement is the merging of the image in a way that would only “update” the filesystem contents and how to scan the USE flags for changes in (non-)build-time flags.
Otherwise I think it should be rather simple to implement it.
Here’s an example for htop which has lsof as a runtime dep:
--- /usr/portage/sys-process/htop/htop-1.0.1.ebuild 2012-03-21 22:01:28.000000000 +0100
+++ /tmp/htop-1.0.1.ebuild 2012-04-14 23:09:59.245042463 +0200
@@ -4,7 +4,7 @@
EAPI=4
-inherit autotools eutils flag-o-matic multilib
+inherit autotools eutils flag-o-matic multilib runtime-deps
DESCRIPTION="interactive process viewer"
HOMEPAGE="http://htop.sourceforge.net"
@@ -13,14 +13,27 @@
LICENSE="BSD GPL-2"
SLOT="0"
KEYWORDS="~alpha ~amd64 ~arm ~hppa ~ia64 ~mips ~ppc ~ppc64 ~sh ~sparc ~x86 ~x86-fbsd ~x86-freebsd ~amd64-linux ~x86-linux"
-IUSE="elibc_FreeBSD kernel_linux openvz unicode vserver"
+IUSE="elibc_FreeBSD kernel_linux openvz unicode vserver lsof"
-RDEPEND="sys-libs/ncurses[unicode?]"
+RDEPEND="sys-libs/ncurses[unicode?]
+ lsof? ( sys-process/lsof )"
DEPEND="${RDEPEND}"
DOCS=( ChangeLog README )
+runtime_deps_setup() {
+ #the body of this function is called only if the package hasn't been installed
+ #or if the state of the USE flags passed as args to check_runtime_deps changed
+ terminate_checking #continues pkg_setup
+ # if build-time flags have changed, exits ebuild processing otherwise
+ #
+ #terminate_checking could be of course implemented in check_runtime_deps
+ # in runtime-deps and it would call this body before calling terminate_checking
+}
+
pkg_setup() {
+ check_runtime_deps lsof #this should call the body above not installed already
+ #or if the state of USE=(-)lsof has changed
if use elibc_FreeBSD && ! [[ -f ${ROOT}/proc/stat && -f ${ROOT}/proc/meminfo ]]; then
eerror
eerror "htop needs /proc mounted to compile and work, to mount it type"
@@ -29,11 +42,6 @@
eerror
die "htop needs /proc mounted"
fi
-
- if ! has_version sys-process/lsof; then
- ewarn "To use lsof features in htop(what processes are accessing"
- ewarn "what files), you must have sys-process/lsof installed."
- fi
}
src_prepare() {
Looking forward to your opinions and comments.
Take care and keep making Gentoo as awesome as it was, is and always will be :)
Ondrej
Now that I look at it, the non-build-time flags wouldn’t have be to passed to check_runtime_deps, but could be declared in e.g. SUSE (suggested USE, no distro pun intended ;) ) .
I would still however suggest that check_runtime_deps has to be called somewhere in the ebuild explicitly, because different packages may want to call it in src_prepare or pkg_setup or wherever as I’m not sure if we’ll be able to find such a function that would suit all packages.
Another thing to consider in connection with the trouble of needing to specify check_runtime_deps in some function for a given package is the possible need to call it in different phases of building as documentation building may require a different phase of package building than just a runtime dep. This could be solved by passing the flags that may have to be considered in another phase of building to check_runtime_deps which would make it not terminate ebuild processing if the passed flags have changed state and continue ebuild processing to the next occurrence of check_runtime_deps (that would not disable processing of the not passed SUSE flags, just modify the termination clause).
Let me know what you think,
Ondrej