Interesting approach :)
It is a smart way to protect users against unwanted overrides, which leads to increased QA.

Good job :)

Thank you for all your excellent work, very much appreciated by all your loyal fans, the Gentoo users.

Great stuff. And asking the user if (s)he would like an obsolete package removed from the world file would be excellent.

We might consider implementing more aggressive parallelization algorithms in the future.

Now I've found a way to make the parallelization algorithm more aggressive in some cases, without making the build order any less optimal. The patch is in svn r11405 and it will be included in >=sys-apps/portage-2.2_rc9.

a) after --resume, and
b) even after --keep-going resumes
since I do not see any parallel emerges anymore. :-(

In fact, IIRC after --resume the compressed output of parallel emerges would not even appear, so the new options probably did not survive the restart.

I can confirm that --resume does not properly preserve --jobs and --load-average options, so I will have that fixed in the next release (that will be 2.2_rc7).

I suspect that the --keep-going behavior that you've observed is due to the topology of the dependency graph, and the following algorithm:

The algorithm used to choose packages that will execute concurrently with other packages is as conservative as possible in the sense that a given package will not be executed if the subgraph composed of its direct and indirect dependencies contains any scheduled merges. By ensuring that the subgraph of deep dependencies is fully up to date in this way, potential problems are avoided which could be triggered by other build orders that are less optimal.

We might consider implementing more aggressive parallelization algorithms in the future. However, those who would consider choosing a more aggressive algorithm should ask themselves why they are in such a hurry. A more aggressive algorithm is inherently less optimal and therefore it only makes sense when there is some kind of deadline to meet. Even with a deadline, there are other techniques such as ccache and distcc which can help improve performance without sacrificing optimal build order.

I have been using these great new features on a multi-core machine during the last days, but it seems that portage has problems
a) after --resume, and
b) even after --keep-going resumes
since I do not see any parallel emerges anymore. :-(

In fact, IIRC after --resume the compressed output of parallel emerges would not even appear, so the new options probably did not survive the restart.

But it's a great feature; in the past I have always tried to mimic that manually, but that would be a lot of work, far from perfect, and sometimes compile some common dependencies more than once if not done carefully.

Thus, I am looking forward to see this feature become more mature.

Should note that @live-rebuild may catch packages that inherit those eclasses, but aren't really "live" packages

It might be good to add support for RESTRICT=live (with the reverse meaning of what was originally suggested in bug 233589) since it will also be useful for repoman's LIVEVCS.stable check. I've just sent an RFC email to the gentoo-dev mailing list.

Well, the @live-rebuild set is really only intended as an interim solution until bug 182028 has been implemented in a future EAPI. See bug 233589 for an alternative way that we might implement @live-rebuild using a new RESTRICT value (that doesn't require an EAPI bump).

Some ebuilds are snapshot ebuilds that point to a certain fixed revision, for example:

ESVN_REPO_URI="svn://mySvnServer/${PN}@{${PV}}"

This will never need svn updating, but it is pulled in anyway to @live-rebuild. The solution you have is so elegent, though, that it may not be worth it to fix this outside case.

However one things that would be nice (maybe worth a bug in bugzilla?) would be to have a @live-update (or something) that checks to see if upstream repo has been updated since last merge and in that case updates it, leaving the rest to be. That way you can run @live-rebuild if you want to rebuild all of them and @live-update if you only want the updated.

We already have plans to implement this in a future EAPI. See bug 182028.

This will be pretty neat features.
However one things that would be nice (maybe worth a bug in bugzilla?) would be to have a @live-update (or something) that checks to see if upstream repo has been updated since last merge and in that case updates it, leaving the rest to be. That way you can run @live-rebuild if you want to rebuild all of them and @live-update if you only want the updated.

This becouse on some system (like one of mines where I have to get a big chunk of xorg from git for some upstream bughunting) there could be a lot of packages inheriting git that are not very often are updated upstream and there is no need to rebuild.
I know there are scripts to do this for you but it would be nice to have them all at one place.

I'm not familiar with the Kuroo or portato codebases so I'm not sure how much effort it would take for them to add support for this. Depending on how they interface with portage, they might not have to change anything.

\o/ I'm happy. We needed USE dependencies a long time ago... Very, very thanks!

A question: which portage frontends will support this dependencies? Kuroo/portato will support it?

Wow, this is pretty neat! I'm really looking forward to test it!

This is awesome. Many of us have wanted this for a very long time.

Wow!
Great work... lately so much progress in the Portage. Package manager USE dependencies is sth that will really improve the Gentoo.

Not having a lot of time and not having interest are two different things. We get lots of feature requests and the list is really quite long. Me might get to that eventually be there are lots of other things with higher priority at this time. Obviously we don't always have time to implement every person's request in a timely manner.

i know and obviously i don

but my point is that if you schedule qt to build in parallel with other smaller package(s) and you build ooo once qt is done there will probably not be such big overload.

When the goal is to maximize the amount of work done in a given amount of time, it may be advantageous to have a slight over-abundance of jobs that are ready and waiting to execute as soon as the load average drops low enough for a new job to be scheduled.

Sorry to see you have no interest in implementing a vcs underneath portage.

Not having a lot of time and not having interest are two different things. We get lots of feature requests and the list is really quite long. We might get to that eventually but there are lots of other things with higher priority at this time. Obviously we don't always have time to implement every person's request in a timely manner.

One of them is likely to cause the load average to increase to a point where additional jobs will not be scheduled. If for some reason they still happen to get scheduled simultaneously, having a --load-average setting in MAKEOPTS will cause make to react to the current load conditions and prevent your system from overloading itself too much.

well, yes, but my point is that if you schedule qt to build in parallel with other smaller package(s) and you build ooo once qt is done there will probably not be such big overload.

[toolchain]
class = portage.sets.shell.CommandOutputSet
command = /usr/local/bin/toolchain-package-set.sh

will try, thanks.
Don

I