Zack Medico - Latest comments on Portage now supports building multiple packages in parallel

In response to: Portage now supports building multiple packages in parallel

Zac MEDICO [Member] — Thu, 14 Aug 2008 05:30:37 +0000

We might consider implementing more aggressive parallelization algorithms in the future.

Now I've found a way to make the parallelization algorithm more aggressive in some cases, without making the build order any less optimal. The patch is in svn r11405 and it will be included in >=sys-apps/portage-2.2_rc9.

In response to: Portage now supports building multiple packages in parallel

Zac MEDICO [Member] — Wed, 06 Aug 2008 22:32:19 +0000

a) after --resume, and
b) even after --keep-going resumes
since I do not see any parallel emerges anymore. :-(

In fact, IIRC after --resume the compressed output of parallel emerges would not even appear, so the new options probably did not survive the restart.

I can confirm that --resume does not properly preserve --jobs and --load-average options, so I will have that fixed in the next release (that will be 2.2_rc7).

I suspect that the --keep-going behavior that you've observed is due to the topology of the dependency graph, and the following algorithm:

The algorithm used to choose packages that will execute concurrently with other packages is as conservative as possible in the sense that a given package will not be executed if the subgraph composed of its direct and indirect dependencies contains any scheduled merges. By ensuring that the subgraph of deep dependencies is fully up to date in this way, potential problems are avoided which could be triggered by other build orders that are less optimal.

We might consider implementing more aggressive parallelization algorithms in the future. However, those who would consider choosing a more aggressive algorithm should ask themselves why they are in such a hurry. A more aggressive algorithm is inherently less optimal and therefore it only makes sense when there is some kind of deadline to meet. Even with a deadline, there are other techniques such as ccache and distcc which can help improve performance without sacrificing optimal build order.

In response to: Portage now supports building multiple packages in parallel

Hans Meine [Visitor] — Wed, 06 Aug 2008 13:06:36 +0000

I have been using these great new features on a multi-core machine during the last days, but it seems that portage has problems
a) after --resume, and
b) even after --keep-going resumes
since I do not see any parallel emerges anymore. :-(

In fact, IIRC after --resume the compressed output of parallel emerges would not even appear, so the new options probably did not survive the restart.

But it's a great feature; in the past I have always tried to mimic that manually, but that would be a lot of work, far from perfect, and sometimes compile some common dependencies more than once if not done carefully.

Thus, I am looking forward to see this feature become more mature.

In response to: Portage now supports building multiple packages in parallel

gringo [Visitor] — Wed, 30 Jul 2008 13:13:01 +0000

Not having a lot of time and not having interest are two different things. We get lots of feature requests and the list is really quite long. Me might get to that eventually be there are lots of other things with higher priority at this time. Obviously we don't always have time to implement every person's request in a timely manner.

i know and obviously i don

In response to: Portage now supports building multiple packages in parallel

Zac MEDICO [Member] — Wed, 30 Jul 2008 10:56:24 +0000

but my point is that if you schedule qt to build in parallel with other smaller package(s) and you build ooo once qt is done there will probably not be such big overload.

When the goal is to maximize the amount of work done in a given amount of time, it may be advantageous to have a slight over-abundance of jobs that are ready and waiting to execute as soon as the load average drops low enough for a new job to be scheduled.

Sorry to see you have no interest in implementing a vcs underneath portage.

Not having a lot of time and not having interest are two different things. We get lots of feature requests and the list is really quite long. We might get to that eventually but there are lots of other things with higher priority at this time. Obviously we don't always have time to implement every person's request in a timely manner.

In response to: Portage now supports building multiple packages in parallel

gringo [Visitor] — Wed, 30 Jul 2008 08:13:52 +0000

One of them is likely to cause the load average to increase to a point where additional jobs will not be scheduled. If for some reason they still happen to get scheduled simultaneously, having a --load-average setting in MAKEOPTS will cause make to react to the current load conditions and prevent your system from overloading itself too much.

well, yes, but my point is that if you schedule qt to build in parallel with other smaller package(s) and you build ooo once qt is done there will probably not be such big overload.

[toolchain]
class = portage.sets.shell.CommandOutputSet
command = /usr/local/bin/toolchain-package-set.sh

will try, thanks.
Don

In response to: Portage now supports building multiple packages in parallel

Zac MEDICO [Member] — Wed, 30 Jul 2008 04:18:07 +0000

I

In response to: Portage now supports building multiple packages in parallel

gringo [Visitor] — Tue, 29 Jul 2008 08:22:11 +0000

thanks for your input !

1- ok, will check this, i was quite sure there was already such feature but didn