Month: March 2009

cmake vs autotools, a benchmark

Recently wesnoth got released and there is already an ebuild in portage.

Since upstream stated that autotools were being deprecated (actually some people come up to avoid that in the end) Mr_Bones crafted an ebuild using cmake.

Here some values:

build time for cmake -> 3.20m (take everything with at least some of variance)

build time for wesnoth using cmake -> 8.15m (again some more, some less depending on the runs)

build time for wesnoth using autotools -> 8.00m (again some less, some more depending on the runs)

my method is quite simple:
– first I fetch the sources and then build some times cmake and wesnoth 1.6a and use time to see how much it takes.
– then I use the older ebuild 1.4.7, remove the no-python from src_prepare since its unneeded, I call it 1.6a-r1 and then build some times that one as well.

Apparently cmake nearly add about 1/3 to the actual build time if you don’t have it already installed (and you shouldn’t) and compared to the autotools system it adds some time by itself.

In short people shouldn’t use cmake if autotools are available.

Benchmarks – howto

Here a bullet list about doing benchmarks

  • Reproducibility: your numbers worth nothing if nobody could reproduce them, so you have to give along them a script or a detailed description of what you did.
  • Statistics: outliers and other artefacts may screw your results, make sure your script is run enough times
  • Indices and values: if you want to proof something you need hard numbers possibly something everybody can understand easily and cannot be misunderstood: cpu cycles, time to complete, memory usage, are quite good, given you aren’t testing on particularly different architectures or systems.

Quite short, isn’t it? Still many people just state their values by inference (like “it ought to do 2x the syscalls thus should be twice slow”), or just tries to benchmark something that isn’t what you want to test (like “glxgears is slower now, mesa is slower at rendering complex scenes”) or have a quite different settings and configurations (think about having your application with a minimal configuration and the same one with a larger one)

Usually the best way to get a meaningful benchmark is preparing a script and give instructions about how to use (like which version of companion software are you using) and then provide the numbers (media with variance if you are inclined) and a summary of the system, this way others can play and try themselves. This is quite useful since the optimization you are working on may be great on gcc4.3 on PowerPC but be problematic on x86 with gcc-2.95.

Other times you want just people to compare something that is _quite_ influenced by the surrounding system or is annoying to setup, in those cases having a full system image is quite a boon, _everything_ could be the same. And given how easy is to use virtualization/emulation software nowadays it just take a bit of bandwidth.

Random ideas about changes, their value, their impact

During the last month we got many arguments about how fundamental was one or another change to portage and how this or that had been a life change for many users.

I’m talking about the whole discussion about glep54 and the following one.

When it was first proposed I found the glep lacking details and given that some people were vehemently against it I asked to have it updated. Since some other people found the idea of improving our support for live ebuilds as whole (not just the versioning) interesting I tried to gather ideas and put them all into a draft. The thing got quite big and covered many steps needed, most of them not detailed yet.

Basically the live template draft aims to have portage do the following:

– let you automatically merge from a specific live branch
– have emerge –fetch work as supposed introducing a phase just for that
– let you track what you installed and possibly re-install as is
– have a way to generate automatically snapshot ebuilds and binpkgs in a non ambiguous way
– some more (like have portage show you which revision and from which branch comes what you installed on emerge -upv)

Glep 54 just aim to make sure versioning is theoretically and thoroughly correct, no matter how crazy overlays, upstream and package maintainer could be. To make an example, right now if someone wants to provide a live ebuild he could prepare an ebuild with a large version value (say 9999) and assume nothing bigger would appear, version usually are small values like 1.2.3.
Now, nothing forbids to have a version that is, say, 10000, making invalid the assumption cat/pkg-9999 is the higher value. The glep54 comes with a solution for this issue, have a new prefix to state that he ebuild is live, thus has to be considered above all the others within it’s range. This change has to be folded within a EAPI change or even better to a file extension change as stated in glep55 since it’s as simple to state as radical: older portage, not knowing anything about what this prefix is, will issue a warning.

Installing from live sources is completely unsupported thus those ebuild are rightfully p.masked and their use is quite specific. The current amount of them is scarce: 90 on 26130 ebuilds are live.

So, if you got the question about if you’d approve this proposal and consider implementing that and all the changes due that would follow or just put it on hold till it would be more useful what would you do?

A pragmatic approach would either be either reject it since does pretty much nothing in the big scheme (0.003 is nearly nothing) a less pragmatic approach, due the tendency to thinker, would be just try to make it more useful so it could be of appeal to a wider interest…

Probably the same time would have been better spent on things that are more interesting like have a better way to state what a dependency is (if it is something I’ll need to link to, something that I need to run) to ease the life of our embedded arches (say ppc, arm, sh, mips) or to fully support multislot and multiabi so we can avoid emul-linux on amd64 and have ppc64 being able to run xlc w/out many annoyances *cough*, or start thinking about how to enable pgo as widely as possible since it appears to make the difference in certain situations (e.g. firefox).