cmake vs autotools, a benchmark

Recently wesnoth got released and there is already an ebuild in portage.

Since upstream stated that autotools were being deprecated (actually some people come up to avoid that in the end) Mr_Bones crafted an ebuild using cmake.

Here some values:

build time for cmake -> 3.20m (take everything with at least some of variance)

build time for wesnoth using cmake -> 8.15m (again some more, some less depending on the runs)

build time for wesnoth using autotools -> 8.00m (again some less, some more depending on the runs)

my method is quite simple:
– first I fetch the sources and then build some times cmake and wesnoth 1.6a and use time to see how much it takes.
– then I use the older ebuild 1.4.7, remove the no-python from src_prepare since its unneeded, I call it 1.6a-r1 and then build some times that one as well.

Apparently cmake nearly add about 1/3 to the actual build time if you don’t have it already installed (and you shouldn’t) and compared to the autotools system it adds some time by itself.

In short people shouldn’t use cmake if autotools are available.

25 thoughts on “cmake vs autotools, a benchmark”

  1. Just a quick question, why shouldn’t people have cmake installed already? It’s a major dependancy of KDE4

  2. Sweet words from cmake maintainer in gentoo comming.
    First i reduced the build time needed for cmake on half (this is not yet in tree but soon :]):
    Thu Mar 12 00:33:38 2009 >>> dev-util/cmake-2.6.3
    merge time: 1 minute and 3 seconds.
    Thu Mar 12 00:41:56 2009 >>> dev-util/cmake-2.6.3
    merge time: 41 seconds.

    For the wesnoth i am quite suprised. My tests were more of oposite. Mostly everything i write cmake build system for benchmaring ended faster than using autotools. I will try to test wesnoth when i am on reasonable net again 😛

  3. That’s great!

    The main issue I think is that there are plenty of ways to make more or less working build script in both cases, getting it right requires some skill.

    The wesnoth autoconf script is relatively fine (you can cut some tests by using pkgconfig instead of building/linking testcases for sdl related stuff), I hadn’t look at the (auto)make part yet.

    If in the end of this rambling we get a quicker (and saner) cmake and a quicker to build wesnoth I’ll be more than happy =)

  4. Personally I think most people tend to have CMake installed as it is not just KDE using it. I also don’t think a benchmark with the time to build CMake and the application is really valid, but I see where you are coming from.

    You have to remember that many projects aim to work on different platforms. In my experience this is much easier to do with CMake (especially when Windows is one of the platforms you care about). Also being able to use CPack, CTest and CDash can be great for a project. Additionally supporting Unix Makefiles along with MSVC, XCode, Eclipse style makefiles gives developers choice without maintaining multiple build systems.

    Without my Gentoo hat on many users couldn’t care less about compilation time. They want an installer/binary of some description anyway…

  5. My main gripe about cmake is that they just invented a perfectly square wheel, got lots of hype because kde people (that wrote complete bogus autotools as well apparently repeated themselves with cmake) used that as the next shiny thing instead of fixing what they got and yet another bogus thing got shoved to people.

    Now, having your application building on windows easily may be something you could consider nice or just an additional pain.
    Some people, me included probably, think windows is a bogus architecture that should disappear and don’t care a dime about having their code running on it and get pestered by users that have problems because of the operating system, others consider it the best way to get more people using it and possibly convince them to switch to something better.

    I could be happy that now it got more or less saner (at least your comments make me think this way).

    Still I found stupid running at adopting something, that is or was completely broken, just because was fashionable and gave you a sort of simpler way to get your code building on a strange platform whose only point to exist is lockin and an mass of users because of that.

    I welcome alternatives and fair competition, but I’m quite vocal against adoption because of hype or ignorance.

    PS: the benchmark methodology is in the post and I welcome people trying themselves and tell me how my results differ from others.

  6. Hi Luca,
    Long time no see!

    Although I’m not very fond of the CMake language, which reminds me of some old Basic language, it does have some very strong points as well. It has excellent support for cross-compiling and out of source builds which is crucial for me (embedded development) and it is very easy to create rules for configuring variations in builds.

    autotools also has very good support for out of source builds and good support for cross-compiling but with a couple of caveats. Some of the detection routines in autotools do not support cross-compiling requiring hacking the scripts and m4 has a much steeper learning curve than CMake making it difficult for new people on the team to get into modifying the build system.

    In conclusion, even though autotools is technically better the amount of time you have to spend with a new guy on the team to teach them m4 and the fact that most of the tests autotools perform aren’t needed for our project I have to say that CMake wins in our particular case.

    Personally I’m interested in trying out SCons but I hear their support for out of source builds is nonexistent.

  7. Hi David!

    From my experience the m4 part _now_ is simpler than the past since you have the autoconf documentation telling you everything you need to know, I heard lots of nice things about waf, you may give it a try.

    If you have a list about the routines not working in cross compiling mode I’d like to have it sent to upstream, they appear quite reactive nowadays (cmake makes them feeling an healthy competition pressure?).

  8. > Still I found stupid running at
    > adopting something, that is or was
    > completely broken, just because was
    > fashionable and gave you a sort of
    > simpler way to get your code building

    This is wrong.
    1) There is no reason to claim that cmake is completely broken. (in case you want to check, go to http://www.cdash.org/CDash/index.php?project=CMake to see how all the tests did today on all supported platforms 😉 )

    2) Using CMake was in no way “fashionable”. We were basically the first bigger free software project which switched to CMake. Maybe because we did that it has become fashionable 🙂

    3) I guess you meant “broken” as in “conceptionally broken”. Still I see absolutely no reason for this. What did we have with autotools ? A whole set of tools (autoconf, automake, autoheader, libtool, make, m4, shell), which had to work together, and in order to master them, you had to learn them all, with all their different languages (make syntax, shell scripts, m4 scripts, autoconf syntax, automake macros). Until today I never understood the need for libtool at all (the makefiles are generated, why no generated the proper rules for the tools ?). With CMake there is just one language you have to learn, this is a huge advantage. It just doesn’t make sense that getting your software to build is more complex than the software itself. Example: building “hello world” with cmake: add_executable(hello main.c)

    So, beside that CMake gives us much better cross platform support, it is very much worth it also if you don’t care about portability to other OSs, it just makes building your sources so much easier and straightforward.

    So, IMO “shoving” CMake onto people is better than continuing to torture people with autotools 😉


    > I welcome alternatives and fair
    > competition, but I’m quite vocal
    > against adoption because of hype or
    > ignorance.

    So there is no problem, at the time KDE switched to CMake there was absolutely no hype for CMake 🙂

    Alex, the KDE CMake guy

  9. Hmm, i just had a quick run comparing wesnoth trunk using autotools vs using cmake, here my data:

    1) using svn://svn.gna.org/svn/wesnoth/trunk revision 34598
    2) using cmake 2.6.3 (gentoo version -r1)
    3) gcc 4.3.3 on a Core2Quad 9300 with 4GB ram

    Commands used to measure:
    1) autotools:
    * exported svn dir (time for exporting *not* measured!)
    * afterwards this command (yes, not installing):
    time { ./autogen.sh && make -j5 ; }
    result:
    real 2m41.259s
    user 8m23.856s
    sys 0m40.550s

    2) cmake 2.6.3-r1:
    * referencing the plain svn checkout, no need to export first to really be using a clean dir
    * this command for compilation (yes, not installing, just as with autotools):
    time { cmake ../wesnoth/ && make -j5 ; }
    result:
    real 2m29.648s
    user 8m41.845s
    sys 0m38.259s

    Basically I’d say that autotools is slightly slower than cmake, just as expected, since autotools spend more time for the checking part then cmake. But that is basically all of the difference there is. Please keep in mind that I ran the commands only once while using my system in parallel just as usual. So those numbers are of course not 100% reliable, they should basically just show that there is not much of a real differen. To make sure that ccache had no impact on the results I ran “ccache -C” before each timing.

    The real difference between autotools and cmake is in maintainability and working on the Wesnoth recipe every now and then (me being the packager) I feel that cmake is easier to use than autotools, it is just cleaner.

    Both buils, no matter if cmake or autotools based were done *only* relying on the default settings. I don’t think they differ by much, though they might differ a little. This can also have some impact on time differences!

    In general I just recommend to every dev out there to simply use the tool he is most familiar with. Those general differences in compile time are not really showing much…

  10. Nils you tried to measure something but w/out running it some times that doesn’t tell much and you didn’t do exactly the same thing (also autotools supports out of tree building, cmake doesn’t support the other way round…).
    The statement about one language to rule them all is easy to counter by the fact you aren’t doing build scripts in C or C++ even if would be easier, you already know this language.

    Alex IIRC

    1- I just hope that cmake isn’t as broken as was (e.g. runtime paths, infinite loop dep resolutions or excessive build time come to mind), still when it got shoved upon kde users it was just that. broken.

    2- I completely agree

    3- You usually use different tools for different purposes, a toolbox isn’t composed by just one hammer. Certain design decision about autotools could be countered and discussed for sure. Having cmake doing stuff in different ways makes an healthy competition for people and that’s good in any possible way. About having thing simple, make is 0 lines to build hello world, but that doesn’t tell much as well =)

    IMHO the best route is always trying to make the wheel rounder, the current outcome shows that other routes may lead to results as good or even better given enough time to polish the rough edges ^^

  11. Ehm, sorry Luca, but I don’t understand your “also autotools supports out of tree building, cmake doesn’t support the other way round…”. For cmake there is no real difference if you run the command right in the checkout or outside. With autotools you are in fact not sure if an out of tree build will work.
    Like I already wrote, I just ran the commands once to get the numbers, but they are reasonably close to my “every day experience” so that I posted them anyway.

    And when it comes to “making the wheel rounder” I can just say that eventually you might end in a situation where plain old polishing does not help with the edges but where a new “prototype” has to be used to achive a really round wheel.
    From my experience at Wesnoth I know that autotools can get *really* ugly. We still have some problems left in our autotools build system (yes, even after this many years) and by now even esr gave up and said “I won’t touch this mess no more”. Okay, he favors the scons recipe, I favor the cmake recipe, but that is a different topic.
    From my experience eg from a “medium sized” project at university I can just state that using cmake is really nice for this stuff since it is easy to write the initial script and even when it grows to creating some lib, building various targets with various options and stuff like this it always worked well and my head was never smoking as it is in general when I have to work on autotools stuff.

    Even if there is a build time difference of maybe 1% in favor of autotools (I have not observed this so far!) I as dev prefer the system that is easier to use for me, and that is ATM cmake.

  12. I’ll have a look again on the autotools in wesnoth again. As I said before having more models of wheel could be good in the end, jumping on different ones just because you do not know how to use the former one (and then you end up redoing the same errors mixed with newer ones) is the problem.

    Now cmake got more or less a less annoying syntax, it doesn’t take ages to build (yet I have to check with ppc) and some glaring issues got addressed, so I shouldn’t complain that much, hopefully.

    Still I don’t see how harder is the autotools set compared to cmake IF it is supposed to give you the same results (dependency tracking/resolution a la make, template language a la m4, running commands a la bash).

    Sadly for the whole toolchain there are sometimes perfectly good (and hidden) reasons why certain stuff is implemented that way and not understanding them while trying to write a replacement could just lead to solutions that may sooner or break later (e.g. playing with rpath).

  13. As for cross-compiling, I’d just quote the FindRuby.cmake module:

    FIND_PATH(RUBY_INCLUDE_PATH
    NAMES ruby.h
    PATHS
    ${RUBY_ARCH_DIR}
    /usr/lib/ruby/1.8/i586-linux-gnu/ )

    It makes absolute sense to hardcode that last path, does it? And where is the Ruby cross-compilation support?

    If you mean you can override everything to force the thing to fake cross-compile support, GNU make based buildsystems are just as fine.

  14. Hi Luca

    I couldn’t really believe my eyes when you went on to say that if you add the build time of cmake, it’s slower than autotools. Oh, and then the 15 seconds of the build time… when I look at my university or the SE jobs I’ve done in industry, I’ve gotten to learn a number of things:

    1) cmake is easy to deploy. Installing it is a onetime operation for a developer, so who cares. But cmake makes it easy as it has easy to deploy packages readily available. Installing autotools on linux is kinda easy as they’re mostly already available, but on windows it’s harder. I know, it’s a one time operation, but it makes a difference. It is especially easy to deploy because it supports project files for different IDE’s. That’s really a biiiig plus for many people.

    2) cmake is a lot easier to learn. That’s a subjective thing I know, but I tried learning autotools but I always failed because of the awquard syntax and the complexity of the system involved. It has several tools and for a beginner it’s not easy to see through this. cmake is one language and a pretty simple one at that. It’s not without flaws, but my first attempt on cmake succeeded, my first attempt at autotools failed. Biiiig difference.

    3) Only one langugate. You say that you already have to know several languages (programming, scripting etc) where is the difference? Well, why learning several if learning one suits? You don’t answer that question at all

    4) cross platform support. It’s not perfect in cmake, but it can be made work quite easily as I’ve already done. Flameeyes has an example above of a broken build system and concludes from this that cmake cross-compilation is kinda broken? Well, how many broken autotools buildsystems are there? Should I conclude autotools is completely broken? Not.

    Yes… cmake is not perfect and has quite a number of flaws. Flameeyes has pointed out a number of them even though I don’t agree to all of them.

    But concluding that cmake is worse because… well installing it takes more time? This is a no-argument

  15. As for cross-compiling, I’d just point you to all the non-ruby modules.

    P.S. as this is free software should you have a problem, you could a) fix it, or b) report the problem.
    I’ve seen repeatedly how the cmake folks are very responsive to either.

    “Still I don’t see how harder is the autotools set compared to cmake IF it is supposed to give you the same results (dependency tracking/resolution a la make, template language a la m4, running commands a la bash).”
    Because you haven’t looked.

    I’m neither programmer nor build tools guru but have had to mess around with Makefiles, Makefile.ams, Makefile.in and my personal _favorite_ the libtool file. It is orders of magnitude easier to beat a CMakeLists.txt file into submission.

    You may point out that the cmakelists file doesn’t provide all the functionality of the other, to which I’d reply: “So?”

    The point is, with cmake, I’ve never needed to do anything outside of this file. And if it takes care of everything which autotools does, not forcing me to figure out how it works, then good.

    If I’m only compelled to learn about autotools features when they break (and I am) then cmake is automatically better by virtue of the fact that I have never found it necessary to learn their analogues when using cmake.

  16. Find${PKG}.cmake modules bundled with cmake are mentioned (in Documentation) to be simple and “quickstart”, rather than reliable robust solutions for production – so your argument is quite invalid here, Diego.
    What’s the most important in CMake – it’s much harder to write bad cmake script rather than bad autotools/automake/libtool script.
    While packaging KDE4 for Gentoo (it will be two releases now, and live KDE releases in overlay for a couple of months), I’ve seen enough to state that CMake is in any way better to get things done. If autotools are that great, easy and all, then why don’t we see yet mysql and openldap split ebuilds in tree?
    Why does it take weeks to covert libmysqld from static to shared library?
    Why X11 team is currently fighting with xcb ABI change consequences caused by broken by design libtool files? (.la)
    And why KDE Gentoo team can easily split/merge packages in any possible way anytime?
    And why we can fix or workaround most buildsystem issues in a few minutes without exchanging unnecessary emails?
    Autotools are just an old relic kept for backward compatibility and should die as soon as possible.
    Also having over 500KiB of buildsystem scripts for simple hello world-size project is pretty ridiculous, don’t you think?

  17. Benjamin
    1) cmake is easy to deploy.

    last time I checked it depends on:
    net-misc/curl
    dev-libs/expat
    dev-libs/libxml2
    dev-libs/xmlrpc-c
    a C++ compiler

    I do not chain back deps, I’ll let you do this exercise by yourself

    autoconf
    >=sys-apps/texinfo-4.3
    >=sys-devel/m4-1.4.6
    dev-lang/perl

    automake
    sys-devel/autoconf

    libtool
    sys-devel/autoconf
    sys-devel/automake

    implicitly bash and a proper C compiler also.

    sooo basically… perl and bash.

    now, on windows maybe the deps are different, maybe you just install the executable (as you can do by using cygwin or msys as well)

    2) cmake is a lot easier to learn.
    Probably is better documented, and more people is giving you better examples for start. Autotools main fault is the fact since few years ago the documentation was quite poor and scattered, add to it the fact nobody is stopping you to abuse bash or make instead of using the features already provided and you see why people perceive autotools (automake in particular) that bad.

    3) Why learning several if learning one suits?
    I like to have a toolbox, not a single big hammer. You see the tendency to have domain specific languages to solve better different issues. In this case makes little difference taking a some small single domain languages and lump them together probably.

    4) cross platform support.
    You cannot claim it is better than the autotools one

    Looks like it again cmake strong point is windows. Everybody loves windows.

  18. illogic-al

    – libtool is a tool made to hide away gory details certain systems had, pkgconfig nowadays solve most of the problems in a nicer way indeed. Still I wonder why you had to fiddle with .la files…

    – I have looked at cmake the first time kde4 started using it and my eye bleeded. the syntax was horrible. I looked it again now and the syntax got at least nicer but still it isn’t any better than autoconf + automake.

    – For a bare autotools project you have 2 files, Makefile.am and configure.ac I please you to take your time and check the new documentation about autoconf or look at the Diego’s articles about it.

  19. Maciej Mrozowski

    > Find${PKG}.cmake modules bundled with cmake are mentioned (in Documentation) to be simple and “quickstart”, rather > than reliable robust solutions for production – so your argument is quite invalid here, Diego.

    Basically you are telling me that you have to redo the equivalent of your .m4 files provided since they are bogus?

    > What’s the most important in CMake – it’s much harder to write bad cmake script rather than bad autotools/automake
    > /libtool script.

    Seems you autocountered this statement telling you have to rewrite basic checks since they aren’t provided by default…

    > If autotools are that great, easy and all, then why don’t we see yet mysql and openldap split ebuilds in tree?

    Because upstream want things differently and you try to address upstream in order to have some agreement before gutting their application randomly mostly. Then, as Diego pointed for mysql, you may have other issues regarding
    how things are built that are to be fixed.

    > Why does it take weeks to covert libmysqld from static to shared library?

    Same reason as before.

    > Why X11 team is currently fighting with xcb ABI change consequences caused by broken by design libtool files? (.la)

    Because nothing is perfect? Because libtool isn’t that necessary anymore and could/should thinned a lot on systems that can do without it IMHO.

    > Also having over 500KiB of buildsystem scripts for simple hello world-size project is pretty ridiculous, don’t
    > you think?

    dev-util/cmake-2.6.3: 429 files, 25 non-files, 24923.813 KB

    Absolutely.

  20. Sorry but if they are just a “quickstart”, it is a *seriously stupid* way to provide support do buildsystem, because then you get 10 projects with at least 8 different modules to do the same bloody thing. Sure there is the same problem with autotools, but it really does not solve anything.

    Why, why, why, why. Why X11 team is doing that? Because they haven’t followed my advice I gave more than an year ago, maybe even two years ago. Yes, it might sound strange to you but it happens that following the advice of somebody that knows some topic better than you can usually resolve problem. Why does it take weeks to convert mysql? _Because they bloody idiots in MySQL did not write autotools!_ they wrote crap based on autotools, which is the same that KDE did with 3.5 series.

    Sorry but autotools are just an old relic as much as the presence of commandline is: it’s always said by people who like shiny stuff and just that. Sure I’d love if we were able to replace autotools but cmake is not up on par yet.

    By the way: why did KDE 4.2 require akonadi to install umbrello? Wasn’t that a buildsystem issue after all?

    illogic-al: you should never have messed with the libtool script in the first place; nor with Makefile.in; if you did it means that either you found bugs (which would be the same as messing with cmake original source code), or you were doing something seriously wrong. As for the problem being simply FindRuby, have you checked FindJNI yet?

    Oh yeah but cmake just provides a basis, I forgot, all the rest of the world has to reinvent stuff, because the _one big huge issue with autotools_ (the presence of N different variations of m4 files) is left untouched, for the nostalgic I guess.

    What autotools seriously have lacked for a long time, and caused a critical mass of crappy buildsystem to spawn, is documentation. CMake seems to have had Kitware to push through by writing the buildsystems or instructing people to in the first place. The fact that KDE is taken as an example is probably the only reason why it can be used.

    Point is: KDE 3 had a seriously crappy (and very flawed) build system, and lots of people copied from that instead of using autotools properly. Which is why we got so many broken buildsystems out there. Fixing an issue in a properly-written buildsystem usually take me five minutes; ask Lennart, for whom I fixed a pulseaudio build regression before he took the plane for BOSSA.

    Only practical reason to choose CMake over autotools is Windows support. And I can accept that well, what I find stupid is to say that it’s _perfect_ and it has not to change. That’s simply stupid.

    Oh by the way the FindRuby module which is broken I reported basically immediately after I first tried CMake since I wrote a better one than the one shipped at the time (which was simply not working); the report was ignored until a kde.org guy submitted a different version; why? why? why?

    Yeah sure, keep on f*cking trying.

  21. Diego, maybe it’s just the way (somewhat offensive maybe?) you presented your point of view, you know – there are some people with “offensive” attitude on gentoo-dev and you imagine how reception looks like.

    >> Also having over 500KiB of buildsystem
    >> scripts for simple hello world-size project
    >> is pretty ridiculous, don’t you think?

    > dev-util/cmake-2.6.3: 429 files, 25 non-files, 24923.813 KB
    > Absolutely.

    If it wasn’t clear enough – I meant buildsystem files distributed with *project* itself – so you should compare several CMakeLists.txt files with shipped .cmake modules with aclocal.*, Makefile.* and configure* scripts (and libtool files of course as well).
    But following your logic, equery list automake shows four different versions slotted, what about this?

    I guess your (Diego and Luca) main objections are:
    – presence of C++ compiler
    – dependency on libstdc++
    – not robust bundled cmake modules
    – cmake depending on xmlrpc-c can be fixed on Gentoo side if it really hurts you that much
    – umbrello depending on kdepimlibs and thus akonadi was not upstream fault but Gentoo devs layness and I fixed it already in case you missed
    – RPATH issues – well, those are just bugs that should be fixed – Niemand ist perfekt, right?

    So it just leaves us with general C++ rant and insufficient cmake bundled modules (valid point imho).

  22. You can add up autotools + actual autotools code (*.ac *.am *.m4) or just the generated code (Makefile.in configure) and maybe make. You may do the same with cmake as well. If you start thinking about how big something is I’d say right now cmake is about 3-4 slots of autotools…

    The fact cmake is built on c++ just makes it having one more dep, nothing more, nothing less.
    the fact it’s about 5 times the size of the autotools… Again it’s just something you may or may not consider, you claim about 500k for the autotools generated files + autotools code + m4 bundles for a simple project, even if a simple project won’t need additional m4…

    You may love your new shiny toy and you may mock kids using vintage ones or you may just try find where a tool is better and where you can improve the other and try to use the one that works best for you for a specific task.

    For wesnoth apparently autotools at least here works better and since autotools is what comes from the rest of the toolchain doesn’t add up. Since I do not use cmake it does add up for wesnoth and it takes some more to get the same program. In the end I’d like to have wesnoth built by autotools (even better built by bash + make as that’s what autotools gives you) instead of cmake since it works better for me.

    1. You don’t know what you are talking about. cmake and autotools are the very same regarding complexity. The featureset is non-overlapping. One let you have native builds on alien systems (windows) using their tools with ease. The other let you target different systems with ease.

      All depends on what you want.

      You might consider using cmake if your project is C++ and Qt, since all you need to get cmake is there, and your target developer works on windows but also on some other platforms.

      You want to use autotools if your project needs to be cross compiled and if your system doesn’t support the features cmake needs (autotools basically requires plain C)

Leave a Reply

Your email address will not be published.