{"id":1240,"date":"2020-10-06T10:01:05","date_gmt":"2020-10-06T08:01:05","guid":{"rendered":"http:\/\/blogs.gentoo.org\/mgorny\/?p=1240"},"modified":"2021-01-03T09:26:57","modified_gmt":"2021-01-03T08:26:57","slug":"speeding-up-emerge-depgraph-calculation-using-pypy3","status":"publish","type":"post","link":"https:\/\/blogs.gentoo.org\/mgorny\/2020\/10\/06\/speeding-up-emerge-depgraph-calculation-using-pypy3\/","title":{"rendered":"Speeding up emerge depgraph calculation using PyPy3"},"content":{"rendered":"<p><ins datetime=\"2021-01-03T08:24:52+00:00\">WARNING: Some of the respondents were&lt; not able to reproduce my results. It is possible that this dependent on the hardware or even a specific emerge state. Please do not rely on my claims that PyPy3 runs faster, and verify it on your system before switching permanently.<\/ins><\/p>\n<p>If you used Gentoo for some time, you&#8217;ve probably noticed that emerge is getting slower and slower. Before I switched to SSD, my emerge could take even 10 minutes before it figured out what to do! Even now it&#8217;s pretty normal for the dependency calculation to take 2 minutes. Georgy Yakovlev recently tested <a rel='external' href='https:\/\/www.pypy.org\/'>PyPy3<\/a> on PPC64, and noticed a great speedup, apparently due to very poor optimization of CPython on that platform. I&#8217;ve attempted the same on amd64, and measured a 35% speedup nevertheless.<br \/>\n<!--more--><br \/>\nPyPy is an alternative implementation of Python that uses a <abbr title=\"Just-in-Time\">JIT<\/abbr> compiler to run Python code. JIT can achieve greater performance on computation-intensive tasks, at the cost of slower program startup. This means that it could be slower for some programs, and faster for others. In case of emerge dependency calculation, it&#8217;s definitely faster. A quick benchmark done using <a rel=\"external\" href=\"https:\/\/packages.gentoo.org\/packages\/dev-perl\/Dumbbench\">dev-perl\/Dumbbench<\/a> (great tool, by the way) shows, for today&#8217;s @world upgrade:<\/p>\n<ul>\n<li>Python 3.9.0: 111.42 s \u00b1 0.87 s (0.8%)<\/li>\n<li>PyPy3.7 7.3.2: 72.30 s \u00b1 0.23 s (0.3%)<\/li>\n<\/ul>\n<p><a rel=\"external\" href=\"https:\/\/packages.gentoo.org\/packages\/dev-python\/pypy3\">dev-python\/pypy3<\/a> is supported on Gentoo, on amd64, arm64, ppc64 and x86 targets. The interpreter itself takes quite a while to build (35\u00ad\u00ad\u201345 minutes on a modern Ryzen), so you may want to suggest emerge to grab <a rel=\"external\" href=\"https:\/\/packages.gentoo.org\/packages\/dev-python\/pypy3-exe-bin\">dev-python\/pypy3-exe-bin<\/a>:<\/p>\n<pre><code>$ emerge -nv dev-python\/pypy3 dev-python\/pypy3-exe-bin<\/code><\/pre>\n<p>If you want to build it from source, it is recommended to grab <a rel=\"external\" href=\"https:\/\/packages.gentoo.org\/packages\/dev-python\/pypy\">dev-python\/pypy<\/a> first (possibly with <a rel=\"external\" href=\"https:\/\/packages.gentoo.org\/packages\/dev-python\/pypy-exe-bin\">dev-python\/pypy-exe-bin<\/a> for faster bootstrap), as building with PyPy itself is much faster:<\/p>\n<pre><code># use prebuilt compiler for fast bootstrap\r\n$ emerge -1v dev-python\/pypy dev-python\/pypy-exe-bin\r\n# rebuild the interpreter\r\n$ emerge -nv dev-python\/pypy dev-python\/pypy-exe\r\n# build pypy3\r\n$ emerge -nv dev-python\/pypy3<\/code><\/pre>\n<p><ins>Update 2020-10-07<\/ins>: Afterwards, you need to rebuild Portage and its dependencies with PyPy3 support enabled. The easiest way of doing it is to enable the PyPy3 target globally, and rebuilding relevant packages:<\/p>\n<pre><code>$ echo '*\/* PYTHON_TARGETS: pypy3' &gt;&gt; \/etc\/portage\/package.use\r\n$ emerge -1vUD sys-apps\/portage<\/code><\/pre>\n<p>Finally, you can use python-exec&#8217;s per-program configuration to use PyPy3 for emerge while continuing to use CPython for other programs:<\/p>\n<pre><code>$ echo pypy3 &gt;&gt; \/etc\/python-exec\/emerge.conf\r\n# yep, that's pypy3.7\r\n$ emerge --info | head -1\r\nPortage 3.0.7 (python 3.7.4-final-0, default\/linux\/amd64\/17.1\/desktop, gcc-9.3.0, glibc-2.32-r2, 5.8.12 x86_64)<\/code><\/pre>\n","protected":false},"excerpt":{"rendered":"<p>WARNING: Some of the respondents were&lt; not able to reproduce my results. It is possible that this dependent on the hardware or even a specific emerge state. Please do not rely on my claims that PyPy3 runs faster, and verify it on your system before switching permanently. If you used Gentoo for some time, you&#8217;ve &hellip; <a href=\"https:\/\/blogs.gentoo.org\/mgorny\/2020\/10\/06\/speeding-up-emerge-depgraph-calculation-using-pypy3\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Speeding up emerge depgraph calculation using PyPy3&#8221;<\/span><\/a><\/p>\n","protected":false},"author":137,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"jetpack_publicize_message":"","jetpack_is_tweetstorm":false,"jetpack_publicize_feature_enabled":true},"categories":[3],"tags":[],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/posts\/1240"}],"collection":[{"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/users\/137"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/comments?post=1240"}],"version-history":[{"count":8,"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/posts\/1240\/revisions"}],"predecessor-version":[{"id":1291,"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/posts\/1240\/revisions\/1291"}],"wp:attachment":[{"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/media?parent=1240"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/categories?post=1240"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.gentoo.org\/mgorny\/wp-json\/wp\/v2\/tags?post=1240"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}