Speeding up emerge depgraph calculation using PyPy3

If you used Gentoo for some time, you’ve probably noticed that emerge is getting slower and slower. Before I switched to SSD, my emerge could take even 10 minutes before it figured out what to do! Even now it’s pretty normal for the dependency calculation to take 2 minutes. Georgy Yakovlev recently tested PyPy3 on PPC64, and noticed a great speedup, apparently due to very poor optimization of CPython on that platform. I’ve attempted the same on amd64, and measured a 35% speedup nevertheless.

PyPy is an alternative implementation of Python that uses a JIT compiler to run Python code. JIT can achieve greater performance on computation-intensive tasks, at the cost of slower program startup. This means that it could be slower for some programs, and faster for others. In case of emerge dependency calculation, it’s definitely faster. A quick benchmark done using dev-perl/Dumbbench (great tool, by the way) shows, for today’s @world upgrade:

  • Python 3.9.0: 111.42 s ± 0.87 s (0.8%)
  • PyPy3.7 7.3.2: 72.30 s ± 0.23 s (0.3%)

dev-python/pypy3 is supported on Gentoo, on amd64, arm64, ppc64 and x86 targets. The interpreter itself takes quite a while to build (35­­–45 minutes on a modern Ryzen), so you may want to suggest emerge to grab dev-python/pypy3-exe-bin:

$ emerge -nv dev-python/pypy3 dev-python/pypy3-exe-bin

If you want to build it from source, it is recommended to grab dev-python/pypy first (possibly with dev-python/pypy-exe-bin for faster bootstrap), as building with PyPy itself is much faster:

# use prebuilt compiler for fast bootstrap
$ emerge -1v dev-python/pypy dev-python/pypy-exe-bin
# rebuild the interpreter
$ emerge -nv dev-python/pypy dev-python/pypy-exe
# build pypy3
$ emerge -nv dev-python/pypy3

Update 2020-10-07: Afterwards, you need to rebuild Portage and its dependencies with PyPy3 support enabled. The easiest way of doing it is to enable the PyPy3 target globally, and rebuilding relevant packages:

$ echo '*/* PYTHON_TARGETS: pypy3' >> /etc/portage/package.use
$ emerge -1vUD sys-apps/portage

Finally, you can use python-exec’s per-program configuration to use PyPy3 for emerge while continuing to use CPython for other programs:

$ echo pypy3 >> /etc/python-exec/emerge.conf
# yep, that's pypy3.7
$ emerge --info | head -1
Portage 3.0.7 (python 3.7.4-final-0, default/linux/amd64/17.1/desktop, gcc-9.3.0, glibc-2.32-r2, 5.8.12 x86_64)

9 Replies to “Speeding up emerge depgraph calculation using PyPy3”

  1. > Even now it’s pretty normal for the dependency calculation to take 2 minutes.

    Heh, I have tens of minutes for the world upgrade sometimes.

  2. Mh doing this in a stable system I still get python 3.7.8-final-0 from emerge –info, am I missing something?

    1. Actually, I’ve missed one important thing — you need to rebuild Portage with PYTHON_TARGETS: pypy3 (which also means some deps).

      1. Looks like on a default amd64 profile python_targets_pypy3 needs to be unmasked as well.

        # tail -n 4 /var/db/repos/gentoo/profiles/arch/amd64/use.stable.mask
        # Michał Górny (2014-03-30)
        # PyPy is unstable on this arch.

  3. pypy3 takes more than twice as much for me:

    # emerge –info|head -1
    Portage 3.0.8 (python 3.7.9-final-0, default/linux/amd64/17.0/desktop/plasma, gcc-9.3.0, glibc-2.31-r6, 5.4.65 x86_64)
    # time emerge -puDU world
    real 3m11.827s
    user 3m10.699s
    sys 0m1.002s

    # echo pypy3 >> /etc/python-exec/emerge.conf
    # emerge –info|head -1
    Portage 3.0.8 (python 3.6.9-final-0, default/linux/amd64/17.0/desktop/plasma, gcc-9.3.0, glibc-2.31-r6, 5.4.65 x86_64)
    # time emerge -puDU world
    real 7m52.081s
    user 7m50.541s
    sys 0m1.395s

    # equery l python
    * Searching for python …
    [IP-] [ ] dev-lang/python-2.7.18-r3:2.7
    [IP-] [ ] dev-lang/python-3.6.12:3.6/3.6m
    [IP-] [ ] dev-lang/python-3.7.9:3.7/3.7m
    # equery l pypy3
    * Searching for pypy3 …
    [IP-] [ ] dev-python/pypy3-7.3.2:0/pypy36-pp73

    did I miss something obvious?

    1. That’s curious. Maybe try with dumbbench (or at least get a few consecutive runs) to eliminate cold cache.

Leave a Reply

Your email address will not be published. Required fields are marked *