Speeding up emerge depgraph calculation using PyPy3

WARNING: Some of the respondents were< not able to reproduce my results. It is possible that this dependent on the hardware or even a specific emerge state. Please do not rely on my claims that PyPy3 runs faster, and verify it on your system before switching permanently.

If you used Gentoo for some time, you’ve probably noticed that emerge is getting slower and slower. Before I switched to SSD, my emerge could take even 10 minutes before it figured out what to do! Even now it’s pretty normal for the dependency calculation to take 2 minutes. Georgy Yakovlev recently tested PyPy3 on PPC64, and noticed a great speedup, apparently due to very poor optimization of CPython on that platform. I’ve attempted the same on amd64, and measured a 35% speedup nevertheless.
Continue reading “Speeding up emerge depgraph calculation using PyPy3”

Sleeping and waking up

Time to write something more personal, for a change. I find it somewhat curious how my sleeping habits have changed over the years, as well as the level of sophistication of the way I have been waking up. Let me run a short recollection of how a teenager who tried to squeeze every single minute out of (morning) sleep turned into a young man who tried to optimize his sleep, and finally into a man who does not mind waking up much earlier than strictly necessary.
Continue reading “Sleeping and waking up”

New vulnerability fixes in Python 2.7 (and PyPy)

As you probably know (and aren’t necessarily happy about it), Gentoo is actively working on eliminating Python 2.7 support from packages until end of 2020. Nevertheless, we are going to keep the Python 2.7 interpreter much longer because of some build-time dependencies. While we do that, we consider it important to keep Python 2.7 as secure as possible.

The last Python 2.7 release was in April 2020. Since then, at least Gentoo and Fedora have backported CVE-2019-20907 (infinite loop in tarfile) fix to it, mostly because the patch from Python 3 applied cleanly to Python 2.7. I’ve indicated that Python 2.7 may contain more vulnerabilities, and two days ago I’ve finally gotten to audit it properly as part of bumping PyPy.

The result is matching two more vulnerabilities that were discovered in Python 3.6, and backporting fixes for them: CVE-2020-8492 (ReDoS in basic HTTP auth handling) and bpo-39603 (header injection via HTTP method). I am pleased to announce that Gentoo is probably the first distribution to address these issues, and our Python 2.7.18-r2 should not contain any known vulnerabilities. Of course, this doesn’t mean it’s safe from undiscovered problems.

While at it, I’ve also audited PyPy. Sadly, all current versions of PyPy2.7 were vulnerable to all aforementioned issues, plus partially to CVE-2019-18348 (header injection via hostname, fixed in 2.7.18). PyPy3.6 was even worse, missing 12 fixes from CPython 3.6. All these issues were fixed in Mercurial now, and should be part of 7.3.2 final.

New tools to help with package cleanups

Did you ever have had Croaker shout at you because you removed an old version that just happened to be still required by some other package? Did you have to run your cleanups past (slow-ish) CI just to avoid that? If you did, I have just released app-portage/mgorny-dev-scripts, version 6 that has a tool just for that!
Continue reading “New tools to help with package cleanups”

Is an umbrella organization a good choice for Gentoo?

The talk of joining an umbrella organization and disbanding the Gentoo Foundation (GF) has been recurring over the last years. To the best of my knowledge, even some unofficial talks have been had earlier. However, so far our major obstacle for joining one was the bad standing of the Gentoo Foundation with the IRS. Now that that is hopefully out of the way, we can start actively working towards it.

But why would we want to join an umbrella in the first place? Isn’t having our own dedicated Foundation better? I believe that an umbrella is better for three reasons:

  1. Long-term sustainability. A dedicated professional entity that supports multiple projects has better chances than a small body run by volunteers from the developer community.
  2. Cost efficiency. Less money spent on organizational support, more money for what really matters to Gentoo.
  3. Added value. Umbrellas can offer us services and status that we currently haven’t been able to achieve.

I’ll expand on all three points.
Continue reading “Is an umbrella organization a good choice for Gentoo?”