Category: Gentoo

DISTUTILS_USE_SETUPTOOLS, QA spam and… more QA spam?

Update: the information provided in this post is out of date. As of today, Python 3.7 is no longer relevant from DISTUTILS_USE_SETUPTOOLS perspective, and ‘rdepend’ is no longer valid when when entry points are used.

I suppose that most of the Gentoo developers have seen at least one of the ‘uses a probably incorrect DISTUTILS_USE_SETUPTOOLS value’ bugs by now. Over 350 have been filed so far, and new ones are filed practically daily. The truth is, I’ve never intended for this QA check to result in bugs being filed against packages, and certainly not that many bugs.

This is not an important problem to be fixed immediately. The vast majority of Python packages depend on setuptools at build time (this is why the build-time dependency is the eclass’ default), and being able to unmerge setuptools is not a likely scenario. The underlying idea was that the QA check would make it easier to update DISTUTILS_USE_SETUPTOOLS when bumping packages.

Nobody has asked me for my opinion, and now we have hundreds of bugs that are not very helpful. In fact, the effort involved in going through all the bugmail, updating packages and closing the bugs greatly exceeds the negligible gain. Nevertheless, some people actually did it. I have bad news for them: setuptools upstream has changed entry point mechanism, and most of the values will have to change again. Let me elaborate on that.
Continue reading “DISTUTILS_USE_SETUPTOOLS, QA spam and… more QA spam?”

Speeding up emerge depgraph calculation using PyPy3

WARNING: Some of the respondents were< not able to reproduce my results. It is possible that this dependent on the hardware or even a specific emerge state. Please do not rely on my claims that PyPy3 runs faster, and verify it on your system before switching permanently.

If you used Gentoo for some time, you’ve probably noticed that emerge is getting slower and slower. Before I switched to SSD, my emerge could take even 10 minutes before it figured out what to do! Even now it’s pretty normal for the dependency calculation to take 2 minutes. Georgy Yakovlev recently tested PyPy3 on PPC64, and noticed a great speedup, apparently due to very poor optimization of CPython on that platform. I’ve attempted the same on amd64, and measured a 35% speedup nevertheless.
Continue reading “Speeding up emerge depgraph calculation using PyPy3”

New vulnerability fixes in Python 2.7 (and PyPy)

As you probably know (and aren’t necessarily happy about it), Gentoo is actively working on eliminating Python 2.7 support from packages until end of 2020. Nevertheless, we are going to keep the Python 2.7 interpreter much longer because of some build-time dependencies. While we do that, we consider it important to keep Python 2.7 as secure as possible.

The last Python 2.7 release was in April 2020. Since then, at least Gentoo and Fedora have backported CVE-2019-20907 (infinite loop in tarfile) fix to it, mostly because the patch from Python 3 applied cleanly to Python 2.7. I’ve indicated that Python 2.7 may contain more vulnerabilities, and two days ago I’ve finally gotten to audit it properly as part of bumping PyPy.

The result is matching two more vulnerabilities that were discovered in Python 3.6, and backporting fixes for them: CVE-2020-8492 (ReDoS in basic HTTP auth handling) and bpo-39603 (header injection via HTTP method). I am pleased to announce that Gentoo is probably the first distribution to address these issues, and our Python 2.7.18-r2 should not contain any known vulnerabilities. Of course, this doesn’t mean it’s safe from undiscovered problems.

While at it, I’ve also audited PyPy. Sadly, all current versions of PyPy2.7 were vulnerable to all aforementioned issues, plus partially to CVE-2019-18348 (header injection via hostname, fixed in 2.7.18). PyPy3.6 was even worse, missing 12 fixes from CPython 3.6. All these issues were fixed in Mercurial now, and should be part of 7.3.2 final.

New tools to help with package cleanups

Did you ever have had Croaker shout at you because you removed an old version that just happened to be still required by some other package? Did you have to run your cleanups past (slow-ish) CI just to avoid that? If you did, I have just released app-portage/mgorny-dev-scripts, version 6 that has a tool just for that!
Continue reading “New tools to help with package cleanups”

Is an umbrella organization a good choice for Gentoo?

The talk of joining an umbrella organization and disbanding the Gentoo Foundation (GF) has been recurring over the last years. To the best of my knowledge, even some unofficial talks have been had earlier. However, so far our major obstacle for joining one was the bad standing of the Gentoo Foundation with the IRS. Now that that is hopefully out of the way, we can start actively working towards it.

But why would we want to join an umbrella in the first place? Isn’t having our own dedicated Foundation better? I believe that an umbrella is better for three reasons:

  1. Long-term sustainability. A dedicated professional entity that supports multiple projects has better chances than a small body run by volunteers from the developer community.
  2. Cost efficiency. Less money spent on organizational support, more money for what really matters to Gentoo.
  3. Added value. Umbrellas can offer us services and status that we currently haven’t been able to achieve.

I’ll expand on all three points.
Continue reading “Is an umbrella organization a good choice for Gentoo?”