Week 4 – Modernization of Portage

Another week of GSOC. Days run really fast. This again was a productive week. The first half was  towards understanding the unit tests for portage and the second half was towards solving a bug.

Testing in portage

Tests are one of the most important components of any software. Portage being no exception  employs unit tests for testing. Till now, I did not bother to look into the tests. We have a bash script runtests. I run it and I watch for things to succeed. Sam felt that I needed to have a bit more  understanding of the tests, for various reasons. So, I started looking into the tests.

Portage’s tests are single threaded. It takes between 300 and 450 seconds to run all the tests portage has, depending on the speed of the machine. It would be nice to have the unit tests run in parallel, but there are several caveats to that. For one, portage needs to virtualize various things including runtime parameters and a filesystem (to test the changes portage makes). Sharing one virtualized  environment among many threads did not seem like a plausible idea. So, for each thread a
new virtual environment has to be created. So threading has to be outside the virtual environment creation phase.

So, I added the functionality to start and stop testing at the nth test file. With this functionality, the  plan is to count the number of tests, split them into groups and assign each group to a separate thread. This leads to a bit of overhead as virtual environments have to created for each thread, but it will make the tests faster. The implementation can be found in this pull request. It is not merged yet because the long term goal is to get rid of runtests and exclusively use standardized python tools
like pytest-xdist for running tests parallely. There is also work going on to make portage tests run properly with python-xdist. I am not sure if this work will block that. It should not, but still, we are holding the merge.

Bug 528836

From day one, I wanted to work on the dependency resolution system of portage. But it is obviously not a simple job and so Sam advised to get familiar with the algorithm by fixing bugs related to that. Sam chose me a bug to fix and it is 528836. The problem is that two conflicting packages are pulled in when only should have been pulled. The bug was not reproduceable with the current state of portage and the ebuild repository. There were a few hurdles along the way, but finally, we were able to reproduce the bug by restoring portage and the ebuild repository to 2017.

We are not yet sure if the bug is due to portage or some misconfiguration in the ebuild repository.  We will continue to work on it and I will keep you posted.

Next week’s plan

The next week’s plan will be to write tests for this bug to make sure it doesn’t happen again. We  will also try to squeeze in a few more quality of life changes if time permits.