Category: Gentoo

One of key problems in PMS Test Suite is getting actual test results. With the whole complexity of build process, including privilege dropping, sandbox, collision protection, auto-pretending it is not that easy to check whether a particular test succeeded without risking a lot of false positives.

The simple attempt: succeed or die!

The simplest method of all would be to assume the test is supposed to either complete and merge successfully or die. Although that will work in many cases, it has many limitations.

First of all, to make it work as expected, the actual test code has to be executed. If for some reason the test code is not executed, we end up with a false positive. Consider the test checking phase function execution order. If for some reason pkg_postinst() isn’t called at all, there is no way we could die about it.

Moreover, if a test is supposed to fail, we can’t be sure if it failed for our reason or with some random PM bug. We could try to implement some method of grabbing the failure message and parsing it but that would imply relying on a particular output format. That’s not really what I’m interested in.

On the other hand, that is most straightforward method of checking test results. It doesn’t introduce additional dependencies, is PM-safe and that’s why the most basic EbuildTestCase class of PMS Test Suite uses that. Well, to be more exact, it checks vardb before and after running the tests to see which ones were merged and which ones failed to.

Passing more complex test results

Due to the problems pointed out above, I’ve decided to introduce a more complex test result checking method. Originally, it was supposed to use files to store ebuild output but during early testing showed that that concept has a few weaknesses.

Most importantly, FEATURES=userpriv resulted in some phase functions being run as root and some other as portage user. I’ve decided that hacking permissions, sandbox and other potential obstacles to get that concept working was not worth the effort.

That’s why the current implementation uses D-Bus for communication between the actual tests and the test runner. I was a little surprised by the fact that neither Portage nor pkgcore had any trouble with letting the test code reach the system bus.

Right now, the DbusEbuildTestCase handles all necessary D-Bus integration. It creates an D-Bus object for each running test, integrates the pms-test-dbus eclass with tests and provides methods to submit and check the test results.

Not all D-Bus test cases have to actually submit any output. Simpler ones just ping the D-Bus object in pkg_setup() to let it know that the test was actually started. This avoids a case when a test is expected to die and is considered so because PM didn’t start it at all (e.g. due to insufficient permissions when emerge assumes --pretend).

As the GSoC coding period starts tomorrow, today seems like the last good day to write a little about the project design. For that reason, I created a little diagram in dia:

As you can see there, the project code is currently broken down into three repositories. The main code repository, called pms-test-suite.git is going to hold the core project code — main scripts, library-handling Python modules and so on. The pms-test-suite-library.git repository is going to contain the bundled test library, and pms-test-suite-overlay.git holds temporary test ebuilds and the eclass.

Looking from the left hand side, you can see a greyed out part with the test-generator script in the middle of it. This module is not a part of GSoC design goals but a possible future guideline. It is supposed to read the human-readable definitions from the Package Manager Specification, match them with helper data and create a complete test library. However, right now the project will simple use a bundled test library.

The test library will consist of modules describing various tests for PMS compliance. Each of these modules, called a test case will provide at least the following information:

1. A human-readable test description (which will be reused in program output and ebuild DESCRIPTIONs,
2. a list of relevant EAPIs for which ebuilds will be generated,
3. definitions of ebuild-specific variables and phase functions necessary to perform the test,
4. a test result checking function, possibly EAPI-conditional.

Those test cases will be reused by two other PMS Test Suite scripts. The ebuild-generator module will use them to generate the ebuilds resembling the particular tests, and the test-runner module will use them to check the test results.

The test-runner script will perform the most important task of testing a particular Package Manager. It is supposed to match the test cases with generated ebuilds (or even call ebuild-generator internally), run the Package Manager on them and collect the results. The results will be used then to determine whether the test case succeeded, failed or its result is ambiguous.

The test ebuilds will use an PM-independent method of exchanging information with the test-runner module. The internal details of this communication will be handled in the pms-test.eclass. Right now, this code uses D-Bus for this task.

The main purpose of the PMS Test Suite project is to test Package Managers for compliance with the PMS. In order to do that, the suite has to supply a particular PM with test ebuilds, run it and check the test results.

Test ebuilds are often very similar one to another. Many of the common tests require running with more than a single EAPI — resulting in ebuilds differing only in a single EAPI line. Considering that, writing test ebuilds manually and maintaining them in that form doesn’t seem like a good idea. That’s why PMS Test Suite will come with a custom test library format.

The test library will be the heart of the whole project. It will supply it with all the test case details necessary to create an appropriate ebuild, run the PM on it and parse the results. That’s why it is very important to choose an elegant and efficient format for the test cases in the library.

Right now, I’m considering using one of the following scripting languages for the test cases:

1. Bash

(bash test case example)

Pros:

• simple syntax,
• same solution as used in ebuilds,
• lazy code parsing — it is possible to declare phase functions as actual functions and grab their code afterwards.

Cons:

• requires a lot of effort to parse,
• slow and CPU-intensive execution,
• limited datatypes,
• more complex operations rely on external utilities.

2. Lua

(lua test case example)

Pros:

• simple yet powerful,
• fast.

Cons:

• introduces another language in the workflow,
• requires installing additional packages.

3. Python

(python test case example)

Pros:

• ability to use class inheritance,
• no additional tools required,
• byte-compiled code.

Cons:

• more complex syntax,
• requirement of fixed namespace or hacks in order to load libraries efficiently.

Summary

Considering that the core PMS Test Suite will be written in Python, using the same language for the test library seems to be the way to go. The test cases can subclass an appropriate, common PMSTest class and then the whole library parsing would be limited to loading the modules and instantiating the classes providing by them.