PMS Test Suite: getting the test results

One of key problems in PMS Test Suite is getting actual test results. With the whole complexity of build process, including privilege dropping, sandbox, collision protection, auto-pretending it is not that easy to check whether a particular test succeeded without risking a lot of false positives.

The simple attempt: succeed or die!

The simplest method of all would be to assume the test is supposed to either complete and merge successfully or die. Although that will work in many cases, it has many limitations.

First of all, to make it work as expected, the actual test code has to be executed. If for some reason the test code is not executed, we end up with a false positive. Consider the test checking phase function execution order. If for some reason pkg_postinst() isn’t called at all, there is no way we could die about it.

Moreover, if a test is supposed to fail, we can’t be sure if it failed for our reason or with some random PM bug. We could try to implement some method of grabbing the failure message and parsing it but that would imply relying on a particular output format. That’s not really what I’m interested in.

On the other hand, that is most straightforward method of checking test results. It doesn’t introduce additional dependencies, is PM-safe and that’s why the most basic EbuildTestCase class of PMS Test Suite uses that. Well, to be more exact, it checks vardb before and after running the tests to see which ones were merged and which ones failed to.

Passing more complex test results

Due to the problems pointed out above, I’ve decided to introduce a more complex test result checking method. Originally, it was supposed to use files to store ebuild output but during early testing showed that that concept has a few weaknesses.

Most importantly, FEATURES=userpriv resulted in some phase functions being run as root and some other as portage user. I’ve decided that hacking permissions, sandbox and other potential obstacles to get that concept working was not worth the effort.

That’s why the current implementation uses D-Bus for communication between the actual tests and the test runner. I was a little surprised by the fact that neither Portage nor pkgcore had any trouble with letting the test code reach the system bus.

Right now, the DbusEbuildTestCase handles all necessary D-Bus integration. It creates an D-Bus object for each running test, integrates the pms-test-dbus eclass with tests and provides methods to submit and check the test results.

Not all D-Bus test cases have to actually submit any output. Simpler ones just ping the D-Bus object in pkg_setup() to let it know that the test was actually started. This avoids a case when a test is expected to die and is considered so because PM didn’t start it at all (e.g. due to insufficient permissions when emerge assumes --pretend).

2 thoughts on “PMS Test Suite: getting the test results

  1. Andreas K. Hüttel (dilfridge)

    Actually, I started trying to prohibit any dbus access in the kde4 eclasses, at least during the src_test phase… The reason is that some of the kde unittests behave very different if they can access a session bus (and in the end they have to pass e.g. on the tinderbox, where no session nor X is running). As I’m not a dbus expert this is probably incomplete so far. We should discuss the potential problem…

    1. Michał Górny Post author

      That doesn’t really apply here. The test suite does set up D-Bus necessarily before starting the test ebuilds, and it uses the system bus to avoid session bus access issues.

Comments are closed.