PMS Test Suite: the project design

As the GSoC coding period starts tomorrow, today seems like the last good day to write a little about the project design. For that reason, I created a little diagram in dia:

PMS Test Suite design diagram

As you can see there, the project code is currently broken down into three repositories. The main code repository, called pms-test-suite.git is going to hold the core project code — main scripts, library-handling Python modules and so on. The pms-test-suite-library.git repository is going to contain the bundled test library, and pms-test-suite-overlay.git holds temporary test ebuilds and the eclass.

Looking from the left hand side, you can see a greyed out part with the test-generator script in the middle of it. This module is not a part of GSoC design goals but a possible future guideline. It is supposed to read the human-readable definitions from the Package Manager Specification, match them with helper data and create a complete test library. However, right now the project will simple use a bundled test library.

The test library will consist of modules describing various tests for PMS compliance. Each of these modules, called a test case will provide at least the following information:

  1. A human-readable test description (which will be reused in program output and ebuild DESCRIPTIONs,
  2. a list of relevant EAPIs for which ebuilds will be generated,
  3. definitions of ebuild-specific variables and phase functions necessary to perform the test,
  4. a test result checking function, possibly EAPI-conditional.

Those test cases will be reused by two other PMS Test Suite scripts. The ebuild-generator module will use them to generate the ebuilds resembling the particular tests, and the test-runner module will use them to check the test results.

The test-runner script will perform the most important task of testing a particular Package Manager. It is supposed to match the test cases with generated ebuilds (or even call ebuild-generator internally), run the Package Manager on them and collect the results. The results will be used then to determine whether the test case succeeded, failed or its result is ambiguous.

The test ebuilds will use an PM-independent method of exchanging information with the test-runner module. The internal details of this communication will be handled in the pms-test.eclass. Right now, this code uses D-Bus for this task.