PMS Test Suite: choosing the test library format

The main purpose of the PMS Test Suite project is to test Package Managers for compliance with the PMS. In order to do that, the suite has to supply a particular PM with test ebuilds, run it and check the test results.

Test ebuilds are often very similar one to another. Many of the common tests require running with more than a single EAPI — resulting in ebuilds differing only in a single EAPI line. Considering that, writing test ebuilds manually and maintaining them in that form doesn’t seem like a good idea. That’s why PMS Test Suite will come with a custom test library format.

The test library will be the heart of the whole project. It will supply it with all the test case details necessary to create an appropriate ebuild, run the PM on it and parse the results. That’s why it is very important to choose an elegant and efficient format for the test cases in the library.

Right now, I’m considering using one of the following scripting languages for the test cases:

1. Bash

(bash test case example)

Pros:

  • simple syntax,
  • same solution as used in ebuilds,
  • lazy code parsing — it is possible to declare phase functions as actual functions and grab their code afterwards.

Cons:

  • requires a lot of effort to parse,
  • slow and CPU-intensive execution,
  • limited datatypes,
  • more complex operations rely on external utilities.

2. Lua

(lua test case example)

Pros:

  • simple yet powerful,
  • fast.

Cons:

  • introduces another language in the workflow,
  • requires installing additional packages.

3. Python

(python test case example)

Pros:

  • ability to use class inheritance,
  • no additional tools required,
  • byte-compiled code.

Cons:

  • more complex syntax,
  • requirement of fixed namespace or hacks in order to load libraries efficiently.

Summary

Considering that the core PMS Test Suite will be written in Python, using the same language for the test library seems to be the way to go. The test cases can subclass an appropriate, common PMSTest class and then the whole library parsing would be limited to loading the modules and instantiating the classes providing by them.