Wednesday, August 24, 2011

screen: a new test suite for crack

In the current trunk, and due for inclusion in the next release, is our new test suite "screen".

It replaces the original system built on a simple bash script. The new test suite offers several advantages:

  1. It is template based, and individual tests all have their own templates
  2. It can skip tests based on functionality available on the host system (e.g. skip SDL tests if you don't have that extension)
  3. It runs multiple tests concurrently (see the -j option). The default concurrency is 4.
  4. It can test one or more builders. Currently this means it can run the tests through JIT, Native AOT, or both.
  5. It can compare expected results literally or to a regex
  6. It can test for explicit stderr output (e.g. parse errors)
  7. It can stop on test failure, and show expected vs actual results
  8. It can run a single test template, or recursively import a directory of them
  9. It's written in crack :)

The main test suite is run via "make check", as usual. You can find the source to screen, as well as the test templates themselves, in the screen/ directory. For more advanced testing (including running individual tests), run crack screen.crk --help