CASA Requirement: Standardized Automated Test Command Line Interface

This page describes a standardized command line interface for all new CASA automated tests.

Status: Draft Proposal

Motivation

Over the years, different CASA developers have introduced several different approaches to automated testing in some area they were interested in. These different approaches focused on the areas that developer was interested in at the time. This has resulted in several different sets of automated tests with different command line interfaces. These interfaces are incompatible, and complicate summarizing CASA test results.

Standardized Command Line Interface

All new CASA automated tests shall implement the following command line interface

  1. Tests shall be located in a test directory under the source directory containing the code being tested.
    • Tests that require a complete CASA build to run shall be located in the system-test directory, which is kept in change control in the same directory that contains code, gcwrap, etc.
  2. Test programs shall be named "test[Feature]"
    • where Feature is the name of the feature being tested.
    • In most cases, one test program will include several tests to verify all aspects of a feature work correctly.
  3. Test programs shall set a UNIX exit code to report the number of sub tests that failed. Test programs shall report 0 if all tests pass.
    • The UNIX exit code for tests will be used by software the runs sets of tests and summarizes results.
  4. Test programs shall report each sub test by name, as well as an explicit, consistent pass/fail results to stdout.
    • This will simplify analysis of test results.
  5. Test programs shall report any run time errors by reporting the sub test by name, as well as an explicit, consistent error to stderr.
    • This will simplify analysis of test results.
  6. Test programs shall report debugging information in a log file separate from data sent to stdout and stderr.
    • Mixing debugging data and intermediate results with pass/fail/error data greatly complicates summarizing test results.

Implementation Notes

UNIX Exit Codes

CASA now uses CMake to build. CMake includes an integrated test driver tool called CTest. CTest is already used in the casacore build system, and partially implemented in the code build system. There are plans to complete the CTest implementation in code.

CTest expects test programs to report pass/fail results in the test UNIX exit code. From this data, CTest can work with either CDash or Dart to produce test reports. Whether CASA will user either of these is to be determined.

Pass/Fail/Error Reporting

I am not aware of any existing test tool that implements reporting exactly as described.

I recommend we, for each CASA language (Bash, C++, Python), we choose a Unit Test tool that minimizes the work of writing tests, then either modify the tool to meet our reporting requirements, or wrap it in a Bash or Python wrapper to meet our reporting requirements. We should build the wrapper in the language that minimizes development and maintenance costs.

-- ScottRankin - 2013-12-04

-- ScottRankin - 2014-02-13
Topic revision: r4 - 2014-02-14, ScottRankin
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback