Requesting a CASA Test Be Added to the Automated Test Framework

Prerequisites

• An account on CASA JIRA (https://open-jira.nrao.edu)
• A working test that is ready to be put in to the framework
• The test has been checked in to the version control system
  • For a unit test: ~/gcwrap/python/scripts/tests/
• Update CMakeLists.txt files to include the new test
  • ~/gcwrap/python/scripts/tests/CMakeLists.txt
  • Add appropriate test name to the file, e.g. test_clean.py
• Any test data needed is in the data repository

Open up a JIRA ticket

1. Create a CASA JIRA ticket as a "Task"
2. Assign it to either Akeem Wells or Andy Hale
3. Ticket title and details should make it clear you want the test to be considered for the test framework
4. Include the test name, full path to the test, as well as the rev number the test will be in

What Happens Next?

Possible Iteration

1. The test is manually run in the framework to ensure it works as intended
   • Run time is considered
   • pending from the test group: Coverage analysis will be done to see how much the test adds to the pool of tests in the framework
2. There may be some iteration with the developer to optimize the test (i.e. shorten run time, etc.)
3. The test at this point will be considered either Accepted (working, but not necessarily Critical), or Experimental (if it has issues running in the framework)
   • Accepted tests will not be run nightly, and will not be required to be passing to elevate a test to a stable - only passing critical tests (those run nightly) are required for a test tarball to become a stable
   • Accepted tests are run anytime a new stable tarball is produced, and they will be run to completion before looking to see if there is another new tarball to test on

Critical Tests

1. If run time is adequately short, and (pending from test group) coverage added is significant, the test can be considered Critical (require it to be passing to elevate a test tarball to a stable tarball and running nightly)
   • There are no firm guidelines considered here
   • Particularly active areas of development (where bugs may be frequent) will help here
   • High profile areas of development (e.g. HPC, etc.) are also helpful to make the case
2. Management can immediately deem tests critical, with the understanding of the impact total test suite run time