Regression testing

In order to test CASA's portability, the regression test suite is executed continously on various supported platforms. Test results including memory and time profiling are tracked as function of CASA version and platform.

Current status

at https://casa-jenkins.nrao.edu/userContent/promotion/stage01/report.html

How to add a regression test to the framework

To add a test to the framework, do the following

1) Put the test name, for example 3C129, and email notification address in .../xmlcasa/scripts/regressions/admin/tests_list.txt
2) Add the test script name to ../xmlcasa/scripts/regressions/tests/CMakeLists.txt.
3) Create the corresponding test script in .../xmlcasa/scripts/regressions/tests/3C129.py. It must define 3 methods:

def description():
   # Returns: Short description of what is being tested
   return "Test of whatever task in whatever mode using whatever data"         # Example

def data():
   # Returns: Input data used by this test (or empty list if no input is needed)
   return ["image1", "image2"]     # Example

def run():
   # Actual test code goes here, might call other scripts
   # Must throw an exception to signal test failure
   # Returns: Data produced by the test. Will be compared against reference output from the test 

   if not all_tests_passed:                                             # Example (throw an exception to indicate failure)
      raise Exception "Something went wrong..." 
   return ["mytask.image1", "mytask.image2"]                            # Example
Optionally a fourth method can be defined if you want to control how the input data is handled by the test framework:

def doCopy():
   # (optional)
   # Returns a list of the same length as that returned by data()
   #   containing 0 if the corresponding file should be just linked
   #   (i.e. is read-only and you want to save test exec time)
   #   or 1 if the corresponding file should be really copied to
   #   the work directory
   return [0,1] # Example: image1 would be only linked, image2 would be copied

Test results will start to appear in the test reports around 1 day after these changes have been committed to the code repository.

The test is executed starting from a clean working directory, which contains the input data defined by data(). (The working directory is a defaulted parameter to publish_summary.runTest().). For example
/tmp/casa_regression/               [working directory]
/tmp/casa_regression/image1/        [input data]
/tmp/casa_regression/image2/        [input data]

The input data is retrieved from the data repository by a simple matching of file- or directory name and it can be located anywhere in the data repository.

Reference Data

You have the option in the run method to specify test products (images, cubes) which will be subjected to additional numerical tests. The corresponding reference data must be located at
.../data/regression/<testname>/reference/<productname>
for example (corresponding to the run() example above):
.../data/regression/mytask/reference/mytask.image1
.../data/regression/mytask/reference/mytask.image2

If the reference data is shared with another regression, you can save disk space by creating a link to the directory containing the reference data, e.g.
cd data/regression
ln -s oldregression newregression
# You might also need to link up subdirectories, i.e. cd oldregression/reference;ln -s oldregression newregression
svn add newregression
svn ci

Adding data to the repository

  1. In some directory with lots of disk space,
          svn co https://svn.cv.nrao.edu/svn/casa-data/trunk/
          
    (or https://svn.cv.nrao.edu/svn/casa-data/trunk/regression if that is all you will need.) This will take a lot of time and disk space, so you may want to fine tune the svn command.
  2. mv your new dataset(s) to the new regression directory.
  3. (optional)
          rm -rf $CASADATA/regression
          
    mv the new regression directory to $CASADATA/regression
  4.       cd regression
          svn ci
          

When the regression servers run your test, the data should then get rsynced in properly.

How to manually run a test through the framework

To run a test from your local CASA installation, do the following in a casapy session:

  • publish_summary.runTest('mytest')

See help publish_summary.runTest for further options.

How to run the full test suite through the framework

2012/12/21 instructions here are flakey; see bottom of page for possible methods to success.

If you are testing a CASA binary distribution, then first copy all of the regression infrastructure to the same directory. If you have compiled CASA yourself, skip this step. For example, for the CASA-2.4 release:

  • cp /path_to/casapy-24.0.8115-001/lib/casapy/bin/*.pl  /path_to/casapy-24.0.8115-001/lib/python2.5/regressions/admin/
  • cp /path_to/casapy-24.0.8115-001/lib/casapy/bin/*.txt /path_to/casapy-24.0.8115-001/lib/python2.5/regressions/admin/
  • cp /path_to/casapy-24.0.8115-001/lib/casapy/bin/*.sh  /path_to/casapy-24.0.8115-001/lib/python2.5/regressions/admin/

Then get the regression test's input data (requires ~28 GB disk space), and link to it from your casapy installation Warning: You need to specify which revision (ZZZZ below) of the data repository you need for the version of CASA that you are testing. For example 3.0.0 requires r2170.

Here /path_to/data/regression/ is the local directory that holds the regression data.

Finally, run the test suite:
  • rm /tmp/tests_next.txt
  • export PATH=$PATH:/path_to/casapy-24.0.8115-001/
  • /path_to/casapy-24.0.8115-001/lib/python2.5/regressions/scheduler.pl -all -noloop -noclean -res_dir=/tmp/results/ -work=/tmp/work/ /tmp/ /path_to/data/regression/ 9999 14400 </dev/null

You will need to adjust the paths to scheduler.pl and to your data repository. (And you may want to use a different working directory than /tmp/) Warning: any working directory which you substitute for /tmp/work/ will be rm -rf ed.)

After the tests have run, you can generate an HTML report by running (in a casapy session)

  • import report
  • report.report('/tmp/results/', '/tmp/report', 'all', True)

Then point a browser to file:///tmp/report/test-report.html

WARNING: Tests that either crash or are killed by timeout do bring down the full casapy session with them, including most of the testing framework. Therefore test logs from crashing tests will be partly missing. However, a session logfile is always produced in /tmp/results/Log/. You should compare the (number of) logfiles in /tmp/results/Log/ with the (number of) tests in the report; the tests that are mentioned in /tmp/results/Log/ but missing from the HTML report are probably due to crashes, and the reason would be given in the corresponding session logfile in /tmp/result/Log/.


-- RemyIndebetouw - 2011-12-21 method [YMMV]:

to run framework in the working dir /export/data_1/rindebet/casa/regress/auto/,

  • copy scheduler.pl to workdir and edit scheduler.pl adding manually these definitions near the top:
$prefix="/export/data_1/rindebet/casa/regress/auto/";
$data_dir="/export/data_1/rindebet/casa/active/data/";
$noclean=1;
$noloop=1;
also replace the auto-found admin_dir with a manual path to my own active build:
#$admin_dir = $FindBin::Bin;
$admin_dir = "/users/rindebet/casa/active-debug/code/xmlcasa/scripts/regressions/admin/";
and change the cmd to run a local copy of runcasa_from_shell that we'll create below in $prefix
       $cmd = 
      "$admin_dir/process_manager.pl $timeout " .
      "$prefix/runcasa_from_shell.sh $xdisplay $work_dir/admin/execute.py $data_dir $work_dir $testname $p";

  • create /export/data_1/rindebet/casa/regress/auto/result/ , copy tests_lists.txt into that dir, and edit it to the tests you want to run.
  • copy gprof2dot.py to /export/data_1/rindebet/casa/regress/auto/ - this seems to be required whether you're doing oprofiling or not
  • copy runcasa_from_shell.sh to /export/data_1/rindebet/casa/regress/auto/ and add
source /users/rindebet/casa/active-debug/casainit.sh 
just before these lines:
if which casapy > /dev/null ; then
    echo "Using `which casapy`"
else
(to force it to use my own build, instead of whatever system installation comes up by default in my shell)

  • in unix, ./scheduler.pl 9999 14400
  • in casapy, import report, and report.report('/users/rindebet/casa/regress/auto/sresult', '/users/rindebet/casa/regress/auto/report', 'all', True)
Topic revision: r23 - 2012-03-14, SandraCastro
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback