In order to test CASA's portability, the regression test suite is executed continously on various supported platforms. Test results including memory and time profiling are tracked as function of CASA version and platform.
How to add a regression test to the framework
To add a test to the framework, do the following
1) Put the test name, for example
, and email notification address in
2) Add the test script name to
3) Create the corresponding test script in
. It must define 3 methods:
# Returns: Short description of what is being tested
return "Test of whatever task in whatever mode using whatever data" # Example
# Returns: Input data used by this test (or empty list if no input is needed)
return ["image1", "image2"] # Example
# Actual test code goes here, might call other scripts
# Must throw an exception to signal test failure
# Returns: Data produced by the test. Will be compared against reference output from the test
if not all_tests_passed: # Example (throw an exception to indicate failure)
raise Exception "Something went wrong..."
return ["mytask.image1", "mytask.image2"] # Example
Optionally a fourth method can be defined if you want to control how the input data
is handled by the test framework:
# Returns a list of the same length as that returned by data()
# containing 0 if the corresponding file should be just linked
# (i.e. is read-only and you want to save test exec time)
# or 1 if the corresponding file should be really copied to
# the work directory
return [0,1] # Example: image1 would be only linked, image2 would be copied
Test results will start to appear in the test reports around 1 day after these changes have been committed to the code repository.
The test is executed starting from a clean working directory, which contains the input data defined by
. (The working directory is a defaulted parameter to
.). For example
/tmp/casa_regression/ [working directory]
/tmp/casa_regression/image1/ [input data]
/tmp/casa_regression/image2/ [input data]
The input data is retrieved from the data repository by a simple matching of file- or directory name and it can be located anywhere in the data repository.
You have the option in the
method to specify test products (images, cubes) which will be subjected to additional numerical tests. The corresponding reference data must be located at
for example (corresponding to the
If the reference data is shared with another regression, you can save disk space by creating a link to the directory containing the reference data, e.g.
ln -s oldregression newregression
# You might also need to link up subdirectories, i.e. cd oldregression/reference;ln -s oldregression newregression
svn add newregression
Adding data to the repository
- In some directory with lots of disk space,
svn co https://svn.cv.nrao.edu/svn/casa-data/trunk/
(or https://svn.cv.nrao.edu/svn/casa-data/trunk/regression if that is all you will need.) This will take a lot of time and disk space, so you may want to fine tune the svn command.
mv your new dataset(s) to the new regression directory.
rm -rf $CASADATA/regression
mv the new regression directory to
When the regression servers run your test, the data should then get rsynced in properly.
How to manually run a test through the framework
To run a test from your local CASA installation, do the following in a casapy session:
for further options.
How to run the full test suite through the framework
2012/12/21 instructions here are flakey; see bottom of page for possible methods to success.
If you are testing a CASA binary distribution, then first copy all of the regression infrastructure to the same directory. If you have compiled CASA yourself, skip this step.
For example, for the CASA-2.4 release:
cp /path_to/casapy-24.0.8115-001/lib/casapy/bin/*.pl /path_to/casapy-24.0.8115-001/lib/python2.5/regressions/admin/
cp /path_to/casapy-24.0.8115-001/lib/casapy/bin/*.txt /path_to/casapy-24.0.8115-001/lib/python2.5/regressions/admin/
cp /path_to/casapy-24.0.8115-001/lib/casapy/bin/*.sh /path_to/casapy-24.0.8115-001/lib/python2.5/regressions/admin/
Then get the regression test's input data (requires ~28 GB disk space), and link to it from your casapy installation
Warning: You need to specify which revision (ZZZZ below) of the data repository you need for the version of CASA that you are testing. For example 3.0.0 requires r2170.
is the local directory that holds the regression data.
Finally, run the test suite:
- rm /tmp/tests_next.txt
- export PATH=$PATH:/path_to/casapy-24.0.8115-001/
/path_to/casapy-24.0.8115-001/lib/python2.5/regressions/scheduler.pl -all -noloop -noclean -res_dir=/tmp/results/ -work=/tmp/work/ /tmp/ /path_to/data/regression/ 9999 14400 </dev/null
You will need to adjust the paths to
and to your data repository. (And you may want to use a different working directory than
) Warning: any working directory which you substitute for
After the tests have run, you can generate an HTML report by running (in a casapy session)
report.report('/tmp/results/', '/tmp/report', 'all', True)
Then point a browser to file:///tmp/report/test-report.html
WARNING: Tests that either crash or are killed by timeout do bring down the full casapy session with them, including most of the testing framework. Therefore test logs from crashing tests will be partly missing. However, a session logfile is always produced in
. You should compare the (number of) logfiles in
with the (number of) tests in the report; the tests that are mentioned in
but missing from the HTML report are probably due to crashes, and the reason would be given in the corresponding session logfile in
- 2011-12-21 method [YMMV]:
to run framework in the working dir /export/data_1/rindebet/casa/regress/auto/,
- copy scheduler.pl to workdir and edit scheduler.pl adding manually these definitions near the top:
also replace the auto-found admin_dir with a manual path to my own active build:
#$admin_dir = $FindBin::Bin;
$admin_dir = "/users/rindebet/casa/active-debug/code/xmlcasa/scripts/regressions/admin/";
and change the cmd to run a local copy of runcasa_from_shell that we'll create below in $prefix
"$admin_dir/process_manager.pl $timeout " .
"$prefix/runcasa_from_shell.sh $xdisplay $work_dir/admin/execute.py $data_dir $work_dir $testname $p";
- create /export/data_1/rindebet/casa/regress/auto/result/ , copy tests_lists.txt into that dir, and edit it to the tests you want to run.
- copy gprof2dot.py to /export/data_1/rindebet/casa/regress/auto/ - this seems to be required whether you're doing oprofiling or not
- copy runcasa_from_shell.sh to /export/data_1/rindebet/casa/regress/auto/ and add
just before these lines:
if which casapy > /dev/null ; then
echo "Using `which casapy`"
(to force it to use my own build, instead of whatever system installation comes up by default in my shell)
- in unix, ./scheduler.pl 9999 14400
- in casapy, import report, and report.report('/users/rindebet/casa/regress/auto/sresult', '/users/rindebet/casa/regress/auto/report', 'all', True)