Nightly Testing Framework
Following is the link to the page with an overview of the Nightly Testing Framework structure. Included is a listing of the required software packages and versions.
ProjectSetupDailyFunctionalTestRunsFramework on Warp
Logging In
Must login as user 'jenkins' to access the correct version of RIDE.
Note by Andy: Darrell suggested also: eval $(perl -I/home/casa/test/lib/perl5 -Mlocal::lib)
If this complains about a missing file, look to confirm the file exists in the appropriate directory. If not, look for an old directory that might have the file, and copy that as needed (as casaadm)
The commands are also contained in the configuration script '.ridesetup', located here: /cbt-el5-1/export/home/cbt-el5-1/casa/mrodrigu/TURNOVER/SCRIPTS
Once configured, RIDE can be started by:
cd to /gcwrap/python/scripts/pybot/casa
Run: /home/casa/test/pybot/2.8.4/bin/ride.py .
Note the 'dot' at the end of the command to force execution in the current directory.
The RIDE GUI allows easier editing and configuration of the pybot test framework, and a visual method of selecting and running individual tests.
Tests are edited by selecting the test in the sidebar and editing the parameters in the main window.
NOTE: When using RIDE to edit or create new tests, the resulting .html files must be retrieved from the working directory and updated in SVN.
Tests are selected for running by adding 'Tags' to the specific test(s) and setting RIDE to run only tagged tests.
RIDE provides a visual indicator while running and displays the results as each test in run.
.html Files
The recommended method of modifying framework tests is to edit the associated .html file directly. (See more detailed instructions below in ' Warp Updates')
Note by Andy: In this directory (e.g. /export/home/cbt-el5-1/casa/mrodrigu/pybot/gcwrap/python/scripts/pybot/casa) there is a file called init.html that could need hard coded paths modified
CASARegression.py
This is the main file, called by pybot/RIDE to execute the suite of regression and functional tests. It controls the download of the latest version, the analysis of result logs, the mailing of error messages and the generation of result reports.
NOTE: To prevent downloading and extracting a new version of TRUNK, set self._rerun = True(on Line 42)
Note by Andy: I had to keep this as FALSE in order for ride to work. This is due to pybot being configured to work with the jenkins tarball.
NOTE: To prevent email notifications being sent to all Maintainers and mailing lists, set the following variables:
Enable or 'uncomment' Line 364 (set default for summary_email_list)
Same as above for Line 420.
MMS Testing
MS to MMS Conversion
Here is a list of the functional tests currently supporting the MMS data structure:
test_bandpass.py
test_concat.py
test_flagdata.py
test_fluxscale.py
test_gaincal.py
test_gencal.py
test_listhistory.py
test_listobs.py
test_listvis.py
test_mst_split.py
test_plotms.py
test_split.py
test_uvcontsub.py
test_virtualconcat.py
test_vishead.py
Following are basic instructions for implementing an MMS test:
1) Create the MMS data for the task you want to test. Example: flagdata task.
Adding or changing tests in the nightly framework:
*START*: Checkout or cd to an updated working copy of the CASA code.
The files to be updated are located here, in the code tree: /gcwrap/python/scripts/pybot/casa
Perform SVN update on this directory.
Open any .html file in this directory.Save file with the new test name, following the convention of existing files.
Edit the following fields: title, h1, td (after Force Tags), td (after TEST), td (after MAINTAINER), td (after MAINTAINER_EMAIL)
Check-in updated file(s) to SVN.
COMPLETE: The new test will be listed according to the ID number and will be run during the nightly testing with pybot.
NOTE: It is recommended to run run the test locally to verify that it works appropriately in the framework.
CASA Guides
Overview: The goal is to be able to automatically extract the executable text from an existing, publicly available CASA Guide directly from the CASA Wiki. Once extracted, the resulting script is merged with a .spec file, that defines how to allocate the tests, do environmental configuration and perform analysis. The result of this merge is an executable test script, that may be used as a functional or regression test in the automated framework.
Spec file: The spec file is edited to add break point symbols or statements that match up with identical statements added to the extracted script. Example spec files are located in the CASA Test Group repository: /cbt-el5-1/export/home/cbt-el5-1/casa/mrodrigu/TURNOVER/G192_Reference_Files/INWORK
Editing: The '#@' symbol is used to note sections in the .spec file that are to be transferred to the test script upon merging with the extracted CASA Guide. Typical items added to the script include the setup and configuration of the specific test and information about cleanup.
Merging: To merge the .spec file with the extracted sript, run generateRegression.py, located here: /cbt-el5-1/export/home/cbt-el5-1/casa/mrodrigu/TURNOVER/G192_Reference_Files. Running it with the '-h' option will display the help information. The output from this operation will be a test script file.
Testing: Testing the CASA Guide involves running the test script in the working directory of the CASA build to be tested. The data (MS) will need to be copied to a directory, located under the main working directtory called 'originalDataDetailed information is contained below in the 'Running G192' Section.
Adding to Framework: To add newly created tests to the nightly testing framework, follow the steps listed in the 'Warp Updates' section above.
Alternative: The proper and desired method for running a CASA Guide test is to perform the minimum amount of editing to the CASATest.py and CASAGUIDETESTScript.py files. To accomplish this, the CASATest.py file will need to be updated to have classes to address the required testing and logging. A faster alternative would be to, after extracting the CASA Guide, massage the resulting extracted script to be runnable as a Functional Test.
To Do: To properly implement the automation of extraction, conversion and running of the CASA Guide test, CASATest.py should be modified to support this effort. New classes should be developed to control logging, analysis of logs and evaluation of CASA task calls. Rough outlines for these classes are contained in the file CASATest-NEW.py, located in the TURNOVER/SCRIPT directory on EL5-1. Here are some recommended classes:
TestTableStatus: Compare tables, check for table creation, check table size, check table content (to match desired result of task call)
Logging: Capture output from the CASA Guide test script, CASATest.py, and CASA into one log for analysis.
Analysis: Analyze log files to determine the status of the overall test as well as each sub-test.
Reporting: Create overall test report with summary.
Running G192:
Copy tarred MS to working directory (named '/originalData')
Copy the mask directories (3ea) /originalData directory.
Copy the .crtf files (3ea) to the /originalData directory.
Copy the initialFlaggingState.ref file to the /originalData directory.
Copy the script (test) file to the working directory.
Copy CASATest.py to the working directory.
Set the working directory environment variable in the /originalData directory.
export casaguideDatapath=`pwd`
cd to the working directory
Run EVLA3-bitTutorialG192Test.py
Framework Future Goals
Near Term
Alternate Branches: Currently, the nightly testing framework on WARP only uses the TRUNK version from SVN. The near term goal is to be able to also test using any branch (Test, Stable, Release...) or extracted code version.
Cron Jobs: Create scheduled tasks on the various ZENITH virtual machines to periodically and automatically run routine tests on a regular basis. The tests might be distributed as in the following scenario:
WARP, Nightly: All pybot compatible (meeting execution time constraints) Regression and Functional tests.
EL6-1, Nightly: All pybot compatible Regression and Functional tests.
EL5-1, Weekly: CASA Guide tests, M100, NGC-3256, Titan.
EL5-2, Nightly: Non-pybot tests.
EL5-2, Daily: On-call / as-needed for running tests requested by Developers.
F17-1, Nightly: All pybot compatible Regression and Functional tests.
Far Term
GUI Testing
Multi-node, parallel testing
Distributed Tests, Central Analysis/Reporting (CBTs Structure)
Metrics & Reporting
New Branches & Release Testing
Jenkins/pybot (nightly)(all regression and functional tests)
The nightly test grabs the latest TRUNK version. This may not be the exact version used for a new branch or release. If there is a mismatch between the tested version and the desired version, the tests will need to be run using a local or ZENITH version of pybot.
M100
Edit file to set data and mask location directories (7 places).
Run as normal, individual, unit or functional test.
Run time typically takes about five hours.
G192
Copy tarred MS to working directory (named '/originalData')
Copy the mask directories (3ea) /originalData directory.
Copy the .crtf files (3ea) to the /originalData directory.
Copy the initialFlaggingState.ref file to the /originalData directory.
Copy the script (test) file to the working directory.
Copy CASATest.py to the working directory.
Set the working directory environment variable in the /originalData directory.
export casaguideDatapath=`pwd`
cd to the working directory
Run EVLA3-bitTutorialG192Test.py
Additional functional tests (non-pybot)
There are six Functional Tests that do not work in the pybot nightly framework. The tests are: imcollapse, immath, imregrid, imstat, split and split2.
These tests are run separately using the script TEST.sh as described above.
Set the branch or version to checkout (trunk, release x-x-x, test, etc...) (default is trunk)
Set the number of cores available to the work machine (default is 4)
cd to the working directory
Set the path for the working directory: export casaWorkDirectory=`pwd`
Set the default path for the wrappers: export PATH=`pwd`/buildToolWrappers-0.16.1/bin:$PATH
Reference the wrappers: source .btwrc
Run: checkoutCasa (typically takes two to four minutes)
Run: buildCasa (typically takes one hour on VOLTA and two hours on ZENITH)
To run:
cd to Working Directory
export casaWorkDirectory=`pwd`
source casainit.sh
cd to Running Directory. Ensure runUnitTest.py is located here, or in the reference path.
Here, tests can be run individually using runUnitTest.py for Functional Tests, or directly from the command line for Regression Tests.
Function Test Format: casapy -c runUnitTest.py test_TESTNAME
Regression Test Format: casapy --nogui --log2term -c TESTNAME.py >& logfile.txt
Optionally, all Functional Tests may be ran using the script, TEST_Functional.sh. This script will run all Functional Tests, log results to a test_TESTNAME.log file, and printout a list failures.
Another option, is to run the tests using the Jenkins built tarball of the latest TRUNK build. This method uses a script (TEST.sh) to download the latest tarball, extract the tarball, configure the environment, move to the run directory, place necessary files in the proper location and run the tests defined in unittest_list.txt. The list may be edited to identify only a subset of tests to run. A runtime command line switch allows naming of the working directory and selection between all or a subset of tests. The script is located here: /cbt-el5-1/export/home/cbt-el5-1/casa/mrodrigu/TURNOVER/SCRIPTS
Inwork (status)
CASA Guides:
Currently, only the G192 CASA Guide can be run on a regular basis. The execution of this casaguide is somewhat difficult due to the large size of the measurement set (18 GB tarred). For each test, the MS must be moved to the working directory, and can only be used for one test run. If the tests are stopped, the MS must be renewed or replaced.
Additionally, a number of reference files and mask directories must be copied along with the MS.
Details on running the test are above under the New Branches and Release Testing section.
Other Regressions Tests (NGC 3256, Titan). Not yet implemented. These test, when integrated into the framework, may be added to the list of 'long tests', for weekly running.
tsd Tests: NAOJ is currently working to complete the tests. Once completed they will need to be tested locally and then integrated into the nightly testing framework. An initial test was performed using a sample test script to verify that the proposed testing method will work in the framework. Details are provided in CAS-5873.
MMS: ESO has developed several Multi-Measurement Set tests that need to be integrated into the nightly framework. The difficulty encountered so far is that the conversion of the MS to a MMS is time consuming and often fails. The goal will be develop the ability to convert the MS and run the tests at will and reliably. Once this is accomplished, the converted MMSs will need to be added to the standard data repository and the tests integrated into the nightly testing framework. NOTE: This task is currently assigned by Mark Rawlings to Shinnosuke, but the scheduled is not defined.
pybot: The main goal here is to duplicate the pybot/RIDE installation on WARP onto the ZENITH virtual machines (CBT-EL5-1, CBT-EL5-1, CBT-EL6-1). Once this is complete and stable, the next goal is to develop the ability to run the test suite using any branch or build. Then, cronjobs can be created to use these installations to test on different platforms (than on WARP). Additionally, different tests can be ran at different intervals. For example, while all regression and functional tests are run on a nightly basis on WARP, a nightly run can also be setup on CBT-EL6-1, to test the same build on a different platform. Additionally, weekly tests can be regularly run for tests that are too long to run on a nightly basis. These 'long' tests include M100, NGC3256, Titan, as well as the CASA Guide tests (G192).
/cbt-el5-1/export/home/cbt-el5-1/casa/mrodrigu/TURNOVER: Holding place for possibly useful files.
/cbt-el5-1/export/home/cbt-el5-1/casa/mrodrigu/TURNOVER/G192_Reference_Files: Repository for CASA Guide G192 reference files and masks.
/cbt-el5-1/export/home/cbt-el5-1/casa/mrodrigu/TURNOVER/LOG: Spreadsheet containing records for Local Build History, Ad-hoc Unit Testing, Functional Test Run Times, Regression Test Run Times, Nightly pybot Results, CASA Guide Test Results, Zenith pybot results.
/cbt-el5-1/export/home/cbt-el5-1/casa/mrodrigu/TURNOVER/NOTES: A variety of rough notes for Building CASA, Installing pybot, using SVN, Jenkins Manual, etc...
/cbt-el5-1/export/home/cbt-el5-1/casa/mrodrigu/TURNOVER/SCRIPTS: Scripts used to configure pybot and run functional tests.