You are here: NRAO Public Wiki>Main Web>WikiUsers>MikeRodriguez>TurnOver (2014-06-19, AndyHale) Edit wiki text Edit Attach Print version

CASA Test Group Tasks:

Nightly Testing Framework

Following is the link to the page with an overview of the Nightly Testing Framework structure. Included is a listing of the required software packages and versions.

ProjectSetupDailyFunctionalTestRuns

Framework on Warp
  • Logging In
    • Must login as user 'jenkins' to access the correct version of RIDE.
  • Running/Editing (RIDE)
    • cd to home
    • ENTER: ride.py pybot/casa
  • Log and Report Locations
    • REPORTS: /warp/home/warp/casa/jenkins/work/userContent/promotion/stage01/results
    • LOGS: /warp/home/warp/casa/jenkins/home/log
  • Keeping Tabs
    • The nightly tests start at 10:00pm (Eastern Time)
    • A typical test runs between 11 and 13 hours
    • The nightly tests typically finish around 9:30am (Eastern Time)
  • Editing cronjob
    • Login to warp as user 'jenkins' (ssh -l jenkins warp)
    • OPTIONAL: Enter export VISUAL='emacs' . This will set the default cronjob editor to emacs, rather than vi.
    • crontab -e
    • Modify time setting and save file
Framework on Other Machines
  • Locations
    • CBT-EL5-1 (Virtual Machine on ZENITH)
      • Main Working Directory: /export/home/cbt-el5-1/casa/USER
      • STATUS
        • BUILD: OK
        • TARBALL: OK
        • PYBOT/RIDE: OK
    • CBT-EL5-2 (Virtual Machine on ZENITH)
      • Main Working Directory: /export/home/cbt-el5-2/casa/USER
      • STATUS
        • BUILD: OK
        • TARBALL: OK
        • PYBOT/RIDE: OK
    • CBT-EL6-1 (Virtual Machine on ZENITH)
      • Main Working Directory: /export/home/cbt-el6-1/casa/USER
      • STATUS
        • BUILD: Needs Work
        • TARBALL: OK (with alternate referencing)
        • PYBOT/RIDE: Needs Work (Starts okay, but can't run tests.
    • CBT-F17-1 (Virtual Machine on ZENITH)
      • Main Working Directory: /export/home/cbt-f17-1/casa/USER
      • STATUS
        • BUILD: Needs Work
        • TARBALL: Needs Work
        • PYBOT/RIDE: Needs Work
    • WARP
      • STATUS
        • BUILD: OK
        • TARBALL: OK
        • PYBOT/RIDE: OK
    • VOLTA
      • STATUS
        • BUILD: OK
        • TARBALL: OK
        • PYBOT/RIDE: OK
  • Running
    • To run RIDE on the ZENITH and VOLTA machines, the environment must be configured for PERL, using the following settings:
      • export PYTHONPATH='/home/casa/test/pybot/2.8.4/lib/python2.7/site-packages'
      • export LD_LIBRARY_PATH="$HOME/pybot/lib"
      • PATH=$HOME/pybot/bin:$PATH
      • PATH=/usr/lib64/casa/01/bin:$PATH
      • Note by Andy: Darrell suggested also: eval $(perl -I/home/casa/test/lib/perl5 -Mlocal::lib)
        • If this complains about a missing file, look to confirm the file exists in the appropriate directory. If not, look for an old directory that might have the file, and copy that as needed (as casaadm)
    • The commands are also contained in the configuration script '.ridesetup', located here: /cbt-el5-1/export/home/cbt-el5-1/casa/mrodrigu/TURNOVER/SCRIPTS
    • Once configured, RIDE can be started by:
      • cd to /gcwrap/python/scripts/pybot/casa
      • Run: /home/casa/test/pybot/2.8.4/bin/ride.py .
        • Note the 'dot' at the end of the command to force execution in the current directory.
Pybot (test framework)
  • Installation (pybot & RIDE)
  • RIDE GUI
    • The RIDE GUI allows easier editing and configuration of the pybot test framework, and a visual method of selecting and running individual tests.
    • Tests are edited by selecting the test in the sidebar and editing the parameters in the main window.
      • NOTE: When using RIDE to edit or create new tests, the resulting .html files must be retrieved from the working directory and updated in SVN.
    • Tests are selected for running by adding 'Tags' to the specific test(s) and setting RIDE to run only tagged tests.
    • RIDE provides a visual indicator while running and displays the results as each test in run.
  • .html Files
    • The recommended method of modifying framework tests is to edit the associated .html file directly. (See more detailed instructions below in ' Warp Updates')
    • Note by Andy: In this directory (e.g. /export/home/cbt-el5-1/casa/mrodrigu/pybot/gcwrap/python/scripts/pybot/casa) there is a file called init.html that could need hard coded paths modified
  • CASARegression.py
    • This is the main file, called by pybot/RIDE to execute the suite of regression and functional tests. It controls the download of the latest version, the analysis of result logs, the mailing of error messages and the generation of result reports.
    • NOTE: To prevent downloading and extracting a new version of TRUNK, set self._rerun = True (on Line 42)
      • Note by Andy: I had to keep this as FALSE in order for ride to work. This is due to pybot being configured to work with the jenkins tarball.
    • NOTE: To prevent email notifications being sent to all Maintainers and mailing lists, set the following variables:
      • Enable or 'uncomment' Line 364 (set default for summary_email_list)
      • Same as above for Line 420.
MMS Testing
  • MS to MMS Conversion
    • Here is a list of the functional tests currently supporting the MMS data structure:
      • test_bandpass.py
      • test_concat.py
      • test_flagdata.py
      • test_fluxscale.py
      • test_gaincal.py
      • test_gencal.py
      • test_listhistory.py
      • test_listobs.py
      • test_listvis.py
      • test_mst_split.py
      • test_plotms.py
      • test_split.py
      • test_uvcontsub.py
      • test_virtualconcat.py
      • test_vishead.py
    • Following are basic instructions for implementing an MMS test:
      • 1) Create the MMS data for the task you want to test. Example: flagdata task.
        • casapy --nogui --log2term -c <casa_install_dir>/python/2.6/regressions/admin/make_mmsdata.py flagdata
        • It will create the data inside ./unittest_mms/flagdata
      • 2) Run the tests normally (MS case):
        • casapy --nogui --log2term -c <casa_install_dir>/python/2.6/regressions/admin/runUnitTest.py test_flagdata
      • 3) Run the tests on the MMS data:
        • casapy --nogui --log2term -c <casa_install_dir>/python/2.6/regressions/admin/runUnitTest.py --datadir ./unittest_mms test_flagdata
  • Link to ESO instructions:
    • Overview of Parallelization in CASA: See the Attachment to this wiki-page titled 'HPC-in-CASA-4.0.pdf'
New tsd Tests (installation)
  • The new single dish test can be add to the nightly test framework as Functional Test files. See more detailed instructions below in ' Warp Updates'
  • In addition to the above instructions, the tests will need to be added to the 'unittest_list.txt' file.
    • Typical file location: /linux_64b/python/2.7/tests
Warp Updates
  • Adding or changing tests in the nightly framework:
    • *START*: Checkout or cd to an updated working copy of the CASA code.
    • The files to be updated are located here, in the code tree: /gcwrap/python/scripts/pybot/casa
    • Perform SVN update on this directory.
    • Open any .html file in this directory.Save file with the new test name, following the convention of existing files.
      • Edit the following fields: title, h1, td (after Force Tags), td (after TEST), td (after MAINTAINER), td (after MAINTAINER_EMAIL)
    • Check-in updated file(s) to SVN.
    • COMPLETE: The new test will be listed according to the ID number and will be run during the nightly testing with pybot.
    • NOTE: It is recommended to run run the test locally to verify that it works appropriately in the framework.

CASA Guides
  • Overview: The goal is to be able to automatically extract the executable text from an existing, publicly available CASA Guide directly from the CASA Wiki. Once extracted, the resulting script is merged with a .spec file, that defines how to allocate the tests, do environmental configuration and perform analysis. The result of this merge is an executable test script, that may be used as a functional or regression test in the automated framework.
  • Extraction: To retrieve a CASA Guide script, use the publically available script extractor: http://casaguides.nrao.edu/images/6/6f/ExtractCASAscript.py
  • Spec file: The spec file is edited to add break point symbols or statements that match up with identical statements added to the extracted script. Example spec files are located in the CASA Test Group repository: /cbt-el5-1/export/home/cbt-el5-1/casa/mrodrigu/TURNOVER/G192_Reference_Files/INWORK
  • Editing: The '#@' symbol is used to note sections in the .spec file that are to be transferred to the test script upon merging with the extracted CASA Guide. Typical items added to the script include the setup and configuration of the specific test and information about cleanup.
  • Merging: To merge the .spec file with the extracted sript, run generateRegression.py, located here: /cbt-el5-1/export/home/cbt-el5-1/casa/mrodrigu/TURNOVER/G192_Reference_Files. Running it with the '-h' option will display the help information. The output from this operation will be a test script file.
  • Testing: Testing the CASA Guide involves running the test script in the working directory of the CASA build to be tested. The data (MS) will need to be copied to a directory, located under the main working directtory called 'originalDataDetailed information is contained below in the 'Running G192' Section.
  • Adding to Framework: To add newly created tests to the nightly testing framework, follow the steps listed in the 'Warp Updates' section above.
  • Alternative: The proper and desired method for running a CASA Guide test is to perform the minimum amount of editing to the CASATest.py and CASAGUIDETESTScript.py files. To accomplish this, the CASATest.py file will need to be updated to have classes to address the required testing and logging. A faster alternative would be to, after extracting the CASA Guide, massage the resulting extracted script to be runnable as a Functional Test.
  • To Do: To properly implement the automation of extraction, conversion and running of the CASA Guide test, CASATest.py should be modified to support this effort. New classes should be developed to control logging, analysis of logs and evaluation of CASA task calls. Rough outlines for these classes are contained in the file CASATest-NEW.py, located in the TURNOVER/SCRIPT directory on EL5-1. Here are some recommended classes:
    • TestTableStatus: Compare tables, check for table creation, check table size, check table content (to match desired result of task call)
    • TestColumnStatus: Same functions as TestTableStatus, except for columns.
    • Logging: Capture output from the CASA Guide test script, CASATest.py, and CASA into one log for analysis.
    • Analysis: Analyze log files to determine the status of the overall test as well as each sub-test.
    • Reporting: Create overall test report with summary.
  • Running G192:
    • Copy tarred MS to working directory (named '/originalData')
    • Copy the mask directories (3ea) /originalData directory.
    • Copy the .crtf files (3ea) to the /originalData directory.
    • Copy the initialFlaggingState.ref file to the /originalData directory.
    • Copy the script (test) file to the working directory.
    • Copy CASATest.py to the working directory.
    • Set the working directory environment variable in the /originalData directory.
      • export casaguideDatapath=`pwd`
    • cd to the working directory
    • Run EVLA3-bitTutorialG192Test.py

Framework Future Goals
  • Near Term
    • Alternate Branches: Currently, the nightly testing framework on WARP only uses the TRUNK version from SVN. The near term goal is to be able to also test using any branch (Test, Stable, Release...) or extracted code version.
    • Cron Jobs: Create scheduled tasks on the various ZENITH virtual machines to periodically and automatically run routine tests on a regular basis. The tests might be distributed as in the following scenario:
      • WARP, Nightly: All pybot compatible (meeting execution time constraints) Regression and Functional tests.
      • EL6-1, Nightly: All pybot compatible Regression and Functional tests.
      • EL5-1, Weekly: CASA Guide tests, M100, NGC-3256, Titan.
      • EL5-2, Nightly: Non-pybot tests.
      • EL5-2, Daily: On-call / as-needed for running tests requested by Developers.
      • F17-1, Nightly: All pybot compatible Regression and Functional tests.
  • Far Term
    • GUI Testing
    • Multi-node, parallel testing
    • Distributed Tests, Central Analysis/Reporting (CBTs Structure)
    • Metrics & Reporting
New Branches & Release Testing
  • Jenkins/pybot (nightly)(all regression and functional tests)
    • The nightly test grabs the latest TRUNK version. This may not be the exact version used for a new branch or release. If there is a mismatch between the tested version and the desired version, the tests will need to be run using a local or ZENITH version of pybot.
  • M100
    • Edit file to set data and mask location directories (7 places).
    • Run as normal, individual, unit or functional test.
    • Run time typically takes about five hours.
  • G192
    • Copy tarred MS to working directory (named '/originalData')
    • Copy the mask directories (3ea) /originalData directory.
    • Copy the .crtf files (3ea) to the /originalData directory.
    • Copy the initialFlaggingState.ref file to the /originalData directory.
    • Copy the script (test) file to the working directory.
    • Copy CASATest.py to the working directory.
    • Set the working directory environment variable in the /originalData directory.
      • export casaguideDatapath=`pwd`
    • cd to the working directory
    • Run EVLA3-bitTutorialG192Test.py
  • Additional functional tests (non-pybot)
    • There are six Functional Tests that do not work in the pybot nightly framework. The tests are: imcollapse, immath, imregrid, imstat, split and split2.
    • These tests are run separately using the script TEST.sh as described above.
Build and Run Notes
  • Typical build process:
    • Create working directory
    • Download or copy the latest Build Tool Wrappers
    • Extract the wrappers (tar -zxvf FILENAME.tar)
    • Edit the .btwrc file;
      • Set the branch or version to checkout (trunk, release x-x-x, test, etc...) (default is trunk)
      • Set the number of cores available to the work machine (default is 4)
    • cd to the working directory
    • Set the path for the working directory: export casaWorkDirectory=`pwd`
    • Set the default path for the wrappers: export PATH=`pwd`/buildToolWrappers-0.16.1/bin:$PATH
    • Reference the wrappers: source .btwrc
    • Run: checkoutCasa (typically takes two to four minutes)
    • Run: buildCasa (typically takes one hour on VOLTA and two hours on ZENITH)
    • To run:
      • cd to Working Directory
      • export casaWorkDirectory=`pwd`
      • source casainit.sh
      • cd to Running Directory. Ensure runUnitTest.py is located here, or in the reference path.
      • Here, tests can be run individually using runUnitTest.py for Functional Tests, or directly from the command line for Regression Tests.
        • Function Test Format: casapy -c runUnitTest.py test_TESTNAME
        • Regression Test Format: casapy --nogui --log2term -c TESTNAME.py >& logfile.txt
      • Optionally, all Functional Tests may be ran using the script, TEST_Functional.sh. This script will run all Functional Tests, log results to a test_TESTNAME.log file, and printout a list failures.
      • Another option, is to run the tests using the Jenkins built tarball of the latest TRUNK build. This method uses a script (TEST.sh) to download the latest tarball, extract the tarball, configure the environment, move to the run directory, place necessary files in the proper location and run the tests defined in unittest_list.txt. The list may be edited to identify only a subset of tests to run. A runtime command line switch allows naming of the working directory and selection between all or a subset of tests. The script is located here: /cbt-el5-1/export/home/cbt-el5-1/casa/mrodrigu/TURNOVER/SCRIPTS
Inwork (status)
  • CASA Guides:
    • Currently, only the G192 CASA Guide can be run on a regular basis. The execution of this casaguide is somewhat difficult due to the large size of the measurement set (18 GB tarred). For each test, the MS must be moved to the working directory, and can only be used for one test run. If the tests are stopped, the MS must be renewed or replaced.
    • Additionally, a number of reference files and mask directories must be copied along with the MS.
    • Details on running the test are above under the New Branches and Release Testing section.
  • Other Regressions Tests (NGC 3256, Titan). Not yet implemented. These test, when integrated into the framework, may be added to the list of 'long tests', for weekly running.
  • tsd Tests: NAOJ is currently working to complete the tests. Once completed they will need to be tested locally and then integrated into the nightly testing framework. An initial test was performed using a sample test script to verify that the proposed testing method will work in the framework. Details are provided in CAS-5873.
  • MMS: ESO has developed several Multi-Measurement Set tests that need to be integrated into the nightly framework. The difficulty encountered so far is that the conversion of the MS to a MMS is time consuming and often fails. The goal will be develop the ability to convert the MS and run the tests at will and reliably. Once this is accomplished, the converted MMSs will need to be added to the standard data repository and the tests integrated into the nightly testing framework. NOTE: This task is currently assigned by Mark Rawlings to Shinnosuke, but the scheduled is not defined.
  • pybot: The main goal here is to duplicate the pybot/RIDE installation on WARP onto the ZENITH virtual machines (CBT-EL5-1, CBT-EL5-1, CBT-EL6-1). Once this is complete and stable, the next goal is to develop the ability to run the test suite using any branch or build. Then, cronjobs can be created to use these installations to test on different platforms (than on WARP). Additionally, different tests can be ran at different intervals. For example, while all regression and functional tests are run on a nightly basis on WARP, a nightly run can also be setup on CBT-EL6-1, to test the same build on a different platform. Additionally, weekly tests can be regularly run for tests that are too long to run on a nightly basis. These 'long' tests include M100, NGC3256, Titan, as well as the CASA Guide tests (G192).
Other Useful References:
Topic attachments
I Attachment Action Size Date Who Comment
HPC-in-CASA-4.0.pdfpdf HPC-in-CASA-4.0.pdf manage 424 K 2014-04-15 - 10:55 UnknownUser HPC in CASA
Edit  | Attach | Print version | History: r10 < r9 < r8 < r7 | Backlinks | View wiki text | Edit wiki text | More topic actions
Topic revision: r10 - 2014-06-19, AndyHale
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback