CASA Software Engineering Project: Setup Daily Functional Test Runs

This project describes the framework for nightly automated testing of the CASA software package. The current and proposed configurations of the framework are provided to help outline the path and effort required for upgrading the system to the latest available versions of supported software, implementing a framework with features based on user needs and feedback.

Status: See CAS-TBD.

---

---

Team

• Mike Rodriguez, Darrel Schiebel, Andy Hale, Scott Rankin, Alexis Tejeda

Customers

• CASA Developers
• CASA Management

Collaborators

• Build Manager
• User Test Group
• Test Maintainers

Plan

Following is an outline of the development (upgrade) and implementation of automated test framework.

1) Define/clarify requirements for existing automated test framework.

2) Define requirements for future versions or upgrades of the existing framework.

3) Develop design for framework upgrades and estimate cost/effort for implementation.

4) Internal design review of proposed upgrades.

5) Scheduling and implementation of proposed upgrades.

Requirements

Current Requirements:

• Perform regression and functional testing on the non-graphics portions of the latest available (Trunk) version of the CASA software available for RHEL5.
• Perform tests nightly at a regularly scheduled time (currently 2200 EST)
• Provide notification of individual test failures to the maintainer of the test.
• Provide an overall test report to the CASA Staff mailing list.

Future/Upgrade Requirements:

• Perform regression and functional testing on any user requested branch(Test, Trunk, Release), version and platform (RHEL5, RHEL6, OSX7, OSX8, Fedora17), including GUI testing.
• Perform standard tests on a regular basis, nightly.
• Perform exceptional tests on a weekly basis using the user defined branch and version. These include tests with longer run times like M-100, NGC3256, and the tests developed from the CASA Guides.
• Include only links to the test reports, instead of the actual results, in the emailed reports.
• Automated analysis of collected reports for identification of trends and the generation of metrics.

Deliverables

• Documentation describing the framework design.
• Framework installation and configuration instructions.
• Necessary custom files (pybot and RIDE configuration files, python regression script, etc...)
• Framework upgrade requirements, plan and associated effort/cost estimates.

Features

• Interface to SVN.
• Automated builds after each code check-in with tarball delivery to specified location.
• Nightly retrieval of latest TRUNK build and extraction to test location.
• Automated implementation of regression and functional tests (Currently 174 total).
• Automated analysis of log files to determine individual test status.
• Automated report generation and mailing of individual failure reports and overall summary reports.

Design

The automated test framework uses Jenkins, pybot and RIDE, along with custom python and BASH scripts. Currently, the system is installed and configured to work on a RHEL5 machine (WARP). Jenkins monitors SVN updates and initiates a new build after each new code check-in. Upon build completion, the resulting code is compressed and copied to the 'current.tar' location (https://casa-jenkins.nrao.edu/userContent/promotion/stage01/current.tar.gz). A cron job initiates the nightly tests by calling a PERL script (run-casa-regression), which uses the Python script CASARegression.py. The Python script first copies and extracts the 'current' tarball to the test location on WARP. The tests are then performed based on the test type (regression or functional), using the configuration files developed in pybot with the RIDE GUI and the custom (developer) built Python test scripts. Regression tests are implemented in CASA using the publish_summary task. Functional tests are implemented in CASA using the runUnitTest.py task. Individual execution logs are created for each test, as well as a comprehensive test log. The CASA logs for each test are also captured. A report is configured by pybot, which includes the summarized status of each test, with links to the underlying CASA logs. Email notifications are sent to the Maintainer of each failed test, and the summary report is sent to the specified mailing list(s).

Implementation

The CASA automated test framework is currently implemented using the following tools and resources:

• RHEL5 Machine (WARP). Version:EL5 2.6.18
• RIDE Version: .35. Installed on WARP.
• Pybot Verison: 2.5.7 . Installed on WARP.
• PERL Verison: 5 . Installed on WARP.
• Jenkins Version: 1.533. Installed on WARP.
• Cron job. Name: TBD . Configured on WARP.
• Location of nightly tarball: https://casa-jenkins.nrao.edu/userContent/promotion/stage01/current.tar.gz
• Python script: CASARegression.py. Located in /gcwrap/python/scripts/pybot/casa/lib
• HTML (RIDE/pybot) configuration files. Located in /gcwrap/python/scripts/casa

Tasks

Project tasks are tracked with JIRA sub-tasks under this project's master task. See CAS-TBD.

-- ScottRankin - 2014-01-08