Improving CASA Testing

STATUS: Draft - "Work required to improve CASA testing" is incomplete.

Scope

This page describes improvements in to CASA testing performed by members of the CASA development team.

This page does not address User testing or any other testing performed by those outside the CASA development team. While this testing is extremely valuable, it is out of the scope of this page.

Ideal CASA testing

Ideal CASA testing will:

• Support the whole CASA development and release process. (Jeff: are these documents current?)

• Test at the granularity of
  • a method,
  • a class,
  • a module,
  • a tool,
  • a task, and
  • the entire system, end-to-end.

• Verify
  • new code is correct,
  • changed code is still correct,
  • all functional and performance requirements are met, and
  • packaged code works correctly in user's environments.

• Test any part of CASA, allowing us to test just one change, all of CASA, or any level in between.

• Work as a system to isolate the source problems quickly.

• Minimize the work required add new tests and update existing tests.

• Run as many tests as possible, as frequently as possible, without using developer's time on routine testing.

• Summarize test results on all branches of CASA development in an easy to digest form while allowing us to drill down into test result details as desired.

Terms

Several of the terms above are not commonly used in CASA, or are used differently from software industry convention. I use them on this page as described below. I hope this will help us communicate better as we discuss improving CASA testing.

Method tests

a new term describing tests of individual methods in isolation. These tests cover a mix of implementation and derived requirements. These tests fit the classic definition of unit tests. Interaction with methods in other classes is prevented by coding tests to use either dummy or mock objects to replace methods the class under test would normally interact with.

Method tests are (usually) the easiest tests to write, and the quickest tests to run. They can shorten the edit/compile/debug cycle by helping developers to verify new code or changed code works as designed.

Class tests

a new term describing tests of interaction between methods within a class. These tests cover a mix of implementation and derived requirements. These tests fit the classic definition of unit tests. Interaction with other classes is prevented by coding tests to use either dummy or mock objects to replace classes the class under test would normally interact with.

Class tests are (usually) the easiest tests to write, and the quickest tests to run. They can shorten the edit/compile/debug cycle by helping developers to verify new code or changed code works as designed.

Module tests

a new term describing tests of interaction between classes within a module or interaction between modules. These tests cover a mix of implementation and derived requirements. Compared with Class tests, Module tests involve sets of real classes rather than one real class and dummy or mock objects.

Module tests are useful to developers to verify a new module or changed modules work as designed. They can be used to verify changes work in a subset of the system before checking them in to change control for more complete automated testing.

Tools tests

a new term describing functional or performance testing of CASA tools in isolation. These tests would be requirements oriented rather than implementation oriented like Class and Module tests.

Tools tests would help us ensure tools behave as designed before we use them in tasks. This is the lowest level that we can run useful user oriented functional and performance requirements tests.

Task tests

a new term describing functional or performance testing of CASA tasks in isolation. These tests would also be requirements oriented rather than implementation oriented like Class and Module tests.

We have instructions for writing automated Task tests (called unit tests in the instructions), along with instructions for running these tests manually. These tests may be unit tests for tasks, but they do not provide unit test coverage for the rest of CASA. This is why I call them Task tests.

Task testing is another good opportunity for user oriented functional and performance requirements testing, while letting us test tasks in isolation from each other.

End-to-end tests

a new term describing testing CASA as a whole. These tests are requirements oriented rather than implementation oriented.

We have instructions for writing automated end-to-end tests (called regression tests in the document), along with instructions for running these tests manually. These instructions may have been superseded by a new test framework written by Wes Young. I have not found instructions for using the new framework.

I call these End-to-end tests rather than regression tests as End-to-end test better describes what the tests cover.

End-to-end testing is another good opportunity for user oriented functional and performance requirements testing, while ensuring that all of CASA works together as a system on real world data. End-to-end testing should be carried out both in dedicated test systems (before software packaging) and in user's environments (after installing a CASA package following published instructions).

GUI tests

an existing term describing testing CASA using graphical user interfaces delivered with CASA.

GUI testing is the last opportunity for user oriented functional and performance requirements testing, while ensuring that all of CASA works together as a system on real world data. GUI testing should be carried out both in dedicated test systems (before software packaging) and in user's environments (after installing a CASA package following published instructions). GUI testing must be carried out both on Linux and Mac computers.

Current CASA testing

This is from the perspective of the new guy in CASA based in information linked to from CasaIndex. This is probably incomplete. Please send corrections to the author.

Process support

I do not yet understand the full CASA development and release process well enough to comment.

Existing tests

I found 462 automated tests in casacore, 52 tests in casa that are run by issuing the command "make test" in the build directory for each major module (casacore, code). These appear to be a mix of Class and Module tests covering C++ code. I don't know how much code is covered by these tests.

I have not found any Python Class or Module tests outside of Task tests. This may be appropriate.

I have not found any existing Tool tests.

I found 55 Task tests in code/xmlcasa/scripts/tests/. By comparing this directory with code/xmlcasa/tasks/, I'm guessing these appear to cover about half of the existing tasks.

I found 62 End-to-end tests in code/xmlcasa/scripts/regressions/. Jeff Kern is leading work to reduce duplication and ensure sufficient coverage in these tests.

I found one manual test for the plotxy/plotcal GUI, and several manual tests for the viewer GUI. The plotxy/plotcal test looks incomplete. The viewer tests look more complete.

Verification

I have not read enough of the existing Class and Module tests to comment on how well they verify code correctness.

I have not found any Tool tests.

Existing End-to-end and Task tests give us functional and some performance requirements coverage. How much is not clear, as I have not found any document linking tests to requirements, nor have I compared CASA requirements with existing tests. I have not yet found CASA requirements documents. GUI test coverage appears to be limited to two GUIs.

Automated distribution of Active, Test, and Stable builds do allow us to test CASA in user's environments on RHEL 5.5 64bit. RHEL 5.5 32bit Test and Stable builds are used by ALMA in Chile. I don't know if these are used in the ALMA Science Centers. I am not aware of formal CASA testing in user's environments for other supported OSs.

Selective testing

I have not found a way to run subsets of Class and Module tests. They may be run together by running "make test" in the build directory. At this time, this does not take long even on a 32bit notebook. They may become prohibitive as we grow Class and Module test coverage

Individual Task and End-to-end tests may be run in isolation, but this must be done outside the build system. This is a distraction during development.

The build system does not support running subsets of tests based on code that has just been changed.

Problem isolation

• End-to-end tests are being reorganized to give good high level test coverage. These tests are good at testing the entire system, but a failure may not provide much information on where the problem occurred.
• GUI tests currently cover the Viewer with some depth, but not much else.
• Task tests will help us find problems at the Task level, but a failure may not provide much information on where the problem occurred.
• Tool tests don't exist.
• Module and Class tests could help with low level problem isolation, but the connection to higher level code is not clear (to the new guy).

These tests do not appear to work together to help us isolate the source of problems quickly.

Work required to add or update tests

I only found instructions for creating Task tests and End-to-end tests. The work looks manageable, but I think it could be reduced with better support from the build system.

I have not found instructions for creating Class or Module tests. It is not clear how much work is required to create and maintain these tests. We may be able to minimize this by adopting cppUnit for Class and Module testing of C++ code, and extending the use of pyUnit Class and Module testing of Python code.

GUI tests are fully manual and documented on the wiki. While editing a wiki is not difficult, we may be able to reduce the work of managing GUI tests using an automated GUI test tool.

Tool tests appear to be a new idea to CASA. There is no support for this type of testing.

Automatic testing

End-to-end tests are run daily on the Test branch on several OSs. I have not yet tracked down where they are run or how the reports are produced. Results are published in Subversion. A link is available on CasaIndex.

A subset of Task tests are run on code check-in on the Active branch on RHEL 5.5 64bit. These are run on a Hudson server at NRAO CV. Results are published on Hudson. Details are in console logs.

Dirk Petry runs a nightly build and smoke test on the Active branch on an unknown OS. I don't know what the smoke test includes. Results are published in his personal web space at ESO. A link is available on CasaIndex.

Some other automated testing was run in the past at ESO, but that was shut down when the responsible developer left CASA.

I have not found any other systems running CASA tests.

Summary reporting

Results are reported separately for the automatically run End-to-end and Tasks tests. I have not found a mechanism for combining and summarizing test results.

Work required to improve CASA testing

Infrastructure

Add support for detecting which tests must be run to test changes

Develop methods to detect which tests must run based on changes in the file system or changes checked in to change control.

• Class tests - we can use naming conventions to detect which tests to run
• Module tests - we can use naming conventions, class import information, and the source code hierarchy to detect which tests to run
• Tool tests - we can use class import, or library linking information to detect which tests to run
• Task tests - we can use class import, or library linking information to detect which tests to run
• End-to-end tests - we can use Task import, class import, or library linking information to detect which tests to run
• GUI tests - we can use Task class import, or library linking information to detect which tests to run

These must be available to developers and automated test systems.

Improve builds system support for running specific tests

Add support for running selected Class and Module tests.

• This requires changes to our use of CMake.
• If possible, add make targets to run Class an Module tests on changed code as part of build
• If not possible, add scripts to run Class and Module tests based on directory tree structure.

Add support for running Tool, Task, and End-to-End tests

• This makes it easier for developers to run tests as desired.

Improve automated testing

Extend the continuous integration server in CV to support more automated testing.

• Provide quick feedback to developers when they check in changes.
  • Automated test runs to be triggered by code checkin to a branch (setup on Active first, then Test).
    • Run builds and tests on
      • OS X 10.6 64 bit
      • RHEL 5.5 64 bit.
  • Run an incremental build of changed code.
  • Run Class tests for changed code. Report if none are found.
  • Run Module tests for modules that use changed classes. Report if no tests are found.
  • Run Tool tests for Tools that use the changed modules. Report if no tests are found.
  • Run a short, representative End-to-end test that covers enough of CASA to give us a confidence test.
  • Update test summary report.
  • Notify the person/people who checked in the change with test results (success or failure) within 10 minutes of checking in the change.
    • What passed, what failed, what tests are missing.

• Provide daily updates on the overall state of CASA code.
  • Automated test runs daily (setup on Test first, then Active).
    • Run builds and tests on
      • OS X 10.6 64 bit
      • RHEL 5.5 64 bit
  • Run a clean, end-to-end build of CASA code and shared development libraries (casacore, asap, ???)
  • Run all Class, Module, Tool, Task, End-to-end, and GUI tests
    • Testing may be distributed across several computers to complete more quickly
  • Update test summary report.
  • Notify the people who checked in changes since the previous build, and others who subscribe for notification, with test results (success or failure).
    • What passed, what failed, what tests are missing

• Provide broad test coverage on the stable branch
  • Automated test runs to be triggered by code check-in to the Stable branch
    • Run builds on
      • OS X 10.5 32 bit
      • OS X 10.6 64 bit
      • RHEL 5.5 32 bit
      • RHEL 5.5 64 bit
    • Run tests on all supported OSs and versions
  • Run a clean, end-to-end build of CASA code and shared development libraries (casacore, asap, ???)
  • Run all Class, Module, and Tool tests.
    • If these pass, run all Task, End-to-end, and GUI tests.
  • Update test summary report.
  • Notify Release folk with results
    • What passed, what failed, what tests are missing

Automate GUI testing

Evaluate automated GUI testing tools. See if any really reduce the work of GUI testing.

Improve Test Summary reporting

Develop a consistent method of reporting test results across tests, and a method to pull that data from automated test systems and consolidate it into one report. Details TBD

Add test coverage measurement tools

These may help us to find holes in our test coverage. I'm not convinced this is necessary.

Developer

Improve test coverage incrementally

Look for high value tests to add. Don't bother testing simple code like getters/setters.

• As we write new code, write new Class and Module tests to demonstrate the new code works.
  • I have had good experiences with test-driven development.

• As we change existing code, write new Class and Module tests to demonstrate the existing code works before and after the change.
  • Start with tests that cover just the code to be changed.

• As we fix problems for users, write new tests at the appropriate level(s) to demonstrate the problem to fix. Then fix the problem, then run the new tests to demonstrate the problem is fixed.

Extend Class test coverage

Extend Module test coverage

Add Tool tests

Extend Task test coverage

Extend End-to-end test coverage

Extend GUI test coverage

-- ScottRankin - 2011-02-14