DRAFT

CASA Test Framework Discussion

Where we should be

What we are trying to accomplish

  • Improved reliability (quality of functionality)
  • Fast, concise feedback to developers when development unintentionally degrades reliability
  • Multi-tiered testing where as we move between packages (test-stable-monthly-prerelease-release) we are more confident in their reliability (at the cost of timeliness)
    • smoke tests
    • modular tests
    • unit and regression tests
      • critical, accepted, experimental level tests
  • Flexibility that understood test failures which are in the process of being fixed will not prevent CASA packages from going out (e.g. quarantined tests)
  • Transparency - easily understood or otherwise industry standard
    • This can include something as simple as reorganization of the test directories or file name changes

Planned test framework workflow

Developer workflow
  1. Developer writes and runs unit tests until they pass (GoogleTest + Python tests)
  2. Developer checks in / creates a pull request for new unit tests and the modified piece of code
  3. Jenkins/Bamboo builds and runs the unit tests and smoke tests (for all platforms)
  4. If the tests fail, a message is sent to the developer who made the latest change and to the developer who wrote the unit test. Send a message to B&T group also (in order to make sure that the failures get resolved)

Nightly build system & UAT workflow
  1. Build the latest test/prerelease
  2. Run Unit tests
  3. If the tests pass, publish the test package (internally for prereleases)
  4. Run regression (module?) tests
  5. If the tests pass, publish stable/prerelease package
  6. User Acceptance (UA) testing starts for a prerelease package
  7. If UA tests pass, publish release

---------------------------------------------------

  • Developer commits code to version control
  • Smoke tests are run
    • all tests pass: kick off modular tests
    • any fails: developer who committed offending code is notified (along with Build and Test team)
    • python tests use robot framework
    • c++ tests use googletest
    • test coverage analysis is done
    • test reporting: done by TBD Continuous Integration (CI) system, available to all developers

  • Modular tests are started
    • any fails: notify developer who committed offending code (along with Build and Test team)
    • c++ testing done by googletest and googlemock
    • modular tests for casacore done on github, using travis CI (as is currently the case)
    • all other modular tests done by TBD CI system (e.g. Bamboo)
    • test reporting: done by TBD CI system, available to all developers
    • test coverage analysis is done

  • Unit and regression tests are started
    • all critical tests pass: Stable tarball can be published
    • all accepted tests pass: Monthly/Prerelease/Release tarballs can be published
    • any tests fail: notify developer responsible for the test (along with Build and Test team)
    • Unit tests are run first, any failures are tagged, to prevent later regression tests known to fail from running
      • This will optimize CPU cycles by not running tests know to fail, based on unit test results
    • python uses robot framework
    • automated Graphical User Interface (GUI) testing:
      • for web technology GUIs (e.g. CARTA, which uses javascript): Selenium
      • for all other GUIs (e.g. plotms, viewer): TBD GUI testing technology (possibly dogtail)
    • test reporting: done by TBD CI system, available to all developers
    • test coverage analysis is done

  • Hardware test resources:
    • Use idle machines on the NRAO network to run a distributed test system, during off hours, where the CI management technology runs the tests in parallel
    • Between hardware and framework optimization, the complete suite of tests should be completed in well under 12 hours (ideally in less then 4)

Where we are

Current automated test schedule

Machine Name OS Dev Environment? Test list Test time (ET) Package to publish if all tests pass NotesSorted ascending
cbt-el5-1 Red Hat 5 NO accepted Every 8 hours, starting at midnight Monthly/Prerelease/Release Only runs if there is an updated tarball, since previous check
cbt-el6-1 Red Hat 6 NO accepted Every 8 hours, starting at midnight Monthly/Prerelease/Release Only runs if there is an updated tarball, since previous check
warp Red Hat 5 YES smoke Checks for update in version control ~3 minutes Test smoke test list has a scripted list (see below)
rh5tst Red Hat 5 NO critical 4:30 AM / 4:30 PM Stable / Prerelease
rh6tst Red Hat 6 NO critical 4:30 AM / 4:30 PM Stable / Prerelease
razor OSX 10.9 YES* critical 1:30 AM / 10:00 AM Stable / Prerelease
hybrid OSX 10.10 YES* critical 1:30 AM / 10:00 AM Stable / Prerelease
cbt-d14-1 OSX 10.10 NO accepted N/A N/A Still under development
cbt-el6-2 Red Hat 6 NO casaguides Every 8 hours, starting at midnight Monthly/Prerelease/Release Still under development
cbt-d14-2 OSX 10.10 YES smoke Checks for update in version control ~3 minutes Test Those tests flagged as smoke in the framework
  • Additional notes:
    • * : These Macs do not use the Managed Software system, to keep packages and versions under manual control (no automatic updates pushed out by the admins)
    • Update 2015/08/11
    • Test schedule is dynamic, so may change without notice
    • warp smoke test list: test_clean[test11], test_clearstat, test_imstat, test_measures, test_setjy, test_vishead, test_visstat[test2], test_visstat[test3], test_visstat[test4]
    • cbt-d14-2 smoke test list: 012 Equinox Vis, 013 Fits-Import-Export
    • Version control: casacore - github, ASAP - SVN at ATNF, all other modules - SVN at NRAO

Existing test technologies

  • Pybot (based on the Robot framework)
    • The tests are created based on this technology
  • RIDE
    • Robot Framework IDE
    • Interface available to manually manage/run tests
  • Nose
    • Extends python unit testing
  • Reporting

Current test framework workflow

  • This covers the mechanical side of getting tests ready to be run in the framework
  • It doesn't include every last piece, just those files related to getting the tests ready, and providing the output for reporting
  • Ideally, if you aren't familiar with the tarball and source code files, you might want them available on your terminal line as you go through these slides

TestFrameworkAnalysisSlide1.png

TestFrameworkAnalysisSlide2.png

TestFrameworkAnalysisSlide3.png

TestFrameworkAnalysisSlide4.png

TestFrameworkAnalysisSlide5.png

DRAFT

-- AndyHale - 2015-08-04
Topic attachments
I Attachment Action Size Date Who Comment
TestFrameworkAnalysisSlide1.pngpng TestFrameworkAnalysisSlide1.png manage 49 K 2015-09-11 - 15:40 AndyHale Test Framework Slide 1
TestFrameworkAnalysisSlide2.pngpng TestFrameworkAnalysisSlide2.png manage 61 K 2015-09-11 - 15:41 AndyHale Test Framework Slide 2
TestFrameworkAnalysisSlide3.pngpng TestFrameworkAnalysisSlide3.png manage 74 K 2015-09-11 - 15:42 AndyHale Test Framework Slide 3
TestFrameworkAnalysisSlide4.pngpng TestFrameworkAnalysisSlide4.png manage 59 K 2015-09-11 - 15:44 AndyHale Test Framework Slide 4
TestFrameworkAnalysisSlide5.pngpng TestFrameworkAnalysisSlide5.png manage 40 K 2015-09-11 - 15:45 AndyHale Test Framework Slide 5
Test_Framework_Analysis_No_cover_slide.jpgjpg Test_Framework_Analysis_No_cover_slide.jpg manage 50 K 2015-09-08 - 12:02 AndyHale JPG Test framework slide
Test_Framework_Analysis_No_cover_slide.pdfpdf Test_Framework_Analysis_No_cover_slide.pdf manage 73 K 2015-09-08 - 12:01 AndyHale Existing CASA test framework slides
Test_Framework_Analysis_No_cover_slide_(2).jpgjpg Test_Framework_Analysis_No_cover_slide_(2).jpg manage 53 K 2015-09-08 - 12:08 AndyHale Slide 2
Test_Framework_Analysis_No_cover_slide_(3).jpgjpg Test_Framework_Analysis_No_cover_slide_(3).jpg manage 78 K 2015-09-08 - 12:09 AndyHale Slide 3
Test_Framework_Analysis_No_cover_slide_(4).jpgjpg Test_Framework_Analysis_No_cover_slide_(4).jpg manage 50 K 2015-09-08 - 12:09 AndyHale Slide 4
Test_Framework_Analysis_No_cover_slide_(5).jpgjpg Test_Framework_Analysis_No_cover_slide_(5).jpg manage 45 K 2015-09-08 - 12:11 AndyHale Slide 5
Topic revision: r9 - 2015-09-14, AndyHale
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback