CASA 4.2.2 Intermediate Release Plan

Status

  • RJS 8/11/14:
    • The 4.2.2 branch is feature complete and has been heavily tested (on RHEL) by the pipeline group. From the pipeline perspective, the 4.2.2 branch is ready for release.
    • A single blocker for the release is a bug in interactive use of plotms.See CAS-6813.
    • OSX release will be built for OSX 10.8. The first OSX pre-release package is expected by the end of this week (8/15/14).
    • The Pipeline is feature complete and tested for release to the telescopes. This will be concurrent with the CASA 4.2.2 release.

  • RJS 6/20/14:
    • Added two SD tickets for Tsys/weights integration.

  • MGR 6/10/14:
    • Added updates to list of inclusions for split2. List for other items to be checked?

  • RJS 6/02/14:
    • Added 1.0.1 additional weights items in anticipation of next pre-release package on 6/6/14.

  • RJS 5/22/14:
    • Updated schedule to account for anticipated delays in pre-release package delivery.

  • RJS 5/20/14:
    • Merging changes from trunk to 4.2.2 branch in progress.
    • Scott is shooting for a pre-release package for testing by 5/23/14. The package may not be feature complete, but will include the new weights implementation.

  • RJS 5/6/14:
    • Revised milestone dates to reflect recent slip due to 4.2.1 resource conflicts.

  • RJS 4/30/14:
    • All initially planned features were ready to merge by the feature freeze date (4/25).
    • Two new tickets under evaluation for inclusion based on NAOJ and CSSC requests (see table below).
    • Merging and subsequent building and testing is on hold due to resource conflicts with 4.2.1 release. Currently this puts us ~2 weeks behind schedule.
      • Will reassess once resource conflict is resolved.

  • RJS 4/21/14:
    • The branch has been setup and is ready to receive revisions.
    • Progress towards the feature freeze is recorded in the requirements table below.
      • RTM indicates that the fix has been developed on trunk and is 'ready to merge'.
      • Open indicates that development is ongoing.

Requirements

  • Casa422DesiredRequirements: Initial collection of requirements from the development team and CSSC.
  • Requirements are summarized below. The inclusion or exclusion of the request from the final scope is also noted.

Item Requested Features Lead Details 4.2.2 Approved Scope Status SVN Revision IDsSorted ascending
4.1     CAS-6164:Running setjy several times on different source / spw selections sometimes clears previous model data (Virtual Model Column requires review by Kumar who is on leave. Omit from release.) Omit - RS 5/2/14 N/A  
6.1 PlotMS: Framebuffer issue resolution Darrell CAS-6363: Plotms framebuffer fix for pipeline (Desirable, but pulled from scope due to resource limits) Omit - JK 4/07/14 N/A  
9.2   Sandra CAS-6179: Make a wrapper of hanningsmooth to use the MSTransform framework. Omit - JK 3/26/14 N/A  
10.0 Other isolated bug fixes.   Must not present an undue burden to merge from Trunk to the branch, and be both necessary and well tested. TBD  
1.0.1     (Weights, continued) Include - RS 6/2/14 RTT

r29851, r29917 (split), r29856 (cvel), r29860 (test_split.py), r29918 + r9870 (data) (time_then_chan_avg.py), r29891 (Tsys weight scale fix)

code: 29851, 29856, 29891, 29917

gcwrap: 29860, 29918

data: 9870

4.0.1     a fix for multiple source fluxdict issue reported in CAS-5941 on Jun 02   Packaged

(Already merged -TT)

gcwrap: r.29941 (4.2.2brach)

r.29940 in trunk

1.0.2   Justo

Adopt new WEIGHT/SIGMA convention in:

- VI/VB2 framework in order to re-converge time. avg algorithm with that of split

- mstransform in order to re-converge spectral averaging and re-gridding algorithms with split and cvel respectively.

- flagdata in order to take into account changes in VI/VB2 for the feature to flag using clip mode based on WEIGHT/WEIGH_SPECTRUM

Include - JK 6/02/14 Closed

r30209 (CAS-6642): Contains changes in the VI/VB2 framework (replace WEIGHT_SPECTRUM_CORRECTED with SIGMA_SPECTRUM) and AveragingTVI (time average)

r30210 (CAS-6642): Contains changes in mstransform module (spectral average/regridding)

r30234 (CAS-5044): Contains changes in the flagging module (clip mode based on WEIGHT column)

1.0.3   Justo Implement a new mode in the calibrator tool to convert WEIGHTS from the old convention to the new (channelized) one. Include - JK 6/02/14 Closed r30272: calibrater tool contains a new method "changeWeightConvention" which modify the WEIGHT column dividing between the number of channels in order to be compliant with 4.2.2 WEIGHT convention
8.1   Kana CAS-5841: Make the scan numbers between scantable, asdm, and MS consistent ... All new unit tests are affected by the change Include - KS/JK 3/19/14 Packaged (Included above)
8.2   Kana CAS-5842: Make sd.splitant available in sdsave ... the new sdsave task depends on the development. Include - KS/JK 3/19/14 Packaged (Included above)
8.3 ASAP: Provide a way to fill the WEIGHT(_SPECTRUM) column in single dish MS. Kana CAS-6181: SD processing needs to incorporate ata weights. (This is a counterpart of CAS-6129 in interferometer task.) \ Include - KS/JK 3/19/14 Packaged (Included above)
8.4 ASAP: Enable channel selection in Tsys calibration. Kana CSV-2986: SD calibration in CASA is not adequately removing the effects of atmospheric lines. Band-7 observations by ALMA are affected by this issue. Include - KS/JK 3/19/14 Packaged (Included above)
1.1   George CAS-3522: Quantitative documentation of data weights Include - JK 3/26/14   (none)
2.1   George CAS-3494: Handle uncalibrate-able spws better Include - JK 3/26/14 Packaged (same as above)
5.0 Viewer: Bug fixes Susan, Darrell All fixes committed to trunk since 4.2.0 branch was created. Include - JK 3/26/14 Packaged

(Sync with 4.2.1 branch) SVN IDs

All changes merged from 4.2.1 to 4.2.2

10.2   Dirk CAS-5776: Linux distro of release 4.2 should contain complete set of include files (Requested by Dirk - problem for anybody who wants to link to our copy of, e.g. the casacore libraries.) Include - RS 5/20/14 Packaged Changes to packaging tools outside the CASA source tree - Completed in first published CASA 4.2.2 prerelease package.
10.8   Darrell CAS-6202:include/casacore is missing Include - RS 5/1/14 Verified Changes to packaging tools outside the CASA source tree - Completed in first published CASA 4.2.2 prerelease package.
3.1   Sandra CAS-5368:Add the ability to give multiple summary commands in flagdata's list mode. Include - JK 3/26/14 Packaged code, gcwrap r27602
9.4   Sandra CAS-5234:cannot run split on ms created from an asap because data column is called 'FLOAT_DATA' Include - RS 5/20/14 Packaged code: 28195
9.0 Mstransform: Bug fixes and features Sandra CAS-6351: partition and mstransform problems with STATE sub-table in an MMS Include - JK 3/31/14 Packaged

code: 29092

gcwrap: 29092

1.0 Calibration: Changes to the handling of WEIGHT and SIGMA. George As described in Weights.pdf. Changes in accordance with Phase I of the Implementation plan described in the document. Include - JK 3/26/14 Packaged

code: 29438, 29494

gcwrap: 29439

data: 9709, 9710, 9725, 9729

2.0 Calibration: Fix to the application of flags in calibration George CAS-6289:chan-dep calibration apply should be capable of flagging Include - JK 3/26/14 Packaged

code: 29440

gcwrap: 29441

1.2   George CAS-6129: Weights improvements (Only the parts of this ticket consistent with the Implementation plan referenced above). Include - JK 3/26/14 Packaged

code: r28834

gcwrap: r28835

10.1   George CAS-6280: gaincal combine='spw,scan' crosses obsid boundaries Include - RS 5/20/14 Packaged code: r28895
3.2   Sandra CAS-6205: Add a time buffer option in flagdata list mode to increase the timerange selections Include - JK 3/26/14 Packaged gcwrap r28816
3.0 Flagging: Fixes Remy CAS-6006: Whitespace handling Include - JK 3/26/14 Packaged gcwrap r28820
9.7   Sandra   Include - RS 5/20/14 Packaged gcwrap: 27537
7.1   Remy CAS-6147:plotbandpass can't handle this table Include - JK 3/26/14 Packaged gcwrap: 28822
7.0 Plotbandpass fixes. Lindsey CAS-6111:plotbandpass sometimes fails to produce plots or an error message when overlay='time,antenna' Include - JK 3/26/14 Packaged gcwrap: 28823
7.2   Todd CAS-6356:plotbandpass overlay='time' option sometimes shows an error in the time legend colors Include - JK 3/26/14 Packaged gcwrap: 29084
7.3   Todd CAS-6368:plotbandpass overlay='spw' and 'baseband' options only show the final spw if the first one is flagged Include - JK 4/11/14 Packaged gcwrap: 29130
7.4   Todd CAS-6394:plotbandpass mode overlay='time' (per antenna) does not show all unflagged timeranges when one is flagged Include - JK 4/11/14 Packaged gcwrap: 29278
7.5   Todd CAS-6476:plotbandpass crashes if final spw solution is flagged and subplot='11' and overlay='spw' Include - TH 4/25/14 Packaged gcwrap: 29460
10.6 Fluxscale: bug fix Tak CAS-5699: problem in dictionary returned by fluxscale in latest casapy-test #26688 Include - JK 3/31/14 Packaged r. 27082 (May already be in the 4.2.2 branch) & r. 27917 (to be merged) (SR - was already merged).
9.3   Sandra CAS-5348:New VI/Vb framework crashes when accessing virtual MODEL_DATA column Include - RS 5/20/14 Packaged

R28189, R28233

code: 28189, 28233

gcwrap: 28189

9.1   Sandra CAS-6206: mstransform does not re-index the SPECTRAL_WINDOW sub-table fully. Omit - JK 3/26/14 N/A R28600
10.4   Michel Change by Michel Caillat for correct initialization of the weight column in casacore. Include - JK 3/19/14 Packaged r28818, r28821
9.5   Sandra Fixes a bug when validating the parameters in mstransform Include - RS 5/20/14 Packaged R29042
9.6   Sandra Fixes a bug that affects MSs with different polarization shapes when a polarization selection is requested in an MMS-case. Include - RS 5/20/14 Packaged R29134
8.5     (Additional ASAP Changes) Include - RS 6/20/14 RTM r30149 and r30150
8.0 ASAP: Interface Changes Kana CAS-5526 and its sub-tickets: Re-design single dish task interfaces Include - KS/JK 3/19/14 Packaged SD SVN IDs
10.3   Dirk Changes by Michel Caillat to the alma module (bug fixes to casacore). Include - JK 04/08/14 Packaged SVN IDs
10.5   Dirk Dirk's bug fixes and small features to casacore and code. Include - JK 4/08/14 Packaged SVN IDs
10.7 Wvrgcal Dirk CAS-6384:wvrgcal seems to not respect flags in some cases Include - JK 4/07/14 Packaged SVN IDs
4.0 Setjy: Bug fixes Tak CAS-5941:intent selection in setjy may have issues Include - JK 3/26/14 Packaged

SVN IDs

code: 28377, 29529

gcwrap: 27961, 28133, 28167-28168, 28170-28172, 28236, 29529, 29533

6.0 PlotMS: hardcopy per iteration for Pipeline Susan CAS-6169: Plotms-Generate multiple pages for hardcopy (Likely to merge all recent plotms changes in the process.) Include - JK 3/26/14 Packaged

SVN IDs

Merged by Susan. Thanks!

6.2   Susan CAS-6662:plotms sometimes seem to give different results for showgui = False and True Include - JK 6/16/14 Open TBD
Key:
  • RTM = Ready To Merge from trunk to branches/release-4_2_2.
  • RTT = Ready to Test on branhces/release-4_2_2 by developers.
  • Packaged = Included in a CASA 4.2.2 prerelease package, and ready for user acceptance testing.
  • Verified = Tested by a developer on branches/release-4_2_2, or in a package built from this branch.
  • Validated = Tested by a user in a package built from branches/release-4_2_2.

Explicitly Excluded Features

  • New Imaging refactor code: The CSSC requested that the weights implementation be released discretely from the imaging refactor to simplify testing.
  • Any experimental additions to the code.

Other Requirements

  • Timing: Available for ALMA Cycle 2 in June, 2014.
  • Testing
    • Typical package acceptance testing for a CASA deliverable.
    • Typical user acceptance testing for a CASA deliverable.
      • Weekly test packages for evaluation.
  • Pipeline
    • Pipeline to be packaged with CASA for this release.
    • Integration of the new PlotMS feature to generate hardcopy per iteration.
    • Other features called for in the ALMA punch-list.

Implementation / Scope

Work elements listed below are not listed in a strictly chronological order, but are intended to give an idea of the overall scope of effort required for the release.
  1. Collect desired features and fixes (requirements) from CSSC and CASA group. (See Casa422DesiredRequirements)
  2. Finalize the list of features and fixes to be included in the scope of the release. (See table above)
  3. Setup 4.2.2 branch in configuration control.
    • Defer setup as late as possible to limit work synchronizing 4.2.1 and 4.2.2 branches.
    • Scott plans to have the branch available by 4/4/14.
  4. Develop the functionality and fixes that are within the scope of the release.
    • Preferably commit changes to to 4.2.2 branch.
    • Changes already committed to trunk will have to be merged as described in the "Procedures" section below.
  5. Identify changes committed to trunk to be included in patch.
    • Identify changes by Subversion Revision ID numbers if they are to be merged by Scott.
    • Developer assistance in merging changes is appreciated.
  6. Iteratively merge existing individual changes from trunk to 4.2.2 branch, building and testing incrementally.
    1. Some automation or logging with svnmerge.
  7. Prepare typical test packages and perform package acceptance testing.
  8. Coordinate user acceptance testing when test package is sufficiently stable.
  9. Build final patch package; bundle with Pipeline.
  10. Release patch package publicly on both OSX and RHEL.

Procedures

Schedule

Gantt Chart

Milestone Summary

Milestone Target Date
Call for Requirements Friday, March 21st
Decision on Features to be Included Friday, March 28th
Release Draft Plan Monday, March 31st
4.2.2 Feature Freeze Friday, April 25th
4.2.2 Pre-Release Package (RHEL) for User Testing & Pipeline Development Friday, June 6th
4.2.2 Released for RHEL Friday, July 18th
4.2.2 Released for OSX Friday, August 6th
4.2.2 w/ Pipeline Bundle Released for RHEL Friday, July 25th
4.2.2 w/ Pipeline Bundle Released for OSX Friday, August 14th
-- RobSelina - 2014-03-28

Acceptance Testing

Package Acceptance Testing

Package acceptance testing consists of:

  1. Building CASA
  2. Running all casacore "make tests"
  3. Running smoke tests on the following to ensure they start and stop with no errors:
    • casa
    • casabrowser
    • casafeather
    • casalogger
    • casaplotms
    • casapy
    • casapyinfo - in rpm and tar tests
    • casaviewer
  4. Running all currently working CASA Guides based tests
    • irc10216 - the only working test at this time
  5. Running a large subset of CASA Regression tests as described elsewhere.
  6. Running all CASA Functional tests not commented out of unittests_list.txt as described elsewhere.

Tests are run in developer builds on EL5, and EL6 before packaging, on CASA rpm packages on EL5, and on CASA tar packages on EL5 and EL6. This is to measure CASA before packaging, and detect any issues that may appear in CASA during packaging.

Once build issues on OS X are resolved, tests will be run in developer builds on OS X 10.7, 10.8, and 10.9, and on CASA dmg packages on OS X 10.7, 10.8, and 10.9.

Test failures are to be reported and tracked in JIRA tickets, and listed below to provide a summary of release issues.

Test Failures

Failure First Seen Who Fixed Comments
CAS-5935 - Crash at CASA exit EL5, EL6 developer builds not assigned    
CAS-6619 - 3C219D_regression.py fails on the CASA 4.2.2 release branch EL5, EL6 developer builds Kumar    
CAS-6623 - accum_regression.py fails on the CASA 4.2.2 release branch EL5, EL6 developer builds George    
CAS-6625 - cvel+clean_regression.py fails on the CASA 4.2.2 release branch EL5, EL6 developer builds Dirk    
CAS-6627 - wf_regression.py fails on the CASA 4.2.2 release branch EL5, EL6 developer builds not assigned    
CAS-6628 - wideband_regression.py fails on the CASA 4.2.2 release branch EL5, EL6 developer builds Urvashi    
CAS-6632 - test_cleanhelper.py fails on the CASA 4.2.2 release branch EL5, EL6 developer builds Scott casa-data/branches/release-4_2@9893  
CAS-6633 - test_csvclean.py fails on the CASA 4.2.2 release branch EL5, EL6 developer builds Kumar    
CAS-6634 - test_flagdata.py fails on the CASA 4.2.2 release branch EL5, EL6 developer builds Sandra casa-data/branches/release-4_2@9892  
CAS-6636 - test_ia_pv.py fails on the CASA 4.2.2 release branch EL5, EL6 developer builds David    
CAS-6637 - test_imfit.py fails on the CASA 4.2.2 release branch EL5, EL6 developer builds Scott casa-data/branches/release-4_2@9894  
CAS-6638 - test_importasdm.py fails on the CASA 4.2.2 release branch EL6 developer build Michel    
CAS-6639 - test_imregrid.py fails on the CASA 4.2.2 release branch EL5, EL6 developer builds Scott casa-data/branches/release-4_2@9895  
CAS-6640 - test_mstransform.py fails on the CASA 4.2.2 release branch EL5, EL6 developer builds Justo    
CAS-6641 - test_simobserve.py fails on the CASA 4.2.2 release branch EL5, EL6 developer builds Scott casa-data/branches/release-4_2@9896  
CAS-6642 - test_simplecluster.py fails on the CASA 4.2.2 release branch EL5, EL6 developer builds Justo    
Notes:
  • Failure - a link to the JIRA ticket describing the failure. Text of the link must include the ticket number and title.
  • First Seen - developer build or package, and platform.
  • Who - "not assigned" | who is working on or has worked on the ticket.
  • Fixed - blank | svn revision including the fix | prerelease package including the fix.
  • Comments - anything useful.
Topic attachments
I Attachment Action Size Date Who Comment
CASA422Patch-Plan-2014-03-29v02.pdfpdf CASA422Patch-Plan-2014-03-29v02.pdf manage 106 K 2014-05-22 - 15:40 RobSelina CASA 4.2.2 Release Schedule (Gantt Chart)
Weights.pdfpdf Weights.pdf manage 835 K 2014-03-29 - 18:19 RobSelina Weights implementation plan
Topic revision: r70 - 2014-08-11, RobSelina
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback