CASA User Testing Meeting Minutes - 2015-06-16, 09:00 Socorro / 11:00 Charlottesville

Post-meeting minutes notes appended in blue text.

Agenda

Participants

Present: Mark R., Crystal, Anand, Juergen, Steve

Apologies: N/A

Locations

https://staff.nrao.edu/cal/RoomListing.htm

  • Socorro: AOC-317; IP: 146.88.6.8
  • Charlottesville: Room 331; IP: 192.33.117.19


Minutes (Main Points Only)

Status

  • A 4.4 release for all platforms is expected this week!
  • 4.4 feature freeze officially passed. Test/stable/monthly packages are being produced for RHEL5 and 6.
  • OS X test packages production ongoing (for 10.10 and 10.9) *Pre-release packages currently being used for user testing. OS X pre-release packages now available at: https://svn.cv.nrao.edu/casa/osx_distro/prerelease/10.10/
  • 4.3.1 is the current public release version for both Linux (RHEL5 and 6) and OS X (10.8). 4.2.2 is still currently being used for pipelined ALMA data reduction at the NAASC, but 4.3 has now been adopted for "manual calibration" reductions.
    • The same general-purpose CASA launcher script has now been deployed at CV, SOC and GB. Details are on the appropriate drop-down menu of the public CASA pages at http://casa.nrao.edu
      • For anyone using test machines zuul01, 03 and 04 are running RHEL6, zuul02 is running RHEL5. REMEMBER: Use /usr/bin/casa on these machines to make sure that you are running the new test/stable packages (there are RPM installations on these).

Number of CASA Jira tickets with "Ready To Test"/"Under Test" status (including EA, EU, JAO, Software-only tickets, but excluding Pipeline) prior to review: 71.

Number of CASA Jira tickets with "Ready To Test"/"Under Test" status (including EA, EU, JAO, Software-only tickets) following review/meeting: 54.

Testing Tracking

  • 4.4 feature freeze deadline passed. No new features can be added to 4.4 from now on - bug fixes only. 4.4 pre-release branch publishing regularly; user testing to be conducted on the 4.4 stable packages for now.
  • New ticket assignments.
  • The "nag list", "input required" items
  • Major test coverage areas for 4.4 pre-release testing
    • Reworked statistics framework - check for changes in behavior. Peter Teuben (plus?) See CAS-7034 Mark R. has repeatedly hassled Peter about the testing of this. Since he led the group that requested this, they should probably also set requirements criteria for associated regression testing as well.
    • PlotMS - Designated tester: Drew Brisbin
      • New PlotMS code implementing VI/VB2 and on-the-fly calibration. Related tickets (with revision 32835+) are: CAS-3034, CAS-4864, CAS-7117, CAS-7118 (already resolved), CAS-7119 (already resolved), (CAS-7386 (already resolved) and CAS-7393 (already resolved) (see below)
      • Besides testing the new functionality, including on-the-fly calibration and plotting sigma and sigma spectrum, please put PlotMS through its paces and make sure none of the old functionality was broken as a result of the underlying code change.
      • Check averaging and flagging. The Pipeline group will also be looking for any such issues.
      • Remaining related issues for release 4.4 included the creation of virtual model data when there is no MODEL_DATA column or SOURCE_MODEL column (CAS-7390 (already resolved) ), and improving the performance of the VI/VB2 iteration through the data when averaging (CAS-7412 (already resolved) ).
      • Also, regarding testing the OTF calibration: A callib parameter has been added to the plotms task in order to use this feature, e.g. plotms("ngc5921.ms", callib="callib.txt", ydatacolumn="corrected") where callib can be a filename or a python dictionary. PlotMS now has a Calibration tab where the OTF calibration can be enabled/disabled, but callib can only be set through the casapy interface for this release.
      • Any other issues should be reported.
    • Viewer: Amanda has been doing user testing of this recently.
    • Weight column (including weight column -> initialized weight spectrum). See CAS-4469. Testers to work with George. Steve was to look into this.
    • New ATM library in 4.4 (CAS-7220). Should be (more or less) a drop-in replacement: testers just need to watch for any oddities during routine use (Already resolved).
    • For a limited group: HPC work (e.g. number of open files, etc.). Pipeline group are also involved in testing this. Probably does not need a lot of dedicated science user testing yet, beyond routinely trying to use split2, day-to-day MSTransform-type work, etc. Amanda had been looking into testing routine split2, cvel2 and other MS_Transform-related issues.
  • Any Other Business:
    • There was a lot of discussion about overall testing strategy and priorities for the 4.5 development cycle. In a nutshell:
      • CASA 4.5 CANNOT BE LATE! IT IS NEEDED FOR THE NEXT ALMA CYCLE, WHICH BEGINS IN OCTOBER!
      • The tclean testing probably needs to be handled as a short-term project all of its own. In principle, a strategy has been agreed upon for ALMA between Jeff & Crystal, and a similar approach is probably needed for the VLA testing effort. An ALMA Pipeline testing package intended purely for internal NRAO testing use only has been put together that makes use of tclean. It is expected that the Pipeline people in Europe will be comparing manual and the new basic automatic imaging. The PA Pipeline people will be performing the "deep dive" testing (e.g. terms:2 at B3, etc.).
      • For the 4.5 CASA release, it is anticipated that one or two CASA Guides will be produced that essentially outline "This is how you do it with the old clean, and here is the tclean equivalent."
      • It will be necessary to ensure that the clean regressions are shown to work with tclean (modulo any appropriate tweaking).
      • Crystal has been promised access on lustre to a CASA 4.5 test/stable package containing tclean that she will be able to run and start testing in earnest by no later than June 29th (the next imaging group meeting will take place on June 30th). The aim is to probably be able to try to make sure it works well enough for ~5 or 6 of the most common-use cases. Action on Mark R. to remind everyone of this near-term deadline at the CASA Monday Morning Meeting.
      • Crystal thinks that Urvashi may have some concerns about how tclean currently deals with reference frames.
      • It may be necessary to set up a dedicated parent ticket with systematic subtasks specifically to track tclean testing efforts. For example, we will need to test multiscale, for single field, mosaics (both forms: heterogeneous beams/A-projection). Moreover, the questions to be addressed will be: (1) Does the user interface work as expected? and (2) Does it actually do the right thing?
      • Mark R. does not need to lobby the NAASC for specific additional effort for this: Crystal has the appropriate experts in her imaging group already. Mark R. may need to provide a Jira framework for testing tracking for them, though. tclean testing will not really begin in earnest until June 30th (at which point, it will then need to be "full steam ahead").
      • Parallelization is also expected to feature more prominently in CASA 4.5. It would be nice if this could be relied upon to help speed up flagdata. Presumably, Pipeline will care quite a lot about this. However, our most immediate concern is that it is not responsible for additional breakage, particularly if such breakage makes it more difficult to test/debug any tclean issues.

PART 1: Items to be Covered During Meetings (as time permits)

Propose to Close Items

  • CAS-7515 (May 2015): Test setjy source model flux scaling fixes for 4.4. release. 4.4 Blocker item.. Under Test. Tester: Juergen. Provisionally resolved as fixed.
  • CAS-3580 (Aug): 4.4 item. Prediction of expected image based noise based on Tsys. Under Test in 4.3. Tester: Eric V. (to be tested at the JAO). Kept open at Remy's express request. Recent testing continuing by Eric. Tracked elsewhere(?), so would like to resolve. Passed over to Pipeline group to discuss.
  • CAS-6707 (Sep): mstransform problems with CALDEVICE and SYSPOWER sub-tables when creating an MMS. Under Test in 4.3, but 4.4 item now. Tester: Previously passed to Steve for VLA user tester assignment. Previously discussed at length: Feature was already shipped in 4.3. To resolve or not to resolve (that is the question)? Steve had talked to James Robnett to get a sense of the status of parallelization. May be more of a developer/Pipeline testing item? Ticket pinged. Provisionally resolved (as shipped with 4.3 and still not tested in time for the 4.4 release). Reopened and passed to Pipeline group.

Items Overdue for Status Updates (i.e. the "Nag List")

  • CAS-7488 Support single dish data transformations (time/chan avg) in plotms/mstransform. Last remaining 4.4 Blocker Currently Under Test. Tester: Juergen. Ticket pinged.Passed to Erik Muller for NAOJ testing.
  • CAS-7318 (Feb): 4.4 item. Verify unpolarized calibrator mode for linear basis (ALMA). Under Test. Tester: Steve. Ticket pinged. Steve and George to nag each other about this one. Ed doesn't know what it's about... Ticket pinged repeatedly. Just pushed back to 4.5.
  • CAS-7034 (Mar 2015): Development of new Common Statistics Framework. Under Test now in current stable package. Tester: Peter Teuben. Sounds like it largely works so far, but a possible issue has been identified with half-fit during full cube access. Fix put in: confirmation awaited. Ticket pinged.
  • CAS-7109 (Mar 2015): Linear full mosaics. Available in stable now. Under Test. Tester: Steve. Ticket pinged. Just pushed back to 4.5.
  • CAS-4983 (Mar 2015): frequencies in gain tables derived using combine='spw' are incorrect. 4.4 item. Available in stable now. Under Test. Tester: was Claire, now Crystal. Some duplication with CAS-7097 (assigned to Claire). Claire reports: "Have tested gaincal with solving for individual spws, looks OK, labels for delays now match complex gain, at least in plotcal plots (used to be edge frequency instead of center frequency). Still have to test with averaged spws."
  • CAS-6385 (May 2015): simobserve listobs file has an incorrect intent? Under Test. Tester: Catarina. Ticket developer pinged.
  • CAS-7522 (May 2015): channel conformance error in MS reported by viewer. Under Test. Tester: Juergen. Ticket pinged.
  • CAS-4283: implement ia.continuumsub()/imcontsub using ImageProfileFitter. RTT. Available in the current stable package now. Tester: Catarina. Help requested from developer for testing (dev nagged).
  • CAS-7517 (May 2015): plotms plots WTSP even when that column is not initialized. Under Test. Tester: Crystal
  • CAS-7544 (May 2015): importfits fails for double precision data when defaultaxes=True. 4.4 item. Under Test. Tester: Brian M.
  • CAS-7243 (June 2015): plotms crashes with combination of adding plot and closing GUI. 4.4 item. Reopened and Under Test (again). Tester: Juergen
  • CAS-7185 (June 2015): reset rest frequency in viewer to original value. 4.4 item. Reopened and Under Test (again). Tester: Juergen
  • CAS-7608 (June 2015): viewer GUI testing CASA 4.4. Mac 10.10. 4.4 item. Under Test (again). Tester: DA (ticket passed to Anand)
  • CAS-7435 (Jun 2015): mstransform sets all antenna indices to zero. 4.4 item. Tester: Josh M. (Reassign?)
  • CAS-7617 (May 2015): (Viewer) Profiler should post warning rather than segfaulting with single channel fit. 4.4 item. Tester: DA (ticket passed to Anand)

Other Items Currently "Under Test"

  • PlotMS Items Drew Brisbin is the science user tester assigned to work closely with Pam on this during the next couple of weeks or so..
  • CAS-7314 (Mar 2015): Add capability to average across fields. Officially Under Test already, but needs the VI/VB2 version of PlotMS to test, so this would be an advance assignment. Question raised during previous meeting: What is the intended science use case for this? Answer: Different observations of the same field source might carry different field IDs. May also have mosaicking applications. Tester: Steve Ticket pinged. Just pushed back to 4.5.
  • CAS-7227 (May 2015): better reporting of tabular spectral axes. 4.5 item. Under Test. Tester: Remy. Ticket pinged.
  • CAS-7573 (May 2015): imval should work on pV images. 4.5 item. Under Test. Tester: Juergen.
  • CAS-7531 (May 2015): poisson fitter for 2d histogram does not appear to be working. Ready To Test, pending a new 4.5 package. Tester: Amanda
  • CAS-7485 (May 2015): Modify SubImageFactory interface. 4.5 item. Under Test. Tester: Erica. Ongoing.
  • CAS-6232 (Mar): writing version numbers in GUIs. Under Test in 4.3. Most recent commit to trunk was r29373. Tester: Juergen. However, this also needs to be checked in the table browser, the work on which is currently unscheduled. Spawned another ticket CAS-7033) for its underlying cause (just pinged).
  • CAS-7589 (Jun 2015): ImageFitter segfaults with empty estimates file. 4.5 item. Tester: DA (ticket passed to Anand)
  • CAS-7620 (Jun 2015): ImageProfileFitter segfaults for a small channel range. 4.5 item. Tester: DA (ticket passed to Anand)
  • CAS-7621 (Jun 2015): Residual image not correct when fitting a single change of an image cube with a 2D fit. 4.5 item. Tester: DA (ticket passed to Anand)

"Ready To Test" Items Pending Tester Assignment

  • CAS-7149 (Jun 2015): mstransform(chanaverage=True) gets the weights wrong if spws differ in number of channels and chanbin is set to the maximum number of channels (instead of a vector). 4.4 item. Tester: Amanda(?)
  • CAS-7577 (May 2015): location problems in 4.4 pre-release plotms. 4.4 item. Tester: TBD (DA?) Works - resolved.

  • CAS-7628 (Jun 2015): msmd.scansforintent returns all matching scans in a multi-obsid dataset, regardless of the obsid parameter. 4.5 item. Tester: DA
  • The below are all tclean related. Testing of these will commence in a couple of weeks, and will be handled as per the above AOB notes.
  • CAS-5048 (May 2015): Clarify when peak residual calcs use/ignore masks, and use consistent thresholding rules. Older ticket, now ready to be tested using new tClean in 4.5. Tester: TBD (was Emmanuel, back in the 4.1 cycle; Amanda?).
  • CAS-3953 (May 2015): Subtraction / Fixing of outlier fields in clean. Subtask of CAS-6622. Ready to test in tClean via interactive viewer. Tester: TBD
  • CAS-3729 (May 2015): clean to write out only those images that are necessary for a given run. Subtask of CAS-4640. Tester: TBD
  • CAS-3290 (May 2015): Logging : clean logging for cube cleaning needs improvement. Subtask of CAS-6615. Tester: TBD
  • CAS-4133 (May 2015): clean inline docs for masks and new region format... Subtask of CAS-6617. Tester: TBD (Brian Kent, who has history with this ticket? Or DA?)
  • CAS-1623 (May 2015): CLEAN should have option to continue from previous CLEAN state. Subtask of CAS-4640. Tester: TBD

  • CAS-2673 (May 2015): The 'cyclefactor' parameter in the clean task. Subtask of CAS-6622. Tester: TBD (Is this really user testable? Maybe not...)

Single Dish (to be skipped during the meeting)

  • See SD ticket list below

Input Required

  • CAS-7036: split in 4.3 stable (4.2.21) now produces .flagversions, need flagbackup parm. Ticket with devs, but Steve has been providing input. Discussion ongoing.
  • CAS-4469 (Jun+ 2013): Split with channel averaging is creating spurious spectral features and weight is wrong when there has been channel flagging. Subtask of CAS-5245. Critical item. was "Under Test". Tester: Crystal. Some stuff works. Ticket back with developer now. Extensive comments provided by George, now "Input Required" from Crystal.
  • CAS-5806 Mathematical prediction of post-gridding single dish beam. Formerly a 4.3+ Blocker, now "Critical". "Input Required?* Developer has done some modelling, but... ? This is a "blocker of a different kind..." High priority (blocking further development). Currently Input Required from Kana. Crystal and Kumar had been in discussion, but this ticket has been quiet since (effectively) December. Post-4.4 item now.
  • CAS-4735 (Nov): imsmooth needs additional option for single dish use. Subtask of CAS-5844 (above). Open again. Tester: Tsuyoshi. Critical item. Code committed to trunk. 4.3 item. Some testing done, mostly works as intended. Input Required from Crystal for some remaining changes. Any remaining future work will have to be for post-4.4.
  • CAS-6124: imfit subtasks:
    • CAS-4538: migrate imfit input estimates from file to input parameters. Input Required (was "On Hold"). Subtask of CAS-6124. Ticket passed to Juergen, pending a CSSC discussion of what needs to be done and a scheduling of the work. Pushed back to 4.5. Ticket pinged. Juergen: More thought about imfit still needed by Science.
  • CAS-4748 (Dec 2012): Polarization selection does not select SIGMA. Input Required. Passed to Jeff for prioritization. Sanjay thinks it's a low-priority item. Version 4.5+.
  • CAS-5886 (Dec 2013): bizarre clean boxes for large images. Awaiting input from one developer to another. Version 4.5+.
  • CAS-2825 Support for Lissajous mapping. Pushed back to 4.5+. Input from Tsuyoshi needed, but he is not able to provide it yet.
  • CAS-5162 (Sep 2013): Improvements in flagging documentation and examples. Input Required from Science (via Jeff). This ticket has been left untouched since May 2013... Version 4.5+.
  • CAS-6794 clean is very slow handling this simple, small ACA mosaic. Input Required from Juergen. Kumar commented on this.

Other Blockers

    • CAS-6688 Turn on new vi/vb within imager and test. Was a 4.4 item, but Unscheduled. Currently with Jim. Some scope modification suggested for this ticket.
  • Viewer registration tickets:
    • CAS-6560 contour image does not unzoom when master coordinate image is re-registered. Subtask of CAS-6604. Viewer issue. Currently Unscheduled.
    • CAS-6720 2 panels don't register. Viewer issue. Currently Open.

Additional work-in-progress 4.5 Blocker items to be added in the future as appropriate.


PART 2: Items Listed for Informational Purposes (note that these will not normally be specifically discussed during the meeting unless someone raises them)

Unable to Test Items

  • CAS-4758 (Sep): 4.4 item. fluxscale fails if one of the reference fields is flagged. Previously "Under Test" (r31121+) in current (new-style) stable. Tester: Claire. No testing progress on this since October. Claire had been trying to find a suitable dataset for testing. Steve thought that this should actually be fairly easy to test and that Claire and/or he may have been able to test this one. The previous suggestion was to ask Drew M. about this, and (if appropriate) reassign it to Brian Kent. Ticket pinged. Place On Hold, pending test data availability. Claire reports: no updates.
  • CAS-5379 - Subtle inconsistencies with other programs in Galactic Coordinates (Adam and Harvey). In a nutshell, shifts of ~0.2" are seen when taking data into other packages (e.g. DS9, IDL which both seem to agree with each other). It is unclear yet if this is an incorrect implementation in one or more packages, or possibly just a precision error. This is a subtle issue, and was pushed back. Ticket stalled. Needs to be looked at by someone with lots of time. To be left as-is for now. This needs significant work, including new science papers on how to actually define the Galactic coordinate system? Needs casacore discussion; this can be part of that. Passed to Jeff for now. Pushed back to some indefinite future release. Unscheduled.
  • CAS-4636 (Oct 2012): Elevation dependent opacity application w/ Tsys. Still awaiting input from Science. Had been "On Hold" since August, awaiting data. Previously switched to "Input Required" and passed to Crystal for comment. Bumped back to 4.4+. Work on this is Unscheduled until further notice (more than a year, now).
  • CAS-3532 (Sep): Incorporation of Alma Primary Beam in Mosaic. Currently Unscheduled. Tester was Scott S., but ticket passed to Crystal for input. Some test imaging done. Ticket with developer. Pushed back to 4.5.

Upcoming Ready to (Re)Test Items

Not Yet Assigned To Testers

  • TBA

Single Dish

See below "assigned" SD list.

Already Assigned To Testers

  • CAS-4592: plotms on 10.7 issues. Previously half fixed due to Qt updates. Was "ARTT" in 4.2.1 and 4.2.2 pre-release. Was "Under Test" , shown to till be an issue under 10.10. Tester: was Nathan, to be Juergen. Exported plots look OK, only the plots in the GUI appear corrupted.
  • CAS-6951 (Oct): 4.4 item now. mstransform creates a 4-polarization DATA column for single-polarization ALMA WVR data, and a 1-polarization DATA column for four-polarization ALMA data. Fix already included in current new-style test/stable packages newer than 4.2.23. Scheduled again now. Tester: Todd. Ticket currently back with developer(s) again.
  • CAS-5893 (May): Now a 4.5 item. Add the ability to write a spectral profile to an ASCII file in the ia tool. Was "Under Test" in 4.3. Now Scheduled. Tester: Assigned to Juergen for testing and a possible CSSC discussion. Might also need a task adding for this, but this will be 4.5 work. Recent testing done, passed back to developer for 4.5 work.
  • CAS-5127 (May): Document ia toolkit methods. Status Scheduled. Mark should start working on this again... Most of Juergen's prioritized list of 26 items had been done. Dates to be added his future documentation updates. Ongoing. ia.regrid docs will require more work. Still needs code commits. Mark to restart work on this under Linux soon.

Single Dish

  • CAS-6572: (Aug 2014): A new baseline task that handles baseline table. Under Test in latest 4.3 casapy-test/new-style stable packages. Mostly done. Ticket with Kana. Some duplication with CAS-5855). Ticket previously pinged.

-- MarkRawlings - 2015-06-15
Topic revision: r5 - 2015-06-16, MarkRawlings
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback