CASA User Testing Meeting Minutes - 2015-10-06, 09:00 Socorro / 11:00 Charlottesville

Post-meeting notes are in blue text.

Agenda

Participants

Present: Mark R., Amanda, Steve, Juergen

Apologies: TBA

Locations

https://staff.nrao.edu/cal/RoomListing.htm

• Socorro: AOC-317; IP: 146.88.6.8
• Charlottesville: Room 331; IP: 192.33.117.19

---

Minutes (Main Points Only)

Status

• 4.4.0 has been released for all supported platforms.
• 4.5 code freeze was scheduled for August 17th 2015. The 4.5 pre-release branch was created at this point. Test/stable/monthly packages are also still being produced.
  • The same general-purpose CASA launcher script has now been deployed at CV, SOC and GB. Details are on the appropriate drop-down menu of the public CASA pages at http://casa.nrao.edu
    • For anyone using test machines zuul01, 03 and 04 are running RHEL6, zuul02 is running RHEL5. REMEMBER: Use /usr/bin/casa on these machines to make sure that you are running the new test/stable packages (there are RPM installations on these).
• 4.6 test and stable packages currently being produced routinely.
• 4.3.1 ALMA and VLA Pipeline version released (RHEL and OS X links posted). Pipeline major release number expected to catch up with the rest of CASA at 4.5.

• Planned 4.5 release timeline:
  • Aug 3: Feature Freeze (all new features are now in the codebase and ready for test) <- PASSED
  • Aug 14: Code Freeze (Bug fixes etc are expected to be in the trunk at this point, we should have all regressions passing as well) <- PASSED
  • Aug 17: Release Branch Created <- PASSED
    • Feature tests complete: obviously detected issues will need to be fixed and retested, but everything should have been at least had one look
  • Aug 17-31: <- PASSED
    • Identified defects are being fixed, documentation generated
    • All commits require authorization
    • First round of GUI testing
  • Aug 31: First release candidate available for random user testing <- PASSED
    • Final spot check of GUIs
  • Sept 9: Release Target <- PASSED

Please conduct all remaining 4.5 user testing with 4.5 pre-release packages whenever possible.

Number of CASA Jira tickets with "Ready To Test"/"Under Test" status (including EA, EU, JAO, Software-only tickets, but excluding Pipeline) prior to review: 62.

Number of CASA Jira tickets with "Ready To Test"/"Under Test" status (including EA, EU, JAO, Software-only tickets) following meeting: 55.

Testing Tracking

• 4.5 Blockers
• New ticket assignments.
• The "nag list", "input required" items
• Major test coverage areas for 4.5 testing
  • tclean testing to be handled as a short-term project all of its own. Strategy was agreed upon for ALMA between Jeff & Crystal.
    • VLA tclean testing report from Claire:
      • gaincal gets different results when I use tclean with savemodel='virtual" vs. savemodel='modelcolumn', with the virtual model apparently worse. If plot both models in plotms as amp vs. uvdist they look the same, but there must be something different about them as far as gaincal is concerned.
      • Help documentation for savemodel seems to be a placeholder, needs to be completed.
      • Unclear what the parameter "overwrite" does based on inline and other help. One would imagine that overwrite=T would overwrite the existing images, but in fact is uses them and starts from the existing model, which is not at all clear from the documentation.
    • Dedicated parent tickets with systematic subtasks specifically to track tclean testing efforts have been set up: CAS-7664 and CAS-7669. The main questions to be addressed will be: (1) Does the user interface work as expected? and (2) Does it actually do the right thing?
    • An ALMA Pipeline testing package intended purely for internal NRAO testing use only has been put together that makes use of tclean. It is expected that the Pipeline people in Europe will be comparing manual and the new basic automatic imaging. The PA Pipeline people will be performing the "deep dive" testing (e.g. terms:2 at B3, etc.).
    • For the 4.5 CASA release, it is anticipated that one or two CASA Guides will be produced that essentially outline "This is how you do it with the old clean, and here is the tclean equivalent."
    • It will be necessary to ensure that the clean regression tests are shown to work with tclean (modulo any appropriate tweaking).
    • Crystal et al. have access on lustre to a CASA 4.5 test/stable package containing tclean that she was to begin testing from June 29th. The overall aim is to at least ensure that it works well enough for ~5 or 6 of the most common-use cases.
    • Crystal previously reported that there were actually three different implementations of the Clark clean algorithm in CASA, and due to contrasting results when attempting to clean mosaics and single fields, two of these have now been linked in for tclean (old Clark -> "clark2" in tclean (suitable for single field); Clark -> "clark" in tclean (suitable for mosaics)). Using the inappropriate one appears to generate deep negative holes in images, and it is suspected that the two implementations were over-optimized for specific cases. Steve had also noted that tclean seems to exhibit problems with restarts (e.g. first run with a mask, then second without).
  • Parallelization is also expected to feature more prominently in CASA 4.5. It would be good if this could be relied upon to help speed up flagdata, although our most immediate concern is that it is not responsible for additional breakage (particularly if any such breakage makes it more difficult to test/debug any tclean issues).
    • For a limited group: HPC work (e.g. number of open files, etc.). The Pipeline group are also involved in testing this. This probably does not need a lot of additional dedicated science user testing yet, beyond routinely trying to use split2, day-to-day MSTransform-type work, etc. Amanda had been looking into testing routine split2, cvel2 and other MS_Transform-related issues.
  • Ephemeris handling
  • Weight column (including weight column -> initialized weight spectrum). See CAS-4469 (now resolved). Testers to work with George. Steve was to look into this.
• Other matters previously arising
  • From Claire: plotms: plotting the virtual model column results in a message ?: MTFTNew :: fromRecord" being printed on the terminal but not in the log. It should probably be in the log only (if at all, as it's not obvious to the user what it means).
  • It had previously been decided that a wiki page would be created for the NA Imaging Team to collect together a list of common use cases to be tested. Socorro staff were then to review the list, and add any VLA-specific entries for which they also see a need. Juergen was to provide a copy of his old plotms testing automation targets to use as a starting point.
  • BDF flagging has been shown to be problematic. Fixes are being put in by Michel, but it has been an ongoing issue. See PRTSIR-6800.
  • Anand was to ask Josh to upgrade the remaining DA OS X 10.8 Mac to 10.9 for user testing purposes.
  • Someone (ahem...) really needs to get going on the tools documentation again soon. The documentation on the imager (in particular) is just plain wrong in places.

• Any Other Business:
  • TBA

PART 1: Items to be Covered During Meetings (as time permits)

4.5 (Non-Pipeline) Blockers

• CAS-7664 tclean science user testing: Continuum (MFS) <- tclean parent ticket. No longer a blocker ticket? Ticket status is marked as "Scheduled" and assigned to Urvashi, but Dirk Muders has been doing testing of this.
• CAS-7870 Multiple beams in cube for restoringbeam='common'. Under Test. Tester: Provisionally Amanda for Science Testing (Dirk M. did some software-level testing) Mark R. has pinged the ticket watchers for suggestions of suitable test data in the meantime.
• CAS-7831: plotms no longer correctly plots the virtual model. Under Test. Tester: (Was Catarina). Virtual model stuff generally flagged as a possible cause for concern. VI/VB interactions? Some fixes put in for 4.5. Workaround noted in 4.5 known issues; Juergen doesn't want to hold up the 4.5 release for this item, though. Bumped to be a 4.6 blocker accordingly.
• CAS-7856: viewer image display is strange in 4.5.0-REL. Currently listed as Scheduled but fix works in 4.6 stable. Is the fix included in (rep)release as well yet? If so, this can be resolved. Confirmed to be in the release following the meeting. Resolved as fixed.

"Ready To Test" Items Pending Tester Assignment

• CAS-7387 (Sep): 4.6 item. Running split by intent can create a non-zero field ID in the resulting ms, and then clean(field='') uses the wrong field coordinates (from field zero). Ready To Test. Tester: TBD (originally reported by Todd) Ticket parked with Mark R. Passed to Todd.
• CAS-7744 (Jul): 4.5 item. Baseline-based types unflag data on apply. Under Test. Former blocker, now demoted. Tester: TBD (was Crystal); Fix confirmed to work for blcal. Still needs to be tested for the chan-independent cross-hand phase case. Still need a tester with a suitable data set for this one... Initially passed to George to ask if he's already confirmed that the fix works for the remaining pol case. If he has, he can resolve the ticket. If not, he will reassign it to Juergen for follow-up.
• CAS-1896 (Sep): clean needs to be more efficient for large cubes. 4.6(?) EVLA item. Subtask of CAS-6620. Ticket currently with Jeff. Tester: TBD Steve proposes closing this one as obsolete, and tracking subsequent work via new, more appropriate tickets.
• CAS-3772 (Sep): add theoretical noise to clean output. 4.5(?) item. Subtask of CAS-6615. Tester: TBD (Original ticket filed by Juergen) One for the imaging team. Left with Urvashi as RTT for now.

• CAS-7925 (Sep 2015): 4.6 item. the help example for ia.rotate is misleading. Under Test. Tester: TBD (DA - ticket passed to Anand for reassignment).
• CAS-3125 (Sep 2015): 4.6 item. plotms with iter and selfscale turned on does not unzoom correctly. Under Test. Tester: TBD (DA - ticket passed to Anand for reassignment).
• CAS-7830 (Sep 2015): 4.6 item. plotms error: Thread::background() not yet implemented. Under Test. Tester: TBD (DA - ticket passed to Anand for reassignment).
• CAS-7851 (Sep 2015): 4.6 item. Restart casaplotms process when it crashes. Under Test. Tester: TBD (DA - ticket passed to Anand for reassignment).

Propose to Close Items

• TBA

Items Overdue for Status Updates (i.e. the "Nag List")

• CAS-7109 (Mar 2015): Linear full mosaics. Available in stable now. Under Test. 4.5 item (former 4.4 item. Testing started, Input requested from developer. Fix put in by Kumar for problem found. Tester: Steve. Ticket pinged repeatedly. Steve has been using this, and noted that it works, and gives something that looks OK. The ticket can't quite be closed yet: he wants to be able to demonstrate that what it does is at least consistent with the old version first. Old clean is now broken, so he'd have to engineer a suitable comparison target.
• CAS-4983 (Mar 2015): frequencies in gain tables derived using combine='spw' are incorrect. 4.5 item (formerly 4.4). Available in stable now. Under Test. Tester: was Claire, now Crystal. Some duplication with CAS-7097 (assigned to Claire). Claire reports: "Have tested gaincal with solving for individual spws, looks OK, labels for delays now match complex gain, at least in plotcal plots (used to be edge frequency instead of center frequency). Still have to test with averaged spws." Crystal had felt that Claire's testing so far was already almost sufficient. While the remaining testing should be done before the release, it is probably less urgent than other issues right now. Unlikely to get additional final testing done for 4.5 release now...
• CAS-7314 (Mar 2015): Add capability to average across fields. Officially Under Test already, but needs the VI/VB2 version of PlotMS to test, so this would be an advance assignment. Intended science use case for this: Different observations of the same field source might carry different field IDs. May also have mosaicking applications. 4.5 item (was 4.4). Tester: Steve. Ticket pinged repeatedly. Steve will do this.
• CAS-3953 (May 2015): Subtraction / Fixing of outlier fields in clean. Subtask of CAS-6622. Ready to test in tclean via interactive viewer. VLA item. Minor issue. Tester: Steve. Ticket pinged repeatedly. tclean item. This will be reassigned to the tclean/imaging team.
• CAS-6215 (Jul): 4.5 item. data averaging in flagdata. Under Test. Tester: was Emmanuel, passed to Steve. Ticket pinged repeatedly. Reassigned to Juergen.

Other Items Currently "Under Test"

• CAS-6130 (Jul): 4.5 item. Parametrized tables in cal library. Under Test. Blocker. Tester: Steve. Following recent discussion, provisionally pushed back to be a 4.6 item, as Steve doesn't think it constitutes a blocker for a 4.5 release.
• CAS-7778: msmd should have a method to retrieve the effective bandwidth or the effective resolution of the channels rather than only their formal bandwidth. 4.6 item. Under Test. Tester: Juergen Reassigned to Todd (Pipeline).
• CAS-7637: msmd should have methods to retrieve REST_FREQUENCY and TRANSITION from the SOURCE table. 4.6 item. Under Test. Tester: Juergen Reassigned to Todd (Pipeline).
• CAS-4663: specfit components/prompt/inf. 4.6 item. Under Test. Tester: Juergen. Some testing done. Seems to basically work on trunk. Ongoing?
• CAS-3366 (Sep): 4.6 item. listfile parameter for imstat. Under Test. Tester: Juergen
• CAS-4735: imsmooth needs additional option for single dish use. Subtask of CAS-5844. Under Test. Tester: Amanda
• CAS-7562 (Sep): 4.5(?) item. Write model to MS in parallel tClean. Ready To Test. Passed to Jeff for reassignment (probably to one of the parallelization folk?). Ticket pinged.
• CAS-5893 (May): Now a 4.5 item. Add the ability to write a spectral profile to an ASCII file in the ia tool. "Under Test" again. Tester: Juergen
• CAS-6896 (Sep): immath should give control over file used for header generation. Under Test. Tester: Juergen
• CAS-7888 (Sep): uvcontsub with excludechannels=T selects wrong subset of channels. Under Test. Tester: Juergen Resolved as fixed.
• CAS-7759 (Sep): viewer does not plot spectrum. Under Test. Tester: Juergen
• CAS-7875 (Sep): glitchy MODEL column in plotms when averaging is used. 4.6 item. Under Test. Tester: Brian M.
• CAS-5996 (Sep): immath unit- (and if relevant beam-) awareness. Under Test. Tester: Remy
• CAS-7837 (Oct): imsmooth produces strange warning message. 4.6 item. Under Test. Tester: Claire.

Single Dish (to be skipped during the meeting)

• See SD ticket list below

Input Required

• CAS-7873 radial velocity - support for polynomial representation. Input Required from Bryan B.
• CAS-2091 casaplotms resets some plotting options, e.g. colorize by every time a data selection changes. Input Required from Lindsey. Very old ticket - may be obsolete?
• CAS-5597 CRTF doco improvements. Input Required from Mark R. Effectively blocked by CAS-7704 (currently assigned to Mark R.).
• CAS-7891 Improvements to baseline/spectral line fitter. Input Required from Erik M.
• CAS-6112 Specfit output. Input Required from Anita.
• CAS-5407 immath should reorder degenerate axes. Input Required from Juergen.
• CAS-7649 clean in interactive mode should not require a clean box for the main field if ran with outliers. Input Required from Juergen.
• CAS-5346 Reviving ia.histograms(). Input Required (still?) from Urvashi
• CAS-7927 ia.coordmeasures frequency shifted. Input Required from Dirk P.
• CAS-5583: Frequency in exportuvfits output. Currently with Brian K., who is liaising with the user in question. No recent feedback from the requesting user, though (since the ticket was created back in Sep. 2013).
• CAS-5806 Mathematical prediction of post-gridding single dish beam. Formerly a 4.3+ Blocker, now "Critical". "Input Required?* Developer has done some modelling, but... ? This is a "blocker of a different kind..." High priority (blocking further development). Crystal and Kumar had been in discussion, but this ticket has been quiet since (effectively) December. 4.5+ item now. Currently Input Required from Daniel.
• CAS-6124: imfit subtasks (parent currently Scheduled):
  • CAS-4538: migrate imfit input estimates from file to input parameters. Input Required (was "On Hold"). Subtask of CAS-6124. Ticket passed to Juergen, pending a CSSC discussion of what needs to be done and a scheduling of the work. Pushed back to 4.6. Ticket pinged. Juergen: More thought about imfit still needed by Science.
• CAS-4748 (Dec 2012): Polarization selection does not select SIGMA. Input Required. Passed to Jeff for prioritization. Sanjay thinks it's a low-priority item. Version 4.6+.
• CAS-5886 (Dec 2013): bizarre clean boxes for large images. Awaiting input from one developer to another. Version 4.6+.
• CAS-5162 (Sep 2013): Improvements in flagging documentation and examples. Input Required from Science (via Jeff). This ticket has been left untouched since May 2013... Version 4.6+.
• CAS-6794 clean is very slow handling this simple, small ACA mosaic. Input Required from Juergen. Kumar commented on this. Ticket pinged
• CAS-7599 imfit deconvolved peak flux unit. Input Required from Juergen (and other CSSC members).

Other Blockers

General questions had previously been raised about callib expectations for 4.5. Implementation in PlotMS? Scratchless operation? Mark R. has asked George about this on CAS-7096. More progress than we suspected had been made on the latter, but is to be regarded as "experimental" for the purposes of a 4.5 release.

• CAS-6688 Turn on new vi/vb within imager and test. Was a 4.4 item, but now Unscheduled. Currently with Jim. Some scope modification suggested for this ticket. Confirmation received that this is not actually a 4.5 release blocker, so bumped back to 4.6.
• CAS-7407 CalibratingVi2 Improvements. Currently Scheduled". 4.6 item (not a 4.5 blocker). Ticket with George M.
• CAS-7287 Deploy Calibration-aware MSTransformManager in solve context. Subtask of CAS-7096. 4.6 item. Currently Scheduled". Ticket with George M. Ticket pinged.
• Viewer registration tickets:
  • CAS-6560 contour image does not unzoom when master coordinate image is re-registered. Subtask of CAS-6604. Viewer issue. Currently Unscheduled. Not listed as a 4.5 blocker.
  • CAS-6720 2 panels don't register. Viewer issue. Currently Unscheduled. Not listed as a 4.5 blocker (but continues to embarrass...).

• For additional (non-user testing) interest, see also:
  • CAS-2609 Factor out code of Conv. func. cache management. Nominally a 4.5 item. (Fix in, some testing still to be done by Sanjay, but appears to work)
  • CAS-7861 CASA dies on VLA ephemeris datasets. Scheduled. No fix version specified, but being worked on.
  • CAS-7835 importasdm() does not handle VLA observations of ephemeris objects properly. Scheduled. No fix version specified, but being worked on.
  • CAS-7760 code and asap atnf libraries clash. Scheduled. Fix committed, and Kana thinks it works. Passed to Mark W. for follow-up.
  • CAS-7755 tclean produces strong negative residuals. Scheduled. No fix version specified, but was being discussed in June.

---

PART 2: Items Listed for Informational Purposes (note that these will not normally be specifically discussed during the meeting unless someone raises them)

tclean

• tclean testing (for info only: to be skipped during the meeting: will be handled by the Imaging Group)
  • CAS-5048 (May 2015): Clarify when peak residual calcs use/ignore masks, and use consistent thresholding rules. Older ticket, now ready to be tested using new tClean in 4.5. Tester: TBD (was Emmanuel, back in the 4.1 cycle). Ticket with Urvashi
  • CAS-3729 (May 2015): clean to write out only those images that are necessary for a given run. Subtask of CAS-4640. Tester: TBD. Ticket with Urvashi
  • CAS-1623 (May 2015): CLEAN should have option to continue from previous CLEAN state. Subtask of CAS-4640. Tester: TBD. Ticket with Urvashi
  • CAS-7292 (July 2015): clean spw parameter cannot stride >1 channel. 4.5 item. Under Test. Tester: Crystal (or other member of imaging team).
  • CAS-3435 (June 2015): Number of iterations in clean appears to behave differently for interactive/non-interactive. Ready To Test. Tester: TBD. Ticket with Urvashi
  • CAS-2044 (June 2015): mask usage differs for csclean and clark clean. Ready To Test. Tester: TBD. Ticket with Urvashi

Unable to Test Items

• CAS-4758 (Sep): 4.4 item. fluxscale fails if one of the reference fields is flagged. Previously "Under Test" (r31121+) in current (new-style) stable. Tester: Claire. No testing progress on this since October. Claire had been trying to find a suitable dataset for testing. Steve thought that this should actually be fairly easy to test and that Claire and/or he may have been able to test this one. Ticket with Brian Kent. Ticket pinged. Currently Unscheduled, pending test data availability.
• CAS-5379 - Subtle inconsistencies with other programs in Galactic Coordinates (Adam and Harvey). In a nutshell, shifts of ~0.2" are seen when taking data into other packages (e.g. DS9, IDL which both seem to agree with each other). It is unclear yet if this is an incorrect implementation in one or more packages, or possibly just a precision error. This is a subtle issue, and was pushed back. Ticket stalled. Needs to be looked at by someone with lots of time. To be left as-is for now. This needs significant work, including new science papers on how to actually define the Galactic coordinate system? Needs casacore discussion; this can be part of that. Passed to Jeff for now. Pushed back to some indefinite future release. Unscheduled.
• CAS-4636 (Oct 2012): Elevation dependent opacity application w/ Tsys. Still awaiting input from Science. Had been "On Hold" since August, awaiting data. Previously switched to "Input Required" and passed to Crystal for comment. Bumped back to 4.4+. Work on this is Unscheduled until further notice (more than a year, now).
• CAS-3532 (Sep): Incorporation of Alma Primary Beam in Mosaic. Currently Unscheduled. Tester was Scott S., but ticket passed to Crystal for input. Some test imaging done. Ticket with developer. Won't be done for 4.5 now.

Upcoming Ready to (Re)Test Items

Not Yet Assigned To Testers

• TBA

Single Dish

See below "assigned" SD list.

Already Assigned To Testers

• CAS-4592: plotms on 10.7 issues. Previously half fixed due to Qt updates. Shown to till be an issue under 10.10. Currently Unscheduled. Tester: was Nathan, to be Juergen. Exported plots all look OK, even if the plots in the GUI appear corrupted. Juergen to discuss prioritization with Jeff? Fix in time for 4.5 now seems unlikely...
• CAS-5127 (May): Document ia toolkit methods. Status Scheduled. Mark should start working on this again... Most of Juergen's prioritized list of 26 items had been done. Dates to be added his future documentation updates. Ongoing. ia.regrid docs will require more work. Still needs code commits. Work really needs to be restarted on this under Linux soon.

Single Dish

• CAS-5763: sdimaging should provide an accurate error message when the requested field does not exist in the measurement set. *Under Test. Tester: Patricio. Input from Todd also invited.

-- MarkRawlings - 2015-10-02