CASA User Testing Meeting Minutes - 2015-02-17, 09:00 Socorro / 11:00 Charlottesville

Agenda

Participants

Locations

• Socorro: AOC-280; IP: 146.88.6.10 or AOC-317; IP: 146.88.6.8
• Charlottesville: Room 331; IP: 192.33.117.19

Minutes (Main Points Only)

Status

• 4.4 is currently under development. Test/stable/monthly packages are being produced for RHEL5 and 6. OS X package production (for 10.10 and 10.9/10.8) is currently a work in progress.
• A 4.3.1 patch release is currently being worked on for Linux and OS X (to address VLA SetJy issue CAS-7298).
• 4.3.0 is the current public release version for both Linux (RHEL5 and 6) and OS X (10.8). 4.2.2 is still currently being used for pipelined ALMA data reduction at the NAASC.
  • A general-purposes CASA launcher script is being developed at the moment.
  • If you are on Red Hat 6 and need CASA 4.2.2 on the new-style lustre nodes (CVPOST), run it from here: /home/casa/packages/RHEL5/release/casapy-42.2.30986-1-64b/casa
  • If you are on Red Hat 6 and need CASA 4.1 on lustre, run it from here: /home/casa/packages/RHEL5/release/casapy-41.0.24668-001-64b/casa
  • Alternatively, the following sort of approach works too:
    • $ CASA_ROOT=/home/casa/packages/RHEL6/release/casapy-42.2.30986-1-64b
    • $ CASAPATH=${CASA_ROOT}
    • $ PATH=$CASAPATH:$PATH
    • $ casa
  • Descriptions of the new-style packages, how to start them, etc. are summarized on the following wiki pages:
    • https://safe.nrao.edu/wiki/bin/view/Software/CASA/StartCasa#Redux - Note that the RPM startup instructions no longer apply to lustre in CV/Socorro, but do apply in Chile and the zuul test machines (see below).
    • https://safe.nrao.edu/wiki/bin/view/Software/CasaVersionNumbering#Redux
    • Version number/Subversion revision number look-up table: https://svn.cv.nrao.edu/cgi-bin/casa-version-series
    • The new-style testing packages are available on four test machines in Charlottesville (zuul01 - 04). These are configured to allow local testing of the new-style 4.4 stable packages. zuul01, 03 and 04 are running RHEL6, zuul02 is running RHEL5. REMEMBER: Use /usr/bin/casa on these machines to make sure that you are running the new test/stable packages.

Agenda

• The 4.4 feature freeze is coming! (March 1st).
• New ticket assignments (see below).
• The "nag list", "input required" items
• Recent discussion items:
  • Crystal had previously noted that she had encountered a problem when taking a data set from CASA to FITS and back to CASA again: problems are reported with axes coordinates when (e.g.) attempting to use impbcorr. A Jira ticket will be created for this, but she also wanted to know how to best draw attention to tickets that have a broad, significant impact on the user experience but which cannot be classified as blockers. Ticket number?
  • Mark was to sound out Josh Malone about NRAO plans for moving to OS X 10.10 (and/or 10.9). Josh's assessment a couple of weeks ago was that OS X 10.10 was that is was not yet ready for common site-wide deployment. However, it now sounds like (with a bit of luck) we may start seeing it on NRAO OS X machines within the next couple of weeks (not to be taken as an official announcement, though: the decision will come from IT).
• Other testing items specifically raised?
  • TBA

New Testing Assignments (see relevant section below)

• Assignment of any other "To Be Assigned" Items.
• The next meeting is scheduled to take place on March 3rd.
• Any Other Business: None.

Old Action Items

• None

New Action Items Arising

• TBA

PART 1: Items to be Covered During Meetings (as time permits)

Propose to Close Items

• CAS-6220 (Mar): Enable importasdm to read flags from the binary ASDM data. Under Test in any of the new-style stable packages. Commit was r28609. Ticket with Dirk for now. Todd had incidentally tested this while working on something else. Runs to completion (and sometimes generates a lot of INFO messages). (Close to?) done, but was kept open for second stage implementation of asdm2MS completion. May be able to close?

Items Overdue for Status Updates (i.e. the "Nag List")

• CAS-7318 (Feb): 4.4 item. Verify unpolarized calibrator mode for linear basis (ALMA). tester; Steve.
• CAS-6602 (Jan): 4.4 item. ia.toworld() does not return correct value for concatenated image with tabular axis. Ready To Test now (probably in the 4.3 release). Tester: DA. Can just go ahead with testing now?
• CAS-4469 (Jun+ 2013): Split with channel averaging is creating spurious spectral features and weight is wrong when there has been channel flagging. Subtask of CAS-5245. Ready To Test. Additional clarification of testing details has been provided on the ticket by developers. Tester: Crystal
• CAS-3580 (Aug): 4.4 item. Prediction of expected image based noise based on Tsys. Under Test in 4.3. Tester: Eric V. (to be tested at the JAO). Kept open at Remy's express request. Recent testing continuing by Eric. Ticket pinged.
• CAS-5191 (Mar): Add gain threshold parameter to fluxscale. Under Test (r28480) in any recent package. Tester: Arielle (was Crystal). Some testing done, bug fix put in, more testing still to be done. With developer for comment. Ticket pinged. Resolved as fixed, following the non-meeting.
• CAS-4758 (Sep): 4.4 item. fluxscale fails if one of the reference fields is flagged. Under Test (r31121+) in current (new-style) stable. Tester: Claire. No testing progress on this since October. Ticket pinged. Note: If no additional progress is made by Feb. 17th, this will be reassigned to Juergen.
• CAS-6912: 4.4 item now. makemask copy always replaces image. 4.3 item. "Under Test" again in stable package 4.4.9+. Tester: Juergen. Ongoing. Needs to be checked again (but went out in the 4.3 release already). Close to done? Testing ongoing (Juergen posted an update, ticket back with developer).
• CAS-4623 (Aug): 4.4 item now. SetJy: Method to set Pol Fraction and Angle as Function of Frequency. Under Test in current packages. Tester: Preshanth. Had been pending access to new testable package. Steve now has a test dataset for this. Ticket passed to Preshanth, although Steve will probably also do a bit of testing of this one. Ticket pinged: Preshanth has tested this, and promised to file his report. Ticket pinged.
• CAS-7221: 4.4 item (just missed 4.3 release). importfits doesn't support RESTWAV fits keyword. Under Test in current stable. Tester: Remy. Close to done? Ticket pinged.
• CAS-6707 (Sep): mstransform problems with CALDEVICE and SYSPOWER sub-tables when creating an MMS. Under Test in 4.3, but 4.4 item now. Tester: Previously passed to Steve for VLA user tester assignment. Previously discussed at length: Dependent on the level of parallelization available. Ongoing item. Feature was already shipped in 4.3, though...
• CAS-6778 (Sep): Add support for new 'OBJECT' reference framew in mstransform. "Under Test". Tester: Arielle. Some testing done. Sounds mostly good, but some issues raised - ticket reopened and passed back to developer for more work. Ongoing. Developer has replied to the tester's comments, response requested.
• CAS-6527 CASA collides with easy_install.pth. 4.4 Blocker%ENDCOLOR item. Under Test. Tester: Remy. Ticket pinged (for Linux testing).
• CAS-6529: 4.4 item now. MS processed with mstransform not handled correctly by uvcontsub, clean, etc... Under Test in latest new-style stables and pre-releases. Tester: TBD (was Kristina). Mostly works now. One more fix just committed (r31369 in trunk, also now in release branch). Reassigned to Juergen, who will get up to speed on this ticket and keep/reassign as appropriate. Ticket pinged.

Other Items Currently "Under Test"

• N/A

"Ready To Test" Items Pending Tester Assignment

• CAS-5295 (Feb 2015): PYTHONPATH is not properly handled in CASA for Mac OSX. Ready To Test now, pending availability of a 4.4 OS X package. Tester: TBD (Remy?)
• CAS-6805 (Feb 2015): importasdm should only overwrite online text files if overwrite=True. Ready To Test now. Tester: TBD (DA?)

• CAS-3571 (Jan): Ionospheric Correction (TECOR). Ready To Test now (probably in the 4.3 release). Ticket left with George, so that he can liaise with testers Jason Kooi, Chris Hales and Huib Intema. Ticket pinged.
• CAS-7258 (Jan): 4.4 item. concat should also scale SIGMA when it scales WEIGHT. Probably available in stable now. Clarification still needed on the necessary testing procedure was requested from Dirk. Tester: Dirk (or his EU designate). Ticket pinged.

Single Dish (to be skipped during the meeting)

• See SD ticket list below

Input Required

• CAS-5806 Mathematical prediction of post-gridding single dish beam. Formerly a 4.3+ Blocker, now "Critical". "Input Required?* Developer has done some modelling, but... ? This is a "blocker of a different kind..." High priority (blocking further development). Currently Input Required from Kana. Crystal and Kumar had been in discussion, but this ticket has been quiet since (effectively) December. 4.4+ item now. Not sure what to do with this one?
• CAS-6975: not evidently possible to overplot from casapy. Input Required from Remy. Ticket pinged.
• CAS-5334 (Feb 2014): Ellipse regions have wrong width (RA). Second fix committed. Was previously tested in casapy-test (r29326), set to Input Required for input from developer. Tester: Paul. Pushed back to 4.4. Cropped up in a Helpdesk ticket recently, so still a problem. Pinged another contributing developer for his thoughts.
• CAS-7282 (Feb 2015): add Gauribidanur Radio Heliograph to CASA. New item, low priority. Ticket with developer, but more input needed from user first. Ticket pinged.
• CAS-5785 (May 2014): drastic split speed discrepancies depending on whether spw is set or not. Input Required. Ticket back with reporter for more info. Known behavior for MSSelection. Some discussion of this item and related topics: split with spw (see also CAS-3666 below). It would be good to be able to select by spw on a large dataset. Will MSTransform have the same problem? Do MMSs help here? Also, Justo's note on this implies that this is a known issue with MSSelection (and not a split problem), but doesn't actually indicate if this is a known issue to be addressed or a known issue that we have to document and just live with. (Steve would also like to be able to do spw selection in asdm2ms - there may well be a separate ticket for this). 4.4+ item. Ticket pinged repeatedly without a response from the reporter.
• CAS-4735 (Nov): imsmooth needs additional option for single dish use. Subtask of CAS-5844 (above). Open again. Tester: Tsuyoshi. Critical item. Code committed to trunk. 4.3 item. Some testing done, mostly works as intended. Input Required from Crystal for some remaining changes. Any remaining future work will have to be for 4.4+. Ticket pinged.
• CAS-6124: imfit subtasks:
  • CAS-4538: migrate imfit input estimates from file to input parameters. Input Required (was "On Hold"). Subtask of CAS-6124. Ticket passed to Juergen, pending a CSSC discussion of what needs to be done and a scheduling of the work. Pushed back to 4.4. Ticket pinged. Juergen: More thought about imfit still needed by Science.
  • CAS-6234: imfit estimates should allow sky coordinates in addition to pixel values. Input Required (was "On Hold"). Subtask of CAS-6124. Ticket with Juergen, pending a CSSC discussion of what needs to be done and a scheduling of the work. Pushed back to 4.4. Ticket pinged.Juergen: More thought about imfit still needed by Science.
• CAS-4748 (Dec 2012): Polarization selection does not select SIGMA. Input Required. Passed to Jeff for prioritization. Version 4.5+. Ticket pinged.
• CAS-5886 (Dec 2013): bizarre clean boxes for large images. Awaiting input from one developer to another. Version 4.5+.
• CAS-6175 Plotms-vis-model (minus) normalization (visnorm) in gain,bandpass (seems to be missing now). Ticket with Jeff for follow-up. Pushed back to 4.4. Ticket pinged.
• CAS-4349 Amplitude calibration for single dish. Pushed back to 4.4+. Input from Tsuyoshi requested. Ticket pinged previously.
• CAS-2825 Support for Lissajous mapping. Pushed back to 4.4+. Input from Tsuyoshi requested. Discussion ongoing.
• CAS-5162 (Sep 2013): Improvements in flagging documentation and examples. Input Required from Science (via Jeff). This ticket has been left untouched since May 2013... Version 4.4+.

Other Blockers

• CAS-7289 plotms should support selecting the plot of FLOAT_DATA. Scheduled. Currently with Pam. Work ongoing. Currently a 4.4 release blocker.
• CAS-7224 Migrate aatm code to casa code tree. Scheduled. Currently with Kana. Pinged Kana in case there's someone in particular who needs to be nudged for a comment on this. Provisional plan proposed for moving forward with this. Work to be started next week. Currently a 4.4 release blocker.
• CAS-7220 Update ATM library in CASA 4.4. 4.4 item. Scheduled. Currently with Kana. Work in progress (Kana to talk to TELCAL team). Currently a 4.4 release blocker.

• CAS-7251 sdimaging crashed on 4.3 prerelease for OS X 10.8. 4.3 item(?). Open. Currently with Kana. Takeshi had been looking into this. Crash reproduced at NAOJ, but not yet in CV. Is this a blocker for the 4.4 release? Will know more once we have 4.4 OS X packages.

• CAS-6688 Turn on new vi/vb within imager and test. 4.4 item, but Unscheduled. Currently with Jim. Probably not a blocker for the 4.4 release.

PART 2: Items Listed for Informational Purposes (note that these will not normally be specifically discussed during the meeting unless someone raises them)

Unable to Test Items

• CAS-3614 Calibration and Analysis of Single Dish Polarization. Pushed back to 4.4+. Input from Tsuyoshi requested: Response was that cal/analysis strategy for SD pol not yet established, so no input possible yet. Leaving as "Input Required" until then.
• CAS-6989 (Jan): 4.4 item. Define infrastructure for new sdutil, sd tool and classes. RTT. Currently awaiting a feature merge back to trunk and general libsakura availability.
• CAS-7240 (Jan): 4.4 item. Dot option for grid lines in plotms does not work on Mac. RTT, pending a suitable OS X test/stable 4.4 package. Tester DA
• CAS-7247 (Jan): 4.4 item. peak value in spectral profile Gaussian estimates window overlapping with slider on Mac. RTT, pending a suitable OS X test/stable 4.4 package. Tester: DA
• CAS-4592: plotms on 10.7 issues. Half fixed due to Qt updates. Was "ARTT" in 4.2.1 and 4.2.2 pre-release. Scheduled again now. Also seen under OS X 10.8+. Ticket with the developer. Tester: Nathan. Awaiting a new OS X package.
• CAS-5084 (May): Testing multi-term support in pclean. Lots of feedback supplied by Kristina and Claire. On Hold (since August 2013) and assigned to Kumar pending class refactoring. To be left on hold, pending the refactored imager classes.
• CAS-6433 (Apr): Implement a function to derive Jy/K factor from amplitude calibrator observation. Open. Currently with developer. Was targeted for November 23rd Pipeline release. Pushed back to March 2015 Pipeline release and flagged as a Pipeline Blocker.
• CAS-6232 (Mar): writing version numbers in GUIs. Under Test in 4.3. Most recent commit to trunk was r29373. Tester: Juergen. However, this also needs to be checked in the table browser, the work on which is currently unscheduled. Spawned another ticket CAS-7033) for its underlying cause.
• CAS-5379 - Subtle inconsistencies with other programs in Galactic Coordinates (Adam and Harvey). In a nutshell, shifts of ~0.2" are seen when taking data into other packages (e.g. DS9, IDL which both seem to agree with each other). It is unclear yet if this is an incorrect implementation in one or more packages, or possibly just a precision error. This is a subtle issue, and was pushed back. Ticket stalled. Needs to be looked at by someone with lots of time. To be left as-is for now. This needs significant work, including new science papers on how to actually define the Galactic coordinate system? Needs casacore discussion; this can be part of that. Passed to Jeff for now. Pushed back to some indefinite future release.
• CAS-4636 (Oct 2012): Elevation dependent opacity application w/ Tsys. Still awaiting input from Science. Had been "On Hold" since August, awaiting data. Switched to Input Required and passed to Crystal for comment. Bumped back to 4.4+. Work on this is "Unscheduled" until further notice. * CAS-6243: sdimaging does not accept type numpy.string_ for fieldid. Unscheduled, pending identification of a suitable example case. Commit was r29988. Discussion ongoing. Pushed back to 4.4(+). Tester: Todd. Todd hasn't seen this happening, but hasn't specifically tested for it either.

Upcoming Ready to (Re)Test Items

Not Yet Assigned To Testers

• TBA

Single Dish

Already Assigned To Testers

• CAS-5836 (Nov): 4.4 item. Create new task imrebin. Under Test again in 4.4.9 or later (so testable again now). Tester: Amanda (was Adam). The dropdeg and box parameters should be hooked up correctly now. Mostly works, but a couple of details identified. Passed back to developer.
• CAS-5893 (May): Now a 4.5 item. Add the ability to write a spectral profile to an ASCII file in the ia tool. Was Under Test in 4.3. Commit was r29681. Tester: Assigned to Juergen for testing and a possible CSSC discussion. Might also need a task adding for this for 4.5? Recent testing done, passed back to dev.
• CAS-6951 (Oct): 4.4 item now. mstransform creates a 4-polarization DATA column for single-polarization ALMA WVR data, and a 1-polarization DATA column for four-polarization ALMA data. Fix already included in current new-style test/stable packages newer than 4.2.23. Scheduled again now. Tester: Todd. Ticket currently back with developer again. May need passing to another developer?
• CAS-6962 (Oct): 4.4 item now. wvrgcal segfaults on a 9-antenna dataset in which only one antenna has a WVR, and minnumants=2. Scheduled again. Tester: Todd. No longer crashes, but odd behavior remains for zero-value case. Ticket currently back with developer. Work seems to be ongoing.
• CAS-3532 (Sep): Incorporation of Alma Primary Beam in Mosaic. Was ARTT, currently Scheduled. Tester was Scott S., but ticket passed to Crystal for input. Some test imaging done. Current status is Scheduled. Pushed back to 4.4. Ticket pinged.
• CAS-3517 (Aug): Improve remapping interface (Cal libraries). Input Required from developer. Tester: Steve. Some testing done, recommendations made, ticket back with developer. Discussion ongoing. Further callibrary work to be pushed out to 4.4 in Jira. Pushed back to 4.4. Previously identified as a duplicate of CAS-5617, but now (also) a sub-task of it.
• CAS-5127 (May): Document ia toolkit methods. Status Scheduled. Mark is working on this. Most of Juergen's prioritized list of 26 items has been done. Dates to be added his future documentation updates. Prioritized by Mark. Ongoing. ia.regrid docs will require more work. Still needs code commits. Mark to restart work on this under Linux soon.

Single Dish

• CAS-6991 (Feb): MS transition: provide capability to create sky calibration table. Under Test. Tester: TBD. Passed to Bunyo for testing assignment.

• CAS-6992 (Jan): MS transition: Define a task to generate Tsys calibration table. Under Test. Tester: Bunyo. Testing ongoing.
• CAS-6987 (Jan): Imaging of multi-beam data taken by Nobeyama 45-m. Under Test. Passed to Bunyo, to liaise with the identified external tester.
• CAS-6572: (Aug): A new baseline task that handles baseline table. Under Test in latest 4.3 casapy-test/new-style stable packages. Mostly done. Ticket with Kana. Ticket pinged.