Thursday Morning Meeting : 20 July 2017

• DIAL-IN NUMBERS & PASSCODES:
• IP: 192.33.117.12##8110
• Phone: (434) 817-6524

Attendance

• Socorro:
• CV:
• Garching:
• SCO:

News / Meetings / Visitors

• Morgan in Socorro - 7/24-728
• Interviews for CASA Lead and CARTA UI position

Build, Release

• 5.0 - any loose ends?

Verification Testing

• Akeem's last day at the NRAO is (tentatively) August 18th
  • We have created a list of things we would like wrapped up before his departure
  • Recruitment for his replacement will begin after his departure
  • This will impact the level of support the test group will be able to provide to CASA
• Automated test movement
  • NGC 3256 has been removed
  • downgraded third 4826 to accepted
  • Full mpi4casa has been moved to accepted
  • subset of mpi4casa tests is in test suite 2
  • predict comp added to accepted
  • asdmsummary added to accepted

Validation Testing

• Testing is going to move to SSR.
• plone: casadocs vs casadocs-devel
  • the site to use for editing the CASA 5.1 documentation is now https://casa.nrao.edu/casadocs-devel
  • the outward facing site for users is now https://casa.nrao.edu/casadocs
• Current Testing Efforts for 5.1 deliverables
  • ALMA Cy5 pipeline
    • (many tickets, many people: Remy, Crystal, Todd, BrianM, Amanda, etc etc)
  • crossover between CASA and PL:
    • CAS-9089 Automatic Generation of Antenna Position Corrections for ALMA -- Eric Villard, under test
    • CAS-9827 plotms iterator changes filename scheme depending on path length -- JDM, under test
    • CAS-9053 Add atmospheric overlays to plotms -- needs some clarification on usage to be assigned (Pam)
  • statwt
    • 26 tickets, all currently assigned to Brian Mason. testing is just starting. trying to get an additional tester at the VLA but having trouble (Morgan?)
    • thank you to Dave for your (infinite) patience with our limited resources
  • parallelization
    • current discussions happening between relevant parties. Bjorn is able to start helping with the validation work once there is a testing plan.
  • autoboxing
    • testing is ongoing (mostly Amanda, Crystal)
  • plone
    • organizing a plan with Kana to get the SD tasks into 5.1
    • Bjorn is going to start helping us with some validation testing
    • otherwise this seems to be a lower priority for 5.1 given the short development cycle
  • miscellaneous
    • please see https://safe.nrao.edu/wiki/bin/view/Software/CASAUserTesting for the up-to-date list of tickets and testing status

Architecture

• Warnings Summary

HPC

• We have created a page in Confluence to follow up our tests in CV.
• CAS-10398 (exportfits has issues with concatenated images) seems fixed. Resuming the parallel tests now.
• The PWG uses custom scripts to run the pipeline and some of the settings they do are not trivial to do when running in parallel. For example, changing ulimit on the shell will only affect where mpicasa is executed, and will not affect the MPI remote servers. We have created CAS-10416 to figure out how to integrate parallel CASA in these scripts.
• Agreement pending between Urvashi and Remy/Crystal whether we (ESO team) can get nodes in CV for the parallel tests or not.

Development

• Parallel pipeline tests :
  • The pipeline working group will not have time to test/commission this in time for our casa 5.1 release (or even their own cycle 5 release). They plan to do a pipeline patch for December. But, we still need testing before our casa 5.1 release, specifically to minimize the chances of needing a casa 5.1.1 patch from our side. So we're doing our own tests.
  • Sandra is coordinating with the PLWG on an agreed-upon list of ALMA pipeline test cases and setting up the tests to run in serial/parallel with some comparison metrics, and weblogs posted somewhere. In CV, Darrell/Andy will take this over once Sandra goes on vacation (in 9 days) and Andy will do some of the scientific validation (with input from Lindsey and the imaging team on what to look for).
  • This has begun. One critical bug turned up immediately (CAS-10398) for which a solution has already been pushed in (thanks to Tak) and ready for Sandra's tests. Follow up work will still be required for a better long-term solution, but it's OK for now.
  • Confirmed that VLASS does not need parallelization for now, and individual users who do need it should be covered by the above tests.

• Master is still failing on acceptance tests. MP is working on possible vi2 issues via CAS-10362. Although the failure is manifesting itself in 'virtualmodel', MP's fix on the branch exposed another failure in vi2 from channel chunking of cubes. This brings this issue into the critical path for the ALMA pipeline.

• Autoboxing goes on : Update from Tak ?
• Documentation -- Anand to review remaining tickets to identify any critical work for 5.1. I think the only high priority item for this cycle is "branching" the documentation along with the code at release time to allow continued work on documentation for the subsequent cycle. Will update / create documentation that describes the current doc system and release / package process.

Pipeline

• On-going ALMA Cycle 5 e2e tests in Chile
• One going ALMA Cycle 5 pipeline testing at the NASSC - focus on autoboxing and imaging sensitivity pre-checking
• On going VLASS testing, many enhancement and improvements
• CASA 5.1. parallel pipeline testing - ALMA parallel pipeline not target for start of Cycle 5, will come in as patch
  • latest issue was slowdown in exportfits - should be resolved
• possible packaging issue in gencal ALMA antenna position correction support

AOB

• (UR) Whoever the first person is to check if this wiki page exists or not, please create the page from the template stored above the calendar list on https://safe.nrao.edu/wiki/bin/view/Software/CASA/CASAMeetingPage#Monday_Morning_Meetings . Lately, I've been doing it, but I usually get to it only late on Wednesday night, leaving little time for Socorro folks to add things in. - (MG) - thanks Urvashi! I should probably be doing it - I'll put a reminder on my calendar to make sure I do.
• (MG) FY2018 POP goals
• (FMP) ALMA Web services:
  • Pipeline would like to know if the SOAP client functionality is not going to be included for CASA 5.1, as it affects PL workflow and development (see CAS-9089).
  • Should we escalate a request for harmonization of ALMA web services?

Developer Reports

Monday Meeting

• Sanjay Bhatnagar
• Sandra Castro
• Lindsey Davis
  • Improvements to ALMA low SNR gain calibration heuristics
  • CASA lead job search interviews and discussions
  • Followup on pipeline tickets
• Pam Ford
• Enrique Garcia
  • CASA-10013: Merging changes from StatWtTVI in mstransform.
• Bob Garwood
• Kumar Golap
• Jeff Kern
• David Mehringer
• George Moellenbrock
• Dirk Petry
• Martin Pokorny
• Federico M Pouzols
  • cvel1/cvel2 regridding question from users: CAS-10411.
  • Tests of pipeline locally and in CV cluster.
  • Issues in pipeline, already fixed in recent revisions CAS-10397. Issue in calibration plotting currently affecting most if not all datasets: CAS-10413.
  • Working on flagging interface and documentation tickets: CAS-10036 CAS-5808, CAS-5809.
• Urvashi Rao
  • Ran a meeting to discuss pipeline testing plans for 5.1, and subsequent discussions about the resulting plans and the new bug (CAS-10398).
  • Participated in two (virtual) in-person CASA Lead interviews.
  • Began work on 5.2 requirements (after finding out from Jeff that he actually did not follow any formal process over the past year, and discussing the situation with Morgan.)
  • Continued work with second summer student.
• Darrell Schiebel
• Ville Suoranta
• Takahiro Tsutsumi

Friday NAOJ Meeting

• Kanako Sugimoto
  • recent discussions
    • ALMA-TP commissioning status and priorities with Tsuyoshi
    • Sent a note to Juergen and Crystal about how to update M100 SD CASA Guide
    • Notification of removal of ASAP from CASA to Malte
    • changes to merge procedure: intention and consequences
    • discussion of test package version in CCC mailing list (OSX support, too)
    • plone document for 5.1 release
  • Update of ATM library in CASA 5.1 ( CAS-9958, CAS-9762, CAS-5092)
  • MS transition: sideband separation algorithm ( CAS-8091): started work
• Wataru Kawasaki
• Masaya Kuniyoshi
• Takeshi Nakazato
  • reported CAS-10409 and verified Dave's fix
  • duplicated row issue (CAS-10289) transferred to PRTSPR-27142
  • completed CAS-10330
  • about to complete CAS-8879
    • asked a way to put release note item when issuing pull request
  • restart CAS-9957
• Renaud Miel
  • CAS-8082 PlotMS Header
    • Fixed bugs in CLI export and MS metadata extraction, extended generic plotter, pushed code
    • Bamboo false negative build failure: Could not find metadata edu.nrao.gradle...
    • Verified that problem 'PlotMS crashes when resized during cache load' is still present in CASA 5.0

Thursday Meeting Minutes

• Attendance: Juergen, Urvashi, Martin, Tak, Lindsey, Sandra, Enrique, Federico, Jen, Dave, Morgan, Anand, Andy, Akeem, Remy
• News
  • hiring committees going forward a lot of meetings for casa lead
  • CARTA
    • interview for new CASA developer Monday, new hire will contribute to CASA GUI development as well as CARTA
    • we have some promising candidates
    • Canada contributes through December
    • SKA contributes
  • MS v3 kickoff meeting happened last week
    • this is a year long project with SKA
    • this is part of a "modernizing CASA" effort, the MS seemed like a good place to start with SKA
    • first goal is establishing requirements
• CASA 5.0
  • Juergen will send out the announcement to the world tomorrow
  • CASA guides will be fixed to over the coming two months
  • CASA guides have redirects to the tools/tasks (which need work to accommodate the new plone documentation)
• Test Group
  • Akeem leaving, he will work through August
  • pipeline test exercise VLA but not ALMA
    • Sandra will contribute an ALMA pipeline test case
    • Sandra has three small datasets which are not antiques
• Validation Testing
  • CASA testing is moving out of the NASC to SSR
    • the hope is that we will get more access to testers
  • documentation changes
    • The outward facing site for end users is casadocs
    • The site which should be edited and updated is casadocs-devel
  • it is difficult to find statwt VLA testers
  • CASA will attempt to do do technical testing of parallelized tclean
  • Bjorn on board as CASA liaison, may be able to contribute to testing
• parallelization
  • options for delivery of parallelization
    • casa 5.2 release in early 2018
    • casa 5.1.1 patch release in early 2018
  • exportfits is extremely slow due to opening a JSON file
  • pipeline sets many environment variables
    • these may be mostly for CASA setup, so may not affect a parallel run
    • they must be examined as part of verifying parallel pipeline behavior, down the road
    • CASA has wrappers around environment variables
  • Tak looking at autoboxing issues, the test image is huge and is taking 7 hours
• documentation process will be added to plone-based documentation in preparation for CASA 5.1