Socorro: Juergen, Urvashi, Martin, Pam, Lindsey, George, Tak
CV: Dave, Darrell, Jen, Bjorn, Anand, Ville, Akeem, Andy, Bob
Garching: Enrique, Federico, Dirk
SCO: ?
News / Meetings / Visitors
Build, Release
Verification Testing
Akeem has been documenting his work (his last day is August 18th)
Andy has been working on HPC pipeline validation
Other than CAS-10470 (Closed - a known issue not related explicitly to HPC pipeline run), and CAS-10481 (RMS noise inconsistencies), nothing too scary in my reviewing the data
19 tickets went RtV this week so far. 22 tickets went through Validation to Resolved this week so far.
July 28-Aug 3: 23 tickets assigned for validation. 28 tickets resolved.
aiming to have most validation testing finished by Friday for branching on 8/15, but if blockers are not fixed, we can wait
Amy Kimball will be helping on the VLA side with getting and organizing testing resources via the new home for CASA testing under SSR after Oct 1
Current Testing Efforts for 5.1 deliverables
ALMA Cy5 pipeline, crossover between CASA and PL
autoboxing (CAS-10415, CAS-10457). plus new seg fault?
mosaic issues CAS-10317, CAS-10352 (ready for DirkM to test this morning). blockers to the 5.1 release.
statwt2
has been merged into 5.1. will not be formally announced, but VLASS will use the pieces they have validated (and keep using statwt). thorough testing and possibly some new requests in 5.2.
will create one testing ticket for 5.2 that encompasses all of the features needing validation
Cycle 4 benchmarks from PWG now all covered (at least one try), cycle 3 should be covered next week as well. Only a couple of tests from Cycle 2. How interested are we in more benchmarks from cycle 2?
Datasets with serial + parallel run results ready for Sci validation: from 6 (last week) to 14 (now) . But 2 probably have issues, and other 5 datasets still failing. Also new issues, see below.
Statwt2 : merged. Will continue validation and incremental features for 5.2
Waiting for one more pull request from NAOJ. Nearly done there.
Bugs from HPC testing of the current ALMA pipeline in parallel
Bjorn/Andy looking at test results. Bjorn's reports CAS-10481 suggest that serial vs parallel cube is scientifically equivalent when no other bugs are encountered. Found 3+ datasets where there appear to be data selection (and csys generation) differences between serial vs parallel. Bjorn - Update on new issues you/Andy have been uncovering ?
Viewer bug with MSs with differently shaped spws : CAS-10470
Applycal errors in parallel : CAS-10453 : Lindsey has followed up with Brian Kent - update ?
CAS-10352/CAS-10507 : Columslicer error in tclean : KG fixed it.
Issues related to running the tests on 64GB RAM machines (or less) vs how the pipeline uses it ( 128GB in Chile, 256GB in Charlottesville ) : CAS-10392 for findcontinuum.py bottleneck, CAS-10484 for the autoboxing segfault.
CASA 5.2 planning
Attached document (https://safe.nrao.edu/wiki/pub/Software/CASA/Aug1017/CASA_5.2_Planning_V.2.pdf) has details for ALMA/VLASS/Internal/Ongoing-noncritical issues, plus a listing per developer. Please read and ask questions if you have any or request changes. This was put together after consulting each developer, but please let me know if you find something unexpected.
Please send me a list of JIRA tickets that cover this planned work. Please make new ones when needed.
Discussion is ongoing about how we should respond to a proposed mid-cycle ALMA pipeline patch, which may need fixes from CASA as well. There are pros/cons to any solution.
There are many old/unused? tasks in CASA, and the plan is to not try to document them, but to identify which need to be replaced or begin the deprecation process. Jen and I made a list and I will follow up offline (or in another weekly meeting, when we have time).
Pipeline
Parallel testing
New / old tclean column slicer error in serial mode (CAS-10507 should be fixed)
Coordinate system errors in serial mode (CAS-10434, CAS-10492, under investigation)
Applycal error in pipeline hif_uvcontsub task is worrying but based on web log review seems to be harmless (note this is VI1 error, with MMS data set, but with parallel processing disabled)
Fixed flux catalog database service error in serial mode, parallel mode behavior seems different still under investigation (found with CAS-10453)
5.1 Release
Dirk M. back from vacation so lots of last minute pipeline imaging tweaks and bug fixes
Autoboxing masks if present have been added to the pipeline data products
Turning on wvrgcal task remcloud feature in pipeline is under discussion
Remaining tickets are for small web log and logging enhancements
On going acceptance testing
AOB
Developer Reports
Thursday Meeting
Sanjay Bhatnagar
Sandra Castro
Lindsey Davis
Added autoboxing masks to the list of ALMA interferometry pipeline imaging data products
Cleaned up various pipeline web log reporting issues
Reviewed ALMA pipeline testing and deployment plan (with Rachel R)
Reviewed remaining 5.1 pipeline tickets (with Remy and others)
Follow up on pipeline issues turned up during parallel pipeline testing
CASA 5.2 planning discussions (CASA coordination meeting) and followup
Bjorn Emonts
Pam Ford
prototype for time axis tick labels (CAS-10488) coded, waiting for feedback from Juergen
ms tool deprecation warnings complete (CAS-10483), casarc plone doc ready to verify (CAS-9294)
Investigating CAS-9053 Test Suite 2 failure
started plotcal replacement effort
Enrique Garcia
CAS-9945: improve the approximations for effective bandwidth and effective resolution in split/mstransform
CAS-5174: UV Continuum Subtraction TVI
Bob Garwood
Kumar Golap
David Mehringer
George Moellenbrock
Resolved CAS-9357: fluxscale no longer leaves MS table open when exception raised
Resolved CAS-10493: repaired gencal's caltype='gc', which broke when CAS-8540 was checked in.
Analyzed ALMA EOP error
5.2 planning
Dirk Petry
Martin Pokorny
Still shepherding changes in casacore for async vi2.
Federico M Pouzols
Parallel pipeline tests
CAS-10464 Fix regression test "MPI Alma M100 Analysis Hpc Regression Tclean", seems fixable, double-checking with tclean experts.
CAS-9089 + ICT-10558: following on, will require further feedback to align gencal with "correctMyAntennaPositions".
Urvashi Rao
A lot of talking and emailing.
Called a CCC meeting to go over V.1 of 5.2 plan.
Darrell Schiebel
Ville Suoranta
Takahiro Tsutsumi
Friday NAOJ Meeting
Kanako Sugimoto
Wataru Kawasaki
Masaya Kuniyoshi
Takeshi Nakazato
Renaud Miel
Minutes
CASA Users committee meeting is likely to be in the week of October 30, in Charlottesville.
We're on track for making the release branch on Aug 15. We have a couple of blockers coming from the HPC tests, but they're being worked on and fixes will just go into both branches over the next two weeks.
Documentation release plan was made (contact Jen/Juergen for details)