Thursday Morning Meeting - 24 August 2017

  • DIAL-IN NUMBERS & PASSCODES:
  • IP: 192.33.117.12##8110
  • Phone: (434) 817-6524

Attendance

  • Socorro:
  • CV:
  • Garching:
  • SCO:

News / Meetings / Visitors

Build, Release

  • 5.1 branch made. [ Ville, please add more details here about rules for submitting edits to multiple branches... ]
  • ALMA pipeline wants a Nov/Dec 2017 product from CASA. We will call it 5.2 and have its development happen on a 5.2 branch (taken from the 5.1 branch when we release 5.1). All other development for the March 2018 release (to be called 5.3) will happen on master. Only fixes required for the 5.2 branch will have to go on 5.2 and master.
  • All tickets currently labeled as fixVersion=5.2 for CASA should go to 5.3. Anand (?) will automatically do this in JIRA. Anand - any instructions for us ? When do you plan to do this ?
  • test_predictcomp started failing
  • CAS-10629/CAS-10630 - CrashReporter temp directory permissions

Verification Testing

  • CAS-10481 : Serial vs Parallel HPC tests : At least one dataset where there was earlier a ~40% difference in rms noise is now showing nearly identical results for serial vs parallel, after being re-run with the latest master containing fixes for parallel data selection issues. Bjorn / Andy - any more details ?

Validation Testing

Architecture

Pipeline

  • Pipeline release branch created in svn
    • Joint developer / PWG telecon on Friday went reasonably well
    • Several last minutes errors processing single polarization ALMA data fixed at the last minue
    • Branch: https://svn.cv.nrao.edu/svn/casa/branches/project/Pipeline-CASA51-P2-B
    • Starting revisions number: 40752
    • This branch will be included in the CASA 5.1 / 5.2 series tarballs starting with CASA 5.1.0-63
  • HPC testing results are promising
    • MMS(s) appear to be filled differently than MS(s), e.g. empty spw(s) are removed from MMS(s) but not from MS(s), see CAS-10453
  • Field table access problem CAS-10620 fixed by Kumar
  • Added an error on exist file capability to the pipeline processing request modules on the pipeline trunk to help with workflow execution
    • Plan to port these to the trunk
  • Ongoing pipeline infrastructure / framework development planning

HPC

  • Serial tests, all with 5-1.0-61 / pipeline r40738 in subpage of https://open-confluence.nrao.edu/display/CASA/ALMA+Pipeline+Cycle+5+Testing
  • Massive speedup in tests as >10 nodes became available in the CV cluster.
  • Also, cvpost065-068 upgraded, 256 GB, IB net. Running smoothly.
  • 21 out of 30 serial tests finished, all others running smoothly so far and expected to finish by this weekend. Technical datasets "TEC" also fine.
  • Not many issues in serial mode, and all fixed. Good landscape for release:
    • Bug in imager field ids, only affecting one test dataset CAS-10620 the fix needs to be merged into 5.1 and master.
    • Issue with transitions in pipeline CAS-10614. Pervasive issue, but fixed in pipeline r40738 (exactly prerelease 5-1.0-61).
  • ... but just found new issue possibly in pipeline, hopefully minor: CAS-10636.
  • For serial vs. parallel validation, updated results for 3 problematic datasets ready CAS-10481. They seem to match better now.
  • We can start now / next week parallel parallel re-testing for 5.2 but resource availability is not so great any longer.
  • Note/Warning: pipeline updates after r40738 (r40753 as of 24th in the morning) could introduce bugs, and there seems to be very limited test coverage.

Development

  • Imager_BugFixes_5.1 -- No blockers for 5.1
    • Open : CAS-10317 : Mosaic slowdown. VLA case : being worked on. ALMA case : all is as expected
    • Open : New : CAS-10624 : Seg-fault from mosaic major cycle upon restart in build #48. Already fixed by KG in build #54 or #57. Requesting retesting by Todd.
    • Open : CAS-10538 : Ongoing w.r.to fix for parallel (array access?) error. Serial mode is OK.
    • Open : CAS-10250 : 7m+12m image cube coordsys definition (ambiguity)? : Tak - any update on this. Move to next release ?

    • Fixed : CAS-10264 : Fix a log message. Under verification by Amanda.
    • Fixed : New : CAS-10620 : Error in Field table access : Fixed by KG.
    • Fixed : New : CAS-10603 : Mosaic residual scaling off : Fixed by KG + setting of the default of conjbeams to False in the interfaces/python.

  • Anything else ?

  • Planning for the March 2018 release :
    • Please continue setting fixVersion=5.2 for tickets planned for the March 2018 release. Someone (Anand?) will do a mass edit in JIRA to move "5.2" to "5.3" at some point.
    • Received more detailed requirements from Remy for ALMA for the March 2018 release. No major changes to existing plans. Will update notes and talk to folks when needed.

  • CAS-10613 : SD imaging of solar data for Cyc5 : Note from Kana : the issue is being actively investigated. If not for 5.1, then this may have to go into our December product (5.2) as data will flow in/after December.

AOB

Developer Reports

Thursday Meeting
  • Sanjay Bhatnagar
  • Sandra Castro
  • Lindsey Davis
    • Participated in readiness review developer / PWG telecon August 18
    • Followup on HPC testing issues, mainly CAS-10453
    • Release notes editing, regression testing, branching
    • Added support for an error on exit file creation to the pipeline processing request execution modules
    • Cycle 6 pipeline infrastructure / framework development planning
  • Bjorn Emonts
  • Pam Ford
    • CAS-10597 (remove ms tool deprecation warnings for 5.1) - Ready for pull request after 5.1 branch merge
    • CAS-10604 (redo ms tool warnings for 5.3) - Ready to Verify
    • CAS-7049 (antenna selection for cal tables) - rescheduled after discussion with Sanjay
  • Enrique Garcia
  • Bob Garwood
    • vacation
    • CAS-10613 - sdimaging cannot image Solar data taken in Cy5 E2E. This is due to duplicate rows in the MS POINTING table. That's caused by duplicate values in the ASDM at subscan boundaries. Discussions are ongoing as to where the appropriate "fix" should be. It's not clear yet whether the filler will be changed to handle this. As commented above, if the filler is changed to handle this that will need to appear in the December product as it's probably too late for any fix to appear in 5.1.
  • Kumar Golap
  • Jeff Kern
  • David Mehringer
  • George Moellenbrock
  • Dirk Petry
  • Martin Pokorny
  • Federico M Pouzols
    • Parallel (little) and serial (mostly) pipeline tests.
    • Trying to organize work on various mstransform/cvel issues, CAS-10446, CAS-10584, CAS-10051, CAS-9241.
    • CAS-10584 for more information about partitions in parallel pipeline runs
  • Urvashi Rao
    • CAS-10264 : fix log message with product of image shape :
    • CAS-10423 : trying to write a unit test script that imports tests from another file, so as to not duplicate the code.... still trying.
    • Email and phone conversations about the December release and planning for the March 2018 release.
  • Darrell Schiebel
  • Ville Suoranta
  • Takahiro Tsutsumi
Friday NAOJ Meeting
  • Kanako Sugimoto
  • Wataru Kawasaki
  • Masaya Kuniyoshi
  • Takeshi Nakazato
  • Renaud Miel

Minutes of the meeting :

  • casa 5.1.0-63 will be the one to first connect to the pipeline, for a pipeline release candidate (for testing).
  • March release is 5.3. December product is 5.2 => Anand will do relabelling today -- 5.2 -> 5.3 fixVersion in JIRA. ( As of 10am MT, he has done this. ).
  • SD imaging issue : Talked later. Decisions made. Emails sent around about it. No last minute action needed from CASA for the 5.1 release.

  • George pointed out that we must add notes on confluence about 5.1.1 , 5.2, 5.3 with clear information about all the release dates and how they relate to the various ALMA/VLA deadlines. Also, the various releases are going to be called, and when branches will be made.

  • Docs : Kelly made a snapshot of docs for the 5.1 release. For 5.2, will clone the 5.1 docs.

  • test_predictcomp failure --> Need to follow up. Tak will send me email about what format this is that has broken. Need to follow up about casacore edits.

  • CrashReporter -> are we paying attention to reports from it right now ? Not really, but it has been used a couple of times to diagnoze problems. Juergen pointed out that we should iron out usability issues by the October meeting of the CUC.

*Lindsey : A Qt app error upon casa exit, is messing up shell scripts that look for exit criteria from casa. Darrell : may have already been fixed --- Ticket from Rafael on this. -- need to retest

  • HPC testing : looking good. Serial OK. Parallel - continue retests of the earlier failed tests. Resume full suite of serial vs parallel tests after all 5.1 fixes are in, and hand this to the PLWG as their starting point.

  • Newsletter contributions due on Sep 11 - if you want to contribute

-- MorganGriffith - 2017-08-15
Topic revision: r12 - 2017-08-24, UrvashiRV
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback