Thursday Morning Meeting 10 August 2017

  • DIAL-IN NUMBERS & PASSCODES:
  • IP: 192.33.117.12##8110
  • Phone: (434) 817-6524

Attendance

  • Socorro: Juergen, Urvashi, Martin, Pam, Lindsey, George, Tak
  • CV: Dave, Darrell, Jen, Bjorn, Anand, Ville, Akeem, Andy, Bob
  • Garching: Enrique, Federico, Dirk
  • SCO: ?

News / Meetings / Visitors

Build, Release

Verification Testing

  • Akeem has been documenting his work (his last day is August 18th)
  • Andy has been working on HPC pipeline validation

Validation Testing

  • Current validation tally
    • 148 Under Validation, 76 Ready to Validate.
    • 19 tickets went RtV this week so far. 22 tickets went through Validation to Resolved this week so far.
    • July 28-Aug 3: 23 tickets assigned for validation. 28 tickets resolved.
    • aiming to have most validation testing finished by Friday for branching on 8/15, but if blockers are not fixed, we can wait
    • Amy Kimball will be helping on the VLA side with getting and organizing testing resources via the new home for CASA testing under SSR after Oct 1
  • Current Testing Efforts for 5.1 deliverables
    • ALMA Cy5 pipeline, crossover between CASA and PL
      • autoboxing (CAS-10415, CAS-10457). plus new seg fault?
      • mosaic issues CAS-10317, CAS-10352 (ready for DirkM to test this morning). blockers to the 5.1 release.
    • statwt2
      • has been merged into 5.1. will not be formally announced, but VLASS will use the pieces they have validated (and keep using statwt). thorough testing and possibly some new requests in 5.2.
      • will create one testing ticket for 5.2 that encompasses all of the features needing validation
    • parallelization
      • no update, team keeps finding bugs
    • miscellaneous
    • plone (lower priority given short development cycle)
      • expecting a few more pages to be included and tickets resolved for 5.1, but not many more
      • plan to include the rest of the tasks in plone in 5.2, rest of the tools in 5.3. stay tuned for this organization to come after 5.1 release.

Architecture

HPC

Development

  • Autoboxing - Tak ?
  • Kumar : Field table efficiency (fixed), selectdata slowdown (done?), mosaic slowdown : investigating
  • Statwt2 : merged. Will continue validation and incremental features for 5.2
  • Waiting for one more pull request from NAOJ. Nearly done there.
  • Bugs from HPC testing of the current ALMA pipeline in parallel
    • Bjorn/Andy looking at test results. Bjorn's reports CAS-10481 suggest that serial vs parallel cube is scientifically equivalent when no other bugs are encountered. Found 3+ datasets where there appear to be data selection (and csys generation) differences between serial vs parallel. Bjorn - Update on new issues you/Andy have been uncovering ?
    • Data selection / csys generation problems : CAS-10434, CAS-10492 : Blockers ? Kumar/Tak - ideas ?
    • Viewer bug with MSs with differently shaped spws : CAS-10470
    • Applycal errors in parallel : CAS-10453 : Lindsey has followed up with Brian Kent - update ?
    • CAS-10352/CAS-10507 : Columslicer error in tclean : KG fixed it.
    • Issues related to running the tests on 64GB RAM machines (or less) vs how the pipeline uses it ( 128GB in Chile, 256GB in Charlottesville ) : CAS-10392 for findcontinuum.py bottleneck, CAS-10484 for the autoboxing segfault.

  • CASA 5.2 planning
    • Attached document (https://safe.nrao.edu/wiki/pub/Software/CASA/Aug1017/CASA_5.2_Planning_V.2.pdf) has details for ALMA/VLASS/Internal/Ongoing-noncritical issues, plus a listing per developer. Please read and ask questions if you have any or request changes. This was put together after consulting each developer, but please let me know if you find something unexpected.
    • Please send me a list of JIRA tickets that cover this planned work. Please make new ones when needed.
    • Discussion is ongoing about how we should respond to a proposed mid-cycle ALMA pipeline patch, which may need fixes from CASA as well. There are pros/cons to any solution.
    • There are many old/unused? tasks in CASA, and the plan is to not try to document them, but to identify which need to be replaced or begin the deprecation process. Jen and I made a list and I will follow up offline (or in another weekly meeting, when we have time).

Pipeline

  • Parallel testing
    • New / old tclean column slicer error in serial mode (CAS-10507 should be fixed)
    • Coordinate system errors in serial mode (CAS-10434, CAS-10492, under investigation)
    • Applycal error in pipeline hif_uvcontsub task is worrying but based on web log review seems to be harmless (note this is VI1 error, with MMS data set, but with parallel processing disabled)
    • Fixed flux catalog database service error in serial mode, parallel mode behavior seems different still under investigation (found with CAS-10453)
  • 5.1 Release
    • Dirk M. back from vacation so lots of last minute pipeline imaging tweaks and bug fixes
    • Autoboxing masks if present have been added to the pipeline data products
    • Turning on wvrgcal task remcloud feature in pipeline is under discussion
    • Remaining tickets are for small web log and logging enhancements
    • On going acceptance testing

AOB

Developer Reports

Thursday Meeting
  • Sanjay Bhatnagar
  • Sandra Castro
  • Lindsey Davis
    • Added autoboxing masks to the list of ALMA interferometry pipeline imaging data products
    • Cleaned up various pipeline web log reporting issues
    • Reviewed ALMA pipeline testing and deployment plan (with Rachel R)
    • Reviewed remaining 5.1 pipeline tickets (with Remy and others)
    • Follow up on pipeline issues turned up during parallel pipeline testing
    • CASA 5.2 planning discussions (CASA coordination meeting) and followup
  • Bjorn Emonts
  • Pam Ford
    • prototype for time axis tick labels (CAS-10488) coded, waiting for feedback from Juergen
    • ms tool deprecation warnings complete (CAS-10483), casarc plone doc ready to verify (CAS-9294)
    • Investigating CAS-9053 Test Suite 2 failure
    • started plotcal replacement effort
  • Enrique Garcia
    • CAS-9945: improve the approximations for effective bandwidth and effective resolution in split/mstransform
    • CAS-5174: UV Continuum Subtraction TVI
  • Bob Garwood
  • Kumar Golap
  • David Mehringer
  • George Moellenbrock
    • Resolved CAS-9357: fluxscale no longer leaves MS table open when exception raised
    • Resolved CAS-10493: repaired gencal's caltype='gc', which broke when CAS-8540 was checked in.
    • Analyzed ALMA EOP error
    • 5.2 planning
  • Dirk Petry
  • Martin Pokorny
    • Still shepherding changes in casacore for async vi2.
  • Federico M Pouzols
    • Parallel pipeline tests
    • CAS-10464 Fix regression test "MPI Alma M100 Analysis Hpc Regression Tclean", seems fixable, double-checking with tclean experts.
    • CAS-9089 + ICT-10558: following on, will require further feedback to align gencal with "correctMyAntennaPositions".
  • Urvashi Rao
    • A lot of talking and emailing.
    • Called a CCC meeting to go over V.1 of 5.2 plan.
  • Darrell Schiebel
  • Ville Suoranta
  • Takahiro Tsutsumi
Friday NAOJ Meeting
  • Kanako Sugimoto
  • Wataru Kawasaki
  • Masaya Kuniyoshi
  • Takeshi Nakazato
  • Renaud Miel

Minutes

  • CASA Users committee meeting is likely to be in the week of October 30, in Charlottesville.
  • We're on track for making the release branch on Aug 15. We have a couple of blockers coming from the HPC tests, but they're being worked on and fixes will just go into both branches over the next two weeks.
  • Documentation release plan was made (contact Jen/Juergen for details)

-- PamFord - 2017-08-09
Topic attachments
I Attachment Action Size Date Who Comment
CASA_5.2_Planning_V.2.pdfpdf CASA_5.2_Planning_V.2.pdf manage 55 K 2017-08-10 - 01:02 UrvashiRV CASA 5.2 development plan details
Topic revision: r12 - 2017-08-10, UrvashiRV
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback