Monday Morning Meeting 27 March 2017

  • DIAL-IN NUMBERS & PASSCODES:
  • IP: 192.33.117.12##8110
  • Phone: (434) 817-6524
Minutes in blue

Attendance

  • Socorro: Jeff, George, Lindsey, Pam, Kumar, Juergen
  • CV: Darrell, Ville, Jen, Andy, Akeem, Bob, Anand, Dave
  • Garching: Sandra, Fredrico,
  • SCO:

News / Meetings / Visitors

  • CASA Dev Talk tomorrow: "VI/VB2 and Transforming VI" by George Moellenbrock. 1500 UTC / 9am MT / 11am ET. Socorro SO-317; Charlottesville ER-209. Video 192.33.117.12##6517; Audio only 434-817-6284.
  • Meeting Time Change: Starting next week, the CASA weekly meeting will be moved to Thursday 8am MT / 10am ET / 1400 UTC in rooms ER-245 and SO-280. The telecon information is unchanged.
  • Organizational Issues:
    • Jeff is expecting to continue doing both jobs for another month or two; he expects to complete the transition by mid-summer.
    • Until Jeff's replacement is found, Jeff is considering the possibility of forming a CASA managerial group consisting of Anand, Jen, Juergen, Sandra, Andy, and maybe others. This group would be tasked with interim CASA leadership responsibilities. This group would probably have a separate meeting to discuss management issues.
    • The interim lead may be either Morgan Griffith or Brian Glendenning.
  • JIRA changes made Friday:
    • Workflow updates: Corrected some errors and added transition Resolved -> Open and administrator-only transition Closed -> Open.
    • Issue Security enabled.
      • The CASA project now has 3 security levels that can be set for every issue:
        • Internal (default): The issue content is not appropriate for public review.
        • Public: The issue content is appropriate for public review.
        • None: Equivalent to Public and built into JIRA.
      • All members of the CASA-Developer or CASA-Users groups can change the security level.
      • As of Friday, all existing tickets were updated with security level "Internal".
  • Juergen plans to announce CASA 4.7.2 today.

Build, Release

  • CASA 5.0 Status
    • Remaining issues: autoboxing, VI/VB2, documentation, cvel
    • Jeff is setting a target of April 7 for cutting a CASA 5.0 branch.
    • After cutting the branch the svn sync may be stopped for the CASA group (not for the pipeline group)
    • The removal of T and F as aliases for True and False has caused some concern about updating scripts and documentation.
  • Data repository
    • Data has been copied to the new git repository. (git does not support large file storage, but there is a community-supported add-on that does allow for large files)
    • All files below the top level are stored as large files.

Verification Testing

  • New test hardware: helpdesk is in the process of setting up 4 linux VMs (2 RHEL 6, 2 RHEL 7)
  • Tracking build and test machine activity
    • Results are currently kept in Akeem's user area in CV: /users/awells/cpu_stats/IMAGES
  • Conversion from regression test framework to unit test framework
  • Accepted tests that have been failing a while:
    • pipelineTest
    • Refimager
    • Alma M100 Analysis Regression
    • Alma Ngc3256
  • CAS-9823 - tool throws a NameError and will not execute if the boolean "True" is not explicitly written out

Validation Testing

Architecture

HPC

  • ALMA Pipeline parallel testing
    • Resumed testing on other datasets. Have added 3 datasets running automatically in ESO's Jenkins. We are having trouble catching failures from the pipeline. It exits with status 0, which Jenkins catches as a success. For example, the failure inside tclean due to the VI/VB2 minMax() isn't rethrown and is never caught later no matter how we try.
    • Have made experiments with cpu pinning (to use only one socket); the gain is very small in some cases. Need to experiment further....
    • Verification of CAS-9604: PyParallelCubeSynthesisImager instance has no attribute 'pbcorImages'. Had similar problems with gaincal seg faulting due to VI/VB2 switch. Lindsey told me about the env variable to by pass it. Running test again....
    • Verified Urvashi's fix to CAS-9587 failure in tclean (parallel continuum mode) when image is not squared. It's fixed. Thanks.
    • Verified Urvashi's fix to CAS-9605 on the creation of node files when running tclean in parallel. It's fixed for our test cases which switch off the web log creation. Further discussion are on-going related to supporting concatenated images in some ia tools.
  • EVLA Pipeline parallel testing
    • Long_runs dataset: it stopped inside stage 6 (applycal). I think it run out of memory on casa-dev-09. This machine has only 8Gb RAM. Will verify it in lustre nodes.

Development

  • Newletter -- Jeff needs to review email from Mark Adams and pick a date.

Pipeline

  • HPC / tclean imaging working again * Still an issue updating the headers of virtually concatenated images
  • Calibrator solve VI/VB2 issues
    • Pipleline segmentation fault, and numerical differences with VI/VB1 run
  • Continued progress in imaging heuristics parameter handling code refactor to better support VLASS

AOB

Developer Reports

Monday Meeting
  • Sanjay Bhatnagar
  • Sandra Castro
    • Plone Plone Plone!
    • Followed up on CAS-9945, a failure in split when width>1 when writing the value of TOTAL_BANDWIDTH column in the SPW table, whenever CHAN_WIDTH = EFFECTIVE_BW. This seems to affect all ALMA data after Cycle 2. As far as I understand it the correct values of EFFECTIVE_BW and RESOLUTION started to be written to the ASDMs for Cycle 3 and this change was never requested to be included in CASA tasks that write an output MS. Currently, these are all mstransform-based tasks. In progress...
  • Lindsey Davis
    • Refactored pipeline importdata ASDM / calibrator catalog flux handling code
    • Calibrator solve VI/VB2 testing
  • Pam Ford
    • ms tool questions
    • plotms bugs
  • Enrique Garcia
    • Fix CAS-9864. mstransform fails if there is channel averaging and spw combination. Fixed but the whole mstransform guessing of compatible inputs is messy and error prone. Most of it should be gone when moving the whole mstransform to TVIs.
    • Create jobs in ESO Jenkins to test off git rather than svn.
    • Created report for the comparison of uvcontsub/uvcontsub3/msttransform (CAS-5174). Reviewing it carefully with Justo. Bottom line: the differences reported by Jennifer are expected.
    • Reformat TVI guide in Plone (CAS-9906).
  • Bob Garwood
    • Looked at the differences between ALMA and CASA shared code and made recommendations to how to resolve the differences. That will lead to tickets in the ALMA jira and code changes there. At which point the ALMA version will be the reference version for CASA and the starting point for the mode, templates, and code move out of the ALMA code tree.
  • Kumar Golap
  • Jeff Kern
  • David Mehringer
    • (CAS-9605) Parallel tclean in specmode='cube' makes only *.n1.pb: Discussion with Ger about how to form initial misc info in concatenated images and how users can change misc info after concatenated images have been created.
    • Lots of code diving to try to understand design of applications (eg ms tool methods) that use TVI framework.
    • (CAS-9416) create plone docs for imval: made requested changes to XML parameters descriptions.
    • (CAS-9410) create plone docs for imsubimage: Responded to queries re: documentation and subparameters
    • Reimplement ImageCollapser::_doMedian to use stats framework. Resolved. Implemented.
    • (CAS-9945) Split inflating the BW?: Replied to query re: from where listobs gets info for band width column in spectral window table it logs
    • (CAS-9391) create plone docs for immoments: responded to documentation queries.
    • (CAS-9881) Existing unit tests that cover importuvfits?: created unit test for importuvfits
    • (CAS-9937) refactor ImageCollapser: Completed. Various refactoring.
    • (CAS-9935) Any unit tests for specsmooth?: Responded to query if there are unit tests for specsmooth. Yes, there are.
    • (CAS-9370) create plone docs for imhistory: Responded to request to spell correct imagenam -> imagename in example in this (and other as it turns out) XML files.
  • George Moellenbrock
  • Dirk Petry
  • Martin Pokorny
  • Federico M Pouzols
    • Quick "fix" for issue in mstransform pre-averaging + regridding ( CAS-9798) in 4.7.2. What are the current plans for 5.0?
    • Testing performance/resource requirements of ALMA pipeline in parallel (as part of CAS-9120 and CAS-9519)
    • Clarification and verification of CAS-9077 with Urvashi (mpi4casa mpi commands, return values and advanced use)
    • Following plans and tests around tclean ( CAS-9852) and cvel ( CAS-9241) + mstransform ( CAS-9798)
    • Looking into issues in parameter initialization & defaults in parallel tasks ( CAS-9871)
  • Urvashi Rao
  • Darrell Schiebel
    • mirroring and verification of git data repository
    • finished plone docs for data repository in git
    • converted /home/casa/data (@nrao) to us git/lfs data repository
    • added T/F, true/false, to ipython command line interface
    • added the ability for use "from tf import *" to import boolean T/F and true/false into python
  • Ville Suoranta
  • Takahiro Tsutsumi
    • Investigated automask pruning issues
Friday NAOJ Meeting
  • Kanako Sugimoto
  • Wataru Kawasaki
  • Masaya Kuniyoshi
  • Takeshi Nakazato
    • CAS-7610 intermittent failure of sdimaging unit test
      • some improvement on unit test code
      • Sanjay has recently fixed a bug in MSSelection::getChanFreqList (pull request #578 in GitHub casacore). I'm not sure it is a cause of the failure in sdimaging unit test. But, at least the test doesn't fail since the fix comes into casa.
  • Renaud Miel
    • CAS-8082 PlotMS Header: PlotMS Cache code review. MeasurementSet and Table API check + test.

-- KanaSugimoto - 2017-03-23
Topic revision: r22 - 2017-03-29, JaredCrossley
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback