Thursday Morning Meeting April 13, 2017

  • DIAL-IN NUMBERS & PASSCODES:
  • IP: 192.33.117.12##8110
  • Phone: (434) 817-6524

Attendance

  • Socorro:
  • CV:
  • Garching:
  • SCO:

News / Meetings / Visitors

  • CARTA Interactive Clean
  • CARTA Face 2 Face
  • ObsMode
  • PLWG Face 2 Face

Build, Release

  • 5.0 Status
    • CAS-9984 and CAS-9985: Issue unflagging time- or channel-averaged data in plotms (flagging works)
    • Autoboxing: issue when nothing to deconvolve; fix is in, but waiting for next pre-release so testing can happen; Look for ticket CAS-9208.
      • Latest pre-release is from yesterday.
    • 2 Issues from CalSolve (Solution Interval Boundaries, Frequency access for pipeline) -- fixes went in Monday.
    • Co-ord issue in tClean -- cannot update spectral coordinates properly in HPC. Not a 5.0 issue. Follow up with Jeff.
    • Akeem to follow up on Test Suite 4:
      • Errors that are assigned to Sandra; she is on vacation right now.
      • Brian Kent is also doing something.
    • Fedrico: Regrid options in msTransform
    • VI/VB2 imaging testing -- Crystal did basic test and found inconsistent results (gridder = standard, multiscale). CAS-10050. Urvashi suggests postponing some of this to 5.1.

Verification Testing

  • Thanks to those who have begun writing task level unit tests
    • I'll begin pinging un-updated tickets next week
  • Continue work on the CASAguides
    • Akeem wants to make some tweaks to script extraction; Sabrina has been recommended as an advisor until Bjorn joins in July.
  • Verifying new tests in the test framework
  • Listcal regression failure update?
    • Traced to a data filling problem; the last row of one uvfits file had a scan set wrong. The test should us a dataset that is less expensive to fill and process.

Validation Testing

  • Federico and Enrique have fixed a cvel problem with 2 or more channels. CAS-9853.

Architecture

HPC

Development

  • Setting memory limits in CASA (pford, kgolap)
    • James Robnett reported that plotting is taking all of the available memory on lustre during his testing and wants to limit it. Jeff suggested environment variable CASA_MAX_MEMORY
    • Some tasks use HostInfo::memoryTotal(true) to check if "system.resources.memory" is set in ~/.casarc; this applies to all sessions on all hosts for a given user
      • The limits are not enforced, just a "suggestion"
      • All tasks which query this value get the same limit
      • HostInfo is up to date with current systems.
    • Would a setting in the casadictionary within the casa session be better than an external environment variable? Could be set dynamically in a python script and updated as tasks allocate and free memory
      • how to do it from c++
    • Darrell suggests possibly doing this through CASAUtils tool.
    • Pam will talk w/ James and look into CASAUtils.

Pipeline

  • CASA 5.0 testing
    • caltable metadata access issue resolved, e.g. CHAN_FREQ column access in SPECTRAL_WINDOW table works as before
    • new problem applying A Mueller table computed with vi/vb1 with applycal using vi/vb2, serial and HPC, confusion between correct and correct2
    • HPC tclean coordinate setting errors
  • first version of VLASS imaging pipeline is working, technically ...
    • Quick Look Imaging Pipeline is working. Single-epoch Imaging Pipeline is coming.
    • This is being tracked under the image refactoring ticket.
  • first EVLA reprocessing use case is working, technically
  • collaboration with ALMA archive developers restarted

AOB

  • CASACore
    • Martin is supposed to be responsible for CASACore now that Jim is gone, but Martin does not have privileges to merge changes.
  • Ville is seeking help dealing with readlocks for Mac NFS mount.

Developer Reports

Thursday Meeting
  • Sanjay Bhatnagar
  • Sandra Castro
  • Lindsey Davis
    • continued work on ALMA representative source identification heuristics
    • debugged pipeline caltable type determination heuristics, possibly pipeline callibrary problem
    • tested calibrator tool changes and fixes
    • discussions on VLASS imaging heuristics, ALMA archive storage and data model issues, ...
  • Pam Ford
    • Created python tests test_plotweather.py and test_plotcal.py (issue with subplot parameter on OSX)
    • Looked into memory limits in plotms and casa
    • Debugged CalibratingVi2 with calibration library file string not producing OTF corrected data in plotms, emailed George
  • Enrique Garcia
  • Bob Garwood
    • Finished reconciling CASA and ALMA versions of the shared ASDM code. Checking in will happen after the ALMA software process has progressed a bit further.
    • Working on a proof of concept test of the ASDM code move to a submodule in CASA.
      • cmake issues continue to cause headaches. Ville has agreed to look at it and see if he has any suggestions.
      • plan B is to move the ASDM to a submodule at the same level as casacore and asap.
        • That solution won't work on the ALMA tree. But ALMA uses Makefiles, which are perhaps less portable but easier to force it to do what needs to be done for a specific case.
    • Following the ASDM development wish list discussions.
    • Preliminary work on an ASDM use chase change that appears to break the lazy filler (looks like a poorly handled exception that's handled better by the non-lazy filler).
    • Some time on some background GBT tasks.
  • Kumar Golap
  • Jeff Kern
  • David Mehringer
    • Asked several more questions re: TVI framework. Answers continue to be slow coming back, so progress on statwt2 continues to move at a snails pace.
    • (CAS-10054) Intermittent importuvfits bug (wrong scan number on last row). Ran George's unit test over 100 times on my Mac. Passed every time.
    • (CAS-10059) Add UVFITS unittest to framework. Communicated with Akeem that failing test had been updated, so test_uvfits should be ready to add to automated test suite.
    • Discussed log messages from Image Analysis with Urvashi and Tak.
    • Need to know who on our team is responsible for reviewing and merging casacore PRs so Ger's PR to fix concat images can be merged.
    • Refactored some Image2DConvolver code.
    • Refactored some StatsImageCreator code.
    • Communicated with Darrell regarding libwcs error he is seeing from viewer code.
  • George Moellenbrock
    • Additional tweaks for VI2 migration:
      • Unchannelized calibration tables initialize nchan=1 in all spws in output caltables so QA2 methods don't choke
      • Refined iteration boundary control so solution combine and solint interact better (as we've become accustomed), e.g., combine='scan' with modest solint (not 'inf') won't break scans on an arbitrary grid
    • Investigated listcal_regression failure, which turned out to be a weird and intermittent bug in importuvfits in which the last row of the MS imported from UVFITS has it scan_number incorrectly incremented. This caused solution boundary issues which (surreptitiously) are minimized by the refined control noted above. This ticket (CAS-10054) now in Dave M.'s hands.
    • Studied Pam's OTF cal problems in plotms. Looks like testing framework within the CalibratingVi2 is getting in the way of general usage by code outside synthesis.
    • Misc. discussions with Martin and others re VI2 issues.
  • Dirk Petry
  • Martin Pokorny
    • CAS-7086: fixed issue when selecting reporting axes that create datasets which span spectral windows of various sizes
    • vi2 async i/o: working on design
    • [VLA: realfast]
  • Federico M Pouzols
  • Urvashi Rao
  • Darrell Schiebel
    • created test for imview and msview
    • added viewer fixes to prevent crash wrt CAS-6966
    • spent time looking into CAS-9361
    • explored wcslib utils
    • jira ticket weeding
    • explored counting lines of code
      • added cloc to casa yum repo
    • explored JanusGraph
  • Ville Suoranta
    • First e2e package build using Bamboo/Docker/Nexus
    • Installed new hard drives on Macs
    • Learned that OSX 10.12 added an undocumented feature which prevents mounting a dmg on a mount point on an external disk. Changed the build scripts to work around this and filed a ticket with Apple.
    • Worked with Tracy on Linux test machine disk configuration
    • In the midst of testing the shared Netapp data directory. Trouble on Macs, getting errors similar to this: https://casaguides.nrao.edu/index.php?title=Waiting_for_read-lock_on_file
    • Started working on a moving the current CI build logic to the common Groovy scripts
  • Takahiro Tsutsumi
    • Checked in modifications to add some warnings and fix a bug in automask
    • Investigating coordsys issue for parallel continuum tclean
Friday NAOJ Meeting
  • Kanako Sugimoto
    • This meeting will be the last one before Erik leaves the project (the last day at the office is 24th Apr.). Thank you very much Erik for all your hard work!
    • CASA 5.0 review discussion at NAOJ (Apr. 6th)
    • CASA 5.1 development cycle started at NAOJ
      • defined targets and assignments of CASA-SD 5.1: waiting for feed back from SsSes and developers
      • Developers should look into their assignments by the end of 17th and feed back SsSes for more description of the tickets (if necessary). If the schedule seems not appropriate, contact KS.
  • Wataru Kawasaki
  • Masaya Kuniyoshi
  • Takeshi Nakazato
    • fixed CAS-10039, ready for validation
    • fixed CAS-10049, waiting for a completion of Bamboo test
  • Renaud Miel

-- JeffKern - 2017-04-11-- JeffKern - 2017-04-11
Topic revision: r18 - 2017-04-24, JaredCrossley
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback