Thursday Morning Meeting 27 April 2017

  • DIAL-IN NUMBERS & PASSCODES:
  • IP: 192.33.117.12##8110
  • Phone: (434) 817-6524

Attendance

  • Socorro: Juergen, George, Lindsey, Martin, Urvashi, Kumar, Pam
  • CV: Anand, Ville, Darrell, Andy, Akeem, Bob
  • Garching: Sandra, Enrique, Federico
  • SCO:

News / Meetings / Visitors

  • CASA Lead and CARTA positions - close April 28
    • Morgan is building the CASA lead hiring committee
  • PM position
    • Morgan is meeting with Lory week after next
  • Pipeline WG Face to Face: May 8-12
  • CARTA Face to Face: June 19-23
  • Obsmode meeting was this week. Good discussion. More topics than time available. Some topics continue today. Prioritization will be done sometime this week.
    • ALMA Director will be taking science leave; Stuart will be replacing director.

Build, Release

CASA-5.0 Status. Holding the branch for these:
  • Autoboxing -- fixes are in. Waiting for next round of testing. The work turned out to be more than was initially expected. There is more to test than what tickets started off with.
    • Amanda leaves Saturday. Ilsang will be doing testing once build is available.
    • Urvashi says VI/VB2 issues are completely separated from autoboxing. Memory issue will not impact autoboxing.
  • VI/VB2 memory issues in tclean - Lindsey is concerned about impact on pipeline. Urvashi and Kumar will discuss offline.
  • Polarization Solve:
    • Fixed! Mainly issues relating to (new) use of weightSpectrum: iterative D solve was neglecting to zero parallel-hand weights, data prep for KCROSS solve was incorrectly imprinting bandpass in the weightSpectrum.
    • Frank has been asked to start his pipeline run again as test. George thinks this issue should not hold the branch any more.

Verification Testing

  • CASA guides are essentially done
    • Some clean calls don't work non-interactively (since clean does not save mask files non-interactively)
    • Are CASA guides going to replace all clean calls to tclean soon?
  • Memory profiler is being tested on a pipeline run (this run is on Osiris), before implementation on test machines
    • It will profile memory usage on each machine (not each test)
  • Continue to put tests in to the framework
  • Status of M100 regression failure? M100 Serial, M100 tclean serial, M100 HPC, M100 HPC tclean - Sandra

Validation Testing

  • VI2 - hopefully testing will resume in earnest next week (once obsmode is over)
  • Calibration testing
    • For pipeline there are changes, but very minor.
    • VLA side has been waiting on polarization.
    • Pipeline is using VI1 in serial mode.
  • Pipeline is not planning to use any of the uvcontsub tasks (including mstransform) because of the way they deal with the ms. Pipeline requirements should be written down. This will be discussed at stakeholder meeting next week.
  • Polarization is ready for manual testing of vi2.
  • We now have a machine for testing OS 10.12. Jen does not know when NRAO will begin supporting 10.12.
  • VLA team has given us support for validating the last few documentation chapters.

Architecture

HPC

  • ALMA pipeline testing
    • new problem with tclean that suddenly became more memory hungry. It seems to be only when using VI2. (CAS-10086)
    • problems with tclean not congerving when doing a cube imaging in parallel. For sure it happens with VI2, the test with VI1 didn't finish, but it was taking very long, so likely it also happened with old VI framework.
    • Verified that CAS-10078, the problem with a coordinate system mismatch in tclean doesn't happen anymore
    • Started to fix some minor issues with the log origin in parallel. (CAS-10074)

Development

Pipeline

  • CASA 5.0 testing
    • vi/vb2 issues
      • polarization calibration for VLASS pipeline
      • tclean memory issues
    • tclean convergence issues (not sure if they are new)
  • VLASS
    • imaging pipeline workflow issue resolved

AOB

  • Ville will push for Bamboo update before branch release (which will be no earlier than Wednesday, we think)
  • CAS-9607 - George added a comment; Anand will check with Brian about when George should check in changed code.
  • Martin now has ability to merge CASA Core changes.

Developer Reports

Thursday Meeting
  • Sanjay Bhatnagar
  • Sandra Castro
  • Lindsey Davis
    • continued work on passing scheduling block scope science goal information to the pipeline
    • investigated some pipeline SCOPS tickets
    • imaging heuristics refactoring discussions
    • continued work on passing scheduling block scope science goal information to the pipeline
    • investigated some pipeline SCOPS tickets
    • imaging heuristics refactoring discussions
  • Pam Ford
    • CAS-10041: implemented CASA_MAX_MEMORY limit check in plotms cache loading
    • CAS-9053: began work to add atmospheric overlays to plotms (like plotbandpass showatm=True)
  • Enrique Garcia
    • Looking into some corner cases found after validation of CAS-9864 with datasets that contain a mix of polarizations settings
    • Troubleshooting a mstransform problem in CAS-9944. It turns out to be a problem with the structure of the ms from LOFAR
  • Bob Garwood
    • (CAS-9882) creating test_importvla
    • continued exploring cmake + asdm module. Working on a suggestion from Darrell to make plan A work.
  • Kumar Golap
  • Jeff Kern
  • David Mehringer
    • (CAS-9996) StatWt Rework: Initial Functionality: Commited initial, "bare bones" functionality. weights computed on a per row, per correlation by basis. tool method: ms.statwt2(), task: statwt2. Includes C++ and python tests.
    • Received helpful, design-like info from George re: dealing with computing weights over multiple time stamps in TVI framework, so began looking into time binning feature for statwt.
  • George Moellenbrock
    • Fixed VI2-related issues in polarization solving
    • Provided some answers and pseudo-code for Dave's statwt efforts
    • (CAS-9607) tec_maps now makes a TEC plot file (for pipeline) (checkin pending 5.0 branch)
  • Dirk Petry
  • Martin Pokorny
  • Federico M Pouzols
    • Various tests of the ALMA pipeline, this uncovered tclean/vi2 memory hungryness: CAS-10086. Note there also seems to be ~25% increase in run time which hopefully goes away when the memory issue is alleviated.
    • Working on CAS-9089: automatic generation of antenna position corrections from web service / ALMA TMC DB. Extending correct_ant_posns.py, and unifying error handling ALMA/EVLA. Trying to figure out with ALMA-ICT developers why it times out. Q: do we have best practices/recommendations wrt Python exception hierarchy in CASA?
  • Urvashi Rao
  • Darrell Schiebel
    • (finally) fixed the history error upon 2nd syntax error (that Bob found)
    • CMake modifications for workspace build/install that is completely separate from source tree (to be committed after 5.0 branch)
    • looked into Bob's CMake rabbit hole (and found what seemed like a workable solution to me)
    • more work with JanusGraph
  • Ville Suoranta
    • Created a script that lists the prerelease packages available in Bamboo regardless of whether they've passed the validation: https://open-bamboo.nrao.edu/browse/CASA-LUP/latest/artifact/JOB1/Prereleaselinks/UntestedPrereleases.html
    • Extended the aforementioned to provide a simple way to get the latest (EL6) package for testing at ESO
    • Added disk space notification on the build agents
    • Added a "delete all workspaces" option to the current Bamboo agent cleanup script. The default option is to delete workspaces if the Bamboo branch doesn't exist.
    • Provided ic2233 OSX core dump backtrace for Kumar
  • Takahiro Tsutsumi
    • Fixed bugs in automask uncovered by the testers and by myself (CAS-9208)
    • Looked into fluxscale modification for handling gain solutions affected by RFI
Friday NAOJ Meeting
  • Kanako Sugimoto
  • Wataru Kawasaki
  • Masaya Kuniyoshi
  • Takeshi Nakazato
  • Renaud Miel

-- PamFord - 2017-04-26-- PamFord - 2017-04-26
Topic revision: r15 - 2017-05-04, JaredCrossley
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback