Monday Morning Meeting 21 November 2016

  • DIAL-IN NUMBERS & PASSCODES:
  • IP: 192.33.117.12##8110
  • Phone: (434) 817-6524

Attendance

  • Socorro: Jeff, George, Martin, Lindsey, Urvashi, Juergen, Pam, Tak
  • CV: Dave, Andy, Darrell, Akeem, Anand, Bob,
  • Garching: Dirk, Sandra, Julian, Justo
  • SCO:

News / Meetings / Visitors

  • Thank's to Susan. She has now moved back to the NM Systems group working on the OPT.
  • Reminder Short week in Us -- Thanksgiving.
  • No Meeting next Two Weeks: Jeff will be at yearly planning meeting in Chile
  • Justo giving a talk on GSL fitting tomorrow at 9 MT (SOC-317, ER-245)

Build, Release

  • Some issues with bamboo configuration last week.

Verification Testing

  • Disk quota issue for the accepted tests fixed
  • M100 SD test was failing
  • Sim Ppdisk was failing, but has started working again
  • Serial Refimager is currently failing, the only failure in the accepted tests list
  • Exploring resource usage of CASA test framework

Validation Testing

Architecture

HPC

  • We have been monitoring the memory use of the pipeline using a few datasets defined in (CAS-9265). Started with a small data set that exposed a bug in casacore on how/when temp images are written to disk. The peak use of memory is in tclean. We have seen very long I/O use, which could be related to CAS-8769 created by Julian sometime ago
  • last run in parallel failed on the applycal stage. It's due to tables left open in the cache. Lindsey verified as well and created a ticket to fluxscale.
  • Dirk Muders has updated the pipeline to support the fixes made by Ger on the concatenated images. He sees a different error now, Exception from task_tclean : PyParallelCubeSynthesisImager instance has no attribute 'pbcorImages', which we will have to investigate.

Development

  • Plan for switching tclean to vi2/vb2 permanently
    • The last time switch was thrown there were problems.
    • There is currently a performance problem that is data set dependent. Large pointing tables cause drastic slowdown.
    • We are not yet ready to make the switch.
    • Continue to make this a priority.
  • Plan to refactor chanchunk in tclean to be using parallel cube code
    • Tak is working on autoboxing now. This must be finished before we can move forward with this. Probably not going into 5.0.
  • Plan forward for interactive tclean:
 
                       if{Pipeline or SRDP forward} 
                                not needed 
                       elseif {carta is going to take 2 years to have it} 
                                then deliver request in old viewer 
                      elseif {carta is going to take 6 months} then 
                                do nothing for now 
                       else 
                                Some other stuff
    • Add a save model button to GUI?
    • Some discussion of adding Python-based GUI. Use case is not well enough developed yet.
  • Plan for scatter plot of big data
    • should not plotms use something smarter (e.g datashader). There is a datashader implementation underway which would plot a color-coded histogram in place of a scatter plot.
    • Datashader does not know about the MS details.
    • Options:
      • Have 2 tools: ploms, datashader
      • Put algorithms of datashader into plotms. Then the user selects which algorithm to use. There is an issue about losing interactivity if datashader is used. However, huge scatter plots may implicitly not have interactivity.
  • Plan for ms tool
    • Will discuss at next meeting
    • view construction, vi2/vb2, on the fly calibration

Pipeline

  • Draft ALMA pipeline cycle 5 requirements are available for review, deadline for all pipeline cycle 5 requirements is Nov 30
  • Implemented ALMA pipeline image size mitigation heuristics
  • Ongoing discussion on pipeline parameter abstraction issues
  • Issues of concern
    • tclean autoboxing
    • clearing the table cache

AOB

Developer Reports

Monday Meeting
  • Sanjay Bhatnagar
  • Sandra Castro
  • Lindsey Davis
    • Image size mitigation heuristics discussions with Dirk M., testing
    • Debugged and fixed pipeline processing request type conversion problem
    • Debugged pipeline HPC problem caused by
      • Tables left in table cache after fluxscale exception, wrote ticket
    • Reviewed draft ALMA Cycle 5 pipeline requirements
    • Pipeline code reorganization and refactoring ./lib/python2.7/task_tclean.pyd
    • Discussed pipeline parameter architecture issues with Jeff
  • Pam Ford
    • Further JIRA/Bitbucket/Bamboo testing with rescheduled ticket
    • plotms ticket cleanup
    • ms tool refactoring
  • Bob Garwood
  • Kumar Golap
  • Justo Gonz├ílez Villalba
  • Jim Jacobs
  • Jeff Kern
    • Liaison Interviews
    • PEPs
  • Susan Loveland
  • David Mehringer
  • George Moellenbrock
    • VI2 in calibration: CAS-9100,9101, and related VisCal tech debt.
    • Responded to some other JIRA noise and user miscellany. (5.0 undated/scheduled cleanup TBD...)
  • Dirk Petry
  • Martin Pokorny
  • Urvashi Rao
    • Checked in a modified tclean divergence termination criterion. Awaiting test results
    • Started on the next ticket - to calculate psfsidelobe level properly
  • Darrell Schiebel
    • reworked casa startup (for systems with new third-party packages installed)
    • reimplemented "assignmentFilter.py" after hooks were lost in ipython upgrade
    • new third-party packages (linux and osx) are ready but have not yet been well tested
  • Ville Suoranta
  • Julian Taylor
  • Takahiro Tsutsumi
    • automask - implementation of the multi-threshold autoboxing is mostly done
      • wrote the binary dilation code to grow the mask
      • the task interface changes were done (usemask="auto-multithresh")
      • pruneRegions may need some improvements
      • not done much testing yet and so far everything was checked in trunk only
Friday NAOJ Meeting
  • Kanako Sugimoto
    • investigated and fixed M100 regression: changes for CAS-9011 initiated the failure. Updated reference values after discussion with Takeshi.
    • fixed ppdisk2 regression: the fix for gap in atmospheric transmission initiated the failure but the changes to result was very slight. Also found a long standing issue in order of random number generation in function calls and fixed it. The outcome of OSX and RHEL are now consistent in corruption classes.
    • MS transition: sideband separation algorithm ( CAS-8091) - started work.
    • Created task documentation tickets of single dish tasks: Erik and tester team kindly suggested taking care of it.
  • Wataru Kawasaki
  • Masaya Kuniyoshi
  • Takeshi Nakazato
  • Renaud Miel
    • CAS-8082
      • Looking into header export in CLI mode.
      • Current approach implemented in PlotMSPlotter for exporting the header should work, but needs to be implemented in the QPPlotter.
Topic revision: r14 - 2016-12-01, JaredCrossley
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback