Monday Morning Meeting Dec 8, 2014

• DIAL-IN NUMBERS & PASSCODES:
• USA Toll Free Number: 866-901-8266
• USA Toll Number: +1-203-566-3863
• PARTICIPANT PASSCODE: 725760

Attendance

• Socorro: Jeff, Lindsey, Susan, Tak, George, Juergen, Jim, Pam, Kumar
• CV: Akeem, Andy, Mark, Dave, Darrell
• Garching: Justo, Dirk, Julian
• SCO:

News / Meetings / Visitors

• Next week, last Monday morning meeting of the year.

Build, Release, Testing

• Pre-release
  • Flag Command, Flag Data: Prerelease
  • Automated Testing in place, some failures
  • New Alma configurations
  • CAS-7475 asdm filler (Done)
  • CAS-7176: Applycal issue
  • RH-5 PlotBandpass Issue
• Telecon about including Parallel processing
  • issue about boost:mpi

Architecture

• Technical Forums:

Development

• Discussion of Julians e-mail on NaN handling
• Review of CASA 4.4 planning:
  • Kumar points out viewer issues (Triage Bugs for a fix list)
  • Darrell Testing CAS-6775, CAS-7091
• Dirk: Discussion of systemic velocity.

Pipeline

• Testing against casa 4.3 pre-releases
  • applycal slowdowns
  • new plotbandpass issue
• Internal testing for next release
  • Target imaging
  • Bandpass SNR heuristics
  • VO calibrator catalog retrieval service

AOB

Developer Reports

Monday Meeting

• Sanjay Bhatnagar
• Sandra Castro
• Lindsey Davis
  • Pipeline intergration, bug fixing, testing
  • CASA HPC meeting, HPC pipeline testing issue
• Pam Ford
  • Investigated time axis scale/ticks in plotms
  • Studied VI/VB implementation in plotms
• Kumar Golap
  • Worked 1.5 days to prove to a user that multiterm ft was doing the right thing
  • Sped up msuvbin for cases when the grid cannot fit into memory
    • started working on a way to incorporate w-projection in the process to remove the need for w-projection at the imaging stage
• Justo González Villalba
  • Attended CASA HPC meeting:
    • We need to agree on a set of tutorials covering most of use cases to test parallel/MMS operation
    • MPI Transport code from Kumar has to be evaluated and tested as a candidate to cover for MPI usage at C++
  • Discussed with George details about weight spectrum generation and virtual corrected data column operation
  • Contributed to investigation of performance problems observed in applycal
  • Weight spectrum:
    • Switched from median to mean for generation of WEIGHT out of WEIGHT_SPECTRUM to actually account for the impact of reduced weight in the spectral regions with significant systematic errors, made tests to validate vs numpy
    • Changed definition of WEIGHT_SPECTRUM / WEIGHT when all contributors are flagged to be the cumulative sum / mean of the spectrum of all the flagged contributors
    • Now in the process of changing definition of time centroid and averaged UVW to account for fraction of samples flagged
  • Working on a parallel version of an ALMA cycle 2 regression
• Jim Jacobs
• Jeff Kern
• Susan Loveland
  • Viewer : CAS-7185
  • CARTA: Work on colormap & histogram settings
• David Mehringer
• George Moellenbrock
  • CAS-7176: applycal slowdowns. Justo and I could not reproduce the CPU=0.0 state we managed to generate just once last Monday, which seemed to be 'hung' in a pathological I/O wait of some sort. Further investigations show lots of excessive reads on DATA column (already noted by James), which grow catastrophically e.g. when non-trivial channel selection is introduced, and keep the CPU pegged near 100%. Outwardly, Lindsey's local tests seem to look more like the excessive read case. Jim has introduced changes to the CASA tile cache size calculation that seem to help the excessive reads. Also, selecting against the ACs seems to ameliorate the excessive reads. We are hopeful this improves the applycal performance problems observed by the pipeline group. That avoiding ACs helps indicates that the ACs are stored awkwardly (cf XCs) in the TSM hypercube file; indeed, we have discovered that ACs are isolated from the XCs and, more generally, the physical order of the data in the file is not monotonic in row number. That the data is sorted awkwardly at this granularity is probably responsible for difficulty improving performance generally, or even understanding why our I/O patterns occur as they do. Since ACs and XCs increase in volume at different rates as nAnt increases, the way these effects scale is almost certainly non-trivial.
  • General discussions with Justo and Julian about visibility averaging and calibration in the transforming VI2 framework.
• Dirk Petry
  • started work on MSFitsInput and Output to enable handling of sysvel info in UVFITS files
  • discussion with Kumar (and Juergen) on how clean would put sysvel into the image
    • need way to store sysvel in image such that exportfits can hand it on to FITS image (and importfits can put it back)
    • propose to give clean a parameter "sysvel" analogous to "restfreq"
      • clean will fill the info from sysvel into newly created location for "sysvel" in CASA image
      • exportfits will take sysvel and fill it into the standard FITS keyword "ZSOURCE" (and the reverse for importfits)
      • to make the connection to the SYSVEL information in the MS SOURCE table, an alternative input format would permit user to enter a pair (Python list) of FIELD and SPW ID to enable clean to take the SYSVEL from the SOURCE table.
• Urvashi Rao
• Darrell Schiebel
• Rob Selina
• Julian Taylor
  • Attended CASA HPC meeting
  • triaged slow performance of asdm2MS (CAS-7191) and helped triage poor disk access pattern of applycal (CAS-7176)
  • made prototype rpm packages of openmpi and mpi4py for new parallel framework, currently deployed on ESO machines for testing
  • investigated simple performance improvement via using fortran rules for complex multiplication
• Takahiro Tsutsumi
  • Fluxscale
    • (CAS-5191) fixed the bug found during testing. Mainly log message fix when the gain threshold parameter was used.
    • Fixed the unit test failure on the Mac OSX prerelease
  • Setting up and building trunk on OSX 10.8 on cbt-d12-2
  • Investigated the SetJy issue reported by Justo for OSX10.8

Friday NAOJ Meeting

• Kanako Sugimoto
• Wataru Kawasaki
• Masaya Kuniyoshi
• Takeshi Nakazato
• Shinnosuke Kawakami