Monday Morning Meeting - March 23, 2015

• DIAL-IN NUMBERS & PASSCODES:
• USA Toll Free Number: 866-901-8266
• USA Toll Number: +1-203-566-3863
• PARTICIPANT PASSCODE: 725760

Attendance

• Socorro: Joe, Susan, Lindsey, Pam, Martin, George, Tak, Jeff, Jim, Kumar,
• CV: Mark, Ville, Darrell, Andy, Akeem, Dave
• Garching: Sandra, Dirk, Julian, Justo
• SCO:

News / Meetings / Visitors

• CASA Survey now closed: Total 252 responses
• CUC April 1

Build, Release, Testing

• 4.4 Status: Mark
  • Blocked by missing OSX package, Darrell will send to Kana
  • Sensitivity and Cal Table Done
  • Plot-MS: Still need flagging, continuing working on it.
    • Justo and George: Flagging issue
  • Failure on Sim51C
  • High nails in terms of general testing (systemic things)
  • Number of open files (needs to be tested see if the multi-file from Ger fixes the problem)
  • MPI for CASA: Sporadic failure Issue making system call at the end of partition.
• 4.4.114 is broken for ALMA
  • Discussion on testing

Architecture

• Nothing this week:

HPC

• Latest minutes are in: https://safe.nrao.edu/wiki/bin/view/HPC/Mar1915
• Running the ALMA pipeline in parallel is 45% faster for this new dataset.
• Julian fixed a TaQL expression problem in partition which speeds up the creation of MMSs for large MSs with many scans in about 40 times.
• James is currently testing EVLA MSs in the cluster using the MPI framework and Multi-MSs. The tests look promising.
• Sanjay has produced a preliminary report of his finding when testing tclean with the MPI interface.

Development

• George: Changes to frequency labeling when we change the reference location.

Pipeline

• CASA 4.3.1 / EVLA pipeline commissioning - received initial punch list: web log displays, plots, intent selection, flux calibration
• CASA 4.4. / pipeline testing - version 116 fixed calibration bugs, image sensitivity code integration
• pipeline development - web log multi-session / multi-ms handling, tclean based imaging tasks, pipeline results document

AOB

• How to set up report

Developer Reports

Monday Meeting

• Sanjay Bhatnagar
• Sandra Castro
  • Tested the ALMA pipeline with different number of sub-MSs, as reported in the HPC report.
  • Started to test with a new data set from Mark Lacy, but run into the bandpass seg fault. Re-starting again today.
  • Preparing the logistics for Urvashi's HPC lecture tomorrow.
• Lindsey Davis
  • updated hif_exportdata task to save data products manifest on disk
  • CASA 4.4 / pipeline integration testing, calibration and tclean issues resolved
  • discussed EVLA punchlist with Brian K., bandpass plotting understood
  • travel arrangements for ALMA ICT 2020 meeting
  • meetings, CASA, pipeline, HPC, SSA
• Joe Deep
  • Lecture at 9:00 HPC lecture, Auditorium
• Pam Ford
  • Upgrading Mac system cbt-d12-2 to OSX 10.10 with Darrell's help; errors compiling 'code' tree.
  • Investigated counting chunks for time-averaged data with plain VIVB2 and plotms averaging code to improve performance. Although speed was fast, number of averaged chunks differed from number returned by MSTransformVI. Have not determined why yet.
  • Started to test writing flags in plotms using MSTransformVI (CAS-7393). Worked for simple unaveraged case. Did not work correctly for channel-averaged data. Investigating whether problem is in plotms or MSTransformVI.
• Kumar Golap
  • Fixed a bug in mosaic that was due to out of bound array access
  • fixeed a bug where outlier with mosaic was not working
  • investigated a bug in mask from region file ...taking a huge time and memory to process 800 to 1000 regions...passed it on to Dave and gave the user a work around of using a list of 800 regions instead of the file
  • investigation a flux scaling factor in msuvbin.
• Justo González Villalba
  • Implemented back-propagation and writing of channelized flags in MSTransformIterator. FLAG_ROW is still pending, I need to discuss with Pam/George about FLAG_ROW/FLAG conventions.
  • As requested for plotms added an option to not-reindex selected data in MSTransformIterator
  • Modified gcwrap automatically generated code to use casalog filter in the parallel context
  • Fixed mpi4casa test which was failing in the multiple node case
• Jim Jacobs
  • Benchmarking averaging VI on various datasets, particularly ALMA one (15 GiB). Averaging VI is definitely somewhat slower. I'm also seeing some odd performance when MSTransform is used: the Feed subtable (fairly small) generates 2 GiB worth of reads); On other relatively small columns there is also a high number of reads. I think this MS is probably suboptimally filled. Each column is independent of the others, some are using the Standard StMan (which tends to handle caching badly by default) and others, like Time, use Incremental StMan which also seems to have caching problems (another huge amount of I/O).
• Jeff Kern
  • SKA PDR
• Susan Loveland
  • CARTA histogram, customizable toolbar, and fix of statistics & layout bugs
  • Viewer CAS-7404
• David Mehringer
  • Completed work on CAS-5053) ms.getscansummary() fails to list all spws of a scan
  • Commented on Darrell's request in CAS-5334) Ellipse regions have wrong width (RA)
  • Completed CAS-7399) remove hundreds of lines of plotting code from LatticeStatistics
  • Completed important cases for CAS-6882) imstat sums over multiple planes in an image cube to produce a "flux"
  • Began working on CAS-7405) rg.fromtexfile uses more than 64GB for 1000 circles
• George Moellenbrock
  • CAS-7402: Added new im.apparentsens() function that provides direct calculation of expected sensitivity from the MS weights (including effects of robust, uniform, taper, etc.). If the weights are properly initialized and calibrated (which should now be the norm for ALMA, and hopefully soon for the EVLA), this will yield a more useful result than the im.sensivity() function. Also discussions re porting this to the tclean framework.
  • CAS-7097,4983: Fixed nominal frequency labeling in gaincal-generated caltables. The recorded frequencies are now spw centers, and when combine='spw', the simple centroid of the spw center frequencies is used. This should improve calibration results for calibrators demanding more than single-spw bandwidth.
  • Fixed a serious problem introduced by SD developments in the calibration VisEquation, which causes a segv in the 4.4.114 stable.
  • CAS-4469: Began work on enhancing initweights task to support simple generation of WEIGHT_SPECTRUM from the existing (calibrated) WEIGHT column. Should be able to finish this on Monday.
  • Performance discussions with James. Apparently, there is a serious performance lapse in gaincal between 4.4.94 and 4.4.114. Unclear what this could be as there has been no direct work on the gaincal solve in this period. Also unclear if this is affecting gaincal generally (or is somehow specific to the large problem James is working with). Will need to investigate.
  • Lots of email w/ Justo and Pam re averaging performance, indexing conventions, and similar in support of finishing plotms for v4.4.
  • A bunch of JIRA gymnastics related to v4.4->v4.5, including priority input to Joe.
• Dirk Petry
  • fixed problem in wvrgcal and made ready for test (CAS-7200) wvrgcal seg.faults when working with MMS
  • some test work for (CAS-6769) Options for restoring beams for cubes
• Urvashi Rau
  • recap/discussion of what to do for cube parallelizationn in tclean (cas-6629), emails with GM about a 'correct' sensitivity calculator.
  • minor bug-fix for Dirk to test cas-6769 for tclean (an automatically chosen 'common' restoring beam option for cubes that can have bad psfs in edge channels)
  • slides for upcoming HPC meeting talk - to document and convey to the non-imaging HPC-folk how Imager currently uses the parallelization infrastructure.
• Darrell Schiebel
• Julian Taylor
  • fixed TaQL dataset size scaling problem in partition, suggest improving TaQL set selection to Ger van Diepen, implemented in by him in googlecode casacore, constant integer sets selects with value < 1MiB now are O(1) instead of O(N)
• Takahiro Tsutsumi
  • test_setjy casalog filter fix
  • makemask expand mode bug fix
  • tclean related : a bit of testing on tclean sumweight handling modification; started to look at tclean cube data/image partition scheme

Friday NAOJ Meeting

• Kanako Sugimoto
• Wataru Kawasaki
• Masaya Kuniyoshi
• Takeshi Nakazato
• Shinnosuke Kawakami