Monday Morning Meeting August 18th, 2014

  • DIAL-IN NUMBERS & PASSCODES:
  • USA Toll Free Number: 866-901-8266
  • USA Toll Number: +1-203-566-3863
  • PARTICIPANT PASSCODE: 725760

Attendance

  • Socorro: Jeff, Rob, Susan, Sanjay, Urvashi, Tak, Lindsey, Juergen, Kumar, George, Jim
  • CV: Dave, Darrell, Andy,
  • Garching: Justo, Sandra, Dirk
  • SCO:

News / Meetings / Visitors

  • CUC
  • Pam Ford Arriving Thur.

Build, Release, Testing

  • 4.2.2 Status
    • RHEL
    • OSX - On the last issue
    • Blockers: Plotms interactive bug CAS-6813

  • Release distribution / availability and configuration control for packages.
    • casapy -r switch vs installations in /home/casa/packages/...
    • Juergen to update documentation

  • Specfit regression test failure * Intermittent Failure

  • Should we have a uniform set of failure conditions for our regressions.

Architecture

  • Common CASA Statistics Framework Discussion - Tuesday 8/19 @ 8AM in DSOC 317.
    • Rob will send out reminder and agenda on Monday.
    • Tasks: calstat, imstat, sdstat, statwt, visstat + Imager & Pipeline.
    • Implement in Scimath (casacore)? Or equivalent under code?
      • To replace imagestatistics would require implementation in casacore.

  • Technical Forums:
    • Next scheduled is VI/VB on 9/4/14.
    • Casacore merge and management - Jeff & Jim - 9/18/14?

Development

Pipeline

AOB

Developer Reports

Monday Meeting
  • Sanjay Bhatnagar
    • Parallel and Non-Parallel are now giving same answer.
  • Sandra Castro
    • CAS-6746: "different data flagged based on order of flagdata commands." I have noticed differences between flagging using the interface and in list mode. The differences seem to be because the MSS function getTimeList() doesn't take into account the exposure time. I've talked to Sanjay about this, who will work on a solution.
    • CAS-6825: "mstransform fails with LOFAR data trying to make 120 SPWs into 1 with 120 channels". I found a bug, inherited from SubMS, which fails when the optional SOURCE table is not present (like in LOFAR data). Fixed on trunk if r30770.
    • CAS-6770: "visstat segfault." Liz reported another MS that fails in the pipeline, but not with a seg fault. I believe the problem is because the selected spws have different number of channels. The initial accumulated data array is created with a smaller number of channels and when a data chunk with more channels is appended it causes the failure. Working on a fix.
    • Fixed test_split2 that were failing in the latest casapy-test package, due to changes in the WEIGHTS.
    • Fixed test_flagcmd which had files left open due to calls to tb.query().
    • Helped Michel to debug importasdm in RHEL 6, which has been failing for awhile. The problem seemed to be in SysCalTable.cc due to libxml2.
    • CAS-6791: "parser: created specific parser with parameter checking for flagdata." Almost finished with this implementation.
    • Created CAS-6835: "wrapper of cvel to use the MSTransform framework." Will make an estimate if it is possible to have this ready for 4.3.
  • Lindsey Davis
    • worked on pipeline / CASA tasks memory usage issues
    • reviewed low calibrator SNR heuristics proposal draft
    • implemented Python executeppr module for EVLA pipeline
    • regression testing for next pipeline heuristics patch
    • released pipeline heuristics release patch 30807 for CASA 42.2 12-1
    • review of outstanding pipeline jira tickets
  • Kumar Golap
    • Fixed msuvbin, clean up some code and did some testing
    • wrote the partition for cube imaging with coordsys and npartition.
  • Justo Gonz├ílez Villalba
    • On Holiday
  • Jim Jacobs
    • On Holiday
  • Jeff Kern
    • In Chile: ALMA Pipeline Deployment
  • Susan Loveland
    • On Holiday
  • David Mehringer
    • Responded to tester comment on CAS-6726 imsubimage: the overwrite parameter does not appear in the inputs list. This parameter indeed does appear if outfile is set to something other than the empty string. Tester later verified that to indeed be the case.
    • Ger pointed out that there is code in ImageSummary::list() that is not being used and asked if I was ok if he removed it. I'm all about removing dead code, so said yes and pointed out I didn't write this code because it is written in a style I do not use nor approve of. He later verified that it was written by someone else long before I (re)joined the project.
    • Created CAS-6830 migrate std::tr1::shared_ptr in MSMetaData to CountedPtr at Ger's request. Although he claims to be able to build google code with a simple search and replace of CountedPtr for std::tr1::shared_ptr, I cannot build our casacore at least because our CountedPtr does not have a reset() method. I've punted this to Jim and suggested to Ger that he contact Jim, not me, for any future issues that involve the future of casacore as I am out of that loop.
    • Set CAS-6354: Rewrite ImageAnalysis::getFreqProfile() to RTT. I can work with our test team re: various functions Martin has implemented since they have relevant images and know what the correct results should be.
    • Added upper limit checking, re: CAS-6459 msmd needs a method to get spw names
    • Requested CSSC approval for requestor's proposed functionality changes re: CAS-5996 immath unit- (and if relevant beam-) awareness
    • Resolved CAS-6838 New frequently occuring warning emitted from ia tool, by adding verbose parameter to ia.calc()
    • Determined and recorded some requirements for unified stats framework.
    • Investigated CAS-5689 spectral fitting results seem off, was unable to reproduce bug where no gaussian was fit when one should have been. Passed back to Susan to investigate spectral fitter GUI-related issues.
    • Fixed Vector initialization defect uncovered during testing of CAS-5238 new feature: ia.imageconcat() to sort images automatically
    • Modified Image2DConvolver::convolve() to issue a warning when the input beam or convolving kernel is poorly sampled, CAS-6848.
    • Changed default behavior of ImageStatsCalculator so that no log messages are emitted, with the exception of messages from ImageRegion which need to be addressed via a casacore API change. I'm waiting for info/guidance on what's happening with casacore before I touch that
      code again.
  • George Moellenbrock
    • Some discussion with Rob re casapy version comprehensibility and consistency in the AOC....
    • CAS-6800: gaincal failure due to flagging: turns out to be caused by the FLAG_ROW IncrStMan bug (CAS-6746)
    • CAS-6746: Ongoing discussions; invented/tested counter-measure for pre-4.2.2 users (i.e., ALMA data reduction): Manually set FLAG_ROW=F globally after flagdata runs (and anything else that might have set FLAG_ROW incorrectly)
    • CAS-6846: gaincal memory leak? WVR caltables are larger than most other tables and have a large memory footprint. But there is not an uncontrolled memory black hole on this scale (subsequent gaincal runs using the same WVR caltable seem to reuse the memory). Further investigation shows that assigned memory (i.e., that actually being used by c++ objects) goes up and down appropriately when caltables are loaded and deleted, but that memory allocated by the OS to the process can stay inflated at about the level reported. In no case does the allocated memory exceed the peak (apparently) required assigned memory, and occasionally allocated memory is released. Resolved/closed this ticket, but created 2 new ones (CAS-6849, CAS-6850) concerned with optimizing the size of caltables, in general, which were revealed to be possibly excessive in the course of this investigation.
    • Discussions re 'pseudo I' mode in clean. From my point of view, the most interesting outcome of this discussion was the realization that use of the formal definitions for formation of Stokes parameters from correlations necessarily means that we do not realize the full apparent aggregate sensitivity of both correlations (~sqrt(2)) unless their weights are strictly equal (or if we opt into a 'pseudo I'-style mode). In fact, we are currently glossing this a bit by asserting equal weight for correlation pairs, even though this is generally not true. In cases where there is a large weight offset between the two correlations (e.g., baselines to an antenna with one bad--but not hopeless--polarization), we may be triggering noticeably disproportionate sensitivity effects.
    • Discussed interim freq-resolved statwt a bit with Emmanuel. He's going to provide a small-ish test dataset; I'm going to write some python to model the basic process.
    • WEIGHT_SPECTRUM and calibrater migration to VI2/VB2: ongoing....
  • Dirk Petry
    • Script to transform MS for spectral reference frame.
  • Urvashi Rao
    • More vi2/vb2 conversion work - still modifying independent copies of 4 classes to get them to use vi2/vb2. Compile-time switch currently forces old vi/vb to be used by default.
    • Tests of tclean with awproject+mtmfs and continuum parallelization : simulated widebandmosaic now gives same output image with the old clean and new clean with and without parallelization. Started testing on real VLA wideband mosaic data.
    • Updated and re-prepared RHEL6 build for us, Frazer, and Emmanuel to use for immediate testing.
    • Other minor tests : (a) Numerical feasibility of Gaussian convolution when the convolving function is undersampled (response to DM's email) (b) Invalidated a bug report from Steve's student regarding instability when deconvolving a snapshot VLA image (using natural weighting, clark clean, and possible overflagging by hand).
    • Some (circular?) discussions related to CAS-2961 (handling stokes imaging with correlation dependent flags)
  • Darrell Schiebel
  • Rob Selina
    • HPC meeting and associated tasks (https://safe.nrao.edu/wiki/bin/view/HPC/Aug1414).
    • Cleaning up HPC/Parallelization wiki page and adding content. Investigating MPI libraries and implementations.
    • Deployed latest 4.2.2 pre-release package in the AOC. Spent a little time trying to understand how version control works on pre-release and test package deployment.
    • NAOJ weekly meeting, Group Leads meeting, etc.
  • Takahiro Tsutsumi
    • Continued working on mask handler implementation
    • Switched to use mstransform's combineSpwCore( ) in buildCoordinateSystem() to resolve the error reported in CAS-6697
    • Clean-up 'setpol' code a bit before marked the corresponding ticket, CAS-4623 as ready to test
  • Ger van Diepen
    • Merged all NRAO changes till 7-Aug-14 into googlecode and removed warnings generated by newer compiler versions.
    • Built CASA (except qwt-related stuff) against googlecode's casacore; required a few minor changes. There are many warnings in the CASA code, some not so innocent. Most changes I made could be committed, because they'll work fine with CASA's casacore. Because the LOFAR imager uses CASA's synthesis code, we need to be able to build CASA.
    • It is the intention that CASA will use googlecode's casacore (rather sooner than later), probably as an external library which is already supported by CASA's cmake system. In this way we will benefit mutually. Jim can explain.
    • I wrote a little test program reproducing the problem Justo and Sandra found in the IncrementalStMan. Next thing is to find the cause.
    • For SKA testing purposes people in Perth are writing storage managers for ADIOS and HDF5 which can potentially support parallel IO.
    • Holiday from 10-Sep till 3-Oct.
Friday NAOJ Meeting
  • Kanako Sugimoto
  • Wataru Kawasaki
  • Takeshi Nakazato
  • Shinnosuke Kawakami

-- RobSelina - 2014-08-13
Topic revision: r17 - 2014-08-18, JeffKern
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback