Lengthy debugging of supposedly flagdata problem in MS with inconsistent FLAG_ROW=True and FLAG_cube=True. Justo and I have both tested different parts of flagdata and VI/VB2 and it seems that the problem is of data corruption. In two reported problems, the problem can be resolved if the spw selection is removed from the call to flagdata(vis,spw='1~22',autocorr=True) in the a priori flagging steps. The spw selection is redundant in this case because autocorr=True will automatically exclude any non-CORRELATION data from the flagging. On the other hand, in one report, the problem is also solved if mstransform is used instead of split, right before the last batch of flag calls. See Justo's report for the latest findings.
Another lengthy debugging of a seg. fault in visstat. After struggling to copy the MS to ESO (running out of disk space), the problem seems to be lack of resources. The task accumulates the whole data cube in order to calculate the median and runs out of memory when resizing a Vector. There is no quick solution for this problem and in my opinion, visstat needs to be re-factored to use the VI/VB2 (to make use of the rowBlocking). There is also the need of full requirements from the pipeline. Do they need the median of the entire data column to be calculated?
Follow up on pipeline release bugs and issues
Fixes to pipeline task CASA interface, BDF flags handling, output image names
Pipeline patch released
Infrastructure, OODT discussions
Fixed virtual model vis to handle the case when SOURCE_ID=-1 in the FIELD table
worked to make a brnch for vi2/vb2 conversion of imaging code
consensus that we should do it for refactored imager only as amount of work to convert existing imager is huge ; thus branching is unnecessary
Verified that the problem is not related with FlagData heuristics, MSSelection issues, or slicer usage in VI/VB.
Detected some unexpected sorting order in the MSs affected (data is not time-sorted within sub-scans:
Reported to Michel Calliac the unexpected sorting order produced by asdm2MS (in other ALMA regressions which are not affected by this bug we see the usual time sorting order within sub-scans).
Apart from potential relation with data corruption problem, the non-time sorted pattern that asdm2MS is producing for the affected MS typically leads to non-contiguous data access and has a big impact of data processing speed.
Found evidence data corruption in the SORTED_TABLE of the affected MSs:
SORTED_TABLE does not have the some number of rows as the original in the affected MSs
Currently in the process of finding out why SORTED_TABLE gets corrupted
Fixed performance issue in VI/VB2
OODT Deployment Design
Pipeline Imaging Meeting
Governance Document / Meet with PMD
Worked on viewer and plotms bugs.
Worked on CARTA
Fixed ia.fitprofile()/specfit issue in which fit results were not reproducable from one run to another in some cases, CAS-6742.
Continued work on image statistic filters design, consulted with Rob re: requirements.
Implemented convolve-using-image in imsmooth, CAS-5844.
Implemented msmd.pointingdirection(), CAS-5878.
Completed implementation for imhead support for NCP projection, CAS-6568.
Provided input for the requirements of the new build and test system, CAS-6788.
Fixed imregrid bug in which image rotation was being performed incorrectly in the opposite direction when converting between direction reference frames and the longitudinal pixel increment was positive, CAS-6533.
FITS File Processing
VI/VB2 Porting in Imaging
new 4.2.2 release candidate for linux
checked out osx application structure
wrote script to collect dependencies from binaries and python modules (for creating OSX app)
corrected one errant dependency (but still others remain)
Trying to close on open items related to robust noise estimating algorithms in imstat.
4.3 status chart update for the CSSC (will be used to give them a status before they comment on 4.4 priorities).
More existing H/W infrastructure research, with a focus on OSX development machines. Will have a catalog of all our shared hardware on the web soon.
Switched the calls in BuildCoordinateSystem () from SubMS::calcChanFreq (and combineSpws) to the corresponding methods in MSTransform
Mask handling in the refactored Imager. Moving some of the existing C++ and Python code related mask generation to the refactored Imager code
Added some descriptions of the 'setpol' parameters in in-line help for setjy
Looked at a new setjy selection issue reported on CAS-5941