Running the ALMA pipeline in parallel is 45% faster for this new dataset.
Julian fixed a TaQL expression problem in partition which speeds up the creation of MMSs for large MSs with many scans in about 40 times.
James is currently testing EVLA MSs in the cluster using the MPI framework and Multi-MSs. The tests look promising.
Sanjay has produced a preliminary report of his finding when testing tclean with the MPI interface.
George: Changes to frequency labeling when we change the reference location.
CASA 4.3.1 / EVLA pipeline commissioning - received initial punch list: web log displays, plots, intent selection, flux calibration
CASA 4.4. / pipeline testing - version 116 fixed calibration bugs, image sensitivity code integration
pipeline development - web log multi-session / multi-ms handling, tclean based imaging tasks, pipeline results document
How to set up report
Tested the ALMA pipeline with different number of sub-MSs, as reported in the HPC report.
Started to test with a new data set from Mark Lacy, but run into the bandpass seg fault. Re-starting again today.
Preparing the logistics for Urvashi's HPC lecture tomorrow.
updated hif_exportdata task to save data products manifest on disk
CASA 4.4 / pipeline integration testing, calibration and tclean issues resolved
discussed EVLA punchlist with Brian K., bandpass plotting understood
travel arrangements for ALMA ICT 2020 meeting
meetings, CASA, pipeline, HPC, SSA
Lecture at 9:00 HPC lecture, Auditorium
Upgrading Mac system cbt-d12-2 to OSX 10.10 with Darrell's help; errors compiling 'code' tree.
Investigated counting chunks for time-averaged data with plain VIVB2 and plotms averaging code to improve performance. Although speed was fast, number of averaged chunks differed from number returned by MSTransformVI. Have not determined why yet.
Started to test writing flags in plotms using MSTransformVI (CAS-7393). Worked for simple unaveraged case. Did not work correctly for channel-averaged data. Investigating whether problem is in plotms or MSTransformVI.
Fixed a bug in mosaic that was due to out of bound array access
fixeed a bug where outlier with mosaic was not working
investigated a bug in mask from region file ...taking a huge time and memory to process 800 to 1000 regions...passed it on to Dave and gave the user a work around of using a list of 800 regions instead of the file
investigation a flux scaling factor in msuvbin.
Justo González Villalba
Implemented back-propagation and writing of channelized flags in MSTransformIterator. FLAG_ROW is still pending, I need to discuss with Pam/George about FLAG_ROW/FLAG conventions.
Modified gcwrap automatically generated code to use casalog filter in the parallel context
Fixed mpi4casa test which was failing in the multiple node case
Benchmarking averaging VI on various datasets, particularly ALMA one (15 GiB). Averaging VI is definitely somewhat slower. I'm also seeing some odd performance when MSTransform is used: the Feed subtable (fairly small) generates 2 GiB worth of reads); On other relatively small columns there is also a high number of reads. I think this MS is probably suboptimally filled. Each column is independent of the others, some are using the Standard StMan (which tends to handle caching badly by default) and others, like Time, use Incremental StMan which also seems to have caching problems (another huge amount of I/O).
CARTA histogram, customizable toolbar, and fix of statistics & layout bugs
Completed work on CAS-5053) ms.getscansummary() fails to list all spws of a scan
Commented on Darrell's request in CAS-5334) Ellipse regions have wrong width (RA)
Completed important cases for CAS-6882) imstat sums over multiple planes in an image cube to produce a "flux"
Began working on CAS-7405) rg.fromtexfile uses more than 64GB for 1000 circles
CAS-7402: Added new im.apparentsens() function that provides direct calculation of expected sensitivity from the MS weights (including effects of robust, uniform, taper, etc.). If the weights are properly initialized and calibrated (which should now be the norm for ALMA, and hopefully soon for the EVLA), this will yield a more useful result than the im.sensivity() function. Also discussions re porting this to the tclean framework.
CAS-7097,4983: Fixed nominal frequency labeling in gaincal-generated caltables. The recorded frequencies are now spw centers, and when combine='spw', the simple centroid of the spw center frequencies is used. This should improve calibration results for calibrators demanding more than single-spw bandwidth.
Fixed a serious problem introduced by SD developments in the calibration VisEquation, which causes a segv in the 4.4.114 stable.
CAS-4469: Began work on enhancing initweights task to support simple generation of WEIGHT_SPECTRUM from the existing (calibrated) WEIGHT column. Should be able to finish this on Monday.
Performance discussions with James. Apparently, there is a serious performance lapse in gaincal between 4.4.94 and 4.4.114. Unclear what this could be as there has been no direct work on the gaincal solve in this period. Also unclear if this is affecting gaincal generally (or is somehow specific to the large problem James is working with). Will need to investigate.
Lots of email w/ Justo and Pam re averaging performance, indexing conventions, and similar in support of finishing plotms for v4.4.
A bunch of JIRA gymnastics related to v4.4->v4.5, including priority input to Joe.
fixed problem in wvrgcal and made ready for test (CAS-7200) wvrgcal seg.faults when working with MMS
some test work for (CAS-6769) Options for restoring beams for cubes
recap/discussion of what to do for cube parallelizationn in tclean (cas-6629), emails with GM about a 'correct' sensitivity calculator.
minor bug-fix for Dirk to test cas-6769 for tclean (an automatically chosen 'common' restoring beam option for cubes that can have bad psfs in edge channels)
slides for upcoming HPC meeting talk - to document and convey to the non-imaging HPC-folk how Imager currently uses the parallelization infrastructure.
fixed TaQL dataset size scaling problem in partition, suggest improving TaQL set selection to Ger van Diepen, implemented in by him in googlecode casacore, constant integer sets selects with value < 1MiB now are O(1) instead of O(N)
test_setjy casalog filter fix
makemask expand mode bug fix
tclean related : a bit of testing on tclean sumweight handling modification; started to look at tclean cube data/image partition scheme