ALMA pipeline script and "balanced" mode in partition

1. (6 Feb) Balanced mode with 4 subMSs, about 14Gb each. Restarted the whole script including calls to setjy. Using uid___A002_X5b1929_X57d.

• Attached is the listpartition of the MMS
• Logfile of the full run showing the error messages. Cannot allocate memory in one of the clean commands. There is an earlier SEVERE message in hif_makecleanlist() that says:

• SEVERE	pipeline.infrastructure.basetask::imager::advise() (file /data1/testhpc1/casa/trunk/code/synthesis/MeasurementEquations/Imager.cc, line 1064)	Maximum uv distance is zero
  WARN	hif_makecleanlist::pipeline.hif.heuristics.makecleanlist::casa@almahpc01:MPIClient	imager.advise failed for field/intent J1427-4206/BANDPASS spw [19] - no valid data?

2. Calls to setjy have been removed from script.

• Script stops in applycal and exit with the following half-trace. It also gives a 'std::bad_alloc' for lack of memory.

• [10] /data1/testhpc1/casa/trunk/linux64/lib/libalma.so.560.225.203(casa::Array<std::complex<float> >::resize(casa::IPosition const&, bool) 0x58) [0x2aaaabade1c8] [11] /data1/testhpc1/casa/trunk/linux64/lib/libdisplay.so.560.225.203(casa::Cube<std::complex<float> >::resize(casa::IPosition const&, bool) 0x58) [0x2aaaada53ed8] [12] /data1/testhpc1/casa/trunk/linux64/lib/libsynthesis.so.560.225.203(casa::ArrayColumn<std::complex<float> >::getColumnCells(casa::RefRows const&, casa::Array<std::complex<float> >&, bool) const 0xe8) [0x2aaab1ab35e8] [13] /data1/testhpc1/casa/trunk/linux64/lib/libmsvis.so.560.225.203(casa::VisibilityIteratorReadImpl::getDataColumn(casa::ROVisibilityIterator::DataColumn, casa::Cube<std::complex<float> >&) const 0x195) [0x2aaab2dcc345] [14] /data1/testhpc1/casa/trunk/linux64/lib/libmsvis.so.560.225.203(casa::VisibilityIteratorReadImpl::visibility(casa::Cube<std::complex<float> >&, casa::ROVisibilityIterator::DataColumn) const 0x40) [0x2aaab2dbf900] [15] /data1/testhpc1/casa/trunk/linux64/lib/libmsvis.so.560.225.203(casa::ROVisibilityIterator::visibility(casa::Cube<std::complex<float> >&, casa::ROVisibilityIterator::DataColumn) const 0x40) [0x2aaab2da5fe0] [16] /data1/testhpc1/casa/trunk/linux64/lib/libmsvis.so.560.225.203(casa::VisSetUtil::initScrCols(casa::VisibilityIterator&, bool, bool) 0x3b0) [0x2aaab2d9ddf0] [17] /data1/testhpc1/casa/trunk/linux64/lib/libmsvis.so.560.225.203(casa::VisSetUtil::initScrCols(casa::MeasurementSet&, bool, bool) 0x8b) [0x2aaab2d9e9fb] [18] /data1/testhpc1/casa/trunk/linux64/lib/libmsvis.so.560.225.203(casa::VisSetUtil::addScrCols(casa::MeasurementSet&, bool, bool, bool, bool) 0xa72) [0x2aaab2d9f672] [19] /data1/testhpc1/casa/trunk/linux64/lib/libsynthesis.so.560.225.203(casa::Calibrater::initialize(casa::MeasurementSet&, bool, bool, bool) 0x607) [0x2aaab1b51407] [20] /data1/testhpc1/casa/trunk/linux64/lib/libtools.so.560.225.203(casac::calibrater::open(std::string const&, bool, bool, bool) 0x24c) [0x2aaab8c1334c] [21] /data1/testhpc1/casa/trunk/linux64/python/2.7/__casac__/_calibrater.so [0x2aaabfcf6fd0]

3. Repeated the script only with importasdm (for context) and applycal, this time using simple_cluster. The engine logs show the NULL selections from subMS 0004. Seg faults without enough memory.

• 2015-01-28 08:17:46	INFO	applycal::Calibrater::selectvis	Performing selection on MeasurementSet
  2015-01-28 08:17:46	INFO	applycal::Calibrater::selectvis+	 Selecting on spw: '17,19,21,23'
  2015-01-28 08:17:46	INFO	applycal::Calibrater::selectvis+	 Selecting on field: 'Ceres,HE2-10,J053851-440507,J0826-225'
  2015-01-28 08:17:46	INFO	applycal::Calibrater::selectvis+	 Selecting on intent: '*PHASE*,*AMPLI*,*TARGET*,*BANDPASS*'
  2015-01-28 08:17:46	INFO	applycal::Calibrater::selectvis+	 Selecting on antenna/baseline: '*&*'
  2015-01-28 08:17:47	INFO	applycal::Calibrater::selectvis	By selection 367950 rows are reduced to 60000
  2015-01-28 08:17:48	INFO	applycal::Calibrater::selectvis	Frequency selection: 
  2015-01-28 08:17:48	INFO	applycal::Calibrater::selectvis	.  Spw 17:0~3839 (3840 channels, step by 1)
  2015-01-28 08:17:48	INFO	applycal::Calibrater::selectvis+	.  Spw 19:0~3839 (3840 channels, step by 1)
  2015-01-28 08:17:48	INFO	applycal::Calibrater::selectvis+	.  Spw 21:0~3839 (3840 channels, step by 1)
  2015-01-28 08:17:48	INFO	applycal::Calibrater::selectvis+	.  Spw 23:0~3839 (3840 channels, step by 1)
  2015-01-28 08:17:48	INFO	applycal::calibrater::correct	Beginning correct---------------------------
  2015-01-28 08:17:48	INFO	applycal::Calibrater::correct2 (VI2/VB2)	The following calibration terms are arranged for apply:
  2015-01-28 08:17:48	INFO	applycal::Calibrater::correct2 (VI2/VB2)	.   (None)
  2015-01-28 08:21:29	SEVERE	applycal::Calibrater::correct2 (VI2/VB2)	Caught exception: Nothing to Apply
  2015-01-28 08:21:29	INFO	applycal::Calibrater::correct2 (VI2/VB2)	Resetting all calibration application settings.
  2015-01-28 08:21:29	SEVERE	applycal::::	Exception Reported: Error in Calibrater::correct.
  2015-01-28 08:21:37	SEVERE	applycal::::casa	An error occurred running task applycal.
  2015-01-28 08:23:40	INFO	flagdata::::casa	
  terminate called after throwing an instance of 'std::bad_alloc'
    what():  St9bad_alloc     

4. Use a smaller MS. uid___A002_X5b1929_X57d.tgz cycle1. Run without setjy and using the balanced mode. Clean seg faults.

• 2015-01-30 06:53:52 INFO: Executing clean(noise='1.0Jy', phasecenter='ICRS 14:27:56.297 -042.06.19.45', scan=['4'], spw='21', vis=['uid___A002_X5b1929_X57d.ms'], imagename='/diskb/testhpc1/casa/work/pipeline/working/oussid.s13_0.J1427_4206_bp.spw21.I.iter0', intent='*BANDPASS*', threshold='0.0mJy', robust=0.0, imsize=[300, 300], stokes='I', nchan=-1, imagermode='csclean', weighting='natural', npixels=1, niter=0, cell=['0.252arcsec'], outframe='LSRK', mode='mfs', selectdata=True, usescratch=True, interactive=False)                                                                                                         
  [almahpc01:29281] *** Process received signal ***                                                                                                           
  [almahpc01:29281] Signal: Segmentation fault (11)                                                                                                           
  [almahpc01:29281] Signal code: Address not mapped (1)                                                                                                       
  [almahpc01:29281] Failing at address: 0x8   
  [almahpc01:29281] [ 0] /lib64/libpthread.so.0 [0x2b43ecdcdca0]                                                                                              
  [almahpc01:29281] [ 1] /data1/testhpc1/casa/trunk/linux64/lib/libcasa_tables.so.560.225.203(casa::TSMCube::makeCache() 0x40) [0x2b440e5b74c0]             
  [almahpc01:29281] [ 2] /data1/testhpc1/casa/trunk/linux64/lib/libcasa_tables.so.560.225.203(casa::TSMCube::setCacheSize(unsigned int, bool, bool) 0x85) [0x2b440e5b9065]       
  [almahpc01:29281] [ 3] /data1/testhpc1/casa/trunk/linux64/lib/libmsvis.so.560.225.203(casa::VisibilityIteratorReadImpl::setTileCache() 0x5be) [0x2b440aad28ae]                                                                                                                                                      
  [almahpc01:29281] [ 4] /data1/testhpc1/casa/trunk/linux64/lib/libmsvis.so.560.225.203(casa::VisibilityIteratorReadImpl::updateSlicer() 0x37c) [0x2b440aacaffc]                                                                                                                                                      
  [almahpc01:29281] [ 5] /data1/testhpc1/casa/trunk/linux64/lib/libmsvis.so.560.225.203(casa::VisibilityIteratorReadImpl::origin() 0xcc) [0x2b440aac3d7c]  
  [almahpc01:29281] [ 6] /data1/testhpc1/casa/trunk/linux64/lib/libmsvis.so.560.225.203(casa::VisibilityIteratorReadImpl::originChunks(bool) 0xb8) [0x2b440aac35b8]                                                                                                                                                       
  [almahpc01:29281] [ 7] /data1/testhpc1/casa/trunk/linux64/lib/libmsvis.so.560.225.203(casa::VisibilityIteratorReadImpl::selectChannel(casa::Block > const&, casa::Block > const&, casa::Block > const&, casa::Block > const&, casa::Block > const&) 0x53f) [0x2b440aaca61f]                                                                                                              
  [almahpc01:29281] [ 8] /data1/testhpc1/casa/trunk/linux64/lib/libmsvis.so.560.225.203(casa::VisibilityIteratorReadImpl::initialize(casa::Block const&) 0x435) [0x2b440aad76a5]
  [almahpc01:29281] [ 9] /data1/testhpc1/casa/trunk/linux64/lib/libmsvis.so.560.225.203(casa::VisibilityIteratorReadImpl::VisibilityIteratorReadImpl(casa::ROVisibilityIterator*, casa::Block const&, casa::Block const&, bool, double) 0x38f) [0x2b440aad81ef]
  [almahpc01:29281] [10] /data1/testhpc1/casa/trunk/linux64/lib/libmsvis.so.560.225.203(casa::ROVisibilityIterator::construct(casa::asyncio::PrefetchColumns const*, casa::Block const&, casa::Block const&, bool, double, bool, casa::ROVisibilityIterator::Factory const&) 0xd0) [0x2b440aaaf9f0]
  [almahpc01:29281] [11] /data1/testhpc1/casa/trunk/linux64/lib/libmsvis.so.560.225.203(casa::ROVisibilityIterator::ROVisibilityIterator(casa::asyncio::PrefetchColumns const*, casa::Block const&, casa::Block const&, bool, double, bool) 0x3a) [0x2b440aaafafa]
  [almahpc01:29281] [12] /data1/testhpc1/casa/trunk/linux64/lib/libmsvis.so.560.225.203(casa::VisibilityIterator::VisibilityIterator(casa::Block const&, casa::Block const&, double) 0x2c) [0x2b440aab00bc]
  [almahpc01:29281] [13] /data1/testhpc1/casa/trunk/linux64/lib/libsynthesis.so.560.225.203(casa::ImagerMultiMS::setDataOnThisMS(casa::MeasurementSet&, casa::String const&, casa::Vector const&, casa::Vector const&, casa::Vector const&, casa::Vector const&, casa::Vector const&, casa::String const&, casa::String const&, casa::String const&, casa::Vector const&, casa::String const&, casa::String const&, casa::String const&, casa::String const&, casa::String const&, casa::String const&) 0x266c) [0x2b4409b341ac]
  [almahpc01:29281] [14] /data1/testhpc1/casa/trunk/linux64/lib/libsynthesis.so.560.225.203(casa::ImagerMultiMS::setDataPerMS(casa::String const&, casa::String const&, casa::Vector const&, casa::Vector const&, casa::Vector const&, casa::Vector const&, casa::Vector const&, casa::String const&, casa::String const&, casa::String const&, casa::Vector const&, casa::String const&, casa::String const&, casa::String const&, casa::String const&, casa::String const&, casa::String const&, bool, bool) 0x27d) [0x2b4409b38cad]
  [almahpc01:29281] [15] /data1/testhpc1/casa/trunk/linux64/lib/libtools.so.560.225.203(casac::imager::selectvis(std::string const&, std::vector > const&, std::vector > const&, std::vector > const&, casac::variant const&, casac::variant const&, casac::variant const&, casac::variant const&, casac::variant const&, std::string const&, casac::variant const&, casac::variant const&, std::string const&, bool, bool, bool) 0xbd4) [0x2b44109ba424]
  

5. Using the smaller MS and with setjy back in the script. I get the same seg fault from setTileCache() as above, but this time during gaical

•  2015-02-05 15:15:45 INFO: Executing gaincal(solint='int', interp='linear,linear', spw='17', vis='uid___A002_X5b1929_X57d.ms', calmode='p', intent='*BANDPASS*', spwmap=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 9, 11, 11, 13, 13, 15, 15, 9, 9, 19, 20, 13, 13], refant='DA41,DV04,DV01,DV19,DV23,DV08,DV07,DV21,DV20,DV06,DV22,DA42,DA48,DV14,DA49,DA46,DA50,DA55,DA45,DA44,DA54,DV13,DV03,DA51,DV10,DV02,DV15,DA43,DV17,DV18,DV05,DV11', gainfield='nearest', minblperant=4, antenna='0~31', gaintype='G', caltable='uid___A002_X5b1929_X57d.ms.hifa_wvrgcalflag.s6_6.spw17.solintint.gpcal.tbl', minsnr=3.0, gaintable='uid___A002_X5b1929_X57d.ms.hifa_tsyscal.s4_1.tsyscal.tbl', field='J1427-4206', solnorm=False)                                                                                             
  The following MS spws have no corresponding cal spws in uid___A002_X5b1929_X57d.ms.hifa_tsyscal.s4_1.tsyscal.tbl: 0 1 2 3 4 5 6 7 8 19 20 23 24             
  2015-02-05 15:15:59     SEVERE  gaincal::Calibrater::solve      Caught exception: Table DataManager error: Invalid operation: TSM: no array in row 214880 of column MODEL_DATA in /diskb/testhpc1/casa/work/pipeline/working/uid___A002_X5b1929_X57d.ms/SUBMSS/uid___A002_X5b1929_X57d.ms.0001.ms/table.f26             
  2015-02-05 15:15:59     SEVERE  gaincal::::     Exception Reported: Error in Calibrater::solve.                                                             
  *** Error *** Error in Calibrater::solve.                                                                                                                   
  2015-02-05 15:15:59 WARNING: qa: wvrgcal QA calculation failed                                                                                              
  2015-02-05 15:16:00     WARN    gaincal::pipeline.hifa.tasks.wvrgcal.wvrgcal::casa@almahpc01:MPIClient  qa: wvrgcal QA calculation failed                   
  2015-02-05 15:15:59 INFO: qa: uid___A002_X5b1929_X57d.ms attempting to calculate wvrgcal QA using spw 21                                                    
  2015-02-05 15:15:59 INFO: Calculating new bandpass for QA analysis                                                                                          
  2015-02-05 15:16:02 INFO: Executing gaincal(solint='int', interp='linear,linear', spw='21', vis='uid___A002_X5b1929_X57d.ms', calmode='p', intent='*BANDPASS*', spwmap=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 9, 11, 11, 13, 13, 15, 15, 9, 9, 19, 20, 13, 13], refant='DA41,DV04,DV01,DV19,DV23,DV08,DV07,DV21,DV20,DV06,DV22,DA42,DA48,DV14,DA49,DA46,DA50,DA55,DA45,DA44,DA54,DV13,DV03,DA51,DV10,DV02,DV15,DA43,DV17,DV18,DV05,DV11', gainfield='nearest', minblperant=4, antenna='0~31', gaintype='G', caltable='uid___A002_X5b1929_X57d.ms.hifa_wvrgcalflag.s6_7.spw21.solintint.gpcal.tbl', minsnr=3.0, gaintable='uid___A002_X5b1929_X57d.ms.hifa_tsyscal.s4_1.tsyscal.tbl', field='J1427-4206', solnorm=False)