• Downloaded test data set uid___A002_X5784d5_X32d.tgz
• Checkout https://svn.cv.nrao.edu/svn/casa/branches/project/Pipeline-Cycle2-R1-B
• export CASAPATH=/diska/scastro/casa/trunk/linux64
• export PATH=/diska/scastro/casa/trunk/linux64/bin:$PATH
• ./runsetup
• Create link in trunk
  • ln -s /diska/scastro/casa/Pipeline-Cycle2-R1-B pipeline
• casapy --pipeline --nogui --log2term
  • tasklist shows all pipeline tasks
• h_init()
• hifa_importdata(vis=['../rawdata/uid___A002_X5784d5_X32d'])

2014-11-12 15:17:10 INFO hifa_importdata::::casa 
2014-11-12 15:17:10 INFO hifa_importdata::::casa+ ##########################################
2014-11-12 15:17:10 INFO hifa_importdata::::casa+ ##### Begin Task: hifa_importdata #####
2014-11-12 15:17:10 INFO hifa_importdata::::casa hifa_importdata(vis=['../rawdata/uid___A002_X5784d5_X32d'],session=[''],pipelinemode="automatic",asis="Antenna Station Receiver Source CalAtmosphere CalWVR",process_caldevice=False, 
2014-11-12 15:17:10 INFO hifa_importdata::::casa+ overwrite=False,bdfflags=False,dryrun=False,acceptresults=True) 
2014-11-12 15:17:09 INFO: Equivalent CASA call: hif_importdata(process_caldevice=False, vis=['uid___A002_X5784d5_X32d'], bdfflags=False, asis='Antenna Station Receiver Source CalAtmosphere CalWVR', overwrite=False) 
2014-11-12 15:17:10 INFO hifa_importdata::pipeline.infrastructure.basetask::casa Equivalent CASA call: hif_importdata(process_caldevice=False, vis=['uid___A002_X5784d5_X32d'], bdfflags=False, asis='Antenna Station Receiver Source CalAtmosphere CalWVR', overwrite=False) 
2014-11-12 15:17:09 INFO: Starting execution for stage 1 
2014-11-12 15:17:10 INFO hifa_importdata::pipeline.infrastructure.basetask::casa Starting execution for stage 1 
[esocasa01:10309] *** Process received signal *** 
[esocasa01:10309] Signal: Segmentation fault (11) 
[esocasa01:10309] Signal code: Address not mapped (1) 
[esocasa01:10309] Failing at address: 0x44 
[esocasa01:10309] [ 0] /lib64/libpthread.so.0[0x3720a0f710] 
[esocasa01:10309] [ 1] /lib64/libuuid.so.1[0x3725e01852] 
[esocasa01:10309] [ 2] /lib64/libuuid.so.1[0x3725e0197a] 
[esocasa01:10309] [ 3] /lib64/libuuid.so.1(__uuid_generate_random+0x50)[0x3725e01b40] 
[esocasa01:10309] [ 4] /lib64/libuuid.so.1(uuid_generate_random+0x16)[0x3725e01bc6]

• h_init()
• hifa_importdata(vis=['../rawdata/uid___A002_X5784d5_X32d'])
• mv uid___A002_X5784d5_X32d.ms uid___A002_X5784d5_X32d-monolithic.ms
• partition('uid___A002_X5784d5_X32d-monolithic.ms',outputvis='uid___A002_X5784d5_X32d.ms',flagbackup=False,separationaxis='scan',numsubms=5)
• exit CASA
• execfile('run_pipeline.py')

2014-11-13 13:49:28     SEVERE  gaincal::::     Exception Reported: All tables in ConCatTable must have same description

• Catching the NULL MS selections will not solve the problem. The input MS/MMS is corrupted once setjy runs through it with NULL selections. It seems that setjy needs to work like applycal and create at open time, a default MODEL column in the MS. If the processing fails later, at least the input MS will be in a consistent state. In applycal this is done with the cb.open() constructor. The corrupted MMS is fixed by adding a cb.open(addcorr=False,addmodel=True). TBD is to remove the several exception raising inside the task. Only the first try/except block should raise the exceptions.
• In order to test the above on the pipeline, I need to be able to edit the setjy source file in CASA, re-build and re-run the pipeline. I haven't found a way to do this.

• Task setjy needs to create the MODEL column at open time to avoid corrupting the MMS in the case of NULL MS selections on some sub-MSs.
• The pipeline seems to stop when exceptions are thrown in the tasks. Task setjy raises exceptions in several parts of the code, which are never returned to the parallel code on the top. This needs to be changed so that the pipeline doesn't fail in these cases.
• The choice of a partition axis is very important and also the number of sub-MSs in the MMS. Applycal, when selecting a field, may sometimes have the selected data in one sub-MS only, so only one node will be working hard while the others are idle.
• The main use-case for the way the MS is partitionted is how applycal selects the data. (include a value 'max' for numsubms in partition ?!!)
• The current 'auto' axis in partition works like this:

         * Find the minimum between number of spws and number of submss 
                                 numSpwPartitions = min(len(spws), numsubms) 
         * Find the integer of the ceiling between the number of submss and the above result 
                                 numScanPartitions = int(math.ceil(numsubms/float(numSpwPartitions))) 
         * if numScanPartitions is greater than 1, separate per scans too. Otherwise, separate only per spw.
     

• This experiment sets numsubms to 500 so that we get the maximum number of sub-MSs, having one scan and one spw in each sub-MS. This is the maximum granularity we can have. Partition creates an MMS with 170 sub-MSs for this dataset. The pipeline fails with the well known error.

2014-11-21 08:19:30     SEVERE  pipeline.domain.state::getcol::EXPOSURE Exception Reported: BucketFile: open error on file /diskb/testhpc1/casa/work/pipeline/working/uid___A002_X5784d5_X32d.ms/SUBMSS/uid___A002_X5784d5_X32d.ms.0150.ms/table.f2: Too many open files
2014-11-21 08:19:30     SEVERE  pipeline.domain.state::utils::setconstraints    Setting constraints ...Unable to read XML file file:///data1/testhpc1/casa/pipeline/pipeline/h/cli/h_save.xml, unable to verify input
2014-11-21 08:19:30     SEVERE  pipeline.domain.state::::       Unable to read XML file file:///data1/testhpc1/casa/pipeline/pipeline/h/cli/h_save.xml, unable to verify input

• The resolution of CAS-4860 is critical for the continuation of work with Multi-MSs.
• When partitioning the MMS in this way, several sub-MSs are created with the spw=0 selection, which for ALMA data contains the WVR data. This sub-MS doesn't have content in the FEED table, so there is a bug in the merging of the FEED sub-tables. CAS-7167
• Flagdata also fails when running on these sub-MSs with the following message:

  2014-11-24 12:48:35     SEVERE  flagdata::::casa@almahpc01:MPIServer-6  void minMax(T &min, T &max, const Array<T> &array) - Array has no elements          
  2014-11-24 12:42:12     SEVERE  agentflagger:: (file /data1/testhpc1/casa/trunk/gcwrap/tools/flagging/agentflagger_cmpt.cc, line 37)    Exception Reported: void minMax(T &min, T &max, const Array<T> &array) - Array has no elements    

• What can be done to avoid the WVR spw in an MMS???
• Fixed the FEED and SYSCAL sub-tables consolidation. Still missing SYSPOWER and CALDEVICE.
• Last run of the pipeline in hpc01/hpc02 is in LOG_last_pipeline_run.log. See here the call to plotms that seg faults. Check the interface on trunk.