1. ALMA MS A001_X122_X291 with 3 EBs

uid___A002_X856bb8_X169d, uid___A002_X86fcfa_X66b, uid___A002_X8719c8_X27f

h_init()
try:
    hifa_importdata(vis=['../../rawdata/uid___A002_X856bb8_X169d', '../../rawdata/uid___A002_X86fcfa_X66b', '../../rawdata/uid___A002_X8719c8_X27f'], session=['session_1', 'session_3', 'session_4'])
    fixsyscaltimes(vis = 'uid___A002_X856bb8_X169d.ms')
    fixsyscaltimes(vis = 'uid___A002_X86fcfa_X66b.ms')
    fixsyscaltimes(vis = 'uid___A002_X8719c8_X27f.ms')
    hifa_flagdata(pipelinemode="automatic")
    hifa_fluxcalflag(pipelinemode="automatic")
    hif_refant(pipelinemode="automatic")
    hifa_tsyscal(pipelinemode="automatic")
    hifa_tsysflag(pipelinemode="automatic")
    hifa_wvrgcalflag(pipelinemode="automatic")
    hif_lowgainflag(pipelinemode="automatic")
    hif_setjy(pipelinemode="automatic")
    hif_bandpass(pipelinemode="automatic")
    hif_bpflagchans(pipelinemode="automatic")
    hifa_gfluxscale(pipelinemode="automatic")
    hifa_timegaincal(pipelinemode="automatic")
    hif_applycal(pipelinemode="automatic")
    hif_makecleanlist(intent='PHASE,BANDPASS,CHECK')
    hif_cleanlist(pipelinemode="automatic")
finally:
    h_save()

1. Sequential run in lustre:
   • Total time: 20 hours
   • 6 calls to applycal: 6min 4min 8min 5min 20min 3.5min
2. Parallel run in lustre, 1 no6min 4min 8min 5min 20min 3.5minde, 10 sub-MSs, casa-test-4.4.123, pipeline rev 32843. There were errors in clean.
   
   • Total time: 14.5 hours; partition average time: 4.5 min; importasdm average time: 9 min
   • 6 calls to applycal: 2.5min 1min 2min 2min 3min 2.5min

     2015-03-31 01:27:56     SEVERE  clean::pipeline.hif.tasks.clean.cleanbase::@nmpost028:MPIClient J1147-3812/BANDPASS/spw23 clean error: [Errno 11] Resource temporarily unavailable                                                                                                                          
     2015-03-31 01:27:56     WARN    pipeline.hif.tasks.clean.cleanbase::ImageAnalysis::open File [/lustre/scastro/testpipe/working/oussid.s15_0.J1147_3812_bp.spw23.I.iter0.model] does not exist. 

3. Parallel run in lustre, 3 nodes, 42 sub-MSs, casa-test-4.4.123, pipeline rev 32843
   • Write-lock from plotms stopped the flagdata processing. Created CAS-7428. After killing plotms the execution continued but failed with the "too many files open" error during hif_makecleanlist.

     2015-04-01 07:35:42     SEVERE  pipeline.infrastructure.basetask::imager::advise() (file /var/rpmbuild/BUILD/casa-test/casa-test-4.4.123/code/synthesis/MeasurementEquations/Imager.cc, line 1064)    Caught exception: FiledesIO: file /lustre/scastro/testpipe/working/uid___A002_X8719c8_X27f.ms/SUBMSS/uid___A002_X8719c8_X27f.ms.0040.ms/table.f21_TSM0 could not be opened: Too many open files

   • Log of the run can be found in /lustre/scastro/testpipe/logs/X291_lustre_42subms_casapy-20150331-081250.log
4. Re-run the above test using the MultiFile implementation from Ger. For that I added the following lines to my .casarc file in the cluster.

   table.storage.option: multifile
   table.storage.blocksize: 262144

   • The script running on 3 EBs run to completion using 3 nodes and 42 cores. It created 3 MMSs with 42 subMSs each. Total execution time was 11h 48 min.