Casa Parallelization meeting minutes - Face2Face Meeting in Socorro

Monday [01 - 05 December 2014], [DSOC 248], [TIME]

Attendees:

• Socorro: Sandra, Justo, Julian, Jeff, James, Jim, Kumar, Lindsey, Tak, Jim
• Charlottesville

Discussion

1. Testing efforts

• Multi-MS testing
  • Running a cycle2 script with MMS: importasdm, partition, listobs, flagdata, wvrgcal, smoothcal, gencal, applycal, split2, setjy, flagmanager, gaincal, bandpass, delmod, fluxscale, plotms.
  • The main driver is the partition schema and how applycal selects the data. When most of the selected data is in one or a few sub-MSs, applycal takes many hours to finish because it is running in a single node.
• lazy filler and MMS testing
  • Run partition to create 10 subMSs, in the same way as the above test. This number of sub-MSs seems to be more optimal for applycal.
  • applycal took many hours for field=0.
  • there was still a write-lock from plotms (maybe my build is not up-to-date?)
  • killing plotms releases the lock. The script ended normally after this.
• Pipeline testing
  • Running the pipeline trunk against CASA trunk. Using uid___A002_X5784d5_X32d/ dataset, 25 spws and 20 scans.
    • The input MMS gets corrupted once setjy runs through it with NULL selections because it doesn't create the MODEL column for the non-selected sub-MSs. A quick fix is to add cb.open(addcorr=False,addmodel=True) to task-setjy right in the beginning so that the MODEL column is created with default values.
    • This task also raises exceptions in several places inside the task. These need to be changed so that the top code dealing with the parallelization can catch them. The pipeline at the moment also seems to be sensitive to exceptions thrown by the tasks.
    • Found a few bugs on the sub-tables consolidation for the MMS which made flagdata and gencal fail. When the first sub-MS contains only the WVR spw, some tables have empty rows (FEED, SYSCAL, SYPOWER, CALDEVICE). The merging functions need to handle this. FEED and SYSCAL were fixed in CAS-7167. The other two still need to be fixed.
    • Have not yet being able to finish the full pipeline script with MMS. The reasons are:
      • number of open files (work-around to increase soft and hard limits in the system)
      • many exceptions thrown by setjy, in the middle of the task code (work-around: changed the SEVERE to WARN)
      • seg-fault from plotms call, right after running setjy. Don't know yet the reason. The backtrack shows:

        /usr/lib64/casa/01/lib/libQtGui.so.4.8.5(QApplicationPrivate::construct(_XDisplay*, unsigned long, unsigned long) 0xe4) [0x2baa9889ec54]
        /usr/lib64/casa/01/lib/libQtGui.so.4.8.5(QApplication::QApplication(int&, char**, int) 0x73) [0x2baa9889fc33]
        /data1/testhpc1/casa/trunk/linux64/lib/libtools.so.472.117.82(casac::plotms::launchApp() 0x488) [0x2baa97aed1e8]
        /data1/testhpc1/casa/trunk/linux64/lib/libtools.so.472.117.82(casac::plotms::setShowGui(bool) 0x1e) [0x2baa97af0f9e]
               

    • How can I comment out the calls to plotms in the pipeline?
• lazy filler and MMS testing with the pipeline
  • How can I make modifications to the way the pipeline calls the CASA tasks? I need to add the lazy=True parameter to the importasdm call for this test.
• Imaging testing
  • Create a spectral version of the script used by Sanjay.
  • Port the parallel_go version of Sanjay's script to use the MPI-based framework
• 3C147 testing
• Documentation on how to test using the MPI parallelization framework, MMS and tclean
  • https://safe.nrao.edu/wiki/bin/view/HPC/TestMPICluster

2. Build, Test and MPI integration (meeting Tuesday 2/12 at 16:00)

• Integration of relevant 3rd party packages into casa repos.
• cmake script changes to respect necessary compiler flags.
• addition of alma m100 (or another script) regression to test suite.
• addition of an MMS test in the automatic tests (maybe at ESO if we can?)

3. Major Blockers

• High priority
  • https://bugs.nrao.edu/browse/CAS-7019: Integration of OpenMPI framework into regular package production
  • https://bugs.nrao.edu/browse/CAS-4860: Multi MS opens all the MS when accessed (file binding to reduce quantity of open files?)
  • https://bugs.nrao.edu/browse/CAS-7016: Option to open main table in write mode and sub-tables in read mode so they can be shared
• Medium
  • https://bugs.nrao.edu/browse/CAS-7018:Update openmpi / mpi4casa implementation to respect passed environmental variables
  • https://bugs.nrao.edu/browse/CAS-6705: Introduce MPI to communicate and synchronize logging directly at C++ level
• Low
  • Keyboard interaction is disturbed when using mpi on CASA. Command history through the arrows does not work. The CASA prompt is not aligned with the cursor which is in the next line.
  • Sometimes there is an error message about creating a link to casapy.log.
  • https://bugs.nrao.edu/browse/CAS-7122: NULL MS selection exceptions showed in MPIClient (stderr).
  • flagmanager needs to be parallelized

4. Plan for next 6-12 months

• 3c147 script also uses parallel_go. Justo and Sandra will work with James in order to port it to use MPI. At the long run, we should find a better dataset for this use-case.
• We should aim to have some casa-guides scripts running on MMSs in our tests. One or some of them could be good candidates to be included in automated tests. Sandra will look into this first. Justo will try later after he fixes some pending tickets in mstransform.
• We should look at the Sanjay's imaging script and see how to make an MPI version of it.
• Sandra and Justo will work with Stewart on the parallelization issues of the pipeline.
• Julian will work with the build team in CV on the MPI distribution in CASA.
• James will test the tileshape parameter in mstransform.
• Kumar will verify/test the fix to CAS-7016.
• Tak will look at the cube side of imaging in place of Urvashi.
• Sandra will use Justo's document to create a simple document guide on how to build CASA with the necessary libraries to run MPI in the cluster in Socorro. The official documentation should have no cluster specific information. For the moment we need a quick-guide to run in Socorro.
• Sandra will talk to Tak about running setjy on MMSs.
• Justo will compile a list of things to do in order to distribute MPI on CASA. For today’s 16:00 meeting.

5. Deferred Items

• Parallelization of plotms will come later.