-- SandraCastro - 2015-03-04

Casa HPC/Parallelization meeting agenda/minutes

---

Thursday [05/03/2015], [ESO Centaurus, C.2.01], [9:00 AM MST]

---

How to connect

• Dial in: +49 89 6834
• Video connection: Use this one (46104@134.171.42.27). Other option is 46104@eso.org

Attendees:

ESO: Justo, Julian, Sandra

Socorro: Lindsey, Tak, Sanjay, Joe

CV: nobody due to bad weather.

Agenda

1. Action Items from previous meeting
   • Mark to talk to Darrell about moving forward with mpicasa.
   • Julian to send email to Jim about increasing the cache.
   • Sandra to check if SEVERE message of makemask also happen in the sequential run of the pipeline.
     • The same errors happen in the sequential run of the pipeline. The error messages are:

WARN   pipeline.infrastructure.basetask::imager::advise():: (file /data/scastro/casa/trunk/code/synthesis/MeasurementEquations/Imager.cc, line 1064)   Visibility data are not yet weighted: using unweighted values
SEVERE   pipeline.infrastructure.basetask::imager::advise():: (file /data/scastro/casa/trunk/code/synthesis/MeasurementEquations/Imager.cc, line 1064)   Maximum uv distance is zero
WARN   hif_makecleanlist::pipeline.hif.heuristics.makecleanlist::casa::   imager.advise failed for field/intent J1427-4206/BANDPASS spw [19] - no valid data?

1. Running the trunk of CASA with mpirun in the cluster. What else do we need in order to have this running smoothly without any patches, tricks, etc.?
2. Testing
   • Scalability tests (James)
   • Imaging (Sanjay)
   • Plotms issues on the pipeline run (Justo and Julian)
   • Pipeline run with lazy=True and MMS (Sandra)
3. AOB (Time of meeting next week)

Minutes

1. Action Items discussion
   • Julian reported that the binaries have been built in a clean environment, which allowed him to send to Darrell the layout expected by openMPI for the libraries. The action is now on Darrell to use this for the fix to mpicasa.
   • Julian will monitor gaincal and see if there is anything to do to improve its performance when reading the input.
2. Running CASA's trunk in the cluster without hacks.
   • Sanjay is asking for a general way to run the CASA trunk in the cluster without having to copy libraries here and there. He doesn't want to keep maintaining the charade script. The decision was to put an action item on Jeff to discuss this matter with the cluster manager.
3. Testing
   • James did not run any tests last week. He sent an email with his plan for testing in the following weeks.
   • Sanjay reported that the serialisation problem is fixed. He and James debugged a problem with multi-threading during gridding, which is designed to run with nCores-1 number of processes. This caused the threads to compete for more processes than the existing number of cores. They decided to switch-off multi-threading which solved the problem. Next he will run scaling curves tests.
   • Sanjay asked for help to automate his script when running in the cluster. Julian will send him an email on how to use qsub with his script.
   • Joe asked how could we automate the use of qsub. Justo replied that we must have priori knowledge of the cluster therefore a manual input is necessary.
   • Justo reported that the issues with plotms when running the pipeline with MMS were due to the tileshape used to create the MMS in mstransform. Basically, the tileshape was using the shape of the first spw (which happens to be the WVR with 4 channels). It uses this shape to swap 1 correlation and 4 channels at a time for many rows, instead of all channels and correlations for a few rows. He and Julian tested this which make plotms run as fast or faster than in sequential.
   • During this investigation they noticed that plotms uses openMP without setting OMP_NUM_THREADS, which makes it use as many threads as possible. This problem has been forwarded to Jim for discussion, but no reply from him has been received yet. An action item is in Julian to find out which third party library may be using openMP in plotms. Action item on Sandra to create a ticket and assign to Jim as critical.
   • Regarding the recent problems with the pipeline that fails when getting the keywords of PHASE_DIR column in the FIELD table. Lindsey reported that there is a ticket for this. She thinks it is a Python binding issue, but Sandra disagreed saying that something changed after the casacore merge on the creation of these keywords in the PHASE_DIR column.
   • As a reminder, this meeting will be one hour earlier for people in Garching. It will be the same time for people in Socorro.

Action Item List

• Darrell to use the layout information provided by Julian to fix mpicasa in the binary distribution of CASA.
• Julian will investigate the too many reads in gaincal.
• Jeff to discuss a general way to have the CASA trunk build in the cluster without the use of tricks and patches as it is currently done. For more information, please talk to Sanjay.
• Julian to send an email to Sanjay on how to use qsub with his imaging script.
• Julian to find out which third party library may be using openMP in plotms.
  • The threads in casaplotms are started by GenSort which is triggered by a table sort via the visibility iterator. Not a particular efficient implementation btw. iI is implemented as a parallel for which it invokes nested parallel regions. Using openmp tasks would probably be more efficient (though unfortunately GCC's openmp thread scheduling is not that great for this type of sort in the first place).
• Sandra to create a ticket about openmp in plotms and assign to Jim as critical. Created CAS-7378

Useful Links

• Major blockers
• Current status
• CASA parallelization task list
• Long-term issue chart
• Bleeding tests area
• Documentation (see Reference Materials topic)