Casa HPC/Parallelization meeting agenda/minutes

Thursday [02/02/2015], [ESO Pavo, A.2.01], [16:00 UT]

How to connect

• Dial in: +49 89 3076833
• Video connection: Use this one (46103@134.171.42.27).
• NRAO-Socorro: room 280
• NRAO-CV: room CV-ER 245

Attendees:

Agenda

1. What will be in next release
   • new baseline separtionaxis in partition and importasdm (CAS-7890)
     • A better implementation is pending on CAS-8292 (msmd.nbaselines should return also the number of auto-correlations).
     • Dave just committed the implementation of CAS-8292, so I will try it in parallel_data_helper.
   • split2 and hanningsmooth2 became the default in CASA (renamed to split and hanningsmooth) (CAS-7886)
   • simple_cluster should be removed or deprecated? (CAS-8226)
   • changes in mpi4casa
   • using tclean in alma-m100-hpc. What needs to be done?
   • OTF calibration for MMS. Should we disable it in 4.6?
2. Change to the structure of an MMS: create only one set of sub-tables, physically located inside the first SubMS and linked to all other SubMSs. Motivation for this change:
   • I've noticed that the MMS structure as it is, may get into an inconsistent state if any sequential tool modifies a sub-table of the MMS. The current structure is that all sub-tables of the SubMSs are created the same and the top reference MS has links to the first subMS sub-tables. If a sequential tool modifies for example the SOURCE table of the MMS, it will ONLY modify the SOURCE of the first subMS. The other subMSs from this point on will be different, therefore the MMS will be inconsistent.
   • We assumed that sub-tables are read-only, but at least in 2 cases they are not: setjy with SOURCE table (usescratch=True) and updates of the HISTORY table.
   • For setjy, Kumar has suggested: "The first solution thst comes to mind is to make parallel setjy to clear the virtual model by one process (the master may be) and make makeVisset have an option of not clearing the virtual model (which will be called when in multi process mode). May be it should be implemented as such in the refactored prediction code." Can we ask Tak to try this?
   • For the HISTORY table, I have tested with flagdata and applycal, by allowing only the MPIClient to write to the HISTORY table. This implementation still needs further changes, I believe when the MS is open. I need to try a few lock options to verify.
3. Martin's C++ and MPI integration in CASA discussion

Minutes

1. 4.6 and 4.7 releases

• New baseline separtionaxis in partition and importasdm (CAS-7890).
  • Will modify the implementation in partition to use the new msmd.nbaselines()
• split2 and hanningsmooth2 became the default in CASA (CAS-7886).
  • cvel2 will be renamed in 4.7 as there is one pending difference between the 2 tasks.
  • Note for the pipelines: hanningsmooth2-->hanningsmooth has a different interface. Please, see CAS-6179.
• simple_cluster will be deprecated in 4.6 and should give a warning when users use it. Any parallelised task running on an MMS will run in sequential if CASA was not started with mpicasa. simple_cluster will be removed in 4.7.
• mpi4casa changes (CAS-8157)
  • Allow to change at run-time the OpenMP settings (in particular maximum number of threads) is in 4.6.
  • Allow to change at run-time the maximum amount of memory that each CASA server can use will go to 4.7. Justo needs feedback from imaging team about this feature.
• tclean in alma-m100-hpc regression
  • It works in sequential and in parallel with both clean and tclean. But there are changes in the peak flux of images when using tclean, in the order of 3 magnitudes. Differences are seen for cube imaging. Jeff says that differences are expected between clean and tclean. The work to convert to tclean in alma-m100-hpc will continue in 4.7 with the help of Urvashi.
• OTF calibration in mstransform
  • George just committed a fix for the MMS case. Justo will verify the fix and depending on the outcome, we still keep OTF in mstransform in 4.6.

2. Change to the structure of an MMS: create only one set of sub-tables and make links to all SubMSs.

• We decided to implement this change in two steps:
  • For now, Sandra will try to fix the HISTORY locking problems. If that is successful, she will make all sub-tables links, except for the SOURCE table.
  • The solution for the SOURCE table locking will take longer because it will depend on the refactoring of setjy. Tak will work on setjy2.

3. Martin's C++ and MPI integration in CASA discussion

• To integrate C++ code using MPI into CASA, we must decide how to manage the MPI communicators required by C++ code. The management of MPI communicators involves several issues that must be addressed, including:
  • communicator lifecycle: how are communicators created, and by which components, and how are they destroyed;
  • how to pass communicators across the Python/C++ divide, assuming that will be necessary.
• The primary decision to make regarding the use of these sorts of communicators in CASA is whether they should be created on the Python side or the C++ side.
• Martin pointed out that in order for a communicator to be created on the C++ side, the C++ code will need to begin with MPI_COMM_WORLD for the creation of any new communicator for an instance of a distributed tool. This immediately imposes on Python code running in any other thread that it refrain from using MPI_COMM_WORLD. One option would be that the Python code passes a communicator to C++.
• Julian also agrees that the managing of resources, tasks and communicators should stay in the Python side. They both point out that CASA's intervening layers make passing a Python object from Python to C++ difficult.
• Justo prefers the option to either do this directly in the C++ side or use an alternative solution. One alternative would be to pass only a list of available ranks to C++ to use.
• Jeff sent an email with a link to discussions related to this about using mpi4py with SWIG for the connection with C++ layers.
• Jeff asked if a standalone C++ code using MPI would still work with this approach. No one is sure about it.
• We have decide to proceed in the following way:
  • Julian will contact Darrell and Kumar to understand how the middle layers work and whether it is possible to pass a Python object from Python to C++ in an elegant way.
  • After that, Julian and Martin will create a prototype showing if their idea works. Justo will help with what is needed in terms of changes in mpi4casa.
  • Sandra pointed out that the prototype should accompany a design document describing the idea. This document should be circulated to the whole team and especially include the imaging team to understand if all use-cases are covered.
• Jeff mentioned that the I/O project should be looked at for 4.7.
• Julian and Sandra mentioned other priorities on their side at ESO. Jeff said the HPC project currently has higher priority over the I/O investigation.

Action Item List

• Julian will contact Darrell and Kumar to understand how the middle layers work and whether it is possible to pass a Python object from Python to C++ in an elegant way.
• Julian and Martin will create a prototype showing if their idea works. Justo will help with what is needed in terms of changes in mpi4casa.

Useful Links

• Major blockers
• Current status
• CASA parallelization task list
• Long-term issue chart
• Bleeding tests area
• Documentation (see Reference Materials topic)