Preferred MPI Implementation

Background

A preferred MPI implementation needs to be identified and incorporated into the CASA build system and development environment (See notes from Justo).

Some consideration was given to the selection of a preferred library or relying on existing system libraries. Options considered included:

1. Pick a single library and package it with CASA: (Preferred) This is the lowest effort solution that keeps maintenance costs to a minimum. We leverage existing cmake wrappers provided with a given MPI library for compilation and package the library with CASA for distribution.
2. Provide multiple binaries compiled against different MPI libraries: This would increase support for clusters with existing MPI implementations, but at greater effort for the CASA group.
3. Compile for generic MPI support to use system libraries: This would require modifying the cmake wrappers to support a generic MPI implementation and may be a maintenance headache. Each library is very similar in the implementation of the MPI standard, but very different in how they spawn processes.
4. Have power users compile CASA if they insist on using their own library: While this seems like low effort, we would have to provide the build tools and support installation that could be a maintenance headache. The MPI calls would also have to coded in an implementation-agnostic way to ensure no dependence on a specific MPI library.

Given the trade offs involved, option #1 is preferred. This wiki page aims to document the requirements for selecting an implementation to incorporate into the CASA build and development environments.

A discussion on the topic can also be found in ticket CAS-5801

Requirements

Implementation Requirements:

• The preferred library should be provided in the development environment, integrated into the build system, and distributed with the CASA package.
• JR: It is desirable if MPI can be bundled with CASA and installed side-by-side with any existing MPI implementations on the system. The intent is to be indifferent to any pre-existing system MPI configuration - neither relying on it nor disturbing it.
• JK: The implementation should rely on the MPI standard calls, not custom calls or extensions specific to a library.

Preferred library requirements:

• JR: It is desirable to use an MPI implementation that is already supported within NRAO/NAOJ/ESO so that we can leverage institutional knowledge.
• JR: Competing libraries are understood to be very similar at the code level in terms of their implementation of the MPI standard. Process spawning approach differs as does the launcher/process manager. A launcher/process manager that is well supported in HPC applications is preferred. Integration with Torque and Maui is highly desirable. This should be considered a driving requirement in selecting a preferred implementation.
• JR: Robust support for infiniband communication is required (Native infiniband)
• Robust support for 10-40Gb Ethernet is required.
• The ability to black-list devices is required. E.g., the MPI implementation should be restricted to Infiniband if both Infiniband and Ethernet interfaces are present.
• JG: An active development community who can assist with issue resolution is desirable. A sizable community using that particular combination of MPI implementation and the python bindings for MPI would be desirable.
• JR: A library with prospects for relatively high longevity is preferred.
• JG: A library with support for MPI calls at both the python and C++ level is required.
• JG: The library should support the MPI-2 standard.
• Support for linux is required, while OSX support is desirable but not required.
• SB: A library that is provided with supported Linux distributions is preferred.
• JG: The chosen library has to support MPI THREAD MULTIPLE.
• JG: Support for selective xterm.

Constraints & Considerations

• JG: It is necessary to compile the code which uses MPI with the compilers supported by the chosen MPI implementation.
• JG: There are big differences between the various MPI implementations. This is due to the fact that some parts of the MPI API are quite flexible. For instance some basic MPI calls like MPI_Send allow the implementation to block or not, and to buffer or not, either on the receiving or sending sides. Therefore, we really need to decide not only a 'preferred' implementation for CASA, but also the implementation (or set of implementations) that we are going to test, in order to guarantee that CASA works properly with them.
• MP: CASA developers should rely on the MPI standard when writing function calls to ensure that code can be easily modified to be compiled against a different MPI implementation should this be necessary at a later date.
• SB: It would be advantageous if MPI dependent code be bracketed in #define's so that dependence on MPI can be switched off at compile time if necessary. This helps in reducing the dependence on third party packages since there are cases where not everything is always required.

Considered Libraries

• OpenMPI
  • http://www.open-mpi.org/
• MPICH / MPICH2
  • http://www.mpich.org/
• MVAPICH
  • Fork of MPICH-2.
  • http://mvapich.cse.ohio-state.edu/
  • https://developer.nvidia.com/mvapich

Implementation Comparison

Requirement	OpenMPI	MPICH/MPICH-2	MVAPICH
NRAO/ESO/NAOJ Institutional Knowledge	Used by Justo for initial development/testing. Walter Briksen has some experience with use on DiFX correlators.	Previously used by NMS group. Switched to MVAPICH due to Infiniband optimization. (TBC)	Heavily used by the NMS group for the CBE.
Launcher / Process Manager support in HPC community (Torque/Maui/etc integration)		Includes Hydra which has excellent integration with Torque/Maui/etc. Strong support in HPC community.	Includes Hydra which has excellent integration with Torque/Maui/etc. Strong support in HPC community.
Infiniband Support	Infiniband / OpenIB, Infiniband / mVAPI	Yes	Yes
Ethernet Support	Yes	Yes	Yes
Black-list Support	Can be restricted See FAQ Section 7
Community Size & Longevity	Safe bet	Seems quiet	Currently active
C++ & Python calls	Included	Included	Included
Compiler Support	GNU Compiler
MPI Standards Supported	Up to MPI-2	Up to MPI-3	Up to MPI-3
OS Support & Bundling	Linux and OSX support. Bundled with OSX.	Linux and OSX support. Bundled with RHEL6.	RHEL5 & 6 support only.
Other Considerations	mpi4casa already tested with OpenMPI.		NVIDIA GPU Support

Conclusions

After reviewing the requirements it appears that all the major implementations will meet our core needs. Justo will test integration of openmpi on the AOC lustre cluster and review the listed requirements in detail. Likely proceeding with OpenMPI.

-- RobSelina - 2014-08-14