CASA HPC/Parallelization Requirements

Introduction

Requirements applicable to the CASA HPC Parallelization effort are described below. An iterative and incremental implementation plan is assumed and requirements are noted by cycle where appropriate. When no cycle is noted, the requirement can be assumed to apply to all cycles.

High Level

• A parallelized deployment of CASA that can make effective use of NRAO, JAO, ESO and NAOJ Clusters.
• A parallelized deployment of CASA that can make effective use of modern PC architectures with multi-core processors.
• Note that these are not separate builds, but rather the incorporation of this capability within the developer build environment as well as the test, stable, pre-release and release packages of CASA.

HPC Architecture

• Many formats of multiple machines accessing shared disk should be supported:
  • Can assume that the HPC system includes a high performance shared disk such as luster, gluster, GPFS, etc.
• Task parallelization level should be tailored to current cluster architectures deployed at NRAO and collaborating institutions. Eg, 16 cores per node, 32 nodes, with 40Gbps interfaces to shared storage media (Infiniband, Ethernet, etc.).
• Integration with common HPC cluster scheduling and resource management tools such as Torque and Maui to determine allocated resources at run time and spawn server processes accordingly.
• The system should allow for selection of different parallelization implementations to make effective use of available resources.
  • E.g., tunable number of engines vs number of threads per engine.
• The parallelization framework shall provide an interface for the use of clusters with scientific GPUs. Note that this interface need not be developed in early iterations, but should not be built-out of the framework.

Parallelization Framework

• A flexible and extensible parallelization framework shall be provided.
• Ability for the nodes to talk to each other (or at least to the controller node) and be able to wait till the controller node finishes some critical operations.
• Graceful failure of parallelization infrastructure in the event of a casa forced shutdown or crash on the client. Need to be able to restart casa with automatic re-initialization of server resources.
• Levels of parallelization (higher level implies that more modifications are needed in the application layer)
  • 1st tier parallelization: Tasks initiated from the client side are internally parallelized so that each server runs the given task with the parameters forwarded from the client in a separated subMS.
    • This mode is already in place, and supported by the most time consuming tasks that can be trivially parallelized internally:
      • mstransform (partition,split2, cvel2, hanningsmooth2)
      • flagdata
      • applycal
      • setjy
  • 2nd tier parallelization: Tasks that cannot be parallelized internally (e.g. plotms) but don't require completion before the next task can be sent to any available server to be dispatched asynchronously
    • It is necessary to identify blocks of tasks that don't require individual completion among them, this has to be done by the application layer
    • Not requiring individual completion implies of course that the eligible tasks cannot modify the common files (e.g. common input MS) in any way
    • It is necessary to establish a barrier/blocking point after the block of asynchronous tasks in order to guarantee completion in any of these cases:
      • If the next block of tasks depends on the outcome of any of the asynchronous parallelized tasks
      • If the next block of tasks modify any of files (e.g. input MS) used by the group of asynchronous parallelized tasks
  • 3rd tier parallelization: Complete data reduction paths can be sent from the client to a group of servers (managed by a master server) making use of the 2 levels of parallelization described before
    • By complete data reduction path we may refer to the complete data reduction of a particular execution block
    • It would be necessary to communicate from the client to each master server the data reduction path in a simple and efficient way (e.g. path to a script)
    • It is necessary to gurantee namespace protection among the data reduction path of each exec block

Resource Management

• The first implementation could simply provide guidance (documentation) on what resources the user should request for common use cases.
  • There is no need to analyze scripts to suggest resource allocation. The system can simply provide guidance for typical use cases (rough calculations).
  • The documentation should also provide guidance on how to partition the MS for common use cases.
• Later implementations should provide a resource manager within the CASA environment that will provide some degree of automated optimization of parallelization for the available resources.
  • Determine the ideal number of threads and processes based on heuristics developed in testing.
    • E.g., likely deploy as many processes (engines) as I/O and memory will support, then thread those engines for unused cores.
  • The resource manager may also guide the division of the MMS, though the user should have the ability to override the recommendation.

Task Parallelization

• Initial task parallelization shall be at the python/task level and be based on the division of the MS in to sub-MSs for asynchronous parallel a processing.
• All CASA tasks must be MMS aware. This is especially applicable to tasks which write to the MS, since the MMS partition structure must be respected rather than creating a new monolithic MS.

Pipeline Parallelization

• The first order parallelization of the pipeline shall simply consist of leveraging the parallelization incorporated into the CASA tasks.
• Where pipeline tasks replace the functionality of CASA tasks and interface to the tool level, the pipeline tasks shall be MMS aware.
• In an effort to allow for routine use of the pipeline and the generation of more useful data products, the flagger shall be revised to be a parallelized RFI flagger.
• A significant portion of current pipeline processing time is plotting. While plotting is an inherently serial task, the pipeline shall spawn parallel plotting processes when able.

Build System Integration

• Integration with the B&T system such that CASA can be built and tested routinely on the cluster nodes during development.
• Nightly builds should be compiled to support the HPC framework and provide the existing parallelization infrastructure in the test builds/packages.
• Parallelized builds should be “Atomic”, incorporating all required library dependencies, and not be reliant on O/S libraries.
• The CASA trunk shall remain functional (buildable and deployable) in the transition from serial to parallel tasks.

Testing

• Unit tests shall be created for the new parallelization infrastructure and shall be incorporated into the Jenkins automated testing infrastructure.
• Regression tests shall be developed to confirm both numerical accuracy of the parallelized tasks as well as the processing time for various tasks.
• Existing parallel implementations shall be characterized early in the development cycle, with the results used to inform subsequent iterative development cycles.

Programmatic

• It is desirable to have the initial infrastructure deployment incorporated into the CASA 4.3 release.