CASA Parallelization Architecture

General Approach

Our general approach will to be to attack the "embarrassingly parallel" portions of data reduction. That is to focus on those portions of the system where discreet chunks of data can be used atomically (no intercommunication) in processing. This is the place where we can garner the largest return most easily.

This first-order parallelization will be achieved using a MultiMS (MMS) approach. When the filler is run we will produce multiple MSs. As a first pass I think that there will be one “Calibration” MS which contains all scans labeled in the SDM with the intents of calibration, and many “Target” MSs which are a subdivision of the scans on target in the original MS. The optimum partioning is TBD, and multiple options shall be supported including SPW, channel, scan, and time.

The Calibration MS will be worked on in an interactive way (or through the pipeline) to generate a set of calibration tables. During this phase we may also generate a set of flag commands, which we will want to apply during the processing.

The steps of applying the calibration tables and flagging data will be performed in parallel on each of the smaller target MSs by having a one to one mapping between process and MS. The CASA team has already demonstrated imaging of these sets in parallel.

Interactive flagging and data inspection will also be required as part of the normal processing, making the tools for this aware of the multiple MSs to simplify processing will also be required, although at a lower priority than the key pipelined modes above.

One of the assumptions I am making here is that we are writing the intermediate data results to disk each time we complete a processing step. I think that this is the approach that many users are currently following. And in part drives the request to remove scratch columns (i.e. just save a MS with the corrected data stored in it, rather than keeping both the raw data and the corrected data). One of the steps that we will eventually want to do is to remove as many of these intermediate data products as possible creating flagged and calibrated data on the fly. This is not part of the current effort to parallelize CASA as it is currently operating.

System Elements

There are at least six architectural elements to a CASA that is optimized for use in a HPC environment. They are:

1. Environment Management
2. User Interface
3. Filling
4. Flagging
5. Calibration
6. Imaging

Environment Management

Environment management pertains to the global portions of the system, affecting more than a single task in the overall reduction system. There are two immediate issues within this section, IO Management and Resource Identification.

IO Management

CASA efficiency is dominated in most cases by CPU usage. However because the disk IO is synchronous, the CPU is often blocked in a call to disk IO. The Lustre file system can help reduce the affects of this, but cannot correct what is a serious design limitation.

One of the top priorities for CASA must be to move to an asynchronous IO model. There are a number of places to do this, the prototype IO manager developed by James is good for testing, but is not portable. Attacking this problem at the Table level in the MS may be the correct solution, but unless Ger decides to tackle this problem, probably is beyond the currently available resources. Within the CASA team I think that we can reasonably introduce a “pre-fetch” capability in the VisBuffer / VisInter framework which will ensure that the CPUs of a given machine are fully engaged in processing data.

Resource Identification

CASA will be used on a wide variety of platforms and the system must identify the available resources and use them efficiently at run time, not at compile time. To do this, the system must have a method whereby it can be configured for a given system (list of nodes, cores available per node etc). This needs to be abstracted out, such that all tasks within CASA can discover what resources are available and design a processing strategy appropriate to the environment.

This should default to the resources available on the current system so that CASA works in an optimized way out of the box, but can be customized by advanced for their particular needs.

User Interface

Specifying all of the MSs within a multiMS set at the input to a task is not a practical approach, CASA must provide a simpler interface to this. I propose a multiMS structure be introduced to CASA at the python level. This structure would contain:

• A list of MSs (or other multiMSs), which are to be processed together.
• An optional dictionary from the MSs to MS specific parameters to be used in whatever operation is being performed. These would override any options specified at a higher level.
• A flag on each MS in the list to simplify looking at only parts of the data .

The recursive design of allowing the containment of multiMSs is to allow combinations of data from different observations to be jointly reduced. The user interface to this class should contain the following methods:

• List: Show the list of MSs and the current state of their flag.
• Enable/Disable: Takes a list of MSs and sets the state of the associated flag.
• Input: Takes the name of a MS and returns the current state of the input parameters, these will be the same as the parameters for the current task except where a MS specific option has been given.
• Set: A bit awkward but takes a list of MSs, the parameter name and the value and sets this parameter for each MS. I don’t think these will need to be used all that often so this ugly syntax may be ok.
• Add: Manually add a additional set of MSs (or MultiMS)

I think we will also want a global task to Load/Save the MultiMS definition to disk to allow easily picking up where reduction left off.

Adding support to plotMS and the viewer for the multiMS structure is probably the first sort of user tool enhancement we should consider. Most other tasks can probably add something like the following snippet of code to support multiMS in a completely trivial way:

if isinstance(vis, multiMS):
  for ms in multiMS.msList:
    if multiMS.isEnabled(ms):
      vis = ms
      <set parameters if necessary>
      <invoke same task again>
 

Filler

Currently the filler is able to separate the data by scan, allowing time division multiplexing. This is the minimum functionality necessary to allow the filler to create a multiMS. The path I see going forward is:

1. The creation of a pFiller task which will automatically create a calibration MS and the target MSs. This task should also create the multiMS object as a starting point for further analysis.
2. Introduction of an option to operate in a frequency-multiplexed mode as well as time-multiplexed mode. If this is done per spectral window then it may make sense to create multiple calibration MSs.
3. When not on the Lustre file system this may run into issues with head contention. The filler may be a candidate for an early second phase (i.e. threaded) approach .

In parallel to this we need to investigate the performance of the filler overall and see if it can be improved.

Flagging

All flagging information (aside from interactive flags) should flow through the FLAG_CMD table. We need a way of inserting new commands into this table (for instance a type of flag SHADOW) for all MSs in an MSCollection, or for each individual MS.

The actual process of applying flags (or removing them) is then launching the tool in parallel on all of the MSs in the multiMS. So the flagging task becomes a two-step process, define all flags to be applied, and then select which of the available flags actually should be applied. We should never go through the code more than twice for a given invocation of flagging, one to gather any necessary statistics and a second for the application of the flag, of course often a single pass may be sufficient.

The tasks in this are:

1. Modify the flagging tool to work with the flag_cmd table. One issue here that needs to be addressed is the number of agents that can / need to be specified.
2. Create the pFlag task, which works on the multiMS input.
3. Enable task level methods to add lines to one or more entries of the multiMS.

Calibration

The generation of CalTables in a parallel environment does not fall within the first phase of our parallelization effort. The three things that must be done in this regard are:

1. The CalTables must be generalized to allow transport between various MSs.
2. The specification of the application of the CalTables must be simplified to allow easy specification within the multiMS context.
3. pApplyCal must be created.

SelfCal will need to be integrated in here, but I’m not sure what the issues are.

Imaging

Imaging is the most advanced in our parallelization path, I think that packaging this up so that is uses the multiMS structure and the environment specification is all that is necessary.