Current Status & Known Work Remaining

• Environment Management & Infrastructure
  • MPI Framework is in place. MPI rpm packages are included in the CASA binaries distribution. There is an issue pending with the wrapper mpicasa.
  • Simple Cluster - to be deprecated in 4,5 or 4.6
    • Notes in the cookbook on the old framework.
  • MPI framework
    • Meant to deprecate simple cluster. Parallel_task_helper used if mpi installed, if not, falls back on simpler cluster at the moment.
    • Justo has a doc describing MPI framework. Latest version available in the SVN tree.
    • MPI interfaces with torque to determine resources (in NMASC case).
  • Multi-MS processing
    • Current documentation is in here
    • All tasks should be MMS aware at this point. Work pending is listed below, per task.
  • Heuristics to automate generation of engines and/or threads based on available resources.
    • Implemented in partition/mstransform with the currentl separationaxis='auto' and numsubms='auto' parameters.
• Filler
  • Task importasdm has been integrated with partition to create a Multi-MS.
  • Parallel filler only parallelizes by SPW at the moment.
  • Can currently use lazy filler, followed by partition to create MMS. Is equivalent to a parallel filler, and uses MPI framework this way. This is a working solution for first release. This needs to be tested.
• Flagmanager
  • Needs to be parallelized.
• Wvrgcal
  • Seems to be solved in CAS-7200
• MSTransform
  • Parallelized.
  • All problems listed in this document have been solved. additional notes on mstransform related work
  • Task cvel2 with full suport to MMS is available in version 4.3.
• Calibration
  • Applycal is parallelized but not tested; but generalization of cal tables is not yet parallelized.
    • Will work with the MMS (MMS aware). Seems to have memory issues.
    • Need basic functional tests for task applycal in order to test it with MMS. CAS-7199
  • Tasks that generate calibration tables are not parallelized.
  • Setjy is fixed.
    • Dominates the VLA pipeline.
• Imager
  • Current efforts are focused in tclean.
  • Parallelization for single/multi-term imaging with single/multiple MSs is working in the new framework for continuum imaging. Some work is required for Cube parallelization, but the basics have been prototyped and they work. The part's that needed is some of KG's heuristics on how best to split up giant cubes and to sync with data selection.
  • See this ticket for current status : https://bugs.nrao.edu/browse/CAS-6616
  • Both Cube and Continuum parallelization are deliverables for 4.3. Sanjay is currently testing 'Continuum' parallelization on big datasets (not just the small test cases Urvashi had).

• Pipeline
  • It runs to completion with MMSs. Implements Tier-0 for some plots and for tclean calibrator imaging.
  • Job executor class could be improved as a 2nd order improvement.
• User Interface(s)
  • Plotting is a significant portion of pipeline processing time.
  • Should at least consider processing multiple plots in parallel, though each plot is likely to be a serial process. (Done with Tier-0)
• Build System
  • Major blocker to system deployment. See notes below on work remaining.
    • Integrate mpi4casa unit tests into nightly system, using a multiple-node hostfile (see https://svn.cv.nrao.edu/svn/casa/trunk/gcwrap/python/scripts/mpi4casa/test_mpi4casa.py)
    • Fix pending issues with mpicasa wrapper when running in multiple nodes.
• Testing
  • Targeted tests of MMS and mstransform, simple_cluster use in Imager, openmp in flagging/gridding, etc...
  • End-to-End testing of typical scripted use cases (post 1st release)
  • Pipeline testing with MMS.