Casa HPC/Parallelization meeting agenda/minutes

Thursday [19/03/2015], [ESO Centaurus, C.2.01], [9:00 AM MST]

How to connect

• Dial in: +49 89 6834
• Video connection: Use this one (46104@134.171.42.27). Other option is 46104@eso.org

Attendees:

Agenda

1. OpenMP defaults:
   • Common problem: Different defaults for desktop/cluster but how do we distinguish?
     • Proposal: Distribute two execution profiles, one for cluster another for desktop
   • Desktop single process:
     • Default: nCores-1
     • "CASA may use by default all cores of your system, this behavior can be overridden by setting the OMP_NUM_THREADS env. variable" (Confirm)
   • Desktop multiple process
     • Default: max((nCore-1)/ nEngine,1)
   • Cluster single process:
     • Default 1 when not set externally (Confirm)
   • Cluster multiple process:
     • Default 1 when not set externally (Confirm)
     • Proposal: If set externally, mpicasa does automatically the following
       • Uses input hostfile w/o modifications
       • Calculates the total number of slots derived from hostfile
       • Uses -n option with value (total number of slots)/OMP_NUM_THREAD
       • Enforces scheduling by host (if it is not the current mpirun default - have to check)
2. Fix in TaQL expression of partition to improve cases of MSs with many scans
3. MultiFile implementation of Ger is already available after the casacore merge, CAS-4860. We need this to be tested.
4. Testing EVLA pipeline (James)
5. Testing ALMA pipeline (Sandra)
6. Imaging (Sanjay)
7. AOB
   1. Science user testing for CASA 4.4? (Mark)

Minutes

• OpenMP defaults: after much discussion we all agreed that in the absence of a OMP_NUM_THREADS setting in the cluster when running with multiple process, mpicasa should set it to 1. James explained that there is no need to set the threads per process for Torque. He explained that all the users running on the cluster should either request all the processes on a node or all the memory, in order to avoid other users being scheduled in the same node. This will be part of his documentation on the cluster usage.
• Julian fixed the TaQL expression creation in partition which was affecting MSs with many scans such as James’s test MS. This should speed up the run by about 40 times.
• James testing on a large EVLA MS look promising, although later there was a problem when running partition, but we suspect this is a rare race condition. Still being investigated.
• Sandra tested the ALMA pipeline on the dataset uid___A002_X969646_X164 and setting OMP_NUM_THREADS=1. For this dataset, running in parallel is about 45% faster than in sequential. Results are:
  • Sequential run:

    total: ~6h    1st applycal: 15min    2nd applycal: 18min 

  • parallel with 4 sub-MSs:

    total: ~4h    1st applycal: 13min    2nd applycal: 3min 

  • parallel with 8 sub-MSs:

    total: ~3.5h    1st applycal: 6min    2nd applycal: 5.5min 

  • parallel with 16 sub-MSs:

    total: ~3h15min    1st applycal: 5.5min    2nd applycal: 5.5min 

• James found this test very fast also in sequential and thinks there is caching involved. In fact, the input data is cached on this machine.
• Sandra reported that Julian created a quick-tool to analyse the log files. Lindsey said the weblogs also print the execution time per task.
• ALMA pipeline testing: we agreed to standardize on two datasets for ALMA and EVLA. Sandra will use Mark Lacy’s ASDM in her next tests with the pipeline. This dataset is A001_X122_X291.
• Science user testing for 4.4: Mark needs a definition of high level targets for the scientists to test. Lindsey said that tclean is not ready to be tested yet. We all agree that the goal now is not to run science testing on tclean, but it’s important to get users testing tclean in a beta mode to help find problems.
• James said that CfA has some dataset to be tested through tclean and James will use those sets for his tests.
• Sandra said she received from Dirk Muders the way to include tclean currently in the pipeline for our tests.
• We decided the best is to create a template of scriptForPI.py to run in parallel and give to some scientists to test the parallelization in 4.4. Mark will coordinate with Amanda to contact us for this work.

Action Item List

• Julian will investigate the too many reads in gaincal.
• Jim/Jeff to discuss a general way to have the CASA trunk build in the cluster without the use of tricks and patches as it is currently done.

Useful Links

• Major blockers
• Current status
• CASA parallelization task list
• Long-term issue chart
• Bleeding tests area
• Documentation (see Reference Materials topic)