Casa HPC/Parallelization meeting agenda/minutes

Thursday [22/01/2015], [ESO Centaurus, C.2.01], [9:00 AM MST]

How to connect

• Dial in: +49 89 307 6834
• Video connection: SIP URI/H323ID: 46104@eso.org (46102@134.171.42.27)

Attendees:

Agenda

1. Question to Jeff. What is the priority for the other CASA developers to fix tickets raised in the HPC context?
2. We will go through the open items decided in the December meeting.
3. AOB

1. Open Items

• Imaging team
  • Port Sanjay's imaging script to use MPI (it currently uses parallel_go). Who should port it?
• James
  • test the tileshape parameter in mstransform
  • 3c147 script also uses parallel_go. It should be ported to use MPI
• Build and Test team
  • status of request from Sandra last week to include test_mpi4casa in the test system
• Tak
  • look at the cube side of imaging in place of Urvashi.
  • New tickets in setjy that affect the Multi-MS cases: CAS-7279, CAS-7278
• Jim
  • Status of ticket: Multi MS opens all the MS when accessed (CAS-4860)
  • Status of ticket: Option to open main table in write mode and sub-tables in read mode so they can be shared (CAS-7016)
• ESO team
  • SC, JG: work with Stewart on the parallelization issues of the pipeline. (pending with changes in partition)
  • JT: Julian will work with the build team in CV on the MPI distribution in CASA. (cmake changes, LD_LIBRARY_PATH)
  • SC: Write a quick-guide explaining how to run tests in the cluster in Socorro and in general. (in progress).
  • SC, JG: test the ALMA pipeline with MMS (will resume once the auto mode in partition is implemented)
  • SC: other tests (functional tests results). Have fixed the following tests: uvcontsub, listhistory, setjy. Will create tickets for the remaining failures.
  • SC: introduce MPI to synchronise the logs in the C++ layer.
  • JG: Results of new "auto" mode in partition.
    • CAS-7280 opened to track the progress with the new partition mode. The rationale for this is explained in the ticket, but in summary: A new partition mode is required to balance the load w/o falling into the robustness problems observed when using a large number of subMSs (see Cycle2MaxNumberSubMS). In particular we aim to avoid the following problems:
      • maximum number of open files ( CAS-4860)
      • wvrgcal crashes when using large number of subMSs ( CAS-7200)
      • empty Feed sub-table when only WVR SPWs are present in a subMS ( CAS-7167)
      • Flagdata performance degradation in list mode (large number of online commands) ( CAS-7201)
      • Applycal performance degradation (see Cycle2MaxNumberSubMS)
    • First version implemented, it allocates scan,spw pairs to balance the load (nVisibilities = nRowsxnChannelsxnCorrs) among all subMS
    • It takes into account 4 criteria: Total subMS size, and homogeneous distribution of visibilities along spw, scan and field axis
    • Example (ALMA cycle 2 data):
      • Distribution of visibilities along spw axis
      • Distribution of visibilities along scan axis
      • Distribution of visibilities along field axis
    • The new mode partition mode 'balanced' is integrated in partition (r32163)

2. AOB

Minutes

• JK says the HPC/parallelization of CASA has high priority but a better triage of the tickets needs to be done. Sandra should send Joe a list of blockers for the HPC work so that they will prioritise them together. The tickets are easy to find. Search for any tickets with HPC component in JIRA to see them all. They are also listed in the twiki of the HPC sub-project.
• SC reported that Sanjay's calib script is easy to parallelise, but the use of the cb tool will have to be changed or replaced. They will discuss it offline
• SB explained he partitions the MS only after it's calibrated and before doing imaging. The locking issue is because the subMSs have some overlapping data and because imaging actually writes to the input MS.
• JR asked for a script to test in lustre. He also needs a CASA installation.
• AH says the test system needs a production tarball to run. JK says, the MPI libraries need to go into the tar balls as soon as possible so that they can be used in the 4.4 release.
• JT need to check if the swig threads option is included in cmake. The LD_LIBRARY_PATH setting for the MPI libs is already in cmake.
• SC explained that the current modes in partition are not sufficient for our needs. The current modes are not flexible enough to allow a load balanced MMS without maximisation of the number of subMSs. Maximization of MMS itself proved to be problematic too.
• SC stressed one more time that we are not running performance tests. We are trying to run the pipeline script through the end but issues in different places interrupt this. The applycal "almost" hanging run is one of the signs that the current partition schema does not work. At this point we need to stop the tests and review partition.
• JG explained the new "balanced" mode in partition which takes the scans, spws and fields into account in order to optimally distribute the data across the subMSs with the best load balance in size. Sandra is testing this with the pipeline.
• SC explained that a "default" in a task means, it runs without giving any other parameters apart from the input and output names. She also explained that we are not trying to find a set of parameters that runs for all use-cases. What she and Justo are trying to do is to make the current "default" parameters of partition better than the current one. Currently, the default parameters of partition separate per scan/spw with a number of subMSs that is impossible to control. The lack of control leads to having the WVR spw alone in a subMS, which causes problems in other tasks (flagdata, wvrgcal and maybe in others that we have not discovered yet).
• JR says it is better to re-partition later rather than trying to predict the best partition up front.
• SC will test the pipeline without setjy.
• SB and JR said for their development they prefer to have the logs from all MPI servers separated in different files. Implementing this has low priority at the moment. Jeff will check with other people in the team if they also need/want this.

Action Item List

• SC to send Joe a list of blockers
• SC to send to James the alma-m100-hpc script for his tests.
• JG to show James how to run the casa build he has in lustre
• MR send a timeline to Jeff for the inclusion of the MPI libraries in the next casapy-test
• LD send Sandra tips to switch off setjy from the pipeline script.
• SB check if parallel_go is used in tclean.
• JK check if other people in the team also prefer separated parallelisation logs

Useful Links

• Major blockers
• Current status
• CASA parallelization task list
• Long-term issue chart
• Bleeding tests area
• Documentation (see Reference Materials topic)