Casa HPC/Parallelization meeting agenda/minutes

Thursday [12/2/2015], [ESO Centaurus, C.2.01], [9:00 AM MST]

How to connect

• Dial in: +49 89 307 6834
• Video connection: Use this one (46104@134.171.42.27). Other option is 46104@eso.org

Attendees:

Agenda

1. List of action itens from last week.
2. Status of mpicasa in the binaries.
3. News from testing campaigns
4. Status of synthesis/Parallel code
5. Imaging scripts (new MPIInterface)
6. AOB

Minutes

1. Action items

• No action items left from last week.

2. Binaries with mpicasa

• Julian reported that the mpicasa errors we had in the cluster are due to the layout in the tarball not being as openmpi expects. He will work with Darrell to change this.
• Julian will check if the tarballs have torque included in?

3. Testing News

• Sandra did not run another test with the pipeline because mpicasa failed in the cluster.
• James reported that running the pipeline in the cluster using mpicasa, it seemed that the pipeline was running in sequential, even though it was reading an MMS. Sandra told him how to check the casapy-log to find information about the running MPIServers. Later in the night we discovered that it was in fact running in parallel, but it probably hit the bug that Justo has just fixed on MPIClient. James is now trying with the latest binaries.
• Sandra reported that she changed the default of numsubms to 'auto' in partition and mstransform.
• Tak will try test_setjy on MMSs. Sandra said that he can use with mpi or simple_cluster for these first tests. The important is to see if all tests pass with MMS.

4. synthesis/Parallel code

• Sandra is working on introducing MPI in the logging, at the C++ level. The initial ideal was to use boost::mpi for this, but it was dropped. She will look into using the code in synthesis/Parallel which uses raw mpi calls, but the unit tests are seg faulting. Sandra pushed the resolution of this to 4.4 and 4.5 as the first assessment of the problem did not look simple to solve.

5. Imaging

• Justo delivered first version of MPIInterface mimicking simple_cluster/parallel_go. It covers all the usage patters need by PyParallelImagerHelper, which are exercises also in a dedicated unit tests in test_mpi4casa. Sanjay tested it and found that tclean uses the convention to count from rank 0 instead of 1 as MPIInterface does. The consensus in the group is that tclean should be changed to use the convention of the MPI framework.

6. AOB

• Justo fixed performance issue for python code executed at the client. This was specially important because some unit tests were severely slowed down, and difficulting development. Now unit tests run faster with mpi4casa than with simple_cluster. Normal data reductions /pipeline can benefict if they have a significant amount of heuristics in python code.
• Justo added some functionality to MSSummary to use DDI instead of SPW for the new balance mode. This is now being tested.

Action Item List

• Julian will check our local tests and see if there are signs of slowness with the latest binaries.
• Andy will check regression run times for any performance differences with latest binaries
• Sandra to ask build/test team to include test_mpi4casa in the list of smoke tests. It should also be included in the nightly tests. Sandra will send also a list of the tests that should run with mpirun in front. Created CAS-7337
• JK check if other people in the team also prefer separated parallelisation logs

Useful Links

• Major blockers
• Current status
• CASA parallelization task list
• Long-term issue chart
• Bleeding tests area
• Documentation (see Reference Materials topic)