Casa HPC/Parallelization meeting agenda/minutes

Thursday [23/07/2015], [ESO Centaurus, C.2.01], [15:00 UT]

How to connect

• Video connection: Use this one (46103@134.171.42.27).
• via Telephone 0049 89 307 6833

Attendees:

Agenda

1. Progress on action items:
   • CAS-7191: excessive amount of fsyncs in CASA (Jim/Jeff)
   • Check if applying a .flagversions created from an MMS also works on a MS (Sandra)
   • Remove the parts of the MPI library that do not work properly in the CASA binaries (Julian)
   • Look into the problem of the many reads of the FEED table (Julian/Jim)
   • Status of failures when memory resident is switched on in CASA (Jeff)
2. New tests
   • James performance tests
   • Sandra's ALMA pipeline run using Tier-0 parallelisation for calibrator imaging
   • SD pipeline results when using Tier-0 parallelisation, as reported by Takeshi
3. Automatic tests for the pipelines and tclean
   • Nightly tests of the pipeline
   • Functional tests of tclean? How is tclean tested when code is committed?
4. AOB

Minutes

1. Progress on action items

• (CAS-7191 excessive amount of fsyncs in CASA). It has been postponed to CAS 4.6
• (CAS-7705: tClean exhibits excessive number of open() calls against MS subtables). Assigned to Jim to evaluate the options.
• Test failures when switching on Memory Resident. Some keywords from MR tables are missing in the implementation. That sounds like it might be a problem with the underlying memory resident table feature. We might be using the keyword feature more than other CasaCore users and thus be the ones that are encountering it. Alternatively, I might have broken the connection between the MR subtable and the auxiliary files (e.g., table.dat) that contain the keyword data.

2. New tests

• James had problems on his testing framework. The second pass through hif_importdata reports a missing Source.xml. He will resume his test with a workaround that copies the missing XML file. He will use the new version of importdata that creates the MMS inside partition.
• Justo suggested that James should set the OMP_NUM_THREADS in his .profile to force also the serial execution to use the same number as the parallel run. This may be the reason why James's tests showed tclean to be slower when running on MMS.
• Sandra reported that the 3EBs testing with the interferometry pipeline gives a performance speed of about 30%, always using 8 servers. Only the Tier-0 parallelisation of tclean for calibrators gives a better improvement of about 64%. This is seen in the weblogs of both runs. These tests were run with OMP_NUM_THREADS=1.
• An additional test is to be done using OMP_NUM_THREADS set to > 1, when tclean's seg fault is resolved.
• Tak just committed a fix for the seg fault in tclean.
• ESO reported that the SD pipeline already implemented Tier-0 parallelisation for the baseline fittings. They used 7 scantables in their tests (coming from 3 ASDMs). The datasets are small, but still showed an improvement in performance of about 17% in total and about 73% for the Tier-0 step. Most of the processing is still sequential, that's why there is no big gain in the total execution time.

3. Automatic tests of the pipeline and tclean

• It was asked if there are any automatic nightly tests of the pipelines. Dirk runs some use-cases tests in Bonn, but they are not daily.
• Akeem has included a daily test of the pipeline using a VLA dataset, but the test is not passing yet. They should be available in https://svn.cv.nrao.edu/test-summary/ but Sandra couldn't find the pipeline test in there.
• There is currently no automatic regression using tclean. As for functional tests, Tak said that Urvashi is writing many tests.

4. AOB

• We discussed the targets for 4.5 and their status regarding the upcoming feature freeze in 1st of August.
• Sandra said that time average will be in, but it is only implemented for the clip mode. Justo explained that adding the time averaging iterator to tfcrop and rflag would require major changes in the code. Sandra also said that the current time averaging in clip cannot work together with other modes in list mode because they use different iterators.
• Sandra will look if there is any ticket open on the HPC component that would affect the 4. 5 release

Action Item List

• Sandra will check if applying a .flagversions created from an MMS also works on a MS.
• Julian should remove the parts of the MPI library that do not work properly in the CASA binaries (Example: mpicc, etc.)
• Jim will look into the problem of the many reads of the FEED table.
• Jeff will follow-up on the problem with too many fsync calls in CASA (CAS-7191)
• James will run pipeline tests using different separation axis in partition (scan and spw), instead of 'auto' (balanced mode).

Useful Links

• Major blockers
• Current status
• CASA parallelization task list
• Long-term issue chart
• Bleeding tests area
• Documentation (see Reference Materials topic)