Casa HPC/Parallelization meeting agenda/minutes

Thursday [26/02/2015], [ESO Centaurus, C.2.01], [9:00 AM MST]

How to connect

• Dial in: +49 89 6834
• Video connection: Use this one (46104@134.171.42.27). Other option is 46104@eso.org

Attendees:

Agenda

1. Action items from last week
   • Mark to talk to Darrell about giving Julian info/access to working with the binaries. Darrell says that this is fine. Julian (included in CC) should drop him a line with details of the changes, & he'd be happy to help with him with it.
   • Andy to ask Darrell about including test_mpi4casa in the smoke checks.
   • Julian to send an email showing how to use the hostfile with mpirun (mpicasa) to the test team. Posted in CAS-7337
2. Logging
   • MPI ranks in CASA log messages (CAS-6705) (Sandra)
   • New log filter for NULL MS Selection exceptions when running in parallel (CAS-7122) (Justo)
3. Testing
   • Pipeline (Sandra). Log is in /lustre/scastro/testpipe/logs/X57d_4.3_mms_casapy-20150225-165534.log
   • Scalability (Justo)
   • Imaging (Sanjay)
   • any other?
4. AOB

Minutes

1. Build System

• We clarified that test_mpi4casa should be used in the nightly system not in the smoke checks. They should not run after every svn commit.
• The mpicasa alias still has issues. Julian cannot do anything because the build environment is not reproduceable. Mark will talk to Darrell about moving forward with this. At the moment, mpicasa works only on one node and cannot be used in a cluster for example.

2. Logging

• Sandra reported that she's waiting for the casacore veto to be released in order to fix in the changes to the LogOrigin. CAS-6705
• Justo created a new logfilter in code/parallel for the NULL MS selection exceptions in subMSs. Maybe we should remove the changes added to setjy and use this instead.

3. Testing

• Sandra run the pipeline+MMS again with success. Lindsey saw some failures in plotms during this run. Justo thinks the problem might come from plotms and GIL. Julian has monitored one of this pipeline runs and thinks the problem is not in GIL locking the run because plotms shows it's doing something during that time. Justo will investigate this.
• Julian also saw that gaincal is very inefficient with bad I/O performance. Gaincal reads the same file hundreds of times.
• It was reported that Jim is looking at some possible issues in the Vi/VB which might be related to the cache, that would explain the problems we see now. Julian will send an email to Jim to discuss this and see if increasing the cache helps.
• There was also a problem coming from makemask. Sandra should check if this problem also happens when running the pipeline in sequential.
• Sandra said that we are ready to include partition in the pipeline. Jeff said we should aim to have "partition" inside importasdm together with lazy mode. Sandra will create a ticket for this and initiate the discussion with Michel.
• Sandra asked what should be the output if two MSs are requested in importasdm wvr mode. The answer is, it should also be two MMSs.
• Sanjay checked in changes to tclean that used the MPIInterface. He and Justo have discussed and there seemed to be a problem with PyParallelContSynthesisImager::initializeImagers. To be verified.
• Justo run some scalability tests with 16, 32 ,64 servers, running partition. There was no strange behavior. The goal was to see that there is nothing strange.
• There was again a discussion about Justo running tests. Jeff said Justo should not run scalability tests, but rather James. James complaints that he needs to know what CASA tarball to use. The ESO team replied that we all use the trunk because the changes are checked in very often on a daily basis.
• Sandra proposed that Joe keep track of the relevant svn commits in the JIRA tickets associated with the HPC development and link them to the created tarballs. A twiki page could be created with a table containing this information that could be used by James and all developers.

Action Item List

• Mark to talk to Darrell about moving forward with mpicasa.
• Julian to send email to Jim about increasing the cache.
• Sandra to check if SEVERE message of makemask also happen in the sequential run of the pipeline.

Useful Links

• Major blockers
• Current status
• CASA parallelization task list
• Long-term issue chart
• Bleeding tests area
• Documentation (see Reference Materials topic)