Casa HPC/Parallelization meeting agenda/minutes

Thursday [09 July 2015], [ESO Centaurus, C.2.01], [15:00 UTC]

How to connect

• Dial in: +49 89 6834
• Video connection: Use this one (46103@134.171.42.27)

Attendees:

Agenda

• Action Items from meeting in June in Socorro
• tclean locking issue when writing in the SOURCE table. What is the action on this? SEE CAS-7562
• tclean fails in the ALMA pipeline when run on MMS, but not on MS. This used the new version of hif_importdata with createmms=True.

2015-07-07 00:39:42    SEVERE    plotms::pipeline.hif.tasks.tclean.cleanbase::@almahpc03::MPIServer-8    J1147-3812/PHASE/spw17 clean error: 'tclean'
2015-07-07 00:39:42    SEVERE    plotms::MPICommandServer::command_request_handler_service::@almahpc03::MPIServer-8    Exception executing command request via eval: Traceback (most recent call last):
2015-07-07 00:39:42    SEVERE    plotms::MPICommandServer::command_request_handler_service::@almahpc03::MPIServer-8+      File "/data/scastro/casa/trunk/linux_64b/python/2.7/mpi4casa/MPICommandServer.py", line 164, in __command_request_handler_service
2015-07-07 00:39:42    SEVERE    plotms::MPICommandServer::command_request_handler_service::@almahpc03::MPIServer-8+        command_response['ret'] = eval(command_request['command'])
2015-07-07 00:39:42    SEVERE    plotms::MPICommandServer::command_request_handler_service::@almahpc03::MPIServer-8+      File "<string>", line 1, in <module>
2015-07-07 00:39:42    SEVERE    plotms::MPICommandServer::command_request_handler_service::@almahpc03::MPIServer-8+      File "/data/scastro/casa/trunk/linux_64b/../pipeline/pipeline/infrastructure/mpihelpers.py", line 178, in mpiexec
2015-07-07 00:39:42    SEVERE    plotms::MPICommandServer::command_request_handler_service::@almahpc03::MPIServer-8+        return executable.execute(dry_run=False)
2015-07-07 00:39:42    SEVERE    plotms::MPICommandServer::command_request_handler_service::@almahpc03::MPIServer-8+      File "/data/scastro/casa/trunk/linux_64b/../pipeline/pipeline/infrastructure/basetask.py", line 46, in attach_timestamp_to_results
2015-07-07 00:39:42    SEVERE    plotms::MPICommandServer::command_request_handler_service::@almahpc03::MPIServer-8+        result = method(self, *args, **kw)
2015-07-07 00:39:42    SEVERE    plotms::MPICommandServer::command_request_handler_service::@almahpc03::MPIServer-8+      File "/data/scastro/casa/trunk/linux_64b/../pipeline/pipeline/infrastructure/basetask.py", line 89, in capture
2015-07-07 00:39:42    SEVERE    plotms::MPICommandServer::command_request_handler_service::@almahpc03::MPIServer-8+        result = method(self, *args, **kw)
2015-07-07 00:39:42    SEVERE    plotms::MPICommandServer::command_request_handler_service::@almahpc03::MPIServer-8+      File "/data/scastro/casa/trunk/linux_64b/../pipeline/pipeline/infrastructure/basetask.py", line 66, in finalise_pipeline_result
2015-07-07 00:39:42    SEVERE    plotms::MPICommandServer::command_request_handler_service::@almahpc03::MPIServer-8+        result = method(self, *args, **kw)
2015-07-07 00:39:42    SEVERE    plotms::MPICommandServer::command_request_handler_service::@almahpc03::MPIServer-8+      File "/data/scastro/casa/trunk/linux_64b/../pipeline/pipeline/infrastructure/basetask.py", line 1056, in execute
2015-07-07 00:39:42    SEVERE    plotms::MPICommandServer::command_request_handler_service::@almahpc03::MPIServer-8+        result = self.prepare(**prepare_parameters)
2015-07-07 00:39:42    SEVERE    plotms::MPICommandServer::command_request_handler_service::@almahpc03::MPIServer-8+      File "/data/scastro/casa/trunk/linux_64b/../pipeline/pipeline/hif/tasks/tclean/tclean.py", line 124, in prepare
2015-07-07 00:39:42    SEVERE    plotms::MPICommandServer::command_request_handler_service::@almahpc03::MPIServer-8+        sequence_manager=sequence_manager, result=result)
2015-07-07 00:39:42    SEVERE    plotms::MPICommandServer::command_request_handler_service::@almahpc03::MPIServer-8+      File "/data/scastro/casa/trunk/linux_64b/../pipeline/pipeline/hif/tasks/tclean/tclean.py", line 153, in _do_iterative_imaging
2015-07-07 00:39:42    SEVERE    plotms::MPICommandServer::command_request_handler_service::@almahpc03::MPIServer-8+        flux=result.flux, cleanmask=None, threshold=None)
2015-07-07 00:39:42    SEVERE    plotms::MPICommandServer::command_request_handler_service::@almahpc03::MPIServer-8+      File "/data/scastro/casa/trunk/linux_64b/../pipeline/pipeline/hif/tasks/tclean/basecleansequence.py", line 40, in iteration_result
2015-07-07 00:39:42    SEVERE    plotms::MPICommandServer::command_request_handler_service::@almahpc03::MPIServer-8+        self.sidelobe_ratio = cbheuristic.psf_sidelobe_ratio(psf=psf)
2015-07-07 00:39:42    SEVERE    plotms::MPICommandServer::command_request_handler_service::@almahpc03::MPIServer-8+      File "/data/scastro/casa/trunk/linux_64b/../pipeline/pipeline/hif/heuristics/cleanbox.py", line 477, in psf_sidelobe_ratio
2015-07-07 00:39:42    SEVERE    plotms::MPICommandServer::command_request_handler_service::@almahpc03::MPIServer-8+        with casatools.ImageReader(psf) as image:
2015-07-07 00:39:42    SEVERE    plotms::MPICommandServer::command_request_handler_service::@almahpc03::MPIServer-8+      File "/usr/lib64/casa/01/lib/python2.7/contextlib.py", line 17, in __enter__
2015-07-07 00:39:42    SEVERE    plotms::MPICommandServer::command_request_handler_service::@almahpc03::MPIServer-8+        return self.gen.next()
2015-07-07 00:39:42    SEVERE    plotms::MPICommandServer::command_request_handler_service::@almahpc03::MPIServer-8+      File "/data/scastro/casa/trunk/linux_64b/../pipeline/pipeline/infrastructure/casatools.py", line 69, in f
2015-07-07 00:39:42    SEVERE    plotms::MPICommandServer::command_request_handler_service::@almahpc03::MPIServer-8+        tool.open(filename, **kwargs)
2015-07-07 00:39:42    SEVERE    plotms::MPICommandServer::command_request_handler_service::@almahpc03::MPIServer-8+      File "/data/scastro/casa/trunk/linux_64b/python/2.7/__casac__/image.py", line 5280, in open
2015-07-07 00:39:42    SEVERE    plotms::MPICommandServer::command_request_handler_service::@almahpc03::MPIServer-8+        return _image.image_open(self, *args, **kwargs)
2015-07-07 00:39:42    SEVERE    plotms::MPICommandServer::command_request_handler_service::@almahpc03::MPIServer-8+    TypeError: argument infile must be a string

Minutes

• With Memory Resident enabled in .casarc, James saw a 20% increase on the number of reads, while Julian saw a slight decrease when running in a local disk. To be verified. (CAS-7705)
• Jim is verifying the excessive calls to fsync in CASA. He is removing them and running tests to verify if anything breaks. On going. (CAS-7191)
• James switched to tclean in the pipeline runs of the nightly tests. The pipeline fails in the import step with casa-stable 4.5.54 and 4.5.55. It runs fine with casa-stable 4.5.52. He still sees that tclean is slower on MMS compared to MS. At the same time, Sandra said that running the ALMA pipeline (trunk of pipeline vs. trunk of CASA) fails in the tclean step. There is a missing _pg line in mpi4casa/task_wrappers.py, but even after fixing this, tclean still fails. Stewart will have a look at the problem.
• In the same topic, Justo run some tests on a small VLA MS, but didn't find any significant difference on the performance of tclean with MS and MMS. The small MS may not be a good test data set for this issue.
• We decided to have stable runs of tclean in the pipelines first, before debugging any possible performance issues of tclean + MMS.
• Sandra reported that she fixed a bug in applycal when creating the .flagversions for an MMS.
• Sandra asked if lazy=True is used by default in the pipelines. Lindsey says this is still being reviewed by the pipeline working group.
• Justo implemented a new view in the summary mode of flagdata, which will potentially improve the run-time of flagdata in the EVLA pipeline for mosaic observations. The implementation is explained in CAS-7489 and should be tested by both pipelines. Sandra asked Lindsey to verify if the new dictionary structure is ok for the pipelines.
• James brought up the issue that we need to log our performance tests. We need some way to state that a specific version caused a performance impact. He suggested to use a Spreadsheet for the moment.
• Next HPC telecon will be on July 23rd, but if anyone sees the need to meet, we can have unscheduled meetings. Please, send Sandra an email and she will organize the video telecon.

Action Items List

• Sandra will check if applying a .flagversions created from an MMS also works on a MS.
• Julian should remove the parts of the MPI library that do not work properly in the CASA binaries (Example: mpicc, etc.)
• Julian/Jim will look into the problem of the many reads of the FEED table.
• Jeff will follow-up on the problem with too many fsync calls in CASA (CAS-7191)
• James will run pipeline tests using different separation axis in partition (scan and spw), instead of 'auto' (balanced mode).
• Kana will organize a video telecon with the ESO team. First telecon should be on the 6th of July.
• Sandra will resume the organization of the HPC telecons, bi-weekly, starting the 9th of July.
• Sandra will create a JIRA container to log the results of James tests. CAS-7716
• James should log the main findings of his performance tests on the new JIRA container.
• Sandra will verify the failures in test_partition and test_fixvis after the memory resident is turned on.
• Justo will verify why calibration imaging using tclean in the pipeline is slower when running on an MMS.

Useful Links

• Major blockers
• Current status
• CASA parallelization task list
• Long-term issue chart
• Bleeding tests area
• Documentation (see Reference Materials topic)

Action Items