You are here: NRAO Public Wiki>HPC Web>CasaParallelization>Jan1515 (2015-01-16, SandraCastro) Edit wiki text Edit Attach Print version
-- SandraCastro - 2015-01-13

Casa Parallelization meeting minutes

Thursday [15 January, 2015], [ESO Centaurus C.2.01], [9:00 MST]

  • Dial in number: +49 89 307 6832

Attendees:

  • ESO: Sandra and Justo
  • NRAO-Socorro: Kumar, Lindsey, James, Joe
  • NRAO-CV: Darrell, Mark, Andy

Minutes captured during the meeting and sent by email 

* ESO team presented a proposal of changes to do in partition for discussion.
1) A Baseline axis for partition. This improves the load balance among subMss. It avoids all problems related to sub-table re-indexing,
which are many. Avoids the problems related with the WVR spw.
* NRAO said this will not work as most calibration tasks require all baselines to work. This approach will slow down these tasks.

2) A smart auto-mode for partition. The current auto mode does not allow flexibility. We need a way to create subMSs using the scan/spw
axes, which avoid having the WVR spw alone in a subMS. It should also take into account the number of servers available. It should spread the science scans in several subMSs so that the work load during applycal is done by several servers. If possible, it should also try to balance the sizes of each subMS.

* There was a confusion about partition's parallelisation axis. NRAO said "time" while they meant scan. 
* SC explained that scan is an axis available in partition. Time is a different thing because it can go across scans. This is not available in partition. SC pointed out that they have asked at one of the previous HPC meetings, if a "time" axis would be used. Sanjay said not for imaging.

* We may need to re-discuss partition in our next meeting. ESO says that the available schema in partition do not work well! Jeff should be present in the next meeting.

* KG: Task partitioning should have baseline partitioning as an(other) option.
* ESO answered that this is what they proposed.

* Several people at NRAO stressed the fact that we need to use more than one type of dataset for tests. 
* ESO responded that they have requested the Build and Test team to propose at least two casa-guides (one EVLA, one ALMA) to be parallelized and
used for automatic tests. CAS-7189.

* DS says that the MPI rpms are already available for ANY developer to test and use them, in the yum repository.

* Julian made some changes in cmake so that building with MPI is already possible for those who install the MPI packages.

* KG asked if we already have a parallel version of the pipeline.
* ESO responded that this is what they are working on at the moment. SC points out that she has not been able to run the pipeline through the end with Multi-MS yet due to the several problems she found. ESO points out that these problems are all somehow related to the partition schema.

* JG says that now the imaging team could/should start using the MPI framework for their tests.

* SC asked Andy/Mark/Darrell if they can run the test_mpi4py tests in Jenkins using mpirun. DS's answer was unrelated.

* DS asked if test_setjy works with MMS. SC said not anymore because the task is broken for MMS.

* JR points out that the most important point now is to have an end-to-end test, not validation. SC responded again that she has not been able to run the pipeline until the end. 

* SC said that we need to keep what we already have working in a healthy state. We noticed that things we parallelized 2 years ago, such as setjy do not work anymore because nobody tests with MMS and there are no automatic tests on MMS. That is why it is critical that we include some tests running automatically in Jenkins.

* JG says somebody else should test the RPMs, apart from us. It is easy because the RPMs are already in the repository. Everything is documented and in the repository.

* LD asks if the MPI packages can be available in the release branch. JG says no, only in the trunk for the moment.

* JR said everyone may have some flawed assumptions because of the breadth of testing.  Half of us were talking from a point of speculation because we can't test, which leads to the next point.

* JR does not recommend verification of bottlenecks first

* ESO said they are/were not verifying bottlenecks, but hit them during the development.

* JR said we shouldn't do correctness verification yet, or more accurately we shouldn't be overly concerned with correctness.  First step is run to completion, then bottlenecks then correctness.

* JG says there are various ways of testing. 1) using the existing unit tests to verify that the MPI framework works as expected, that the job queuing mechanism works, etc. This is a thin layer of tests that must go to Jenkins. We are not doing any validation or optimisation tests. We are trying to choose the most appropriate partition schema (axis, numsubms) so that we can test that every CASA task runs to completion. The boundaries between the subMSs are causing the problems.

* KG: The Wiki page you spoke of was incomplete.  Please add JIRA tickets.  
* ESO answers: All the major blockers for the HPC work are listed in yesterday's Wiki page, in item 3, listed per priority. You can also go to JIRA and choose the HPC component to see all tickets.

* KG: Can you provide a list of “bugs” that are stopping you?  Recommendations are that you fix these bugs first.
* See previous answer. The ESO team cannot fix all the problems related to parallelisation. This has to be a CASA effort. We have been re-assigning the tickets but people are not working on them.

* KG: SetJy be placed as a high priority (critical or major)
* SC: Will do it, as well as with all other problems raised in other tasks. 

* KG doesn't understand why the ALMA pipeline is using setjy with scratch=True.

* SC says if both options are there, we have to support both.

* JG setjy is not the main problem. There are other problems such as the ones in wvrgcal.

* KG said these things should be placed in tickets as bugs.

* SC said we have created tickets but people do not work on them.

* JG said again that many of those problems are related to the partition schema, that's why we are trying to come up with a better approach.

* We decided to have weekly meetings every Thursday at 9:00 am MST. SC to send email to JD about the next meetings.

Discussion

New Partition Schema to discuss

  • Baseline separation axis
    • In principle, this will solve all the problems we have/had with consolidating the sub-tables after the MMS is partitioned per spw or scan,spw. It will also allow a better load balance in the subMSs.
      • Load balancing is guaranteed regardless of scan/spw selection (quite common)
      • Since load balancing is achieved w/o using a large number of subMSs we avoid problems associated with using a large number of file descriptors
      • There is no need to consolidate spw/ddi/polId sub-tables
      • WVR SPW is always together with other SPWs, therefore there is no problem with empty FEED sub-table
      • PROBLEM: When the MMS is iterated as a monolithic MS it is required to read data from all sub-MSs for each timestamp.
  • List of number of scans, spws for each sub-MS
    • Example: numsubms=[3,2], meaning each sub-MS will contain 3 scans and 2 spws. This will allow us to better balance the load in each sub-MS and also to increasing the partitioning for spws without running into problems of having the WVR spw 0 alone in one subMS.
    • We could introduce a smarter "auto" mode in partition which will calculate the optimal partition schema using the scan/spw axes with the number of sub-MSs according to the number of available servers.

1. Testing Efforts

  • ALMA Pipeline Testing
    • After the problems with plotms and importasdm were solved, I run another test, this time creating an MMS with 19 sub-MSs, partitioned in the scan axis. The process still fails in setjy. Issues to be solved in setjy. All these issues are in progress:
      • Create the MODEL column at open time for MMS cases only. Probably right before calling ParallelTaskHelper
      • Change the way the task catches NULL MS selections.
      • Verify the return values of the task. It used to be a Bool, now it's a dictionary. It seems that the dictionaries of each sub-MS do not need to be consolidated. This requires some work at the task level, inside the parallel section.
      • Run test_setjy with MMSs.
  • Test the pipeline with lazy filler option in importasdm
    • How to make modifications to the way the pipeline calls the CASA tasks? I need to add the lazy=True parameter to the importasdm call for this test.
  • Imaging tests. I have not been able to get the test scripts from Sanjay.
    • Create a spectral version of the script used by Sanjay.
    • Port the parallel_go version of Sanjay's script to use the MPI-based framework
  • Documentation on how to test using the MPI parallelization framework, MMS and tclean
    • Consolidate all the documentation into a single simple tutorial. This document should contain what is needed to test any task with MMS.

2. Build, Test and MPI integration

  • Integration of relevant 3rd party packages into casa repos.
    • Status:
      • rpms generated by Julian at ESO were tested by Justo at almahpc02 - no major problems found all mpi4casa unit tests passed successfully
      • Julian sent rpm specification to Darrell in order to generate the rpms in the official environment - they were generated successfully but bringing a dependency on numactl-dev which in principle is not needed (thought is harmless)
  • cmake script changes to respect necessary compiler flags.
    • Status:
      • Julian sent also a svn diff of code/CMakeLists including the mpi checks to be tested by Darrell - (status of testing still to be reported)
        • this is more properly a test to be performed by those developers who will be using the HPC framework (says Darrell)
      • swig flag -thread param was not included in the svn diff to be tested
  • addition of alma m100 (or another script) regression to test suite.
    • CAS-7189: Use one ALMA, one EVLA casa-guides examples with MMS for the tests.
  • addition of an MMS test in the automatic tests (maybe at ESO if we can?)
    • not yet done.

3. Major Blockers

4. Plan for next 6-12 months

  • 3c147 script also uses parallel_go. Justo and Sandra will work with James in order to port it to use MPI. At the long run, we should find a better dataset for this use-case.
  • CAS-7189: We should aim to have some casa-guides scripts running on MMSs in our tests. One or some of them could be good candidates to be included in automated tests. Sandra will look into this first. Justo will try later after he fixes some pending tickets in mstransform.
  • We should look at the Sanjay's imaging script and see how to make an MPI version of it.
  • Sandra and Justo will work with Stewart on the parallelization issues of the pipeline.
  • In progress: Julian will work with the build team in CV on the MPI distribution in CASA.
  • James will test the tileshape parameter in mstransform.
  • Done: Kumar will verify/test the fix to CAS-7016.
    • Kumar verified that this is not fixed. It's assigned back to Jim.
  • Tak will look at the cube side of imaging in place of Urvashi.
  • Sandra will use Justo's document to create a simple document guide on how to build CASA with the necessary libraries to run MPI in the cluster in Socorro. The official documentation should have no cluster specific information. For the moment we need a quick-guide to run in Socorro.
  • Done: Sandra will talk to Tak about running setjy on MMSs.
  • Done: Justo will compile a list of things to do in order to distribute MPI on CASA. For today’s 16:00 meeting.

Deferred Items

  • Parallelization of plotms will come later.
Edit  | Attach | Print version | History: r9 < r8 < r7 < r6 | Backlinks | View wiki text | Edit wiki text | More topic actions
Topic revision: r9 - 2015-01-16, SandraCastro
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback