CASA Parallelization Meeting Minutes

Thursday November 6th, DSOC 317, 8:00AM MT

• 192.33.117.12##8108
• 434-817-6523

Attendees:

• Socorro: Rob, James, Sanjay, Lindsey
• Charlottesville: N/A
• Garching: Sandra,

• Apologies:
  • Justo is sick at home.

Discussion

• Sandra taking CASA Parallelization sub-project lead, effective 11/20/14.
  • Rob will help setup working meeting for week of Dec 1st.

• HPC working meeting - Dec 1 - Dec 5 @ DSOC
  • Reserved room 248 for week of 12/1. (3 desks)
  • Do we need workstations? (No)
    • Will bring laptops, but cannot run CASA.
  • Topics for the week?
    • Previously limited on testing environment. Do need some documentation for testers, how to set things up and how to test.
    • (1) Sanjay to put together a regression that others can run. Use to verify results. (2) James will modify to use openmip.
      • Can compare ipython and openmpi frameworks, along with other changes to the code base.
      • (3) Eventually setup a couple of VLA and ALMA data sets (golden data sets) for regression runs. Lindsey has proposed a couple of pipeline data sets. Sandra has a candidate too from ALMA Cycle II.
      • Long term, need to start validating based on pipeline performance. But these regression scripts are a good solution while development is proceeding on both CASA HPC and Pipeline.
      • Migration can be gradual. Use pipeline for calibration and flagging verification, but keep regression scripts for imaging end-to-end verification.
      • Desirable if the data sets have been used for serial testing, but more important that they are interesting on a 5year period.
        • Wideband and widefield imaging will have to be considered.
    • Sandra notes that split should be exercised since it is heavily used.
    • 3C147 test can be updated by James.
    • Testing of MSTransform MMS features
    • OUTCOME OF MEETING: Plan (for 6-12 months) testing platform.

• From 4.3 release notes:
  • "plotms cannot plot asynchronously. If a plot is issued while a previous one is still ongoing, plotms will crash and CASA needs to be restarted. Please wait for each plotms job to be finished before issuing a new one. "
    • Not an issue for scripted use cases. Only applicable to interactive use with the GUI. When run w/o gui, execution is synchronous.
    • ~2:1 or 3:1 ratio for i/o:plotting per plot. Data loading is the majority of plotting time.
    • Should test scripted case of multiple processes running plotms on an MMS.

• Priority tasks for parallelization effort in 4.4 cycle.
  • Should be updated after December working meeting. Need testing effort to identify areas for improvement.
  • High Priority:
    • https://bugs.nrao.edu/browse/CAS-7019: Integration of OpenMPI framework into regular package production
    • https://bugs.nrao.edu/browse/CAS-4860: Multi MS opens all the MS when accessed (file binding to reduce quantity of open files?)
    • https://bugs.nrao.edu/browse/CAS-7016: Option to open main table in write mode and sub-tables in read mode so they can be shared
  • Medium
    • https://bugs.nrao.edu/browse/CAS-7018:Update openmpi / mpi4casa implementation to respect passed environmental variables
    • https://bugs.nrao.edu/browse/CAS-6705: Introduce MPI to communicate and synchronize logging directly at C++ level
  • Low

• Development & Testing - Status updates
  • First time I run successfully a cycle 2 data reduction script using MMS, mstransform and mpi. The script performs the following steps: importasdm, flagging, wvrgcal, generation of Tsys and ant tables, applycal, split science spws out, plotms, flagmanager, bandpass and gaincal calibrations, applycal and a last split of the corrected column. I run the script without using the analysisUtilies packages which many times leave open tables in the cache which interfere with the parallel servers. These scripts always run in sequential.
  • We think there is an issue that happens randomly. There is a method in BaseTable that doesn't flush the sub-table and when mstransform tries to write a big sub-table such as the POINTING table, another subsequent call fails because the sub-table is not yet flushed in disk and fails. We need to run more tests to be sure about this.
  • James will broaden out his HPC testing to include various file systems. May expose issues.

• Next Meeting
  • Week of 12/01/14

Deferred Items for a future meeting

• • Multiple xterms
  • Schedule for first OpenMPI enabled test/stable package.

• Logger
  • Single log file & concerns regarding ordering of entries.
  • Pipeline think of log as a data product. Single file.
  • No synchronization at c++ level amongst loggers, but no indication of log overlap in current implementation. Can happen with intense logging activity.
  • Improvements may not be urgent, revisit use cases/requirements when Sandra is getting ready to start work.

• Imaging
  • Discuss requirement imposed on VI/VB2 to pass required information for MMS processing.

• MSTransform
  • Discuss time separation axis for MMS creation.

Action Item List

Closed Item Record