CASA Coordination Team - 19 April 2018 10:00 ET/8:00 MT/16:00 CET

Rooms: ER-245, SOC-280 Hub:

Phone: 434-817-6524

  • Plan for integration of HPC in to operational ALMA
  • How to address requirements for restore and rerun in different HPC modes
  • BLOCKER: requirement for ALMA workflow: save flags with a given MS/MMS, and later restore flags to a different MS/MMS. In particular, we currently want to run the calibration pipeline and the imaging pipeline (which begins with a restore MS from ASDM and flagmanager::restore), with different arbitrary MS/MMS configurations CAS-11247Proposed solutions include:
    • run parallel imaging pipeline with MS only (is this possible?), and throw away the capability to do parallel calibration which requires MMS
    • store the MMS config in the flagversions comment and have hif_restoredata first create that MMS, apply the flags, and then convert to a regular MS (will this incur such a performance hit that it negates doing parallel calibration in the first place?)
    • store the MMS config in the flagersions comment and have hif_restoredata create that identical MMS and leave it that way (this is probably untenable since a. we don't a priori know at the start of calibration what kind of MMS to create to facilitate subsequent imaging, and b. the MMS has not bee sufficiently tested that if we ask users to start using it in many places in CASA we risk many bugs)
  • Options for restoredata workarounds (Lindsey)
    • ALMA calibration pipeline does not use HPC
      • No restore data issue, nothing needs to change in restore data
    • ALMA calibration pipeline uses HPC but not the MMS
      • hifa_importdata sets createmms to False, nothing needs to change in restore data
      • retains plotting and calibrator imaging parallelization, loses flagdata and applycal parallelization
    • ALMA calibration pipeline uses HPC and MMS * MMS usage captured by pipeline manifest, restore data restores using MMS option of parent pipeline run * Maximum parallelization retained * ALMA Imaging pipeline * restore data must restore with the MMS mode of the parent run * split out target data can be turned into an MMS if need be via usage of the createmms parameter in mstransform

-- RyanRaba - 2018-04-18
Topic revision: r3 - 2018-04-19, LindseyDavis
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback