SRDP Foundational Documents

This is a collection of documents and references to the SRDP project prior to instantiation of the project itself. Collected here for convenience and (after the project charter is accepted) historical perspective.

  • From the NRAO Long Range Plan (FY2018-FY2022):
    • From Section 3.2 Scientific User Support (p. 25):
Science Ready Data Products
In collaboration with DMS, SSR will oversee the development and implementation of SRDPs, an initiative designed to remove a significant amount of burden and overhead from the user that is associated with data calibration and imaging, allowing NRAO users to focus much less on data reduction and much more on the cutting edge science enabled by the VLA and ALMA. As such, SRDPs are an important means of expanding the NRAO user base.

While much progress has been made in automatic (pipeline) data processing for both ALMA and VLA a great deal of work remains. For the VLA, pipeline processing of continuum observations in Stokes I are routinely produced. Heuristics for continuum polarization calibration will be in place in FY2017 with ongoing heuristic development needed for spectroscopy, weak calibrator sources, and more complex observational setups. For ALMA science operations has transitioned from a “human pipeline” that provided PIs with reduced and imaged data to automated calibration and imaging. Ultimately, however, the challenge to producing SRDPs will very much depend on user intent to inform the automatic data processing and a robust quality assurance (QA) protocols.

During this LRP timeframe, the SRDP project will progress through standard project management/systems engineering processes such as CoDR, PDR, CDR, and PRR for an implementation phase starting in 2020.
    • From Section 7.1 Data Management & Software Department (p. 63)
Science Ready Data Products
The NRAO will continue the move from delivering raw observational (UV plane) data to PIs, to providing science-quality images for PIs. This has two principal advantages:
  1. It makes ALMA and the VLA accessible to non-radio astronomers (both as PIs and from archive data re-use by others).
  2. Even for radio-astronomers fully conversant in data calibration, flagging, and imaging, it should make them more efficient users of telescope time (from observation to publication), effectively shifting effort from time consuming data reduction activities to scientific analyses.
We call this curated observing because the Observatory takes responsibility for the production of, and quality of, derived products in the archive. To fulfill this vision NRAO will:
  1. Modify the VLA PST and OPT to allow non-experts to prepare proposals and scheduling blocks mostly using scientific parameters (sky regions, sensitivity, resolution, frequency), and minimizing the use of instrumental (sidebands) and data reduction parameters (weighting functions).
  2. While the ALMA equivalent to the PST and OPT (OT), Phase I & II) already has this as a design goal, and is the responsibility of the European Executive, NRAO will work with our scientific community to request changes where necessary to ensure that this vision is fulfilled.
  3. Ensure that the ALMA and VLA pipelines produce well-characterized images, normally sufficient for scientific use and publication, for all common observing modes.
  4. Whenever the flagging, calibration, or imaging algorithms (in CASA) or heuristics (in the pipeline) change significantly, improved products will be made available, either upon request or via reprocessing all archive data.
In addition to producing well-characterized images, the NRAO in conjunction with our community stakeholders will work to produce higher-level data products derived from the images. The deliverables of the ADMIT ALMA development project and the VLASS enhanced data products (produced by the community) are examples that are being pursued.

We recognize that there will always be cases where unusual science goals or technical requirements require human-directed data reduction. We also understand that some radio astronomy experts will want to verify the quality of the automatic reductions versus what they can do “by hand,” perhaps in a non-NRAO supported software package. This requires that:
  1. It should be possible for users to re-execute the pipeline, tweaking parameters, when the standard pipeline is “almost” good enough.
  2. For users who want to reduce their data with the NRAO pipelines or CASA, NRAO will provide an implementation ported to commercial cloud and national HPC infrastructure, via Extreme Science and Engineering Discovery Environment (XSEDE), as well as in-house cluster facilities, see External Data Processing below.
  3. For users who want to process data using software packages other than CASA, or have very unusual processing requirements, the Observatory will continue to provide the raw data in SDM/BDF format. Additionally, CASA will continue to provide limited support for translation into UVFITS and FITS-IDI formats.

-- JeffKern - 2017-03-24

Topic revision: r2 - 2017-03-29, JeffKern
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback