KFPA Pipeline Overview

Pipelines

• Imperative for large focal plane arrays (high data rate).
• More than just a handy quick-look tool.
• Defaults should be sensible to avoid the need to reprocess the data.
  • Defaults may be context specific (vary with use case).
  • Reprocessing is painful.
• Not a data processing environment (i.e. not interactive like GBTIDL or AIPS).
• Likely to be useful in processing data from other receivers with more leisure data rates.
• Other telescopes routines provide roughly-calibrated data to their users - most institutions consider this the starting point of data reduction.

Goals of the Prototype Pipeline (KFPA Critical Design Review - Jan. 2009)

• Support KFPA commissioning
• Explore new processing tools/techniques not yet widely available in GB (vector calibration, statistical data flagging and editing, visualization, parallel processing).
• Prototype an automated pipeline - add necessary meta data to capture user intent
• Prototype tools necessary to support larger focal plane arrays (e.g. parallel computing)
• Based on prototype tools, estimate costs associated with delivering a pipeline and necessary computing hardware to handle the expected data rates for a larger focal plane array.

Out-of-scope items (deferred) (Critical Design Review)

• complicate calibration schemes (e.g. "basketweaving" or it's FPA equivalent)
• cross-correlation
• continuum

"The initial observing modes should be kept simple yet effective - even at the cost of reduced flexibility for observers." - GBT KFPA Critical Design Review Final Report.