KFPA Science and Pipeline Workshop (M. McCarty's Notes)

Monday, November 26th


Overview of the KFPA project

  • Program Personnel
  • Goals:
    • 7 Pixel array construction complete in 2 yrs.
    • Production use in 3 yrs.
  • Deliverables
    • Frontend - cryogenic package, modular downconverter, modular noise calibration, mechanical packaging.
    • software - package for engineering and M&C; package for data analysis.
    • cost per channel - reduce cost, know what it will cost to expand array.
  • Milestones
  • baseline instrument specs
    • freq - 18 - 26.5 GHz
    • Trx - < 25K (75% of band), < 36K (entire band)
    • 7 beams
    • dual, circular polarization
  • Focal Plane coverage
    • beam spacing - 3 HPBWs
    • beam efficiency of outermost elements issues.
    • compromises
  • key subsystems
  • Bandwidth Limitations
  • Future FPA Developments
    • Significantly enhanced spectrometer with this new array and existing if system.
    • New digital I.F. distribution system.
    • Expanded FPA.
    • Phased, expandable software development to match hardware capabilities.
  • IF System
    • Current analog IF system limits expansion.
    • Digitization at antenna with filter/compression schemes probable solution of IF transmission.
  • Backend strawman
    • 3 GHz analog bandwidth
  • Software
    • similar systems do exit at other telescopes.
    • challenge in software is to transfer
  • Open Issues
  • Comments from UMass
    • hexagonal array - "pluming issues" with bringing the IF in and out - modular and serviceable.

The GBT at K Band

  • Current Rx
  • Backends
    • DCR, Spectral Processor, Spectrometer (mostly used)
  • Observing Techinques
    • Doppler tracking, switching modes, observing types
  • Antenna
    • nearly perfect at K Band; aperture eff 65 - 58%, beam eff 89%79%
  • Atmospheric Limitations
    • Opacity, winds, day/night time (not an issue k-band)
  • Precipitable Water/Cloud stats
    • under 5 mm of year 25% of the time
  • Atmospheric Conditions - Opacity
    • 15 - 30 K system contributions due to weather during the winter.
  • Forecasting
    • 4mm accuracy in 48 hrs prediction
    • 5K accuracy in 48 hr Tsys predictions
  • Relative Effective Tsys
    • Tsys normalized using the best Tsys at that frequency
    • Used to determine what percentage of the time conditions are favorable for observing at a given frequency.
  • Winds cause a 5% loss of efficiency
  • RFI
    • 22.21 - 22.5 GHz: shared protected band
    • 17.7 - 20.2 Ghz

Science with the KFPA

  • Characteristics of the GBT
  • High frequency over subscription factor is on the rise.
  • 5 large GBT projects (> 200 hrs)
  • 1.5% sampling with a 7 pixel array.
  • K Band had the highest demand for mapping in 2007.

Chemistry with the KFPA

  • Sources of large interstellar molecules
    • SgrB2 and TMC-1 are the primary sources for the Remijan et al. chemistry survey.
  • Chemistry of molecules discovered with the GBT
    • New molecules have opened new questions regarding their formation.
    • The GBT is "sensitive" to detecting "large" (>5 atoms) molecules.
    • It is important to understand the distribution and excitation of new molecules.
  • VLA/BIMA mapping projects.
  • Interstellar molecule "myths".
  • Why a GBT spectral line mapping array?
    • Weak Line Intensities
    • Low State Temperatures
    • Low-Energy Level Transitions
    • Information about Cloud Density, Kinematics, and Structure
    • No Interferometric Arrays

Continuum polarimetry

  • Scientific Motivation
    • Galactic Astrophysics - Spinning dust emissions in the Galactic ISM.
    • CMB
    • Extragalactic Sources
      • Not much known about extragalactic sources at 20 GHz.
  • Mapping Speed
    • for 50 MHz bandwidth - ~1 hour for one-sigma to 1 mJy in 1 sq degree
    • for 1800 Mhz bandwidth ~ 10 minutes

High-redshift studies with the KFPA

Molecular clouds and star formation. I.

Molecular clouds and star formation. II.

Observing modes and array configuration

  • On-the-fly OTF observing
  • Why OTF?
    • efficiency in signal-tonoise
    • efficiency in telescope motion
    • better data consistency
  • OTF Sampling Issues
    • convolution functions; beam broadening/noise aliasing; sample at lambda / 2D; sampling interval only needs to be lambda/(d+D)
    • How much rope do you give to the observing?
  • OTF differential Doppler correction
    • spectra must be fully sampled in frequency so that the frequency axis can be resampled.
  • Observing Modes
    • Point-and-Shoot mapping
      • Frequency switching
    • Multi-beam nodding
    • OTF
  • OTF Scanning Patterns
    • Raster
    • Spiral
    • Hypocycloid
  • Array Feed configuration Issues
    • Array Rotation Capabilities?
      • Only necesary if point-and-shoot mapping long integration science a requirement.
      • Array rotator on NRAO 8-Beam receiver considered a design flaw (mechanically unreliable).
      • APEX
        • No rotator on bolometer arrays.
        • 490 GHz spectral line array does have rotator.
    • Why have a feed rotator?
      • Point-and-Shoot while rotating to track along paralactic angle.
      • Attempting to fine the edge of a region.
    • UMass prefers OTF over Point-and-shoot.
    • Proposal: Leave the rotator out for the prototype and consider it for the 61 element array.
  • Antenna Control Issues
    • Scanning efficiency
      • Minimize turnaround time; Multiple scanning patterns avaiable.
  • Monitor and Control Issues
    • Data dump rate
      • Selectable to balance data volume with sampling necessities.
      • Selectable scanning patterns in variety of coordinate frames.
      • Tagging total power samples with "observing intent" information ( ON, OFF, etc.).
        • JCMT: Tagging is a major issue because tagging is done in the corrilator. Its very hard to recover bad tags. Problems with tagging with a rotator.
      • Output to FITS format with (u,v) binary extension (formerly reffered to as "uvdata" format) for portability to any imaging analysis package.

  • Feed rotator
Con Pro
Expensive Full control of array on sky.
- Don't have to make a complete map.
Not required for some/most projects  
Can be added later  

Discussion on configuration

Discussion on calibration

  • Calibration Issues
    • Tcal - Noise diode?
      • Number of diodes? Arrangement?
        • 1 per feed, shared by both polarizations? DONE
        • 1 for the array, shared by all feeds and polarizations?
      • Intensity?
        • 1 dB dynamic range for linearity : Tcal / Tsys < 0.25; <= 10K
      • Feed, polarization, and frequency dependent
        • Varies by a few %, few MHz
        • Vector Tcal (freq, feed, pol)? -- wide bandwidth, multi-lines, continuum sources
      • Stabile - time scale?
      • Usage
        • Blinking? Duty cycle?
      • How to determine? How often to measure? Stability?
        • Lab Hot-Cold loads
        • On telescopes Hot-Cold loads
        • Astronomical - standard astronomical calibrators
  • Opacity
    • Frequency and time dependent
    • Tippings
      • Requires Nufss, Tatm, Tcal
      • Takes telescope time or ancillary radiometer
      • Measures...
      • Accuracy? 2-4 %
    • Forecasting
      • Accuracy? 2-4%
  • Efficiency
    • Elevation, frequency, feed dependent
    • Stable - Once modeled
    • Depends upon source size
      • Nua - Point sources
      • Nubeam - Extended to first null
      • Nufss - Very extended
      • Deconvolution problem?
    • How to measure with sufficient accuracy?
      • Beam shape is important.
      • A specification on the dynamic range is needed to determine beam shape.
  • Frequency Calibration
    • Doppler track a single window
    • Not needed with current Spectrometer - single window
  • How to apply vector tu, Nu, Tcal?
    • Use GBTIDL model (in development)?
      • Doppler track in software after the fact.
      • Current LO knows nothing about the antenna position.

Discussion on polarimetry

  • Calibration signal to go to both polarization to determine phase changes.
  • Calculate a Mu-Matrix for each element.
  • Off axis beams will have worse asymmetry than the center beams.
  • Cross polarization terms could get as high as 10%.
  • Proposal: Drop one of the polarizations to improve Tsys?
    • If the continuum performance would improve significantly, yes.
    • Will removing the OMT help Tsys at K-band?
    • VLBI experiments need dual polarization.
    • Lost polarimetry in Ka-Band.
    • The only science case for dual polarization is polarimetry.
    • Better frequency coverage without the OMT.
      • There is a bandwidth limitation that is not due to the OMT.
    • Polarization is needed to identify RFI.
      • Correlated signals across feeds could be used instead.
  • Will the prototype be thrown away?
    • Not a prototype, but phase A.
  • If dual polarization is dropped, could you have 16 elements?
    • No, more like 8 - 9 because current backend limitations.
    • More integrations per pixel.
    • 2 more samplers.
  • Polarization vs Number of feeds.

The KFPA and future GBT instrumentation

Tuesday, November 27th

Recap of previous day's science discussions

  • Frequency coverage should be pushed to its highest possible limit. 27.5 - 28 GHz
  • Dropping the feed rotator will compromise science (projects that require deep integrations where the array foot print has rotated on the sky). However, this might still be the way to go to avoid rotator complications for the prototype.
  • By dropping the OMT it might be possible to lose ~15K in Tsys.
  • Calibration and observing modes are well understood.

State of GBT data reduction software

  • M&C produces "raw" FITS files
    • Set of files for each scan
    • "Static" files, fixed at start of scan
    • "Dynamic" files, grow as scans progress.
  • Backends
    • Spectral line: Spectrometer, Spectral Processor, Zpectrometer
    • Continuum: DCR, Mustang
    • Pulsar backends
  • Real-time monitoring
    • GFM - The GBT FITS Monitor
  • Data capture
    • sdfits
  • Data Reduction and Analysis
    • GBTIDL
      • IDL scripts
      • Modeled on package of Tom Bania
      • Heavily influenced by UniPOPS and sips++ dish plotter
      • Primarily spectral line including raw zpectrometer data.
    • Aips++
      • Dish: Precursor to GBTIDL, has graphical flagging, statistical flagging, can us aips++ imaging tool
      • GBT continuum tools: Can be used with DCR data, Calibration and imaging
    • Aips: Imaging spectral line GBT data
    • OBIT - continuum imaging
    • Pulsar - Scott Ransom's package. Others.

Pipeline basics and VLA pipeline

  • Work on VLA pipeline to make a survey of the VLA archive.
  • What is a data pipeline?
    • One or more programs that perform a task with reduced user interaction.
  • Why use it?
    • Saves time: large (repetitive) data sets; Interactive data reduction may take a lot of time.
    • Consistency
    • Increased accessibility of a data reduction system.
  • Building a pipeline: start simple
    • Build a pipeline in layers.
    • Example:
      • allow the user to specify input parameters needed by the following tasks.
      • find the best default parameter values for most data sets.
    • Focus on a subset of input data.
    • The pipeline will evolve with time
    • Use metadata to initialize parameters.
  • Areas of concern
    • How much control should the user be given?
    • How many output diagnostics should the pipeline produce?
  • More on output
    • Consider outputting calibrated data and log files.
  • Comment: Standardized pipeline is needed at the observatory.
  • This set of heuristics that could be translated into a nicer environment i.e. Python with the AIPS API.


  • When imaging OTF data the convolution function is critical.
  • Continuum calibration additive terms are very difficult to estimate.
  • Determining the "zero level" is an important issue in continuum imaging.
  • Redundant measurements of the same part of the sky along different trajectories help separate sky from backgrounds.
  • Single Dish OTF in Obit
  • Single dish OTF image package that is currently experimental.
  • Obit offers a toolkit for scripting in Python.
  • OTF FITS data format
    • Multiple tables with all data for an observing session organized like a relational database.
  • Imaging of Mustang data using low pass filtering.


  • Lessons learned
    • OTF mapping is essential to acquisition of the highest quality data with focal plane arrays
    • Processing to final data cubes must be straightforward for users.
      • Relational database facilitates management of large quantities of data.
    • Constructing the data cubes and images is the first step to science goals.

Design and Implementation of the ACSIS Real-Time Data Reduction Pipeline

HARP Data Pipeline and LOFAR Real-Time Pipeline


  • Processing software needs to be: fast, parallel, floating point, project specific, automated, new techniques.
  • Software decisions
    • written in C/C++
    • optimized for x86_64, uses the AMCL (AMD Core math library) which incorporates BLAS, LAPACK and FFT routines
    • multi-threaded
    • automated process
  • Quick look plot consists of perl and html pages.

Pipeline Demos

Key Requirements

  • Data Products?
    • "raw" spectra - current output of sdfits
    • Minimally calibrated spectra Tcal * (on-off)/off
    • Gridded image cubes




Wednesday, November 28th

Garwood | Review of yesterday's progress and unresolved issues

Technical Hurdles

  • Parallel Computing Environments
    • ICE - Internet Communication Engine?
      • Python Interface
    • XML RPC
  • Handling large data sets.

Non-technical Hurdles

  • How much reduction does the pipeline do?
    • The minimal amount to reduction to archive. ???
    • The time series data.

The User Experience

Data Formats (throughout pipeline and final)


Data Visualization

Tilanus | Demo of GAIA. Other unique visualization tools could be demoed here if there is interest.

Where do we go from here?

Topic revision: r9 - 2007-11-28, MikeMcCarty
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback