Zpectrometer Data Flow Description

This document describes the current state of the Zpectrometer data flow in to IDL for additional processing and analysis. The filler (a special version of sdfits) and GBTIDL enhancements (in a special branch of gbtidl) are the focus of this note.

Details on how to invoke the two parts of this data flow are described elsewhere.

Filling the data

The raw FITS files produced by the managers are combined by a special version of sdfits. This version of sdfits was branched from the main sdfits development trunk in late September of 2006 (using the code base of sdfits version 1.6 [sparrow 6.6] as the starting point).

In addition to the capabilities of sdfits at the time of the branch, this version of sdfits can also process Zpectrometer raw data. The output in that case is not an sdfits file (see this work in progress for a description of the current content of the spectral line output of sdfits which follows the SDFITS convention for storing single dish spectra in a FITS binary table).

Instead, sdfits takes the raw Zpectrometer FITS file and copies that information, row by row, to a new output FITS file (which will, in general, contain other rows from other Zpectrometer FITS file so that one large FITS file is produced containing all of the scans to be filled). In the process of copying those fields some information not available when the scan's FITS file was written is filled in and some additional information is added. Although the result resembles the SDFITS convention, it is not fully compliant with SDFITS. In addition, an ASCII index file is created that is similar to what sdfits creates for the other spectral-line backends. This index file is used by GBTIDL and contains information useful in data selection operations in GBTIDL.

The specific details as to what the Zpectrometer version of sdfits does are described in the MR related to that work.

Note that none of these changes change the way sdfits processes data from other backends (DCR, ACS, SP).

Since that branch, there has been 1 release of sdfits. Read the sdfits version 1.7 release notes for a description of changes as of that version. In addition, there have been some accumulated changes since that release that will result in a new version of sdfits by roughly the end of May. None of these changes are relevant to the processing of Zpectrometer data described here. On the other hand, none of the Zpectrometer changes impact the behavior of sdfits with data from the other backends and so it should be possible to merge the zpectrometer-specific code into the main sparrow trunk without any adverse effects. That would result in an sdfits that produces (for the zpectrometer) non-SDFITS-convention data, which may be confusing to users.

In addition to the main output file, this version of sdfits also copies over some of the additional Zpectrometer data files as described in the previously mentioned MR.

Instead of merging the Zpectrometer code in to sdfits, it may make more sense to create a separate application who's only job is processing Zpectrometer raw FITS files as the current "bastard" sdfits does. This would be fairly easy to do and it would put the Zpectrometer code back into the main sparrow code trunk.

Using IDL to process the data.

The output of the Zpectrometer version of sdfits can be read in to GBTIDL using a special version as described in this MR.

These tools were also implemented in a special Zpectrometer-specific version of GBTIDL so that they remained isolated from the distributed version of GBTIDL.

Those additions are isolated to a few IDL scripts, primarily in the io layer of GBTIDL. The only other additions to GBTIDL were 3 files in the "contrib" directory that aid in converting the zpectrometer data container returned by the new i/o classes to a standard spectral-line data container and in displaying Zpectrometer data in the GBTIDL plotter. None of these changes impact the rest of GBTIDL and they could easily all (including the io classes) be moved into the contrib directory and made part of the main GBTIDL development trunk.

Since this code branch, there has been one release of GBTIDL (version 2.1) and one minor patch to that release (version 2.1.1). It is likely that there will be a new release (2.2) by the end of May.

Options for future development

  • Continue on the same code branches. Since the zpectrometer code doesn't impact the rest of the functionality of GBTIDL or sdfits this seems unnecessary. The shared code will continue to drift and that may lead to code maintenance overhead for little real gain.
  • Merge the code back in to the main branch.
    • sdfits
      • back in to the main sdfits program. The output isn't sdfits, so that may be confusing since it can't be used "as is" for input into the GBTIDL spectral line data i/o.
      • A separate application that just processes zpectrometer data. I think this makes the most sense. One can then imagine a pipeline that uses this and IDL to produce a final end-user product that is fully compatible with the SDFITS convention and standard GBTIDL data i/o. The sdfits program could even invoke such a thing for the Zpectrometer backend so that to the end-user that would be indistinguishable from how other backends are handled by sdfits.
    • gbtidl
      • in the contrib directory, distributed with GBTIDL
      • in a special zpectrometer directory with special zpectrometer start-up script to include that directory in the path. This gets the code in the main trunk where it's easier to maintain, it's clear what the zpectrometer-specific code is, we could easily not distribute that code. We could have another, separate pared down start-up script that would not include the plotter code so that it could be used in the pipeline as described above. The other code that has been developed to turn the zpectrometer data into spectral line data would then also be added to this same directory. Note that all of this should include unit tests so that we have some confidence that changes throughout GBTIDL do not break this code.

-- BobGarwood - 10 May 2007
Topic revision: r1 - 2007-05-10, BobGarwood
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback