Combines the information from the various M&C FITS files
into a data stream that the pipeline can be used.
At the moment, the online sdfits filler does this task and the pipeline starts from the raw FITS binary tables that it produces. See here
for a discussion of the details on how the information in the FITS files is used to produce the sdfits files.
The following features are known to be missing or inadequate in sdfits for use by the KFPA pipeline
- Antenna beam positions are not available directly.
- The pointing positions in SDFITS for each row are found in the CRVAL2 and CRVAL3 columns (single-element axes 2 and 3, CTYPE2 and CTYPE3 describe the type of coordinate system). Currently, those values are the MAJOR and MINOR position averages over that integration as reported in the Antenna FITS file. They do not include the beam offsets nor do they include any offsets that may be present because of motions of the secondary during the course of the scan. The beam offsets, in elevation and cross-elevation coordinates, are available in the SDFITS file in the BEAMEOFF and BEAMXOFF columns. These are found in the beam offset table of the Antenna file. If the subreflector is moving following the "SubNod" motion, then the state of the subreflector (on source, moving, and offsource) is recorded in the SUBREF_STATE column but the actual changes to the pointing direction caused by the subreflector motion are not recorded in the SDFITS file.
- MR 3C608 describes the effort to modify sdfits so that the output SDFITS file's CRVAL2 and CRVAL3 will include the beam offsets. This is useful for the pipeline and also has been required by GBT observers who wish to use the existing 2-beam systems in a way where they need to know where each beam is pointed at all times. It's also more in keeping with the intent of SDFITS to record the true pointing direction in a standard column. Incorporating subreflector motion into those values is deferred until it is needed.
- vector Tcal values. sdfits currently records a scalar Tcal value which is an average of the Tcal values found in the calibration (receiver) FITS file. The values are first interpolated onto the observed frequency axis and then an average is taken over the inner 80% of all channels.
- sdfits won't write these to the sdfits file. The sdfits convention doesn't easily support it and if included it would double the size of the sdfits file. Instead, sufficient information will be passed to the calibration module in the pipeline so that that module can supply the vector Tcal values when requested by the other components of the pipeline.
- Missing zenith opacities. Sdfits doesn't attempt to write out any zenith opacity. Within GBTIDL, users can supply a single, scalar value to the calibration routines which is then used throughout that scan for the indicated polarization, feed, and spectral window. If not supplied, GBTIDL simply assumes that the zenith opacity is 0.1. Reasonable estimates of the zenith opacity are now available through Ron's weather-based predictions.
- sdfits will be modified to produce a scalar zenith opacity appropriate for the center of that spectral window. Sufficient information will also be passed to the calibration module in the pipeline so that that module can supply an appropriate vector zenith opacity across the bandpass when that is requested by the other components of the pipeline. An MR for the scalar zenith opacity work needs to be written.
- Filling while data is taking place. Currently sdfits waits until a scan is complete before it attempts to write out the sdfits file. For long scans involving high data rates, filling the scan can take as long, or longer, than it took to take the data. It is unacceptable that a 40 minute scan might require an additional 40 minutes before the user (or a pipeline) could interact with the data. Work is underway to modify sdfits so that it will fill data as it arrives on disk thereby making it appear that sdfits is running faster because the data are available sooner. In reality, because of the overhead involved in the current implementation of sdfits, it actually takes more CPU cycles to fill the data in that mode as compared to waiting until all the data is present.
- Filling separate groups of samplers in separate threads. The KFPA pipeline will be parallelized by splitting the data processing up by sampler or related groups of samplers. The current spectrometer backend creates up to 4 FITS files, one for each bank. Each bank contains one or more samplers (beams + polarization + spectral window).