Single Dish Calibration/Imaging Package Effort

CASA Modification Request 28C108, November 2007



1. Introduction

A single-dish ("total power") package is required in order to support the commissioning and operations of ALMA. This capability is being provided in CASA by the inclusion of an external package called http://www.atnf.csiro.au/computing/software/asap/ ASAP (ATNF Spectral Line Analysis Package) developed by the Australia Telescope National Facility. http://www.atnf.csiro.au/computing/software/asap/ ASAP continues to be developed by NRAO in order to improve its integration with the rest of CASA and to add additional capabilities for the processing of ALMA and GBT data.

2. Background

http://www.atnf.csiro.au/computing/software/asap/ ASAP is a software package developed by ATNF to reduce single-dish spectral data from ATNF telescopes such as the Mopra, Parkes, and Tidbinbilla telescopes. Available for Linux/UNIX and OS X, http://www.atnf.csiro.au/computing/software/asap/ ASAP was developed in C++ using several CASA libraries, and, like CASA, employs Python for scripting and the user interface. http://www.atnf.csiro.au/computing/software/asap/ ASAP can handle data in Single-Dish FITS (SDFITS) and in Measurement Sets (MS). For all these reasons, it is a relatively easy package to integrate into CASA. The current stable version is http://www.atnf.csiro.au/computing/software/asap/ ASAP 2.2, although the version incorporated into CASA is presently version 2.1.

http://www.atnf.csiro.au/computing/software/asap/ ASAP takes an object-oriented approach in which objects have associated functions which can operate on data. A hierarchical collection of object tools are used to operate on data through a command-line syntax. In order to provide an interface consistent with that used for synthesis data processing in CASA, much effort is going into the development of single-dish tasks using the individual tools. Internal testing has confirmed that this is the right approach.

During the March 2007 internal CASA test (http://projectoffice.aips2.nrao.edu/ALMA2007.03/ALMA2007.03.html Test5), the following five single-dish tasks were tested:
      - sdcal.py to select, calibrate, average, smooth, and baseline data;
      - sdfit.py for line fitting analysis;
      - sdlist.py to print a summary of a single-dish dataset;
      - sdplot.py to visualize spectra, including overlay of line catalogue data;
      - sdstat.py to compute spectral statistics.
Some enhancements were made for calibrating position-switched data, frequency-switched data, and nod data from the GBT. During http://projectoffice.aips2.nrao.edu/ALMA2007.03/ALMA2007.03.html Test5, no software tools were available to calibrate ALMA nutator data. The recommendations of the testers were that priority for single-dish support be given to improve the robustness of the software and to continue packaging the single-dish tools into tasks. It was also suggested that some omnibus tools (such as sdcal.py) could be broken up into a series of smaller tasks in order to make them less unwieldy to use and to provide a finer-grained control by the user of the data reduction process.

3. Requirements

      - sdaverage.py for data selection, calibration and averaging; 
      - sdsmooth.py for smoothing;
      - sdbaseline.py for baseline fitting;
      - sdcoadd.py for coadding multiple existing spectra;
      - sdflag.py to flag bad channels;
      - sdscale.py for scaling (to multiplying a fixed number) to the spectrum;
      - sdsave.py to save the resultant spectra in various formats;
      - sdlist.py to print a summary of a dataset;
      - sdplot.py for plotting of spectra;
      - sdstat.py to derive statistics (noise rms, etc.) of specified regions of a spectrum; 
      - sdcal.py as a combination of sdaverage, sdsmooth and sdbaseline; 
      - and others.

  • Enhanced support for OTF data single-dish data.

4. Design

  • Some prototype tasks (sdcal, sdlist, sdplot, sdstat, sdfit) already exist. For these, the user interface will be improved by compliance with current task standards.

  • More inputs are needed to define specific tasks/tools needed for enhancement of the OTF data treatment from the sponsor and others.

5. Deployment Checklist

For this developmemt cycle, the focus was on development of the single-dish tasks. Enhancements to the processing of total power data, including OTF data, is planned for the next development cycle.

Existing tasks modified:

DONE Tasks sdcal, sdlist, sdplot, sdstat, and sdfit.

New tasks created:

DONE Tasks sdaverage, sdsmooth, and sdbaseline.

DONE Task sdcoadd with limited functionality.

DONE Task sdflag: non-interactive channel flagging with the option to restrict flagging ranges by scan, IF, field, and polarization.

DONE Task sdscale: scaling of the spectrum and Tsys data.

DONE Task sdsave: save the spectral data to http://www.atnf.csiro.au/computing/software/asap/ ASAP, MS, SDFITS, or ASCII formats.

The regression scripts for GBT data (Orion-S and IRC+10216) using the new tasks were also checked in.

ALERT! Enhancement of the OTF data treatment.

6. Test Plan

6.1 Internal Testing

  • Basic functionality tests for each task were done (as well as with the regression scripts, see below). It is ready for sponsor testing.

6.2 Sponsor Testing

The following is the report of sponsor testing. The tests were carried out on CASA Verson 2.0 Revision 4638, running on Linux. As this is a "daily" rather than "stable" version, before starting on tests of the new functionality, some of the standard regression tests were run. These tests were passed. In the next two sections are some general comments (those not specific to a particular SD task) and some minor comments. After that come detailed comments on all of the SD tasks which have been implemented thus far. All of the tasks seem to perform their job as advertised, although it would be valuable to have a wider variety of test data to work with. The exception is the SDCOADD task, which I was not able to test properly (see below).

GENERAL COMMENTS

  • Tools help: the help text for the SD tools is extremely verbose and is written in a style difficult for the user to understand. At some point the style and content of the help text for the http://www.atnf.csiro.au/computing/software/asap/ ASAP tools and the rest of the CASA tools need to be made consistent.

  • Polarization support: support for polarization in the SD tasks needs to be enhanced, as detailed below.

  • Output files: control of overwriting of files through the "overwrite" parameter needs to be included in all of the SD tasks that write files. Also, all SD tasks that write data files should be able to do so in all four supported formats (http://www.atnf.csiro.au/computing/software/asap/ ASAP, ASCII, MS, SDFITS). No file should be overwritten or modified by a task unless told to by the user.

  • In tasks that combine spectra together, it should be possible to control the weightings.

  • The parameter "sdfile" is sometimes used as an input file and sometimes as an output file. Other times, "infile" and "outfile" are used instead. The nomenclature should be systematized.

MINOR COMMENTS

  • The "restore" command no longer seems to work.

  • The "listobs" task works with single-dish data sets, but the parameter "vis" is in this case a misnomer as the input file does not contain visibilities.

  • It is annoying that the help text for SD tasks disappears from the screen once you press "q".

  • The "align" parameter in "sd.average_time" does not seem to work.

  • Give an astronomer too much rope... I guess the following is a lesson in the dangers of providing the user with access to "low level" commands. Accidentally writing "infile = sdlist" (instead of "infile = sdfile") results in "infile" being redefined to be "infile = ". Seeing hex numbers when you don't expect them is usually not good. Following that, it was impossible to set "infile" to a sensible value, apparently because I had renamed it to be a function ("WARNING: Using infile for a variable name is not allowed; it is a protected CASA function. SUGGEST: try infile( ... ) or type help infile"). Typing "help infile" resulted in the "sdlist" help information. Typing "infile(sdfile)" fixed the problem!

SDAVERAGE

  • It should be possible to set specunit = '' and have it interpreted as the current or default value.

SDBASELINE

  • The "blmode" parameter defaults to 'list', but the help text says the default is 'auto'. I suggest the default be 'none'.

  • When baselining, it should be possible to specify that regions of the spectra in which baseline extrapolations are performed (e.g. at the edges of the spectra) be automatically dropped.

  • The parameters of the baseline fits (regions, polynomial order and terms, rms, etc) should be automatically stored in the output data or in an additional ascii file. This is particularly important when using the 'auto' option.

  • We will eventually probably want a real interactive fitting procedure where fits can defined and redone on-the-fly.

SDCAL

  • The parameters "scanaverage", "timeaverage", and "polaverage" could conveniently be placed within a single expandable parameter called, e.g., "average".

  • The "kernel" parameter does not accept the value 'none'. I think 'none' should be the default, since it should be possible to turn off smoothing without having to select, e.g., 'boxcar' with kwidth = 0.

  • The "blmode" parameter defaults to 'list', but the help text says the default is 'auto'. I suggest the default be 'none'.

SDCOADD

  • This task could not be properly tested because I wasn't able to figure out the correct syntax of the "infilelist" parameter. For example, infilelist = ['file0', 'file1', 'file2'] didn't work.

  • Can infiles with different formats (e.g. MS and SD?) be processed together by SDCOADD? The help text seems to suggest yes, but the inputs list suggests not.

  • The word "coadd" is usually used to refer to the averaging together of spectra. This task is merging scantables together, so perhaps this task should be renamed to, e.g., "SDMERGE"?

SDFIT

  • It should be possible to set specunit = '' and have it interpreted as the current or default value.

  • The task will only fit to the first spectrum in an input file, not all of them.

SDFLAG

  • If "outfile" is not specified, the input file should not be overwritten.

  • Once a file has a maskflag defined, how do you later find out what its value is?

SDLIST

  • The output from this command should not vanish once you type "q".

  • The "async" parameter which appears in the input list is not mentioned in the help text.

SDPLOT

  • The parameter "calmode" is listed in the help text, but it is not in the task inputs.

  • How can you plot 'ps' and 'psr' spectra separately?

  • Would be useful if spectra could be plotted in histogram mode.

  • It would also be useful if one could have a screen readout of the spectral values as the cursor is moved along in channels.

  • The way the task decides how many channels to plot is odd. For example, for an 8192-channel spectrum, the task draws a box from channel 0 to 9000 (or 8999?), when I would have expected 0 to 8191. If sprange = [0, 8000], channels 0 to 8000 are drawn. However, if sprange = [0, 6900], channels 0 to 7000 (or 6999?) are drawn.

SDSAVE

  • The task needs an "overwrite" parameter.

  • We should be able to select by polarization when saving the data.

  • If you set outfile = 'none', the task writes out a file named "none". Not what I expected!

SDSCALE

  • It is probably most logical that scalesys = True be the default.

  • How can we find the values of Tsys?

  • The option outfile = 'none' should not overwrite the input file.

  • The behaviour of outfile = '' is different (but safer) than in other tasks. We could consider using it in the others.

SDSMOOTH

  • Why are there no parameters "timeaverage" and "polaverage"?

  • The "kernel" parameter does not accept the value 'none' (which should be the default).

  • When examining the ascii output file, what are the meanings of "yf0" and "yf1"?

  • Handling of polarization needs to be improved. It is possible to split off data by polarization, but the output file does not appear to carry along the polarization id with it.

SDSTAT

  • It would be useful to be able to dump the output to an ascii file.

6.3 Integration/Regression Tests

Tested with:

  • Scripts ori_hc3n_task_regression.py, ori_sio_task_regression.py, and ori_ch3oh_task_regression.py.

  • Scripts irc_hc3n_task_regression.py, irc_sio_task_regression.py, and irc_cs_task_regression.py.

6.4 Testing for Scientific Validity

To be defined.

7. Suggestions for Further Development

7.1 Enhancements from Patch1 Testing

Based on the tests from Patch1 above, here is a list of detailed items suggested for further single dish development:

ALERT! More thorough support for polarization throughout the single dish tasks (e.g., SDSAVE, SDSMOOTH).

ALERT! Overwrite (True/False) parameters in all of the relevant SD tasks.

ALERT! Write/read support for all four supported SD data formats (http://www.atnf.csiro.au/computing/software/asap/ ASAP, ASCII, MS, SDFITS) throughout the package.

ALERT! Ability to assign weights when coadding/averaging spectra.

ALERT! More consistent use of parameter names (e.g., infile and outfile versus sdfile).

Other improvements:

SDBASELINE

ALERT! Save the parameters of baseline fits in the baselined spectra.

ALERT! Real interactive baseline fitting.

SDCAL

ALERT! Implement an expandable parameter for all types of averaging methods.

SDFIT

ALERT! Fix bug that results in only the first spectrum in a file being fit.

SDFLAG

ALERT! Fix overwriting bug.

SDPLOT

ALERT! Implement histogram mode support.

ALERT! Screen readout of spectral values at cursor position.

SDSTAT

ALERT! Add an option to dump the output to ASCII.

7.2 Requirements for SD support for AIV

There are a number of requirements for total power/SD functionality for http://www.alma.cl/aiv/ AIV. The following is a list of what is thought to be needed, along with comments on what, if anything, needs to be done to ensure that the functionality is available within CASA. This list of requirements is a work in progress.

ALERT! OTF Mapping

For terrestrial (e.g., ground-based beacons), sidereal, and solar system objects. We include in this such tasks as raster scans for, e.g., mapping out the single dish beam profiles. We might also do some radio pointing using small scans.

At present, support for OTF has not been implemented for the ASDM -> MS -> http://www.atnf.csiro.au/computing/software/asap/ ASAP data stream. At least, we expect that there would need to be modifications of the MS-to-http://www.atnf.csiro.au/computing/software/asap/ ASAP filler in order to identify the "on" and "off" data in the http://www.atnf.csiro.au/computing/software/asap/ ASAP scantable. Calibration specific to OTF observations would also have to be implemented.

OTF imaging support is needed. The EKH technique should be implemented first, and others (such as MEM, Pixon, matrix methods, etc) should be investigated with a view to implementing them as an option for reconstruction of OTF images.

ALERT! Fast Switching Calibration

This needs to be supported, but it is unclear at present what support is already present, and what needs to be added.

ALERT! Radio Pointing

Has been no specific implementation of this, although it might be done with with scripting outside of http://www.atnf.csiro.au/computing/software/asap/ ASAP directly accessing the MS data rather than the http://www.atnf.csiro.au/computing/software/asap/ ASAP scantable.

ALERT! Focus versus Zenith Distance

Probably can be done with some scripting.

ALERT! OOF Beam Maps

OOF (Out Of Focus) beam maps are envisaged as an antenna characterization tool. Requirements for CASA are unclear, but this is apparently a low priority task for http://www.alma.cl/aiv/ AIV.

Here are some detailed requirements:

ALERT! Averaging (also median, clipped mean) spectra with different velocity shifts. This can be done with the tools, and needs to be added to the tasks. May want to extend this to spectra with different channel widths.

ALERT! Calculating integrated flux over a range of channels. This can be done using SDSTAT. Also useful to be able to calculate other moments of the spectra.

ALERT! Calculating the FWHM of a spectral line. This could be done by simple scripting and added to SDSTAT.

ALERT! Basic spectral math such as addition, subtraction, multiplication, and division. Multiplication/division of spectra by scalars is done in SDSCALE. How would other operations spectra-spectra and spectra-scalar be done?

ALERT! Splitting data into separate spectra (by polarization, IF, channel range). Splitting by polarization and IF can be done by the tools and would need to be added to the tasks (e.g., adding polarization to SDSAVE). Splitting by channel range may not be possible in http://www.atnf.csiro.au/computing/software/asap/ ASAP?

DONE Concatenate scantables. This is done in SDCOADD.

ALERT! Multiple rest frequencies stored in order to suupport multiple lines per IF. Not yet implemented. May require changes in the http://www.atnf.csiro.au/computing/software/asap/ ASAP data structure?

ALERT! Averaging spectra when plotting. Can be done using tools, needs to be added to SDPLOT (e.g., add weighting parameters).

ALERT! Control of line and histogram details in SDPLOT (thickness, style, size, color). Can be done in tools, needs to be added to SDPLOT.

ALERT! Catalogue overlays in plotting. SDPLOT uses the JPL line catalogue; others currently require use of the tools but the latter functionality should be added to SDPLOT.

ALERT! Plot residual data with/without subtraction of fit functions. Can be done at tool level, needs to be added to SDFIT.

ALERT! Display basic header data and annotations on/next to plot. Can be done with scripting at tool level, needs to be added to SDPLOT.


Signatures

APPROVED: I acknowledge that my request is fully contained in this MR, and if the CASA development group delivers exactly what I specified, I will be happy.

ACCEPTED: I acknowledge that I have validated the completed code according to the acceptance tests, and I am happy with the results.

Written - - - - -
Checked - - - - -
Approved by Scientific Sponsor - - - - -
Accepted/Delivered by Sponsor - - - - -

Symbols:
  • Use %X% if MR is not complete (will display ALERT!)

  • Use %Y% if MR is complete (will display DONE)


Discussion Area

Topic revision: r13 - 2008-05-04, LewisKnee
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback