GBT Data Handling Status and Future Option



This is a work in progress. I will be evaluating the status of GBT data handling as of early 2007 (what's missing, what needs to be improved) and looking at possible directions that GBT data handling might take in the future.

Things remain fairly un-unified across the various backends. Single pointing spectral-line data from the SP and ACS can be handled well by GBTIDL although there are a number of refinements that are sorely needed. One can also still use dish and there are a few die-hard dish users that still need support. Imaging GBT data requires either that the SDFITS file be converted to something that classic AIPS can read or that aips++ be used (the latter is not recommended by GBT staff although several GBT users still do just that). The path from GBTIDL-processed spectral-line data to imaging in AIPS is not well documented and may not be robust. It could be much more obvious how to do that. It could also be possible to go from that same GBTIDL-processed stage to CASA and it should be straightforward to add appropriate gridding kernels to CASA so that the end result should be identical to that produced by AIPS.

It's not clear to me what GBT continuum users are told. The documentation that I could find on the web is not at all helpful. I've been told that people use Bill Cotton's OBIT but I can't find words to that effect or any quick links to documentation on how to do that. That may be the recommended path for continuum imaging using the DCR and perhaps the other continuum backends (CCB, Mustang) but there are simpler continuum tasks that it might be convenient if there was some easier tie-in to the spectral-line reduction (e.g. tipping scans for determining opacities).

As always, pulsar observers have their own set of tools. Do we continue to just allow that to happen or do we need a more standardized approach for non-pulsar-black-belts? Are there some simple pulsar observations that we should standardize (e.g. using a pulsar as a background source where it's really just the set of spectra at various phases of the pulsar period that one is interested) and provide access to the data in GBTIDL (or whatever the spectral-line environment is at the time)?

We have no VO presence. That can't continue. What do we publish to the VO? How much of a pipeline do we want to have to publish standard products? It's worth noting that at this point the VO is more than just a place where people can get at archived data. There are emerging tools that can share data because they all speak "VO" and perhaps the future of data reduction in astronomy is to make sure you can exchange data via VO standards and then pick the tools you need. There will certainly be a need for telescope-specific tools but perhaps the data visualizing and browsing and imaging tools could be picked up from elsewhere. We already do a small amount of code re-use in GBTIDL by making use of the Goddard Astronomy IDL library for things like FITS readers and some coordinate transformations and we use other peoples functional fitting code and color-table code, so we're used to code-reuse. We may be able to reuse higher-level tools.

Or do we just buy in to the CASA/Asap effort? Or will that just put us back where we were with aips++/dish (not happy)? Minimally, it needs to be trivial for GBT users to get their data (raw or calibrated) into casa so that they can make that decision for themselves. The more we buy in, the more leverage we will have for the end product being something appropriate for GBT data. Alternatively, we could find a way to borrow what we like (e.g. Measures and Quanta library) without having to go all the way and switch to casa.

Finally, at what point to we want to ask users what they want. What they think is missing. What they find really cool elsewhere that they can't do easily with their GBT data. Or are they all really happy with GBTIDL and want to see that improved. Or would they rather go with python. And is there a GUI that would make some of the routine tasks simpler or would that just get in the way (data browsing is still much simpler in GFM, for example).


Possibly Useful Resources

  • GDL - Gnu Data Language. This is an attempt to implement a free IDL-compatible interpreter. This is getting close to where it could be a viable alternative to IDL. GDL has an interface to python (python modules can be called from GDL) and GDL can be built as a python module (GDL can be called from python). Widgets are not supported yet. See the web site for the latest details and information.
  • IDL to python translator. i2py does a source to source translation. It doesn't handle objects and it doesn't attempt to sort through the IDL system calls and translate those to something appropriate for python.
  • Calling IDL from python. See this link. The second half of this page is a link to code that purports to allow you to invoke IDL from python and to move simply data types between the two interpreters.

Future GBTIDL development

  • Obviously missing: visual flagging (GUI-based). This is something that aips++ has and this is the most-cited reason die-hard dish users give for not switching over to GBTIDL. Turning boxes and regions into flags in GBTIDL is not the hard step to code. The detail that keeps from happening any time soon is the user interface. It needs a 2-d (image-like) display that you can zoom to the region of interest. It needs to be easy to change the view (e.g. individual channels, averaged channels, various alignments in frequency and possibly also regularly gridding in time). It must do that without ever needing to suck all of the data into memory at one time so that the user can view arbitrarily large chunks of data (with accompanying averaging as the 2-D image is generated for display and interaction). That's a fair amount of code. I have been unable to even find a suitable interactive image viewer in IDL much less think about the mechanics of feeding the data to such an imager as the user pans and zooms and asks for different views. Casa has such a tool, but the data formats are quite different so only the viewer could be used without having to recast (fill) all of the data of interest into a MeasurementSet. At the recent ADASS there were various VO tools shown that might be useful for viewing the data if it could be packaged appropriately. There is no obvious answer.

The role of pipelines


VO tools

-- BobGarwood - 21 Feb 2007
Topic revision: r3 - 2007-04-16, BobGarwood
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback