Traditional Holography Scripts User's Guide

Overview

This is a guide on how to use the new Traditional Holography scripts. These are essentially a port of the old Matlab scripts, which are documented here.

For information regarding the location of the scripts, git repositories, code architecture, etc., visit the Developer's Guide (TBF).

Background

The original Traditional Holography scripts were created in Matlab by (primarily) Todd Hunter. They were used successfully for many years, and survived at least one Matlab version change (R2008 -> R2011B). However, Todd Hunter has moved on, and we have had yet another Matlab version change (2011B-> R2012B), which has broken the Matlab Holography scripts. It seems that due to the strangeness of the ScanLog.fits format, our current version of Matlab can apparently no longer read the DMJD and FILENAME columns. There is probably a workaround for this, but it was decided that the GB Software Development Division would subsume responsibility for the maintenance of the scripts. In light of this, it seemed prudent to port the Matlab scripts to a more portable, stable, and maintainable language: Python.

Development Summary

  • Mark Whitehead began development of the scripts in August of 2015 with a proof-of-concept IPython session that captured the core functionality of the mainBuildAsciiCorrelator.m Matlab script.

  • In early September of 2015 the project was handed off to Thomas Chamberlin.

  • The project is projected to conclude development in early November of 2015.

Script Entry Points

Note that there is not a huge amount of difference between the functionality of these scripts and the old scripts. However, for the sake of completeness, they will be fully documented here.

There are, effectively, two scripts that comprise "the Holography scripts": processHoloScan and processHoloMap. These scripts are globally available on the default path of any computer within the GB intranet.

I'll be using TPTCSHOL_141004 for all of the examples here unless otherwise specified.

processHoloScan

This script is responsible for analyzing a single Holography scan. It generates a single figure that it saves to disk as both a PDF and a PNG, then shows it interactively on-screen. Within the figure are two plots (shown below) - one of phase vs time, and one of amplitude vs time.

Here are two figures (one from each of the two typical use cases):
Phase Stability Central Stripe SNRSorted ascending
TPTCSHOL_141004.6.phaseamp.png TPTCSHOL_141004.113.phaseamp.png
$ processHoloScan TPTCSHOL_141004 6
$ processHoloScan TPTCSHOL_141004 113

These plots are intended to be used for diagnostic purposes and provide a way to, at-a-glance, confirm that the data taken during a Holography project is valid and useful. Typically two scans are analyzed - one to check phase stability (a 'Track' scan that tracks the satellite for ~2 minutes), and one to check the signal-to-noise ratio of the center stripe (an 'RALongMap' that ???).

If you wanted to, for example, ignore the first half of the data and output to your local disk:
$ processHoloScan TPTCSHOL_141004 6 -o /tmp -i .5

processHoloMap

This script is responsible for analyzing a single Holography map (i.e. a set of Holography scans). In the simplest case, you would pass it the range of scans that comprise the Holography data for the given project. However, it can also handle non-contiguous scan ranges.

For reference, here are the figures output by this script: [Images in the same style as above (too lazy to do this in the sandbox)]

Example .dat files are attached (but would be incredibly boring to read), as well as the summary .pdf containing all of the above images (in vector format, so give it time to render).

If you wanted to, for example, save only the .dat files to your current working directory (no figures shown or saved):
$ processHoloMap TPTCSHOL_141004 7-218 -o . --save dat

Differences between old (Matlab) and new (Python) scripts

Although the primary reason behind the port was maintainability, there have also been some usability enhancements. As a result, the new scripts function somewhat differently than the old ones. The specifics of how to use these is described in Usage.

The most important differences are in the following features:

Usage

Note that usage information may change - be sure to check it yourself using the below command.

Arguments link to a more detailed description, including any differences in usage from the old scripts.

General

The following are features shared by both scripts.

Project Identification

An absolute path to the project directory is no longer required. While it is still possible to enter an absolute path, it is much easier to simply enter the project ID. As long as your project exists in a standard directory (i.e. /home/gbtdata or /home/archive/science-data or /home/archive/test-data), it should be found.

This is the first positional argument and requires no flag.

Scan Numbers

For scan processing nothing has changed: a single scan number is specified.

For map processing, the range of scan numbers to process no longer has to be contiguous. That is, a series of scan number ranges can now be input, which will then be concatenated together and processed in the same manner as a contiguous range.

This is the second positional argument and requires no flag.

Scan Type Checking (--validtypes)

Scan types are now checked for validity during processing. Holography scans, in this context, are either a 'Track' (typically a reference scan in this context) or an 'RALongMap'. If necessary you can also specify a different set of acceptable scan types by using the '--validtypes' flag. For scan processing, if either of these types is encountered a warning will be raised (but processing will continue). For map processing, after a summary of the requested scan types has been printed, an error will be raised, stopping the program and providing a summary of the invalid types. This behavior can be overridden with the '--force' flag.

Output Location (--output)

The output directory can now be specified, rather than simply saving to the current working directory. It will default to '/home/groups/ptcs/holography/data'.

Within the specified directory a folder will be created using the project ID as the name. Within that directory the structure is flat, with all of the output residing immediately inside.

Show Plots (--showplots)

Interactive plots are now optional and disabled by default. To enable, simply pass in the '-p' flag.

If enabled, at the end of the script you will be prompted to press ENTER to show the plots. All the data that will be saved to disk has already been saved; feel free to exit the script at this point.

In contrast to the Matlab scripts, the plotting library used by the Python scripts (matplotlib.pyplot) shows the plots via a blocking call - in other words, while the plots are showing the program cannot continue. Rather than add threading to the scripts to avoid this, I've simply put the plots at the end to avoid needless complication (I've made sure to include plenty of that elsewhere in the code).

processHoloScan

View the help:
$ processHoloScan -h

Partial Data Frame Analysis (--ignore)

For the purposes of this document, the Holography correlations will be referred to as the "correlation data frame". As in the old Matlab scripts, only a portion of this data frame is analyzed for the calculations that appear in the info panes next to their respective plots. Originally, 25% of the data at the beginning of the data frame was ignored - that is, only the latter 75% of data was used for the calculations. This is now a user-configurable value, and the default is now set to 20%. Note that, as was the case previously, this value does not effect the plots themselves.

processHoloMap

View the help:
$ processHoloMap -h

Phase Corrections (--phasecorr)

Overview of what phase corrections are and why you would use them

Pointing Corrections (--pointcorr)

OVERVIEW OF WHAT POINTING CORRECTIONS ARE AND WHY YOU WOULD USE THEM

Situation in which you would use pointing correction mode 1

Situation in which you would use pointing correction mode 2

Situation in which you would use pointing correction mode 2.5

Selective Processing (--save)

By default, all possible figures and .dat files are saved to disk. If for whatever reason you would not like something to be saved to disk, you can specify what output you would like.

The only practical use case I can think of for this would be if you only want to generate the .dat files. So, simply specify that you only want to save .dat files - if there is no need for figures to be generated, the script will skip them, saving you literally entire seconds of your life.

Known Issues

  1. The first run of the scripts per "session" might take a very long time. Even after the first run, subsequent runs will still take a few seconds to begin. This is due to the large number of imports that the scripts require, and might also be related to NFS.

TODO

The programmer's guide has not yet been created.

The following features are either incomplete or missing altogether.

Phase corrections

Need to be tested to confirm they are working properly. Preliminary tests have been done, but not yet exhaustive.

Pointing corrections

Need to be tested to confirm they are working properly

Future Development Plans

GFM Integration

Richard Prestage would like the new Holography scripts to be integrated with GFM, as described in this MR. This has not yet been scheduled; no estimate for start or completion.

-- ThomasChamberlin - 2015-10-26
Topic attachments
I Attachment Action Size Date Who Comment
TPTCSHOL_141004.113.phaseamp.pngpng TPTCSHOL_141004.113.phaseamp.png manage 241 K 2015-10-26 - 15:18 ThomasChamberlin processHoloScan figure showing SNR of central stripe
TPTCSHOL_141004.6.phaseamp.pngpng TPTCSHOL_141004.6.phaseamp.png manage 213 K 2015-10-26 - 15:18 ThomasChamberlin processHoloScan figure showing phase stability
Topic revision: r1 - 2015-10-26, ThomasChamberlin
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback