VLA Pipeline Resources

No longer maintained and probably outdated.

Nathan's Musings on the VLA and ALMA Pipelines

VLA

We inspect calibration tables from VLA PL because our first goal is to identify problems with the instruments and/or system

• that means we want to see the corrections that need to be applied so we can potentially identify patterns that could indicate issues in the system
• looking at the corrected data would only show the quality of the data after all corrections are applied and would not reveal any potential problems we need to address on our side

ALMA

We primarily inspect the corrected data and resulting images when manually reducing ALMA data because our first goal is to meet the science specifications laid out by the PI (i.e. desired sensitivity)

• so we are concerned with the quality of the data when reducing the data and not in what causes the need for the corrections to be applied

Email Lists/Notifications

• Pipeline execution status notifications (ask Bryan Butler to be setup with this; hard-coded into PL)
• OST failures email list
• Daily known issues email list
• [optional] Archive issues email list

Online Resources

• NMObsAccount wiki
• Pipeline Operations Manual (Google document)
• 2014_VLA_CASA_Calibration_Pipeline (Google spreadsheet)
• VLA Quick Reference (Google document)

Tips for Reviewing Weblogs

General Information

• ignore switched power and system temperature plots
• P band and test SBs should not be run through the PL (essentially guarenteed to crash the PL)
• nominally every Friday/Saturday a stress test is run (should be named 14A-000) and this information should go into the stress tab of the spreadsheet
• check the msinfo log for missing scans
  • compare this to listobs to see what's missing
  • e.g. if amplitude calibrator is not observed then PL will fail and it should probably be reobserved/substitute other amp. cal. data
• can see list of refAnts picked by PL in fluxgain log
  • also in qa2script.csh output
• qa2script.csh gives a useful quick summary of info on a PL run
  • it takes the full pipeline Job ID for the "Pipe ID" prompt (e.g. 13B-085_20140303_1393901724016)
  • to run, ssh onto machine in AOC, cd to /users/nbrunett, type "./qa2script.csh" at the command line, enter Job ID at prompt
• can usually get away with just looking at first-run cal tables and the final cal. tables
• an underscore between sb and eb (instead of a period) could indicate a hand-edited data set
• 3bit has 1 and 2 in baseband names
• 8bit has 0 in baseband names

Finding Bad Data

• delays of ≥10 nsec are not good
• phase jumping is not good
• bandpass amplitude vs. frequency shows "V" shape (or strongly sloping SPWs) for bad deformatters
• C band ea11 or ea16 have almost sine wave appearance on bandpass amp. vs. freq. and that's fine
• ea23 DTS frequently

Templates

Pipeline Results Email

Hello,

One of your Scheduling Blocks,  <FSID>, observed on YYYY-MM-DD, L S C X U K Ka Q band, ### GB, has been processed through the VLA CASA Calibration Pipeline, which is designed to handle Stokes I continuum data, and has received a QA2 score of 'SCORE'.  Some data may need further flagging before imaging: please check your data and the target calibration carefully before using the output from the pipeline for science! The following list includes information about any issues found while checking the quality of your calibrated data. 
- Issue one
- Issues two
.
.
.
- Issue n

- If your observations used a resolved calibrator source (i.e., has UV limits) but does not have standard model images in CASA, the calibrated data through the pipeline would not be accurate. In such instances,  re-calibration using UV limits, or imaging the calibrator first and using the resulting image model, will be needed.

- If your science involves spectral lines, you should be aware of the following:

1) The pipeline applies Hanning-smoothing by default, which may make the calibrated data set not optimal for certain spectral-line science.
2) During the calibration process, several edge channels in each sub band get flagged by default because they are noisier. Therefore, breaks in the frequency span get introduced in the pipeline calibrated data, which in turn may make the output not suitable for certain spectral-line science.
3) The pipeline runs an RFI flagging algorithm which should flag strong lines and may remove spectral lines of interest to your science. 
___________
Your calibrated dataset is not ready for retrieval. If you would like to have access to it, please create a helpdesk ticket within 15 days of this email under the VLA Pipeline department (https://help.nrao.edu/) and tell us which of these two options you would like:

1) Download the dataset over the internet. You may choose to download the calibrated Measurement Set (MS), just the calibration and flag tables, or weblog (online viewable set of Quality Assurance plots,  diagnostic plots, and log files).
2.) Have the dataset written to a hard disk and mailed to your location (at a cost of $125.00USD per 2TB hard disk). If you choose this option, more than one dataset may be written to a hard disk, including non-pipeline products if requested.  If you choose to have your data delivered via hard disk, please include the following information, without which, we are unable to process your request:
 
Name
Mailing Address
Telephone Number
Hard disk format: Linux or Mac OS X
Specific Dataset(s) you would like. 

For more information about prices and other hard disk options, please see https://science.nrao.edu/facilities/vla/data-archive/data-shipment

After 15 days, only the weblog, calibration and flag tables, and a few other small files are archived for future retrieval. These may be requested any time. 
____________
The information above also appears in the file "comments.txt" (basic text file), which will be available (and archived) should you wish to have access to the pipeline products in the future. For more information about the pipeline, including access to the scripts for rerunning the pipeline or modification to suit your particular science goals, please go here: https://science.nrao.edu/facilities/vla/data-processing/pipeline

Please remember, after the 15 day period has expired, only the weblog and relevant calibration and flag tables (not the calibrated MS) will be archived for your retrieval in the future.

Please let us know if you have any questions or concerns through the helpdesk (https://help.nrao.edu/).

Pipeline Issues basic text

Common notes to type up for issues that come up with SBs:

• In cases where flagging the pointing calibration spectral windows is enough to push total flagged data past certain threshold causing QA2 Fail.

  • This default QA2 score is due to the number of spectral windows for which the pipeline finds no calibration. Here, this includes the setup scan/spws and does not impact calibration quality.

• Currently for all 3bit observations, we'll stop this at the start of 14B / D configuration later this summer ('14).
  • The beginning of scans immediately following requantizer gain setup scans may require additional flagging to remove data taken while these requantizer gains were still being set! Please check the data carefully and flag as needed.
• When more than 1 integration time in observation, pipeline uses the first it finds in all calculations.
  • The pipeline uses the longest integration time found for the observation: here, s has been used as the short solint for all data calibrated.
• These are separate but all pretty self-explanatory.
  • More than one field has a scan intent of Bandpass calibrator: a model will be needed for some of these since the pipeline has chosen the first one found for this calibration!
  • Since 3C84 was used as a delay and bandpass calibrator, uvrange limits may be used.
  • The spectral indices reported have SNR of 0.0: this may have implications for continuum science on your target. This may require reprocessing to achieve the best continuum science goals. This may or may not be applicable to your science goals.
• Wide frequency range causes plots to be difficult to read because the points get squeezed together.
  • Since this SB uses multiple observing bands, the pipeline output plots may have limited use. We recommend plotting the calibration tables and calibrated data per observing band for further inspection before using the output from the pipeline for science.
• Only one solution interval is used instead of the typical two.
  • The pipeline has used s for both the short solution interval and the long solution interval during calibration. This may be too long of a time for this observation: leading to possible decorrelation.