I16293 Water VLA Observation Summary
Last Modified: AlWootten
- 20 Nov 2018
Archive Extraction and Download
All data can be extracted from the EVLA archive available from the NRAO User Portal
. Since these data sets are extremely large, one needs to use the CV lustre to both store and process (with CASA) these data. Steps to extraction and download are:
- Make the archive extraction request from NRAO User Portal, and wait for the data to be extracted into the archive directory.
- Login to firstname.lastname@example.org to request a lustre (cvpost) node: ssh email@example.com
- List all used and free lustre nodes: qstat -a.
- Request a node: nodescheduler --request 45 1 to request one node for 45 days (the current maximum), for example. You will get an email telling you which node you have been assigned. Let's assume in the following that you have been assigned "cvpost005".
- ssh into your assigned node: ssh cvpost005
- If you are going to use a VNC connection later, start the VNC server: vncserver
- cd to your data directory (you should have already been given a login on the lustre). NOTE: Data directories are something like /lustre/cv/projects/awootten.
- Use the following wget command to copy files from the archive:
wget -b -e robots=off --wait 1 -rc -nH --cut-dirs=1 --header="accept-encoding: gzip" --user=Chicken --password=Butt https://archive.nrao.edu/secured/Chicken/
- If you are copying files from the public archive, use the following command (use wildcards on the MSNAME if necessary):
Links to data reduction details provided in link in table.
wget -b -e robots=off --wait 1 -rc -nH --cut-dirs=1 --header="accept-encoding: gzip" ftp://ftp.aoc.nrao.edu/e2earchive/MSNAME
- 2018-11-20: Updated for 18A-507
| Session Log
|| Duration (hrs)
Comments: Only 26 antennas available. First session. Copied from EVLA pipeline and located at /.lustre/cv/projects/awootten/I16293/2018Water/. EVLA pipeline flagged most water maser channels. Made cont.dat file to use rerunning pipeline, defining continuum channels so line channels will not be flagged. In addition to creating a 'cont.dat' file, we advise to modify the following tasks in 'casa_pipescript.py' for spectral line data:
- hifv_hanning: Hanning smoothing lessens the Gibbs ringing from strong spectral features, usually strong, narrow RFI, or very strong spectral lines such as masers. Hanning smoothing, however, reduces the spectral resolution. Therefore, depending on the data and the science case, one may or may not choose to apply Hanning smoothing. Disable the application of Hanning-smoothing in the pipeline if such smoothing is not needed or desired by commenting out hifv_hanning or simply removing the step from 'casa_pipescript.py'. Do not need to modify.
- hifv_flagata: The pipeline, by default, flags 5% of the data on each spw edge as well as the first and last 10 channels of each baseband. In some cases, for example spectral surveys, lines may fall right on such frequencies. The edgespw, fracspw, and baseband parameters in hifv_flagata can be adjusted to flag different percentages of the edges. Do not need to modify
- hifv_targetflag: Flagging prior to this step was only applied to the calibrator scans, which should be line-free. But hifv_targetflag attempts to auto-flag all fields including target fields. The rflag mode in CASA's flagdata is designed to remove outliers that deviate from a mean level. Strong spectral lines can fulfill this criterion and be flagged. The 'cont.dat' file will ensure that rflag will only be applied to the continuum frequency ranges specified in it. Alternatively, hifv_targetflag can be turned off completely, or, by specifying intents='*CALIBRATE*' (and thereby omitting '*TARGET*'), one can restrict the flagging to the calibrator data only, leaving the target data (scans with the TARGET intent) untouched. In either case, we recommend manual flagging for the spectral line frequency ranges after the pipeline has finished processing.
- hifv_statwt: A similar argument applies to the hifv_statwt step, where the visibilities are weighted by the square of the inverse of their RMS noise. Strong spectral lines will increase the RMS and will therefore be down-weighted. The cont.dat file will restrict statwt to only use the continuum frequency ranges for the rms and weight calculations and thus prevent the inclusion of spectral features. Alternatively, hifv_statwt can be excluded from the pipeline altogether and the CASA task statwt can be executed manually after the pipeline has finished, where statwt's parameter fitspw should be set to continuum channels only.
- casa -r 5.1.2-4 --pipeline run in /lustre/cv/projects/awootten/I16293/2018Water/rawdata/UJ3510 using execfile('casa_pipescript-20181110S194136.py') (Nov 20, 2018) One needs to avoid spectral line channels--the 'cont.dat' construct only works with casa 5.4.2 or later, Tobin tells me. I just eliminated flagging of the source altogether.
| Comments: Second session. Extracted from archive and located as above.