Instructions for Extragalactic HI Observing at GBT JHibbard, Aug2006. Updated Jan 2008 (pipeline scripts)

1) Preobserving: get setup file and schedule blocks.

Example files (see attachments at end):
scheduler.cat Catalogue file for cleo scheduler. Not used by the system at all (optional).
GBT06C_034_setup.py reads in catalogues; defines global variables (e.g. hoff), sets up receivers. This file is read in by all other scheduling blocks (s.b.'s)
2PIGG.cat Catalogue of source positions, velocities
calsources.cat Catalogue of calibrator sources positions, velocities. Includes both flux calibrators and spectral line calibrators.
2PIGG.sb observing s.b. for sources. Move to source, balance, on/off for some amount of time.
peakcal.sb observing s.b. for flux calibrator
peakspcal.sb observing s.b. for spectral line calibrator

2) Preobserving: validating schedule blocks

Log into computer that can run astrid (e.g. thalassa, despina, thales, euler, fourier, etc. all at .gb.nrao.edu). Start up astrid:

%astrid
Select the option to work offline.
Click on "Edit" tab. Select your project from the drop-down menu. If you have already saved any schedule blocks for this project, the will show up in the left column. Click on the one you want to edit. If you have the files on disk, then click "Import from file", then click on the validate button. When it validates, click "Save to Database". It is now available for use when observing.

Note: you do not validate your catalogue files by reading them in. Instead, you read in a s.b that reads the catalogue and validate that.

3) Optional: Observing remotely.

!!NOTE!! GBT remote observing instructions changed May 2008. I have not propagated these changes into my instructions. I quote from the email sent by Ron Maddalena. See the referenced link.

Change in Remote Observing Instructions

We have recently changed the method that observers should use for remote
observing.  Observers should now use stargate.gb.nrao.edu to access the
control room computers.  We are also seeing a rash of observers not
knowing that they need to explicitly kill their VNC servers at the end
of their observing run.  Multiple servers from previous observers puts
excessive strain on our computer resources and noticeably interferes
with the work of the current observers.  

I suggest that all remote observers read
http://www.gb.nrao.edu/gbt/remoteobserving.shtml for updated
instructions.  Observers should pay special attention to the section on
how to close their VNC session.  By the end of June 2008, VNC will no
longer work with the old instructions and when using ssh.gb.nrao.edu.

R. Maddalena

(see GbtRemoteObserving)
  • Must have account on an nrao machine to do this.
  • Initially:
    • NOT NEEDED FOR LINUX SYSTEMS?: Install the VPN software on your machine.
    • Install the VNC software on your machine (e.g. chicken of the VNC).
      • Before running vnc the first time:
        despina%mkdir .vnc 
        despina%cp ~jbraatz/.vnc/xstartup  ~/.vnc/.
  • ~20min before your scheduled set-up time:
    • Call the GBT operator (304-456-2346) and let them know you will be observing remotely (you should have ok'd this with Carl Bignell before), which receiver you'll be using.
    • NOT NECESSARY?: Initiate VPN client on your home machine. Connect to e.g.
      vpn.gb.nrao.edu
    • Start up a vnc desktop:
      • ssh into a greenbank computer (e.g. thalassa, despina, thales, euler, fourier, leibniz, planck, all at .gb.nrao.edu).
        ssh despina.gb.nrao.edu
        (instructions from GBT page said to type this:
        ssh -N -C -L 5900:titanioa.gb.nrao.edu:590N jhibbard@ssh.gb.nrao.edu
        but we didn't need to do this).
      • start up the vnc server:
        despina%vncserver -geometry 1280x1024 -depth 8
        (for Mac 15" running at 1280x854, then
        despina%vncserver -geometry 1200x800 -depth 8
        seems to work better).
      • The first time you do this you will set a password. Write that someplace safe.
      • Make note of the desktop number that it reports.
      • On your home machine:
        • For Linux:
          superglide%vncviewer [machine].gb.nrao.edu:#
          where [machine] is the machine in gb that you logged into and # is the desktop number that was reported above.
        • For Mac:
          • double click on vnc
          • give hostname (.gb.nrao.edu), password and desktop no.
        • This will pop-up a display that shows a desktop on the machine you logged into in greenbank.
    • Within that desktop open an xterm and start-up your cleo tools (more on this below)
      despina%cleo
    • If you want, you can start up astrid and run it offline to check out your schedule blocks.
  • At your scheduled start-up time:
    • Within the vnc desktop, open an xterm and log into one of the machines that controls the GBT (e.g. titania):
      despina%ssh titania
    • Call the GBT operator (304-456-2346) and ask to be put into the gateway for whatever machine you logged into (e.g., titania).
    • Start up astrid from the xterm on titania:
      titania%astrid
  • Follow instruction below. Except - only ssh titania for running astrid online. If you want to run astrid offline, or run cleo, use a normal xterm from the machine you logged into (e.g., despina). So...see the next few items. Run those from the vnc desktop.
  • To disconnect: within any xterm in your gb desktop type:
    depina%vncserver -kill :#
    where # is the desktop number

4) Observing:

Go to observer room 30min before your run. Ask Operator to put your receiver in. Start up astrid on observing console, but do not go online till they tell you you can.

Meanwhile, in another window start cleo in another window.
%cleo

launch=>status
If observing remotely: launch=>utilities/tools=>Talk and Draw

%cleo scheduler
In skyview: 
- select "real time"
- Catalog...
  => Add/select/deselect catalog
     =>Add user catalog
        navigate to your catalog file, double click on it, and click "ok".
  To see your sources, you may have to change the entries in the
  "Intensity" field and press return.

You use the scheduler to decide what to observe, but should be looking at the status window during your observations - keep an eye on the balance, make sure no untoward messages appear, see how long until the next scan.

By now its probably time to take control of the telescope, so go back to the astrid window and take the telescope online with control.

Go to the "Edit" tab. Highlight your first scheduling block (should include an autopeakfocus followed by a ~60sec observation of your flux calibrator - see peakcal.sb in attachments). Click on Validate. Save to database. Give it a new name if need be.

Now click on the "Run" tab. Select the scheduling block and press "submit".

For autopeakfocus, you can see the scans by clicking on the "Data Display -1" Tab.

To go back to the Edit/Run pages, click on Observation Managment.

You are observing!! Keep tabs on things with the cleo "status" display. Of particular interest will be the levels for the spectrometer after doing a balance on a source. These should both be around 1.3-1.5 and similar to each other.

4.1) Notes on Observing Sequence and using attached sb's

In general, I'll first submit the peakcal.sb file, wait to see the first spectra come up to make sure the spectrometer is working fine, then submit peakspcal.sb (after first uncommenting the spectral line calibrator I want to observe). This shows me that the system is fully functional (and will provide a calibration check when I reduce the data). Then I'll repeatedly edit (to uncomment the desired source) and submit source.sb for the rest of the night (in the attachments I have included 2PIGG.sb as the example of a source schedule block. There is also an example of a 3x3 mapping schedule block). Its a good check to go back at least once to a calibrator - either peakcal.sb, or peakspcal.sb. So here is a possible sequence

  • uncomment desired calibrator and AutoPeakFocus command in peakcal.sb
  • submit peakcal.sb to queue
  • uncomment desired spectral calibrator in peakspcal.sb (but keep AutoPeak commented out)
  • submit peakspcal.sb
  • uncomment desired source in source.sb (but keep AutoPeak commented out)
  • submit source.sb
  • repeat last two steps for about half the night
  • uncomment desired calibrator in peakcal.sb [and/or peakspcal.sb] and submit
  • go back for rest of your sources

Whether and how often to go back to a calibration source, and whether to calibrate on a radio galaxy or a spectral line calibrator depends on what you are trying to do. Include at least one observation of a spectral line calibrator as a test on your calibration. Include more if you want to understand your systematics better.

At L-band I am definitely not worried about the focus after doing it at the start of the night, unless the operator tells me I should. If the observer before you was using L-band, you may not even have to both with focus. But its cheap so do it. I am also generally not worried about pointing (AutoPeak) after I do it at the start of the run, so often will not comment out the AutoPeak functions in the attached source.sb and peakspcal.sb schedule blocks. However, if you have a largish galaxy (including spectral line calibrators), you may want to AutoPeak near it to make sure you are centered - I have seen spectra taken of the same source that show differences in the shape of the line profile that must be due to the receding/approaching side falling in a different region of the beam.

You may also find an apparent absorption profile at a different velocity than your target source. This is probably due to an emission source in the "Off" position. If the profile is not bothering your target spectra, you can leave it as is and get a "free" HI observation of another source. But if you need the real estate, uncomment out the other "OnOff" command to use a different offset (off=ra_offset instead of off=dec_offset).

5) Looking at your data with gbtidl:

(also see example scripts in attachments)

Open an xterm.

ssh to another gb machine (so you don't interfere with the observing machine).

cd to the directory with any idl programs you wish to use (you might want to create a directory for each night you observe and cd there, e.g. mkdir aug11, cd aug11)

Start up gbtidl:
%gbtidl

;; gbtidl_basic.pro
;; basic idl commands for gbtidl
;; JHibbard  Aug2006

;; to work online
online
;;or
online,/acs
;;or
line
online

;; to work with a specific sdfits dataset
filein,'/home/sdfits/AGBT06C_034_01.raw.acs.fits'

;; see what is in your database
summary

;; get a scan and display it (only shows YY pol by default)
getps,22

;; spcifically display YY, then XX
getps,22,plnum=0
getps,22,plnum=1

;; display a single integration of a single polarization of a scan

getps,22,plnum=0,int=17

;; save the displayed plot to a file (can also do with "print" button
;; on display
write_ps
$mv gbtidl.ps out.ps

;; save data from last manipulated plot to an ascii file
write_ascii,'out.ascii'

;; display all ints, one after the other

for i=0,29 do begin $
getps,22,plnum=0,int=i & endfor

;; fit a baseline of order 5

getps, 25
boxcar, 20, /decimate
chan
nregion,[50,300,500,800]  ;; could have used "setregion" to set interactively
baseline, nfit=5
velo

;; get statistics for whatever is plotted:
stats

;; accumlate consequent scans from 60-73, boxcar smooth by 20 channels
;; and display. this requires the file sumcans.pro to be in your directory or 
;; you will need to specify the path. Note: Instead of specifying nregion, you 
;; could use "setregion" to set it interactively.

.compile <path/>sumscans.pro
chan
nregion,[50,300,500,800]  
nfit,1
sumscans, 60, 73, 20


;;; Reducing 9-pt map:

getsigref, 19, 22

;; where 19 is your on-source pointing and ref is your reference pointing. 
;; Do each of these, and accumulate those you want to accumulate. 

;; To get a single scan:

getscan, 19

##############################################################

;; A more complicated example for summing scans w/o RFI:
;; Sum scans [8,10,12,26,28,30]. But scan 25/26 has RFI in ints=20-22, 
;; so individually sum the int outside of that range.

for i=long(8),12,2 do begin $
    getps,i,plnum=0 & accum & $
    getps,i,plnum=1 & accum  & endfor
;;only accumulate ints w/o RFI for scan 25/26
i=25
for j=long(0),19 do begin $
    getps,i,plnum=0,int=j & accum & $
    getps,i,plnum=1,int=j & accum & endfor
for j=long(23),29 do begin $
    getps,i,plnum=0,int=j & accum & $
    getps,i,plnum=1,int=j & accum & endfor
for i=long(28),30,2 do begin $
    getps,i,plnum=0 & accum & $
    getps,i,plnum=1 & accum  & endfor
ave
boxcar,20,/decimate
chan
nregion,[50,300,500,800]
baseline,nfit=5
velo
write_ps
$mv gbtidl.ps out.ps
write_ascii,'out.ascii'


;;This will accumulate XX,YY separately:

freeze
for k=0,1 do begin 
   for i=24,29,2 do begin $
       getps,i,plnum=k & accum,k  & endfor 
   i=25 
   for j=long(0),19 do begin $
       getps,i,plnum=k,int=j & accum,k & endfor
   for j=long(23),29 do begin $
       getps,i,plnum=k,int=j & accum,k & endfor
   for i=long(28),30,2 do begin $
       getps,i,plnum=k & accum,k  & endfor
   ave,k
   boxcar,20,/decimate
   chan
   nregion,[50,300,500,800]
   baseline,nfit=5
   velo
   if k eq 0 then XXspec=dcextract(!g.s[0])
   if k eq 1 then YYspec=dcextract(!g.s[0])
endfor
unfreeze
!g.s[0]=XXspec
show
!g.s[0]=YYspec
oshow

5a) GBTIDL position-switched pipeline

I have written some GBTIDL scripts to automate reduction of on/off HI data reductions. The main files are gbt_setup.pro (one for each night of observing) and GBTreduce.pro. The latter calls the following: accumscans.pro, write_spectra.pro, spcalflux.pro, and ottflux.pro. Its also useful to use the macro GBTreduce1.pro.

To use these, do the following:

  • Edit gbt_setup.pro to identify scans with RFI, calibrator scans, source scans, and input/output data. Use rough values for default_region - you will refine these later
  • start up gbtidl and execute the setup file via: "@gbt_setup". This reduces all the data specified in gbt_setup.
  • use GBTreduce1.pro to look at each source/calibrator separately to determine best region:
    • GBTIDL ->i=1 & GBTreduce1, source[i] & chan
  • Enter these regions into gbt_setup.pro and run again!

For more than one night, I use the same keepfile, so that all the data are archived together.

6) Flagging:

Copied from GBTIDLBlankFlagIntro

  flagrec, record, bchan=bchan, echan=echan, chans=chans, chanwidth=chanwidth,
        idstring=idstring, /keep

One uses idstring to associate with a rule an identifying string that is typically a reminder of the reason for the flag.

Examples:

The following example shows how to flag a channel range for a small number of scans and integrations. Note that either the scan parameter or scanrange keyword is required but both can not be used at the same time. Otherwise, if a parameter is not specified, "all" is assumed. So in this example, all polarizations are flagged:

  flag, [18,19,20], int=[1,3], bchan=512, echan=514, idstring="RFI"

Equivalently, using the scanrange keyword:
  flag, scanrange=[18,20], int=[1,3], bchan=512, echan=514, idstring="RFI"
The next example shows how to flag all channels for a given integration in one scan:
  flag, 15, int=3, idstring="spectrometer glitch"
The next example shows how one could flag all data for the given three scans:
  flag, [101,105,107]
The next example shows how one might flag a record in a processed data file (a "keep" file):
  flagrec, 15, idstring="Glitch", /keep
The next example shows how mone might flag different channel ranges for a record:
   flagrec, 16, bchan=[0,100], echan=[10,110], idstring="Two RFI Spikes"
The next example flags the same channel ranges:
   flagrec, 16, chans=[5,105], chanwidth=11, idstring="Two RFI Spikes"
Flagging a few more scans, not in a nice sequence:
   flag, [40,42,44,47,48,50,56], int=[1,3], bchan=512, echan=514, idstring="More RFI"

-- JohnHibbard - 22 Aug 2006
Topic attachments
ISorted descending Attachment Action Size Date Who Comment
GBT06B-034_setup.py.txttxt GBT06B-034_setup.py.txt manage 1 K 2009-03-13 - 14:50 JohnHibbard setup file for GBT HI extragalactic observations with ACS
2PIGG.sbsb 2PIGG.sb manage 2 K 2009-03-13 - 14:48 JohnHibbard example xgal HI source schedule block for astrid
9pmap2PIGG.sbsb 9pmap2PIGG.sb manage 2 K 2009-03-13 - 14:48 JohnHibbard example GBT xgal HI source schedule block for astrid - 3x3 map
peakcal.sbsb peakcal.sb manage 721 bytes 2009-03-13 - 14:49 JohnHibbard example GBT xgal HI flux calibrator schedule block for astrid
peakspcal.sbsb peakspcal.sb manage 774 bytes 2009-03-13 - 14:49 JohnHibbard example GBT xgal HI sp.line calibrator schedule block for astrid
GBTreduce.propro GBTreduce.pro manage 8 K 2008-01-23 - 17:26 JohnHibbard master file for reducing an entire GBT HI on/off dataset
GBTreduce1.propro GBTreduce1.pro manage 583 bytes 2008-01-23 - 17:27 JohnHibbard utility file to reduce data from one source/calib
accumscans.propro accumscans.pro manage 568 bytes 2008-01-23 - 17:27 JohnHibbard accumulate and average scans given by an array
avepointmap.propro avepointmap.pro manage 1 K 2006-08-23 - 09:41 JohnHibbard gbtidl macro to reduce each pointing of a NxNxn map
gbt_setup.propro gbt_setup.pro manage 4 K 2008-01-23 - 17:30 JohnHibbard setup parameters for use in GBTreduce.pro
spcalflux.propro spcalflux.pro manage 10 K 2009-03-13 - 14:21 JohnHibbard get fluxes etc for some spectral line calibrators
sumscans.propro sumscans.pro manage 485 bytes 2006-08-23 - 09:31 JohnHibbard gbtidl macro to sum both pols, range of scans, smooth, fit baseline
sumscansPol.propro sumscansPol.pro manage 474 bytes 2006-08-23 - 09:40 JohnHibbard gbtidl macro to sum single pol, range of scans, smooth, fit baseline
write_spectra.propro write_spectra.pro manage 1004 bytes 2008-01-23 - 17:29 JohnHibbard make postscript and ascii files for displayed spectra
2PIGG.catcat 2PIGG.cat manage 4 K 2006-08-23 - 09:25 JohnHibbard source catalogue file for astrid (read in by setup file)
calsources.catcat calsources.cat manage 1 K 2009-03-13 - 14:31 JohnHibbard flux and spectral line calibrator catalogue file for astrid (read in by setup file)
scheduler.catcat scheduler.cat manage 11 K 2006-08-23 - 09:22 JohnHibbard example catalogue file for cleo scheduler
Topic revision: r9 - 2009-03-13, JohnHibbard
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback