VEGAS Operations and Trouble Shooting Guide

If you are stuck

If you cannot find the help you need here, first port of call is the on-duty support scientist: schedule

If they are stuck, call:
  • Richard Prestage: 304-456-2222 (w), 304-456-4711 (h)
  • Adam Kobelski 406-624-9858
  • Sam Bates: 304-456-4219
  • Anish Roshi 434-244-6858 (w), 434-227-4664 (h)

Quick reference to VEGAS Modes

Introduction

This wiki page is intended to provide assistance to GBT Support Scientists and Operators while observing with VEGAS. We expect this page to evolve rapidly as we gain experience with the instrument. If you find errors in this document, please fix them yourself, or email Richard Prestage with details. There is a space at the bottom of this document ( Comments) where you can add comments, ask questions and so on.

Overview of VEGAS

VEGAS is an eight-bank Spectrometer, similar in many respects to the GBT Spectrometer, but with significantly enhanced capabilities. More information is available at:

VEGAS has twenty-nine different "observing modes". These have different bandwidths, frequency resolutions, and so on, and are listed in the VEGAS mode table.

VEGAS High Performance Computers

An important difference between VEGAS and the GBT Spectrometer is that part of the real-time data processing is done using GPUs (Graphics Processing Units) installed in so-called High Perfomance Computers (HPCs). VEGAS has eight "Banks", Bank A through Bank H, and each Bank has an HPC computer associated with it. These are called vegas-hpc1 through vegas-hpc8 .

Observing with VEGAS

Observing with VEGAS is extremely similar to observing with the GBT spectrometer. In brief,

  • Command VEGAS through Astrid:
    • define a configuration in your Astrid observing script
    • configure
    • balance
    • execute a scan to acquire data
  • "Fill" your data, either relying on the on-line filler, or running SDFITS manually
  • Process your data using GBTIDL

Some notes on each of these steps are provided in the following sections

Configuring VEGAS

Please see the GBT Observer's Guide for details as to how to configure VEGAS.

Balancing VEGAS

As for other instruments, VEGAS is balanced using the Astrid Balance() observing directive. the default target level is -20dBm. As an ongoing commissioning exercise, we are exploring whether different balance levels will give better baselines. To balance VEGAS to a different power level, e.g.:

Balance("VEGAS", {"target_level": -17})

Executing Scans

Executing scans with VEGAS should be identical to current use of the GBT Spectrometer.

Filling Data

As for the GBT Spectrometer, the online filler should fill VEGAS data automatically. However, the integration times for VEGAS can be much shorter, and the data-rate can be substantially greater than for the GBT Spectrometer.

The online filler skips all VEGAS data with an integration length < 0.5s. If it falls behind it behaves as it does now for the spectrometer. It just keeps trying to play catch up until the project ends. If a new project starts before it has caught up, it gives up on catching up and moves on to the new projects. It only checks after it has finished processing a scan - so if it's catching up on a particularly long scan it can miss the start of the next project while it finishes filling that scan.

Unlike the GBT Spectrometer, the online filler can not fill VEGAS data until the scan has completely finished. Consequently, the online data may not be available in GBTIDL for a substantial time after a scan has finished (depending on the length of the scan and the volume of data recorded for that scan).

If necessary, you can fill your data manually as normal using the sdfits command.

Note: VEGAS data is stored on the lustre file system. This is only mounted on the most powerful machines in Green Bank, which does not include the observing machines titania or ariel. To fill data, you must be logged onto a data reduction machine, and we recommend one of newton, planck and fourier, which have a direct, fast, infiniband connection to the lustre file-system

Processing your data using GBTIDL

Processing your data with GBTIDL should be essentially the same as for the GBT Spectrometer. So far in commissioning tests we have checked gettp, getps, getfs and getnod, and we believe these should all function as expected.

Like the GBT Spectrometer, VEGAS produces one FITS file for each Bank, labelled A through H. Unlike the GBT Spectrometer, sdfits fills each VEGAS Bank to a separate SDFITS file. Therefore, to import data into GBTIDL, you are most likely to want to use the dirin command e.g.:

GBTIDL> dirin, "TGBT13B_502_31.raw.vegas"

If you use filein but provide a directory name, as above, it will silently work as if you had used dirin. The distinction between dirin and filein lies mainly in the GUI file chooser you get if you don't provide an argument for either command. The dirin GUI requires you to choose a directory and the filein GUI requires you to choose a file.

Real-time Configuration, Status and Diagnostic displays.

There are three main tools to monitor the real-time configuration of VEGAS: the VEGAS CLEO application, "VEGASDM", the VEGAS Data Monitor, and "Vegas Status", an application which monitors the state of VEGAS shared memory. Each of these is described in more detail below.

NOTE: The GbtStatus and DataDisplay tabs in Astrid currently DO NOT HANDLE VEGAS data well. VEGAS is disabled in live mode because the VEGAS fits files never appear on disk during the scan and gfm will appear to hang as it constantly tries to find the non-existent FITS files while the scan is running. However, in offline mode you can browse VEGAS data with gfm in the same way you can with spectrometer data.

VEGAS CLEO application

You can launch the VEGAS CLEO application via CLEO Launcher -> Backends -> VEGAS.

The CLEO application has the standard functionality of a backend application, and should be relatively self-explanatory. Contact Ron Maddalena if you have any questions.

The "VEGAS Data Monitor" button will launch VEGASDM, and the "Status" button for each bank will launch VEGAS status (see the next two sections).

VEGAS Data Monitor (VEGASDM)

VEGASDM provides a real-time display of the current power levels as measued by VEGAS, and a histogram of the distribution of ADC (Analog to Digital Converter) Counts. To start VEGASDM:
% source /home/gbt/gbt.bash (or .csh)
% vegasdm

This will start a new window, displaying the above information. There is one tab for each Bank, plus an overview Tab showing the measured power for all banks. If VEGAS is configured and working correctly:

  • The measured power (in dBm) should match the target power level issued in your Balance() command;
  • The ADC histogram should look ~ Gaussian, with a FWHM of around 50 counts.

VEGAS Status

VEGAS status displays the contents of the shared memory of each of the VEGAS hpcs. To use VEGAS Status, you need to be logged on to the HPC whose status you are interested in (which might be all of them!) To run VEGAS Status (this will be improved soon):

log into the appropriate hpc machine (e.g. vegas-hpc1)
% source /home/gbt/gbt.bash (or .csh)
% vegas_status

This will start a status window which will display the contents of the VEGAS shared memory.

Manager Control

You can also use the CLEO Manager Control to check on the status of the individual VEGAS banks. Go to Launch -> Manager Control. Select the Sub Sys button to the right of the main VEGAS manager.

What VEGAS should look like in normal operation

  • Using the CLEO IF Manager application, click on the VEGAS input ports. The IF routing should be as you expect for your configuration.
  • In VEGASDM, the measured power for each bank in use should match your target Balance level
  • In VEGASDM, the ADC count histograms should look ~ Gaussian, with a FWHM of around 50 counts
  • In VEGAS Status:
    • describe what a good setup should look like here.

Troubleshooting Guide

There is a large spike in the center of my spectrum.

This is due to the fact that we are using interleaved ADCs. The spike appears at exactly the channel BW/2.0, where BW is the bandwidth. The solution is simply to flag this channel. The spike will be significantly reduced, if not entirely eliminated, when the ADCs are calibrated, which should happen around February 24th. As of late July 2014, a cosmetic fix has been applied which interpolates over this spike, effectively removing the spike. If the spike is found, please contact the VEGAS team.

I get the error message "BankAMgr is Hung in Aborting"

This seems to happen fairly frequently when VEGAS has not been used for a while. The cure is:
  • Launch CLEO DeviceExplorer
  • double-click on VEGAS to select this Coordinator
  • Using the "Managers" pull-down menu at the top-left, turn the Coordinator OFF and ON again
  • retry your observation

I cannot find my data to fill it

I ran e.g. % sdfits TGBT13B_502_33 and it didn't find any VEGAS data, even though I know it is there.

This means that you are running on a machine which does not have the lustre file system mounted. Use newton, planck or fourier to fill your data.

I am trying to observe, and I get an error message about invalid bandwidth.

This may be because you do not have the correct value of vegas.subband. This is "remembered" from configuration to configuration, so you may need to set it to exactly the value you need (1 or 8) in your observing script.

Vegas is not paying attention to what bank I want to use.

The vegas.vfreq keyword for bank is "bank", not "Bank", i.e., make sure you're using all lowercase.

VEGAS is not switching

Make sure that if you're using a multi-beam receiver that Bank A is being used. If Bank A is not being used, switch the beam being used (B2 in KFPA is a good choice, B1 otherwise) or change your configuration to use all eight banks.

I can't run vegasdm

Tried to start vegasdm. Got the error message:
panelAccess for VEGAS.BankAMgr
RPCclient::rexec:Connection not yet established, Call not made

Then it just hung.

This means the Managers are not running. Ask a member of the Software Group to restart them

GBTIDL doesn't see my data

I filled my data ok, but when I try and use e.g. "gettp", GBTIDL says "no data found, can not continue".

For some receiver/VEGAS combinations, not all IFNUM, PLNUM and FDNUM combinations will have data. For example, with Ka-band, valid data is in FDNUM 0, PLNUM 1, and FDNUM 1, PLNUM 0. So, you need to specify these explicitly with the gettp command.

Comments, Questions

Feel free to edit in comments or questions here.
Topic revision: r22 - 2014-08-12, AdamKobelski
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback