VEGAS Bank Discrepancy Problem (page created 21 March 2015)

For earlier discussions, and a description of the astronomical manifestation, see see VegasBankDiscrepancyProblem2015Mar21, and subsequent links

Introduction

During the week starting Monday 16th March, I ran a series of experiments designed to (hopefully) definitively tie down the cause of the VEGAS bank discrepancy problem. The advantage of these experiments over previous attempts is that we believe all components (with one caveat see below ) are now well tested. For example, we believe snapblocks are now being archived correctly, (complex) powers are being computed correctly, we understand the performance of the ditigal gain block, and so on..

I designed the experiments to isolate each individual component of the GBT receiver chain / VEGAS, including if necessary subcomponents within the FPGA. So if we are successful, we will be able to isolate the problem to a specific component (or at worst, the interaction between two components).

Below, I describe each experiment and propose the appropriate data analysis approach, including the best order to perform the various data reduction steps. Further details will be added as reductions progress. Hopefully, this page can also serve as a complete historical record of how the experiments and analysis were were performed, so that we will not need to revisit this topic in the future. If necessary, we can always expand the list of tests.

[ caveat: We are now using the GBT Digital Continuum Receiver (DCR) as a total power meter, to get an independent measure of the analog power feeding in to the VEGAS ADCs from the VEGAS Analog IF module. But this is being tapped off from the IF output via a coupler, and detect by a V/F, which we have not fully characterized. If we have any concerns below about the DCR measurements, we can easily review the specifications of these components, and/or verify their performance using the noise source.]

Experimental Setup

We used two separate inputs to the VEGAS analog IF module; the IF from the X-band (and L-band) receivers, and the "noise source". In addition, we used the GBT Digital Continuum Receiver (DCR) to record the VEGAS analog IF monitor outputs. Each of these, as well as the VEGAS configuration, are described below.

Data Directories

Data for these experiments were taken into TGBT14A_912, sessions 83, 84 and 85.

All of these data have already been filled, and are available using the GBTIDL "offline" command.

The Noise Source

The noise source was used (a) as an input directly into the ADC inputs of Banks E through H, and (b) as input to the VEGAS analog IF module.

The noise source consists of:
  • The GB Lab Spectrometer noise source (I do not have specs on this, but could probably get them).
  • An RFI group amplifier (as above)
  • A 300 MHz high pass filter (to remove some strong low-frequency components from the above)
  • A RUDAT-6000-30 0 to 30dB programmable attenuator.
  • An eight way power splitter
  • eight cables into the eight selected input ports.
  • When connected to the input of the VEGAS analog IF module, the amplifier was removed from the IF chain.

When connected directly to the ADCs, a RUDAT attenuation of 0dB corresponds to an ADC power of ~ -15dBFS.

The RUDAT attenuation accuracy is listed as ~ 0.25 - 0.5 dB (~ 6 to 12%). So, we should not expect power ratios to be more accurate than this.

Receiver IF setup

The X band receiver produces circular polarization, the L-band native linear. Each as a low (~10% Tsys?) and high (~ Tsys?) power cal diode. The two polarizations are transmitted over separate analog optical fibers to the GBT Equipment room. The signal from each optical receiver is split across four Converter Rack modules where the IF is mixed down to baseband (150 MHz to 2 GHz) and connected to the VEGAS analog IF inputs. Thus each polarization of the receiver IF can be connected to four banks (E through H) of VEGAS. Various filters e.g. (320MHz) may be inserted in the IF chain, to restricte the passband for non-wideband (e.g. 187.5 MHz) operation.

When VEGAS was connected to the receiver IF, the Astrid "Balance()" command was used to vary the input levels to VEGAS. Note that the Balance() command does not always achieve the target attenuation, and in some cases does not even attempt to change the attenuation (e.g. if the input was at one end of the allowable range for one scan, and the other end for the second scan). Further, the results for each bank / polarization are not uniform.

Therefore, when changing the VEGAS input power levels from the receiver IF, there is no guarantee that the input to each bank / polarization will be stepped by the same amount. The simultaneous DCR observations made in this configuration can be used to resolve this ambiguity.

VEGAS analog IF monitor

The VEGAS analog IF module contains a coupler immediately preceding the output. This monitor point was connected into the "external" inputs of the SGFilters in the Analog Filter Rack. These filters are then connected to the inputs of the DCR. (Again, I don't have specs on these components, but we could find them, or if necessary measure them). In this way, the DCR can be used to measure the total power coming out of the VEGAS analog IF modules.

The monitor outputs and DCR inputs were connected up as follows:

VEGAS SGFilter DCR
Bank E pol 0 SGFilter 1 A9
Bank E pol 1 SGFilter 2 A13
Bank F pol 0 SGFilter 3 A10
Bank F pol 1 SGFilter 4 A14
Bank G pol 0 SGFilter 5 A11
Bank G pol 1 SGFilter 6 A15
Bank H pol 0 SGFilter 7 A12
Bank H pol 1 SGFilter 8 A16
(Note therefore, that the DCR sampler order is Pol 0 Banks E,F,G,H, Pol 1 Banks E,F,G,H). Thanks to Ron Maddalena for pointing this out.)

VEGAS Configuration

VEGAS was configured in either Mode 1 (H1k BOF) or Mode 4 (L1) BOF. The switch period and integration time were both set to one second, and 60 second integrations were performed. The switching mode was either "tp_nocal" (a single phase, with the noise diode switch off) or "tp" (two phases, with the cal diode firing). For every observation, ADC, filtersnap and filtersnap_nogain snapblocks were recorded by the Archivist.

The lbw_gain was set to the the same for all banks/polarizations for a specific observation. By simulation, Randy McCullough has determined that for input white noise, an lbw_gain of 5.664 (time 2**16 prior to writing to the Roach) is appropriate to have each of the real and imaginary terms to have the same variance (in counts) as the ADCs. Since for noise, there should be the same power in both real and imaginary terms, this results in the filtersnap power being 3dB higher than the ADC power. The lbw_gain was either set to this value, or 5.664 / 2 * sqrt(2) (I need to remember why this seemed appropriate. )

Specific Experiments

  1. Mode 1 - noise source to ADC input
  2. Mode 4 - noise source to ADC input
  3. Mode 4 - noise source to ADC input, cables swapped
  4. Mode 1 - noise source to VEGAS Analog IF input, and VEGAS IF monitor output to DCR
  5. Mode 4 - noise source to VEGAS Analog IF input, and VEGAS IF monitor output to DCR
  6. Mode 1 - X-band receiver IF connected to VEGAS Analog IF input, and VEGAS IF monitor output to DCR
  7. Mode 4 - X-band receiver IF connected to VEGAS Analog IF input, and VEGAS IF monitor output to DCR
  8. Mode 1 X-band receiver, total power with cal observation, high cal, VEGAS IF monitor output to DCR
  9. Mode 4 X-band receiver, total power with cal observation, high cal, VEGAS IF monitor output to DCR
  10. Mode 4 X-band receiver, total power with cal observation, low cal, VEGAS IF monitor output to DCR
  11. Mode 1, L-band receiver, total power with cal observation, high cal, VEGAS IF monitor output to DCR
  12. Mode 4, L-band receiver, total power with cal observation, high cal, VEGAS IF monitor output to DCR
  13. Mode 4 - noise source to ADC input, repeat
  14. Mode 1 - noise source to ADC iinput, repear

1. Mode 1 - noise source to ADC input

The noise source was connected directly to the VEGAS ADC inputs, and VEGAS was configured for Mode 1. The attenuator was stepped from 0dB - 12dB in 3dB steps
  • Session 85 Scans 65 - 69.

2. Mode 4 - noise source to ADC input

The noise source was connected directly to the VEGAS ADC inputs, and VEGAS was configured for Mode 4. The attenuator was stepped from 0dB - 12dB in 3dB steps
  • Session 85 Scans 71 - 75.

3. Mode 4 - noise source to ADC input, cables swapped

The noise source was connected directly to the VEGAS ADC inputs, and VEGAS was configured for Mode 4. The cables were swapped as follows: E1 -> G1, E2 -> G2, F1 -> H1, F2 -> H2. The attenuator was stepped from 0dB - 12dB in 3dB steps
  • Session 85 Scans 76 - 80.

4. Mode 1 - noise source to VEGAS Analog IF input, and VEGAS IF monitor output to DCR

Similar to experiment (1), but the noise source was moved to the inputs of the VEGAS Analog IF modules. The amplifier was removed to get reasonable input levels into the IF. Attenuator setting of 18dB appears to correspond to ADC power of ~ -20dBFS. The attenuator was stepped from 12 dB to 27dB in 3dB steps.
  • Session 83 scans 21 to 26.

5. Mode 4 - noise source to VEGAS Analog IF input, and VEGAS IF monitor output to DCR

As experiment (4), but with VEGAS configured for Mode 4. lbw_gain set to 5.664 * 2**24 in all banks.Attenuator setting of 19dB corresponds to ADC power of ~ -20dBFS. The attenuator was stepped from 12dB to 27dB in 3dB steps.
  • Session 83 scans 15 to 20

6. Mode 1 - X-band receiver IF connected to VEGAS Analog IF input, and VEGAS IF monitor output to DCR

The X-band receiver was configured for "tp_nocal" (i.e. a single phase, no switching), and connected to the VEGAS Analog IF. VEGAS was configured for Mode 1. The Astrid "Balance()" command was use to step the Converter Rack attenuators from a nominal input level at the VEGAS ADCs of -15dBFS to -27dB, in 3dBFS steps. Note that the Balance() command does not always succeed, and this varies depending on target, bank, and polarization. Thus the VEGAS ADC or DCR measured values need to be used to determine the actual input power levels.
  • Session 83, scans 45 to 49

7. Mode 4 - X-band receiver IF connected to VEGAS Analog IF input, and VEGAS IF monitor output to DCR

As for experiment (6), but with VEGAS configured for Mode 4. In this case, the nominal target levels were stepped from -18dBFS to -27dBFS
  • Session 83, scans 50 - 53

8. Mode 1 X-band receiver, cal switching, high cal

The X-band receiver was configured for "tp" (with cal, two phases), and the high cal noise diode. The Balance() command was executed with target levels of -15dB to -27dB in 3dB steps. Three one-minute scans were executed for each target level.
  • Session 85, scans 20 - 34.

9. Mode 4 X-band receiver, cal switching, high cal

As experiment (8), but with VEGAS configured for Mode 4.
  • Session 85, scans 2- 16.

10. Mode 4 X-band receiver, cal switching, low cal

As experiment (9), but with the low cal diode. Only one scan each was performed at each of -15,-18 and -21dB
  • Session 85, scans 17 - 19

11. Mode 1, L-band receiver, cal switching, high cal

As experiment (8), but with the L-band receiver. Nominal target levels of -15dB to -27dB, three scans on each.
  • Session 85, scans 35 - 49

12. Mode 4, L-band receiver, high cal

As experiment (11), but with VEGAS in Mode 4. Nominal target levels of -15dB to -27dB, three scans on each.
  • Session 85, scans 50 - 64

13. Mode 1 - noise source to ADC input, repeated

As experiment (1), but performed on a different day.
  • Session 83, scans 7-12

14. Mode 4 - noise source to ADC input, repeated

As experiment (2), but performed on a different day
  • Session 83, scans 1-6

Data Processing Steps

  1. Process Experiment 1 by dividing each successive scan by the 0dB scan, for each bank / polarization. The result should be a flat spectrum, with a value equal to the power ratio between the two scans. The ratio should be identical for each bank / polarization, since the only difference is the cable lengths. But the ratio may be slightly off, to an extent consistent with the specification of the attenuator (see above).
    • Hypothesis: VEGAS will pass this test, since we believe the HI BOF is working correctly.
  2. Perform the same analysis for Experiment 2.
    • Hypothesis: VEGAS may or may not pass, since we know there is a problem somewhere with mode 4.
  3. If VEGAS fails step (2), repeat the analysis for the ADC, filtersnap and filtersnap_nogain snapblocks.
    • Hypothesis: If the problem is in the VEGAS HPC code, these test should pass. If the problem is in the FPGA, one or more of these should fail.
  4. Perform the same analysis for Experiment 3.
    • Hypothesis If the problem is in VEGAS, this should give the same results as for experiment 2. If there is something weird about the splitter / cables, the problem should move with the cables.

Note from Adam: In experiments 4 and 5, are the ADC powers the same? Nothing has changed apart from in the VEGAS analog IF module. Are they using the same filter? Does this go in the correct direction?

I think we should probably halt the analysis when we have the above results, so we can discuss and come to agreement as to what they mean.

If the VEGAS hardware / HPC software checks out ok, we can start moving incrementally up the signal chain.

If the VEGAS hardware / HPC software does not check out ok, we can start isolating different sections of the VEGAS system.

Results

Noise Source into VEGAS (bypassing VEGAS analog system)

Data Processing Step 1 was performed by dividing each scan from session 85 scans 66-69 by it's bank and polarization counterpart in scan 65. This produced the plot below. Note that there is a high pass filter at 300 MHz, so the data below that frequency is not reliable. This plot is from Mode 1 (H1 BOF). While they resulting ratios of spectra are noisy, all banks and polarizations are noisy together, and show similar behavior across the band. The error bars on the plot represent the standard deviation for each spectral bin across all 61 integrations. The mean value of the ratios corresponds to approximately 2.8, 5.6, 8.4, and 11.5 dB for the 3, 6, 9, and 12 dB attenuation settings respectively. This suggests (to me) that the 0 attenuation setting is closer to 0.3dB. This is within the quoted error in the computer controlled attenuator (+/-0.5dB). One bank, Bank G Polarization 1 is systematically higher than the others by a few standard deviations.

Exp1A.png

To better show these deviations, I have included a zoomed in version of this figure, shown below:

Exp1Azoom.png

The data using mode 4 (which uses the HPCs) show the same behavior and deviations, as shown in the figure below:

Exp2A.png

The ripple shapes and magnitudes are comparable to those seen with the mode 1 data, and the linearity of the system appears intact. The Bank G pol 1 signal is elevated (not as attenuated), just as in the previous test. To verify if the Bank G pol 1 signal is due to the cabling, the data from Experiment 3 (which swapped E with G and H with F) were employed. These data show that the elevated signal is not from the cabling, as Bank G pol 1 is still the elevated signal:

Exp4A.png

The color coding is the same, and Bank G pol 1 is still the upper curve, but the difference is small (not yet calculated, though). Looking at how the power levels between banks scaled with the input attenuation levels showed very linear behavior, as shown below:

Exp1B E0.png

This plot compares the total power / MHz in Bank E Pol 0 to the total power / MHz for every other bank/polarization combination in Experiment 1 (Mode 1 data hooked up directly to the noise source). The numbers quoted in the legend refer to the y-intercept from the linear fit to the data points. These data show an offset to zero power point of a few (1-2) percent. Should this type of offset be expected with VEGAS architecture? Plotting the same thing, but for the Mode 4 data, shows very similar relationships:

Exp2B E0.png

Note that the intercepts are comparable in sign and relative magnitude. The magnitudes in this plot have been scaled by the estimate of the LBW gain (5.664*2**16), which as can be seen does not account for the difference in observed power. I do not think this is an issue, as I assume there is more in the system than the lbw_gain to account for, but please correct me if that should have accounted for more of the difference between the two plots.

This analysis has been performed using the script fullTests.py in ~akobelsk/b2b/ . More plots can be found in that directory. More work on this tomorrow, but please feel free to help guide the next steps. I will begin to look into the VEGAS analog system tomorrow for comparison.

Conclusions

-- RichardPrestage - 2015-03-21

-- AdamKobelski - 2015-03-24
Topic attachments
I Attachment Action Size Date Who Comment
Exp1A.pngpng Exp1A.png manage 1 MB 2015-03-24 - 17:08 AdamKobelski Mode1atten
Exp1Azoom.pngpng Exp1Azoom.png manage 566 K 2015-03-24 - 18:10 AdamKobelski zoomed
Exp1B_E0.pngpng Exp1B_E0.png manage 764 K 2015-03-24 - 18:32 AdamKobelski  
Exp2A.pngpng Exp2A.png manage 912 K 2015-03-24 - 18:14 AdamKobelski mode4
Exp2B_E0.pngpng Exp2B_E0.png manage 736 K 2015-03-24 - 18:38 AdamKobelski  
Exp3A.pngpng Exp3A.png manage 891 K 2015-03-24 - 18:29 AdamKobelski mode 4 cables swapped
Exp4A.pngpng Exp4A.png manage 891 K 2015-03-24 - 18:30 AdamKobelski  
Topic revision: r13 - 2015-03-24, AdamKobelski
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback