TGBT15A_915_88 - Test for FFT Overflow and Source of Artifacts

Goals

  • Test for FFT overflow using a new BOF file provided by Randy and Jason (c0800x0512_fft_ovf_01_x14_7_2017_Feb_16_1132.bof)
    • If FFT is overflowing, try to eliminate with different values of fftshift
  • Try to isolate an RFI source that may be causing artifacts with use of narrowband filters
  • If time, try to excite spurs using noise source + test tone

Details

Session begins at 09:32
  • Switched VEGAS to pulsar branch
    • Astrid had trouble connecting to SB executor at first
    • Had to restart Grail
  • Performing initial scan using standard BOF file (c0800x0512_x14_7_2016_Mar_14_1421.bof), to make sure artifacts are present today. GUPPI is also used.
    • Using 00-CoherentModeArtifactTesting scheduling block
    • Set up for L-Band at fctr=1500 MHz
    • SB submitted at 09:46
    • Scan is #1
    • VEGAS failed to balance but levels look OK
    • Bank D did not write any data, but no error messages. Just seeing a lot of dropped packets.
    • SB ended at 09:48
    • Artifacts do seem to be present
    • Taking another scan to try and get data from Bank D
    • SB submitted at 09:49
    • Scan is #2
    • VEGAS failed to balance again
    • Data is being taken on Bank D this time
    • SB ended at 09:51
    • Artifacts are present
  • Switching to new BOF file (c0800x0512_fft_ovf_01_x14_7_2017_Feb_16_1132.bof)
    • Using 00-CoherentModeArtifactTesting scheduling block
    • Set up for L-Band at fctr=1500 MHz
    • BOF file is being used
    • SB submitted at 09:54
    • Balance failed
    • Banks faulted and aborted
    • All banks failed to start VEGAS HPC subprocess
    • Trying a system stop/start
    • Resubmitting SB at 09:58
    • Running successfully this time
    • Scan is #4
    • Artifacts are still present
    • Jason reports that FFT is overflowing
  • Taking another scan with fftshift set to all f's (i.e. fftshift = 0xffffffff)
    • New fftshift was successfully pulled in
    • Using 00-CoherentModeArtifactTesting scheduling block
    • Set up for L-Band at fctr=1500 MHz
    • SB submitted at 10:04
    • Balance still failing
    • Banks aborted again with HPC subprocess taking too long to be ready
    • Doing a system stop/start
    • Resubmitting SB at 10:09
    • Running successfully this time
    • Scan is #6
    • SB ended at 10:11
    • Now just seeing noise (not even the diode). Probably shifted by too much, so now everything is in the noise.
    • We are still seeing some spurs at a low level (similar to when we only had the tone generator hooked up)
  • Trying a new fftshift which is halfway between all a's and all f's
    • Setting fftshift = 0xD5555554
    • Just going straight to system stop/start this time
    • Jason reports that the indicator light is showing an FFT overflow
    • Using 00-CoherentModeArtifactTesting scheduling block
    • Set up for L-Band at fctr=1500 MHz
    • SB submitted at 10:17
    • Abort from Bank B
    • Doing a stop/start on bank B only
    • Had to restart Astrid and Turtle
    • Resubmitted SB at 10:25
    • Ran successfully
    • Scan is #8
    • SB ended at 10:27
    • Artifacts are present
  • Taking a break at 10:25 so that Ryan can meet with new prospective software group hire.
  • Restarting at 11:04
  • Setting fftshift = 0xeaaaaa9
    • Did a system stop/start
    • Using 00-CoherentModeArtifactTesting scheduling block
    • Set up for L-Band at fctr=1500 MHz
    • SB submitted at 11:08
    • Scan is #9
    • SB ended at 11:10
    • Used wrong shift (needed one more a)
    • Artifacts are present, power levels were fluctuating a lot
    • Using correct shift (0xeaaaaaa9)
    • SB submitted at 11:14
    • Scan is #10
    • Power levels look more stable
    • SB ended at 11:16
    • Artifacts still present
  • Setting fftshift = 0xeaaaaaaf
    • Did a system stop/start
    • Using 00-CoherentModeArtifactTesting scheduling block
    • Set up for L-Band at fctr=1500 MHz
    • SB submitted at 11:26
    • Scan is #11
    • Scan ended at 11:27
    • Jason reports that there was overflow but that the LED indicator was not always on, so perhaps it wasn't always overflowing
    • There are still some artifacts but they are not as prominent. It may only be the highest power bins that overflowing.
  • Setting fftshift = 0xe555559f
    • Did system stop/start
    • Using 00-CoherentModeArtifactTesting scheduling block but with vegas.scale = 5.0 to compensate for lower power through FFT
    • Set up for L-Band at fctr=1500 MHz
    • SB submitted at 11:35
    • Scan is #12
    • SB ended at 11:36
    • Jason reports that LED was still blanking with lower duty cycle
    • Data look odd. Noise diode appears to shift and drop out at certain phases. Some search mode data might be good here. But artifacts seem much reduced.
    • Taking another scan
    • SB submitted at 11:42
    • Scan is #13
    • SB ended at 11:44
    • Breaking for lunch
    • Still seeing that weird shift in the cal. But at first glance the artifacts look much better.
    • Trying another scan with a tfold of 1 s
    • SB submitted at 13:09
    • Scan is #14
    • SB ended at 13:11
    • Looks like there may have been a little bit of artifacts this time but no cal phase shift
  • Setting fftshift = 0xe5555abf
    • Did system stop/start
    • Using 00-CoherentModeArtifactTesting scheduling block but with vegas.scale = 20.0 to compensate for lower power through FFT
    • Set up for L-Band at fctr=1500 MHz
    • SB submitted at 13:17
    • Scan is #15
    • SB ended at 13:18
    • Jason reports that FFT is overflowing maybe 5% of the time
    • Minimal artifacts, no phase shift
  • Setting fftshift = 0xe555557f
    • Did system stop/start
    • Using 00-CoherentModeArtifactTesting scheduling block but with vegas.scale = 40.0 to compensate for lower power through FFT
    • Set up for L-Band at fctr=1500 MHz
    • SB submitted at 13:30
    • Scan is #16
    • SB ended at 13:33
    • Still get occasional overflows
    • Minimal to know artifacts. Possible getting some features from the cal. vegas.scale could have been lower
  • Setting fftshift = 0xaaaaaaff
    • Did system stop/start
    • Using 00-CoherentModeArtifactTesting scheduling block but with vegas.scale = 40.0 to compensate for lower power through FFT
    • Set up for L-Band at fctr=1500 MHz
    • SB submitted at 13:50
    • Scan is #17
    • SB ended at 13:52
    • Jason did not see any overflow from the LED indicator light
    • No obvious artifacts, possible some ripple in the cal profile. Possible some sort of phase shifting in first integration. But overall looks pretty decent
  • We seem to have found a workable shift pattern. I am going to reset the shift pattern to all a's and play around with narrow band filters in the IF rack to see if I can isolate an RFI source that causes the artifacts.
  • Setting fftshift = 0xaaaaaaaa, resetting vegas.scale to 2.25
  • Will try this with the incoherent cal mode
  • Taking a scan to verify that the artifacts are back
    • Using 01-IncoherentModeArtifactTesting scheduling block
    • Set up for L-Band at fctr=1500 MHz
    • SB submitted at 14:04
    • Scan is #18
    • SB ended at 14:06
    • Artifacts are present
  • Setting IF rack filter from 2360-3640 to 2840-3160. This should let the central 320 MHz through and get rid of the RFI just below the notch filter
    • Using 01-IncoherentModeArtifactTesting scheduling block
    • Set up for L-Band at fctr=1500 MHz
    • SB submitted at 14:11
    • Scan is #19
    • SB ended at 14:13
    • Filter was successful. Artifacts are still present. This may suggest that the RFI around 1500-1600 MHz is the culprit.
  • Keeping IF filter at 2840-3160 but modifying fctr to be 1360. This should include the notch filter but keep out most of the RFI above 1520 MHz.
    • Using 01-IncoherentModeArtifactTesting scheduling block
    • Set up for L-Band at fctr=1360 MHz
    • SB submitted at 14:16
    • Scan is #20
    • SB ended at 14:18
    • Data are noisy. It looks like the artifacts may still be present, but they don't really seem obvious in the bandpass where we have sensitivity. We are still getting RFI around 1550 MHz, so maybe we need to use a narrow filter.
  • Using 2960-3040 MHz filter centered on 1400 MHz
    • Using 01-IncoherentModeArtifactTesting scheduling block
    • Set up for L-Band at fctr=1400 MHz
    • SB submitted at 14:22
    • Scan is #21
    • SB ended at 14:25
    • No obvious artifacts.
  • Going to try the 2960-3040 MHz filter but now centered on 1600 MHz. This will allow a bunch of RFI through.
    • Using 01-IncoherentModeArtifactTesting scheduling block
    • Set up for L-Band at fctr=1600 MHz
    • SB submitted at 14:30
    • Scan is #22
    • SB ended at 14:32
    • Artifacts seem to be present, though not in every channel. I think we are seeing the 50 MHz repeating signal, though. This suggest the RFI at around 1600 MHz is the culprit.
  • Going to do something weird. Will set fctr=1400 MHz and will using the 2360-3640 MHz filter in the X polarization, but the 2960-3040 MHz filter in the Y polarization. I am doing this because Dave MacMahon seemed to see the artifacts mostly in the Y polarization.
    • Using 01-IncoherentModeArtifactTesting scheduling block
    • Set up for L-Band at fctr=1400 MHz
    • SB submitted at 14:36
    • Scan is #23
    • SB ended at 14:38
    • Artifacts are still present.
  • Will do the same as above but with the 2960-3040 MHz filter only in the X polarization. Maybe Dave had the polarizations flipped.
    • Using 01-IncoherentModeArtifactTesting scheduling block
    • Set up for L-Band at fctr=1600 MHz
    • SB submitted at 14:38
    • Scan is #24
    • SB ended at 14:40
    • Still seeing artifacts. So it doesn't seem like it is polarization specific.
  • There is also strong RFI at around 1100 MHz. Let's see what happens when we let this in with the 2960-3040 MHz filter in both polarizations.
    • Using 01-IncoherentModeArtifactTesting scheduling block
    • Set up for L-Band at fctr=1100 MHz
    • SB submitted at 14:43
    • Scan is #25
    • SB ended at 14:45
    • Tons of artifacts. Looks like crap. This suggests the 1100 MHz RFI is also a big problem (maybe the biggest)
  • Going to step in frequency from 1100 MHz to 1900 MHz in 40 MHz increments, setting the IF rack filter to 2960-3040 MHz in an automated way with SetValues. Let's see if we systematically determine which frequency ranges are bad.
    • Using 02-IncoherentModeArtifactTesting scheduling block
    • First scan will be #26, last scan will be #46
    • SB submitted at 14:55
    • Note that we still have been having VEGAS balancing issues...
    • vegas.scale values seem to be kind of high in general.
    • Unexplained abort on last scan. Continued and data seem OK.
    • SB ended at 15:38
    • We definitely see artifacts around 1100 MHz.
  • I think that is enough for today. Switching back to release branch at 15:47
  • Taking verification scan in spectral line mode
    • Using CheckMode1 scheduling block
    • SB submitted at 15:51
    • Scan is #46 (duplicate with a pulsar scan, probably because of that last abort message)
    • Data look OK
Session ended at 15:53

Conclusions

Progress has been made. We seem to have determined that the artifacts are due to overflow somewhere early in the FFT chain. We have determined some fftshift values that, at least today, for the current RFI conditions, minimize the amount of overflow without seeming to impact anything else in an obvious way (though we still need to understand why some of the scans had odd signals at certain pulse phases). We further seemed to show that the overflow/artifacts do not occur in a relatively clean RFI environment but are pretty bad when strong RFI is present. We also seemed to show that RFI at around 1100 MHz, 1600 MHz, and 1800 MHz, with 1100 and 1800 MHz being the most problematic. I think we now have a pretty clear picture of what is causing the overflow, why it is variable and seen only at L-Band (so far), and what the problematic sources of RFI are. Now the question is, how to mitigate this most effectively.

-- RyanLynch - 2017-02-17
Topic revision: r3 - 2017-02-18, RyanLynch
 

This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback