TGBT15A_915_128 - Matrix Adapter and Other Tests

Goals

  • More tests of the Matrix adapter that should allow for automatic switching between Matrix spectral line modes and Classic pulsar modes
  • Test config-tool support for 32-channel modes
  • Test changes to guppi_daq to use a new file naming format that includes UT seconds of midnight in the file name
  • Do A/B tests with out/with the guppi_daq server agent running on the HPCs to see if this triggers the incorrect MAC addresses on vegas-hpc3 and vegas-hpc4
  • Collect some new RAW mode data for Natalia
  • If time, look into c1500 packet loss issues

Details

  • Session begins 11:10
  • Ray begins switch to regression_test branch at 11:18
  • Version switch completed at 11:49
  • Will first test switching between spectral line and pulsar modes. Will use CheckModeN scripts for spectral line and CoherentModeTests/IncoherentModeTests for pulsar. Dual backend operation with GUPPI will not be used at first.
    • Scan #1 is Mode 1, submitted at 12:01
      • Scan ran successfully
    • Scan #2 is Mode 4, submitted at 12:02
      • Scan ran successfully
    • Scan #3 is i0800x0512 (8 banks), submitted at 12:06
      • Bank H did not configure for pulsar mode properly. Other banks seem OK. Data look OK (used an old BOF file by accident, but other than that seems good).
      • New file names with UTC start time look good!
      • Ray is investigating Bank H
      • In the meantime, I turned the manager Off/On to pull in an updated config file with the newer BOF
    • Scan #4 is i0800x0512 (8 banks) again, submitted at 12:19
      • Using newer BOF file this time
      • Scan ran better this time. Still seems to be artifacts in the cal scans. S/N is lower than it should be. Will need to investigate at some point.
    • Scan #5 is Mode 1, submitted at 12:24
      • Scan ran successfully
    • Scan #6 is c0800x0512, submitted at 12:26
      • In previous scan, Banks E-H were not in use and the managers were turned off. The managers came back on but the mode was left in incoherent cal, and not in coherent cal. ROACHs were left in service, but the banks were not balanced. The banks that were already on were configured properly for coherent cal. The Coordinator had the correct settings.
      • Scan completed with Banks A-D writing coherent mode data, and Banks E-H writing incoherent mode data.
      • After the scan completed, Ray did a conform params, and Banks E-H went back in the correct, coherent cal mode, and the ROACHs were taken out of service.
    • Scan #7 is Mode 1, submitted at 12:32
      • Scan ran successfully but the valon was not reset to the proper frequency on Banks B-D. Instead, it stayed at 800 MHz (appropriate for the previous pulsar scan). Bank A was configured properly in every way and its data look good.
      • The banks that were not in use in this spectral line scan remained in coherent pulsar mode, and the samplers on those ROACHs remained off. This is fine from an operational perspective, as long as it doesn't scare observers.
    • Scan #8 is c0800x0512, submitted at 12:36
      • Since the last scan that used Banks E-H was c0800x0512, there was no issue with them being configured properly this time.
      • Scan ran successfully
      • This SB was apparently configured for C-Band. S/N looks good, no RFI, but there seems to be a glitch in the diode. It looks like there is signal starting at a phase of 0.4, but at a lower amplitude, before the main square wave starts at phase 0.5. This appears at all frequencies and in all sub-integrations. There is no corresponding drop in the signal at phase 0.9, as you might expect if there was one cycle where the diode started or stopped too early. The glitch is present in the individual files from each HPC, so it is not an artifact from merging them together. The merge appears successful. The relative power of the off-phase pulse is about 0.5 that of the main pulse. It is spread out among many phase bins, so it is not just short glitch. It would have lasted 4 ms.
      • It would be good to try this mode again to see if the glitch reappears.
    • Scan #9 is Mode 1, submitted at 12:38
      • Valons on Banks B-D once again failed to reset to 1500 MHz
      • Ray thinks this may be related to the problem in coherent mode when the Banks are turned off to start with.
      • Conform params and prepare does not trigger a reset of the valon frequency, however.
      • Ray is investigating.
      • Ray isn't quite sure what is up, but he cycled the managers Off/On and the messages cleared.
    • Scan #10 is Mode 1, submitted at 12:52
      • Everything was fine this time
    • Scan #11 is i0800x0512, submitted at 12:54
      • Banks E-H were still in coherent mode from two scans previous. They did not configure properly for incoherent mode.
      • Tried to conform params and prepare while everything was configuring but the balance failed.
      • Scan aborted
      • Resubmitted at 12:57, but got a failed balance on Bank H
      • Resubmitted again at 12:58. Everything ran OK this time
    • Scan #12 is Mode 1, submitted at 13:00
      • Everything is getting set properly this time, data look good.
    • Scan #13 is i0800x0512, submitted at 13:04
      • Scan ran successfully
    • Scan #14 is c0800x0512, submitted at 13:06
      • Scan ran successfully
      • No cal glitch this time. There is a little bit of extra signal showing up for about 0.05 phases, at 4 times, spaced about 0.25 phases.
    • Scan #15 is i0800x0512, submitted at 13:09
      • Scan ran successfully. No issue switching between modes.
    • Scan #16 is Mode 1, submitted at 13:13
      • Scan ran successfully, data look fine. No issues.
    • Scan #17 is c0800x0512, submitted at 13:15
      • Once again, Banks E-H remained in incoherent, and ROACHs were not taken out of service. This was the last mode these banks ran in. Same behaviour as in scan #6 (same sequence leading up to it as well).
      • As with the last time, a conform params put everything in the right state.
    • Scan #18 is Mode 1, submitted at 13:22
      • Warnings reappearing about valon not being set properly.
      • Same behavior as in scan #7 (same sequence as well)
      • Ray was able to set the valon frequency by hand, but not through the manager
    • Ray cycled the managers Off/On, and the valon frequency was set properly
    • Made some changes needed for 32 channel coherent modes
    • Scan #19 is c0800x0512, submitted at 13:59
      • Ran OK
    • Scans #20 and 21 are c0800x0032 and c1500x0032, submitted at 14:00
      • Manager was complaining that the chosen number of channels is not allowed. No data taken.
  • Need to start switching back to 16.3 in time for holography checkout
  • Scans #20, 21, 22, and 23 are checkouts of Mode 1, 4, 10, and 20.
    • Modes 1, 10, and 20 look good. Mode 4 only had data for 4 IFs in gbtidl but should have had 7. Ran again, and it looked OK.
    • Scan #24 ran in Mode 4 but nothing showed up in gbtidl.
    • Scan #25 ran fine in Mode 4.
  • Session ends at 14:27

Conclusions

  • There seems to be a problem whenever we go between two modes, and some of the managers are turned off or the ROACH is out of service. It seems like parameters are being set properly in the Coordinator but this is not always communicated the Managers, so they are often not setting up properly for the chosen mode. We see this when going from a spectral line mode that doesn't use all the banks to a coherent mode, as well as from a coherent mode where the ROACHs are not in service on Banks B-H to a spectral line mode, in which case all but Bank A get the wrong valon frequency. There does not seem to be an issue with going back and forth between spectral line and incoherent mode, or with going between incoherent and coherent modes.
  • The guppi_daq version with the UTC seconds after midnight encoded in the file names worked just fine. File names were as expected.
  • Started trying to test 32 channel modes but ran into some config tool and manager issues.
  • Did not have time to look at MAC address issues in detail, although we saw no errors when running VEGAS only without any of the daq_servers for checking the status of the HPCs.
-- RyanLynch - 2017-11-03
Topic revision: r2 - 2017-11-03, RyanLynch
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback