VEGAS Pulsar Project Meeting - Sprint Review: 2017 December 4th 14:00 - 15:00pm ET

Present

  • Ray, Jason, Joe, Ryan, Laura

Sprint Story Overview

  • VE-305: Update release notes (INCOMPLETE)
  • VE-312: Matrix spectral line regression testing (INCOMPLETE)
  • VE-354: Troubleshoot performance (IN PROGRESS)
  • VE-355: Continue testing (IN PROGRESS)
  • VE-353: Testing over Thanksgiving shutdown (COMPLETE)

Story Details

VE-305: Update release notes

  • Waiting to complete acceptance tests

VE-312: Matrix spectral line regression testing

  • Waiting to complete software tests

VE-354: Troubleshoot performance

  • Every once and a while it appears as if the GPU thread doesn't keep up with incoming data in the spectral line modes.
  • Joe is still debugging this
  • If this happens the data will be blanked and you will see less integration time per phase

VE-355: Continue testing

  • See https://safe.nrao.edu/wiki/bin/view/CICADA/CICADAGreenBankSpectrometer2017dec01for detailed notes from latest run. Summary:
    • Significant issues
      • There still seemed to be some issues balancing VEGAS and the IF rack (though not sure if that is related to any M&C issues)
      • Still getting crashes in some of the FSW LBW modes.
      • Some banks in the coherent mode scans seemed to lose connection to GBT status database and have "Unknown" in the file name strings.
      • Lots of bad bandpasses today. I know there is filming going on at the telescope, so it is possible that this was locally generated RFI. That it was affecting HBW, LBW, INCO, and CODD modes on multipe IF paths/ROACH's in a very intermittent way seems to support this, though it is very unsatisfying to just blame RFI.
    • Minor issues
      • Still getting "Test message (please ignore)" from VEGAS
    • Issues for Ryan
      • Autoplotter is not reliably updating (but seemed OK after a restart and addition of debug statements)
      • Look into HPC monitor crash

VE-353: Testing over Thanksgiving shutdown

  • See https://safe.nrao.edu/wiki/bin/edit/CICADA/CICADAGreenBankSpectrometer2017nov22for detailed notes. Summary:
    • All and all, this was a very successful session.
      • Total power spectral line scans executed in all modes without any problems.
      • Most frequency switched scans executed without any problems. See below for a few exceptions.
      • Most incoherent cal, fold, and search mode pulsar scans executed without any problems once vegas.conf was modified to include paths for tempo. There are some potential issues with the 64 and 256 channel BOF files but these seem unrelated to the M&C system (see below for details).
      • Most coherent cal, fold, and search mode pulsar scans executed without any problems. See below for a few exceptions.
      • Related to the c1500 modes, a spot check of the data show no sign of the 1-ms dropped packet spurs. Some of the higher channel modes had high dropped packet rates, but this seems to be more closely related to data rates than networking issues.
      • Switching between spectral and pulsar modes went mostly smoothly, but there were some errors that we will need to look into more closely.
      • Offline validation works.
      • No issues with coherent modes while running the monitoring utilities.
      • c0800x0064 raw mode scan seemed to run well.
    • A few issues that we do need to look at:
      • VEGAS is regularly sending out "Test message (please ignore)". It would be good to turn this off.
      • Bank B seemed to have issues with some of the frequency switched scans. Specifically with modes 24 and 29.
      • Bandpasses in i0800x0064 and i1500x0064 are bad. This may be a case of having the wrong BOF file. Ryan will need to check this.
      • There seem to be a lot of artifacts in the i0800x0256 and i1500x0256 scans. Need to investigate if this is related to FFT overflow issues.
      • There were no packets being sent in the c1500x4096 mode. This seemed to be an issue with the ROACH. Need to investigate further.
      • During the mode-switching tests, there more frequent than usual errors while balancing, and a couple of manager crashes. Mostly related to illegal values of actual_switching_period.
      • All these errors seemed to self correct without any intervention but we'll need to look into the causes. Bank G also failed to activate in time during a coherent mode scan. That did not self correct and required an off/on cycle, but I was able to resume after that.
      • c1500x0064 raw mode scan had lots of dropped packets and NETSTAT was frequently blocked. Maybe a data rate issue.
    • Some minor things:
      • Need to check the current vegas.conf into the working branch, since I made changes to the path lists.
      • Need to ask Ray and Joe if samplers should now remain on for Banks B -- H in coherent modes.
      • Can probably remove DBUG and non-critical error messages in Manager logs.

VEGAS Pulsar Project Meeting - Sprint Planning: 2017 December 4th 2:00 - 3:00pm ET

Note: The team's "velocity" has been consistent at 11--13 points per sprint. We should aim for 12 points per sprint for planning purposes.

Backlog
  • High time resolution modes
  • Release notes
  • Continue work on Matrix LBW pulsar modes

Critical Issues

  • Fix balancing issues in Matrix modes
  • Fix frequency switched LBW spectral line modes (may already be done)
  • Identify a source tree for VPM utilities
  • Install VPM utilities in new source tree
  • Add a Task Master task for vpm_auto_plot, automatic VPM bandpass monitor, and a DAQ server used to monitor processes on the VPM HPCs in coherent dedispersion modes.

New Stories

-- RyanLynch - 2017-12-04
Topic revision: r2 - 2017-12-04, RyanLynch
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback