VEGAS Pulsar Project Meeting - Sprint Review: 2017 April 3rd 2:00 - 3:00pm ET

Present

• Ryan, Jason, Joe, Ray, Natalia, Randy

Sprint Story Overview

• VE-204: Mount 2nd ROACH and HPC in test/dev environment (PARTIALLY COMPLETE)
• VE-206: Get rebased VPM M&C branch running on simulator (COMPLETE; FIXING BUGS)
• VE-207: Add book keeping info for PFB component of matrix (IN PROGRESS)
• VE-208: Test performance of PFB/Thrust (INCOMPLETE)
• VE-210: Update packet generator to support switching signals (COMPLETE)
• VE-197: Continue work on high time resolution modes (IN PROGRESS)
• VE-159: Troubleshoot data visualization tools (DONE)
• VE-205: Get all spectral line modes to load and initialize correctly (IN PROGRESS)
• VE-203: Clone /home/sim for regression testing
• VE-187: Trouble shoot bad Stokes U and V data (DONE)
• VE-209: Install release candidate in test environment

Story Details

VE-187: Troubleshoot bad Stokes U and V data

• With much help from Dave MacMahon we were able to isolate and solve the source of the problem.
• By investigating ADC snap block data we determined that there was a 288 ADC sample delay between the X and Y data coming out of the ADCs. This absolved any issue with downstream firmware/software.
• We also determined that the delay was not present in Breakthrough Listen data that used the same VEGAS BOF file.
• Since BTL does not currently perform MMCM calibration, we began investigating the MMCM calibration routine in detail.
• The X/Y delays were not seen when no MMCM calibration was performed. They were also not seen when MMCM calibration was performed manually using Paul M.'s python script. But they were seen when MMCM calibration was performed with the Manager.
• Dave was able to look at the individual commands sent to the ROACH by the manager and eventually isolated the problem.
  • The LSB in some of the commands sent to the ROACH was set to a constant value of 1. This was appropriate when communicating with the ADC's SPI bus, as it sent a "strobe" command to the SPI. Strobe command tells the BOF to read-out of each parallel port on the SPI.
  • When writing to the ROACH registers not on the SPI, the high state of the LSB incorrectly held the ADC in a reset mode.
  • We believe that this broke phase lock between the two ADC clocks. Phase lock was required to trigger readout of two FIFOs, one for each polarization channel. We hypothesize that without phase lock, one of the FIFOs simply accumulated ADC samples and only started to read out when the FIFO was full, or reach some other condition sufficient to trigger a read out. The depth of the FIFO is not 288 ADC samples so it isn't entirely clear why this was the magnitude of the delay.
  • Ray patched the MMCM code to set the LSB to the appropriate value.
  • Data taken with the patched code still had some small X/Y delay (it seems to vary from 16-28 samples between scans) but looks far more consistent with GUPPI.
• We believe this issue would have affected all VEGAS observing since the implementation of MMCM calibration, but most spectral line observers do not look at polarization cross products and so probably would not notice.
• Patched code is now released and this issue is considered closed.

VE-204: Mount 2nd ROACH and HPC in test/dev environment

• 2nd ROACH is installed. We are trying to identify a 2nd HPC to pair with the ROACH but the machine currently known as west seems like a good candidate. It has two 10-gig cards but needs an upgraded GPU and memory.
• Jason is coordinating with Wolfgang.
• We will likely rename the system.

VE-206: Get rebased VPM M&C branch running on simulator

• VPM branch is installed on simulator
• Spectral line modes have been tested but modes 10 -- 19 (L8/LBW1) are still not recording data.

VE-207: Add book keeping info for PFB component of matrix

• This includes data for many parameters (e.g. which subband to write, scan number, switching, etc.)
• Data is being transmitted through matrix framework but FITS writer for matrix was adopted from DIBAS
  • Needs various "GBTisms" added
  • Joe plans to focus on getting FITS files written correctly and then we can inspect data. (Probably a couple weeks off).

VE-208: Test performance of PFB/Thrust

• Incomplete

VE-210: Update packet generator to support switching signals

No notes.

VE-197: Continue work on high time resolution modes

• Natalia is working on writing some tools to read in raw files for analysis

VE-159: Troubleshoot data visualization tools

• Ryan found a simple fix for the "vegas_monitor" program that allows it to read data buffers and plot real-time bandpass.
  • Still need to test in live system to make sure data packet headers are being populated correctly, but this is not expected to be an issue
• We decided that the equivalent of "guppi_gpu_status", which prints guppi_daq log info and information on NETSTAT, DROPTOT, and DROPAVG for each GPU node will instead by incorporated into error capturing by the Manager. This has three main advantages:
  • It does not require setting up servers on each HPC to communicate with a head node (GUPPI currently runs separate servers on each GPU node using syscntld)
  • It allows for better error capturing and behavior, such as raising faults or abort signals
  • It more fully integrates VEGAS with the existing M&C framework

VE-205: Get all spectral line modes to load and initialize correctly

• See VE-206 (still debugging modes 10 -- 19)

VE-203: Clone /home/sim for regression testing

• Incomplete

VE-209: Install release candidate in test environment

• Waiting on VE-205

Oustanding issues

• Get regression test environment ready for use by the time a release candidate is ready
• Prepare a VPM M&C release candidate for spectral line regression testing

VEGAS Pulsar Project Meeting - Sprint Planning: 2017 April 3rd 2:00 - 3:00pm ET

Note: The team's "velocity" has been consistent at 11--13 points per sprint. We should aim for 12 points per sprint for planning purposes.

Backlog

• Get all regression test environment hardware and software
• Continue work on acquiring hardware for upgraded development environment
• Prepare a release candidate for the re-based VPM branch that can run spectral line modes as expected
• Spectral line regression tests of release candidate
• Continue work on high time resolution modes

Critical Issues

• M&C system and regression tests

New Stories

• Continue work on spectral line modes in VPM branch
• Regression testing if possible
• High time resolution
• Matrix work: book keeping, FITS writer normalization, manager interface, fix bug in accumulator. Hoping to put together spectral line modes in next 2--3 weeks.
• Digital: Work on 32-channel mode. Randy is working on a simulator for ADC data.
  • Need to rebuild BOF files with truncate instead of wrap behavior in offset/gain stage
• Post-processing: Upgrade search merging scripts to handle overlapping/broken frequency coverage
• Collect all observing tools/scripts into a single software area and make available to users
• Clone /home/sim for regression test environment

-- RyanLynch - 2017-04-03