VEGAS Pulsar Project Meeting - Sprint Review: 2017 August 30th 11:00 - 12:00am ET

Present

• Laura, Joe, Mark, Ray, Richard, Luke, Randy

Sprint Story Overview

• VE-306: Fix FPGA design offset issue (COMPLETE)
• VE-281/285: Continue BOF file testing (COMPLETE)
• VE-292: Investigate BOF files for possible source of 1 ms artifacts (NO UPDATES)
• VE-310: High time resolution modes (NO UPDATE)
• VE-300: Troubleshoot Manager config issue (COMPLETE)
• VE-298: Continue debugging pulsar modes (COMPLETE)
• VE-302: Configure Manager to support Matrix and Classic HPCs (NO UPDATES)
• VE-309: Prepare for final regression test (COMPLETE)
• VE-295: Reinstate nchan software register in 32-channel modes (COMPLETE)
• VE-303: Review list of changes that are needed for CLEO (COMPLETE)
• VE-301: Complete spectral line regression testing (IN PROGRESS)

Story Details

VE-306: Fix FPGA design offset issue

• Randy found a delay that would have caused a 375 MHz offset
• Ryan has tested a new build and the shift is not present

VE-281/285: Continue BOF file testing

• Ryan has tested all the new BOF files designs supplied by Luke, Randy, and Jason.
  • The incoherent 800 and 1500 MHz modes with 512 channels or greater all pass basic tests. The bandpasses are as expected and the cal phase across the band looks reasonable.
  • The incoherent 800 and 1500 MHz modes 128 and 256 channels seem mostly OK, but there may be FFT overflow issues, especially in the 256-channel modes for each bandwidth. Need to explore different fftshift values again.
  • The incoherent 800 and 1500 MHz modes with 64 channels had some bad bandpasses. These need to be checked out in more detail.
  • The coherent 800 MHz modes all looked good (i.e., good bandpasses and cal phases). We still had some packet loss in the high-nchan modes, but this is nothing new.
  • The coherent 1500 MHz modes had good bandpasses and the cal phase was well behaved, but there is still 1-ms packet loss artifacts in all the c1500 modes except c1500x0064

VE-292: Investigate BOF files for possible source of 1 ms artifacts

• No updates

VE-310: High time resolution modes

• No updates

VE-300: Troubleshoot Manager config issue

• Astrid complains that balancing fails when trying to use GUPPI and VEGAS. To recreate this
  • Configure for both GUPPI and VEGAS with "backend = 'VEGAS/GUPPI'"
  • Issue a Balance() command. This will balance GUPPI but not VEGAS.
  • Issue a Balance("VEGAS") command. The power levels on VEGAS then seem OK but Astrid complains that balance command failed and offers to abort. If the abort is ignored, scans continue and data look OK.
• This version of config tool also needs to allow 32-channel pulsar modes

VE-298: Continue debugging pulsar mode

• Ray identified two Manager problems that were probably causing excessive packet loss on most of the VEGAS banks in coherent dedispersion modes.
  • The manager was not using the correct name for the ethernet register on the BOF when loading the ARP table. This meant that the BOF did not have the correct MAC addresses of the HPC computers.
  • When banks B--H receive a message to program the BOF file coherent dedispersion modes they ignore it, which is the correct behavior as they don't control the Roach that is the data source. However, they should unload the BOF on the Roach that they do control to prevent that Roach from sending out packets.
• In testing, Ryan was seeing "hpc taking too long to be ready" errors in many coherent and incoherent dedispersion modes. Increasing the "need_arm_delay" time in vegas.conf seems to have solved this issue.
• Fixing the above issues eliminated the excessive packet loss for most coherent modes. These now work as they did with Justin's version of the Manager.
• Ray also fixed an issue that was preventing shared memory from being locked, and that was causing the CALMODE parameter to be set incorrectly.
• There was a separate issue with incoherent dedispersion search mode in which scans would end, but guppi_daq would not properly close the output files. This has also now been fixed.
• Ryan needs to do 1--2 more rounds of testing before moving on to on-sky acceptance tests.

VE-302: Configure Manager to support Matrix and Classic HPCs

• Next on Ray's list of things to do. Expects to get to this in the current sprint.

VE-309: Prepare for final regression test

• Ryan polled the scientific staff and investigated historical use of VEGAS to inform which modes we would focus on for final spectral line regression testing.
  • The scientific staff suggested heavily testing setups used for KFPA NH3 mapping, ARGUS, and RRL projects, as well as testing behavior when doing fast frequency switching.
  • Ryan also produced the following census of VEGAS mode usage and based on this decided to focus on mode 1, 4, 10, 20, and 28.

VE-295: Reinstate nchan software register in 32-channel modes

• Complete

VE-303: Review list of changes that are needed for CLEO

• Ron, Ryan, and Ray believe we have a version of CLEO that supports the new Manager and pulsar modes, and that works with classic VEGAS.

VE-301: Complete spectral line regression testing

• Dave performed before/after tests with ARGUS to check for any changes in baseline shapes. He is happy with the results and has given his OK to go ahead with the release of the improved Manager and Matrix backend. He reported many fewer VEGAS aborts when using the new Manager, which is consistent with Richard's experience.
• Richard's additional spectral line regression tests were hampered by work on the SCU.
• Testing did reveal some problems.
  • Ray and Joe found LBW gains were not being changed by balancing
  • The issue is that config tool was not updating these parameters. Seems to have been introduced when modes were changed to enums.
  • The error was missed because of a try/except statement
  • Ray thinks he has a quick fix and can also do a more detailed review of the code to look for additional issues.
• Richard found an intermittent problem that affects mode 4 and 10 but not 20 and 28. Ray believes it affects all LBW modes.
• Richard believes targeted Rx tests need to look at an astronomical source to truly verify that these are working.
  • Richard found that some of the (supposedly) in-use configurations used by projects that Toney suggested give warnigns or do not seem to configure successfully at all. This is not specific to VEGAS manager/HPC changes but raises questions about how best to do regression tests.
  • Richard thinks we really need motion tests in good weather for true astronomical commissioning.
  • Best to revisit this once we are confident engineering tests show no problems.

VEGAS Pulsar Project Meeting - Sprint Planning: 2017 August 14th 2:00 - 3:00pm ET

Note: The team's "velocity" has been consistent at 11--13 points per sprint. We should aim for 12 points per sprint for planning purposes.

Backlog

• High time resolution modes
• Investigate packet loss in c1500 modes
• Reinstate nchan software register in 32 channel modes

Critical Issues

• Final pulsar mode tests

New Stories

• Richard cannot validate some scheduling blocks when online
• Fix VEGAS LBW gain balancing
• Complete spectral line regression tests and release new Manager/Matrix backend for spectral line modes
• Settle on best values for fftshift in pulsar modes to avoid FFT overflow artifacts
• Fix incoherent 64-channel modes
• Confirm that multi-backend balancing with GUPPI and VEGAS works.
• Fix Astrid errors about being unable to balance VEGAS when configuring for GUPPI and VEGAS
• Automate pulsar mode data combination procedure
• Set-up process to continuously run pulsar mode data viewer webpage
• Possibly get to acceptance tests of pulsar modes
• Possibly modify the VEGAS power sampler that is reported in CLEO VEGAS to display output VEGAS levels to ensure that balancing is indeed correct
• Joe beginning work on Matrix pulsar modes
• Luke wants to talk about appending delays in BOF files

-- RyanLynch - 2017-09-01