VE-302: Configure Manager to support Matrix and Classic HPCs (NO UPDATES)
VE-309: Prepare for final regression test (COMPLETE)
VE-295: Reinstate nchan software register in 32-channel modes (COMPLETE)
VE-303: Review list of changes that are needed for CLEO (COMPLETE)
VE-301: Complete spectral line regression testing (IN PROGRESS)
VE-306: Fix FPGA design offset issue
Randy found a delay that would have caused a 375 MHz offset
Ryan has tested a new build and the shift is not present
VE-281/285: Continue BOF file testing
Ryan has tested all the new BOF files designs supplied by Luke, Randy, and Jason.
The incoherent 800 and 1500 MHz modes with 512 channels or greater all pass basic tests. The bandpasses are as expected and the cal phase across the band looks reasonable.
The incoherent 800 and 1500 MHz modes 128 and 256 channels seem mostly OK, but there may be FFT overflow issues, especially in the 256-channel modes for each bandwidth. Need to explore different fftshift values again.
The incoherent 800 and 1500 MHz modes with 64 channels had some bad bandpasses. These need to be checked out in more detail.
The coherent 800 MHz modes all looked good (i.e., good bandpasses and cal phases). We still had some packet loss in the high-nchan modes, but this is nothing new.
The coherent 1500 MHz modes had good bandpasses and the cal phase was well behaved, but there is still 1-ms packet loss artifacts in all the c1500 modes except c1500x0064
VE-292: Investigate BOF files for possible source of 1 ms artifacts
VE-310: High time resolution modes
VE-300: Troubleshoot Manager config issue
Astrid complains that balancing fails when trying to use GUPPI and VEGAS. To recreate this
Configure for both GUPPI and VEGAS with "backend = 'VEGAS/GUPPI'"
Issue a Balance() command. This will balance GUPPI but not VEGAS.
Issue a Balance("VEGAS") command. The power levels on VEGAS then seem OK but Astrid complains that balance command failed and offers to abort. If the abort is ignored, scans continue and data look OK.
This version of config tool also needs to allow 32-channel pulsar modes
VE-298: Continue debugging pulsar mode
Ray identified two Manager problems that were probably causing excessive packet loss on most of the VEGAS banks in coherent dedispersion modes.
The manager was not using the correct name for the ethernet register on the BOF when loading the ARP table. This meant that the BOF did not have the correct MAC addresses of the HPC computers.
When banks B--H receive a message to program the BOF file coherent dedispersion modes they ignore it, which is the correct behavior as they don't control the Roach that is the data source. However, they should unload the BOF on the Roach that they do control to prevent that Roach from sending out packets.
In testing, Ryan was seeing "hpc taking too long to be ready" errors in many coherent and incoherent dedispersion modes. Increasing the "need_arm_delay" time in vegas.conf seems to have solved this issue.
Fixing the above issues eliminated the excessive packet loss for most coherent modes. These now work as they did with Justin's version of the Manager.
Ray also fixed an issue that was preventing shared memory from being locked, and that was causing the CALMODE parameter to be set incorrectly.
There was a separate issue with incoherent dedispersion search mode in which scans would end, but guppi_daq would not properly close the output files. This has also now been fixed.
Ryan needs to do 1--2 more rounds of testing before moving on to on-sky acceptance tests.
VE-302: Configure Manager to support Matrix and Classic HPCs
Next on Ray's list of things to do. Expects to get to this in the current sprint.
VE-309: Prepare for final regression test
Ryan polled the scientific staff and investigated historical use of VEGAS to inform which modes we would focus on for final spectral line regression testing.
The scientific staff suggested heavily testing setups used for KFPA NH3 mapping, ARGUS, and RRL projects, as well as testing behavior when doing fast frequency switching.
Ryan also produced the following census of VEGAS mode usage and based on this decided to focus on mode 1, 4, 10, 20, and 28.
VE-295: Reinstate nchan software register in 32-channel modes
VE-303: Review list of changes that are needed for CLEO
Ron, Ryan, and Ray believe we have a version of CLEO that supports the new Manager and pulsar modes, and that works with classic VEGAS.
VE-301: Complete spectral line regression testing
Dave performed before/after tests with ARGUS to check for any changes in baseline shapes. He is happy with the results and has given his OK to go ahead with the release of the improved Manager and Matrix backend. He reported many fewer VEGAS aborts when using the new Manager, which is consistent with Richard's experience.
Richard's additional spectral line regression tests were hampered by work on the SCU.
Testing did reveal some problems.
Ray and Joe found LBW gains were not being changed by balancing
The issue is that config tool was not updating these parameters. Seems to have been introduced when modes were changed to enums.
The error was missed because of a try/except statement
Ray thinks he has a quick fix and can also do a more detailed review of the code to look for additional issues.
Richard found an intermittent problem that affects mode 4 and 10 but not 20 and 28. Ray believes it affects all LBW modes.
Richard believes targeted Rx tests need to look at an astronomical source to truly verify that these are working.
Richard found that some of the (supposedly) in-use configurations used by projects that Toney suggested give warnigns or do not seem to configure successfully at all. This is not specific to VEGAS manager/HPC changes but raises questions about how best to do regression tests.
Richard thinks we really need motion tests in good weather for true astronomical commissioning.
Best to revisit this once we are confident engineering tests show no problems.
VEGAS Pulsar Project Meeting - Sprint Planning: 2017 August 14th 2:00 - 3:00pm ET
Note: The team's "velocity" has been consistent at 11--13 points per sprint. We should aim for 12 points per sprint for planning purposes.
High time resolution modes
Investigate packet loss in c1500 modes
Reinstate nchan software register in 32 channel modes
Final pulsar mode tests
Richard cannot validate some scheduling blocks when online
Fix VEGAS LBW gain balancing
Complete spectral line regression tests and release new Manager/Matrix backend for spectral line modes
Settle on best values for fftshift in pulsar modes to avoid FFT overflow artifacts
Fix incoherent 64-channel modes
Confirm that multi-backend balancing with GUPPI and VEGAS works.
Fix Astrid errors about being unable to balance VEGAS when configuring for GUPPI and VEGAS
Automate pulsar mode data combination procedure
Set-up process to continuously run pulsar mode data viewer webpage
Possibly get to acceptance tests of pulsar modes
Possibly modify the VEGAS power sampler that is reported in CLEO VEGAS to display output VEGAS levels to ensure that balancing is indeed correct
Joe beginning work on Matrix pulsar modes
Luke wants to talk about appending delays in BOF files