VEGAS Pulsar Project Meeting: 2016 April 25th 2:00 - 3:00pm ET

Room / Connection Details

  • GB-137 / Soc 280
  • 192.33.117.12##7144
  • 434-817-6443

Housekeeping Details

  • Main wiki page
  • gbsapp Mailing List
  • Location and regular date and time of meeting - agreed weekly on Mondays; GB-137 / Soc-280; 2-3pm ET
  • ETK codes:
    • Electronics: 432510.GB3015
    • Software: 893252.GB3015
    • Science: 432540.GB3015

Resource Allocations

BLOSS, MARTIN ALAN 0.05
BRANDT, JOE 0.10
CREAGER, RAY 0.50
DEMOREST, PAUL 0.05
RICHMOND-DECKER, JUSTIN 0.50
FORD, JOHN M 0.05
MCCULLOUGH, RANDY L 0.50
RAY, JASON 0.10
PRESTAGE, RICHARD 0.05
LYNCH, Ryan 0.25
VAN TONDER, VEREESE 0.20

Present

  • Marty, Randy, Ryan, Justin, Paul, Scott

Agenda

BOF file status updates

Coherent Modes

Mode BOF File Status
c1500x4096 c1500x4096_x14_7_2016_Feb_19_1357.bof GBT Testing; >95% dropped packets in simulator
c1500x2048 c1500x2048_x14_7_2016_Mar_02_1218.bof GBT Testing; >90% dropped packets in simulator
c1500x1024 c1500x1024_x14_7_2016_Mar_16_1434.bof GBT Testing; 22% dropped packets in simulator
c1500x0512 c1500x0512_x14_7_2016_Feb_24_1437.bof GBT Testing; 6% dropped packets in simulator
c1500x0256 c1500x0256_x14_7_2016_Mar_04_1344.bof GBT Testing
c1500x0128 c1500x0128_x14_7_2016_Mar_05_1423.bof GBT Testing
c1500x0064 c1500x0064_x14_7_2016_Mar_09_0820.bof GBT Testing
c0800x4096 c0800x4096_x14_7_2016_Mar_09_1415.bof GBT Testing; occasional < 0.1% packet loss in simulator
c0800x2048 c0800x2048_x14_7_2016_Mar_09_1445.bof GBT Testing
c0800x1024 c0800x1024_x14_7_mt01_2016_Apr_01_1216.bof GBT Testing; MMCM calibration problems
c0800x0512 c0800x0512_x14_7_2016_Mar_14_1421.bof GBT Testing
c0800x0256 c0800x0256_x14_7_2016_Mar_10_1136.bof GBT Testing; MMCM calibration problems
c0800x0128 c0800x0128_x14_7_2016_Mar_10_1144.bof GBT Testing
c0800x0064 c0800x0064_x14_7_mt01_2016_Apr_01_1209.bof GBT Testing
  • Outstanding issues
    • Packet loss in c1500x4096 and c1500x2048 modes not unexpected. Action item: Talk to Dave McMahon in April about optimization for BL that we can pull in for VPM.
    • Packet loss in c1500x0128 mode is somewhat surprising, given that higher-channel modes seem to work well.
      • Ryan to talk with Joe/Randy about possible reasons for this on the guppi_daq side.
    • MMCM calibration failing for zdok1 for c0800x1024 and c0800x0256
      • Low-level checks of MMCM calibration show glitches across all channels on the second ADC card.
      • Randy Suggested that placement of some blocks in these BOFs was not optimal. Two new builds to test this week.
    • Randy unsure how to build a 32-channel coherent mode
      • One clock cycle short; In 64 channel-mode 8 clock cycles are needed to read out a spectra. In 32 channel mode, 4 clock cycles are needed, which would require speeding up processing of data through the rest of the design
        • Paul and Scott wonder where the bottle neck is
          • Issue is at the end of the packet. There are always 3 64-bit words of overhead attached to each packet. There are not enough clock cycles to add all the spectra given this overhead.
          • Packetizing is handled slighly differently depending on number of channels.
      • Randy could add a short vector accumulator but this would actually average two spectra
        • Paul recommends not doing that, as it is like cutting the bandwidth of a channel in half
      • Randy could also add two additional memory banks, each interleaved.
        • There are enough resources on the roach for this, but it would make data handling more complex and likely lead to timing closure issues.
      • Could also drop FPGA DSP and allow everythign to be done on the GPU, but GPU probably can't process that much bandwidth.
      • This mode could be used for very high time resolution, but probably not a common use case.
        • Could develop if time at higher priority than incoherent 32-channel mode.

Incoherent Modes

ModeSorted descending BOF File Status
i1500x8192 i1500x8192_x14_7_2016_Apr_18_0901.bof GBT Testing
i1500x4096 i1500x4096_x14_7_2016_Apr_18_0932.bof GBT Testing
i1500x2048 i1500x2048_x14_7_2016_Apr_18_1509.bof GBT Testing
i1500x1024 i1500x1024_x14_7_2016_Apr_18_1521.bof GBT Testing
i1500x0512 i1500x0512_x14_7_2016_Apr_18_1253.bof GBT Testing
i1500x0256 i1500x0256_x14_7_2016_Apr_19_0753.bof GBT Testing
i1500x0128 i1500x0128_x14_7_2016_Apr_20_1411.bof GBT Testing
i1500x0064 i1500x0064_x14_7_2016_Apr_19_0813.bof GBT Testing
i0800x8192 i0800x8192_x14_7_2016_Apr_19_0843.bof GBT Testing
i0800x4096 i0800x4096_x14_7_2016_Apr_19_1038.bof GBT Testing
i0800x2048 i0800x2048_x14_7_2016_Apr_19_1048.bof GBT Testing
i0800x1024 i0800x1024_x14_7_2016_Apr_19_1054.bof GBT Testing
i0800x0512 i0800x0512_x14_7_2016_Apr_19_1104.bof GBT Testing
i0800x0256 i0800x0256_x14_7_2016_Apr_19_1242.bof GBT Testing
i0800x0128 i0800x0128_x14_7_2016_Apr_19_1249.bof GBT Testing
i0800x0064 i0800x0064_x14_7_2016_Apr_19_1256.bof GBT Testing
  • Outstanding Issues
    • Packet rate was 25% that of expected
      • Ryan tested new designs. Packet rate appears correct and BOFs look good in simmulator testing
    • Randy building i1500x0032 mode
      • Will require more detailed testing
      • Custom transform for channel ordering

Low bandwidth modes

  • Randy has built 100 MHz and 200 MHz BOFs
    • All channelization will happen in software
    • Need to check software register compatability
      • These are VEGAS designs, so Randy and Jason have not added all the software registers used in pulsar modes (and some do not make sense for the LBW modes)
      • Requires discussion with Justin to make sure software knows how to interface with these designs

Fast4K and Fast8K mmodes

  • GUPPI only uses Fast4K for 100 MHz modes
    • Probably does not require a separate hardware mode
    • Action item: Discuss how to implement in software

Software

  • Coherent mode scans are failing to initialize in the simulator
    • Justin tracked this down to changes to the valon synthezier that were being done when a scan was started. This cased the roach to freeze up.
    • Justin implemented a fix so that all operations on the valon happen while the roach is not programmed with a BOF. BOFs are now only loaded after all valon operations have been completed.
      • This has been applied to both spectral line and pulsar modes, and all BOFs.
      • Ryan has begun preliminary tests with coherent mode BOFs and they seem to be working well.
  • Configtool changes
    • Astrid still reports that VEGAS failed to balance when trying to adjust LBW gain. Still need to work on disabling this for pulsar modes.
    • Astrid still issues a warning about a discrepancy in the requested and actual IF rack power levels, though this did not cause a failure
      • This is because GUPPI and VEGAS have different power requirements. Dave Frayer suggested that VEGAS in wide bandwidth modes did not need to have it's targets set so high (currently 3 V). Could we use the same target (1 V) for GUPPI and VEGAS?
      • Justin has fixed this by setting target levels to 1 V for both GUPPI and VEGAS
        • Does this only apply in pulsar mode? Or in dual backend mode?
  • Spectral line modes
    • Shared memory parameters were not being updated properly. This is fixed
    • Bandpass shapes now also look good after adding back in control for new IF modules.
  • Matrix HPC
    • No updates
  • Low Bandwidth Modes
    • Justin and Ryan talked to Joe last week. Joe estimates more than one week of work but agrees that most of the code should exist.
      • Requires that the PFB/FFT be added to guppi_net_thread
      • Care needs to be taken to keep operations on the GPU for optimization
    • BOFs will be the same for coherent and incoherent mode; all the channelization and coherent dedispersion will happen in software.
  • Manager maintenace
    • Manager core dumps when killed with SIGINT
      • Justin still tyring to track down the cause, but does not interfere with data taking.

GBT testing

  • Test spectral line modes, coherent bank behavior, balancing, ADC calibration in coherent modes.

Wideband AP

  • Vereese and Jason are at WVU today looking for a student

VEGAS/DIBAS-style backend for FAST

  • Work progressing as independent project
  • Work starting. Possible delay in acquiring roach boards.

Multicasting for spectral line and pulsar modes

  • Modifications to ROACH configuration and net task must bind to a multicast group instead of a specific host
    • All the 10-gig addresses on the ROACH would be multicast groups, and config will be told which to bind to
    • Some changes to net task code (in VEGAS HPC and guppi_daq code) to initialize sockets

Date / Time of Next Meeting

  • May 2

-- RyanLynch - 2016-04-25
Topic revision: r3 - 2016-04-26, RyanLynch
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback