TGBT15A_915_123 - Check Balancing in Pulsar and Spectral Line Modes

Goals

  • In previous tests, VEGAS does not independently balance when used in dual backend mode with GUPPI. Issuing a Balance("VEGAS") causes Astrid to report that balancing has failed, with an option to abort, even though the IF rack appears to be balanced. Ray made changes to the balancing routine so today we will try to test these and resolve any remaining balancing issues.
  • In previous spectral line tests, the LBW gain balancing failed in a quiet and not obvious way. We will check that it is now balancing properly.

Details

  • Session begins at 10:00
  • Ray switched versions to the regression test candidate
  • Will use the CoherentModeTests scheduling block for VPM balancing tests
  • SB submitted at 10:35
    • Scan #1 is c0800x0512
    • Using a simple Balance() command, i.e. no separate Balance("VEGAS")
    • GUPPI balanced, but not VEGAS
    • Got an abort with "VEGAS HPC program taking too long to be ready"
  • Ray recommends using incoherent mode for now.
  • Will use IncoherentModeTests scheduling block
  • SB submitted at 10:42
    • Scan #2 is i0800x0512
    • Still using a simple Balance() command
    • GUPPI balanced, but not VEGAS
    • Still got an abort with "VEGAS HPC program taking too long to be ready"
  • ROACH does not seem to be sending packets
    • Rebooted BANK A ROACH
    • Reboot did not help
    • Packets are not being sent in spectral line modes, either
    • Apparently there is some work being done on the Timing Center - not sure if this is impacting the ROACH
  • While ROACH issue is being diagnosed Ryan tried to look only at whether issuing a Balance("VEGAS") call causes an abort
  • SB submitted at 11:28
    • No scan being taken
    • When using Balance("VEGAS") the IF rack balances properly but we are still getting an abort message from Astrid saying that VEGAS balancing failed
    • This time there is more error information in the Astrid log:
[15:28:55] ERROR: Balance: unknown exception.
[15:28:55] (<class 'SOAPpy.Types.faultType'>, <Fault SOAP-ENV:Client: Data type 'SOAP-ENC:Array' mismatch in element 'v1'>, <traceback object at 0x7f6500326878>)
  • Trying to balance again, but this time only using a Balance() followed by a Balance("IFRack") instead of Balance("VEGAS")
  • SB submitted at 11:30
    • VEGAS IF inputs do not balance when using Balance("IFRack")
  • Ray will try to add some more error capturing to investigate what is happening with balancing
  • Trying to balance again with a more error capturing
  • SB submitted at 11:59
    • No scan being taken
    • Here is the added error messages
[15:59:11] WARNING Balance.balance(): unknown exception:
[15:59:11] Traceback (most recent call last):
[15:59:11]   File "/home/gbt/lib/python/telescope/gbt/config/Balance.py", line 134, in balance
    status += mgr.balance(options, cfg=cfg)
[15:59:11]   File "/home/gbt/lib/python/telescope/gbt/config/Balance.py", line 1099, in balance
    lbw_status = self.balanceAllBanks(self.banks)
[15:59:11]   File "/home/gbt/lib/python/telescope/gbt/config/Balance.py", line 798, in balanceAllBanks
    needs_adj = self.ballParkBalance(p, cmd)
[15:59:11]   File "/home/gbt/lib/python/telescope/gbt/config/Balance.py", line 723, in ballParkBalance
    d = self.gc.get_values_array(paths)
[15:59:11]   File "/home/sparrow/16.3/lib/python/gbt/ygor/GrailClient.py", line 348, in get_values_array
    rtrn = self.cl.get_values_array(a, " " ," ",b)
[15:59:11] faultType: <Fault SOAP-ENV:Client: Data type 'SOAP-ENC:Array' mismatch in element 'v1'>
[15:59:11] *** Error: Balance failed!
  • Broke for lunch, resumed at 13:15
  • ROACH is sending packets again (1 PPS was down due to work on Timing Center)
  • Tried resubmitting with more error capturing. Ray found a typo.
  • Resubmitted to just look at data flow
    • SB submitted at 13:18
    • Conflicted w/ Ray's Turtle restart. Oops.
  • Submitted for real at 13:19
[17:19:51] Balancing VEGAS
[17:19:52] INFO: ballParkBalance(): paths = [], cmdata = []
[17:19:52] WARNING Balance.balance(): unknown exception:
[17:19:52] Traceback (most recent call last):
[17:19:52]   File "/home/gbt/lib/python/telescope/gbt/config/Balance.py", line 134, in balance
    status += mgr.balance(options, cfg=cfg)
[17:19:52]   File "/home/gbt/lib/python/telescope/gbt/config/Balance.py", line 1101, in balance
    lbw_status = self.balanceAllBanks(self.banks)
[17:19:52]   File "/home/gbt/lib/python/telescope/gbt/config/Balance.py", line 800, in balanceAllBanks
    needs_adj = self.ballParkBalance(p, cmd)
[17:19:52]   File "/home/gbt/lib/python/telescope/gbt/config/Balance.py", line 725, in ballParkBalance
    d = self.gc.get_values_array(paths)
[17:19:52]   File "/home/sparrow/16.3/lib/python/gbt/ygor/GrailClient.py", line 348, in get_values_array
    rtrn = self.cl.get_values_array(a, " " ," ",b)
[17:19:52] faultType: <Fault SOAP-ENV:Client: Data type 'SOAP-ENC:Array' mismatch in element 'v1'>
    • Data flow seemed OK
    • Bugs in real-time bandpass plotter. Will take another scan.
  • SB submitted at 13:22
    • Scan #4 is i0800x0512
    • More little bugs in bandpass plotter. Will take another scan.
  • SB submitted at 13:26
    • Scan #5 is i0800x0512. More error capturing by Ray in this scan.
    • Balancing seemed to work! No error messages anyway. But IF was already balanced, so we should try again
    • Scan seemed to go OK but Astrid threw an abort. No error messages or obvious reason why.
  • Changing attenuation in converter rack to make sure that balancing is working for VEGAS now.
  • Submitted SB at 13:30
    • Scan #6 is i0800x0512
    • Balancing worked!
    • Bandpass auto plotter worked!
    • Still getting mysterious abort
  • Will try incoherent mode again to try and track down the abort. Ray noticed Antenna was in scan coordinator. Took it out.
  • SB submitted at 13:33
    • Scan #7 is i0800x0512
    • No issues. Aborts maybe due to Antenna being in scan coordinator.
    • Data also look good (didn't look at acutal psrfits output in previous scans)
  • Will try coherent mode
  • SB submitted at 13:36
    • Using CoherentModeTests scheduling block
    • Scan #8 is c0800x0512
    • Balance("VEGAS") was commented out. Oops!
    • Scan ran but NETSTAT was blocked on all HPCs.
    • guppi_daq seems to have crashed on HPCs.
  • Ray restarted guppi_daq
  • SB submitted at 13:51
    • Scan #9 is c0800x0512
    • VEGAS balancing worked
    • Got abort. Ray says it was his fault.
  • SB submitted at 13:53
    • Scan # is c0800x0512
    • Got another abort but no messages again. Antenna is not in scan coordinator
    • No packets were flowing on Banks C and D. It seems like the ARP tables are wrong on those banks, but not clear why.
    • Ray found a bug in the way MAC addresses were being set on those banks. He is working on fixing it.
  • In the meantime, Richard is going to come up and test the LBW gain balancing.

Conclusions

  • Need to figure out MAC address problems on Bank C and D, but balancing now seems to be working. VEGAS auto plot, monitor, and HPC status utilities also seem to be working.
-- RyanLynch - 2017-09-28
Topic revision: r4 - 2017-10-03, RyanLynch
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback