Performance Evaluations for CycSpec

Testing DSPSR and GUPPI_DAQ performance via Grafana

http://grafana.gb.nrao.edu/d/DhE1lXbnz/vegas-hpc21-22-testing-dash?orgId=1&refresh=5s



Testing Round 1: one 2080 GPU

Testing using dspsr and guppi-daq on vegas-hpc11
Diagnostic Tests for: one 2080 GPU

TINT_20211215

Grafana Snapshot: http://grafana.gb.nrao.edu/dashboard/snapshot/AW3VsoMopR9eXirYqkxQcCAb7N1hX3dL
Description guppi time - dspsr time dspsr end time - guppi end time Banks in Use (vegas-hpc1#) Timing (s) Memory Used Notes
Normal Pulsar Run     1-8 610 802 MB  
Only dspsr     1 150 5.33 GB pulse_profile lowered with -b
Run guppi+dspsr +70s -70s 2-8 310 806 MB  
      1 240 5.75 GB  
dspsr run with 3 threads +60s -40s 2-8 200 811 MB  
      1 140 5.75 GB  
dspsr run with 2 threads +60s -30s 2-8 210 815 MB  
      1 150 3.45 GB  
dspsr run with 1 thread +160s (?)   2-8 180 819 MB  
    -150s (?) 1 20 2.16 GB notes say: 'writing to shared memory'
    -0s (?) 1 20 2.16 GB saw 2 'spikes' in GPU memory
dspsr run with 1 thread (180s) +30s (note: dspsr started after guppi) +20s 2-8 200 823 MB  
      1 170 2.17 GB notes say: 'barely able to keep up'
run not using GPU     1-8 230 827 MB  



Testing Round 2: two 2080 GPUs

Testing using dspsr and guppi-daq on vegas-hpc21
Diagnostic Tests for: two 2080 GPU

TINT_20220113

Grafana Snapshot: http://grafana.gb.nrao.edu/dashboard/snapshot/7mRIcNE1lBHh4geSVcz2470GNzk7K1Je?viewPanel=11&orgId=1

Note before table: vegas-hpc21 GPU-B models the same behavior as GPUs on vegas-hpc1[1-7] (hpc18 was unresponsive until the return to production as hpc21 took its place as BankH) - The other hpcs were to the order of 1.87GB at their peaks, while GPU-B peaks at 746MB - anytime GPU-B peaks the reader can assume the other hpc1[1-7] also peak to their peak value.

Important for reader to note that GPU-A can be regarded as dspsr and GPU-B can be regarded as guppi-daq as they each only run on one GPU
Description guppi time - dspsr time dspsr end time - guppi end time GPUs in Use (vegas-hpc21) A or B Timing (s) Memory Used Notes
Pulsar Coherent Scan -70 -70 A 130 4.92 GB  
      B 200 737 MB  
180s Scan -80 -80 A 120 4.92 GB  
      B 200 741 MB  
Run dspsr on core 23     A 110 4.92 GB 4 @ 25% each
Run dspsr on core 20-23     A 80 4.92 GB Only runs on core 20
Run dspsr on core 23 + guppi 0 0 A 200 4.89 GB  
      B 200 750 MB  
dsprs + guppi 180s scan +20 +20 A 200 4.92 GB bottle neck
      B 180 758 MB  
only guppi @ 30s     B 60 746 MB  
only dspsr for 30s     A 50 4.93 GB  
dspsr+guppi+profiler @ 30s -30 -30 A 50 4.94 GB  
      B 80 746 MB  
dspsr>1 core+guppi+profiler @30s -70 -40 (dspsr start +30 after) A 50 4.94 GB  
      B 120 746 MB  
only guppi     B 90 746 MB  
dspsr>1 core+guppi @30s -30 -10 A 40 4.92 GB  
      B 70 725 MB  
sanity check     vegas-hpc[1-8] 120 1.16 GB back to production



Testing Round 3: one 3080 GPU

Testing using guppi-daq on vegas-hpc22
Diagnostic Tests for: guppi_daq can be run on a 3080 - and on a mixed system with other 2080s

TINT_20220202

Comments: no dspsr for testing



-- KathlynPurcell - 2022-02-08
Topic revision: r3 - 2022-02-10, KathlynPurcell
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback