Performance Evaluations for CycSpec

Testing DSPSR and GUPPI_DAQ performance via Grafana

http://grafana.gb.nrao.edu/d/DhE1lXbnz/vegas-hpc21-22-testing-dash?orgId=1&refresh=5s

---

---

Testing Round 1: one 2080 GPU

Testing using dspsr and guppi-daq on vegas-hpc11
Diagnostic Tests for: one 2080 GPU

TINT_20211215

Grafana Snapshot: http://grafana.gb.nrao.edu/dashboard/snapshot/AW3VsoMopR9eXirYqkxQcCAb7N1hX3dL

Description	guppi time - dspsr time	dspsr end time - guppi end time	Banks in Use (vegas-hpc1#)	Timing (s)	Memory Used	Notes
Normal Pulsar Run			1-8	610	802 MB
Only dspsr			1	150	5.33 GB	pulse_profile lowered with -b
Run guppi+dspsr	+70s	-70s	2-8	310	806 MB
			1	240	5.75 GB
dspsr run with 3 threads	+60s	-40s	2-8	200	811 MB
			1	140	5.75 GB
dspsr run with 2 threads	+60s	-30s	2-8	210	815 MB
			1	150	3.45 GB
dspsr run with 1 thread	+160s (?)		2-8	180	819 MB
		-150s (?)	1	20	2.16 GB	notes say: 'writing to shared memory'
		-0s (?)	1	20	2.16 GB	saw 2 'spikes' in GPU memory
dspsr run with 1 thread (180s)	+30s (note: dspsr started after guppi)	+20s	2-8	200	823 MB
			1	170	2.17 GB	notes say: 'barely able to keep up'
run not using GPU			1-8	230	827 MB

---

---

Testing Round 2: two 2080 GPUs

Testing using dspsr and guppi-daq on vegas-hpc21
Diagnostic Tests for: two 2080 GPU

TINT_20220113

Grafana Snapshot: http://grafana.gb.nrao.edu/dashboard/snapshot/7mRIcNE1lBHh4geSVcz2470GNzk7K1Je?viewPanel=11&orgId=1

Note before table: vegas-hpc21 GPU-B models the same behavior as GPUs on vegas-hpc1[1-7] (hpc18 was unresponsive until the return to production as hpc21 took its place as BankH) - The other hpcs were to the order of 1.87GB at their peaks, while GPU-B peaks at 746MB - anytime GPU-B peaks the reader can assume the other hpc1[1-7] also peak to their peak value.

Important for reader to note that GPU-A can be regarded as dspsr and GPU-B can be regarded as guppi-daq as they each only run on one GPU

Description	guppi time - dspsr time	dspsr end time - guppi end time	GPUs in Use (vegas-hpc21) A or B	Timing (s)	Memory Used	Notes
Pulsar Coherent Scan	-70	-70	A	130	4.92 GB
			B	200	737 MB
180s Scan	-80	-80	A	120	4.92 GB
			B	200	741 MB
Run dspsr on core 23			A	110	4.92 GB	4 @ 25% each
Run dspsr on core 20-23			A	80	4.92 GB	Only runs on core 20
Run dspsr on core 23 + guppi	0	0	A	200	4.89 GB
			B	200	750 MB
dsprs + guppi 180s scan	+20	+20	A	200	4.92 GB	bottle neck
			B	180	758 MB
only guppi @ 30s			B	60	746 MB
only dspsr for 30s			A	50	4.93 GB
dspsr+guppi+profiler @ 30s	-30	-30	A	50	4.94 GB
			B	80	746 MB
dspsr>1 core+guppi+profiler @30s	-70	-40 (dspsr start +30 after)	A	50	4.94 GB
			B	120	746 MB
only guppi			B	90	746 MB
dspsr>1 core+guppi @30s	-30	-10	A	40	4.92 GB
			B	70	725 MB
sanity check			vegas-hpc[1-8]	120	1.16 GB	back to production

---

---

Testing Round 3: one 3080 GPU

Testing using guppi-daq on vegas-hpc22
Diagnostic Tests for: guppi_daq can be run on a 3080 - and on a mixed system with other 2080s

TINT_20220202

Comments: no dspsr for testing

---

---

-- KathlynPurcell - 2022-02-08