2019-07-18 @ 14:00 = FINAL

* = note

[ ] = a task to consider. Some are optional. Some are not.

Good luck finding stuff on the wiki:

https://safe.nrao.edu/wiki/bin/view/ALMA/NAASC/Oracle

https://safe.nrao.edu/wiki/bin/view/HPC/WebHome
https://safe.nrao.edu/wiki/bin/view/HPC/HPCSupport
https://safe.nrao.edu/wiki/bin/view/HPC/ALMABenchmarking

NGAS from ESO - https://ictwiki.alma.cl/twiki/bin/view/CommonInf/NgasOperationalProcedures

[X] I have a huge stack of documentation to hand over to you, plus misc stuff.


Lustre:

See printed notes on quotas, I can probably find ATM's e-mails.

Don't let anyone on vault, naasc or lard lustre without quotas

Enabling on NAASC Lustre will take a long time as you put people in the groups they need to be. Work with Tony and Pat.


HPC:

Nothing to add.


OVM:

[ ] Open a low priority ticket with Oracle and exlapin your OVM GUI does not work any more - normally accessed at

The URL to access OVM is https://naasc-ovm.cv.nrao.edu:7002/ovm/console/ - you should have the password already.

* You already have docs for cli start-ups.

* You should have the map for which VM is on which VS.

* Josh is the keeper of all CV SAN (NetApp) storage if you need space or help. He won't add you on to CV SAN until you demonstrate minimum skillz, which is 100% in your best interest.


NGAS:

[ ] Get this job automated on NANAGS11 to run on Fridays at 06:00 - /root/bin/ngas_diskspace.sh | /bin/mail -s "{metric} NAASC NGAS Filesystem Sizes" naasc-admin@nrao.edu,dhalstea@nrao.edu,pmurphy@nrao.edu

[ ] Work with Stewart to finish the NGAS retirement of NANGAS[01-04].
* I will try to get the last 360 chased down this week.

[ ] Migrate to a new NGAS Master of NANGAS11, also with Stewart.


Oracle:

* CJ, get as much training as you can. Caleb, until you pass RHEL 7, I wouldn't get out of sequence by going to Oracle first. You won't be doing your self any favors.

[ ] I would swap in 6 TB drives on NAO2 and make an mdadm RAID6 set - Ask Josh for help if you get stuck. He is the Mdadm Master.

Currently your backups spool to /export/data_1/oracle/ which is 3ware /c7/u0 = /dev/sde1 - do not blow this away on accident.

root@naasc-oracle-2 [/export/data_1/oracle/RMAN_Database_Exports]# df -h

Filesystem Size Used Avail Use% Mounted on
/dev/sdb1 419G 398G 0 100% /
tmpfs 28G 0 28G 0% /dev/shm
/dev/sde1 22T 18T 4.9T 78% /export/data_1
shmfs 28G 0 28G 0% /dev/shm

root@naasc-oracle-2 [/export/data_1/oracle]# pwd
/export/data_1/oracle

root@naasc-oracle-2 [/export/data_1/oracle]# ll
total 32
drwxr-xr-x 2 54321 54321 6 Feb 13 2018 ALMA_dumps
drwxr-xr-x 12 54321 54321 4096 Sep 14 2017 DA_Transfers
drwxr-x--- 3 54321 54321 17 Feb 26 2016 FRA
drwxr-xr-x 3 54321 54321 16 Feb 4 2016 JAO
drwxr-xr-x 2 54321 54321 4096 Jul 11 15:35 NGAS_Dumps
drwxr-xr-x 2 root root 6 Feb 8 04:01 OL6
drwxr-xr-x 2 oracle oinstall 6 Sep 18 2017 OL7
drwxr-xr-x 2 54321 54321 4096 Jun 18 2017 Old
drwxr-xr-x 2 54321 54321 103 Nov 25 2014 Old_RMAN
drwxr-xr-x 2 54321 54321 56 May 26 2017 Oracle_Dumps
drwxr-xr-x 2 54321 54321 4096 Jul 15 05:24 RMAN_Database_Exports
drwxr-xr-x 2 54321 54321 4096 Jul 11 15:39 Request_Handler_Dumps
drwxr-xr-x 2 54321 54321 4096 Jul 11 15:41 Source_Catalogue_Dumps
drwxr-xr-x 2 54321 54321 4096 Jul 11 15:43 TAP_Dumps
drwxrwxrwt 4 54321 54321 4096 Feb 8 04:28 tmp

root@naasc-oracle-2 [/export/data_1/oracle]# ll RMAN_Database_Exports/

total 18165963460
-rw-r----- 1 54321 54321 687837126656 Jul 7 01:52 RMAN_NAO5_ALMA-2019-07-07_set-2360_1
-rw-r----- 1 54321 54321 673550344192 Jul 7 03:45 RMAN_NAO5_ALMA-2019-07-07_set-2361_1
-rw-r----- 1 54321 54321 690853322752 Jul 7 05:35 RMAN_NAO5_ALMA-2019-07-07_set-2362_1
-rw-r----- 1 54321 54321 2290385920 Jul 7 05:36 RMAN_NAO5_ALMA-2019-07-07_set-2363_1
-rw-r----- 1 54321 54321 5657486336 Jul 8 00:01 RMAN_NAO5_ALMA-2019-07-08_set-2365_1
-rw-r----- 1 54321 54321 688419127296 Jul 8 01:56 RMAN_NAO5_ALMA-2019-07-08_set-2366_1
-rw-r----- 1 54321 54321 674491383808 Jul 8 03:50 RMAN_NAO5_ALMA-2019-07-08_set-2367_1
-rw-r----- 1 54321 54321 691608158208 Jul 8 05:44 RMAN_NAO5_ALMA-2019-07-08_set-2368_1
-rw-r----- 1 54321 54321 3156311552 Jul 8 05:45 RMAN_NAO5_ALMA-2019-07-08_set-2369_1
-rw-r----- 1 54321 54321 4825960448 Jul 9 00:01 RMAN_NAO5_ALMA-2019-07-09_set-2371_1
-rw-r----- 1 54321 54321 689091338240 Jul 9 01:52 RMAN_NAO5_ALMA-2019-07-09_set-2372_1
-rw-r----- 1 54321 54321 675207233536 Jul 9 03:41 RMAN_NAO5_ALMA-2019-07-09_set-2373_1
-rw-r----- 1 54321 54321 692034928640 Jul 9 05:32 RMAN_NAO5_ALMA-2019-07-09_set-2374_1
-rw-r----- 1 54321 54321 2285334528 Jul 9 05:33 RMAN_NAO5_ALMA-2019-07-09_set-2375_1
-rw-r----- 1 54321 54321 6246432768 Jul 10 00:01 RMAN_NAO5_ALMA-2019-07-10_set-2377_1
-rw-r----- 1 54321 54321 689890820096 Jul 10 01:56 RMAN_NAO5_ALMA-2019-07-10_set-2378_1
-rw-r----- 1 54321 54321 676052525056 Jul 10 03:46 RMAN_NAO5_ALMA-2019-07-10_set-2379_1
-rw-r----- 1 54321 54321 692649508864 Jul 10 05:36 RMAN_NAO5_ALMA-2019-07-10_set-2380_1
-rw-r----- 1 54321 54321 1673796096 Jul 10 05:36 RMAN_NAO5_ALMA-2019-07-10_set-2381_1
-rw-r----- 1 54321 54321 4725296640 Jul 11 00:01 RMAN_NAO5_ALMA-2019-07-11_set-2383_1
-rw-r----- 1 54321 54321 690917130240 Jul 11 01:48 RMAN_NAO5_ALMA-2019-07-11_set-2384_1
-rw-r----- 1 54321 54321 675747282944 Jul 11 03:39 RMAN_NAO5_ALMA-2019-07-11_set-2385_1
-rw-r----- 1 54321 54321 693360287744 Jul 11 05:29 RMAN_NAO5_ALMA-2019-07-11_set-2386_1
-rw-r----- 1 54321 54321 1741429248 Jul 11 05:29 RMAN_NAO5_ALMA-2019-07-11_set-2387_1
-rw-r----- 1 54321 54321 4555800576 Jul 12 00:01 RMAN_NAO5_ALMA-2019-07-12_set-2389_1
-rw-r----- 1 54321 54321 691253944320 Jul 12 01:49 RMAN_NAO5_ALMA-2019-07-12_set-2390_1
-rw-r----- 1 54321 54321 676616290304 Jul 12 03:37 RMAN_NAO5_ALMA-2019-07-12_set-2391_1
-rw-r----- 1 54321 54321 693765963776 Jul 12 05:28 RMAN_NAO5_ALMA-2019-07-12_set-2392_1
-rw-r----- 1 54321 54321 1521342464 Jul 12 05:28 RMAN_NAO5_ALMA-2019-07-12_set-2393_1
-rw-r----- 1 54321 54321 4173675008 Jul 13 00:01 RMAN_NAO5_ALMA-2019-07-13_set-2395_1
-rw-r----- 1 54321 54321 691744235520 Jul 13 01:53 RMAN_NAO5_ALMA-2019-07-13_set-2396_1
-rw-r----- 1 54321 54321 676922974208 Jul 13 03:38 RMAN_NAO5_ALMA-2019-07-13_set-2397_1
-rw-r----- 1 54321 54321 694233456640 Jul 13 05:34 RMAN_NAO5_ALMA-2019-07-13_set-2398_1
-rw-r----- 1 54321 54321 2015549952 Jul 13 05:35 RMAN_NAO5_ALMA-2019-07-13_set-2399_1
-rw-r----- 1 54321 54321 3699884544 Jul 14 00:07 RMAN_NAO5_ALMA-2019-07-14_set-2401_1
-rw-r----- 1 54321 54321 692319576064 Jul 14 02:02 RMAN_NAO5_ALMA-2019-07-14_set-2402_1
-rw-r----- 1 54321 54321 677438177280 Jul 14 03:47 RMAN_NAO5_ALMA-2019-07-14_set-2403_1
-rw-r----- 1 54321 54321 694587465728 Jul 14 05:35 RMAN_NAO5_ALMA-2019-07-14_set-2404_1
-rw-r----- 1 54321 54321 1777279488 Jul 14 05:36 RMAN_NAO5_ALMA-2019-07-14_set-2405_1
-rw-r----- 1 54321 54321 4321286656 Jul 15 00:01 RMAN_NAO5_ALMA-2019-07-15_set-2407_1
-rw-r----- 1 54321 54321 692800847872 Jul 15 01:48 RMAN_NAO5_ALMA-2019-07-15_set-2408_1
-rw-r----- 1 54321 54321 677964505088 Jul 15 03:36 RMAN_NAO5_ALMA-2019-07-15_set-2409_1
-rw-r----- 1 54321 54321 695285604352 Jul 15 05:24 RMAN_NAO5_ALMA-2019-07-15_set-2410_1
-rw-r----- 1 54321 54321 1529803776 Jul 15 05:25 RMAN_NAO5_ALMA-2019-07-15_set-2411_1

These rman exports are automatically purged based on 7 days retention. Eventually, some day, you WILL run out of backup space.


===========================================

Best e-mail is mike.hatz@gmail.com I plan to be 100% gone NLT August 30th.

=======================================

Notes for Mattermost (sorry for some dupes):

[ ] Workshop topic list!
[ ] NGAS 360 file migrations
[ ] Jano list of stuff to tweak - will need to be monitored over time aka "inconsistency list"
[ ] "Andy list", which atually likely ties in to "Jano List"
[ ] NGAS disk usage weekly report - needs a keyless ssh
[ ] OVM GUI start-up - will need a trouble ticket with Oracle
[ ] Xfer Oracle account POC to naasc-dba rather than mhatz
[X] NAOTEST refreshes: +grants, etc.
[X] NAOTEST - review application of OFFLINE updates for testing
[X] Oracle Backups for n00bz
[X] Final review of EMS
[ ] OEM if you're so inclined
[ ] Oracle POC - naasc-dba for NM / Oracle Chile

[ ] Work with Josh or get a Nagios plugin to trap and alert on "ORA-####" entries in nao5:/home/oracle/ALMA.log and on nao5:/home/oracle/ASM.log

lrwxrwxrwx. 1 oracle oinstall 57 Jan 22 2018 ALMA.log -> /u01/app/oracle/diag/rdbms/alma/ALMA/trace/alert_ALMA.log
lrwxrwxrwx. 1 oracle oinstall 55 Jan 22 2018 ASM.log -> /u01/app/oracle/diag/asm/+asm/+ASM/trace/alert_+ASM.log

[ ] And that huge ass bloated log should really really get rotated, like on a monthly basis - ALMA & ASM
Topic revision: r4 - 2019-07-19, MikeHatz
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback