2019-07-18 @ 14:00 = FINAL
* = note
[ ] = a task to consider. Some are optional. Some are not.
Good luck finding stuff on the wiki:
https://safe.nrao.edu/wiki/bin/view/ALMA/NAASC/Oracle
https://safe.nrao.edu/wiki/bin/view/HPC/WebHome https://safe.nrao.edu/wiki/bin/view/HPC/HPCSupport https://safe.nrao.edu/wiki/bin/view/HPC/ALMABenchmarking
NGAS from ESO -
https://ictwiki.alma.cl/twiki/bin/view/CommonInf/NgasOperationalProcedures
[X] I have a huge stack of documentation to hand over to you, plus misc stuff.
Lustre:
See printed notes on quotas, I can probably find ATM's e-mails.
Don't let anyone on vault, naasc or lard lustre without quotas
Enabling on NAASC Lustre will take a long time as you put people in the groups they need to be. Work with Tony and Pat.
HPC:
Nothing to add.
OVM:
[ ] Open a low priority ticket with Oracle and exlapin your OVM GUI does not work any more - normally accessed at
The URL to access OVM is
https://naasc-ovm.cv.nrao.edu:7002/ovm/console/ - you should have the password already.
* You already have docs for cli start-ups.
* You should have the map for which VM is on which VS.
* Josh is the keeper of all CV SAN (
NetApp) storage if you need space or help. He won't add you on to CV SAN until you demonstrate minimum skillz, which is 100% in your best interest.
NGAS:
[ ] Get this job automated on NANAGS11 to run on Fridays at 06:00 - /root/bin/ngas_diskspace.sh | /bin/mail -s "{metric} NAASC NGAS Filesystem Sizes"
naasc-admin@nrao.edu,dhalstea@nrao.edu,pmurphy@nrao.edu
[ ] Work with Stewart to finish the NGAS retirement of NANGAS[01-04].
* I will try to get the last 360 chased down this week.
[ ] Migrate to a new NGAS Master of NANGAS11, also with Stewart.
Oracle:
* CJ, get as much training as you can. Caleb, until you pass RHEL 7, I wouldn't get out of sequence by going to Oracle first. You won't be doing your self any favors.
[ ] I would swap in 6 TB drives on NAO2 and make an mdadm RAID6 set - Ask Josh for help if you get stuck. He is the Mdadm Master.
Currently your backups spool to /export/data_1/oracle/ which is 3ware /c7/u0 = /dev/sde1 - do not blow this away on accident.
root@naasc-oracle-2 [/export/data_1/oracle/RMAN_Database_Exports]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sdb1 419G 398G 0 100% /
tmpfs 28G 0 28G 0% /dev/shm
/dev/sde1 22T 18T 4.9T 78% /export/data_1
shmfs 28G 0 28G 0% /dev/shm
root@naasc-oracle-2 [/export/data_1/oracle]# pwd
/export/data_1/oracle
root@naasc-oracle-2 [/export/data_1/oracle]# ll
total 32
drwxr-xr-x 2 54321 54321 6 Feb 13 2018 ALMA_dumps
drwxr-xr-x 12 54321 54321 4096 Sep 14 2017 DA_Transfers
drwxr-x--- 3 54321 54321 17 Feb 26 2016 FRA
drwxr-xr-x 3 54321 54321 16 Feb 4 2016 JAO
drwxr-xr-x 2 54321 54321 4096 Jul 11 15:35 NGAS_Dumps
drwxr-xr-x 2 root root 6 Feb 8 04:01 OL6
drwxr-xr-x 2 oracle oinstall 6 Sep 18 2017 OL7
drwxr-xr-x 2 54321 54321 4096 Jun 18 2017 Old
drwxr-xr-x 2 54321 54321 103 Nov 25 2014 Old_RMAN
drwxr-xr-x 2 54321 54321 56 May 26 2017 Oracle_Dumps
drwxr-xr-x 2 54321 54321 4096 Jul 15 05:24 RMAN_Database_Exports
drwxr-xr-x 2 54321 54321 4096 Jul 11 15:39 Request_Handler_Dumps
drwxr-xr-x 2 54321 54321 4096 Jul 11 15:41 Source_Catalogue_Dumps
drwxr-xr-x 2 54321 54321 4096 Jul 11 15:43 TAP_Dumps
drwxrwxrwt 4 54321 54321 4096 Feb 8 04:28 tmp
root@naasc-oracle-2 [/export/data_1/oracle]# ll RMAN_Database_Exports/
total 18165963460
-rw-r----- 1 54321 54321 687837126656 Jul 7 01:52 RMAN_NAO5_ALMA-2019-07-07_set-2360_1
-rw-r----- 1 54321 54321 673550344192 Jul 7 03:45 RMAN_NAO5_ALMA-2019-07-07_set-2361_1
-rw-r----- 1 54321 54321 690853322752 Jul 7 05:35 RMAN_NAO5_ALMA-2019-07-07_set-2362_1
-rw-r----- 1 54321 54321 2290385920 Jul 7 05:36 RMAN_NAO5_ALMA-2019-07-07_set-2363_1
-rw-r----- 1 54321 54321 5657486336 Jul 8 00:01 RMAN_NAO5_ALMA-2019-07-08_set-2365_1
-rw-r----- 1 54321 54321 688419127296 Jul 8 01:56 RMAN_NAO5_ALMA-2019-07-08_set-2366_1
-rw-r----- 1 54321 54321 674491383808 Jul 8 03:50 RMAN_NAO5_ALMA-2019-07-08_set-2367_1
-rw-r----- 1 54321 54321 691608158208 Jul 8 05:44 RMAN_NAO5_ALMA-2019-07-08_set-2368_1
-rw-r----- 1 54321 54321 3156311552 Jul 8 05:45 RMAN_NAO5_ALMA-2019-07-08_set-2369_1
-rw-r----- 1 54321 54321 4825960448 Jul 9 00:01 RMAN_NAO5_ALMA-2019-07-09_set-2371_1
-rw-r----- 1 54321 54321 689091338240 Jul 9 01:52 RMAN_NAO5_ALMA-2019-07-09_set-2372_1
-rw-r----- 1 54321 54321 675207233536 Jul 9 03:41 RMAN_NAO5_ALMA-2019-07-09_set-2373_1
-rw-r----- 1 54321 54321 692034928640 Jul 9 05:32 RMAN_NAO5_ALMA-2019-07-09_set-2374_1
-rw-r----- 1 54321 54321 2285334528 Jul 9 05:33 RMAN_NAO5_ALMA-2019-07-09_set-2375_1
-rw-r----- 1 54321 54321 6246432768 Jul 10 00:01 RMAN_NAO5_ALMA-2019-07-10_set-2377_1
-rw-r----- 1 54321 54321 689890820096 Jul 10 01:56 RMAN_NAO5_ALMA-2019-07-10_set-2378_1
-rw-r----- 1 54321 54321 676052525056 Jul 10 03:46 RMAN_NAO5_ALMA-2019-07-10_set-2379_1
-rw-r----- 1 54321 54321 692649508864 Jul 10 05:36 RMAN_NAO5_ALMA-2019-07-10_set-2380_1
-rw-r----- 1 54321 54321 1673796096 Jul 10 05:36 RMAN_NAO5_ALMA-2019-07-10_set-2381_1
-rw-r----- 1 54321 54321 4725296640 Jul 11 00:01 RMAN_NAO5_ALMA-2019-07-11_set-2383_1
-rw-r----- 1 54321 54321 690917130240 Jul 11 01:48 RMAN_NAO5_ALMA-2019-07-11_set-2384_1
-rw-r----- 1 54321 54321 675747282944 Jul 11 03:39 RMAN_NAO5_ALMA-2019-07-11_set-2385_1
-rw-r----- 1 54321 54321 693360287744 Jul 11 05:29 RMAN_NAO5_ALMA-2019-07-11_set-2386_1
-rw-r----- 1 54321 54321 1741429248 Jul 11 05:29 RMAN_NAO5_ALMA-2019-07-11_set-2387_1
-rw-r----- 1 54321 54321 4555800576 Jul 12 00:01 RMAN_NAO5_ALMA-2019-07-12_set-2389_1
-rw-r----- 1 54321 54321 691253944320 Jul 12 01:49 RMAN_NAO5_ALMA-2019-07-12_set-2390_1
-rw-r----- 1 54321 54321 676616290304 Jul 12 03:37 RMAN_NAO5_ALMA-2019-07-12_set-2391_1
-rw-r----- 1 54321 54321 693765963776 Jul 12 05:28 RMAN_NAO5_ALMA-2019-07-12_set-2392_1
-rw-r----- 1 54321 54321 1521342464 Jul 12 05:28 RMAN_NAO5_ALMA-2019-07-12_set-2393_1
-rw-r----- 1 54321 54321 4173675008 Jul 13 00:01 RMAN_NAO5_ALMA-2019-07-13_set-2395_1
-rw-r----- 1 54321 54321 691744235520 Jul 13 01:53 RMAN_NAO5_ALMA-2019-07-13_set-2396_1
-rw-r----- 1 54321 54321 676922974208 Jul 13 03:38 RMAN_NAO5_ALMA-2019-07-13_set-2397_1
-rw-r----- 1 54321 54321 694233456640 Jul 13 05:34 RMAN_NAO5_ALMA-2019-07-13_set-2398_1
-rw-r----- 1 54321 54321 2015549952 Jul 13 05:35 RMAN_NAO5_ALMA-2019-07-13_set-2399_1
-rw-r----- 1 54321 54321 3699884544 Jul 14 00:07 RMAN_NAO5_ALMA-2019-07-14_set-2401_1
-rw-r----- 1 54321 54321 692319576064 Jul 14 02:02 RMAN_NAO5_ALMA-2019-07-14_set-2402_1
-rw-r----- 1 54321 54321 677438177280 Jul 14 03:47 RMAN_NAO5_ALMA-2019-07-14_set-2403_1
-rw-r----- 1 54321 54321 694587465728 Jul 14 05:35 RMAN_NAO5_ALMA-2019-07-14_set-2404_1
-rw-r----- 1 54321 54321 1777279488 Jul 14 05:36 RMAN_NAO5_ALMA-2019-07-14_set-2405_1
-rw-r----- 1 54321 54321 4321286656 Jul 15 00:01 RMAN_NAO5_ALMA-2019-07-15_set-2407_1
-rw-r----- 1 54321 54321 692800847872 Jul 15 01:48 RMAN_NAO5_ALMA-2019-07-15_set-2408_1
-rw-r----- 1 54321 54321 677964505088 Jul 15 03:36 RMAN_NAO5_ALMA-2019-07-15_set-2409_1
-rw-r----- 1 54321 54321 695285604352 Jul 15 05:24 RMAN_NAO5_ALMA-2019-07-15_set-2410_1
-rw-r----- 1 54321 54321 1529803776 Jul 15 05:25 RMAN_NAO5_ALMA-2019-07-15_set-2411_1
These rman exports are automatically purged based on 7 days retention. Eventually, some day, you WILL run out of backup space.
===========================================
Best e-mail is
mike.hatz@gmail.com I plan to be 100% gone NLT August 30th.
=======================================
Notes for Mattermost (sorry for some dupes):
[ ] Workshop topic list!
[ ] NGAS 360 file migrations
[ ] Jano list of stuff to tweak - will need to be monitored over time aka "inconsistency list"
[ ] "Andy list", which atually likely ties in to "Jano List"
[ ] NGAS disk usage weekly report - needs a keyless ssh
[ ] OVM GUI start-up - will need a trouble ticket with Oracle
[ ] Xfer Oracle account POC to naasc-dba rather than mhatz
[X] NAOTEST refreshes: +grants, etc.
[X] NAOTEST - review application of OFFLINE updates for testing
[X] Oracle Backups for n00bz
[X] Final review of EMS
[ ] OEM if you're so inclined
[ ] Oracle POC - naasc-dba for NM / Oracle Chile
[ ] Work with Josh or get a Nagios plugin to trap and alert on "ORA-####" entries in nao5:/home/oracle/ALMA.log and on nao5:/home/oracle/ASM.log
lrwxrwxrwx. 1 oracle oinstall 57 Jan 22 2018 ALMA.log -> /u01/app/oracle/diag/rdbms/alma/ALMA/trace/alert_ALMA.log
lrwxrwxrwx. 1 oracle oinstall 55 Jan 22 2018 ASM.log -> /u01/app/oracle/diag/asm/+asm/+ASM/trace/alert_+ASM.log
[ ] And that huge ass bloated log should really really get rotated, like on a monthly basis - ALMA & ASM