Scope

Feb 2018 upgrade of various Oracle machines and service

  1. New OVM VM, NAASC-ORACLE-5 (NAO5), running OEL 7.4 and Oracle 12.2c - Highest priority
  2. New OVM VM, NAASC-GOLDENGATE-02 (NAGG02), running OEL 7.4(?) and Golden Gate 12.3.0.1.0 - High priority
  3. Upgrade the OVM infrastructure from 3.4.2 to 3.4.4 - Medium priority
  4. Update OVM VM, NAASC-OEM, running OEL X.X and Oracle Enterprise Manager 13.2.0.0 - Low priority

Preparation

Ordering needs to be cleaned up.

  1. Make a migration checklist. How in the world, do we not have one located in one location for all the ARCs to use?
tnsnames.ora, listener.ora, sqlnet.ora (see http://www.dba-oracle.com/t_create_sqlnet_ora.htm) , ALMAinit.ora, /etc/oratab,..???

  1. Do we have enough space on CVSAN for the 3 groups of Oracle, OEM and Golden Gate?
  2. Has all of the existing storage been re-labelled to prevent confusion as to which device goes to which VM?
    1. <# include screencap(s)>
  3. Has all of the media from JAO been downloaded?
    1. ssh root@nadd
    2. ssh naarc@arcproc
    3. mkdir 122c ; cd 122c
    4. rsync -avP rsync://10.200.114.98:/122sw .
    5. rsync from arcproc back to nadd as
      1. rsync -avPz naarc@arcproc:122c .
  4. Has DNS address space been vetted by Derek and new A / CNAME / PTR records been updated?
  5. Making room for NAO5 on NAASC-OVS-01
    1. Has the RAM and CPU count been reduced on NAO4 in preparation for NAO5 spinning up? Cut down to 8 cores and 88000 memory.
    2. Has /dev/shm and Oracle memory parameters been turned down on NAO4 in preparation for NAO5 spinning up?
  6. Has a database export been performed successfully?
  7. Have the various Oracle images been staged at https://bulk.cv.nrao.edu/almadata/sciver/mhatz/oracle/2018, because OVM is fundamentally stupid about the choices it gives you to load data into repositories? That web repository maps to /lustre/naasc/web/SV/mhatz/oracle/2018 The source is nadd:/export/data_1/oracle/tmp/122c that is copied on to Lustre for web staging.
  8. grid user for ASM vs oracle user for ASM. Make a decision!

NAO5

  1. Various gyrations to prepare the image and loaded in to Oracle OVM repository as a VM template. Image(s) are staged from https://bulk.cv.nrao.edu/almadata/sciver/mhatz/oracle/2018/template122c
  2. Create new VM as NAO5
  3. Set cores = 4 and memory = 320000
  4. Make sure the partitions are large enough for the VM. Sometimes they are too small for our use as shipped by Oracle. frown, sad smile frown, sad smile frown, sad smile
  5. Label virtual disks in OVM as NAO5-$BLAH
  6. Attach VNIC (Virtual NIC) to 10.2.140.0 network.
  7. Boot new VM and configure
  8. Resize partitions of the actual VM.
  9. /etc/security/limits.conf has been updated by installer, but not on NAO4...???
  10. Map and configure ASM through OVM. Make sure to label all NAO5 storage as such!
  11. Archive loggings - OFF during import, on after that.
  12. Verify FRA / flashback. http://dbaclass.com/article/ora-38774-cannot-disable-media-recovery-flashback-database-enabled/
Disks for the Oracle VM (NAO5)

Our numbers are appended.

/dev/xvdc (4TB,physical disk):   /backup (ext4 for FRA)                 5000
/dev/xvdd (100GB,physical disk): GROUP1_DISK1 for GROUP1_SYSTEM         100
/dev/xvde (100GB,physical disk): GROUP2_DISK1 for GROUP2_UNDO           200
/dev/xvdf (100GB,physical disk): GROUP3_DISK1 for GROUP3_TEMP           500
/dev/xvdg (300GB,physical disk): GROUP4_DISK1 for GROUP4_REDO1          1000
/dev/xvdh (300GB,physical disk): GROUP5_DISK1 for GROUP5_REDO2          1000
/dev/xvdi (300GB,physical disk): GROUP6_DISK1 for GROUP6_TABLE          300
/dev/xvdj (300GB,physical disk): GROUP7_DISK1 for GROUP7_INDEX          300
/dev/xvdk (2000GB,physical disk):GROUP8_DISK1 for GROUP8_LOB            2000
/dev/xvdl (2000GB,physical disk):GROUP8_DISK2 for GROUP8_LOB            2000
/dev/xvdm (2000GB,physical disk):GROUP8_DISK3 for GROUP8_LOB            2000

[Are we using?] /dev/xvdn (100GB,virtual disk): GROUP1_TEMP for GROUP1_SYSTEM (temporary use)
  1. Install Oracle database software with Oracle Deploy Tool.
  2. Create the ALMA database - Morita-san@NAOJ is going through his final review of 12.2c scripts. Stay tuned! smile
  3. DISABLE LOGGING DURING THE DATABASE IMPORT!
  4. Apply Morita-san fix to OVS0[1-2] gnttab_max_nr_frames=128
  5. Before the cut over take memory and cores back up to pre-migration settings on NAO5 and down on NAO4, to the bare minimum! Do not forget /dev/shm and various internal MEMORY tuning targets in Oracle!

NAGG02

  1. Loaded in to OVM repository. Image(s) are staged from https://bulk.cv.nrao.edu/almadata/sciver/mhatz/oracle/2018
  2. Create new VM as NAGG02
  3. Make sure the partitions are large enough for the VM. Sometimes they are too small for our use as shipped by Oracle. frown, sad smile frown, sad smile frown, sad smile
  4. Label virtual disks in OVM as NAGG02-$BLAH
  5. Attach VNIC (Virtual NIC) to 10.2.140.0 network.
  6. Boot new VM and configure - the canned image with 2 GB RAM and 4 GB swap is way too small. Make it 8GB x 8 GB.
  7. Install Oracle Golden Gate software
  8. Needs Oracle Client - NOT the Express Edition
  9. Update network IP addresses and ports on JAO wiki.
  10. Closely review data pumps vs extracts vs replicats.
  11. Pay attention to warnings / comments in the configs from NAGG01 when you copy them over.
  12. Copy /etc/security/limits.conf from NAGG01
  13. Are we getting transactions for both 12.1 and 12.2 now?
  14. At cut over time, flip CNAME for NAGG to go to NAGG02. This CNAME is NOT used for GG TXs to JAO. Those are hardcoded by IP. The CNAME is local SysAdmin use.
  15. jotey logrotate scripts for NAGG02
  16. Backups for $GG

OVM

  1. Loaded in to OVM repository. Image(s) are staged from https://bulk.cv.nrao.edu/almadata/sciver/mhatz/oracle/2018
  2. Repos for yum. Are they right? Why the hell doesn't Urkel provide a .repo for /etc/yum.repos.d?

NAASC-OEM

  1. Loaded in to OVM repository. Image(s) are staged from https://bulk.cv.nrao.edu/almadata/sciver/mhatz/oracle/2018
  2. Create new VM as NAASC-OEM
  3. Make sure the partitions are large enough for the VM. Sometimes they are too small for our use as shipped by Oracle. frown, sad smile frown, sad smile frown, sad smile
  4. Label virtual disks in OVM as NAOEM-$BLAH
  5. Attach VNIC (Virtual NIC) to 10.2.140.0 network.
  6. Boot new VM and configure

Miscellaneous

  1. ssh keys for root / oracle
  2. .dot files like .bash*, .wgetrc, .vimrc, etc...
  3. /etc/chrony.conf via chrony for UTC
  4. RPMs - review /var/log/yum.log from VALKYRIE or NAO4 or NANGAS10 for installs of packages. Make sure /etc/yum.conf is good to go against other local copies.
    1. rcs
    2. htop
    3. iotop
    4. iftop
    5. rlwrap
    6. mlocate
    7. tree
    8. unzip
    9. rlwrap
    10. mailx (if mail is missing)
    11. lsof
    12. screen
    13. diffstat
  5. DNS for /etc/resolv.conf
  6. Disable iptables /firewalld(erp)
  7. Disable selinux
  8. Backups of ~root, ,~oracle, /etc, /var - refer to existing setups.
  9. Common symlinks in ~oracle for tnsnames.ora, ALMA.log and +ASM.log
  10. Install and configure postfix
  11. Validate hosnames with 'hostnamectl' and ensure they are short names, not FQDN nor localhost.localdomain. frown, sad smile
  12. cp nao4:/etc/wgetrc nao5:/etc/wgetrc
  13. DISABLE Oracle backups when you start the export!

Issues / Items of Interest

  1. Database export speed - 34 hours. Redo logs will need to be watched. See e-mail from MZ (Mauricio Zambrano).
  2. NAO5 boot error - memory
OVMRU_005071E Virtual Machine: NAO5, cannot be started/resumed on any server in server pool: NAASC OVM Pool, because there are no compatible servers for this VM.
Reasons: - Memory required: 31.753 GB; Memory available: 12.570 GB, on Server: NAASC-OVS-02.  
VM: NAO5, has a VNIC: 00:21:f6:2f:c6:4c, with a network: Undefined network, that has no assigned ports - VM: NAO5, has a VNIC: 00:21:f6:99:ea:eb, with a network: Undefined network (2), that has no assigned ports [Thu Jan 04 19:09:15 UTC 2018]

GRRRRRRRRRRRR - had to cut memory down to 8000.

And the VM decided to start on OVS-02, not OVS-01 like I wanted it to... I wonder if that is what is screwing with the memory ceiling???

Save yourself a lot of aggravation and make sure you shut down any VM you are working on and CLONE it BEFORE you make any major changes. In the event things go sideways, it will be easy to restore to a last known good.

External Links

MZ Sample Installation - https://adcwiki.alma.cl/bin/view/APO/122installation

Morita-san scripts / PPTs - get permission to post, first.

-- MikeHatz - 2018-01-03

This topic: ALMA/NAASC > WebHome > NAASCComputing > Oracle > NAASCOracle122Upgrade
Topic revision: 2018-02-13, MikeHatz
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback