Add Node to 11gR2 Oracle RAC Cluster (11.2.0.3)
0. Environment
1. Pre-installation tasks for GI for a cluster
0.i) Backup OCR
i) Install the cvuqdisk Package for Linux
ii) Verify New Node (HWOS)
iii) Verify Peer (REFNODE)
iv) Verify New Node (New Pre-Node)
v) Run fixup scripts
2. Cluster Node Addition for GI Home.
i) Run addnode.sh script
ii) Run orainstRoot.sh #On nodes rac2
iii) Run root.sh #On nodes rac2
iv) Check Clusterware Resources after ran root.sh
v) Run cluvfy post-addNode script
vi) Check Cluster Nodes
vii) Check TNS Listener
viii) Check ASM Status
ix) Check OCR
x) Check Vote disk
3. Cluster Node Addition for RDBMS Home.
i) Run addnode.sh script
ii) root.sh #On nodes rac2 from RDBMS home
4. Add Instance to Database through Command-Line or you can add via dbca.
i)Pre-task
ii) Add redo thread
iii) Add undo tablespace
iv) Add instance to OCR
v) Add service to new instance via srvctl or you can add via dbca
vi) Check the cluster stack
Let start !!!
One node RAC 11.2.0.3 (Not RAC ONE), single node RAC. Earlier it was two node RAC setup, recently i have deleted 2nd node from cluster for testing.
Node name: RAC1
OS: RHEL 5
DATABASE: nike, instance: nike1
Task: We are going to add node “RAC2” to our existing cluster.
Current status
[root@rac1 ~]# crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
ONLINE ONLINE rac1
ora.DATA1.dg
ONLINE ONLINE rac1
ora.LISTENER.lsnr
ONLINE ONLINE rac1
ora.asm
ONLINE ONLINE rac1 Started
ora.gsd
OFFLINE OFFLINE rac1
ora.net1.network
ONLINE ONLINE rac1
ora.ons
ONLINE ONLINE rac1
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE rac1
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE rac1
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE rac1
ora.cvu
1 ONLINE ONLINE rac1
ora.nike.db
1 ONLINE ONLINE rac1 Open
ora.nike.nike_srv.svc
1 ONLINE ONLINE rac1
ora.oc4j
1 ONLINE ONLINE rac1
ora.rac1.vip
1 ONLINE ONLINE rac1
ora.scan1.vip
1 ONLINE ONLINE rac1
ora.scan2.vip
1 ONLINE ONLINE rac1
ora.scan3.vip
1 ONLINE ONLINE rac1
[root@rac1 ~]#
1. Pre-installation tasks for GI for a cluster
A) Install and Configure the Linux Operating System on the New Node << This is step already done by SA
B) Configure Access to the Shared Storage << This is step already done by SA
C) Install and Configure ASMLib << This is step already done by SA
D) SSH configure << This is step already done by SA
From: Node RAC1
As root
[root@rac1 ~]# ocrconfig -manualbackup
rac1 2015/06/19 23:38:03 /u01/app/11.2.0/grid/cdata/rac-scan/backup_20150619_233803.ocr
[root@rac1 ~]#
i) Install the cvuqdisk Package for Linux
[root@rac2 oracle]# rpm -ivh cvuqdisk-1.0.9-1.rpm Preparing... ########################################### [100%] Using default group oinstall to install package 1:cvuqdisk ########################################### [100%] [root@rac2 oracle]# Note: Without cvuqdisk package, CVU cannot discover shared disks and you will receive the error message "Package cvuqdisk not installed" when the CVU is run. Example below: ============== Checking shared storage accessibility... WARNING: rac2:PRVF-7017 : Package cvuqdisk not installed rac2 No shared storage found Shared storage check failed on nodes "rac2"
As GI Home owner From active node RAC1. [oracle@rac1 ~]$ cluvfy stage -post hwos -n rac2 Performing post-checks for hardware and operating system setup Checking node reachability... Node reachability check passed from node "rac1" Checking user equivalence... User equivalence check passed for user "oracle" Checking node connectivity... Checking hosts config file... Verification of the hosts config file successful Node connectivity passed for subnet "192.168.2.0" with node(s) rac2 TCP connectivity check passed for subnet "192.168.2.0" Node connectivity passed for subnet "192.168.0.0" with node(s) rac2 TCP connectivity check passed for subnet "192.168.0.0" Node connectivity passed for subnet "10.0.4.0" with node(s) rac2 TCP connectivity check passed for subnet "10.0.4.0" Interfaces found on subnet "10.0.4.0" that are likely candidates for VIP are: rac2 eth2:10.0.4.15 Interfaces found on subnet "192.168.2.0" that are likely candidates for a private interconnect are: rac2 eth0:192.168.2.102 Interfaces found on subnet "192.168.0.0" that are likely candidates for a private interconnect are: rac2 eth1:192.168.0.102 Node connectivity check passed Checking multicast communication... Checking subnet "192.168.2.0" for multicast communication with multicast group "230.0.1.0"... Check of subnet "192.168.2.0" for multicast communication with multicast group "230.0.1.0" passed. Checking subnet "192.168.0.0" for multicast communication with multicast group "230.0.1.0"... Check of subnet "192.168.0.0" for multicast communication with multicast group "230.0.1.0" passed. Checking subnet "10.0.4.0" for multicast communication with multicast group "230.0.1.0"... Check of subnet "10.0.4.0" for multicast communication with multicast group "230.0.1.0" passed. Check of multicast communication passed. Check for multiple users with UID value 0 passed Time zone consistency check passed Checking shared storage accessibility... Disk Sharing Nodes (1 in count) ------------------------------------ ------------------------ /dev/sda rac2 Disk Sharing Nodes (1 in count) ------------------------------------ ------------------------ /dev/sdb rac2 Shared storage check was successful on nodes "rac2" Post-check for hardware and operating system setup was successful. [oracle@rac1 ~]$
From active node RAC1
As GI Home owner
[oracle@rac1 ~]$ cluvfy comp peer -refnode rac1 -n rac2 -orainv oinstall -osdba dba -verbose Verifying peer compatibility Checking peer compatibility... Compatibility check: Physical memory [reference node: rac1] Node Name Status Ref. node status Comment ------------ ------------------------ ------------------------ ---------- rac2 3.8633GB (4050940.0KB) 3.8633GB (4050940.0KB) matched Physical memory check passed Compatibility check: Available memory [reference node: rac1] Node Name Status Ref. node status Comment ------------ ------------------------ ------------------------ ---------- rac2 3.6215GB (3797424.0KB) 2.6165GB (2743628.0KB) mismatched Available memory check failed Compatibility check: Swap space [reference node: rac1] Node Name Status Ref. node status Comment ------------ ------------------------ ------------------------ ---------- rac2 1.9994GB (2096472.0KB) 5.5385GB (5807488.0KB) mismatched Swap space check failed Compatibility check: Free disk space for "/u01/app/11.2.0/grid" [reference node: rac1] Node Name Status Ref. node status Comment ------------ ------------------------ ------------------------ ---------- rac2 17.165GB (1.7998848E7KB) 21.2725GB (2.2305792E7KB) mismatched Free disk space check failed Compatibility check: Free disk space for "/tmp" [reference node: rac1] Node Name Status Ref. node status Comment ------------ ------------------------ ------------------------ ---------- rac2 3.707GB (3887104.0KB) 4.4717GB (4688896.0KB) mismatched Free disk space check failed Compatibility check: User existence for "oracle" [reference node: rac1] Node Name Status Ref. node status Comment ------------ ------------------------ ------------------------ ---------- rac2 oracle(1100) oracle(1100) matched User existence for "oracle" check passed Compatibility check: Group existence for "oinstall" [reference node: rac1] Node Name Status Ref. node status Comment ------------ ------------------------ ------------------------ ---------- rac2 oinstall(1000) oinstall(1000) matched Group existence for "oinstall" check passed Compatibility check: Group existence for "dba" [reference node: rac1] Node Name Status Ref. node status Comment ------------ ------------------------ ------------------------ ---------- rac2 dba(1200) dba(1200) matched Group existence for "dba" check passed Compatibility check: Group membership for "oracle" in "oinstall (Primary)" [reference node: rac1] Node Name Status Ref. node status Comment ------------ ------------------------ ------------------------ ---------- rac2 yes yes matched Group membership for "oracle" in "oinstall (Primary)" check passed Compatibility check: Group membership for "oracle" in "dba" [reference node: rac1] Node Name Status Ref. node status Comment ------------ ------------------------ ------------------------ ---------- rac2 yes yes matched Group membership for "oracle" in "dba" check passed Compatibility check: Run level [reference node: rac1] Node Name Status Ref. node status Comment ------------ ------------------------ ------------------------ ---------- rac2 5 5 matched Run level check passed Compatibility check: System architecture [reference node: rac1] Node Name Status Ref. node status Comment ------------ ------------------------ ------------------------ ---------- rac2 x86_64 x86_64 matched System architecture check passed Compatibility check: Kernel version [reference node: rac1] Node Name Status Ref. node status Comment ------------ ------------------------ ------------------------ ---------- rac2 2.6.32-200.13.1.el5uek 2.6.32-200.13.1.el5uek matched Kernel version check passed Compatibility check: Kernel param "semmsl" [reference node: rac1] Node Name Status Ref. node status Comment ------------ ------------------------ ------------------------ ---------- rac2 250 250 matched Kernel param "semmsl" check passed Compatibility check: Kernel param "semmns" [reference node: rac1] Node Name Status Ref. node status Comment ------------ ------------------------ ------------------------ ---------- rac2 32000 32000 matched Kernel param "semmns" check passed Compatibility check: Kernel param "semopm" [reference node: rac1] Node Name Status Ref. node status Comment ------------ ------------------------ ------------------------ ---------- rac2 100 100 matched Kernel param "semopm" check passed Compatibility check: Kernel param "semmni" [reference node: rac1] Node Name Status Ref. node status Comment ------------ ------------------------ ------------------------ ---------- rac2 128 128 matched Kernel param "semmni" check passed Compatibility check: Kernel param "shmmax" [reference node: rac1] Node Name Status Ref. node status Comment ------------ ------------------------ ------------------------ ---------- rac2 2074081280 1054504960 mismatched Kernel param "shmmax" check failed Compatibility check: Kernel param "shmmni" [reference node: rac1] Node Name Status Ref. node status Comment ------------ ------------------------ ------------------------ ---------- rac2 4096 4096 matched Kernel param "shmmni" check passed Compatibility check: Kernel param "shmall" [reference node: rac1] Node Name Status Ref. node status Comment ------------ ------------------------ ------------------------ ---------- rac2 2097152 2097152 matched Kernel param "shmall" check passed Compatibility check: Kernel param "file-max" [reference node: rac1] Node Name Status Ref. node status Comment ------------ ------------------------ ------------------------ ---------- rac2 6815744 6815744 matched Kernel param "file-max" check passed Compatibility check: Kernel param "ip_local_port_range" [reference node: rac1] Node Name Status Ref. node status Comment ------------ ------------------------ ------------------------ ---------- rac2 between 9000.0 & 65500.0 between 9000.0 & 65500.0 matched Kernel param "ip_local_port_range" check passed Compatibility check: Kernel param "rmem_default" [reference node: rac1] Node Name Status Ref. node status Comment ------------ ------------------------ ------------------------ ---------- rac2 262144 262144 matched Kernel param "rmem_default" check passed Compatibility check: Kernel param "rmem_max" [reference node: rac1] Node Name Status Ref. node status Comment ------------ ------------------------ ------------------------ ---------- rac2 4194304 4194304 matched Kernel param "rmem_max" check passed Compatibility check: Kernel param "wmem_default" [reference node: rac1] Node Name Status Ref. node status Comment ------------ ------------------------ ------------------------ ---------- rac2 262144 262144 matched Kernel param "wmem_default" check passed Compatibility check: Kernel param "wmem_max" [reference node: rac1] Node Name Status Ref. node status Comment ------------ ------------------------ ------------------------ ---------- rac2 1048586 1048586 matched Kernel param "wmem_max" check passed Compatibility check: Kernel param "aio-max-nr" [reference node: rac1] Node Name Status Ref. node status Comment ------------ ------------------------ ------------------------ ---------- rac2 1048576 1048576 matched Kernel param "aio-max-nr" check passed Compatibility check: Package existence for "make" [reference node: rac1] Node Name Status Ref. node status Comment ------------ ------------------------ ------------------------ ---------- rac2 make-3.81-3.el5 make-3.81-3.el5 matched Package existence for "make" check passed Compatibility check: Package existence for "binutils" [reference node: rac1] Node Name Status Ref. node status Comment ------------ ------------------------ ------------------------ ---------- rac2 binutils-2.17.50.0.6-14.el5 binutils-2.17.50.0.6-14.el5 matched Package existence for "binutils" check passed Compatibility check: Package existence for "gcc (x86_64)" [reference node: rac1] Node Name Status Ref. node status Comment ------------ ------------------------ ------------------------ ---------- rac2 gcc-4.1.2-51.el5 (x86_64) gcc-4.1.2-51.el5 (x86_64) matched Package existence for "gcc (x86_64)" check passed Compatibility check: Package existence for "libaio (x86_64)" [reference node: rac1] Node Name Status Ref. node status Comment ------------ ------------------------ ------------------------ ---------- rac2 libaio-0.3.106-5 (x86_64),libaio-0.3.106-5 (i386) libaio-0.3.106-5 (x86_64),libaio-0.3.106-5 (i386) matched Package existence for "libaio (x86_64)" check passed Compatibility check: Package existence for "glibc (x86_64)" [reference node: rac1] Node Name Status Ref. node status Comment ------------ ------------------------ ------------------------ ---------- rac2 glibc-2.5-65 (x86_64),glibc-2.5-65 (i686) glibc-2.5-65 (x86_64),glibc-2.5-65 (i686) matched Package existence for "glibc (x86_64)" check passed Compatibility check: Package existence for "compat-libstdc++-33 (x86_64)" [reference node: rac1] Node Name Status Ref. node status Comment ------------ ------------------------ ------------------------ ---------- rac2 compat-libstdc++-33-3.2.3-61 (x86_64),compat-libstdc++-33-3.2.3-61 (i386) compat-libstdc++-33-3.2.3-61 (x86_64),compat-libstdc++-33-3.2.3-61 (i386) matched Package existence for "compat-libstdc++-33 (x86_64)" check passed Compatibility check: Package existence for "elfutils-libelf (x86_64)" [reference node: rac1] Node Name Status Ref. node status Comment ------------ ------------------------ ------------------------ ---------- rac2 elfutils-libelf-0.137-3.el5 (x86_64),elfutils-libelf-0.137-3.el5 (i386) elfutils-libelf-0.137-3.el5 (x86_64),elfutils-libelf-0.137-3.el5 (i386) matched Package existence for "elfutils-libelf (x86_64)" check passed Compatibility check: Package existence for "elfutils-libelf-devel" [reference node: rac1] Node Name Status Ref. node status Comment ------------ ------------------------ ------------------------ ---------- rac2 elfutils-libelf-devel-0.137-3.el5 elfutils-libelf-devel-0.137-3.el5 matched Package existence for "elfutils-libelf-devel" check passed Compatibility check: Package existence for "glibc-common" [reference node: rac1] Node Name Status Ref. node status Comment ------------ ------------------------ ------------------------ ---------- rac2 glibc-common-2.5-65 glibc-common-2.5-65 matched Package existence for "glibc-common" check passed Compatibility check: Package existence for "glibc-devel (x86_64)" [reference node: rac1] Node Name Status Ref. node status Comment ------------ ------------------------ ------------------------ ---------- rac2 glibc-devel-2.5-65 (x86_64),glibc-devel-2.5-65 (i386) glibc-devel-2.5-65 (x86_64),glibc-devel-2.5-65 (i386) matched Package existence for "glibc-devel (x86_64)" check passed Compatibility check: Package existence for "glibc-headers" [reference node: rac1] Node Name Status Ref. node status Comment ------------ ------------------------ ------------------------ ---------- rac2 glibc-headers-2.5-65 glibc-headers-2.5-65 matched Package existence for "glibc-headers" check passed Compatibility check: Package existence for "gcc-c++ (x86_64)" [reference node: rac1] Node Name Status Ref. node status Comment ------------ ------------------------ ------------------------ ---------- rac2 gcc-c++-4.1.2-51.el5 (x86_64) gcc-c++-4.1.2-51.el5 (x86_64) matched Package existence for "gcc-c++ (x86_64)" check passed Compatibility check: Package existence for "libaio-devel (x86_64)" [reference node: rac1] Node Name Status Ref. node status Comment ------------ ------------------------ ------------------------ ---------- rac2 libaio-devel-0.3.106-5 (i386),libaio-devel-0.3.106-5 (x86_64) libaio-devel-0.3.106-5 (i386),libaio-devel-0.3.106-5 (x86_64) matched Package existence for "libaio-devel (x86_64)" check passed Compatibility check: Package existence for "libgcc (x86_64)" [reference node: rac1] Node Name Status Ref. node status Comment ------------ ------------------------ ------------------------ ---------- rac2 libgcc-4.1.2-51.el5 (x86_64),libgcc-4.1.2-51.el5 (i386) libgcc-4.1.2-51.el5 (x86_64),libgcc-4.1.2-51.el5 (i386) matched Package existence for "libgcc (x86_64)" check passed Compatibility check: Package existence for "libstdc++ (x86_64)" [reference node: rac1] Node Name Status Ref. node status Comment ------------ ------------------------ ------------------------ ---------- rac2 libstdc++-4.1.2-51.el5 (x86_64),libstdc++-4.1.2-51.el5 (i386) libstdc++-4.1.2-51.el5 (x86_64),libstdc++-4.1.2-51.el5 (i386) matched Package existence for "libstdc++ (x86_64)" check passed Compatibility check: Package existence for "libstdc++-devel (x86_64)" [reference node: rac1] Node Name Status Ref. node status Comment ------------ ------------------------ ------------------------ ---------- rac2 libstdc++-devel-4.1.2-51.el5 (x86_64) libstdc++-devel-4.1.2-51.el5 (x86_64) matched Package existence for "libstdc++-devel (x86_64)" check passed Compatibility check: Package existence for "sysstat" [reference node: rac1] Node Name Status Ref. node status Comment ------------ ------------------------ ------------------------ ---------- rac2 sysstat-7.0.2-11.el5 sysstat-7.0.2-11.el5 matched Package existence for "sysstat" check passed Compatibility check: Package existence for "ksh" [reference node: rac1] Node Name Status Ref. node status Comment ------------ ------------------------ ------------------------ ---------- rac2 ksh-20100202-1.el5_6.6 ksh-20100202-1.el5_6.6 matched Package existence for "ksh" check passed Verification of peer compatibility was unsuccessful. Checks did not pass for the following node(s): rac2 [oracle@rac1 ~]$
iv) Verify New Node (New Pre-Node)
As GI Home owner
From Node RAC1
[oracle@rac1 ~]$ cluvfy stage -pre nodeadd -n rac2 -fixup -verbose Performing pre-checks for node addition Checking node reachability... Check: Node reachability from node "rac1" Destination Node Reachable? ------------------------------------ ------------------------ rac2 yes Result: Node reachability check passed from node "rac1" Checking user equivalence... Check: User equivalence for user "oracle" Node Name Status ------------------------------------ ------------------------ rac2 passed Result: User equivalence check passed for user "oracle" Checking node connectivity... Checking hosts config file... Node Name Status ------------------------------------ ------------------------ rac1 passed rac2 passed Verification of the hosts config file successful Interface information for node "rac1" Name IP Address Subnet Gateway Def. Gateway HW Address MTU ------ --------------- --------------- --------------- --------------- ----------------- ------ eth0 192.168.2.101 192.168.2.0 0.0.0.0 10.0.4.2 08:00:27:02:46:97 1500 eth0 192.168.2.106 192.168.2.0 0.0.0.0 10.0.4.2 08:00:27:02:46:97 1500 eth0 192.168.2.107 192.168.2.0 0.0.0.0 10.0.4.2 08:00:27:02:46:97 1500 eth0 192.168.2.103 192.168.2.0 0.0.0.0 10.0.4.2 08:00:27:02:46:97 1500 eth0 192.168.2.105 192.168.2.0 0.0.0.0 10.0.4.2 08:00:27:02:46:97 1500 eth1 192.168.0.101 192.168.0.0 0.0.0.0 10.0.4.2 08:00:27:9A:66:6A 1500 eth1 169.254.6.127 169.254.0.0 0.0.0.0 10.0.4.2 08:00:27:9A:66:6A 1500 eth2 10.0.4.15 10.0.4.0 0.0.0.0 10.0.4.2 08:00:27:D3:B8:9F 1500 Interface information for node "rac2" Name IP Address Subnet Gateway Def. Gateway HW Address MTU ------ --------------- --------------- --------------- --------------- ----------------- ------ eth0 192.168.2.102 192.168.2.0 0.0.0.0 10.0.4.2 08:00:27:93:7F:00 1500 eth1 192.168.0.102 192.168.0.0 0.0.0.0 10.0.4.2 08:00:27:63:80:76 1500 eth2 10.0.4.15 10.0.4.0 0.0.0.0 10.0.4.2 08:00:27:4C:B3:01 1500 Check: Node connectivity for interface "eth0" Source Destination Connected? ------------------------------ ------------------------------ ---------------- rac1[192.168.2.101] rac1[192.168.2.106] yes rac1[192.168.2.101] rac1[192.168.2.107] yes rac1[192.168.2.101] rac1[192.168.2.103] yes rac1[192.168.2.101] rac1[192.168.2.105] yes rac1[192.168.2.101] rac2[192.168.2.102] yes rac1[192.168.2.106] rac1[192.168.2.107] yes rac1[192.168.2.106] rac1[192.168.2.103] yes rac1[192.168.2.106] rac1[192.168.2.105] yes rac1[192.168.2.106] rac2[192.168.2.102] yes rac1[192.168.2.107] rac1[192.168.2.103] yes rac1[192.168.2.107] rac1[192.168.2.105] yes rac1[192.168.2.107] rac2[192.168.2.102] yes rac1[192.168.2.103] rac1[192.168.2.105] yes rac1[192.168.2.103] rac2[192.168.2.102] yes rac1[192.168.2.105] rac2[192.168.2.102] yes Result: Node connectivity passed for interface "eth0" Check: TCP connectivity of subnet "192.168.2.0" Source Destination Connected? ------------------------------ ------------------------------ ---------------- rac1:192.168.2.101 rac1:192.168.2.106 passed rac1:192.168.2.101 rac1:192.168.2.107 passed rac1:192.168.2.101 rac1:192.168.2.103 passed rac1:192.168.2.101 rac1:192.168.2.105 passed rac1:192.168.2.101 rac2:192.168.2.102 passed Result: TCP connectivity check passed for subnet "192.168.2.0" Checking subnet mask consistency... Subnet mask consistency check passed for subnet "192.168.2.0". Subnet mask consistency check passed. Result: Node connectivity check passed Checking multicast communication... Checking subnet "192.168.2.0" for multicast communication with multicast group "230.0.1.0"... Check of subnet "192.168.2.0" for multicast communication with multicast group "230.0.1.0" passed. Check of multicast communication passed. Checking CRS integrity... Clusterware version consistency passed The Oracle Clusterware is healthy on node "rac1" CRS integrity check passed Checking shared resources... Checking CRS home location... "/u01/app/11.2.0/grid" is shared Result: Shared resources check for node addition passed Checking node connectivity... Checking hosts config file... Node Name Status ------------------------------------ ------------------------ rac1 passed rac2 passed Verification of the hosts config file successful Interface information for node "rac1" Name IP Address Subnet Gateway Def. Gateway HW Address MTU ------ --------------- --------------- --------------- --------------- ----------------- ------ eth0 192.168.2.101 192.168.2.0 0.0.0.0 10.0.4.2 08:00:27:02:46:97 1500 eth0 192.168.2.106 192.168.2.0 0.0.0.0 10.0.4.2 08:00:27:02:46:97 1500 eth0 192.168.2.107 192.168.2.0 0.0.0.0 10.0.4.2 08:00:27:02:46:97 1500 eth0 192.168.2.103 192.168.2.0 0.0.0.0 10.0.4.2 08:00:27:02:46:97 1500 eth0 192.168.2.105 192.168.2.0 0.0.0.0 10.0.4.2 08:00:27:02:46:97 1500 eth1 192.168.0.101 192.168.0.0 0.0.0.0 10.0.4.2 08:00:27:9A:66:6A 1500 eth1 169.254.6.127 169.254.0.0 0.0.0.0 10.0.4.2 08:00:27:9A:66:6A 1500 eth2 10.0.4.15 10.0.4.0 0.0.0.0 10.0.4.2 08:00:27:D3:B8:9F 1500 Interface information for node "rac2" Name IP Address Subnet Gateway Def. Gateway HW Address MTU ------ --------------- --------------- --------------- --------------- ----------------- ------ eth0 192.168.2.102 192.168.2.0 0.0.0.0 10.0.4.2 08:00:27:93:7F:00 1500 eth1 192.168.0.102 192.168.0.0 0.0.0.0 10.0.4.2 08:00:27:63:80:76 1500 eth2 10.0.4.15 10.0.4.0 0.0.0.0 10.0.4.2 08:00:27:4C:B3:01 1500 Check: Node connectivity for interface "eth0" Source Destination Connected? ------------------------------ ------------------------------ ---------------- rac1[192.168.2.101] rac1[192.168.2.106] yes rac1[192.168.2.101] rac1[192.168.2.107] yes rac1[192.168.2.101] rac1[192.168.2.103] yes rac1[192.168.2.101] rac1[192.168.2.105] yes rac1[192.168.2.101] rac2[192.168.2.102] yes rac1[192.168.2.106] rac1[192.168.2.107] yes rac1[192.168.2.106] rac1[192.168.2.103] yes rac1[192.168.2.106] rac1[192.168.2.105] yes rac1[192.168.2.106] rac2[192.168.2.102] yes rac1[192.168.2.107] rac1[192.168.2.103] yes rac1[192.168.2.107] rac1[192.168.2.105] yes rac1[192.168.2.107] rac2[192.168.2.102] yes rac1[192.168.2.103] rac1[192.168.2.105] yes rac1[192.168.2.103] rac2[192.168.2.102] yes rac1[192.168.2.105] rac2[192.168.2.102] yes Result: Node connectivity passed for interface "eth0" Check: TCP connectivity of subnet "192.168.2.0" Source Destination Connected? ------------------------------ ------------------------------ ---------------- rac1:192.168.2.101 rac1:192.168.2.106 passed rac1:192.168.2.101 rac1:192.168.2.107 passed rac1:192.168.2.101 rac1:192.168.2.103 passed rac1:192.168.2.101 rac1:192.168.2.105 passed rac1:192.168.2.101 rac2:192.168.2.102 passed Result: TCP connectivity check passed for subnet "192.168.2.0" Check: Node connectivity for interface "eth1" Source Destination Connected? ------------------------------ ------------------------------ ---------------- rac1[192.168.0.101] rac2[192.168.0.102] yes Result: Node connectivity passed for interface "eth1" Check: TCP connectivity of subnet "192.168.0.0" Source Destination Connected? ------------------------------ ------------------------------ ---------------- rac1:192.168.0.101 rac2:192.168.0.102 passed Result: TCP connectivity check passed for subnet "192.168.0.0" Checking subnet mask consistency... Subnet mask consistency check passed for subnet "192.168.2.0". Subnet mask consistency check passed for subnet "192.168.0.0". Subnet mask consistency check passed. Result: Node connectivity check passed Checking multicast communication... Checking subnet "192.168.2.0" for multicast communication with multicast group "230.0.1.0"... Check of subnet "192.168.2.0" for multicast communication with multicast group "230.0.1.0" passed. Checking subnet "192.168.0.0" for multicast communication with multicast group "230.0.1.0"... Check of subnet "192.168.0.0" for multicast communication with multicast group "230.0.1.0" passed. Check of multicast communication passed. Check: Total memory Node Name Available Required Status ------------ ------------------------ ------------------------ ---------- rac2 3.8633GB (4050940.0KB) 1.5GB (1572864.0KB) passed rac1 3.8633GB (4050940.0KB) 1.5GB (1572864.0KB) passed Result: Total memory check passed Check: Available memory Node Name Available Required Status ------------ ------------------------ ------------------------ ---------- rac2 3.6216GB (3797500.0KB) 50MB (51200.0KB) passed rac1 2.4853GB (2606016.0KB) 50MB (51200.0KB) passed Result: Available memory check passed Check: Swap space Node Name Available Required Status ------------ ------------------------ ------------------------ ---------- rac2 1.9994GB (2096472.0KB) 3.8633GB (4050940.0KB) failed <<<<< rac1 5.5385GB (5807488.0KB) 3.8633GB (4050940.0KB) passed Result: Swap space check failed Check: Free disk space for "rac2:/u01/app/11.2.0/grid" Path Node Name Mount point Available Required Status ---------------- ------------ ------------ ------------ ------------ ------------ /u01/app/11.2.0/grid rac2 /u01 17.165GB 5.5GB passed Result: Free disk space check passed for "rac2:/u01/app/11.2.0/grid" Check: Free disk space for "rac1:/u01/app/11.2.0/grid" Path Node Name Mount point Available Required Status ---------------- ------------ ------------ ------------ ------------ ------------ /u01/app/11.2.0/grid rac1 /u01 21.2705GB 5.5GB passed Result: Free disk space check passed for "rac1:/u01/app/11.2.0/grid" Check: Free disk space for "rac2:/tmp" Path Node Name Mount point Available Required Status ---------------- ------------ ------------ ------------ ------------ ------------ /tmp rac2 / 3.707GB 1GB passed Result: Free disk space check passed for "rac2:/tmp" Check: Free disk space for "rac1:/tmp" Path Node Name Mount point Available Required Status ---------------- ------------ ------------ ------------ ------------ ------------ /tmp rac1 / 4.4717GB 1GB passed Result: Free disk space check passed for "rac1:/tmp" Check: User existence for "oracle" Node Name Status Comment ------------ ------------------------ ------------------------ rac2 passed exists(1100) rac1 passed exists(1100) Checking for multiple users with UID value 1100 Result: Check for multiple users with UID value 1100 passed Result: User existence check passed for "oracle" Check: Run level Node Name run level Required Status ------------ ------------------------ ------------------------ ---------- rac2 5 3,5 passed rac1 5 3,5 passed Result: Run level check passed Check: Hard limits for "maximum open file descriptors" Node Name Type Available Required Status ---------------- ------------ ------------ ------------ ---------------- rac1 hard 65536 65536 passed rac2 hard 65536 65536 passed Result: Hard limits check passed for "maximum open file descriptors" Check: Soft limits for "maximum open file descriptors" Node Name Type Available Required Status ---------------- ------------ ------------ ------------ ---------------- rac1 soft 1024 1024 passed rac2 soft 1024 1024 passed Result: Soft limits check passed for "maximum open file descriptors" Check: Hard limits for "maximum user processes" Node Name Type Available Required Status ---------------- ------------ ------------ ------------ ---------------- rac1 hard 16384 16384 passed rac2 hard 16384 16384 passed Result: Hard limits check passed for "maximum user processes" Check: Soft limits for "maximum user processes" Node Name Type Available Required Status ---------------- ------------ ------------ ------------ ---------------- rac1 soft 2047 2047 passed rac2 soft 2047 2047 passed Result: Soft limits check passed for "maximum user processes" Check: System architecture Node Name Available Required Status ------------ ------------------------ ------------------------ ---------- rac2 x86_64 x86_64 passed rac1 x86_64 x86_64 passed Result: System architecture check passed Check: Kernel version Node Name Available Required Status ------------ ------------------------ ------------------------ ---------- rac2 2.6.32-200.13.1.el5uek 2.6.18 passed rac1 2.6.32-200.13.1.el5uek 2.6.18 passed Result: Kernel version check passed Check: Kernel parameter for "semmsl" Node Name Current Configured Required Status Comment ---------------- ------------ ------------ ------------ ------------ ------------ rac1 250 250 250 passed rac2 250 250 250 passed Result: Kernel parameter check passed for "semmsl" Check: Kernel parameter for "semmns" Node Name Current Configured Required Status Comment ---------------- ------------ ------------ ------------ ------------ ------------ rac1 32000 32000 32000 passed rac2 32000 32000 32000 passed Result: Kernel parameter check passed for "semmns" Check: Kernel parameter for "semopm" Node Name Current Configured Required Status Comment ---------------- ------------ ------------ ------------ ------------ ------------ rac1 100 100 100 passed rac2 100 100 100 passed Result: Kernel parameter check passed for "semopm" Check: Kernel parameter for "semmni" Node Name Current Configured Required Status Comment ---------------- ------------ ------------ ------------ ------------ ------------ rac1 128 128 128 passed rac2 128 128 128 passed Result: Kernel parameter check passed for "semmni" Check: Kernel parameter for "shmmax" Node Name Current Configured Required Status Comment ---------------- ------------ ------------ ------------ ------------ ------------ rac1 1054504960 1054504960 2074081280 failed Current value too low. Configured value too low. <<< rac2 2074081280 1054504960 2074081280 failed Configured value too low. <<< Result: Kernel parameter check failed for "shmmax" Check: Kernel parameter for "shmmni" Node Name Current Configured Required Status Comment ---------------- ------------ ------------ ------------ ------------ ------------ rac1 4096 4096 4096 passed rac2 4096 4096 4096 passed Result: Kernel parameter check passed for "shmmni" Check: Kernel parameter for "shmall" Node Name Current Configured Required Status Comment ---------------- ------------ ------------ ------------ ------------ ------------ rac1 2097152 2097152 2097152 passed rac2 2097152 2097152 2097152 passed Result: Kernel parameter check passed for "shmall" Check: Kernel parameter for "file-max" Node Name Current Configured Required Status Comment ---------------- ------------ ------------ ------------ ------------ ------------ rac1 6815744 6815744 6815744 passed rac2 6815744 6815744 6815744 passed Result: Kernel parameter check passed for "file-max" Check: Kernel parameter for "ip_local_port_range" Node Name Current Configured Required Status Comment ---------------- ------------ ------------ ------------ ------------ ------------ rac1 between 9000.0 & 65500.0 between 9000.0 & 65500.0 between 9000.0 & 65500.0 passed rac2 between 9000.0 & 65500.0 between 9000.0 & 65500.0 between 9000.0 & 65500.0 passed Result: Kernel parameter check passed for "ip_local_port_range" Check: Kernel parameter for "rmem_default" Node Name Current Configured Required Status Comment ---------------- ------------ ------------ ------------ ------------ ------------ rac1 262144 262144 262144 passed rac2 262144 262144 262144 passed Result: Kernel parameter check passed for "rmem_default" Check: Kernel parameter for "rmem_max" Node Name Current Configured Required Status Comment ---------------- ------------ ------------ ------------ ------------ ------------ rac1 4194304 4194304 4194304 passed rac2 4194304 4194304 4194304 passed Result: Kernel parameter check passed for "rmem_max" Check: Kernel parameter for "wmem_default" Node Name Current Configured Required Status Comment ---------------- ------------ ------------ ------------ ------------ ------------ rac1 262144 262144 262144 passed rac2 262144 262144 262144 passed Result: Kernel parameter check passed for "wmem_default" Check: Kernel parameter for "wmem_max" Node Name Current Configured Required Status Comment ---------------- ------------ ------------ ------------ ------------ ------------ rac1 1048586 1048586 1048576 passed rac2 1048586 1048586 1048576 passed Result: Kernel parameter check passed for "wmem_max" Check: Kernel parameter for "aio-max-nr" Node Name Current Configured Required Status Comment ---------------- ------------ ------------ ------------ ------------ ------------ rac1 1048576 1048576 1048576 passed rac2 1048576 1048576 1048576 passed Result: Kernel parameter check passed for "aio-max-nr" Check: Package existence for "make" Node Name Available Required Status ------------ ------------------------ ------------------------ ---------- rac2 make-3.81-3.el5 make-3.81 passed rac1 make-3.81-3.el5 make-3.81 passed Result: Package existence check passed for "make" Check: Package existence for "binutils" Node Name Available Required Status ------------ ------------------------ ------------------------ ---------- rac2 binutils-2.17.50.0.6-14.el5 binutils-2.17.50.0.6 passed rac1 binutils-2.17.50.0.6-14.el5 binutils-2.17.50.0.6 passed Result: Package existence check passed for "binutils" Check: Package existence for "gcc(x86_64)" Node Name Available Required Status ------------ ------------------------ ------------------------ ---------- rac2 gcc(x86_64)-4.1.2-51.el5 gcc(x86_64)-4.1.2 passed rac1 gcc(x86_64)-4.1.2-51.el5 gcc(x86_64)-4.1.2 passed Result: Package existence check passed for "gcc(x86_64)" Check: Package existence for "libaio(x86_64)" Node Name Available Required Status ------------ ------------------------ ------------------------ ---------- rac2 libaio(x86_64)-0.3.106-5 libaio(x86_64)-0.3.106 passed rac1 libaio(x86_64)-0.3.106-5 libaio(x86_64)-0.3.106 passed Result: Package existence check passed for "libaio(x86_64)" Check: Package existence for "glibc(x86_64)" Node Name Available Required Status ------------ ------------------------ ------------------------ ---------- rac2 glibc(x86_64)-2.5-65 glibc(x86_64)-2.5-24 passed rac1 glibc(x86_64)-2.5-65 glibc(x86_64)-2.5-24 passed Result: Package existence check passed for "glibc(x86_64)" Check: Package existence for "compat-libstdc++-33(x86_64)" Node Name Available Required Status ------------ ------------------------ ------------------------ ---------- rac2 compat-libstdc++-33(x86_64)-3.2.3-61 compat-libstdc++-33(x86_64)-3.2.3 passed rac1 compat-libstdc++-33(x86_64)-3.2.3-61 compat-libstdc++-33(x86_64)-3.2.3 passed Result: Package existence check passed for "compat-libstdc++-33(x86_64)" Check: Package existence for "elfutils-libelf(x86_64)" Node Name Available Required Status ------------ ------------------------ ------------------------ ---------- rac2 elfutils-libelf(x86_64)-0.137-3.el5 elfutils-libelf(x86_64)-0.125 passed rac1 elfutils-libelf(x86_64)-0.137-3.el5 elfutils-libelf(x86_64)-0.125 passed Result: Package existence check passed for "elfutils-libelf(x86_64)" Check: Package existence for "elfutils-libelf-devel" Node Name Available Required Status ------------ ------------------------ ------------------------ ---------- rac2 elfutils-libelf-devel-0.137-3.el5 elfutils-libelf-devel-0.125 passed rac1 elfutils-libelf-devel-0.137-3.el5 elfutils-libelf-devel-0.125 passed Result: Package existence check passed for "elfutils-libelf-devel" Check: Package existence for "glibc-common" Node Name Available Required Status ------------ ------------------------ ------------------------ ---------- rac2 glibc-common-2.5-65 glibc-common-2.5 passed rac1 glibc-common-2.5-65 glibc-common-2.5 passed Result: Package existence check passed for "glibc-common" Check: Package existence for "glibc-devel(x86_64)" Node Name Available Required Status ------------ ------------------------ ------------------------ ---------- rac2 glibc-devel(x86_64)-2.5-65 glibc-devel(x86_64)-2.5 passed rac1 glibc-devel(x86_64)-2.5-65 glibc-devel(x86_64)-2.5 passed Result: Package existence check passed for "glibc-devel(x86_64)" Check: Package existence for "glibc-headers" Node Name Available Required Status ------------ ------------------------ ------------------------ ---------- rac2 glibc-headers-2.5-65 glibc-headers-2.5 passed rac1 glibc-headers-2.5-65 glibc-headers-2.5 passed Result: Package existence check passed for "glibc-headers" Check: Package existence for "gcc-c++(x86_64)" Node Name Available Required Status ------------ ------------------------ ------------------------ ---------- rac2 gcc-c++(x86_64)-4.1.2-51.el5 gcc-c++(x86_64)-4.1.2 passed rac1 gcc-c++(x86_64)-4.1.2-51.el5 gcc-c++(x86_64)-4.1.2 passed Result: Package existence check passed for "gcc-c++(x86_64)" Check: Package existence for "libaio-devel(x86_64)" Node Name Available Required Status ------------ ------------------------ ------------------------ ---------- rac2 libaio-devel(x86_64)-0.3.106-5 libaio-devel(x86_64)-0.3.106 passed rac1 libaio-devel(x86_64)-0.3.106-5 libaio-devel(x86_64)-0.3.106 passed Result: Package existence check passed for "libaio-devel(x86_64)" Check: Package existence for "libgcc(x86_64)" Node Name Available Required Status ------------ ------------------------ ------------------------ ---------- rac2 libgcc(x86_64)-4.1.2-51.el5 libgcc(x86_64)-4.1.2 passed rac1 libgcc(x86_64)-4.1.2-51.el5 libgcc(x86_64)-4.1.2 passed Result: Package existence check passed for "libgcc(x86_64)" Check: Package existence for "libstdc++(x86_64)" Node Name Available Required Status ------------ ------------------------ ------------------------ ---------- rac2 libstdc++(x86_64)-4.1.2-51.el5 libstdc++(x86_64)-4.1.2 passed rac1 libstdc++(x86_64)-4.1.2-51.el5 libstdc++(x86_64)-4.1.2 passed Result: Package existence check passed for "libstdc++(x86_64)" Check: Package existence for "libstdc++-devel(x86_64)" Node Name Available Required Status ------------ ------------------------ ------------------------ ---------- rac2 libstdc++-devel(x86_64)-4.1.2-51.el5 libstdc++-devel(x86_64)-4.1.2 passed rac1 libstdc++-devel(x86_64)-4.1.2-51.el5 libstdc++-devel(x86_64)-4.1.2 passed Result: Package existence check passed for "libstdc++-devel(x86_64)" Check: Package existence for "sysstat" Node Name Available Required Status ------------ ------------------------ ------------------------ ---------- rac2 sysstat-7.0.2-11.el5 sysstat-7.0.2 passed rac1 sysstat-7.0.2-11.el5 sysstat-7.0.2 passed Result: Package existence check passed for "sysstat" Check: Package existence for "ksh" Node Name Available Required Status ------------ ------------------------ ------------------------ ---------- rac2 ksh-20100202-1.el5_6.6 ksh-20060214 passed rac1 ksh-20100202-1.el5_6.6 ksh-20060214 passed Result: Package existence check passed for "ksh" Checking for multiple users with UID value 0 Result: Check for multiple users with UID value 0 passed Check: Current group ID Result: Current group ID check passed Starting check for consistency of primary group of root user Node Name Status ------------------------------------ ------------------------ rac2 passed rac1 passed Check for consistency of root user's primary group passed Checking OCR integrity... OCR integrity check passed Checking Oracle Cluster Voting Disk configuration... Oracle Cluster Voting Disk configuration check passed Check: Time zone consistency Result: Time zone consistency check passed Starting Clock synchronization checks using Network Time Protocol(NTP)... NTP Configuration file check started... Network Time Protocol(NTP) configuration file not found on any of the nodes. Oracle Cluster Time Synchronization Service(CTSS) can be used instead of NTP for time synchronization on the cluster nodes No NTP Daemons or Services were found to be running Result: Clock synchronization check using Network Time Protocol(NTP) passed Checking to make sure user "oracle" is not in "root" group Node Name Status Comment ------------ ------------------------ ------------------------ rac2 passed does not exist rac1 passed does not exist Result: User "oracle" is not part of "root" group. Check passed Checking consistency of file "/etc/resolv.conf" across nodes Checking the file "/etc/resolv.conf" to make sure only one of domain and search entries is defined File "/etc/resolv.conf" does not have both domain and search entries defined Checking if domain entry in file "/etc/resolv.conf" is consistent across the nodes... domain entry in file "/etc/resolv.conf" is consistent across nodes Checking if search entry in file "/etc/resolv.conf" is consistent across the nodes... search entry in file "/etc/resolv.conf" is consistent across nodes Checking file "/etc/resolv.conf" to make sure that only one search entry is defined All nodes have one search entry defined in file "/etc/resolv.conf" Checking all nodes to make sure that search entry is "rajasekhar.com" as found on node "rac1" All nodes of the cluster have same value for 'search' Checking DNS response time for an unreachable node Node Name Status ------------------------------------ ------------------------ rac1 failed rac2 failed PRVF-5636 : The DNS response time for an unreachable node exceeded "15000" ms on following nodes: rac1,rac2 File "/etc/resolv.conf" is not consistent across nodes Fixup information has been generated for following node(s): rac2,rac1 Please run the following script on each node as "root" user to execute the fixups: '/tmp/CVU_11.2.0.3.0_oracle/runfixup.sh' Pre-check for node addition was unsuccessful on all the nodes. [oracle@rac1 ~]$
On RAC 1, as root user
[root@rac1 ~]# /tmp/CVU_11.2.0.3.0_oracle/runfixup.sh
Response file being used is :/tmp/CVU_11.2.0.3.0_oracle/fixup.response
Enable file being used is :/tmp/CVU_11.2.0.3.0_oracle/fixup.enable
Log file location: /tmp/CVU_11.2.0.3.0_oracle/orarun.log
Setting Kernel Parameters...
kernel.shmmax = 68719476736
kernel.shmmax = 1054504960
/tmp/CVU_11.2.0.3.0_oracle/orarun.sh: line 230: [: 68719476736kernel.shmmax: integer expression expected
The value for shmmax in response file is not greater than value for shmmax in /etc/sysctl.conf file. Hence not changing it.
kernel.shmmax = 2074081280
[root@rac1 ~]#
On RAC2, as root user
[root@rac2 ~]# /tmp/CVU_11.2.0.3.0_oracle/runfixup.sh
Response file being used is :/tmp/CVU_11.2.0.3.0_oracle/fixup.response
Enable file being used is :/tmp/CVU_11.2.0.3.0_oracle/fixup.enable
Log file location: /tmp/CVU_11.2.0.3.0_oracle/orarun.log
Setting Kernel Parameters...
kernel.shmmax = 68719476736
kernel.shmmax = 1054504960
/tmp/CVU_11.2.0.3.0_oracle/orarun.sh: line 230: [: 68719476736kernel.shmmax: integer expression expected
The value for shmmax in response file is not greater than value for shmmax in /etc/sysctl.conf file. Hence not changing it.
The value for shmmax in response file is not greater than value of shmmax for current session. Hence not changing it.
[root@rac2 ~]#
2. Cluster Node Addition for GI Home.
Note: Pre-node check failed. Hence need to set the environment variable IGNORE_PREADDNODE_CHECKS=Y before running addNode.sh otherwise, the silent node addition will fail without showing any errors to the console.
As GI Home owner
From active node RAC1
[oracle@rac1 ~]$ cd /u01/app/11.2.0/grid/oui/bin/ [oracle@rac1 bin]$ IGNORE_PREADDNODE_CHECKS=Y [oracle@rac1 bin]$ export IGNORE_PREADDNODE_CHECKS [oracle@rac1 bin]$ ./addNode.sh -silent "CLUSTER_NEW_NODES={rac2}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={rac2-vip}" Starting Oracle Universal Installer... Checking swap space: must be greater than 500 MB. Actual 5671 MB Passed Oracle Universal Installer, Version 11.2.0.3.0 Production Copyright (C) 1999, 2011, Oracle. All rights reserved. Performing tests to see whether nodes rac2 are available ............................................................... 100% Done. . ----------------------------------------------------------------------------- Cluster Node Addition Summary Global Settings Source: /u01/app/11.2.0/grid New Nodes Space Requirements New Nodes rac2 /u01: Required 6.99GB : Available 15.98GB Installed Products Product Names Oracle Grid Infrastructure 11.2.0.3.0 Sun JDK 1.5.0.30.03 Installer SDK Component 11.2.0.3.0 Oracle One-Off Patch Installer 11.2.0.1.7 Oracle Universal Installer 11.2.0.3.0 Oracle USM Deconfiguration 11.2.0.3.0 Oracle Configuration Manager Deconfiguration 10.3.1.0.0 Enterprise Manager Common Core Files 10.2.0.4.4 Oracle DBCA Deconfiguration 11.2.0.3.0 Oracle RAC Deconfiguration 11.2.0.3.0 Oracle Quality of Service Management (Server) 11.2.0.3.0 Installation Plugin Files 11.2.0.3.0 Universal Storage Manager Files 11.2.0.3.0 Oracle Text Required Support Files 11.2.0.3.0 Automatic Storage Management Assistant 11.2.0.3.0 Oracle Database 11g Multimedia Files 11.2.0.3.0 Oracle Multimedia Java Advanced Imaging 11.2.0.3.0 Oracle Globalization Support 11.2.0.3.0 Oracle Multimedia Locator RDBMS Files 11.2.0.3.0 Oracle Core Required Support Files 11.2.0.3.0 Bali Share 1.1.18.0.0 Oracle Database Deconfiguration 11.2.0.3.0 Oracle Quality of Service Management (Client) 11.2.0.3.0 Expat libraries 2.0.1.0.1 Oracle Containers for Java 11.2.0.3.0 Perl Modules 5.10.0.0.1 Secure Socket Layer 11.2.0.3.0 Oracle JDBC/OCI Instant Client 11.2.0.3.0 Oracle Multimedia Client Option 11.2.0.3.0 LDAP Required Support Files 11.2.0.3.0 Character Set Migration Utility 11.2.0.3.0 Perl Interpreter 5.10.0.0.2 PL/SQL Embedded Gateway 11.2.0.3.0 OLAP SQL Scripts 11.2.0.3.0 Database SQL Scripts 11.2.0.3.0 Oracle Extended Windowing Toolkit 3.4.47.0.0 SSL Required Support Files for InstantClient 11.2.0.3.0 SQL*Plus Files for Instant Client 11.2.0.3.0 Oracle Net Required Support Files 11.2.0.3.0 Oracle Database User Interface 2.2.13.0.0 RDBMS Required Support Files for Instant Client 11.2.0.3.0 RDBMS Required Support Files Runtime 11.2.0.3.0 XML Parser for Java 11.2.0.3.0 Oracle Security Developer Tools 11.2.0.3.0 Oracle Wallet Manager 11.2.0.3.0 Enterprise Manager plugin Common Files 11.2.0.3.0 Platform Required Support Files 11.2.0.3.0 Oracle JFC Extended Windowing Toolkit 4.2.36.0.0 RDBMS Required Support Files 11.2.0.3.0 Oracle Ice Browser 5.2.3.6.0 Oracle Help For Java 4.2.9.0.0 Enterprise Manager Common Files 10.2.0.4.3 Deinstallation Tool 11.2.0.3.0 Oracle Java Client 11.2.0.3.0 Cluster Verification Utility Files 11.2.0.3.0 Oracle Notification Service (eONS) 11.2.0.3.0 Oracle LDAP administration 11.2.0.3.0 Cluster Verification Utility Common Files 11.2.0.3.0 Oracle Clusterware RDBMS Files 11.2.0.3.0 Oracle Locale Builder 11.2.0.3.0 Oracle Globalization Support 11.2.0.3.0 Buildtools Common Files 11.2.0.3.0 Oracle RAC Required Support Files-HAS 11.2.0.3.0 SQL*Plus Required Support Files 11.2.0.3.0 XDK Required Support Files 11.2.0.3.0 Agent Required Support Files 10.2.0.4.3 Parser Generator Required Support Files 11.2.0.3.0 Precompiler Required Support Files 11.2.0.3.0 Installation Common Files 11.2.0.3.0 Required Support Files 11.2.0.3.0 Oracle JDBC/THIN Interfaces 11.2.0.3.0 Oracle Multimedia Locator 11.2.0.3.0 Oracle Multimedia 11.2.0.3.0 HAS Common Files 11.2.0.3.0 Assistant Common Files 11.2.0.3.0 PL/SQL 11.2.0.3.0 HAS Files for DB 11.2.0.3.0 Oracle Recovery Manager 11.2.0.3.0 Oracle Database Utilities 11.2.0.3.0 Oracle Notification Service 11.2.0.3.0 SQL*Plus 11.2.0.3.0 Oracle Netca Client 11.2.0.3.0 Oracle Net 11.2.0.3.0 Oracle JVM 11.2.0.3.0 Oracle Internet Directory Client 11.2.0.3.0 Oracle Net Listener 11.2.0.3.0 Cluster Ready Services Files 11.2.0.3.0 Oracle Database 11g 11.2.0.3.0 ----------------------------------------------------------------------------- Instantiating scripts for add node (Sunday, June 21, 2015 12:35:19 PM IST) . 1% Done. Instantiation of add node scripts complete Copying to remote nodes (Sunday, June 21, 2015 12:35:22 PM IST) ............................................................................................... 96% Done. Home copied to new nodes Saving inventory on nodes (Sunday, June 21, 2015 12:45:29 PM IST) . 100% Done. Save inventory complete WARNING:A new inventory has been created on one or more nodes in this session. However, it has not yet been registered as the central inventory of this system. To register the new inventory please run the script at '/u01/app/oraInventory/orainstRoot.sh' with root privileges on nodes 'rac2'. If you do not register the inventory, you may not be able to update or patch the products you installed. The following configuration scripts need to be executed as the "root" user in each new cluster node. Each script in the list below is followed by a list of nodes. /u01/app/oraInventory/orainstRoot.sh #On nodes rac2 /u01/app/11.2.0/grid/root.sh #On nodes rac2 To execute the configuration scripts: 1. Open a terminal window 2. Log in as "root" 3. Run the scripts in each cluster node The Cluster Node Addition of /u01/app/11.2.0/grid was successful. Please check '/tmp/silentInstall.log' for more details. [oracle@rac1 bin]$
Note: If you are using GNS:
cd $GI_HOME/oui/bin
./addNode.sh -silent “CLUSTER_NEW_NODES={rac2}”
ii) Run orainstRoot.sh #On nodes rac2
On node RAC2
As root
[root@rac2 ~]# /u01/app/oraInventory/orainstRoot.sh Creating the Oracle inventory pointer file (/etc/oraInst.loc) Changing permissions of /u01/app/oraInventory. Adding read,write permissions for group. Removing read,write,execute permissions for world. Changing groupname of /u01/app/oraInventory to oinstall. The execution of the script is complete. [root@rac2 ~]#
iii) Run root.sh #On nodes rac2
On node RAC2
As root
[root@rac2 ~]# /u01/app/11.2.0/grid/root.sh Performing root user operation for Oracle 11g The following environment variables are set as: ORACLE_OWNER= oracle ORACLE_HOME= /u01/app/11.2.0/grid Enter the full pathname of the local bin directory: [/usr/local/bin]: The contents of "dbhome" have not changed. No need to overwrite. The contents of "oraenv" have not changed. No need to overwrite. The contents of "coraenv" have not changed. No need to overwrite. Creating /etc/oratab file... Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root script. Now product-specific root actions will be performed. Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params Creating trace directory User ignored Prerequisites during installation OLR initialization - successful Adding Clusterware entries to inittab CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node rac1, number 1, and is terminating An active cluster was found during exclusive startup, restarting to join the cluster clscfg: EXISTING configuration version 5 detected. clscfg: version 5 is 11g Release 2. Successfully accumulated necessary OCR keys. Creating OCR keys for user 'root', privgrp 'root'.. Operation successful. Configure Oracle Grid Infrastructure for a Cluster ... succeeded [root@rac2 ~]#
Note: The root.sh script will configuring Grid Infrastructure on the new node and includes adding High Availability Services to the /etc/inittab so that CRS starts up when the machine starts.
iv) Check Clusterware Resources after ran root.sh
[root@rac2 ~]# crsctl stat res -t -------------------------------------------------------------------------------- NAME TARGET STATE SERVER STATE_DETAILS -------------------------------------------------------------------------------- Local Resources -------------------------------------------------------------------------------- ora.DATA.dg ONLINE ONLINE rac1 ONLINE ONLINE rac2 ora.DATA1.dg ONLINE ONLINE rac1 ONLINE ONLINE rac2 ora.LISTENER.lsnr ONLINE ONLINE rac1 ONLINE ONLINE rac2 ora.asm ONLINE ONLINE rac1 Started ONLINE ONLINE rac2 Started ora.gsd OFFLINE OFFLINE rac1 OFFLINE OFFLINE rac2 ora.net1.network ONLINE ONLINE rac1 ONLINE ONLINE rac2 ora.ons ONLINE ONLINE rac1 ONLINE ONLINE rac2 -------------------------------------------------------------------------------- Cluster Resources -------------------------------------------------------------------------------- ora.LISTENER_SCAN1.lsnr 1 ONLINE ONLINE rac2 ora.LISTENER_SCAN2.lsnr 1 ONLINE ONLINE rac1 ora.LISTENER_SCAN3.lsnr 1 ONLINE ONLINE rac1 ora.cvu 1 ONLINE ONLINE rac1 ora.nike.db 1 ONLINE ONLINE rac1 Open ora.nike.nike_srv.svc 1 ONLINE ONLINE rac1 ora.oc4j 1 ONLINE ONLINE rac1 ora.rac1.vip 1 ONLINE ONLINE rac1 ora.rac2.vip 1 ONLINE ONLINE rac2 ora.scan1.vip 1 ONLINE ONLINE rac2 ora.scan2.vip 1 ONLINE ONLINE rac1 ora.scan3.vip 1 ONLINE ONLINE rac1 [root@rac2 ~]# [root@rac2 ~]# crsctl check crs CRS-4638: Oracle High Availability Services is online CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online [root@rac2 ~]#
v) Run cluvfy post-node add script
As GI Home owner
From node RAC1 as best practice, because initially we ran pre-node cluvfy from RAC 1 only
[oracle@rac1 ~]$ cluvfy stage -post nodeadd -n rac2 -verbose Performing post-checks for node addition Checking node reachability... Check: Node reachability from node "rac1" Destination Node Reachable? ------------------------------------ ------------------------ rac2 yes Result: Node reachability check passed from node "rac1" Checking user equivalence... Check: User equivalence for user "oracle" Node Name Status ------------------------------------ ------------------------ rac2 passed Result: User equivalence check passed for user "oracle" Checking node connectivity... Checking hosts config file... Node Name Status ------------------------------------ ------------------------ rac2 passed rac1 passed Verification of the hosts config file successful Interface information for node "rac2" Name IP Address Subnet Gateway Def. Gateway HW Address MTU ------ --------------- --------------- --------------- --------------- ----------------- ------ eth0 192.168.2.102 192.168.2.0 0.0.0.0 10.0.4.2 08:00:27:93:7F:00 1500 eth0 192.168.2.107 192.168.2.0 0.0.0.0 10.0.4.2 08:00:27:93:7F:00 1500 eth0 192.168.2.104 192.168.2.0 0.0.0.0 10.0.4.2 08:00:27:93:7F:00 1500 eth1 192.168.0.102 192.168.0.0 0.0.0.0 10.0.4.2 08:00:27:63:80:76 1500 eth1 169.254.215.111 169.254.0.0 0.0.0.0 10.0.4.2 08:00:27:63:80:76 1500 eth2 10.0.4.15 10.0.4.0 0.0.0.0 10.0.4.2 08:00:27:4C:B3:01 1500 Interface information for node "rac1" Name IP Address Subnet Gateway Def. Gateway HW Address MTU ------ --------------- --------------- --------------- --------------- ----------------- ------ eth0 192.168.2.101 192.168.2.0 0.0.0.0 10.0.4.2 08:00:27:02:46:97 1500 eth0 192.168.2.106 192.168.2.0 0.0.0.0 10.0.4.2 08:00:27:02:46:97 1500 eth0 192.168.2.103 192.168.2.0 0.0.0.0 10.0.4.2 08:00:27:02:46:97 1500 eth0 192.168.2.105 192.168.2.0 0.0.0.0 10.0.4.2 08:00:27:02:46:97 1500 eth1 192.168.0.101 192.168.0.0 0.0.0.0 10.0.4.2 08:00:27:9A:66:6A 1500 eth1 169.254.6.127 169.254.0.0 0.0.0.0 10.0.4.2 08:00:27:9A:66:6A 1500 eth2 10.0.4.15 10.0.4.0 0.0.0.0 10.0.4.2 08:00:27:D3:B8:9F 1500 Check: Node connectivity for interface "eth0" Source Destination Connected? ------------------------------ ------------------------------ ---------------- rac2[192.168.2.102] rac2[192.168.2.107] yes rac2[192.168.2.102] rac2[192.168.2.104] yes rac2[192.168.2.102] rac1[192.168.2.101] yes rac2[192.168.2.102] rac1[192.168.2.106] yes rac2[192.168.2.102] rac1[192.168.2.103] yes rac2[192.168.2.102] rac1[192.168.2.105] yes rac2[192.168.2.107] rac2[192.168.2.104] yes rac2[192.168.2.107] rac1[192.168.2.101] yes rac2[192.168.2.107] rac1[192.168.2.106] yes rac2[192.168.2.107] rac1[192.168.2.103] yes rac2[192.168.2.107] rac1[192.168.2.105] yes rac2[192.168.2.104] rac1[192.168.2.101] yes rac2[192.168.2.104] rac1[192.168.2.106] yes rac2[192.168.2.104] rac1[192.168.2.103] yes rac2[192.168.2.104] rac1[192.168.2.105] yes rac1[192.168.2.101] rac1[192.168.2.106] yes rac1[192.168.2.101] rac1[192.168.2.103] yes rac1[192.168.2.101] rac1[192.168.2.105] yes rac1[192.168.2.106] rac1[192.168.2.103] yes rac1[192.168.2.106] rac1[192.168.2.105] yes rac1[192.168.2.103] rac1[192.168.2.105] yes Result: Node connectivity passed for interface "eth0" Check: TCP connectivity of subnet "192.168.2.0" Source Destination Connected? ------------------------------ ------------------------------ ---------------- rac1:192.168.2.101 rac2:192.168.2.102 passed rac1:192.168.2.101 rac2:192.168.2.107 passed rac1:192.168.2.101 rac2:192.168.2.104 passed rac1:192.168.2.101 rac1:192.168.2.106 passed rac1:192.168.2.101 rac1:192.168.2.103 passed rac1:192.168.2.101 rac1:192.168.2.105 passed Result: TCP connectivity check passed for subnet "192.168.2.0" Checking subnet mask consistency... Subnet mask consistency check passed for subnet "192.168.2.0". Subnet mask consistency check passed. Result: Node connectivity check passed Checking multicast communication... Checking subnet "192.168.2.0" for multicast communication with multicast group "230.0.1.0"... Check of subnet "192.168.2.0" for multicast communication with multicast group "230.0.1.0" passed. Check of multicast communication passed. Checking cluster integrity... Node Name ------------------------------------ rac1 rac2 Cluster integrity check passed Checking CRS integrity... Clusterware version consistency passed The Oracle Clusterware is healthy on node "rac2" The Oracle Clusterware is healthy on node "rac1" CRS integrity check passed Checking shared resources... Checking CRS home location... "/u01/app/11.2.0/grid" is not shared Result: Shared resources check for node addition passed Checking node connectivity... Checking hosts config file... Node Name Status ------------------------------------ ------------------------ rac2 passed rac1 passed Verification of the hosts config file successful Interface information for node "rac2" Name IP Address Subnet Gateway Def. Gateway HW Address MTU ------ --------------- --------------- --------------- --------------- ----------------- ------ eth0 192.168.2.102 192.168.2.0 0.0.0.0 10.0.4.2 08:00:27:93:7F:00 1500 eth0 192.168.2.107 192.168.2.0 0.0.0.0 10.0.4.2 08:00:27:93:7F:00 1500 eth0 192.168.2.104 192.168.2.0 0.0.0.0 10.0.4.2 08:00:27:93:7F:00 1500 eth1 192.168.0.102 192.168.0.0 0.0.0.0 10.0.4.2 08:00:27:63:80:76 1500 eth1 169.254.215.111 169.254.0.0 0.0.0.0 10.0.4.2 08:00:27:63:80:76 1500 eth2 10.0.4.15 10.0.4.0 0.0.0.0 10.0.4.2 08:00:27:4C:B3:01 1500 Interface information for node "rac1" Name IP Address Subnet Gateway Def. Gateway HW Address MTU ------ --------------- --------------- --------------- --------------- ----------------- ------ eth0 192.168.2.101 192.168.2.0 0.0.0.0 10.0.4.2 08:00:27:02:46:97 1500 eth0 192.168.2.106 192.168.2.0 0.0.0.0 10.0.4.2 08:00:27:02:46:97 1500 eth0 192.168.2.103 192.168.2.0 0.0.0.0 10.0.4.2 08:00:27:02:46:97 1500 eth0 192.168.2.105 192.168.2.0 0.0.0.0 10.0.4.2 08:00:27:02:46:97 1500 eth1 192.168.0.101 192.168.0.0 0.0.0.0 10.0.4.2 08:00:27:9A:66:6A 1500 eth1 169.254.6.127 169.254.0.0 0.0.0.0 10.0.4.2 08:00:27:9A:66:6A 1500 eth2 10.0.4.15 10.0.4.0 0.0.0.0 10.0.4.2 08:00:27:D3:B8:9F 1500 Check: Node connectivity for interface "eth0" Source Destination Connected? ------------------------------ ------------------------------ ---------------- rac2[192.168.2.102] rac2[192.168.2.107] yes rac2[192.168.2.102] rac2[192.168.2.104] yes rac2[192.168.2.102] rac1[192.168.2.101] yes rac2[192.168.2.102] rac1[192.168.2.106] yes rac2[192.168.2.102] rac1[192.168.2.103] yes rac2[192.168.2.102] rac1[192.168.2.105] yes rac2[192.168.2.107] rac2[192.168.2.104] yes rac2[192.168.2.107] rac1[192.168.2.101] yes rac2[192.168.2.107] rac1[192.168.2.106] yes rac2[192.168.2.107] rac1[192.168.2.103] yes rac2[192.168.2.107] rac1[192.168.2.105] yes rac2[192.168.2.104] rac1[192.168.2.101] yes rac2[192.168.2.104] rac1[192.168.2.106] yes rac2[192.168.2.104] rac1[192.168.2.103] yes rac2[192.168.2.104] rac1[192.168.2.105] yes rac1[192.168.2.101] rac1[192.168.2.106] yes rac1[192.168.2.101] rac1[192.168.2.103] yes rac1[192.168.2.101] rac1[192.168.2.105] yes rac1[192.168.2.106] rac1[192.168.2.103] yes rac1[192.168.2.106] rac1[192.168.2.105] yes rac1[192.168.2.103] rac1[192.168.2.105] yes Result: Node connectivity passed for interface "eth0" Check: TCP connectivity of subnet "192.168.2.0" Source Destination Connected? ------------------------------ ------------------------------ ---------------- rac1:192.168.2.101 rac2:192.168.2.102 passed rac1:192.168.2.101 rac2:192.168.2.107 passed rac1:192.168.2.101 rac2:192.168.2.104 passed rac1:192.168.2.101 rac1:192.168.2.106 passed rac1:192.168.2.101 rac1:192.168.2.103 passed rac1:192.168.2.101 rac1:192.168.2.105 passed Result: TCP connectivity check passed for subnet "192.168.2.0" Check: Node connectivity for interface "eth1" Source Destination Connected? ------------------------------ ------------------------------ ---------------- rac2[192.168.0.102] rac1[192.168.0.101] yes Result: Node connectivity passed for interface "eth1" Check: TCP connectivity of subnet "192.168.0.0" Source Destination Connected? ------------------------------ ------------------------------ ---------------- rac1:192.168.0.101 rac2:192.168.0.102 passed Result: TCP connectivity check passed for subnet "192.168.0.0" Checking subnet mask consistency... Subnet mask consistency check passed for subnet "192.168.2.0". Subnet mask consistency check passed for subnet "192.168.0.0". Subnet mask consistency check passed. Result: Node connectivity check passed Checking multicast communication... Checking subnet "192.168.2.0" for multicast communication with multicast group "230.0.1.0"... Check of subnet "192.168.2.0" for multicast communication with multicast group "230.0.1.0" passed. Checking subnet "192.168.0.0" for multicast communication with multicast group "230.0.1.0"... Check of subnet "192.168.0.0" for multicast communication with multicast group "230.0.1.0" passed. Check of multicast communication passed. Checking node application existence... Checking existence of VIP node application (required) Node Name Required Running? Comment ------------ ------------------------ ------------------------ ---------- rac2 yes yes passed rac1 yes yes passed VIP node application check passed Checking existence of NETWORK node application (required) Node Name Required Running? Comment ------------ ------------------------ ------------------------ ---------- rac2 yes yes passed rac1 yes yes passed NETWORK node application check passed Checking existence of GSD node application (optional) Node Name Required Running? Comment ------------ ------------------------ ------------------------ ---------- rac2 no no exists rac1 no no exists GSD node application is offline on nodes "rac2,rac1" Checking existence of ONS node application (optional) Node Name Required Running? Comment ------------ ------------------------ ------------------------ ---------- rac2 no yes passed rac1 no yes passed ONS node application check passed Checking Single Client Access Name (SCAN)... SCAN Name Node Running? ListenerName Port Running? ---------------- ------------ ------------ ------------ ------------ ------------ rac-scan.rajasekhar.com rac2 true LISTENER_SCAN1 1521 true rac-scan.rajasekhar.com rac1 true LISTENER_SCAN2 1521 true rac-scan.rajasekhar.com rac1 true LISTENER_SCAN3 1521 true Checking TCP connectivity to SCAN Listeners... Node ListenerName TCP connectivity? ------------ ------------------------ ------------------------ rac1 LISTENER_SCAN1 yes rac1 LISTENER_SCAN2 yes rac1 LISTENER_SCAN3 yes TCP connectivity to SCAN Listeners exists on all cluster nodes Checking name resolution setup for "rac-scan.rajasekhar.com"... SCAN Name IP Address Status Comment ------------ ------------------------ ------------------------ ---------- rac-scan.rajasekhar.com 192.168.2.107 passed rac-scan.rajasekhar.com 192.168.2.105 passed rac-scan.rajasekhar.com 192.168.2.106 passed Verification of SCAN VIP and Listener setup passed Checking to make sure user "oracle" is not in "root" group Node Name Status Comment ------------ ------------------------ ------------------------ rac2 passed does not exist Result: User "oracle" is not part of "root" group. Check passed Checking if Clusterware is installed on all nodes... Check of Clusterware install passed Checking if CTSS Resource is running on all nodes... Check: CTSS Resource running on all nodes Node Name Status ------------------------------------ ------------------------ rac2 passed Result: CTSS resource check passed Querying CTSS for time offset on all nodes... Result: Query of CTSS for time offset passed Check CTSS state started... Check: CTSS state Node Name State ------------------------------------ ------------------------ rac2 Active CTSS is in Active state. Proceeding with check of clock time offsets on all nodes... Reference Time Offset Limit: 1000.0 msecs Check: Reference Time Offset Node Name Time Offset Status ------------ ------------------------ ------------------------ rac2 0.0 passed Time offset is within the specified limits on the following set of nodes: "[rac2]" Result: Check of clock time offsets passed Oracle Cluster Time Synchronization Services check passed Post-check for node addition was successful. <<< [oracle@rac1 ~]$
[oracle@rac2 ~]$ olsnodes -n
rac1 1
rac2 2
[oracle@rac2 ~]$
On node RAC2
[oracle@rac2 ~]$ ps -ef | grep tns
root 13 2 0 Jun20 ? 00:00:00 [netns]
oracle 24168 1 0 12:57 ? 00:00:00 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER_SCAN1 -inherit
oracle 24639 1 0 12:57 ? 00:00:00 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER -inherit
oracle 28355 28292 0 13:19 pts/1 00:00:00 grep tns
[oracle@rac2 ~]$
On node RAC2
[oracle@rac2 ~]$ srvctl status asm -a
ASM is running on rac2,rac1
ASM is enabled.
[oracle@rac2 ~]$
On node RAC2
[oracle@rac2 ~]$ ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 4076
Available space (kbytes) : 258044
ID : 1037097601
Device/File Name : +DATA
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
[oracle@rac2 ~]$
On node RAC2
[oracle@rac2 ~]$ crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE a785661b81264f8ebfa8538128d4e1fe (/dev/oracleasm/disks/DISK1) [DATA]
2. ONLINE 7a41922f13254f61bf4cee3f53b9aa74 (/dev/oracleasm/disks/DISK2) [DATA]
3. ONLINE 492244b5021f4fc7bf7d75b74cfe841a (/dev/oracleasm/disks/DISK3) [DATA]
Located 3 voting disk(s).
[oracle@rac2 ~]$
3. Cluster Node Addition for RDBMS Home.
From Node 1 RAC1
As RDBMS HOME owner
[oracle@rac1 ~]$ cd /u01/app/oracle/product/11.2.0/db_1/oui/bin/ <<< This is RDBMS home [oracle@rac1 bin]$ ./addNode.sh -silent "CLUSTER_NEW_NODES={rac2}" Starting Oracle Universal Installer... Checking swap space: must be greater than 500 MB. Actual 5670 MB Passed Oracle Universal Installer, Version 11.2.0.3.0 Production Copyright (C) 1999, 2011, Oracle. All rights reserved. Performing tests to see whether nodes rac2 are available ............................................................... 100% Done. ........ ----------------------------------------------------------------------------- Cluster Node Addition Summary Global Settings Source: /u01/app/oracle/product/11.2.0/db_1 New Nodes Space Requirements New Nodes rac2 /u01: Required 5.04GB : Available 10.57GB Installed Products Product Names Oracle Database 11g 11.2.0.3.0 Sun JDK 1.5.0.30.03 Installer SDK Component 11.2.0.3.0 Oracle One-Off Patch Installer 11.2.0.1.7 Oracle Universal Installer 11.2.0.3.0 Oracle USM Deconfiguration 11.2.0.3.0 Oracle Configuration Manager Deconfiguration 10.3.1.0.0 Oracle DBCA Deconfiguration 11.2.0.3.0 Oracle RAC Deconfiguration 11.2.0.3.0 Oracle Database Deconfiguration 11.2.0.3.0 Oracle Configuration Manager Client 10.3.2.1.0 Oracle Configuration Manager 10.3.5.0.1 Oracle ODBC Driverfor Instant Client 11.2.0.3.0 LDAP Required Support Files 11.2.0.3.0 SSL Required Support Files for InstantClient 11.2.0.3.0 Bali Share 1.1.18.0.0 Oracle Extended Windowing Toolkit 3.4.47.0.0 Oracle JFC Extended Windowing Toolkit 4.2.36.0.0 Oracle Real Application Testing 11.2.0.3.0 Oracle Database Vault J2EE Application 11.2.0.3.0 Oracle Label Security 11.2.0.3.0 Oracle Data Mining RDBMS Files 11.2.0.3.0 Oracle OLAP RDBMS Files 11.2.0.3.0 Oracle OLAP API 11.2.0.3.0 Platform Required Support Files 11.2.0.3.0 Oracle Database Vault option 11.2.0.3.0 Oracle RAC Required Support Files-HAS 11.2.0.3.0 SQL*Plus Required Support Files 11.2.0.3.0 Oracle Display Fonts 9.0.2.0.0 Oracle Ice Browser 5.2.3.6.0 Oracle JDBC Server Support Package 11.2.0.3.0 Oracle SQL Developer 11.2.0.3.0 Oracle Application Express 11.2.0.3.0 XDK Required Support Files 11.2.0.3.0 RDBMS Required Support Files for Instant Client 11.2.0.3.0 SQLJ Runtime 11.2.0.3.0 Database Workspace Manager 11.2.0.3.0 RDBMS Required Support Files Runtime 11.2.0.3.0 Oracle Globalization Support 11.2.0.3.0 Exadata Storage Server 11.2.0.1.0 Provisioning Advisor Framework 10.2.0.4.3 Enterprise Manager Database Plugin -- Repository Support 11.2.0.3.0 Enterprise Manager Repository Core Files 10.2.0.4.4 Enterprise Manager Database Plugin -- Agent Support 11.2.0.3.0 Enterprise Manager Grid Control Core Files 10.2.0.4.4 Enterprise Manager Common Core Files 10.2.0.4.4 Enterprise Manager Agent Core Files 10.2.0.4.4 RDBMS Required Support Files 11.2.0.3.0 regexp 2.1.9.0.0 Agent Required Support Files 10.2.0.4.3 Oracle 11g Warehouse Builder Required Files 11.2.0.3.0 Oracle Notification Service (eONS) 11.2.0.3.0 Oracle Text Required Support Files 11.2.0.3.0 Parser Generator Required Support Files 11.2.0.3.0 Oracle Database 11g Multimedia Files 11.2.0.3.0 Oracle Multimedia Java Advanced Imaging 11.2.0.3.0 Oracle Multimedia Annotator 11.2.0.3.0 Oracle JDBC/OCI Instant Client 11.2.0.3.0 Oracle Multimedia Locator RDBMS Files 11.2.0.3.0 Precompiler Required Support Files 11.2.0.3.0 Oracle Core Required Support Files 11.2.0.3.0 Sample Schema Data 11.2.0.3.0 Oracle Starter Database 11.2.0.3.0 Oracle Message Gateway Common Files 11.2.0.3.0 Oracle XML Query 11.2.0.3.0 XML Parser for Oracle JVM 11.2.0.3.0 Oracle Help For Java 4.2.9.0.0 Installation Plugin Files 11.2.0.3.0 Enterprise Manager Common Files 10.2.0.4.3 Expat libraries 2.0.1.0.1 Deinstallation Tool 11.2.0.3.0 Oracle Quality of Service Management (Client) 11.2.0.3.0 Perl Modules 5.10.0.0.1 JAccelerator (COMPANION) 11.2.0.3.0 Oracle Containers for Java 11.2.0.3.0 Perl Interpreter 5.10.0.0.2 Oracle Net Required Support Files 11.2.0.3.0 Secure Socket Layer 11.2.0.3.0 Oracle Universal Connection Pool 11.2.0.3.0 Oracle JDBC/THIN Interfaces 11.2.0.3.0 Oracle Multimedia Client Option 11.2.0.3.0 Oracle Java Client 11.2.0.3.0 Character Set Migration Utility 11.2.0.3.0 Oracle Code Editor 1.2.1.0.0I PL/SQL Embedded Gateway 11.2.0.3.0 OLAP SQL Scripts 11.2.0.3.0 Database SQL Scripts 11.2.0.3.0 Oracle Locale Builder 11.2.0.3.0 Oracle Globalization Support 11.2.0.3.0 SQL*Plus Files for Instant Client 11.2.0.3.0 Required Support Files 11.2.0.3.0 Oracle Database User Interface 2.2.13.0.0 Oracle ODBC Driver 11.2.0.3.0 Oracle Notification Service 11.2.0.3.0 XML Parser for Java 11.2.0.3.0 Oracle Security Developer Tools 11.2.0.3.0 Oracle Wallet Manager 11.2.0.3.0 Cluster Verification Utility Common Files 11.2.0.3.0 Oracle Clusterware RDBMS Files 11.2.0.3.0 Oracle UIX 2.2.24.6.0 Enterprise Manager plugin Common Files 11.2.0.3.0 HAS Common Files 11.2.0.3.0 Precompiler Common Files 11.2.0.3.0 Installation Common Files 11.2.0.3.0 Oracle Help for the Web 2.0.14.0.0 Oracle LDAP administration 11.2.0.3.0 Buildtools Common Files 11.2.0.3.0 Assistant Common Files 11.2.0.3.0 Oracle Recovery Manager 11.2.0.3.0 PL/SQL 11.2.0.3.0 Generic Connectivity Common Files 11.2.0.3.0 Oracle Database Gateway for ODBC 11.2.0.3.0 Oracle Programmer 11.2.0.3.0 Oracle Database Utilities 11.2.0.3.0 Enterprise Manager Agent 10.2.0.4.3 SQL*Plus 11.2.0.3.0 Oracle Netca Client 11.2.0.3.0 Oracle Multimedia Locator 11.2.0.3.0 Oracle Call Interface (OCI) 11.2.0.3.0 Oracle Multimedia 11.2.0.3.0 Oracle Net 11.2.0.3.0 Oracle XML Development Kit 11.2.0.3.0 Database Configuration and Upgrade Assistants 11.2.0.3.0 Oracle JVM 11.2.0.3.0 Oracle Advanced Security 11.2.0.3.0 Oracle Internet Directory Client 11.2.0.3.0 Oracle Enterprise Manager Console DB 11.2.0.3.0 HAS Files for DB 11.2.0.3.0 Oracle Net Listener 11.2.0.3.0 Oracle Text 11.2.0.3.0 Oracle Net Services 11.2.0.3.0 Oracle Database 11g 11.2.0.3.0 Oracle OLAP 11.2.0.3.0 Oracle Spatial 11.2.0.3.0 Oracle Partitioning 11.2.0.3.0 Enterprise Edition Options 11.2.0.3.0 ----------------------------------------------------------------------------- Instantiating scripts for add node (Sunday, June 21, 2015 2:06:51 PM IST) . 1% Done. Instantiation of add node scripts complete Copying to remote nodes (Sunday, June 21, 2015 2:08:47 PM IST) ............................................................................................... 96% Done. Home copied to new nodes Saving inventory on nodes (Sunday, June 21, 2015 8:10:31 PM IST) . 100% Done. Save inventory complete WARNING: The following configuration scripts need to be executed as the "root" user in each new cluster node. Each script in the list below is followed by a list of nodes. /u01/app/oracle/product/11.2.0/db_1/root.sh #On nodes rac2 To execute the configuration scripts: 1. Open a terminal window 2. Log in as "root" 3. Run the scripts in each cluster node The Cluster Node Addition of /u01/app/oracle/product/11.2.0/db_1 was successful. Please check '/tmp/silentInstall.log' for more details. [oracle@rac1 bin]$
ii) root.sh #On nodes rac2 from RDBMS home
[root@rac2 ~]# /u01/app/oracle/product/11.2.0/db_1/root.sh Performing root user operation for Oracle 11g The following environment variables are set as: ORACLE_OWNER= oracle ORACLE_HOME= /u01/app/oracle/product/11.2.0/db_1 Enter the full pathname of the local bin directory: [/usr/local/bin]: The contents of "dbhome" have not changed. No need to overwrite. The contents of "oraenv" have not changed. No need to overwrite. The contents of "coraenv" have not changed. No need to overwrite. Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root script. Now product-specific root actions will be performed. Finished product-specific root actions. [root@rac2 ~]#
4. Add Instance to Database through Command-Line or you can add via dbca.
On RAC2
As RDBMS Home owner
[oracle@rac2 ~]$ cd /u01/app/oracle/product/11.2.0/db_1/dbs [oracle@rac2 dbs]$ ls -ltr total 28 -rw-r----- 1 oracle oinstall 1536 Jun 21 19:56 orapwnike1 << -rw-r----- 1 oracle oinstall 161 Jun 21 19:56 initDBUA5216639.ora -rw-r----- 1 oracle oinstall 36 Jun 21 19:56 initnike1.ora << -rw-rw---- 1 oracle oinstall 1544 Jun 21 19:56 hc_DBUA5216639.dat -rw-rw---- 1 oracle oinstall 1544 Jun 21 19:56 hc_nike1.dat -rw-r--r-- 1 oracle oinstall 2851 Jun 21 19:56 init.ora -rw-r----- 1 oracle oinstall 1536 Jun 21 19:56 orapwDBUA5216639 [oracle@rac2 dbs]$ [oracle@rac2 dbs]$ mv initnike1.ora initnike2.ora [oracle@rac2 dbs]$ mv orapwnike1 orapwnike2 [oracle@rac2 dbs]$ ls -ltr total 28 -rw-r----- 1 oracle oinstall 1536 Jun 21 19:56 orapwnike2 -rw-r----- 1 oracle oinstall 161 Jun 21 19:56 initDBUA5216639.ora -rw-r----- 1 oracle oinstall 36 Jun 21 19:56 initnike2.ora -rw-rw---- 1 oracle oinstall 1544 Jun 21 19:56 hc_DBUA5216639.dat -rw-rw---- 1 oracle oinstall 1544 Jun 21 19:56 hc_nike1.dat -rw-r--r-- 1 oracle oinstall 2851 Jun 21 19:56 init.ora -rw-r----- 1 oracle oinstall 1536 Jun 21 19:56 orapwDBUA5216639 [oracle@rac2 dbs]$ [oracle@rac2 dbs]$ cat initnike2.ora SPFILE='+DATA1/nike/spfilenike.ora' [oracle@rac2 dbs]$ [oracle@rac2 dbs]$ echo "nike2:/u01/app/oracle/product/11.2.0/db_1:N" >> /etc/oratab [oracle@rac2 dbs]$ echo "nike:/u01/app/oracle/product/11.2.0/db_1:N" >> /etc/oratab cat /etc/oratab .. # +ASM2:/u01/app/11.2.0/grid:N # line added by Agent nike2:/u01/app/oracle/product/11.2.0/db_1:N nike:/u01/app/oracle/product/11.2.0/db_1:N [oracle@rac2 ~]$ mkdir -p /u01/app/oracle/admin/nike/adump [oracle@rac2 ~]$ mkdir -p /u01/app/oracle/admin/nike/dpdump [oracle@rac2 ~]$ mkdir -p /u01/app/oracle/admin/nike/hdump [oracle@rac2 ~]$ mkdir -p /u01/app/oracle/admin/nike/pfile
On RAC1, As ORACLE HOME owner
SQL> set lines 180 SQL> col MEMBER for a60 SQL> select b.thread#, a.group#, a.member, b.bytes FROM v$logfile a, v$log b WHERE a.group# = b.group#; THREAD# GROUP# MEMBER BYTES ---------- ---------- ------------------------------------------------------------ ---------- 1 2 +DATA1/nike/onlinelog/group_2.290.847761035 52428800 1 1 +DATA1/nike/onlinelog/group_1.289.847761031 52428800 SQL> SQL> alter database add logfile thread 2 group 3 ('+DATA1') size 52428800, group 4 ('+DATA1') size 52428800; Database altered. SQL> SQL> select b.thread#, a.group#, a.member, b.bytes FROM v$logfile a, v$log b WHERE a.group# = b.group#; THREAD# GROUP# MEMBER BYTES ---------- ---------- ------------------------------------------------------------ ---------- 1 2 +DATA1/nike/onlinelog/group_2.290.847761035 52428800 1 1 +DATA1/nike/onlinelog/group_1.289.847761031 52428800 2 3 +DATA1/nike/onlinelog/group_3.271.883012175 52428800 2 4 +DATA1/nike/onlinelog/group_4.270.883012181 52428800 SQL> SQL> alter database enable public thread 2; Database altered. SQL>
On node RAC1
SQL> set pages 0 SQL> set long 9999999 SQL> select dbms_metadata.get_ddl('TABLESPACE','UNDOTBS1') from dual; CREATE UNDO TABLESPACE "UNDOTBS1" DATAFILE SIZE 26214400 AUTOEXTEND ON NEXT 5242880 MAXSIZE 32767M BLOCKSIZE 8192 EXTENT MANAGEMENT LOCAL AUTOALLOCATE ALTER DATABASE DATAFILE '+DATA1/nike/datafile/undotbs1.357.847760839' RESIZE 41943040 SQL> create undo tablespace undotbs2 datafile '+DATA1' size 25M autoextend on next 5m maxsize 40M; Tablespace created. SQL> SQL> alter system set undo_tablespace=undotbs2 scope=spfile sid='nike2'; System altered. SQL> alter system set instance_number=2 scope=spfile sid='nike2'; System altered. SQL> alter system set thread=2 scope=spfile sid='nike2'; System altered. SQL> SQL> alter system set cluster_database_instances=2 scope=spfile sid='*'; System altered. SQL> SQL> select inst_id,name,value from gv$parameter where name like 'undo_table%'; INST_ID NAME VALUE ------- -------------------- --------------- 2 undo_tablespace UNDOTBS2 1 undo_tablespace UNDOTBS1 SQL>
From node RAC1 as ORACLE_HOME owner
[oracle@rac1 ~]$ which srvctl /u01/app/oracle/product/11.2.0/db_1/bin/srvctl <<< You should run from RDBMS Home [oracle@rac1 ~]$ [oracle@rac1 ~]$ srvctl add instance -d nike -i nike2 -n rac2 [oracle@rac1 ~]$ [oracle@rac1 ~]$ srvctl status database -d nike -v Instance nike1 is running on node rac1 with online services nike_srv. Instance status: Open. Instance nike2 is not running on node rac2 <<< [oracle@rac1 ~]$ [oracle@rac1 ~]$ srvctl start instance -d nike -i nike2 [oracle@rac1 ~]$ [oracle@rac1 ~]$ srvctl status database -d nike -v Instance nike1 is running on node rac1 with online services nike_srv. Instance status: Open. Instance nike2 is running on node rac2. Instance status: Open. <<< [oracle@rac1 ~]$ [oracle@rac1 ~]$ srvctl config database -d nike Database unique name: nike Database name: nike Oracle home: /u01/app/oracle/product/11.2.0/db_1 <<< Oracle user: oracle Spfile: +DATA1/nike/spfilenike.ora Domain: Start options: open Stop options: immediate Database role: PRIMARY Management policy: AUTOMATIC Server pools: nike Database instances: nike1,nike2 <<<< Disk Groups: DATA1 Mount point paths: Services: nike_srv << Type: RAC Database is administrator managed <<< [oracle@rac1 ~]$ SQL> col host_name format a22 SQL> set lines 180 SQL> select host_name, inst_id, instance_name, status, to_char(startup_time, 'DD-MON-YYYY HH24:MI:SS') as "START_TIME" from gv$instance order by inst_id; HOST_NAME INST_ID INSTANCE_NAME STATUS START_TIME ---------------------- ---------- ---------------- ------------ -------------------- rac1.rajasekhar.com 1 nike1 OPEN 21-JUN-2015 11:38:48 rac2.rajasekhar.com 2 nike2 OPEN 22-JUN-2015 01:25:13 SQL>
v) Add New Instance to service via srvctl or you can add via dbca
From node RAC1, as ORACLE_HOME owner
[oracle@rac1 ~]$ which srvctl /u01/app/oracle/product/11.2.0/db_1/bin/srvctl <<< You should run from RDBMS Home [oracle@rac1 ~]$ [oracle@rac1 ~]$ srvctl add service -d nike -s nike_srv -a nike2 -u <<<< -a means available instance. [oracle@rac1 ~]$ [oracle@rac1 ~]$ srvctl config service -d nike -s nike_srv Service name: nike_srv <<< Service is enabled Server pool: nike_nike_srv Cardinality: 1 Disconnect: false Service role: PRIMARY Management policy: AUTOMATIC DTP transaction: false AQ HA notifications: false Failover type: NONE Failover method: NONE TAF failover retries: 0 TAF failover delay: 0 Connection Load Balancing Goal: LONG Runtime Load Balancing Goal: NONE TAF policy specification: NONE Edition: Preferred instances: nike1 Available instances: nike2 <<<< [oracle@rac1 ~]$
Note: If you want add instance as preferred then follow below
[oracle@rac1 ~]$ which srvctl /u01/app/oracle/product/11.2.0/db_1/bin/srvctl <<< You should run from RDBMS Home [oracle@rac1 ~]$ [oracle@rac1 ~]$ srvctl add service -d nike -s nike_srv -r nike2 -u [oracle@rac1 ~]$ srvctl config service -d nike -s nike_srv Service name: nike_srv Service is enabled Server pool: nike_nike_srv Cardinality: 2 Disconnect: false Service role: PRIMARY Management policy: AUTOMATIC DTP transaction: false AQ HA notifications: false Failover type: NONE Failover method: NONE TAF failover retries: 0 TAF failover delay: 0 Connection Load Balancing Goal: LONG Runtime Load Balancing Goal: NONE TAF policy specification: NONE Edition: Preferred instances: nike2,nike1 <<< Available instances: [oracle@rac1 ~]$ srvctl status service -d nike Service nike_srv is running on instance(s) nike1 <<<< [oracle@rac1 ~]$ srvctl start service -d nike [oracle@rac1 ~]$ srvctl status service -d nike -v Service nike_srv is running on instance(s) nike1,nike2 <<<< [oracle@rac1 ~]$
Note: Modify tnsnames.ora file if required.
[oracle@rac1 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
ONLINE ONLINE rac1
ONLINE ONLINE rac2
ora.DATA1.dg
ONLINE ONLINE rac1
ONLINE ONLINE rac2
ora.LISTENER.lsnr
ONLINE ONLINE rac1
ONLINE ONLINE rac2
ora.asm
ONLINE ONLINE rac1 Started
ONLINE ONLINE rac2 Started
ora.gsd
OFFLINE OFFLINE rac1
OFFLINE OFFLINE rac2
ora.net1.network
ONLINE ONLINE rac1
ONLINE ONLINE rac2
ora.ons
ONLINE ONLINE rac1
ONLINE ONLINE rac2
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE rac2
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE rac1
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE rac1
ora.cvu
1 ONLINE ONLINE rac1
ora.nike.db
1 ONLINE ONLINE rac1 Open
2 ONLINE ONLINE rac2 Open
ora.nike.nike_srv.svc
1 ONLINE ONLINE rac1
ora.oc4j
1 ONLINE ONLINE rac1
ora.rac1.vip
1 ONLINE ONLINE rac1
ora.rac2.vip
1 ONLINE ONLINE rac2
ora.scan1.vip
1 ONLINE ONLINE rac2
ora.scan2.vip
1 ONLINE ONLINE rac1
ora.scan3.vip
1 ONLINE ONLINE rac1
[oracle@rac1 ~]$
Caution: Your use of any information or materials on this website is entirely at your own risk. It is provided for educational purposes only. It has been tested internally, however, we do not guarantee that it will work for you. Ensure that you run it in your test environment before using.
Still page under construction !!! 🙂