Tag Archives: adding node

Add Node

Add Node to 11gR2 Oracle RAC Cluster (11.2.0.3)

0. Environment

1. Pre-installation tasks for GI for a cluster

0.i) Backup OCR
i) Install the cvuqdisk Package for Linux
ii) Verify New Node (HWOS)
iii) Verify Peer (REFNODE)
iv) Verify New Node (New Pre-Node)
v) Run fixup scripts

2. Cluster Node Addition for GI Home.

i) Run addnode.sh script
ii) Run orainstRoot.sh #On nodes rac2
iii) Run root.sh #On nodes rac2
iv) Check Clusterware Resources after ran root.sh
v) Run cluvfy post-addNode script
vi) Check Cluster Nodes
vii) Check TNS Listener
viii) Check ASM Status
ix) Check OCR
x) Check Vote disk

3. Cluster Node Addition for RDBMS Home.

i) Run addnode.sh script
ii) root.sh #On nodes rac2 from RDBMS home

4. Add Instance to Database through Command-Line or you can add via dbca.

i)Pre-task
ii) Add redo thread
iii) Add undo tablespace
iv) Add instance to OCR
v) Add service to new instance via srvctl or you can add via dbca
vi) Check the cluster stack

Let start !!!


0. Environment

One node RAC 11.2.0.3 (Not RAC ONE), single node RAC. Earlier it was two node RAC setup, recently i have deleted 2nd node from cluster for testing.
Node name: RAC1
OS: RHEL 5
DATABASE: nike, instance: nike1

Task: We are going to add node “RAC2” to our existing cluster.

Current status

[root@rac1 ~]# crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       rac1
ora.DATA1.dg
               ONLINE  ONLINE       rac1
ora.LISTENER.lsnr
               ONLINE  ONLINE       rac1
ora.asm
               ONLINE  ONLINE       rac1                     Started
ora.gsd
               OFFLINE OFFLINE      rac1
ora.net1.network
               ONLINE  ONLINE       rac1
ora.ons
               ONLINE  ONLINE       rac1
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       rac1
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       rac1
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       rac1
ora.cvu
      1        ONLINE  ONLINE       rac1
ora.nike.db
      1        ONLINE  ONLINE       rac1                     Open
ora.nike.nike_srv.svc
      1        ONLINE  ONLINE       rac1
ora.oc4j
      1        ONLINE  ONLINE       rac1
ora.rac1.vip
      1        ONLINE  ONLINE       rac1
ora.scan1.vip
      1        ONLINE  ONLINE       rac1
ora.scan2.vip
      1        ONLINE  ONLINE       rac1
ora.scan3.vip
      1        ONLINE  ONLINE       rac1
[root@rac1 ~]#


1. Pre-installation tasks for GI for a cluster

A) Install and Configure the Linux Operating System on the New Node     << This is step already done by SA
B) Configure Access to the Shared Storage     << This is step already done by SA
C) Install and Configure ASMLib     << This is step already done by SA
D) SSH configure    << This is step already done by SA


0.i) Backup OCR

From: Node RAC1
As root

[root@rac1 ~]# ocrconfig -manualbackup
rac1     2015/06/19 23:38:03     /u01/app/11.2.0/grid/cdata/rac-scan/backup_20150619_233803.ocr
[root@rac1 ~]#


i) Install the cvuqdisk Package for Linux

[root@rac2 oracle]# rpm -ivh cvuqdisk-1.0.9-1.rpm
Preparing...                ########################################### [100%]
Using default group oinstall to install package
   1:cvuqdisk               ########################################### [100%]
[root@rac2 oracle]#

Note: Without cvuqdisk package, CVU cannot discover shared disks and you will receive the error message "Package cvuqdisk not installed" when the CVU is run. 

Example below:
==============
Checking shared storage accessibility...

WARNING:
rac2:PRVF-7017 : Package cvuqdisk not installed
        rac2
No shared storage found
Shared storage check failed on nodes "rac2"


ii) Verify New Node (HWOS)

As GI Home owner
From active node RAC1.

[oracle@rac1 ~]$ cluvfy stage -post hwos -n rac2

Performing post-checks for hardware and operating system setup

Checking node reachability...
Node reachability check passed from node "rac1"


Checking user equivalence...
User equivalence check passed for user "oracle"

Checking node connectivity...

Checking hosts config file...

Verification of the hosts config file successful

Node connectivity passed for subnet "192.168.2.0" with node(s) rac2
TCP connectivity check passed for subnet "192.168.2.0"

Node connectivity passed for subnet "192.168.0.0" with node(s) rac2
TCP connectivity check passed for subnet "192.168.0.0"

Node connectivity passed for subnet "10.0.4.0" with node(s) rac2
TCP connectivity check passed for subnet "10.0.4.0"


Interfaces found on subnet "10.0.4.0" that are likely candidates for VIP are:
rac2 eth2:10.0.4.15

Interfaces found on subnet "192.168.2.0" that are likely candidates for a private interconnect are:
rac2 eth0:192.168.2.102

Interfaces found on subnet "192.168.0.0" that are likely candidates for a private interconnect are:
rac2 eth1:192.168.0.102

Node connectivity check passed

Checking multicast communication...

Checking subnet "192.168.2.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "192.168.2.0" for multicast communication with multicast group "230.0.1.0" passed.

Checking subnet "192.168.0.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "192.168.0.0" for multicast communication with multicast group "230.0.1.0" passed.

Checking subnet "10.0.4.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "10.0.4.0" for multicast communication with multicast group "230.0.1.0" passed.

Check of multicast communication passed.
Check for multiple users with UID value 0 passed
Time zone consistency check passed

Checking shared storage accessibility...

  Disk                                  Sharing Nodes (1 in count)
  ------------------------------------  ------------------------
  /dev/sda                              rac2

  Disk                                  Sharing Nodes (1 in count)
  ------------------------------------  ------------------------
  /dev/sdb                              rac2


Shared storage check was successful on nodes "rac2"

Post-check for hardware and operating system setup was successful.
[oracle@rac1 ~]$


iii) Verify Peer (REFNODE)

From active node RAC1
As GI Home owner

[oracle@rac1 ~]$ cluvfy comp peer -refnode rac1 -n rac2 -orainv oinstall -osdba dba -verbose

Verifying peer compatibility

Checking peer compatibility...

Compatibility check: Physical memory [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          3.8633GB (4050940.0KB)    3.8633GB (4050940.0KB)    matched
Physical memory check passed

Compatibility check: Available memory [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          3.6215GB (3797424.0KB)    2.6165GB (2743628.0KB)    mismatched
Available memory check failed

Compatibility check: Swap space [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          1.9994GB (2096472.0KB)    5.5385GB (5807488.0KB)    mismatched
Swap space check failed

Compatibility check: Free disk space for "/u01/app/11.2.0/grid" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          17.165GB (1.7998848E7KB)  21.2725GB (2.2305792E7KB)  mismatched
Free disk space check failed

Compatibility check: Free disk space for "/tmp" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          3.707GB (3887104.0KB)     4.4717GB (4688896.0KB)    mismatched
Free disk space check failed

Compatibility check: User existence for "oracle" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          oracle(1100)              oracle(1100)              matched
User existence for "oracle" check passed

Compatibility check: Group existence for "oinstall" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          oinstall(1000)            oinstall(1000)            matched
Group existence for "oinstall" check passed

Compatibility check: Group existence for "dba" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          dba(1200)                 dba(1200)                 matched
Group existence for "dba" check passed

Compatibility check: Group membership for "oracle" in "oinstall (Primary)" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          yes                       yes                       matched
Group membership for "oracle" in "oinstall (Primary)" check passed

Compatibility check: Group membership for "oracle" in "dba" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          yes                       yes                       matched
Group membership for "oracle" in "dba" check passed

Compatibility check: Run level [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          5                         5                         matched
Run level check passed

Compatibility check: System architecture [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          x86_64                    x86_64                    matched
System architecture check passed

Compatibility check: Kernel version [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          2.6.32-200.13.1.el5uek    2.6.32-200.13.1.el5uek    matched
Kernel version check passed

Compatibility check: Kernel param "semmsl" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          250                       250                       matched
Kernel param "semmsl" check passed

Compatibility check: Kernel param "semmns" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          32000                     32000                     matched
Kernel param "semmns" check passed

Compatibility check: Kernel param "semopm" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          100                       100                       matched
Kernel param "semopm" check passed

Compatibility check: Kernel param "semmni" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          128                       128                       matched
Kernel param "semmni" check passed

Compatibility check: Kernel param "shmmax" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          2074081280                1054504960                mismatched
Kernel param "shmmax" check failed

Compatibility check: Kernel param "shmmni" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          4096                      4096                      matched
Kernel param "shmmni" check passed

Compatibility check: Kernel param "shmall" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          2097152                   2097152                   matched
Kernel param "shmall" check passed

Compatibility check: Kernel param "file-max" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          6815744                   6815744                   matched
Kernel param "file-max" check passed

Compatibility check: Kernel param "ip_local_port_range" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          between 9000.0 & 65500.0  between 9000.0 & 65500.0  matched
Kernel param "ip_local_port_range" check passed

Compatibility check: Kernel param "rmem_default" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          262144                    262144                    matched
Kernel param "rmem_default" check passed

Compatibility check: Kernel param "rmem_max" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          4194304                   4194304                   matched
Kernel param "rmem_max" check passed

Compatibility check: Kernel param "wmem_default" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          262144                    262144                    matched
Kernel param "wmem_default" check passed

Compatibility check: Kernel param "wmem_max" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          1048586                   1048586                   matched
Kernel param "wmem_max" check passed

Compatibility check: Kernel param "aio-max-nr" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          1048576                   1048576                   matched
Kernel param "aio-max-nr" check passed

Compatibility check: Package existence for "make" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          make-3.81-3.el5           make-3.81-3.el5           matched
Package existence for "make" check passed

Compatibility check: Package existence for "binutils" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          binutils-2.17.50.0.6-14.el5  binutils-2.17.50.0.6-14.el5  matched
Package existence for "binutils" check passed

Compatibility check: Package existence for "gcc (x86_64)" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          gcc-4.1.2-51.el5 (x86_64)  gcc-4.1.2-51.el5 (x86_64)  matched
Package existence for "gcc (x86_64)" check passed

Compatibility check: Package existence for "libaio (x86_64)" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          libaio-0.3.106-5 (x86_64),libaio-0.3.106-5 (i386)  libaio-0.3.106-5 (x86_64),libaio-0.3.106-5 (i386)  matched
Package existence for "libaio (x86_64)" check passed

Compatibility check: Package existence for "glibc (x86_64)" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          glibc-2.5-65 (x86_64),glibc-2.5-65 (i686)  glibc-2.5-65 (x86_64),glibc-2.5-65 (i686)  matched
Package existence for "glibc (x86_64)" check passed

Compatibility check: Package existence for "compat-libstdc++-33 (x86_64)" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          compat-libstdc++-33-3.2.3-61 (x86_64),compat-libstdc++-33-3.2.3-61 (i386)  compat-libstdc++-33-3.2.3-61 (x86_64),compat-libstdc++-33-3.2.3-61 (i386)  matched
Package existence for "compat-libstdc++-33 (x86_64)" check passed

Compatibility check: Package existence for "elfutils-libelf (x86_64)" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          elfutils-libelf-0.137-3.el5 (x86_64),elfutils-libelf-0.137-3.el5 (i386)  elfutils-libelf-0.137-3.el5 (x86_64),elfutils-libelf-0.137-3.el5 (i386)  matched
Package existence for "elfutils-libelf (x86_64)" check passed

Compatibility check: Package existence for "elfutils-libelf-devel" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          elfutils-libelf-devel-0.137-3.el5  elfutils-libelf-devel-0.137-3.el5  matched
Package existence for "elfutils-libelf-devel" check passed

Compatibility check: Package existence for "glibc-common" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          glibc-common-2.5-65       glibc-common-2.5-65       matched
Package existence for "glibc-common" check passed

Compatibility check: Package existence for "glibc-devel (x86_64)" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          glibc-devel-2.5-65 (x86_64),glibc-devel-2.5-65 (i386)  glibc-devel-2.5-65 (x86_64),glibc-devel-2.5-65 (i386)  matched
Package existence for "glibc-devel (x86_64)" check passed

Compatibility check: Package existence for "glibc-headers" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          glibc-headers-2.5-65      glibc-headers-2.5-65      matched
Package existence for "glibc-headers" check passed

Compatibility check: Package existence for "gcc-c++ (x86_64)" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          gcc-c++-4.1.2-51.el5 (x86_64)  gcc-c++-4.1.2-51.el5 (x86_64)  matched
Package existence for "gcc-c++ (x86_64)" check passed

Compatibility check: Package existence for "libaio-devel (x86_64)" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          libaio-devel-0.3.106-5 (i386),libaio-devel-0.3.106-5 (x86_64)  libaio-devel-0.3.106-5 (i386),libaio-devel-0.3.106-5 (x86_64)  matched
Package existence for "libaio-devel (x86_64)" check passed

Compatibility check: Package existence for "libgcc (x86_64)" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          libgcc-4.1.2-51.el5 (x86_64),libgcc-4.1.2-51.el5 (i386)  libgcc-4.1.2-51.el5 (x86_64),libgcc-4.1.2-51.el5 (i386)  matched
Package existence for "libgcc (x86_64)" check passed

Compatibility check: Package existence for "libstdc++ (x86_64)" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          libstdc++-4.1.2-51.el5 (x86_64),libstdc++-4.1.2-51.el5 (i386)  libstdc++-4.1.2-51.el5 (x86_64),libstdc++-4.1.2-51.el5 (i386)  matched
Package existence for "libstdc++ (x86_64)" check passed

Compatibility check: Package existence for "libstdc++-devel (x86_64)" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          libstdc++-devel-4.1.2-51.el5 (x86_64)  libstdc++-devel-4.1.2-51.el5 (x86_64)  matched
Package existence for "libstdc++-devel (x86_64)" check passed

Compatibility check: Package existence for "sysstat" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          sysstat-7.0.2-11.el5      sysstat-7.0.2-11.el5      matched
Package existence for "sysstat" check passed

Compatibility check: Package existence for "ksh" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          ksh-20100202-1.el5_6.6    ksh-20100202-1.el5_6.6    matched
Package existence for "ksh" check passed

Verification of peer compatibility was unsuccessful.
Checks did not pass for the following node(s):
        rac2
[oracle@rac1 ~]$


iv) Verify New Node (New Pre-Node)

As GI Home owner
From Node RAC1

[oracle@rac1 ~]$ cluvfy stage -pre nodeadd -n rac2 -fixup -verbose

Performing pre-checks for node addition

Checking node reachability...

Check: Node reachability from node "rac1"
  Destination Node                      Reachable?
  ------------------------------------  ------------------------
  rac2                                  yes
Result: Node reachability check passed from node "rac1"


Checking user equivalence...

Check: User equivalence for user "oracle"
  Node Name                             Status
  ------------------------------------  ------------------------
  rac2                                  passed
Result: User equivalence check passed for user "oracle"

Checking node connectivity...

Checking hosts config file...
  Node Name                             Status
  ------------------------------------  ------------------------
  rac1                                  passed
  rac2                                  passed

Verification of the hosts config file successful


Interface information for node "rac1"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.2.101   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:02:46:97 1500
 eth0   192.168.2.106   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:02:46:97 1500
 eth0   192.168.2.107   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:02:46:97 1500
 eth0   192.168.2.103   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:02:46:97 1500
 eth0   192.168.2.105   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:02:46:97 1500
 eth1   192.168.0.101   192.168.0.0     0.0.0.0         10.0.4.2        08:00:27:9A:66:6A 1500
 eth1   169.254.6.127   169.254.0.0     0.0.0.0         10.0.4.2        08:00:27:9A:66:6A 1500
 eth2   10.0.4.15       10.0.4.0        0.0.0.0         10.0.4.2        08:00:27:D3:B8:9F 1500


Interface information for node "rac2"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.2.102   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:93:7F:00 1500
 eth1   192.168.0.102   192.168.0.0     0.0.0.0         10.0.4.2        08:00:27:63:80:76 1500
 eth2   10.0.4.15       10.0.4.0        0.0.0.0         10.0.4.2        08:00:27:4C:B3:01 1500


Check: Node connectivity for interface "eth0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rac1[192.168.2.101]             rac1[192.168.2.106]             yes
  rac1[192.168.2.101]             rac1[192.168.2.107]             yes
  rac1[192.168.2.101]             rac1[192.168.2.103]             yes
  rac1[192.168.2.101]             rac1[192.168.2.105]             yes
  rac1[192.168.2.101]             rac2[192.168.2.102]             yes
  rac1[192.168.2.106]             rac1[192.168.2.107]             yes
  rac1[192.168.2.106]             rac1[192.168.2.103]             yes
  rac1[192.168.2.106]             rac1[192.168.2.105]             yes
  rac1[192.168.2.106]             rac2[192.168.2.102]             yes
  rac1[192.168.2.107]             rac1[192.168.2.103]             yes
  rac1[192.168.2.107]             rac1[192.168.2.105]             yes
  rac1[192.168.2.107]             rac2[192.168.2.102]             yes
  rac1[192.168.2.103]             rac1[192.168.2.105]             yes
  rac1[192.168.2.103]             rac2[192.168.2.102]             yes
  rac1[192.168.2.105]             rac2[192.168.2.102]             yes
Result: Node connectivity passed for interface "eth0"


Check: TCP connectivity of subnet "192.168.2.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rac1:192.168.2.101              rac1:192.168.2.106              passed
  rac1:192.168.2.101              rac1:192.168.2.107              passed
  rac1:192.168.2.101              rac1:192.168.2.103              passed
  rac1:192.168.2.101              rac1:192.168.2.105              passed
  rac1:192.168.2.101              rac2:192.168.2.102              passed
Result: TCP connectivity check passed for subnet "192.168.2.0"

Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "192.168.2.0".
Subnet mask consistency check passed.

Result: Node connectivity check passed

Checking multicast communication...

Checking subnet "192.168.2.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "192.168.2.0" for multicast communication with multicast group "230.0.1.0" passed.

Check of multicast communication passed.

Checking CRS integrity...

Clusterware version consistency passed
The Oracle Clusterware is healthy on node "rac1"

CRS integrity check passed

Checking shared resources...

Checking CRS home location...
"/u01/app/11.2.0/grid" is shared
Result: Shared resources check for node addition passed


Checking node connectivity...

Checking hosts config file...
  Node Name                             Status
  ------------------------------------  ------------------------
  rac1                                  passed
  rac2                                  passed

Verification of the hosts config file successful


Interface information for node "rac1"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.2.101   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:02:46:97 1500
 eth0   192.168.2.106   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:02:46:97 1500
 eth0   192.168.2.107   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:02:46:97 1500
 eth0   192.168.2.103   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:02:46:97 1500
 eth0   192.168.2.105   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:02:46:97 1500
 eth1   192.168.0.101   192.168.0.0     0.0.0.0         10.0.4.2        08:00:27:9A:66:6A 1500
 eth1   169.254.6.127   169.254.0.0     0.0.0.0         10.0.4.2        08:00:27:9A:66:6A 1500
 eth2   10.0.4.15       10.0.4.0        0.0.0.0         10.0.4.2        08:00:27:D3:B8:9F 1500


Interface information for node "rac2"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.2.102   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:93:7F:00 1500
 eth1   192.168.0.102   192.168.0.0     0.0.0.0         10.0.4.2        08:00:27:63:80:76 1500
 eth2   10.0.4.15       10.0.4.0        0.0.0.0         10.0.4.2        08:00:27:4C:B3:01 1500


Check: Node connectivity for interface "eth0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rac1[192.168.2.101]             rac1[192.168.2.106]             yes
  rac1[192.168.2.101]             rac1[192.168.2.107]             yes
  rac1[192.168.2.101]             rac1[192.168.2.103]             yes
  rac1[192.168.2.101]             rac1[192.168.2.105]             yes
  rac1[192.168.2.101]             rac2[192.168.2.102]             yes
  rac1[192.168.2.106]             rac1[192.168.2.107]             yes
  rac1[192.168.2.106]             rac1[192.168.2.103]             yes
  rac1[192.168.2.106]             rac1[192.168.2.105]             yes
  rac1[192.168.2.106]             rac2[192.168.2.102]             yes
  rac1[192.168.2.107]             rac1[192.168.2.103]             yes
  rac1[192.168.2.107]             rac1[192.168.2.105]             yes
  rac1[192.168.2.107]             rac2[192.168.2.102]             yes
  rac1[192.168.2.103]             rac1[192.168.2.105]             yes
  rac1[192.168.2.103]             rac2[192.168.2.102]             yes
  rac1[192.168.2.105]             rac2[192.168.2.102]             yes
Result: Node connectivity passed for interface "eth0"


Check: TCP connectivity of subnet "192.168.2.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rac1:192.168.2.101              rac1:192.168.2.106              passed
  rac1:192.168.2.101              rac1:192.168.2.107              passed
  rac1:192.168.2.101              rac1:192.168.2.103              passed
  rac1:192.168.2.101              rac1:192.168.2.105              passed
  rac1:192.168.2.101              rac2:192.168.2.102              passed
Result: TCP connectivity check passed for subnet "192.168.2.0"


Check: Node connectivity for interface "eth1"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rac1[192.168.0.101]             rac2[192.168.0.102]             yes
Result: Node connectivity passed for interface "eth1"


Check: TCP connectivity of subnet "192.168.0.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rac1:192.168.0.101              rac2:192.168.0.102              passed
Result: TCP connectivity check passed for subnet "192.168.0.0"

Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "192.168.2.0".
Subnet mask consistency check passed for subnet "192.168.0.0".
Subnet mask consistency check passed.

Result: Node connectivity check passed

Checking multicast communication...

Checking subnet "192.168.2.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "192.168.2.0" for multicast communication with multicast group "230.0.1.0" passed.

Checking subnet "192.168.0.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "192.168.0.0" for multicast communication with multicast group "230.0.1.0" passed.

Check of multicast communication passed.

Check: Total memory
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          3.8633GB (4050940.0KB)    1.5GB (1572864.0KB)       passed
  rac1          3.8633GB (4050940.0KB)    1.5GB (1572864.0KB)       passed
Result: Total memory check passed

Check: Available memory
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          3.6216GB (3797500.0KB)    50MB (51200.0KB)          passed
  rac1          2.4853GB (2606016.0KB)    50MB (51200.0KB)          passed
Result: Available memory check passed

Check: Swap space
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          1.9994GB (2096472.0KB)    3.8633GB (4050940.0KB)    failed  <<<<<
  rac1          5.5385GB (5807488.0KB)    3.8633GB (4050940.0KB)    passed
Result: Swap space check failed

Check: Free disk space for "rac2:/u01/app/11.2.0/grid"
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /u01/app/11.2.0/grid  rac2          /u01          17.165GB      5.5GB         passed
Result: Free disk space check passed for "rac2:/u01/app/11.2.0/grid"

Check: Free disk space for "rac1:/u01/app/11.2.0/grid"
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /u01/app/11.2.0/grid  rac1          /u01          21.2705GB     5.5GB         passed
Result: Free disk space check passed for "rac1:/u01/app/11.2.0/grid"

Check: Free disk space for "rac2:/tmp"
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /tmp              rac2          /             3.707GB       1GB           passed
Result: Free disk space check passed for "rac2:/tmp"

Check: Free disk space for "rac1:/tmp"
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /tmp              rac1          /             4.4717GB      1GB           passed
Result: Free disk space check passed for "rac1:/tmp"

Check: User existence for "oracle"
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rac2          passed                    exists(1100)
  rac1          passed                    exists(1100)

Checking for multiple users with UID value 1100
Result: Check for multiple users with UID value 1100 passed
Result: User existence check passed for "oracle"

Check: Run level
  Node Name     run level                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          5                         3,5                       passed
  rac1          5                         3,5                       passed
Result: Run level check passed

Check: Hard limits for "maximum open file descriptors"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rac1              hard          65536         65536         passed
  rac2              hard          65536         65536         passed
Result: Hard limits check passed for "maximum open file descriptors"

Check: Soft limits for "maximum open file descriptors"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rac1              soft          1024          1024          passed
  rac2              soft          1024          1024          passed
Result: Soft limits check passed for "maximum open file descriptors"

Check: Hard limits for "maximum user processes"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rac1              hard          16384         16384         passed
  rac2              hard          16384         16384         passed
Result: Hard limits check passed for "maximum user processes"

Check: Soft limits for "maximum user processes"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rac1              soft          2047          2047          passed
  rac2              soft          2047          2047          passed
Result: Soft limits check passed for "maximum user processes"

Check: System architecture
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          x86_64                    x86_64                    passed
  rac1          x86_64                    x86_64                    passed
Result: System architecture check passed

Check: Kernel version
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          2.6.32-200.13.1.el5uek    2.6.18                    passed
  rac1          2.6.32-200.13.1.el5uek    2.6.18                    passed
Result: Kernel version check passed

Check: Kernel parameter for "semmsl"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac1              250           250           250           passed
  rac2              250           250           250           passed
Result: Kernel parameter check passed for "semmsl"

Check: Kernel parameter for "semmns"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac1              32000         32000         32000         passed
  rac2              32000         32000         32000         passed
Result: Kernel parameter check passed for "semmns"

Check: Kernel parameter for "semopm"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac1              100           100           100           passed
  rac2              100           100           100           passed
Result: Kernel parameter check passed for "semopm"

Check: Kernel parameter for "semmni"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac1              128           128           128           passed
  rac2              128           128           128           passed
Result: Kernel parameter check passed for "semmni"

Check: Kernel parameter for "shmmax"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac1              1054504960    1054504960    2074081280    failed        Current value too low. Configured value too low.  <<< 
  rac2              2074081280    1054504960    2074081280    failed        Configured value too low.  <<<
Result: Kernel parameter check failed for "shmmax"

Check: Kernel parameter for "shmmni"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac1              4096          4096          4096          passed
  rac2              4096          4096          4096          passed
Result: Kernel parameter check passed for "shmmni"

Check: Kernel parameter for "shmall"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac1              2097152       2097152       2097152       passed
  rac2              2097152       2097152       2097152       passed
Result: Kernel parameter check passed for "shmall"

Check: Kernel parameter for "file-max"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac1              6815744       6815744       6815744       passed
  rac2              6815744       6815744       6815744       passed
Result: Kernel parameter check passed for "file-max"

Check: Kernel parameter for "ip_local_port_range"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac1              between 9000.0 & 65500.0  between 9000.0 & 65500.0  between 9000.0 & 65500.0  passed
  rac2              between 9000.0 & 65500.0  between 9000.0 & 65500.0  between 9000.0 & 65500.0  passed
Result: Kernel parameter check passed for "ip_local_port_range"

Check: Kernel parameter for "rmem_default"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac1              262144        262144        262144        passed
  rac2              262144        262144        262144        passed
Result: Kernel parameter check passed for "rmem_default"

Check: Kernel parameter for "rmem_max"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac1              4194304       4194304       4194304       passed
  rac2              4194304       4194304       4194304       passed
Result: Kernel parameter check passed for "rmem_max"

Check: Kernel parameter for "wmem_default"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac1              262144        262144        262144        passed
  rac2              262144        262144        262144        passed
Result: Kernel parameter check passed for "wmem_default"

Check: Kernel parameter for "wmem_max"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac1              1048586       1048586       1048576       passed
  rac2              1048586       1048586       1048576       passed
Result: Kernel parameter check passed for "wmem_max"

Check: Kernel parameter for "aio-max-nr"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac1              1048576       1048576       1048576       passed
  rac2              1048576       1048576       1048576       passed
Result: Kernel parameter check passed for "aio-max-nr"

Check: Package existence for "make"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          make-3.81-3.el5           make-3.81                 passed
  rac1          make-3.81-3.el5           make-3.81                 passed
Result: Package existence check passed for "make"

Check: Package existence for "binutils"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          binutils-2.17.50.0.6-14.el5  binutils-2.17.50.0.6      passed
  rac1          binutils-2.17.50.0.6-14.el5  binutils-2.17.50.0.6      passed
Result: Package existence check passed for "binutils"

Check: Package existence for "gcc(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          gcc(x86_64)-4.1.2-51.el5  gcc(x86_64)-4.1.2         passed
  rac1          gcc(x86_64)-4.1.2-51.el5  gcc(x86_64)-4.1.2         passed
Result: Package existence check passed for "gcc(x86_64)"

Check: Package existence for "libaio(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          libaio(x86_64)-0.3.106-5  libaio(x86_64)-0.3.106    passed
  rac1          libaio(x86_64)-0.3.106-5  libaio(x86_64)-0.3.106    passed
Result: Package existence check passed for "libaio(x86_64)"

Check: Package existence for "glibc(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          glibc(x86_64)-2.5-65      glibc(x86_64)-2.5-24      passed
  rac1          glibc(x86_64)-2.5-65      glibc(x86_64)-2.5-24      passed
Result: Package existence check passed for "glibc(x86_64)"

Check: Package existence for "compat-libstdc++-33(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          compat-libstdc++-33(x86_64)-3.2.3-61  compat-libstdc++-33(x86_64)-3.2.3  passed
  rac1          compat-libstdc++-33(x86_64)-3.2.3-61  compat-libstdc++-33(x86_64)-3.2.3  passed
Result: Package existence check passed for "compat-libstdc++-33(x86_64)"

Check: Package existence for "elfutils-libelf(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          elfutils-libelf(x86_64)-0.137-3.el5  elfutils-libelf(x86_64)-0.125  passed
  rac1          elfutils-libelf(x86_64)-0.137-3.el5  elfutils-libelf(x86_64)-0.125  passed
Result: Package existence check passed for "elfutils-libelf(x86_64)"

Check: Package existence for "elfutils-libelf-devel"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          elfutils-libelf-devel-0.137-3.el5  elfutils-libelf-devel-0.125  passed
  rac1          elfutils-libelf-devel-0.137-3.el5  elfutils-libelf-devel-0.125  passed
Result: Package existence check passed for "elfutils-libelf-devel"

Check: Package existence for "glibc-common"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          glibc-common-2.5-65       glibc-common-2.5          passed
  rac1          glibc-common-2.5-65       glibc-common-2.5          passed
Result: Package existence check passed for "glibc-common"

Check: Package existence for "glibc-devel(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          glibc-devel(x86_64)-2.5-65  glibc-devel(x86_64)-2.5   passed
  rac1          glibc-devel(x86_64)-2.5-65  glibc-devel(x86_64)-2.5   passed
Result: Package existence check passed for "glibc-devel(x86_64)"

Check: Package existence for "glibc-headers"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          glibc-headers-2.5-65      glibc-headers-2.5         passed
  rac1          glibc-headers-2.5-65      glibc-headers-2.5         passed
Result: Package existence check passed for "glibc-headers"

Check: Package existence for "gcc-c++(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          gcc-c++(x86_64)-4.1.2-51.el5  gcc-c++(x86_64)-4.1.2     passed
  rac1          gcc-c++(x86_64)-4.1.2-51.el5  gcc-c++(x86_64)-4.1.2     passed
Result: Package existence check passed for "gcc-c++(x86_64)"

Check: Package existence for "libaio-devel(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          libaio-devel(x86_64)-0.3.106-5  libaio-devel(x86_64)-0.3.106  passed
  rac1          libaio-devel(x86_64)-0.3.106-5  libaio-devel(x86_64)-0.3.106  passed
Result: Package existence check passed for "libaio-devel(x86_64)"

Check: Package existence for "libgcc(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          libgcc(x86_64)-4.1.2-51.el5  libgcc(x86_64)-4.1.2      passed
  rac1          libgcc(x86_64)-4.1.2-51.el5  libgcc(x86_64)-4.1.2      passed
Result: Package existence check passed for "libgcc(x86_64)"

Check: Package existence for "libstdc++(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          libstdc++(x86_64)-4.1.2-51.el5  libstdc++(x86_64)-4.1.2   passed
  rac1          libstdc++(x86_64)-4.1.2-51.el5  libstdc++(x86_64)-4.1.2   passed
Result: Package existence check passed for "libstdc++(x86_64)"

Check: Package existence for "libstdc++-devel(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          libstdc++-devel(x86_64)-4.1.2-51.el5  libstdc++-devel(x86_64)-4.1.2  passed
  rac1          libstdc++-devel(x86_64)-4.1.2-51.el5  libstdc++-devel(x86_64)-4.1.2  passed
Result: Package existence check passed for "libstdc++-devel(x86_64)"

Check: Package existence for "sysstat"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          sysstat-7.0.2-11.el5      sysstat-7.0.2             passed
  rac1          sysstat-7.0.2-11.el5      sysstat-7.0.2             passed
Result: Package existence check passed for "sysstat"

Check: Package existence for "ksh"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          ksh-20100202-1.el5_6.6    ksh-20060214              passed
  rac1          ksh-20100202-1.el5_6.6    ksh-20060214              passed
Result: Package existence check passed for "ksh"

Checking for multiple users with UID value 0
Result: Check for multiple users with UID value 0 passed

Check: Current group ID
Result: Current group ID check passed

Starting check for consistency of primary group of root user
  Node Name                             Status
  ------------------------------------  ------------------------
  rac2                                  passed
  rac1                                  passed

Check for consistency of root user's primary group passed

Checking OCR integrity...

OCR integrity check passed

Checking Oracle Cluster Voting Disk configuration...

Oracle Cluster Voting Disk configuration check passed
Check: Time zone consistency
Result: Time zone consistency check passed

Starting Clock synchronization checks using Network Time Protocol(NTP)...

NTP Configuration file check started...
Network Time Protocol(NTP) configuration file not found on any of the nodes. Oracle Cluster Time Synchronization Service(CTSS) can be used instead of NTP for time synchronization on the cluster nodes
No NTP Daemons or Services were found to be running

Result: Clock synchronization check using Network Time Protocol(NTP) passed


Checking to make sure user "oracle" is not in "root" group
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rac2          passed                    does not exist
  rac1          passed                    does not exist
Result: User "oracle" is not part of "root" group. Check passed
Checking consistency of file "/etc/resolv.conf" across nodes

Checking the file "/etc/resolv.conf" to make sure only one of domain and search entries is defined
File "/etc/resolv.conf" does not have both domain and search entries defined
Checking if domain entry in file "/etc/resolv.conf" is consistent across the nodes...
domain entry in file "/etc/resolv.conf" is consistent across nodes
Checking if search entry in file "/etc/resolv.conf" is consistent across the nodes...
search entry in file "/etc/resolv.conf" is consistent across nodes
Checking file "/etc/resolv.conf" to make sure that only one search entry is defined
All nodes have one search entry defined in file "/etc/resolv.conf"
Checking all nodes to make sure that search entry is "rajasekhar.com" as found on node "rac1"
All nodes of the cluster have same value for 'search'
Checking DNS response time for an unreachable node
  Node Name                             Status
  ------------------------------------  ------------------------
  rac1                                  failed
  rac2                                  failed
PRVF-5636 : The DNS response time for an unreachable node exceeded "15000" ms on following nodes: rac1,rac2

File "/etc/resolv.conf" is not consistent across nodes

Fixup information has been generated for following node(s):
rac2,rac1
Please run the following script on each node as "root" user to execute the fixups:
'/tmp/CVU_11.2.0.3.0_oracle/runfixup.sh'

Pre-check for node addition was unsuccessful on all the nodes.
[oracle@rac1 ~]$


v) Run fixup scripts

On RAC 1, as root user

[root@rac1 ~]# /tmp/CVU_11.2.0.3.0_oracle/runfixup.sh
Response file being used is :/tmp/CVU_11.2.0.3.0_oracle/fixup.response
Enable file being used is :/tmp/CVU_11.2.0.3.0_oracle/fixup.enable
Log file location: /tmp/CVU_11.2.0.3.0_oracle/orarun.log
Setting Kernel Parameters...
kernel.shmmax = 68719476736
kernel.shmmax = 1054504960
/tmp/CVU_11.2.0.3.0_oracle/orarun.sh: line 230: [: 68719476736kernel.shmmax: integer expression expected
The value for shmmax in response file is not greater than value for shmmax in /etc/sysctl.conf file. Hence not changing it.
kernel.shmmax = 2074081280
[root@rac1 ~]#

On RAC2, as root user

[root@rac2 ~]# /tmp/CVU_11.2.0.3.0_oracle/runfixup.sh
Response file being used is :/tmp/CVU_11.2.0.3.0_oracle/fixup.response
Enable file being used is :/tmp/CVU_11.2.0.3.0_oracle/fixup.enable
Log file location: /tmp/CVU_11.2.0.3.0_oracle/orarun.log
Setting Kernel Parameters...
kernel.shmmax = 68719476736
kernel.shmmax = 1054504960
/tmp/CVU_11.2.0.3.0_oracle/orarun.sh: line 230: [: 68719476736kernel.shmmax: integer expression expected
The value for shmmax in response file is not greater than value for shmmax in /etc/sysctl.conf file. Hence not changing it.
The value for shmmax in response file is not greater than value of shmmax for current session. Hence not changing it.
[root@rac2 ~]#


2. Cluster Node Addition for GI Home.

Note: Pre-node check failed. Hence need to set the environment variable IGNORE_PREADDNODE_CHECKS=Y before running addNode.sh otherwise, the silent node addition will fail without showing any errors to the console.


i) Run addnode.sh script

As GI Home owner
From active node RAC1

[oracle@rac1 ~]$ cd /u01/app/11.2.0/grid/oui/bin/
[oracle@rac1 bin]$ IGNORE_PREADDNODE_CHECKS=Y
[oracle@rac1 bin]$ export IGNORE_PREADDNODE_CHECKS
[oracle@rac1 bin]$ ./addNode.sh -silent "CLUSTER_NEW_NODES={rac2}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={rac2-vip}"
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 5671 MB    Passed
Oracle Universal Installer, Version 11.2.0.3.0 Production
Copyright (C) 1999, 2011, Oracle. All rights reserved.


Performing tests to see whether nodes rac2 are available
............................................................... 100% Done.

.
-----------------------------------------------------------------------------
Cluster Node Addition Summary
Global Settings
   Source: /u01/app/11.2.0/grid
   New Nodes
Space Requirements
   New Nodes
      rac2
         /u01: Required 6.99GB : Available 15.98GB
Installed Products
   Product Names
      Oracle Grid Infrastructure 11.2.0.3.0
      Sun JDK 1.5.0.30.03
      Installer SDK Component 11.2.0.3.0
      Oracle One-Off Patch Installer 11.2.0.1.7
      Oracle Universal Installer 11.2.0.3.0
      Oracle USM Deconfiguration 11.2.0.3.0
      Oracle Configuration Manager Deconfiguration 10.3.1.0.0
      Enterprise Manager Common Core Files 10.2.0.4.4
      Oracle DBCA Deconfiguration 11.2.0.3.0
      Oracle RAC Deconfiguration 11.2.0.3.0
      Oracle Quality of Service Management (Server) 11.2.0.3.0
      Installation Plugin Files 11.2.0.3.0
      Universal Storage Manager Files 11.2.0.3.0
      Oracle Text Required Support Files 11.2.0.3.0
      Automatic Storage Management Assistant 11.2.0.3.0
      Oracle Database 11g Multimedia Files 11.2.0.3.0
      Oracle Multimedia Java Advanced Imaging 11.2.0.3.0
      Oracle Globalization Support 11.2.0.3.0
      Oracle Multimedia Locator RDBMS Files 11.2.0.3.0
      Oracle Core Required Support Files 11.2.0.3.0
      Bali Share 1.1.18.0.0
      Oracle Database Deconfiguration 11.2.0.3.0
      Oracle Quality of Service Management (Client) 11.2.0.3.0
      Expat libraries 2.0.1.0.1
      Oracle Containers for Java 11.2.0.3.0
      Perl Modules 5.10.0.0.1
      Secure Socket Layer 11.2.0.3.0
      Oracle JDBC/OCI Instant Client 11.2.0.3.0
      Oracle Multimedia Client Option 11.2.0.3.0
      LDAP Required Support Files 11.2.0.3.0
      Character Set Migration Utility 11.2.0.3.0
      Perl Interpreter 5.10.0.0.2
      PL/SQL Embedded Gateway 11.2.0.3.0
      OLAP SQL Scripts 11.2.0.3.0
      Database SQL Scripts 11.2.0.3.0
      Oracle Extended Windowing Toolkit 3.4.47.0.0
      SSL Required Support Files for InstantClient 11.2.0.3.0
      SQL*Plus Files for Instant Client 11.2.0.3.0
      Oracle Net Required Support Files 11.2.0.3.0
      Oracle Database User Interface 2.2.13.0.0
      RDBMS Required Support Files for Instant Client 11.2.0.3.0
      RDBMS Required Support Files Runtime 11.2.0.3.0
      XML Parser for Java 11.2.0.3.0
      Oracle Security Developer Tools 11.2.0.3.0
      Oracle Wallet Manager 11.2.0.3.0
      Enterprise Manager plugin Common Files 11.2.0.3.0
      Platform Required Support Files 11.2.0.3.0
      Oracle JFC Extended Windowing Toolkit 4.2.36.0.0
      RDBMS Required Support Files 11.2.0.3.0
      Oracle Ice Browser 5.2.3.6.0
      Oracle Help For Java 4.2.9.0.0
      Enterprise Manager Common Files 10.2.0.4.3
      Deinstallation Tool 11.2.0.3.0
      Oracle Java Client 11.2.0.3.0
      Cluster Verification Utility Files 11.2.0.3.0
      Oracle Notification Service (eONS) 11.2.0.3.0
      Oracle LDAP administration 11.2.0.3.0
      Cluster Verification Utility Common Files 11.2.0.3.0
      Oracle Clusterware RDBMS Files 11.2.0.3.0
      Oracle Locale Builder 11.2.0.3.0
      Oracle Globalization Support 11.2.0.3.0
      Buildtools Common Files 11.2.0.3.0
      Oracle RAC Required Support Files-HAS 11.2.0.3.0
      SQL*Plus Required Support Files 11.2.0.3.0
      XDK Required Support Files 11.2.0.3.0
      Agent Required Support Files 10.2.0.4.3
      Parser Generator Required Support Files 11.2.0.3.0
      Precompiler Required Support Files 11.2.0.3.0
      Installation Common Files 11.2.0.3.0
      Required Support Files 11.2.0.3.0
      Oracle JDBC/THIN Interfaces 11.2.0.3.0
      Oracle Multimedia Locator 11.2.0.3.0
      Oracle Multimedia 11.2.0.3.0
      HAS Common Files 11.2.0.3.0
      Assistant Common Files 11.2.0.3.0
      PL/SQL 11.2.0.3.0
      HAS Files for DB 11.2.0.3.0
      Oracle Recovery Manager 11.2.0.3.0
      Oracle Database Utilities 11.2.0.3.0
      Oracle Notification Service 11.2.0.3.0
      SQL*Plus 11.2.0.3.0
      Oracle Netca Client 11.2.0.3.0
      Oracle Net 11.2.0.3.0
      Oracle JVM 11.2.0.3.0
      Oracle Internet Directory Client 11.2.0.3.0
      Oracle Net Listener 11.2.0.3.0
      Cluster Ready Services Files 11.2.0.3.0
      Oracle Database 11g 11.2.0.3.0
-----------------------------------------------------------------------------


Instantiating scripts for add node (Sunday, June 21, 2015 12:35:19 PM IST)
.                                                                 1% Done.
Instantiation of add node scripts complete

Copying to remote nodes (Sunday, June 21, 2015 12:35:22 PM IST)
...............................................................................................                                 96% Done.
Home copied to new nodes

Saving inventory on nodes (Sunday, June 21, 2015 12:45:29 PM IST)
.                                                               100% Done.
Save inventory complete
WARNING:A new inventory has been created on one or more nodes in this session. However, it has not yet been registered as the central inventory of this system.
To register the new inventory please run the script at '/u01/app/oraInventory/orainstRoot.sh' with root privileges on nodes 'rac2'.
If you do not register the inventory, you may not be able to update or patch the products you installed.
The following configuration scripts need to be executed as the "root" user in each new cluster node. Each script in the list below is followed by a list of nodes.
/u01/app/oraInventory/orainstRoot.sh #On nodes rac2
/u01/app/11.2.0/grid/root.sh #On nodes rac2
To execute the configuration scripts:
    1. Open a terminal window
    2. Log in as "root"
    3. Run the scripts in each cluster node

The Cluster Node Addition of /u01/app/11.2.0/grid was successful.
Please check '/tmp/silentInstall.log' for more details.
[oracle@rac1 bin]$

Note: If you are using GNS:
cd $GI_HOME/oui/bin
./addNode.sh -silent “CLUSTER_NEW_NODES={rac2}”


ii) Run orainstRoot.sh #On nodes rac2

On node RAC2
As root

[root@rac2 ~]# /u01/app/oraInventory/orainstRoot.sh
Creating the Oracle inventory pointer file (/etc/oraInst.loc)
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.
[root@rac2 ~]#


iii) Run root.sh #On nodes rac2

On node RAC2
As root

[root@rac2 ~]# /u01/app/11.2.0/grid/root.sh
Performing root user operation for Oracle 11g

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
User ignored Prerequisites during installation
OLR initialization - successful
Adding Clusterware entries to inittab
CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node rac1, number 1, and is terminating
An active cluster was found during exclusive startup, restarting to join the cluster
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 11g Release 2.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
[root@rac2 ~]#

Note: The root.sh script will configuring Grid Infrastructure on the new node and includes adding High Availability Services to the /etc/inittab so that CRS starts up when the machine starts.


iv) Check Clusterware Resources after ran root.sh

[root@rac2 ~]# crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       rac1
               ONLINE  ONLINE       rac2
ora.DATA1.dg
               ONLINE  ONLINE       rac1
               ONLINE  ONLINE       rac2
ora.LISTENER.lsnr
               ONLINE  ONLINE       rac1
               ONLINE  ONLINE       rac2
ora.asm
               ONLINE  ONLINE       rac1                     Started
               ONLINE  ONLINE       rac2                     Started
ora.gsd
               OFFLINE OFFLINE      rac1
               OFFLINE OFFLINE      rac2
ora.net1.network
               ONLINE  ONLINE       rac1
               ONLINE  ONLINE       rac2
ora.ons
               ONLINE  ONLINE       rac1
               ONLINE  ONLINE       rac2
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       rac2
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       rac1
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       rac1
ora.cvu
      1        ONLINE  ONLINE       rac1
ora.nike.db
      1        ONLINE  ONLINE       rac1                     Open
ora.nike.nike_srv.svc
      1        ONLINE  ONLINE       rac1
ora.oc4j
      1        ONLINE  ONLINE       rac1
ora.rac1.vip
      1        ONLINE  ONLINE       rac1
ora.rac2.vip
      1        ONLINE  ONLINE       rac2
ora.scan1.vip
      1        ONLINE  ONLINE       rac2
ora.scan2.vip
      1        ONLINE  ONLINE       rac1
ora.scan3.vip
      1        ONLINE  ONLINE       rac1
[root@rac2 ~]#

[root@rac2 ~]# crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
[root@rac2 ~]#


v) Run cluvfy post-node add script

As GI Home owner
From node RAC1 as best practice, because initially we ran pre-node cluvfy from RAC 1 only

[oracle@rac1 ~]$ cluvfy stage -post nodeadd -n rac2 -verbose

Performing post-checks for node addition

Checking node reachability...

Check: Node reachability from node "rac1"
  Destination Node                      Reachable?
  ------------------------------------  ------------------------
  rac2                                  yes
Result: Node reachability check passed from node "rac1"


Checking user equivalence...

Check: User equivalence for user "oracle"
  Node Name                             Status
  ------------------------------------  ------------------------
  rac2                                  passed
Result: User equivalence check passed for user "oracle"

Checking node connectivity...

Checking hosts config file...
  Node Name                             Status
  ------------------------------------  ------------------------
  rac2                                  passed
  rac1                                  passed

Verification of the hosts config file successful


Interface information for node "rac2"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.2.102   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:93:7F:00 1500
 eth0   192.168.2.107   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:93:7F:00 1500
 eth0   192.168.2.104   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:93:7F:00 1500
 eth1   192.168.0.102   192.168.0.0     0.0.0.0         10.0.4.2        08:00:27:63:80:76 1500
 eth1   169.254.215.111 169.254.0.0     0.0.0.0         10.0.4.2        08:00:27:63:80:76 1500
 eth2   10.0.4.15       10.0.4.0        0.0.0.0         10.0.4.2        08:00:27:4C:B3:01 1500


Interface information for node "rac1"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.2.101   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:02:46:97 1500
 eth0   192.168.2.106   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:02:46:97 1500
 eth0   192.168.2.103   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:02:46:97 1500
 eth0   192.168.2.105   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:02:46:97 1500
 eth1   192.168.0.101   192.168.0.0     0.0.0.0         10.0.4.2        08:00:27:9A:66:6A 1500
 eth1   169.254.6.127   169.254.0.0     0.0.0.0         10.0.4.2        08:00:27:9A:66:6A 1500
 eth2   10.0.4.15       10.0.4.0        0.0.0.0         10.0.4.2        08:00:27:D3:B8:9F 1500


Check: Node connectivity for interface "eth0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rac2[192.168.2.102]             rac2[192.168.2.107]             yes
  rac2[192.168.2.102]             rac2[192.168.2.104]             yes
  rac2[192.168.2.102]             rac1[192.168.2.101]             yes
  rac2[192.168.2.102]             rac1[192.168.2.106]             yes
  rac2[192.168.2.102]             rac1[192.168.2.103]             yes
  rac2[192.168.2.102]             rac1[192.168.2.105]             yes
  rac2[192.168.2.107]             rac2[192.168.2.104]             yes
  rac2[192.168.2.107]             rac1[192.168.2.101]             yes
  rac2[192.168.2.107]             rac1[192.168.2.106]             yes
  rac2[192.168.2.107]             rac1[192.168.2.103]             yes
  rac2[192.168.2.107]             rac1[192.168.2.105]             yes
  rac2[192.168.2.104]             rac1[192.168.2.101]             yes
  rac2[192.168.2.104]             rac1[192.168.2.106]             yes
  rac2[192.168.2.104]             rac1[192.168.2.103]             yes
  rac2[192.168.2.104]             rac1[192.168.2.105]             yes
  rac1[192.168.2.101]             rac1[192.168.2.106]             yes
  rac1[192.168.2.101]             rac1[192.168.2.103]             yes
  rac1[192.168.2.101]             rac1[192.168.2.105]             yes
  rac1[192.168.2.106]             rac1[192.168.2.103]             yes
  rac1[192.168.2.106]             rac1[192.168.2.105]             yes
  rac1[192.168.2.103]             rac1[192.168.2.105]             yes
Result: Node connectivity passed for interface "eth0"


Check: TCP connectivity of subnet "192.168.2.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rac1:192.168.2.101              rac2:192.168.2.102              passed
  rac1:192.168.2.101              rac2:192.168.2.107              passed
  rac1:192.168.2.101              rac2:192.168.2.104              passed
  rac1:192.168.2.101              rac1:192.168.2.106              passed
  rac1:192.168.2.101              rac1:192.168.2.103              passed
  rac1:192.168.2.101              rac1:192.168.2.105              passed
Result: TCP connectivity check passed for subnet "192.168.2.0"

Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "192.168.2.0".
Subnet mask consistency check passed.

Result: Node connectivity check passed

Checking multicast communication...

Checking subnet "192.168.2.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "192.168.2.0" for multicast communication with multicast group "230.0.1.0" passed.

Check of multicast communication passed.

Checking cluster integrity...

  Node Name
  ------------------------------------
  rac1
  rac2

Cluster integrity check passed


Checking CRS integrity...

Clusterware version consistency passed
The Oracle Clusterware is healthy on node "rac2"
The Oracle Clusterware is healthy on node "rac1"

CRS integrity check passed

Checking shared resources...

Checking CRS home location...
"/u01/app/11.2.0/grid" is not shared
Result: Shared resources check for node addition passed


Checking node connectivity...

Checking hosts config file...
  Node Name                             Status
  ------------------------------------  ------------------------
  rac2                                  passed
  rac1                                  passed

Verification of the hosts config file successful


Interface information for node "rac2"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.2.102   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:93:7F:00 1500
 eth0   192.168.2.107   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:93:7F:00 1500
 eth0   192.168.2.104   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:93:7F:00 1500
 eth1   192.168.0.102   192.168.0.0     0.0.0.0         10.0.4.2        08:00:27:63:80:76 1500
 eth1   169.254.215.111 169.254.0.0     0.0.0.0         10.0.4.2        08:00:27:63:80:76 1500
 eth2   10.0.4.15       10.0.4.0        0.0.0.0         10.0.4.2        08:00:27:4C:B3:01 1500


Interface information for node "rac1"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.2.101   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:02:46:97 1500
 eth0   192.168.2.106   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:02:46:97 1500
 eth0   192.168.2.103   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:02:46:97 1500
 eth0   192.168.2.105   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:02:46:97 1500
 eth1   192.168.0.101   192.168.0.0     0.0.0.0         10.0.4.2        08:00:27:9A:66:6A 1500
 eth1   169.254.6.127   169.254.0.0     0.0.0.0         10.0.4.2        08:00:27:9A:66:6A 1500
 eth2   10.0.4.15       10.0.4.0        0.0.0.0         10.0.4.2        08:00:27:D3:B8:9F 1500


Check: Node connectivity for interface "eth0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rac2[192.168.2.102]             rac2[192.168.2.107]             yes
  rac2[192.168.2.102]             rac2[192.168.2.104]             yes
  rac2[192.168.2.102]             rac1[192.168.2.101]             yes
  rac2[192.168.2.102]             rac1[192.168.2.106]             yes
  rac2[192.168.2.102]             rac1[192.168.2.103]             yes
  rac2[192.168.2.102]             rac1[192.168.2.105]             yes
  rac2[192.168.2.107]             rac2[192.168.2.104]             yes
  rac2[192.168.2.107]             rac1[192.168.2.101]             yes
  rac2[192.168.2.107]             rac1[192.168.2.106]             yes
  rac2[192.168.2.107]             rac1[192.168.2.103]             yes
  rac2[192.168.2.107]             rac1[192.168.2.105]             yes
  rac2[192.168.2.104]             rac1[192.168.2.101]             yes
  rac2[192.168.2.104]             rac1[192.168.2.106]             yes
  rac2[192.168.2.104]             rac1[192.168.2.103]             yes
  rac2[192.168.2.104]             rac1[192.168.2.105]             yes
  rac1[192.168.2.101]             rac1[192.168.2.106]             yes
  rac1[192.168.2.101]             rac1[192.168.2.103]             yes
  rac1[192.168.2.101]             rac1[192.168.2.105]             yes
  rac1[192.168.2.106]             rac1[192.168.2.103]             yes
  rac1[192.168.2.106]             rac1[192.168.2.105]             yes
  rac1[192.168.2.103]             rac1[192.168.2.105]             yes
Result: Node connectivity passed for interface "eth0"


Check: TCP connectivity of subnet "192.168.2.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rac1:192.168.2.101              rac2:192.168.2.102              passed
  rac1:192.168.2.101              rac2:192.168.2.107              passed
  rac1:192.168.2.101              rac2:192.168.2.104              passed
  rac1:192.168.2.101              rac1:192.168.2.106              passed
  rac1:192.168.2.101              rac1:192.168.2.103              passed
  rac1:192.168.2.101              rac1:192.168.2.105              passed
Result: TCP connectivity check passed for subnet "192.168.2.0"


Check: Node connectivity for interface "eth1"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rac2[192.168.0.102]             rac1[192.168.0.101]             yes
Result: Node connectivity passed for interface "eth1"


Check: TCP connectivity of subnet "192.168.0.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rac1:192.168.0.101              rac2:192.168.0.102              passed
Result: TCP connectivity check passed for subnet "192.168.0.0"

Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "192.168.2.0".
Subnet mask consistency check passed for subnet "192.168.0.0".
Subnet mask consistency check passed.

Result: Node connectivity check passed

Checking multicast communication...

Checking subnet "192.168.2.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "192.168.2.0" for multicast communication with multicast group "230.0.1.0" passed.

Checking subnet "192.168.0.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "192.168.0.0" for multicast communication with multicast group "230.0.1.0" passed.

Check of multicast communication passed.

Checking node application existence...

Checking existence of VIP node application (required)
  Node Name     Required                  Running?                  Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          yes                       yes                       passed
  rac1          yes                       yes                       passed
VIP node application check passed

Checking existence of NETWORK node application (required)
  Node Name     Required                  Running?                  Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          yes                       yes                       passed
  rac1          yes                       yes                       passed
NETWORK node application check passed

Checking existence of GSD node application (optional)
  Node Name     Required                  Running?                  Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          no                        no                        exists
  rac1          no                        no                        exists
GSD node application is offline on nodes "rac2,rac1"

Checking existence of ONS node application (optional)
  Node Name     Required                  Running?                  Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          no                        yes                       passed
  rac1          no                        yes                       passed
ONS node application check passed


Checking Single Client Access Name (SCAN)...
  SCAN Name         Node          Running?      ListenerName  Port          Running?
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac-scan.rajasekhar.com  rac2          true          LISTENER_SCAN1  1521          true
  rac-scan.rajasekhar.com  rac1          true          LISTENER_SCAN2  1521          true
  rac-scan.rajasekhar.com  rac1          true          LISTENER_SCAN3  1521          true

Checking TCP connectivity to SCAN Listeners...
  Node          ListenerName              TCP connectivity?
  ------------  ------------------------  ------------------------
  rac1          LISTENER_SCAN1            yes
  rac1          LISTENER_SCAN2            yes
  rac1          LISTENER_SCAN3            yes
TCP connectivity to SCAN Listeners exists on all cluster nodes

Checking name resolution setup for "rac-scan.rajasekhar.com"...
  SCAN Name     IP Address                Status                    Comment
  ------------  ------------------------  ------------------------  ----------
  rac-scan.rajasekhar.com  192.168.2.107             passed
  rac-scan.rajasekhar.com  192.168.2.105             passed
  rac-scan.rajasekhar.com  192.168.2.106             passed

Verification of SCAN VIP and Listener setup passed

Checking to make sure user "oracle" is not in "root" group
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rac2          passed                    does not exist
Result: User "oracle" is not part of "root" group. Check passed

Checking if Clusterware is installed on all nodes...
Check of Clusterware install passed

Checking if CTSS Resource is running on all nodes...
Check: CTSS Resource running on all nodes
  Node Name                             Status
  ------------------------------------  ------------------------
  rac2                                  passed
Result: CTSS resource check passed


Querying CTSS for time offset on all nodes...
Result: Query of CTSS for time offset passed

Check CTSS state started...
Check: CTSS state
  Node Name                             State
  ------------------------------------  ------------------------
  rac2                                  Active
CTSS is in Active state. Proceeding with check of clock time offsets on all nodes...
Reference Time Offset Limit: 1000.0 msecs
Check: Reference Time Offset
  Node Name     Time Offset               Status
  ------------  ------------------------  ------------------------
  rac2          0.0                       passed

Time offset is within the specified limits on the following set of nodes:
"[rac2]"
Result: Check of clock time offsets passed


Oracle Cluster Time Synchronization Services check passed

Post-check for node addition was successful. <<<
[oracle@rac1 ~]$


vi) Check Cluster Nodes

[oracle@rac2 ~]$ olsnodes -n
rac1    1
rac2    2
[oracle@rac2 ~]$


vii) Check TNS Listener

On node RAC2

[oracle@rac2 ~]$ ps -ef | grep tns
root        13     2  0 Jun20 ?        00:00:00 [netns]
oracle   24168     1  0 12:57 ?        00:00:00 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER_SCAN1 -inherit
oracle   24639     1  0 12:57 ?        00:00:00 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER -inherit
oracle   28355 28292  0 13:19 pts/1    00:00:00 grep tns
[oracle@rac2 ~]$


viii) Check ASM Status

On node RAC2

[oracle@rac2 ~]$ srvctl status asm -a
ASM is running on rac2,rac1
ASM is enabled.
[oracle@rac2 ~]$


ix) Check OCR

On node RAC2

[oracle@rac2 ~]$ ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       4076
         Available space (kbytes) :     258044
         ID                       : 1037097601
         Device/File Name         :      +DATA
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check bypassed due to non-privileged user

[oracle@rac2 ~]$


x) Check Vote disk

On node RAC2

[oracle@rac2 ~]$ crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   a785661b81264f8ebfa8538128d4e1fe (/dev/oracleasm/disks/DISK1) [DATA]
 2. ONLINE   7a41922f13254f61bf4cee3f53b9aa74 (/dev/oracleasm/disks/DISK2) [DATA]
 3. ONLINE   492244b5021f4fc7bf7d75b74cfe841a (/dev/oracleasm/disks/DISK3) [DATA]
Located 3 voting disk(s).
[oracle@rac2 ~]$


3. Cluster Node Addition for RDBMS Home.


i) Run addnode.sh script

From Node 1 RAC1
As RDBMS HOME owner

[oracle@rac1 ~]$ cd /u01/app/oracle/product/11.2.0/db_1/oui/bin/    <<< This is RDBMS home

[oracle@rac1 bin]$ ./addNode.sh -silent "CLUSTER_NEW_NODES={rac2}"
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 5670 MB    Passed
Oracle Universal Installer, Version 11.2.0.3.0 Production
Copyright (C) 1999, 2011, Oracle. All rights reserved.


Performing tests to see whether nodes rac2 are available
............................................................... 100% Done.

........
-----------------------------------------------------------------------------
Cluster Node Addition Summary
Global Settings
   Source: /u01/app/oracle/product/11.2.0/db_1
   New Nodes
Space Requirements
   New Nodes
      rac2
         /u01: Required 5.04GB : Available 10.57GB
Installed Products
   Product Names
      Oracle Database 11g 11.2.0.3.0
      Sun JDK 1.5.0.30.03
      Installer SDK Component 11.2.0.3.0
      Oracle One-Off Patch Installer 11.2.0.1.7
      Oracle Universal Installer 11.2.0.3.0
      Oracle USM Deconfiguration 11.2.0.3.0
      Oracle Configuration Manager Deconfiguration 10.3.1.0.0
      Oracle DBCA Deconfiguration 11.2.0.3.0
      Oracle RAC Deconfiguration 11.2.0.3.0
      Oracle Database Deconfiguration 11.2.0.3.0
      Oracle Configuration Manager Client 10.3.2.1.0
      Oracle Configuration Manager 10.3.5.0.1
      Oracle ODBC Driverfor Instant Client 11.2.0.3.0
      LDAP Required Support Files 11.2.0.3.0
      SSL Required Support Files for InstantClient 11.2.0.3.0
      Bali Share 1.1.18.0.0
      Oracle Extended Windowing Toolkit 3.4.47.0.0
      Oracle JFC Extended Windowing Toolkit 4.2.36.0.0
      Oracle Real Application Testing 11.2.0.3.0
      Oracle Database Vault J2EE Application 11.2.0.3.0
      Oracle Label Security 11.2.0.3.0
      Oracle Data Mining RDBMS Files 11.2.0.3.0
      Oracle OLAP RDBMS Files 11.2.0.3.0
      Oracle OLAP API 11.2.0.3.0
      Platform Required Support Files 11.2.0.3.0
      Oracle Database Vault option 11.2.0.3.0
      Oracle RAC Required Support Files-HAS 11.2.0.3.0
      SQL*Plus Required Support Files 11.2.0.3.0
      Oracle Display Fonts 9.0.2.0.0
      Oracle Ice Browser 5.2.3.6.0
      Oracle JDBC Server Support Package 11.2.0.3.0
      Oracle SQL Developer 11.2.0.3.0
      Oracle Application Express 11.2.0.3.0
      XDK Required Support Files 11.2.0.3.0
      RDBMS Required Support Files for Instant Client 11.2.0.3.0
      SQLJ Runtime 11.2.0.3.0
      Database Workspace Manager 11.2.0.3.0
      RDBMS Required Support Files Runtime 11.2.0.3.0
      Oracle Globalization Support 11.2.0.3.0
      Exadata Storage Server 11.2.0.1.0
      Provisioning Advisor Framework 10.2.0.4.3
      Enterprise Manager Database Plugin -- Repository Support 11.2.0.3.0
      Enterprise Manager Repository Core Files 10.2.0.4.4
      Enterprise Manager Database Plugin -- Agent Support 11.2.0.3.0
      Enterprise Manager Grid Control Core Files 10.2.0.4.4
      Enterprise Manager Common Core Files 10.2.0.4.4
      Enterprise Manager Agent Core Files 10.2.0.4.4
      RDBMS Required Support Files 11.2.0.3.0
      regexp 2.1.9.0.0
      Agent Required Support Files 10.2.0.4.3
      Oracle 11g Warehouse Builder Required Files 11.2.0.3.0
      Oracle Notification Service (eONS) 11.2.0.3.0
      Oracle Text Required Support Files 11.2.0.3.0
      Parser Generator Required Support Files 11.2.0.3.0
      Oracle Database 11g Multimedia Files 11.2.0.3.0
      Oracle Multimedia Java Advanced Imaging 11.2.0.3.0
      Oracle Multimedia Annotator 11.2.0.3.0
      Oracle JDBC/OCI Instant Client 11.2.0.3.0
      Oracle Multimedia Locator RDBMS Files 11.2.0.3.0
      Precompiler Required Support Files 11.2.0.3.0
      Oracle Core Required Support Files 11.2.0.3.0
      Sample Schema Data 11.2.0.3.0
      Oracle Starter Database 11.2.0.3.0
      Oracle Message Gateway Common Files 11.2.0.3.0
      Oracle XML Query 11.2.0.3.0
      XML Parser for Oracle JVM 11.2.0.3.0
      Oracle Help For Java 4.2.9.0.0
      Installation Plugin Files 11.2.0.3.0
      Enterprise Manager Common Files 10.2.0.4.3
      Expat libraries 2.0.1.0.1
      Deinstallation Tool 11.2.0.3.0
      Oracle Quality of Service Management (Client) 11.2.0.3.0
      Perl Modules 5.10.0.0.1
      JAccelerator (COMPANION) 11.2.0.3.0
      Oracle Containers for Java 11.2.0.3.0
      Perl Interpreter 5.10.0.0.2
      Oracle Net Required Support Files 11.2.0.3.0
      Secure Socket Layer 11.2.0.3.0
      Oracle Universal Connection Pool 11.2.0.3.0
      Oracle JDBC/THIN Interfaces 11.2.0.3.0
      Oracle Multimedia Client Option 11.2.0.3.0
      Oracle Java Client 11.2.0.3.0
      Character Set Migration Utility 11.2.0.3.0
      Oracle Code Editor 1.2.1.0.0I
      PL/SQL Embedded Gateway 11.2.0.3.0
      OLAP SQL Scripts 11.2.0.3.0
      Database SQL Scripts 11.2.0.3.0
      Oracle Locale Builder 11.2.0.3.0
      Oracle Globalization Support 11.2.0.3.0
      SQL*Plus Files for Instant Client 11.2.0.3.0
      Required Support Files 11.2.0.3.0
      Oracle Database User Interface 2.2.13.0.0
      Oracle ODBC Driver 11.2.0.3.0
      Oracle Notification Service 11.2.0.3.0
      XML Parser for Java 11.2.0.3.0
      Oracle Security Developer Tools 11.2.0.3.0
      Oracle Wallet Manager 11.2.0.3.0
      Cluster Verification Utility Common Files 11.2.0.3.0
      Oracle Clusterware RDBMS Files 11.2.0.3.0
      Oracle UIX 2.2.24.6.0
      Enterprise Manager plugin Common Files 11.2.0.3.0
      HAS Common Files 11.2.0.3.0
      Precompiler Common Files 11.2.0.3.0
      Installation Common Files 11.2.0.3.0
      Oracle Help for the  Web 2.0.14.0.0
      Oracle LDAP administration 11.2.0.3.0
      Buildtools Common Files 11.2.0.3.0
      Assistant Common Files 11.2.0.3.0
      Oracle Recovery Manager 11.2.0.3.0
      PL/SQL 11.2.0.3.0
      Generic Connectivity Common Files 11.2.0.3.0
      Oracle Database Gateway for ODBC 11.2.0.3.0
      Oracle Programmer 11.2.0.3.0
      Oracle Database Utilities 11.2.0.3.0
      Enterprise Manager Agent 10.2.0.4.3
      SQL*Plus 11.2.0.3.0
      Oracle Netca Client 11.2.0.3.0
      Oracle Multimedia Locator 11.2.0.3.0
      Oracle Call Interface (OCI) 11.2.0.3.0
      Oracle Multimedia 11.2.0.3.0
      Oracle Net 11.2.0.3.0
      Oracle XML Development Kit 11.2.0.3.0
      Database Configuration and Upgrade Assistants 11.2.0.3.0
      Oracle JVM 11.2.0.3.0
      Oracle Advanced Security 11.2.0.3.0
      Oracle Internet Directory Client 11.2.0.3.0
      Oracle Enterprise Manager Console DB 11.2.0.3.0
      HAS Files for DB 11.2.0.3.0
      Oracle Net Listener 11.2.0.3.0
      Oracle Text 11.2.0.3.0
      Oracle Net Services 11.2.0.3.0
      Oracle Database 11g 11.2.0.3.0
      Oracle OLAP 11.2.0.3.0
      Oracle Spatial 11.2.0.3.0
      Oracle Partitioning 11.2.0.3.0
      Enterprise Edition Options 11.2.0.3.0
-----------------------------------------------------------------------------


Instantiating scripts for add node (Sunday, June 21, 2015 2:06:51 PM IST)
.                                                                 1% Done.
Instantiation of add node scripts complete

Copying to remote nodes (Sunday, June 21, 2015 2:08:47 PM IST)
...............................................................................................                                 96% Done.
Home copied to new nodes

Saving inventory on nodes (Sunday, June 21, 2015 8:10:31 PM IST)
.                                                               100% Done.
Save inventory complete
WARNING:
The following configuration scripts need to be executed as the "root" user in each new cluster node. Each script in the list below is followed by a list of nodes.
/u01/app/oracle/product/11.2.0/db_1/root.sh #On nodes rac2
To execute the configuration scripts:
    1. Open a terminal window
    2. Log in as "root"
    3. Run the scripts in each cluster node

The Cluster Node Addition of /u01/app/oracle/product/11.2.0/db_1 was successful.
Please check '/tmp/silentInstall.log' for more details.
[oracle@rac1 bin]$


ii) root.sh #On nodes rac2 from RDBMS home

[root@rac2 ~]# /u01/app/oracle/product/11.2.0/db_1/root.sh
Performing root user operation for Oracle 11g

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/oracle/product/11.2.0/db_1

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Finished product-specific root actions.
[root@rac2 ~]#


4. Add Instance to Database through Command-Line or you can add via dbca.


i) Pre-task

On RAC2
As RDBMS Home owner

[oracle@rac2 ~]$ cd /u01/app/oracle/product/11.2.0/db_1/dbs
[oracle@rac2 dbs]$ ls -ltr
total 28
-rw-r----- 1 oracle oinstall 1536 Jun 21 19:56 orapwnike1 <<
-rw-r----- 1 oracle oinstall  161 Jun 21 19:56 initDBUA5216639.ora
-rw-r----- 1 oracle oinstall   36 Jun 21 19:56 initnike1.ora << 
-rw-rw---- 1 oracle oinstall 1544 Jun 21 19:56 hc_DBUA5216639.dat 
-rw-rw---- 1 oracle oinstall 1544 Jun 21 19:56 hc_nike1.dat 
-rw-r--r-- 1 oracle oinstall 2851 Jun 21 19:56 init.ora 
-rw-r----- 1 oracle oinstall 1536 Jun 21 19:56 orapwDBUA5216639 [oracle@rac2 dbs]$ 
[oracle@rac2 dbs]$ mv initnike1.ora initnike2.ora
[oracle@rac2 dbs]$ mv orapwnike1 orapwnike2 
[oracle@rac2 dbs]$ ls -ltr 
total 28 
-rw-r----- 1 oracle oinstall 1536 Jun 21 19:56 orapwnike2 
-rw-r----- 1 oracle oinstall  161 Jun 21 19:56 initDBUA5216639.ora 
-rw-r----- 1 oracle oinstall   36 Jun 21 19:56 initnike2.ora 
-rw-rw---- 1 oracle oinstall 1544 Jun 21 19:56 hc_DBUA5216639.dat 
-rw-rw---- 1 oracle oinstall 1544 Jun 21 19:56 hc_nike1.dat 
-rw-r--r-- 1 oracle oinstall 2851 Jun 21 19:56 init.ora 
-rw-r----- 1 oracle oinstall 1536 Jun 21 19:56 orapwDBUA5216639 [oracle@rac2 dbs]$ 
[oracle@rac2 dbs]$ cat initnike2.ora 
SPFILE='+DATA1/nike/spfilenike.ora' 
[oracle@rac2 dbs]$ 
[oracle@rac2 dbs]$ echo "nike2:/u01/app/oracle/product/11.2.0/db_1:N" >> /etc/oratab
[oracle@rac2 dbs]$ echo "nike:/u01/app/oracle/product/11.2.0/db_1:N" >> /etc/oratab

cat /etc/oratab
..
#
+ASM2:/u01/app/11.2.0/grid:N            # line added by Agent
nike2:/u01/app/oracle/product/11.2.0/db_1:N
nike:/u01/app/oracle/product/11.2.0/db_1:N


[oracle@rac2 ~]$ mkdir -p /u01/app/oracle/admin/nike/adump
[oracle@rac2 ~]$ mkdir -p /u01/app/oracle/admin/nike/dpdump
[oracle@rac2 ~]$ mkdir -p /u01/app/oracle/admin/nike/hdump
[oracle@rac2 ~]$ mkdir -p /u01/app/oracle/admin/nike/pfile


ii) Add redo thread

On RAC1, As ORACLE HOME owner

SQL> set lines 180
SQL> col MEMBER for a60
SQL> select b.thread#, a.group#, a.member, b.bytes FROM v$logfile a, v$log b WHERE a.group# = b.group#;

   THREAD#     GROUP# MEMBER                                                            BYTES
---------- ---------- ------------------------------------------------------------ ----------
         1          2 +DATA1/nike/onlinelog/group_2.290.847761035                    52428800
         1          1 +DATA1/nike/onlinelog/group_1.289.847761031                    52428800

SQL>

SQL> alter database add logfile thread 2 group 3 ('+DATA1') size 52428800, group 4 ('+DATA1') size 52428800;

Database altered.

SQL>

SQL> select b.thread#, a.group#, a.member, b.bytes FROM v$logfile a, v$log b WHERE a.group# = b.group#;

   THREAD#     GROUP# MEMBER                                                            BYTES
---------- ---------- ------------------------------------------------------------ ----------
         1          2 +DATA1/nike/onlinelog/group_2.290.847761035                    52428800
         1          1 +DATA1/nike/onlinelog/group_1.289.847761031                    52428800
         2          3 +DATA1/nike/onlinelog/group_3.271.883012175                    52428800
         2          4 +DATA1/nike/onlinelog/group_4.270.883012181                    52428800

SQL>

SQL> alter database enable public thread 2;

Database altered.

SQL>


iii) Add undo tablespace

On node RAC1

SQL> set pages 0
SQL> set long 9999999
SQL> select dbms_metadata.get_ddl('TABLESPACE','UNDOTBS1') from dual;

  CREATE UNDO TABLESPACE "UNDOTBS1" DATAFILE
  SIZE 26214400
  AUTOEXTEND ON NEXT 5242880 MAXSIZE 32767M
  BLOCKSIZE 8192
  EXTENT MANAGEMENT LOCAL AUTOALLOCATE
   ALTER DATABASE DATAFILE
  '+DATA1/nike/datafile/undotbs1.357.847760839' RESIZE 41943040


SQL> create undo tablespace undotbs2 datafile '+DATA1' size 25M autoextend on next 5m maxsize 40M;

Tablespace created.

SQL>

SQL> alter system set undo_tablespace=undotbs2 scope=spfile sid='nike2';

System altered.

SQL> alter system set instance_number=2 scope=spfile sid='nike2';

System altered.

SQL> alter system set thread=2 scope=spfile sid='nike2';

System altered.

SQL>

SQL> alter system set cluster_database_instances=2 scope=spfile sid='*';

System altered.

SQL>

SQL> select inst_id,name,value from gv$parameter where name like 'undo_table%';

INST_ID NAME                 VALUE
------- -------------------- ---------------
      2 undo_tablespace      UNDOTBS2
      1 undo_tablespace      UNDOTBS1

SQL>


iv) Add instance to OCR

From node RAC1 as ORACLE_HOME owner

[oracle@rac1 ~]$ which srvctl
/u01/app/oracle/product/11.2.0/db_1/bin/srvctl   <<< You should run from RDBMS Home
[oracle@rac1 ~]$
[oracle@rac1 ~]$ srvctl add instance -d nike -i nike2 -n rac2
[oracle@rac1 ~]$
[oracle@rac1 ~]$ srvctl status database -d nike -v
Instance nike1 is running on node rac1 with online services nike_srv. Instance status: Open.
Instance nike2 is not running on node rac2 <<<
[oracle@rac1 ~]$ 
[oracle@rac1 ~]$ srvctl start instance -d nike -i nike2
[oracle@rac1 ~]$
[oracle@rac1 ~]$ srvctl status database -d nike -v
Instance nike1 is running on node rac1 with online services nike_srv. Instance status: Open.
Instance nike2 is running on node rac2. Instance status: Open. <<<
[oracle@rac1 ~]$
[oracle@rac1 ~]$ srvctl config database -d nike
Database unique name: nike
Database name: nike
Oracle home: /u01/app/oracle/product/11.2.0/db_1 <<<
Oracle user: oracle
Spfile: +DATA1/nike/spfilenike.ora
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: nike
Database instances: nike1,nike2  <<<<
Disk Groups: DATA1
Mount point paths:
Services: nike_srv <<
Type: RAC
Database is administrator managed <<< [oracle@rac1 ~]$ SQL> col host_name format a22
SQL> set lines 180
SQL> select host_name, inst_id, instance_name, status, to_char(startup_time, 'DD-MON-YYYY HH24:MI:SS') as "START_TIME" from gv$instance order by inst_id;

HOST_NAME                 INST_ID INSTANCE_NAME    STATUS       START_TIME
---------------------- ---------- ---------------- ------------ --------------------
rac1.rajasekhar.com             1 nike1            OPEN         21-JUN-2015 11:38:48
rac2.rajasekhar.com             2 nike2            OPEN         22-JUN-2015 01:25:13

SQL>


v) Add New Instance to service via srvctl or you can add via dbca

From node RAC1, as ORACLE_HOME owner

[oracle@rac1 ~]$ which srvctl
/u01/app/oracle/product/11.2.0/db_1/bin/srvctl   <<< You should run from RDBMS Home
[oracle@rac1 ~]$
[oracle@rac1 ~]$ srvctl add service -d nike -s nike_srv -a nike2 -u  <<<< -a means available instance.
[oracle@rac1 ~]$
[oracle@rac1 ~]$ srvctl config service -d nike -s nike_srv
Service name: nike_srv <<<
Service is enabled
Server pool: nike_nike_srv
Cardinality: 1
Disconnect: false
Service role: PRIMARY
Management policy: AUTOMATIC
DTP transaction: false
AQ HA notifications: false
Failover type: NONE
Failover method: NONE
TAF failover retries: 0
TAF failover delay: 0
Connection Load Balancing Goal: LONG
Runtime Load Balancing Goal: NONE
TAF policy specification: NONE
Edition:
Preferred instances: nike1
Available instances: nike2 <<<<
[oracle@rac1 ~]$

Note: If you want add instance as preferred then follow below

[oracle@rac1 ~]$ which srvctl
/u01/app/oracle/product/11.2.0/db_1/bin/srvctl   <<< You should run from RDBMS Home
[oracle@rac1 ~]$
[oracle@rac1 ~]$ srvctl add service -d nike -s nike_srv -r nike2 -u
[oracle@rac1 ~]$ srvctl config service -d nike -s nike_srv
Service name: nike_srv
Service is enabled
Server pool: nike_nike_srv
Cardinality: 2
Disconnect: false
Service role: PRIMARY
Management policy: AUTOMATIC
DTP transaction: false
AQ HA notifications: false
Failover type: NONE
Failover method: NONE
TAF failover retries: 0
TAF failover delay: 0
Connection Load Balancing Goal: LONG
Runtime Load Balancing Goal: NONE
TAF policy specification: NONE
Edition:
Preferred instances: nike2,nike1 <<<
Available instances:
[oracle@rac1 ~]$ srvctl status service -d nike
Service nike_srv is running on instance(s) nike1 <<<< 
[oracle@rac1 ~]$ srvctl start service -d nike
[oracle@rac1 ~]$ srvctl status service -d nike -v
Service nike_srv is running on instance(s) nike1,nike2 <<<<
[oracle@rac1 ~]$

Note: Modify tnsnames.ora file if required.


vi) Check the cluster stack

[oracle@rac1 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       rac1
               ONLINE  ONLINE       rac2
ora.DATA1.dg
               ONLINE  ONLINE       rac1
               ONLINE  ONLINE       rac2
ora.LISTENER.lsnr
               ONLINE  ONLINE       rac1
               ONLINE  ONLINE       rac2
ora.asm
               ONLINE  ONLINE       rac1                     Started
               ONLINE  ONLINE       rac2                     Started
ora.gsd
               OFFLINE OFFLINE      rac1
               OFFLINE OFFLINE      rac2
ora.net1.network
               ONLINE  ONLINE       rac1
               ONLINE  ONLINE       rac2
ora.ons
               ONLINE  ONLINE       rac1
               ONLINE  ONLINE       rac2
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       rac2
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       rac1
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       rac1
ora.cvu
      1        ONLINE  ONLINE       rac1
ora.nike.db
      1        ONLINE  ONLINE       rac1                     Open
      2        ONLINE  ONLINE       rac2                     Open
ora.nike.nike_srv.svc
      1        ONLINE  ONLINE       rac1
ora.oc4j
      1        ONLINE  ONLINE       rac1
ora.rac1.vip
      1        ONLINE  ONLINE       rac1
ora.rac2.vip
      1        ONLINE  ONLINE       rac2
ora.scan1.vip
      1        ONLINE  ONLINE       rac2
ora.scan2.vip
      1        ONLINE  ONLINE       rac1
ora.scan3.vip
      1        ONLINE  ONLINE       rac1
[oracle@rac1 ~]$

Caution: Your use of any information or materials on this website is entirely at your own risk. It is provided for educational purposes only. It has been tested internally, however, we do not guarantee that it will work for you. Ensure that you run it in your test environment before using.

Still page under construction !!! 🙂