Category Archives: RAC

RAC Standby 12.2

Create RAC Physical Standby Database using RMAN Active Duplicate Command

Table of Contents

___________________________________________________________________________________________________

1. Overview

2. Environment

On Primary (Step 3 to Step 9)

3. Enable Forced Logging on Primary
4. Copy Password File from Primary to standby
5. Configure Standby Redo Log on Primary
6. Verify Archive Mode Enabled on Primary
7. Set Primary Database Initialization Parameters
8. Configure LISTENER Entries on Primary
9. Configure TNS Entries on Primary

On STANDBY (Step 10 to Step 22)

10. Set Standby Database Initialization Parameters
11. Create required directories on Standby
12. Add below entry in ORATAB on Standby
13. Startup nomount
14. Configure LISTENER Entries on Standby
15. Configure TNS Entries on Standby
16. Verify TNS connectvity
17. Run the duplicate command
18. Verify Standby redo logs
19. Create spfile
20. Add init parameters for Instance 2 (DELL_DG2)
21. Add database to OCR
22. Enable MRP on Standby

Verification

23. Verify Sync

___________________________________________________________________________________________________


1. Overview

AIM:Without shutting down primary, we need to create physical standby database using RMAN DUPLICATE FROM ACTIVE DATABASE command (No need to take backup of primary database) 

Active Data Guard is a new option from Oracle Database 11g Enterprise Edition.

PLEASE NOTE in 12c Data Guard is set up at the Container level and not the individual Pluggable database level as the redo log files only belong to the Container database and the individual pluggable databases do not have their own online redo log files.

Definition of Active Dataguard:

Oracle Active Data Guard enables read-only access to a physical standby database for queries, sorting, reporting, web-based access, etc., while continuously applying changes received from the production/primary database.


2. Environment

Primary RAC cluster : rac-cluster

Platform		: Linuxx86_64
Server Name		: RAC1.RAJASEKHAR.COM,RAC2.RAJASEKHAR.COM
DB Version		: Oracle 12.2.0.1
File system             : ASM
Disk Groups 	        : +DATA,+FRA
Database Name	        : DELL
DB_UNIQUE_NAME          : DELL
INSTANCES		: DELL1,DELL2
Flashback		: Disabled
Oracle Home Path        : /u01/app/oracle/product/12.2.0/dbhome_1

Primary Cluster Status: 

[oracle@rac1 ~]$ crsctl check cluster -all
**************************************************************
rac1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
rac2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
[oracle@rac1 ~]$


[oracle@rac1 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
               ONLINE  ONLINE       rac1                     STABLE
               ONLINE  ONLINE       rac2                     STABLE
ora.DATA.dg
               ONLINE  ONLINE       rac1                     STABLE
               ONLINE  ONLINE       rac2                     STABLE
ora.FRA.dg
               ONLINE  ONLINE       rac1                     STABLE
               ONLINE  ONLINE       rac2                     STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       rac1                     STABLE
               ONLINE  ONLINE       rac2                     STABLE
ora.net1.network
               ONLINE  ONLINE       rac1                     STABLE
               ONLINE  ONLINE       rac2                     STABLE
ora.ons
               ONLINE  ONLINE       rac1                     STABLE
               ONLINE  ONLINE       rac2                     STABLE
ora.proxy_advm
               OFFLINE OFFLINE      rac1                     STABLE
               OFFLINE OFFLINE      rac2                     STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       rac2                     STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       rac1                     STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       rac1                     STABLE
ora.asm
      1        ONLINE  ONLINE       rac1                     Started,STABLE
      2        ONLINE  ONLINE       rac2                     Started,STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.cvu
      1        ONLINE  ONLINE       rac1                     STABLE
ora.dell.db
      1        ONLINE  ONLINE       rac1                     Open,HOME=/u01/app/o
                                                             racle/product/12.2.0
                                                             /dbhome_1,STABLE
      2        ONLINE  ONLINE       rac2                     Open,HOME=/u01/app/o
                                                             racle/product/12.2.0
                                                             /dbhome_1,STABLE
ora.qosmserver
      1        OFFLINE OFFLINE                               STABLE
ora.rac1.vip
      1        ONLINE  ONLINE       rac1                     STABLE
ora.rac2.vip
      1        ONLINE  ONLINE       rac2                     STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       rac2                     STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       rac1                     STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       rac1                     STABLE
--------------------------------------------------------------------------------
[oracle@rac1 ~]$ 

Standby RAC Cluster: racdg-cluster

Platform		: Linuxx86_64
Server Name		: RAC1.RAJASEKHAR.COM,RAC2.RAJASEKHAR.COM
DB Version		: Oracle 12.2.0.1
File system             : ASM
Disk Groups 	        : +DATA,+DATA_DG	
Database Name	        : DELL
DB_UNIQUE_NAME          : DELL_DG
INSTANCES		: DELL_DG1,DELL_DG2
Flashback		: Disabled
Oracle Home Path        : /u01/app/oracle/product/12.2.0/dbhome_1

Standby Cluster Status

[grid@racdg1 ~]$ crsctl check cluster -all
**************************************************************
racdg1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
racdg2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
[grid@racdg1 ~]$

[grid@racdg1 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
               ONLINE  ONLINE       racdg1                   STABLE
               ONLINE  ONLINE       racdg2                   STABLE
ora.DATA.dg
               ONLINE  ONLINE       racdg1                   STABLE
               ONLINE  ONLINE       racdg2                   STABLE
ora.DATA_DG.dg
               ONLINE  ONLINE       racdg1                   STABLE
               ONLINE  ONLINE       racdg2                   STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       racdg1                   STABLE
               ONLINE  ONLINE       racdg2                   STABLE
ora.net1.network
               ONLINE  ONLINE       racdg1                   STABLE
               ONLINE  ONLINE       racdg2                   STABLE
ora.ons
               ONLINE  ONLINE       racdg1                   STABLE
               ONLINE  ONLINE       racdg2                   STABLE
ora.proxy_advm
               OFFLINE OFFLINE      racdg1                   STABLE
               OFFLINE OFFLINE      racdg2                   STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       racdg2                   STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       racdg1                   STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       racdg1                   STABLE
ora.asm
      1        ONLINE  ONLINE       racdg1                   Started,STABLE
      2        ONLINE  ONLINE       racdg2                   Started,STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.cvu
      1        ONLINE  ONLINE       racdg1                   STABLE
ora.qosmserver
      1        OFFLINE OFFLINE                               STABLE
ora.racdg1.vip
      1        ONLINE  ONLINE       racdg1                   STABLE
ora.racdg2.vip
      1        ONLINE  ONLINE       racdg2                   STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       racdg2                   STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       racdg1                   STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       racdg1                   STABLE
--------------------------------------------------------------------------------
[grid@racdg1 ~]$

On Primary (Step 3 to Step 12)


3. Enable Forced Logging on Primary

SQL> select name, open_mode,cdb from v$database;

NAME      OPEN_MODE            CDB
--------- -------------------- ---
DELL      READ WRITE           NO

SQL> select force_logging from v$database;

FORCE_LOGGING
---------------------------------------
NO   <---------

SQL> ALTER DATABASE FORCE LOGGING;

Database altered.

SQL> select force_logging from v$database;

FORCE_LOGGING
---------------------------------------
YES  <-------

SQL>


4. Copy Password File from Primary to standby

ASMCMD> pwd
+data/dell/password
ASMCMD> pwcopy pwddell.258.1000514183 /tmp
copying +data/dell/password/pwddell.258.1000514183 -> /tmp/pwddell.258.1000514183
ASMCMD>

[root@rac2 ~]# cd /tmp
[root@rac2 tmp]# ls -ltr pwddell.258.1000514183
-rw-r-----. 1 grid oinstall 2048 Feb 20 13:16 pwddell.258.1000514183
[root@rac2 tmp]#
[root@rac2 tmp]# chown oracle:oinstall pwddell.258.1000514183

[oracle@rac2 tmp]$ ls -ltr pwddell.258.1000514183
-rw-r-----. 1 oracle oinstall 2048 Feb 20 13:16 pwddell.258.1000514183
[oracle@rac2 tmp]$

[oracle@rac2 tmp]$ scp -p pwddell.258.1000514183 oracle@racdg1:/u01/app/oracle/product/12.2.0/dbhome_1/dbs/orapwDELL_DG1
[oracle@rac2 tmp]$ scp -p pwddell.258.1000514183 oracle@racdg2:/u01/app/oracle/product/12.2.0/dbhome_1/dbs/orapwDELL_DG2


5. Configure Standby Redo Log on Primary

The standby redo logs must be the same size as the primary database online logs. The recommended number of standby redo logs is:

(maximum # of logfiles +1) * maximum # of threads

This example uses two online log files for each thread. Thus, the number of standby redo logs should be (2 + 1) * 2 = 6. 
That is, one more standby redo log file for each thread. 

-- Standy Redo logs created in the primary and RMAN will create them in standby automatically while running duplicate command.

-- Standy Redo logs files come into picture only when protection mode is Maximum Availability and Maximum Protection.


SQL> set lines 180
col MEMBER for a60
select b.thread#, a.group#, a.member, b.bytes FROM v$logfile a, v$log b WHERE a.group# = b.group#;SQL> SQL>

   THREAD#     GROUP# MEMBER                                                            BYTES
---------- ---------- ------------------------------------------------------------ ----------
         1          2 +DATA/DELL/redo02.log                                         209715200
         1          1 +DATA/DELL/redo01.log                                         209715200
         2          3 +DATA/DELL/redo03.log                                         209715200
         2          4 +DATA/DELL/redo04.log                                         209715200

SQL>

SQL> ALTER DATABASE ADD STANDBY LOGFILE THREAD 1
GROUP 5 ('+DATA/DELL/redo05.log') SIZE 200M,
GROUP 6 ('+DATA/DELL/redo06.log') SIZE 200M,
GROUP 7 ('+DATA/DELL/redo07.log') SIZE 200M;

Database altered.

SQL> ALTER DATABASE ADD STANDBY LOGFILE THREAD 2
GROUP 8 ('+DATA/DELL/redo08.log') SIZE 200M,
GROUP 9 ('+DATA/DELL/redo09.log') SIZE 200M,
GROUP 10 ('+DATA/DELL/redo10.log') SIZE 200M;

Database altered.

SQL> 


SQL> select * from v$logfile;

    GROUP# STATUS  TYPE    MEMBER                     IS_     CON_ID
---------- ------- ------- -------------------------- --- ----------
         2         ONLINE  +DATA/DELL/redo02.log      NO           0
         1         ONLINE  +DATA/DELL/redo01.log      NO           0
         3         ONLINE  +DATA/DELL/redo03.log      NO           0
         4         ONLINE  +DATA/DELL/redo04.log      NO           0
         5         STANDBY +DATA/DELL/redo05.log      NO           0
         6         STANDBY +DATA/DELL/redo06.log      NO           0
         7         STANDBY +DATA/DELL/redo07.log      NO           0
         8         STANDBY +DATA/DELL/redo08.log      NO           0
         9         STANDBY +DATA/DELL/redo09.log      NO           0
        10         STANDBY +DATA/DELL/redo10.log      NO           0

10 rows selected.

SQL>

SQL> select b.thread#,a.group#, a.member, b.bytes FROM v$logfile a, v$standby_log b WHERE a.group# = b.group#;

   THREAD#     GROUP# MEMBER                                                            BYTES
---------- ---------- ------------------------------------------------------------ ----------
         1          5 +DATA/DELL/redo05.log                                         209715200
         1          6 +DATA/DELL/redo06.log                                         209715200
         1          7 +DATA/DELL/redo07.log                                         209715200
         2          8 +DATA/DELL/redo08.log                                         209715200
         2          9 +DATA/DELL/redo09.log                                         209715200
         2         10 +DATA/DELL/redo10.log                                         209715200

6 rows selected.

SQL>


6. Verify Archive Mode Enabled on Primary

SQL> archive log list
Database log mode              Archive Mode <------
Automatic archival             Enabled
Archive destination            +FRA
Oldest online log sequence     5
Next log sequence to archive   6
Current log sequence           6
SQL>


7. Set Primary Database Initialization Parameters

SQL> create pfile='/home/oracle/initDELL.ora.bkp' from spfile;

File created.

SQL> alter system set db_unique_name='DELL' scope=spfile sid='*';

System altered.

SQL> ALTER SYSTEM SET LOG_ARCHIVE_CONFIG='DG_CONFIG=(DELL,DELL_DG)' scope=both sid='*';

System altered.

SQL> ALTER SYSTEM SET LOG_ARCHIVE_DEST_1='LOCATION=+FRA VALID_FOR=(ALL_LOGFILES,ALL_ROLES) DB_UNIQUE_NAME=DELL' scope=both sid='*';

System altered.

SQL> ALTER SYSTEM SET LOG_ARCHIVE_DEST_2='SERVICE=DELL_DG LGWR ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=DELL_DG' scope=both sid='*';

System altered.

SQL> ALTER SYSTEM SET LOG_ARCHIVE_DEST_STATE_1=ENABLE scope=both sid='*';

System altered.

SQL> ALTER SYSTEM SET LOG_ARCHIVE_DEST_STATE_2=ENABLE scope=both sid='*';

System altered.

SQL> ALTER SYSTEM SET fal_client=DELL scope=both sid='*';

System altered.

SQL>

Please note: The FAL_CLIENT database initialization parameter is no longer required from 11gR2

SQL> ALTER SYSTEM SET fal_server=DELL_DG scope=both sid='*';

System altered.

SQL> ALTER SYSTEM SET DB_FILE_NAME_CONVERT='+DATA_DG','+DATA' SCOPE=SPFILE sid='*';

System altered.

SQL> ALTER SYSTEM SET LOG_FILE_NAME_CONVERT='+DATA_DG','+DATA' SCOPE=SPFILE sid='*';

System altered.

SQL> ALTER SYSTEM SET STANDBY_FILE_MANAGEMENT=AUTO scope=both sid='*';

System altered.

SQL> create pfile='/home/oracle/initDELL.ora' from spfile;

File created.

SQL>

[oracle@rac1 ~]$ cat /home/oracle/initDELL.ora
DELL1.__data_transfer_cache_size=0
DELL2.__data_transfer_cache_size=0
DELL2.__db_cache_size=541065216
DELL1.__db_cache_size=520093696
DELL1.__inmemory_ext_roarea=0
DELL2.__inmemory_ext_roarea=0
DELL1.__inmemory_ext_rwarea=0
DELL2.__inmemory_ext_rwarea=0
DELL1.__java_pool_size=4194304
DELL2.__java_pool_size=4194304
DELL1.__large_pool_size=8388608
DELL2.__large_pool_size=8388608
DELL1.__oracle_base='/u01/app/oracle'#ORACLE_BASE set from environment
DELL2.__oracle_base='/u01/app/oracle'#ORACLE_BASE set from environment
DELL1.__pga_aggregate_target=301989888
DELL2.__pga_aggregate_target=301989888
DELL1.__sga_target=905969664
DELL2.__sga_target=905969664
DELL2.__shared_io_pool_size=37748736
DELL1.__shared_io_pool_size=37748736
DELL2.__shared_pool_size=301989888
DELL1.__shared_pool_size=322961408
DELL1.__streams_pool_size=0
DELL2.__streams_pool_size=0
*.audit_file_dest='/u01/app/oracle/admin/DELL/adump'
*.audit_trail='db'
*.cluster_database=true
*.compatible='12.2.0'
*.control_files='+DATA/DELL/control01.ctl','+DATA/DELL/control02.ctl'
*.db_block_size=8192
*.db_file_name_convert='+DATA_DG','+DATA'
*.db_name='DELL'
*.db_recovery_file_dest='+FRA'
*.db_recovery_file_dest_size=8016m
*.db_unique_name='DELL'
*.diagnostic_dest='/u01/app/oracle'
*.dispatchers='(PROTOCOL=TCP) (SERVICE=DELLXDB)'
*.fal_client='DELL'
*.fal_server='DELL_DG'
family:dw_helper.instance_mode='read-only'
DELL1.instance_number=1
DELL2.instance_number=2
*.local_listener='-oraagent-dummy-'
*.log_archive_config='DG_CONFIG=(DELL,DELL_DG)'
*.log_archive_dest_1='LOCATION=+FRA VALID_FOR=(ALL_LOGFILES,ALL_ROLES) DB_UNIQUE_NAME=DELL'
*.log_archive_dest_2='SERVICE=DELL_DG LGWR ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=DELL_DG'
*.log_archive_dest_state_1='ENABLE'
*.log_archive_dest_state_2='ENABLE'
*.log_archive_format='%t_%s_%r.dbf'
*.log_file_name_convert='+DATA_DG','+DATA'
*.nls_language='AMERICAN'
*.nls_territory='AMERICA'
*.open_cursors=300
*.pga_aggregate_target=288m
*.processes=300
*.remote_listener='rac-scan:1622'
*.remote_login_passwordfile='exclusive'
*.sga_target=864m
*.standby_file_management='AUTO'
DELL2.thread=2
DELL1.thread=1
DELL2.undo_tablespace='UNDOTBS2'
DELL1.undo_tablespace='UNDOTBS1'
[oracle@rac1 ~]$


8. Configure LISTENER Entries on Primary

[oracle@rac1 ~]$ ps -ef | grep tns
root        15     2  0 11:31 ?        00:00:00 [netns]
grid      6429     1  0 11:33 ?        00:00:03 /u01/app/12.2.0/grid/bin/tnslsnr ASMNET1LSNR_ASM -no_crs_notify -inherit
grid      6451     1  0 11:33 ?        00:00:00 /u01/app/12.2.0/grid/bin/tnslsnr LISTENER -no_crs_notify -inherit
grid      6453     1  0 11:33 ?        00:00:00 /u01/app/12.2.0/grid/bin/tnslsnr LISTENER_SCAN3 -no_crs_notify -inherit
grid      6477     1  0 11:33 ?        00:00:00 /u01/app/12.2.0/grid/bin/tnslsnr LISTENER_SCAN2 -no_crs_notify -inherit
oracle   31300 16939  0 15:07 pts/0    00:00:00 grep tns
[oracle@rac1 ~]$ lsnrctl status LISTENER

LSNRCTL for Linux: Version 12.2.0.1.0 - Production on 20-FEB-2019 15:07:47

Copyright (c) 1991, 2016, Oracle.  All rights reserved.

Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1622))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 12.2.0.1.0 - Production
Start Date                20-FEB-2019 11:33:28
Uptime                    0 days 3 hr. 34 min. 19 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/12.2.0/grid/network/admin/listener.ora
Listener Log File         /u01/app/grid/diag/tnslsnr/rac1/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.2.101)(PORT=1521)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.2.203)(PORT=1521)))
Services Summary...
Service "+ASM" has 1 instance(s).
  Instance "+ASM1", status READY, has 1 handler(s) for this service...
Service "+ASM_DATA" has 1 instance(s).
  Instance "+ASM1", status READY, has 1 handler(s) for this service...
Service "+ASM_FRA" has 1 instance(s).
  Instance "+ASM1", status READY, has 1 handler(s) for this service...
Service "DELL" has 1 instance(s).
  Instance "DELL1", status READY, has 1 handler(s) for this service...
Service "DELLXDB" has 1 instance(s).
  Instance "DELL1", status READY, has 1 handler(s) for this service...
The command completed successfully
[oracle@rac1 ~]$

[oracle@rac1 ~]$ lsnrctl status LISTENER_SCAN3

LSNRCTL for Linux: Version 12.2.0.1.0 - Production on 20-FEB-2019 15:37:28

Copyright (c) 1991, 2016, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN3)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER_SCAN3
Version                   TNSLSNR for Linux: Version 12.2.0.1.0 - Production
Start Date                20-FEB-2019 11:33:28
Uptime                    0 days 4 hr. 4 min. 0 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/12.2.0/grid/network/admin/listener.ora
Listener Log File         /u01/app/grid/diag/tnslsnr/rac1/listener_scan3/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_SCAN3)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.2.207)(PORT=1622)))
Services Summary...
Service "DELL" has 2 instance(s).
  Instance "DELL1", status READY, has 1 handler(s) for this service...
  Instance "DELL2", status READY, has 1 handler(s) for this service...
Service "DELLXDB" has 2 instance(s).
  Instance "DELL1", status READY, has 1 handler(s) for this service...
  Instance "DELL2", status READY, has 1 handler(s) for this service...
The command completed successfully
[oracle@rac1 ~]$
[oracle@rac1 ~]$ lsnrctl status LISTENER_SCAN2

LSNRCTL for Linux: Version 12.2.0.1.0 - Production on 20-FEB-2019 15:37:40

Copyright (c) 1991, 2016, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN2)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER_SCAN2
Version                   TNSLSNR for Linux: Version 12.2.0.1.0 - Production
Start Date                20-FEB-2019 11:33:29
Uptime                    0 days 4 hr. 4 min. 11 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/12.2.0/grid/network/admin/listener.ora
Listener Log File         /u01/app/grid/diag/tnslsnr/rac1/listener_scan2/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_SCAN2)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.2.206)(PORT=1622)))
Services Summary...
Service "DELL" has 2 instance(s).
  Instance "DELL1", status READY, has 1 handler(s) for this service...
  Instance "DELL2", status READY, has 1 handler(s) for this service...
Service "DELLXDB" has 2 instance(s).
  Instance "DELL1", status READY, has 1 handler(s) for this service...
  Instance "DELL2", status READY, has 1 handler(s) for this service...
The command completed successfully
[oracle@rac1 ~]$

[oracle@rac1 ~]$ cat /u01/app/12.2.0/grid/network/admin/listener.ora
LISTENER_SCAN3=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN3))))                # line added by Agent
LISTENER_SCAN2=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN2))))                # line added by Agent
# listener.ora Network Configuration File: /u01/app/12.2.0/grid/network/admin/listener.ora
# Generated by Oracle configuration tools.

ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN1 = ON

VALID_NODE_CHECKING_REGISTRATION_LISTENER_SCAN1 = OFF

VALID_NODE_CHECKING_REGISTRATION_ASMNET1LSNR_ASM = SUBNET

ASMNET1LSNR_ASM =
  (DESCRIPTION =
    (ADDRESS_LIST =
      (ADDRESS = (PROTOCOL = IPC)(KEY = ASMNET1LSNR_ASM))
    )
  )

VALID_NODE_CHECKING_REGISTRATION_LISTENER = SUBNET

LISTENER =
  (DESCRIPTION =
    (ADDRESS_LIST =
      (ADDRESS = (PROTOCOL = IPC)(KEY = LISTENER))
    )
  )

ENABLE_GLOBAL_DYNAMIC_ENDPOINT_ASMNET1LSNR_ASM = ON

ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER = ON

LISTENER_SCAN1 =
  (DESCRIPTION =
    (ADDRESS_LIST =
      (ADDRESS = (PROTOCOL = IPC)(KEY = LISTENER_SCAN1))
    )
  )

ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN2=ON                # line added by Agent
VALID_NODE_CHECKING_REGISTRATION_LISTENER_SCAN2=OFF             # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN3=ON                # line added by Agent
VALID_NODE_CHECKING_REGISTRATION_LISTENER_SCAN3=OFF             # line added by Agent
[oracle@rac1 ~]$


[grid@rac2 admin]$ cat listener.ora  <--- 2nd node of Primary 
LISTENER_SCAN3=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN3))))                # line added by Agent
LISTENER_SCAN2=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN2))))                # line added by Agent
LISTENER_SCAN1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1))))                # line added by Agent
LISTENER=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER))))            # line added by Agent
ASMNET1LSNR_ASM=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=ASMNET1LSNR_ASM))))              # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_ASMNET1LSNR_ASM=ON               # line added by Agent
VALID_NODE_CHECKING_REGISTRATION_ASMNET1LSNR_ASM=SUBNET         # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER=ON              # line added by Agent
VALID_NODE_CHECKING_REGISTRATION_LISTENER=SUBNET                # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN1=ON                # line added by Agent
VALID_NODE_CHECKING_REGISTRATION_LISTENER_SCAN1=OFF             # line added by Agent
REGISTRATION_INVITED_NODES_LISTENER_SCAN1=()            # line added by Agent
REGISTRATION_INVITED_NODES_LISTENER=()          # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN2=ON                # line added by Agent
VALID_NODE_CHECKING_REGISTRATION_LISTENER_SCAN2=OFF             # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN3=ON                # line added by Agent
VALID_NODE_CHECKING_REGISTRATION_LISTENER_SCAN3=OFF             # line added by Agent
[grid@rac2 admin]$


9. Configure TNS Entries on Primary

[oracle@rac1 admin]$ cat tnsnames.ora
# tnsnames.ora Network Configuration File: /u01/app/oracle/product/12.2.0/dbhome_1/network/admin/tnsnames.ora
# Generated by Oracle configuration tools.

DELL =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = rac-scan)(PORT = 1622))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = DELL)
    )
  )

DELL_DG =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = racdg-scan)(PORT = 1622))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = DELL_DG)(UR=A)
    )
  )

[oracle@rac1 admin]$

[oracle@rac2 admin]$ cat tnsnames.ora  <--- 2nd node of Primary
# tnsnames.ora Network Configuration File: /u01/app/oracle/product/12.2.0/dbhome_1/network/admin/tnsnames.ora
# Generated by Oracle configuration tools.

DELL =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = rac-scan)(PORT = 1622))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = DELL)
    )
  )

DELL_DG =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = racdg-scan)(PORT = 1622))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = DELL_DG)(UR=A)
    )
  )

[oracle@rac2 admin]$

[oracle@rac1 ~]$ tnsping dell

TNS Ping Utility for Linux: Version 12.2.0.1.0 - Production on 20-FEB-2019 15:24:20

Copyright (c) 1997, 2016, Oracle.  All rights reserved.

Used parameter files:
/u01/app/oracle/product/12.2.0/dbhome_1/network/admin/sqlnet.ora


Used TNSNAMES adapter to resolve the alias
Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = rac-scan)(PORT = 1622)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = DELL)))
OK (10 msec)
[oracle@rac1 ~]$
[oracle@rac1 ~]$ tnsping dell_dg

TNS Ping Utility for Linux: Version 12.2.0.1.0 - Production on 20-FEB-2019 16:26:23

Copyright (c) 1997, 2016, Oracle.  All rights reserved.

Used parameter files:
/u01/app/oracle/product/12.2.0/dbhome_1/network/admin/sqlnet.ora


Used TNSNAMES adapter to resolve the alias
Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = racdg-scan)(PORT = 1622)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = DELL_DG)(UR=A)))
OK (0 msec)
[oracle@rac1 ~]$

On STANDBY (Step 10 – Step 22)


10. Set Standby Database Initialization Parameters

[oracle@racdg1 dbs]$ cat initDELL_DG1.ora
DELL_DG1.__data_transfer_cache_size=0
DELL_DG1.__db_cache_size=520093696
DELL_DG1.__inmemory_ext_roarea=0
DELL_DG1.__inmemory_ext_rwarea=0
DELL_DG1.__java_pool_size=4194304
DELL_DG1.__large_pool_size=8388608
DELL_DG1.__oracle_base='/u01/app/oracle'#ORACLE_BASE set from environment
DELL_DG1.__pga_aggregate_target=301989888
DELL_DG1.__sga_target=905969664
DELL_DG1.__shared_io_pool_size=37748736
DELL_DG1.__shared_pool_size=322961408
DELL_DG1.__streams_pool_size=0
*.audit_file_dest='/u01/app/oracle/admin/DELL_DG/adump'
*.audit_trail='db'
*.cluster_database=false
*.compatible='12.2.0'
*.control_files='+DATA_DG/DELL_DG/control01.ctl','+DATA_DG/DELL_DG/control02.ctl'
*.db_block_size=8192
*.db_file_name_convert='+DATA/DELL','+DATA_DG/DELL_DG'
*.db_name='DELL'
*.db_recovery_file_dest='+DATA_DG'
*.db_recovery_file_dest_size=8016m
*.db_unique_name='DELL_DG'
*.diagnostic_dest='/u01/app/oracle'
*.dispatchers='(PROTOCOL=TCP) (SERVICE=DELL_DGXDB)'
*.fal_client='DELL_DG'
*.fal_server='DELL'
family:dw_helper.instance_mode='read-only'
*.instance_name='DELL_DG1'
DELL_DG1.instance_number=1
*.log_archive_config='DG_CONFIG=(DELL,DELL_DG)'
*.log_archive_dest_1='LOCATION=+DATA_DG VALID_FOR=(ALL_LOGFILES,ALL_ROLES) DB_UNIQUE_NAME=DELL_DG'
*.log_archive_dest_2='SERVICE=DELL LGWR ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=DELL'
*.log_archive_dest_state_1='ENABLE'
*.log_archive_dest_state_2='ENABLE'
*.log_archive_format='%t_%s_%r.dbf'
*.log_file_name_convert='+DATA/DELL','+DATA_DG/DELL_DG'
*.nls_language='AMERICAN'
*.nls_territory='AMERICA'
*.open_cursors=300
*.pga_aggregate_target=288m
*.processes=300
*.remote_listener='racdg-scan:1622'
*.remote_login_passwordfile='exclusive'
*.sga_target=864m
*.standby_file_management='AUTO'
DELL_DG1.thread=1
DELL_DG1.undo_tablespace='UNDOTBS1'
[oracle@racdg1 dbs]$


11. Crete required directories on Standby

[oracle@racdg1 ~]$ mkdir -p /u01/app/oracle/admin/DELL_DG/adump
[oracle@racdg2 ~]$ mkdir -p /u01/app/oracle/admin/DELL_DG/adump


12. Add below entry in ORATAB on Standby

[oracle@racdg1 ~]$ echo "DELL:/u01/app/oracle/product/12.2.0/dbhome_1:N" >> /etc/oratab
[oracle@racdg1 ~]$ echo "DELL_DG1:/u01/app/oracle/product/12.2.0/dbhome_1:N" >> /etc/oratab

[oracle@racdg2 ~]$ echo "DELL:/u01/app/oracle/product/12.2.0/dbhome_1:N" >> /etc/oratab
[oracle@racdg2 ~]$ echo "DELL_DG2:/u01/app/oracle/product/12.2.0/dbhome_1:N" >> /etc/oratab


13. Startup nomount

SQL> startup nomount pfile='/u01/app/oracle/product/12.2.0/dbhome_1/dbs/initDELL_DG1.ora';
ORACLE instance started.

Total System Global Area  905969664 bytes
Fixed Size                  8627008 bytes
Variable Size             348130496 bytes
Database Buffers          545259520 bytes
Redo Buffers                3952640 bytes
SQL> 


14. Configure LISTENER Entries on Standby

[oracle@racdg1 ~]$ ps -ef | grep tns
root        15     2  0 11:36 ?        00:00:00 [netns]
oracle    2239 31551  0 15:38 pts/0    00:00:00 grep tns
grid      6070     1  0 11:38 ?        00:00:04 /u01/app/12.2.0/grid/bin/tnslsnr ASMNET1LSNR_ASM -no_crs_notify -inherit
grid      6090     1  0 11:38 ?        00:00:00 /u01/app/12.2.0/grid/bin/tnslsnr LISTENER_SCAN3 -no_crs_notify -inherit
grid      6099     1  0 11:38 ?        00:00:00 /u01/app/12.2.0/grid/bin/tnslsnr LISTENER -no_crs_notify -inherit
grid      6122     1  0 11:38 ?        00:00:00 /u01/app/12.2.0/grid/bin/tnslsnr LISTENER_SCAN2 -no_crs_notify -inherit
[oracle@racdg1 ~]$

[grid@racdg1 ~]$ lsnrctl status LISTENER

LSNRCTL for Linux: Version 12.2.0.1.0 - Production on 23-FEB-2019 23:55:46

Copyright (c) 1991, 2016, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 12.2.0.1.0 - Production
Start Date                23-FEB-2019 12:52:44
Uptime                    0 days 11 hr. 3 min. 2 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/12.2.0/grid/network/admin/listener.ora
Listener Log File         /u01/app/grid/diag/tnslsnr/racdg1/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.2.103)(PORT=1621)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.2.105)(PORT=1621)))
Services Summary...
Service "+ASM" has 1 instance(s).
  Instance "+ASM1", status READY, has 1 handler(s) for this service...
Service "+ASM_DATA" has 1 instance(s).
  Instance "+ASM1", status READY, has 1 handler(s) for this service...
Service "+ASM_DATA_DG" has 1 instance(s).
  Instance "+ASM1", status READY, has 1 handler(s) for this service...
Service "DELL_DG" has 1 instance(s).
  Instance "DELL_DG1", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully
[grid@racdg1 ~]$


[grid@racdg1 ~]$ cat /u01/app/12.2.0/grid/network/admin/listener.ora
# listener.ora Network Configuration File: /u01/app/12.2.0/grid/network/admin/listener.ora
# Generated by Oracle configuration tools.

ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN3 = ON

ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN2 = ON

ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN1 = ON

VALID_NODE_CHECKING_REGISTRATION_LISTENER_SCAN3 = OFF

VALID_NODE_CHECKING_REGISTRATION_LISTENER_SCAN2 = OFF

SID_LIST_LISTENER =
  (SID_LIST =
    (SID_DESC =
      (GLOBAL_DBNAME = DELL_DG)
      (ORACLE_HOME = /u01/app/oracle/product/12.2.0/dbhome_1)
      (SID_NAME = DELL_DG)
    )
  )

VALID_NODE_CHECKING_REGISTRATION_LISTENER_SCAN1 = OFF

VALID_NODE_CHECKING_REGISTRATION_ASMNET1LSNR_ASM = SUBNET

ASMNET1LSNR_ASM =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = IPC)(KEY = ASMNET1LSNR_ASM))
  )

VALID_NODE_CHECKING_REGISTRATION_LISTENER = SUBNET

LISTENER =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = IPC)(KEY = LISTENER))
  )

ADR_BASE_LISTENER = /u01/app/grid

ENABLE_GLOBAL_DYNAMIC_ENDPOINT_ASMNET1LSNR_ASM = ON

ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER = ON

ADR_BASE_ASMNET1LSNR_ASM = /u01/app/grid

LISTENER_SCAN3 =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = IPC)(KEY = LISTENER_SCAN3))
  )

LISTENER_SCAN2 =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = IPC)(KEY = LISTENER_SCAN2))
  )

ADR_BASE_LISTENER_SCAN3 = /u01/app/grid

LISTENER_SCAN1 =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = IPC)(KEY = LISTENER_SCAN1))
  )

ADR_BASE_LISTENER_SCAN2 = /u01/app/grid

ADR_BASE_LISTENER_SCAN1 = /u01/app/grid

[grid@racdg1 ~]$

[grid@racdg2 admin]$ cat listener.ora  <--- 2nd of standby
LISTENER_SCAN2=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN2))))                # line added by Agent
LISTENER_SCAN3=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN3))))                # line added by Agent
LISTENER_SCAN1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1))))                # line added by Agent
# listener.ora Network Configuration File: /u01/app/12.2.0/grid/network/admin/listener.ora
# Generated by Oracle configuration tools.

VALID_NODE_CHECKING_REGISTRATION_ASMNET1LSNR_ASM = SUBNET

ASMNET1LSNR_ASM =
  (DESCRIPTION =
    (ADDRESS_LIST =
      (ADDRESS = (PROTOCOL = IPC)(KEY = ASMNET1LSNR_ASM))
    )
  )

VALID_NODE_CHECKING_REGISTRATION_LISTENER = SUBNET

LISTENER =
  (DESCRIPTION =
    (ADDRESS_LIST =
      (ADDRESS = (PROTOCOL = IPC)(KEY = LISTENER))
    )
  )

ENABLE_GLOBAL_DYNAMIC_ENDPOINT_ASMNET1LSNR_ASM = ON

ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER = ON

ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN1=ON                # line added by Agent
VALID_NODE_CHECKING_REGISTRATION_LISTENER_SCAN1=OFF             # line added by Agent
REGISTRATION_INVITED_NODES_LISTENER_SCAN1=()            # line added by Agent
REGISTRATION_INVITED_NODES_LISTENER=()          # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN3=ON                # line added by Agent
VALID_NODE_CHECKING_REGISTRATION_LISTENER_SCAN3=OFF             # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN2=ON                # line added by Agent
VALID_NODE_CHECKING_REGISTRATION_LISTENER_SCAN2=OFF             # line added by Agent
[grid@racdg2 admin]$


15. Configure TNS Entries on Standby

[oracle@racdg1 admin]$ cat tnsnames.ora
# tnsnames.ora Network Configuration File: /u01/app/oracle/product/12.2.0/dbhome_1/network/admin/tnsnames.ora
# Generated by Oracle configuration tools.

DELL =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = rac-scan)(PORT = 1622))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = DELL)
    )
  )

DELL_DG =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = racdg-scan)(PORT = 1622))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = DELL_DG) (UR=A)
    )
  )


[oracle@racdg1 admin]$

[oracle@racdg2 admin]$ cat tnsnames.ora  <--- 2nd node of Standby 
# tnsnames.ora Network Configuration File: /u01/app/oracle/product/12.2.0/dbhome_1/network/admin/tnsnames.ora
# Generated by Oracle configuration tools.

DELL =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = rac-scan)(PORT = 1622))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = DELL)
    )
  )

DELL_DG =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = racdg-scan)(PORT = 1622))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = DELL_DG) (UR=A)
    )
  )


[oracle@racdg2 admin]$


16. Verify TNS connectvity

On Primary

[oracle@rac1 ~]$ sqlplus sys/sys@dell as sysdba

SQL*Plus: Release 12.2.0.1.0 Production on Wed Feb 20 16:28:46 2019

Copyright (c) 1982, 2016, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

SQL>


[oracle@rac1 ~]$ sqlplus sys/sys@dell_dg as sysdba

SQL*Plus: Release 12.2.0.1.0 Production on Wed Feb 20 16:28:54 2019

Copyright (c) 1982, 2016, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

SQL>

On Standby

[oracle@racdg1 ~]$ sqlplus sys/sys@dell as sysdba

SQL*Plus: Release 12.2.0.1.0 Production on Wed Feb 20 16:29:28 2019

Copyright (c) 1982, 2016, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

SQL>

[oracle@racdg1 ~]$ sqlplus sys/sys@dell_dg as sysdba

SQL*Plus: Release 12.2.0.1.0 Production on Wed Feb 20 16:29:35 2019

Copyright (c) 1982, 2016, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

SQL>


17. Run the duplicate command

Please note DB_CREATE_FILE_DEST parameter cannot be set together with DB_FILE_NAME_CONVERT during RMAN active duplication.

[oracle@racdg1 ~]$ rman target sys/sys@DELL auxiliary sys/sys@DELL_DG

Recovery Manager: Release 12.2.0.1.0 - Production on Sat Feb 23 23:41:12 2019

Copyright (c) 1982, 2017, Oracle and/or its affiliates.  All rights reserved.

connected to target database: DELL (DBID=3971311101)
connected to auxiliary database: DELL (not mounted)

RMAN> duplicate target database for standby from active database nofilenamecheck;

Starting Duplicate Db at 23-FEB-19
using target database control file instead of recovery catalog
allocated channel: ORA_AUX_DISK_1
channel ORA_AUX_DISK_1: SID=50 device type=DISK

contents of Memory Script:
{
   backup as copy reuse
   targetfile  '+DATA/DELL/PASSWORD/pwddell.260.1000570117' auxiliary format
 '/u01/app/oracle/product/12.2.0/dbhome_1/dbs/orapwDELL_DG1'   ;
}
executing Memory Script

Starting backup at 23-FEB-19
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=35 instance=DELL1 device type=DISK
Finished backup at 23-FEB-19

contents of Memory Script:
{
   restore clone from service  'DELL1' standby controlfile;
}
executing Memory Script

Starting restore at 23-FEB-19
using channel ORA_AUX_DISK_1

channel ORA_AUX_DISK_1: starting datafile backup set restore
channel ORA_AUX_DISK_1: using network backup set from service DELL1
channel ORA_AUX_DISK_1: restoring control file
channel ORA_AUX_DISK_1: restore complete, elapsed time: 00:00:04
output file name=+DATA_DG/DELL_DG/control01.ctl
output file name=+DATA_DG/DELL_DG/control02.ctl
Finished restore at 23-FEB-19

contents of Memory Script:
{
   sql clone 'alter database mount standby database';
}
executing Memory Script

sql statement: alter database mount standby database

contents of Memory Script:
{
   set newname for tempfile  1 to
 "+DATA_DG/DELL_DG/temp01.dbf";
   switch clone tempfile all;
   set newname for datafile  1 to
 "+DATA_DG/DELL_DG/system01.dbf";
   set newname for datafile  3 to
 "+DATA_DG/DELL_DG/sysaux01.dbf";
   set newname for datafile  4 to
 "+DATA_DG/DELL_DG/undotbs01.dbf";
   set newname for datafile  5 to
 "+DATA_DG/DELL_DG/undotbs02.dbf";
   set newname for datafile  7 to
 "+DATA_DG/DELL_DG/users01.dbf";
   restore
   from  nonsparse   from service
 'DELL1'   clone database
   ;
   sql 'alter system archive log current';
}
executing Memory Script

executing command: SET NEWNAME

renamed tempfile 1 to +DATA_DG/DELL_DG/temp01.dbf in control file

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

Starting restore at 23-FEB-19
using channel ORA_AUX_DISK_1

channel ORA_AUX_DISK_1: starting datafile backup set restore
channel ORA_AUX_DISK_1: using network backup set from service DELL1
channel ORA_AUX_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_AUX_DISK_1: restoring datafile 00001 to +DATA_DG/DELL_DG/system01.dbf
channel ORA_AUX_DISK_1: restore complete, elapsed time: 00:00:16
channel ORA_AUX_DISK_1: starting datafile backup set restore
channel ORA_AUX_DISK_1: using network backup set from service DELL1
channel ORA_AUX_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_AUX_DISK_1: restoring datafile 00003 to +DATA_DG/DELL_DG/sysaux01.dbf
channel ORA_AUX_DISK_1: restore complete, elapsed time: 00:00:09
channel ORA_AUX_DISK_1: starting datafile backup set restore
channel ORA_AUX_DISK_1: using network backup set from service DELL1
channel ORA_AUX_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_AUX_DISK_1: restoring datafile 00004 to +DATA_DG/DELL_DG/undotbs01.dbf
channel ORA_AUX_DISK_1: restore complete, elapsed time: 00:00:04
channel ORA_AUX_DISK_1: starting datafile backup set restore
channel ORA_AUX_DISK_1: using network backup set from service DELL1
channel ORA_AUX_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_AUX_DISK_1: restoring datafile 00005 to +DATA_DG/DELL_DG/undotbs02.dbf
channel ORA_AUX_DISK_1: restore complete, elapsed time: 00:00:01
channel ORA_AUX_DISK_1: starting datafile backup set restore
channel ORA_AUX_DISK_1: using network backup set from service DELL1
channel ORA_AUX_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_AUX_DISK_1: restoring datafile 00007 to +DATA_DG/DELL_DG/users01.dbf
channel ORA_AUX_DISK_1: restore complete, elapsed time: 00:00:02
Finished restore at 23-FEB-19

sql statement: alter system archive log current

contents of Memory Script:
{
   switch clone datafile all;
}
executing Memory Script

datafile 1 switched to datafile copy
input datafile copy RECID=1 STAMP=1001029332 file name=+DATA_DG/DELL_DG/system01.dbf
datafile 3 switched to datafile copy
input datafile copy RECID=2 STAMP=1001029332 file name=+DATA_DG/DELL_DG/sysaux01.dbf
datafile 4 switched to datafile copy
input datafile copy RECID=3 STAMP=1001029332 file name=+DATA_DG/DELL_DG/undotbs01.dbf
datafile 5 switched to datafile copy
input datafile copy RECID=4 STAMP=1001029332 file name=+DATA_DG/DELL_DG/undotbs02.dbf
datafile 7 switched to datafile copy
input datafile copy RECID=5 STAMP=1001029332 file name=+DATA_DG/DELL_DG/users01.dbf
Finished Duplicate Db at 23-FEB-19

RMAN>


18. Verify Standby redo logs

SQL> select * from v$logfile;

    GROUP# STATUS  TYPE    MEMBER                                                       IS_     CON_ID
---------- ------- ------- ------------------------------------------------------------ --- ----------
         2         ONLINE  +DATA_DG/DELL_DG/redo02.log                                  NO           0
         1         ONLINE  +DATA_DG/DELL_DG/redo01.log                                  NO           0
         3         ONLINE  +DATA_DG/DELL_DG/redo03.log                                  NO           0
         4         ONLINE  +DATA_DG/DELL_DG/redo04.log                                  NO           0
         5         STANDBY +DATA_DG/DELL_DG/redo05.log                                  NO           0
         6         STANDBY +DATA_DG/DELL_DG/redo06.log                                  NO           0
         7         STANDBY +DATA_DG/DELL_DG/redo07.log                                  NO           0
         8         STANDBY +DATA_DG/DELL_DG/redo08.log                                  NO           0
         9         STANDBY +DATA_DG/DELL_DG/redo09.log                                  NO           0
        10         STANDBY +DATA_DG/DELL_DG/redo10.log                                  NO           0

10 rows selected.

SQL> select b.thread#,a.group#, a.type, a.member, b.bytes FROM v$logfile a, v$standby_log b WHERE a.group# = b.group#;

   THREAD#     GROUP# TYPE    MEMBER                                                            BYTES
---------- ---------- ------- ------------------------------------------------------------ ----------
         1          5 STANDBY +DATA_DG/DELL_DG/redo05.log                                   209715200
         1          6 STANDBY +DATA_DG/DELL_DG/redo06.log                                   209715200
         1          7 STANDBY +DATA_DG/DELL_DG/redo07.log                                   209715200
         2          8 STANDBY +DATA_DG/DELL_DG/redo08.log                                   209715200
         2          9 STANDBY +DATA_DG/DELL_DG/redo09.log                                   209715200
         2         10 STANDBY +DATA_DG/DELL_DG/redo10.log                                   209715200

6 rows selected.

SQL>


19. Create spfile

SQL> create spfile='+DATA_DG/DELL_DG/PARAMETERFILE/spfileDELL_DG.ora' from pfile='/u01/app/oracle/product/12.2.0/dbhome_1/dbs/initDELL_DG1.ora';

File created.

SQL> shut immediate;
ORA-01109: database not open


Database dismounted.
ORACLE instance shut down.
SQL> 

[oracle@racdg1 ~]$ cd $ORACLE_HOME/dbs
[oracle@racdg1 dbs]$ ls -ltr initDELL_DG1.ora
-rw-r--r--. 1 oracle oinstall 1802 Feb 23 23:40 initDELL_DG1.ora
[oracle@racdg1 dbs]$ mv initDELL_DG1.ora initDELL_DG1.ora.bkp
[oracle@racdg1 dbs]$ echo "SPFILE='+DATA_DG/DELL_DG/PARAMETERFILE/spfileDELL_DG.ora'" > initDELL_DG1.ora
[oracle@racdg1 dbs]$ scp initDELL_DG1.ora oracle@racdg2:/u01/app/oracle/product/12.2.0/dbhome_1/dbs/initDELL_DG2.ora
initDELL_DG1.ora                                             100%   58     0.1KB/s   00:00
[oracle@racdg1 dbs]$

SQL> startup mount;
ORACLE instance started.

Total System Global Area 1269366784 bytes
Fixed Size                  2252864 bytes
Variable Size             805310400 bytes
Database Buffers          452984832 bytes
Redo Buffers                8818688 bytes
Database mounted.
SQL> 


20. Add init parameters for Instance 2 (DELL_DG2)

SQL> alter system set undo_tablespace=UNDOTBS2 sid='DELL_DG2' scope=spfile;

System altered.

SQL> alter system set instance_number=1 sid='DELL_DG1' scope=spfile;

System altered.

SQL> alter system set instance_number=2 sid='DELL_DG2' scope=spfile;

System altered.

SQL> alter system set instance_name='DELL_DG1' sid='DELL_DG1' scope=spfile;

System altered.

SQL> alter system set instance_name='DELL_DG2' sid='DELL_DG2' scope=spfile;

System altered.

SQL> alter system set thread=1 sid='DELL_DG1' scope=spfile;

System altered.

SQL> alter system set thread=2 sid='DELL_DG2' scope=spfile;

System altered.

SQL> alter system set cluster_database=TRUE scope=spfile;

System altered.

SQL> alter system set remote_listener='racdg-scan:1622' scope=spfile;

System altered.

SQL> 


SQL> shut immediate;
ORA-01109: database not open


Database dismounted.
ORACLE instance shut down.
SQL>

SQL> startup mount;
ORACLE instance started.

Total System Global Area 1269366784 bytes
Fixed Size                  2252864 bytes
Variable Size             805310400 bytes
Database Buffers          452984832 bytes
Redo Buffers                8818688 bytes
Database mounted.
SQL> 

SQL> select name,open_mode,database_role,cdb from v$database;

NAME      OPEN_MODE            DATABASE_ROLE    CDB
--------- -------------------- ---------------- ---
DELL      MOUNTED              PHYSICAL STANDBY NO

SQL>


21. Add database to OCR

[oracle@racdg1 dbs]$ srvctl add database -db DELL_DG -oraclehome /u01/app/oracle/product/12.2.0/dbhome_1 -role physical_standby -startoption mount -spfile +DATA_DG/DELL_DG/PARAMETERFILE/spfileDELL_DG.ora
[oracle@racdg1 dbs]$
[oracle@racdg1 dbs]$ srvctl add instance -db DELL_DG -instance DELL_DG1 -node racdg1
[oracle@racdg1 dbs]$ srvctl add instance -db DELL_DG -instance DELL_DG2 -node racdg2
[oracle@racdg1 dbs]$
[oracle@racdg1 dbs]$ srvctl start database -d DELL_DG
[oracle@racdg1 dbs]$ srvctl status database -d DELL_DG
Instance DELL_DG1 is running on node racdg1
Instance DELL_DG2 is running on node racdg2
[oracle@racdg1 dbs]$


[grid@racdg1 trace]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
               ONLINE  ONLINE       racdg1                   STABLE
               ONLINE  ONLINE       racdg2                   STABLE
ora.DATA.dg
               ONLINE  ONLINE       racdg1                   STABLE
               ONLINE  ONLINE       racdg2                   STABLE
ora.DATA_DG.dg
               ONLINE  ONLINE       racdg1                   STABLE
               ONLINE  ONLINE       racdg2                   STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       racdg1                   STABLE
               ONLINE  ONLINE       racdg2                   STABLE
ora.net1.network
               ONLINE  ONLINE       racdg1                   STABLE
               ONLINE  ONLINE       racdg2                   STABLE
ora.ons
               ONLINE  ONLINE       racdg1                   STABLE
               ONLINE  ONLINE       racdg2                   STABLE
ora.proxy_advm
               OFFLINE OFFLINE      racdg1                   STABLE
               OFFLINE OFFLINE      racdg2                   STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       racdg2                   STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       racdg1                   STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       racdg1                   STABLE
ora.asm
      1        ONLINE  ONLINE       racdg1                   Started,STABLE
      2        ONLINE  ONLINE       racdg2                   Started,STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.cvu
      1        ONLINE  ONLINE       racdg1                   STABLE
ora.dell_dg.db
      1        ONLINE  INTERMEDIATE racdg1                   Mounted (Closed),HOM
                                                             E=/u01/app/oracle/pr
                                                             oduct/12.2.0/dbhome_
                                                             1,STABLE
      2        ONLINE  INTERMEDIATE racdg2                   Mounted (Closed),HOM
                                                             E=/u01/app/oracle/pr
                                                             oduct/12.2.0/dbhome_
                                                             1,STABLE
ora.qosmserver
      1        OFFLINE OFFLINE                               STABLE
ora.racdg1.vip
      1        ONLINE  ONLINE       racdg1                   STABLE
ora.racdg2.vip
      1        ONLINE  ONLINE       racdg2                   STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       racdg2                   STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       racdg1                   STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       racdg1                   STABLE
--------------------------------------------------------------------------------
[grid@racdg1 trace]$


22. Enable MRP on Standby

SQL> select name,open_mode,database_role,cdb from v$database;

NAME      OPEN_MODE            DATABASE_ROLE    CDB
--------- -------------------- ---------------- ---
DELL      MOUNTED              PHYSICAL STANDBY NO

SQL>

SQL> alter database recover managed standby database disconnect from session;

Database altered.

SQL>


23. Verify Sync

On Primary

SQL> select thread#,max(sequence#) from v$archived_log where archived='YES' group by thread#;

   THREAD# MAX(SEQUENCE#)
---------- --------------
         1             65  <----
         2             49  <----

SQL>

On Primary Instance 1:

SQL> alter system switch logfile;

System altered.

SQL> /

System altered.

SQL> /

System altered.

SQL>

On Primary Instance 2:


SQL> alter system switch logfile;

System altered.

SQL> /

System altered.

SQL> /

System altered.

SQL> 


SQL> select thread#,max(sequence#) from v$archived_log where archived='YES' group by thread#;

   THREAD# MAX(SEQUENCE#)
---------- --------------
         1             69  <----
         2             53  <----

SQL>

On Standby

SQL> select thread#,max(sequence#) from v$archived_log where applied='YES' group by thread#;

   THREAD# MAX(SEQUENCE#)
---------- --------------
         1             68 <------
         2             53 <------

SQL>


SQL> SELECT ARCH.THREAD# "Thread", ARCH.SEQUENCE# "Last Sequence Received", APPL.SEQUENCE# "Last Sequence Applied", (ARCH.SEQUENCE# - APPL.SEQUENCE#) "Difference" FROM (SELECT THREAD# ,SEQUENCE# FROM V$ARCHIVED_LOG WHERE (THREAD#,FIRST_TIME ) IN (SELECT THREAD#,MAX(FIRST_TIME) FROM V$ARCHIVED_LOG GROUP BY THREAD#)) ARCH,(SELECT THREAD# ,SEQUENCE# FROM V$LOG_HISTORY WHERE (THREAD#,FIRST_TIME ) IN (SELECT THREAD#,MAX(FIRST_TIME) FROM V$LOG_HISTORY GROUP BY THREAD#)) APPL WHERE ARCH.THREAD# = APPL.THREAD# ORDER BY 1;

    Thread Last Sequence Received Last Sequence Applied Difference
---------- ---------------------- --------------------- ----------
         1                     69                    69          0 <---
         2                     53                    53          0 <---

SQL>

Caution: Your use of any information or materials on this website is entirely at your own risk. It is provided for educational purposes only. It has been tested internally, however, we do not guarantee that it will work for you. Ensure that you run it in your test environment before using.

Thank you,
Rajasekhar Amudala
Email: br8dba@gmail.com
WhatsApp : +65-94693551
Linkedin: https://www.linkedin.com/in/rajasekhar-amudala/

How to take OCR backup on 11.2.0.4

How to take OCR backup on 11.2.0.4

1. Overview

11g Release 2 on wards voting disk data is automatically backed up in the OCR whenever there is a configuration change.
No need to take voting disk backup separately, OCR backup will contain both OCR and Voting data. We can restore voting disk using OCR backup.

OCR BACKUP

Automatic backups :

a) CRSD automatically creates OCR backups every 4 hours.
b) Each full day.
c) End of each week.
d) Retains the last three copies of OCR.

Manual backups:

a) can be taken using the "ocrconfig -manualbackup" command

2. Show Backup

All Backups

[root@rac1 ~]# ocrconfig -showbackup
PROT-24: Auto backups for the Oracle Cluster Registry are not available

rac1     2018/11/03 08:10:12     /u01/app/11.2.0.4/grid/cdata/rac-scan/backup_20181103_081012.ocr
[root@rac1 ~]#

Auto Backups

[root@rac1 ~]# ocrconfig -showbackup auto

Manual Backups

[root@rac1 ~]# ocrconfig -showbackup manual

rac1     2018/11/03 08:10:12     /u01/app/11.2.0.4/grid/cdata/rac-scan/backup_20181103_081012.ocr
[root@rac1 ~]#

3. Take OCR backup manually

[root@rac1 ~]# ocrconfig -manualbackup

rac1     2018/11/03 08:30:42     /u01/app/11.2.0.4/grid/cdata/rac-scan/backup_20181103_083042.ocr  <------

rac1     2018/11/03 08:10:12     /u01/app/11.2.0.4/grid/cdata/rac-scan/backup_20181103_081012.ocr
[root@rac1 ~]#

Caution: Your use of any information or materials on this website is entirely at your own risk. It is provided for educational purposes only. It has been tested internally, however, we do not guarantee that it will work for you. Ensure that you run it in your test environment before using.

Thank you,
Rajasekhar Amudala
Email: br8dba@gmail.com

root.sh failed with ORA-29783 on RAC

Issue Description:

Environment: Single node RAC on 11.2.0.4

root.sh failed with error ORA-29783 on node 1 while install RAC GI 11.2.0.4

Creation of ASM spfile in disk group failed. Following error occured: ORA-29783: GPnP attribute SET failed with error [CLSGPNP_NOT_FOUND]


Configuration of ASM ... failed
see asmca logs at /u01/app/oracle/cfgtoollogs/asmca for details
Did not succssfully configure and start ASM at /u01/app/11.2.0.4/grid/crs/install/crsconfig_lib.pm line 6912.
/u01/app/11.2.0.4/grid/perl/bin/perl -I/u01/app/11.2.0.4/grid/perl/lib -I/u01/app/11.2.0.4/grid/crs/install /u01/app/11.2.0.4/grid/crs/install/rootcrs.pl execution failed
[root@rac1 ~]#

Action Plan:

1. Disable firewall

[root@rac1 ~]# service iptables status <----
Table: filter
Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination
1    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           state RELATED,ESTABLISHED
2    ACCEPT     icmp --  0.0.0.0/0            0.0.0.0/0
3    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0
4    ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:22
5    REJECT     all  --  0.0.0.0/0            0.0.0.0/0           reject-with icmp-host-prohibited

Chain FORWARD (policy ACCEPT)
num  target     prot opt source               destination
1    REJECT     all  --  0.0.0.0/0            0.0.0.0/0           reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination

[root@rac1 ~]# 

[root@rac1 ~]# service iptables stop
iptables: Setting chains to policy ACCEPT: filter          [  OK  ]
iptables: Flushing firewall rules:                         [  OK  ]
iptables: Unloading modules:                               [  OK  ]
[root@rac1 ~]#

[root@rac1 ~]# service iptables status
iptables: Firewall is not running.
[root@rac1 ~]# 

[root@rac1 ~]# chkconfig iptables off

2. Deconfig root.sh changes

[root@rac1 ~]# /u01/app/11.2.0.4/grid/crs/install/rootcrs.pl -deconfig -force
Using configuration parameter file: /u01/app/11.2.0.4/grid/crs/install/crsconfig_params
PRCR-1119 : Failed to look up CRS resources of ora.cluster_vip_net1.type type
PRCR-1068 : Failed to query resources
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.gsd is registered
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.ons is registered
Cannot communicate with crsd

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac1'
CRS-2673: Attempting to stop 'ora.asm' on 'rac1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac1'
CRS-2677: Stop of 'ora.mdnsd' on 'rac1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac1'
CRS-2677: Stop of 'ora.cssd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rac1'
CRS-2677: Stop of 'ora.gipcd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac1'
CRS-2677: Stop of 'ora.gpnpd' on 'rac1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
Successfully deconfigured Oracle clusterware stack on this node
[root@rac1 ~]# 
[root@rac1 ~]#

3. Re-run the root.sh on node 1

[root@rac1 ~]# /u01/app/11.2.0.4/grid/root.sh <-------
Performing root user operation for Oracle 11g

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /u01/app/11.2.0.4/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/11.2.0.4/grid/crs/install/crsconfig_params
User ignored Prerequisites during installation
Installing Trace File Analyzer
OLR initialization - successful
Adding Clusterware entries to upstart
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac1'
CRS-2676: Start of 'ora.mdnsd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac1'
CRS-2676: Start of 'ora.gpnpd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1'
CRS-2672: Attempting to start 'ora.gipcd' on 'rac1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac1'
CRS-2676: Start of 'ora.diskmon' on 'rac1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac1' succeeded

ASM created and started successfully.

Disk Group DATA mounted successfully.

clscfg: -install mode specified
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4256: Updating the profile
Successful addition of voting disk 91d9f39644994f76bf7775f3a8e3929f.
Successfully replaced voting disk group with +DATA.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   91d9f39644994f76bf7775f3a8e3929f (/dev/oracleasm/disks/DISK1) [DATA]
Located 1 voting disk(s).
CRS-2672: Attempting to start 'ora.asm' on 'rac1'
CRS-2676: Start of 'ora.asm' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.DATA.dg' on 'rac1'
CRS-2676: Start of 'ora.DATA.dg' on 'rac1' succeeded
Preparing packages for installation...
cvuqdisk-1.0.9-1
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
[root@rac1 ~]#

Caution: Your use of any information or materials on this website is entirely at your own risk. It is provided for educational purposes only. It has been tested internally, however, we do not guarantee that it will work for you. Ensure that you run it in your test environment before using.

Thank you,
Rajasekhar Amudala
Email: br8dba@gmail.com

Cluster Name

/*How to display Oracle Cluster name*/

1. The command “cemutlo” provides cluster name and version.

$GI_HOME/bin/cemutlo [-n] [-w]

[oracle@rac1 ~]$ cemutlo -n
rac-scan <—– This is Cluster name.

2. $CRS_HOME/cdata/<cluster_name> directory

3. ocrdump
which will create a text file called OCRDUMPFILE open that file and look for this entry
+[SYSTEM.css.clustername]+ ORATEXT : crs_cluster In this case, “crs_cluster” is the cluster name.

4. gpnptool get
search for keyword “ClusterName

5. ASM SP File location
[root@rac1 ]# gpnptool getpval -asm_spf (or) SQL> show parameter spfile 
+DATA/<clusterName>/asmparameterfile/registry.253.783619900

Note: We cannot change the cluster name. The only to do that is to reinstall the clusterware.

 

Move/Relocate OCR

How to Move/Relocate OCR from +DATA to +VOTE diskgroup

Contents
___________________________________________________________________________________________________________________________________

1. Verify Available DiskGroups
2. Verify Current OCR location
3. Add OCR to DiskGroup +VOTE
4. Delete the old OCR location
___________________________________________________________________________________________________________________________________


1. Verify Available DiskGroups

ASMCMD> lsdg
State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  NORMAL  N         512   4096  1048576      2940     2110              980             565              0             N  DATA/
MOUNTED  EXTERN  N         512   4096  1048576      7295       13                0              13              0             N  DATA1/
MOUNTED  EXTERN  N         512   4096  1048576      1019      892                0             892              0             Y  VOTE/
ASMCMD>


2. Verify Current OCR location

[oracle@rac1 ~]$ ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       4280
         Available space (kbytes) :     257840
         ID                       : 1037097601
         Device/File Name         :      +DATA <--------
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check bypassed due to non-privileged user

[oracle@rac1 ~]$


3. Add OCR to DiskGroup +VOTE

As root user

[root@rac1 ~]# which ocrconfig
/u01/app/11.2.0/grid/bin/ocrconfig
[root@rac1 ~]# ocrconfig -add +VOTE
[root@rac1 ~]#
[root@rac1 ~]# ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       4280
         Available space (kbytes) :     257840
         ID                       : 1037097601
         Device/File Name         :      +DATA <-----------
                                    Device/File integrity check succeeded
         Device/File Name         :      +VOTE <-----------
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check succeeded

[root@rac1 ~]#


4. Delete the old OCR location

As root user

[root@rac1 ~]# ocrconfig -delete +DATA
[root@rac1 ~]#
[root@rac1 ~]# ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       4280
         Available space (kbytes) :     257840
         ID                       : 1037097601
         Device/File Name         :      +VOTE <--------
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check succeeded

[root@rac1 ~]#

Caution: Your use of any information or materials on this website is entirely at your own risk. It is provided for educational purposes only. It has been tested internally, however, we do not guarantee that it will work for you. Ensure that you run it in your test environment before using.

OSWatcher

How to Configure OSWatcher

Note: You have to follow the same steps to configure OSW in all remaining nodes in the cluster.

Contents
_________________________________________________________________________________________________________________________

1. Download OSWatcher
2. Install OSWatcher
3. Create file private.net
4. Start OS Watcher
5. Verify OSW Status
6. Stop OSWather
_________________________________________________________________________________________________________________________


Step 1: Download OSWatcher

OSWatcher (Includes: [Video]) (Doc ID 301137.1)

oswbb733.tar


Step 2: Install OSWatcher

Move the OSW tar file to target location where you want to install OSW. After this just untar

[root@rac1 share]# tar xvf oswbb733.tar
oswbb/
oswbb/docs/
oswbb/docs/The_Analyzer/
oswbb/docs/The_Analyzer/OSWatcherAnalyzerOverview.pdf
oswbb/docs/The_Analyzer/oswbbaUserGuide.pdf
oswbb/docs/The_Analyzer/oswbba_README.txt
oswbb/docs/OSWatcher/
oswbb/docs/OSWatcher/oswbb_README.txt
oswbb/docs/OSWatcher/OSWatcherUserGuide.pdf
oswbb/Exampleprivate.net
oswbb/nfssub.sh
oswbb/stopOSWbb.sh
oswbb/call_du.sh
oswbb/iosub.sh
oswbb/OSWatcherFM.sh
oswbb/ifconfigsub.sh
oswbb/ltop.sh
oswbb/mpsub.sh
oswbb/call_uptime.sh
oswbb/psmemsub.sh
oswbb/tar_up_partial_archive.sh
oswbb/oswnet.sh
oswbb/vmsub.sh
oswbb/call_sar.sh
oswbb/oswib.sh
oswbb/startOSWbb.sh
oswbb/Example_extras.txt
oswbb/oswsub.sh
oswbb/oswbba.jar
oswbb/OSWatcher.sh
oswbb/tarupfiles.sh
oswbb/xtop.sh
oswbb/src/
oswbb/src/Thumbs.db
oswbb/src/OSW_profile.htm
oswbb/src/tombody.gif
oswbb/src/missing_graphic.gif
oswbb/src/coe_logo.gif
oswbb/src/watch.gif
oswbb/src/oswbba_input.txt
oswbb/oswrds.sh
[root@rac1 share]#


3. Create file private.net for monitoring private interconnet

OS Watcher User Guide (Doc ID 301137.1)
Note: By default private interconnect statistics are not collected by OSW. You have to set it manually as per the document.If you open the OSW guide mentioned in the above document you can see the sub title 'Setting up OSW'. In that it is clearly mentioned how to set the private.net stats.document.

vi private.net     <-- add below entries and then save and exit

#Linux Example
###########################################
echo "zzz ***"`date`
traceroute -r -F rac1-priv.rajasekhar.com
traceroute -r -F rac2-priv.rajasekhar.com
############################################
rm locks/lock.file

[root@rac1 oswbb]# cat private.net
#Linux Example
###########################################
echo "zzz ***"`date`
traceroute -r -F rac1-priv.rajasekhar.com
traceroute -r -F rac2-priv.rajasekhar.com
############################################
rm locks/lock.file
[root@rac1 oswbb]#

[root@rac1 oswbb]# chown -R oracle:oinstall private.net
[root@rac1 oswbb]# chmod -R 755 private.net
[root@rac1 oswbb]# ls -ltr private.net
-rwxr-xr-x 1 oracle oinstall 228 Aug 12 02:04 private.net
[root@rac1 oswbb]#


4. Start OS Watcher

Example 1: This would start the tool and collect data at default 30 second intervals and log the last 48 hours of data to archive files.

./startOSWbb.sh 

Example 2: This would start the tool and collect data at 60 second intervals and log the last 10 hours of data to archive files and automatically compress the files.
./startOSWbb.sh 60 10 gzip

Example 3: This would start the tool and collect data at 60 second intervals and log the last 10 hours of data to archive files, compress the files and set the archive directory to a non-default location.

./startOSWbb.sh 60 10 gzip /u02/tools/oswbb/archive

Example 4: This would start the tool and collect data at 60 second intervals and log the last 48 hours of data to archive files, NOT compress the files and set the archive directory to a non-default location.

./startOSWbb.sh 60 48 NONE /u02/tools/oswbb/archive

Example 5: This would start the tool, put the process in the background, enable to the tool to continue running after the session has been terminated, collect data at 60 second intervals, and log the last 10 hours of data to archive files.

nohup ./startOSWbb.sh 60 10 &
As root user

[root@rac1 ~]# cd /u01/share/oswbb
[root@rac1 oswbb]# ls -ltr startOSWbb.sh
-rwxr-xr-x 1 oracle oinstall 2574 Feb 26 23:50 startOSWbb.sh
[root@rac1 oswbb]#
[root@rac1 oswbb]# nohup ./startOSWbb.sh 30 72 gzip & <--- Hit ENTER twice
[1] 28446
[root@rac1 oswbb]# nohup: appending output to `nohup.out'

[1]+  Done                    nohup ./startOSWbb.sh 30 72 gzip
[root@rac1 oswbb]#

Note: OSW will keep running until stop/crash and will keep data for last 72 hours only. Data automatically compress after 72 hours.


5. Verify OSW Running

[root@rac1 archive]# ps -elf | grep OSWatcher  | grep -v grep
0 S root     28450     1  0  80   0 -  2213 wait   02:48 pts/2    00:00:00 /bin/sh ./OSWatcher.sh 30 72 gzip   <-- 30 Sec, 72 Hours, output gzip format 
0 S root     28499 28450  0  80   0 -  2179 wait   02:49 pts/2    00:00:00 /bin/sh ./OSWatcherFM.sh 72 /u01/share/oswbb/archive  <-- OSW output location
[root@rac1 archive]#

[root@rac1 ~]# ls -ltr /u01/share/oswbb/archive
total 40
drwxr-xr-x 2 root root 4096 Aug 12 02:49 oswvmstat
drwxr-xr-x 2 root root 4096 Aug 12 02:49 oswtop
drwxr-xr-x 2 root root 4096 Aug 12 02:49 oswslabinfo
drwxr-xr-x 2 root root 4096 Aug 12 02:49 oswps
drwxr-xr-x 2 root root 4096 Aug 12 02:49 oswprvtnet
drwxr-xr-x 2 root root 4096 Aug 12 02:49 oswnetstat
drwxr-xr-x 2 root root 4096 Aug 12 02:49 oswmpstat
drwxr-xr-x 2 root root 4096 Aug 12 02:49 oswmeminfo
drwxr-xr-x 2 root root 4096 Aug 12 02:49 oswiostat
drwxr-xr-x 2 root root 4096 Aug 12 02:49 oswifconfig
[root@rac1 ~]#

[root@rac1 ~]# cd /u01/share/oswbb/archive/oswprvtnet
[root@rac1 oswprvtnet]# ls -ltr
total 8
-rw-r--r-- 1 root root 4272 Aug 12 02:55 rac1.rajasekhar.com_prvtnet_15.08.12.0200.dat
[root@rac1 oswprvtnet]# tail -10 rac1.rajasekhar.com_prvtnet_15.08.12.0200.dat
zzz ***Wed Aug 12 02:55:15 IST 2015
traceroute to rac1-priv.rajasekhar.com (192.168.0.101), 30 hops max, 40 byte packets
 1  rac1-priv.rajasekhar.com (192.168.0.101)  0.023 ms  0.012 ms  0.005 ms
traceroute to rac2-priv.rajasekhar.com (192.168.0.102), 30 hops max, 40 byte packets
 1  rac2-priv.rajasekhar.com (192.168.0.102)  0.278 ms  0.185 ms  0.124 ms
zzz ***Wed Aug 12 02:55:45 IST 2015
traceroute to rac1-priv.rajasekhar.com (192.168.0.101), 30 hops max, 40 byte packets
 1  rac1-priv.rajasekhar.com (192.168.0.101)  0.022 ms  0.007 ms  0.005 ms
traceroute to rac2-priv.rajasekhar.com (192.168.0.102), 30 hops max, 40 byte packets
 1  rac2-priv.rajasekhar.com (192.168.0.102)  0.396 ms  0.310 ms  0.226 ms
[root@rac1 oswprvtnet]#


6. Stop OSWather

[root@rac1 oswbb]# pwd
/u01/share/oswbb
[root@rac1 oswbb]# ls -ltr stopOSWbb.sh
-rwxr-xr-x 1 oracle oinstall 558 Apr 17  2014 stopOSWbb.sh
[root@rac1 oswbb]#
[root@rac1 oswbb]# ./stopOSWbb.sh
[root@rac1 oswbb]#
[root@rac1 oswbb]# ps -ef | grep OSW <-- now OSW is not running
root     30248  2602  0 02:57 pts/2    00:00:00 grep OSW
[root@rac1 oswbb]#

Caution: Your use of any information or materials on this website is entirely at your own risk. It is provided for educational purposes only. It has been tested internally, however, we do not guarantee that it will work for you. Ensure that you run it in your test environment before using.

Reference:
OSWatcher (Includes: [Video]) (Doc ID 301137.1)

Restore loss of all VOTE disks

Restore Loss of All Vote Disks

Contents:
_________________________________________________________________________________________________________________

0. Environment
1. Current Status of OCR/VOTE DISK
2. Backup OCR
3. Simulate VOTE DISK corruption
4. Reboot both nodes in order to see corruption << This step is not mandatory
5. Restore loss of all Voting disk
            A. Stop CRS on all the nodes
            B. Start CRS in exclusive mode only
            C. Create New Diskgroup
            D. Restore/Move/Replace Votedisk
            E. Stop CRS on Node 1
            F. Start CRS on both nodes
6. Check Cluster Status

_________________________________________________________________________________________________________________


0. Environment

Two Node RAC 11.2.0.3
OS : RHEL5


1. Current Status of OCR/VOTE DISK.

[oracle@rac1 ~]$ ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       4156
         Available space (kbytes) :     257964
         ID                       : 1037097601
         Device/File Name         :      +DATA  <<< OCR located in ASM diskgroup DATA.
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check bypassed due to non-privileged user

[oracle@rac1 ~]$

[oracle@rac1 ~]$ crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   7a14418b50a54f9dbfda2a6b97b4f620 (/dev/oracleasm/disks/DISK5) [VOTE]  <<<  voting disk /dev/oracleasm/disks/DISK5
Located 1 voting disk(s). <<<<
[oracle@rac1 ~]$

Note: Now both OCR and Voting disks are in two different diskgroups. 
      OCR in DATA diskgroup
	  Voting disk /dev/oracleasm/disks/DISK5 in VOTE diskgroup.


2. Backup OCR

[root@rac1 ~]# ocrconfig -manualbackup

rac2     2015/06/24 03:08:27     /u01/app/11.2.0/grid/cdata/rac-scan/backup_20150624_030827.ocr

rac1     2015/06/23 05:46:12     /u01/app/11.2.0/grid/cdata/rac-scan/backup_20150623_054612.ocr

rac1     2015/06/23 02:39:07     /u01/app/11.2.0/grid/cdata/rac-scan/backup_20150623_023907.ocr

rac1     2015/06/19 23:38:03     /u01/app/11.2.0/grid/cdata/rac-scan/backup_20150619_233803.ocr
[root@rac1 ~]#

Note: With OCR backup we can recover Voting Disk in case of vote disk lose.


3. Simulate VOTE DISK corruption

DISCLAIMER: The dd command given below is just for learning purposes and should only be used on testing systems. I will not take any responsibility of any consequences or loss of data caused by this command.

Corrupt the voting disk /dev/oracleasm/disks/DISK5

dd if=/dev/zero of=/dev/oracleasm/disks/DISK5 bs=4096 count=1000000

Why only 4096 bytes? because the ASM disk header is in the first block of the first AU, and the block size is 4096 bytes.

[oracle@rac1 ~]$ kfed read /dev/oracleasm/disks/DISK5 | grep kfdhdb.blksize
kfdhdb.blksize:                    4096 ; 0x0ba: 0x1000
[oracle@rac1 ~]$

[oracle@rac1 ~]$ kfed read /dev/oracleasm/disks/DISK5  <<<< KFED confirms that disk got corrupted.
kfbh.endian:                          0 ; 0x000: 0x00
kfbh.hard:                            0 ; 0x001: 0x00
kfbh.type:                            0 ; 0x002: KFBTYP_INVALID
kfbh.datfmt:                          0 ; 0x003: 0x00
kfbh.block.blk:                       0 ; 0x004: blk=0
kfbh.block.obj:                       0 ; 0x008: file=0
kfbh.check:                           0 ; 0x00c: 0x00000000
kfbh.fcn.base:                        0 ; 0x010: 0x00000000
kfbh.fcn.wrap:                        0 ; 0x014: 0x00000000
kfbh.spare1:                          0 ; 0x018: 0x00000000
kfbh.spare2:                          0 ; 0x01c: 0x00000000
7F999709F400 00000000 00000000 00000000 00000000  [................]
  Repeat 255 times
KFED-00322: Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][0]

[oracle@rac1 ~]$

[root@rac1 ~]# crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   7a14418b50a54f9dbfda2a6b97b4f620 (/dev/oracleasm/disks/DISK5) [VOTE]  <<< Don't know why still status showing as ONLINE
Located 1 voting disk(s).
[root@rac1 ~]# 

KFED read command failed. Voting disk got corrupted, i have waited around 1 hour but some how CLUSTER DID NOT WENT DOWN. Don't know why, but i am missing something here.

Please correct me if i am wrong. Let’s bring down everything in order to see the corruption.

Note: I tried to stop the CRS on both nodes at the same time, on Node 2 CRS stopped, but Node 1 restarted while shutting down CRS.

However i have rebooted both the nodes.


4. Reboot both nodes in order to see corruption

After reboot cluster status on both nodes.

From RAC1
=========
[root@rac1 ~]# crsctl stat res -t -init
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
      1        ONLINE  OFFLINE
ora.cluster_interconnect.haip
      1        ONLINE  OFFLINE
ora.crf
      1        ONLINE  ONLINE       rac1
ora.crsd
      1        ONLINE  OFFLINE
ora.cssd
      1        ONLINE  OFFLINE  <<<<<<
ora.cssdmonitor
      1        ONLINE  ONLINE       rac1
ora.ctssd
      1        ONLINE  OFFLINE
ora.diskmon
      1        OFFLINE OFFLINE
ora.drivers.acfs
      1        ONLINE  OFFLINE
ora.evmd
      1        ONLINE  OFFLINE
ora.gipcd
      1        ONLINE  ONLINE       rac1
ora.gpnpd
      1        ONLINE  ONLINE       rac1
ora.mdnsd
      1        ONLINE  ONLINE       rac1
[root@rac1 ~]#

From RAC2
===========
[root@rac2 ~]# crsctl stat res -t -init
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
      1        ONLINE  OFFLINE                               Instance Shutdown
ora.cluster_interconnect.haip
      1        ONLINE  OFFLINE
ora.crf
      1        ONLINE  ONLINE       rac2
ora.crsd
      1        ONLINE  OFFLINE
ora.cssd
      1        ONLINE  OFFLINE                               STARTING   <<<<< It will not start because "No voting files found"
ora.cssdmonitor
      1        ONLINE  ONLINE       rac2
ora.ctssd
      1        ONLINE  OFFLINE
ora.diskmon
      1        OFFLINE OFFLINE
ora.evmd
      1        ONLINE  OFFLINE
ora.gipcd
      1        ONLINE  ONLINE       rac2
ora.gpnpd
      1        ONLINE  ONLINE       rac2
ora.mdnsd
      1        ONLINE  ONLINE       rac2
[root@rac2 ~]#


alertrac1.log
==============
2015-06-25 04:25:07.002
[cssd(6313)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /u01/app/11.2.0/grid/log/rac1/cssd/ocssd.log
2015-06-25 04:25:22.291
[cssd(6313)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /u01/app/11.2.0/grid/log/rac1/cssd/ocssd.log

ocssd.log from RAC1
====================
2015-06-25 04:25:06.961: [   SKGFD][1093830976]OSS discovery with :/dev/oracleasm/disks*:
2015-06-25 04:25:06.961: [   SKGFD][1093830976]Handle 0x7fbfd8002e50 from lib :UFS:: for disk :/dev/oracleasm/disks/DISK1:
2015-06-25 04:25:06.962: [   SKGFD][1093830976]Handle 0x7fbfd80ead10 from lib :UFS:: for disk :/dev/oracleasm/disks/DISK2:
2015-06-25 04:25:06.962: [   SKGFD][1093830976]Handle 0x7fbfd80eb540 from lib :UFS:: for disk :/dev/oracleasm/disks/DISK3:
2015-06-25 04:25:06.962: [   SKGFD][1093830976]Handle 0x7fbfd80e6240 from lib :UFS:: for disk :/dev/oracleasm/disks/DISK4:
                            <<<<<<<<< DISK5 is missing.
2015-06-25 04:25:06.962: [   SKGFD][1093830976]Handle 0x7fbfd80e6a70 from lib :UFS:: for disk :/dev/oracleasm/disks/DISK6:
2015-06-25 04:25:06.963: [   SKGFD][1093830976]Handle 0x7fbfd80c7d10 from lib :UFS:: for disk :/dev/oracleasm/disks/DISK7:
..
2015-06-25 04:25:07.001: [    CSSD][1093830976]clssnmvDiskVerify: Successful discovery of 0 disks
2015-06-25 04:25:07.002: [    CSSD][1093830976]clssnmCompleteInitVFDiscovery: Completing initial voting file discovery
2015-06-25 04:25:07.002: [    CSSD][1093830976]clssnmvFindInitialConfigs: No voting files found
2015-06-25 04:25:07.002: [    CSSD][1093830976](:CSSNM00070:)clssnmCompleteInitVFDiscovery: Voting file not found. Retrying discovery in 15 seconds

alertrac2.log
==============
2015-06-25 04:25:06.999
[cssd(6539)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /u01/app/11.2.0/grid/log/rac2/cssd/ocssd.log
2015-06-25 04:25:22.279
[cssd(6539)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /u01/app/11.2.0/grid/log/rac2/cssd/ocssd.log

ocssd.log from RAC2
=====================
2015-06-25 04:25:06.573: [   SKGFD][1087797568]OSS discovery with :/dev/oracleasm/disks*:
2015-06-25 04:25:06.573: [   SKGFD][1087797568]Handle 0x19e8640 from lib :UFS:: for disk :/dev/oracleasm/disks/DISK1:
2015-06-25 04:25:06.573: [   SKGFD][1087797568]Handle 0x1993310 from lib :UFS:: for disk :/dev/oracleasm/disks/DISK2:
2015-06-25 04:25:06.574: [   SKGFD][1087797568]Handle 0x1a49550 from lib :UFS:: for disk :/dev/oracleasm/disks/DISK3:
2015-06-25 04:25:06.574: [   SKGFD][1087797568]Handle 0x18aaa40 from lib :UFS:: for disk :/dev/oracleasm/disks/DISK4:
                                                                          <<<<<<<< DISK5 is missing.
2015-06-25 04:25:06.575: [   SKGFD][1087797568]Handle 0x19f6e90 from lib :UFS:: for disk :/dev/oracleasm/disks/DISK6:
2015-06-25 04:25:06.575: [   SKGFD][1087797568]Handle 0x196cbf0 from lib :UFS:: for disk :/dev/oracleasm/disks/DISK7:
..
2015-06-25 04:25:06.999: [    CSSD][1087797568]clssnmvDiskVerify: Successful discovery of 0 disks <<<
2015-06-25 04:25:06.999: [    CSSD][1087797568]clssnmCompleteInitVFDiscovery: Completing initial voting file discovery
2015-06-25 04:25:06.999: [    CSSD][1087797568]clssnmvFindInitialConfigs: No voting files found <<<
2015-06-25 04:25:07.000: [    CSSD][1087797568](:CSSNM00070:)clssnmCompleteInitVFDiscovery: Voting file not found. Retrying discovery in 15 seconds


5. Restore loss of all Voting disk.


A. Stop CRS on all the nodes

From RAC1
==========
[root@rac1 ~]# crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac1'
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'rac1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac1'
CRS-2673: Attempting to stop 'ora.crf' on 'rac1'
CRS-2677: Stop of 'ora.cssdmonitor' on 'rac1' succeeded
CRS-2677: Stop of 'ora.crf' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rac1'
CRS-2677: Stop of 'ora.mdnsd' on 'rac1' succeeded
CRS-2677: Stop of 'ora.gipcd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac1'
CRS-2677: Stop of 'ora.gpnpd' on 'rac1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@rac1 ~]#

From RAC2
===========
[root@rac2 ~]# crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac2'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac2'
CRS-2673: Attempting to stop 'ora.crf' on 'rac2'
CRS-2677: Stop of 'ora.crf' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rac2'
CRS-2677: Stop of 'ora.mdnsd' on 'rac2' succeeded
CRS-2677: Stop of 'ora.gipcd' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac2'
CRS-2677: Stop of 'ora.gpnpd' on 'rac2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@rac2 ~]#


B. Start CRS in exclusive mode only

From RAC1 as root user

Note: From 11.2.0.2 onwards we should include flag “nocrs” in exclusive CRS startup

[root@rac1 ~]# crsctl start crs -excl -nocrs
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac1'
CRS-2676: Start of 'ora.mdnsd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac1'
CRS-2676: Start of 'ora.gpnpd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1'
CRS-2672: Attempting to start 'ora.gipcd' on 'rac1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac1'
CRS-2676: Start of 'ora.diskmon' on 'rac1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'rac1'
CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'rac1'
CRS-2672: Attempting to start 'ora.ctssd' on 'rac1'
CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rac1'
CRS-2674: Start of 'ora.drivers.acfs' on 'rac1' failed
CRS-2676: Start of 'ora.ctssd' on 'rac1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rac1' succeeded
CRS-2679: Attempting to clean 'ora.asm' on 'rac1'
CRS-2681: Clean of 'ora.asm' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rac1'
CRS-2676: Start of 'ora.asm' on 'rac1' succeeded
[root@rac1 ~]#

[oracle@rac1 ~]$ crsctl stat res -t -init
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
      1        ONLINE  ONLINE       rac1                     Started  <<
ora.cluster_interconnect.haip
      1        ONLINE  ONLINE       rac1
ora.crf
      1        OFFLINE OFFLINE
ora.crsd
      1        OFFLINE OFFLINE  <<<< We have started CRS exclusive mode then CSSD and ASM started, but CRSD won't start
ora.cssd
      1        ONLINE  ONLINE       rac1   <<<
ora.cssdmonitor
      1        ONLINE  ONLINE       rac1
ora.ctssd
      1        ONLINE  ONLINE       rac1                     ACTIVE:0
ora.diskmon
      1        OFFLINE OFFLINE
ora.drivers.acfs
      1        ONLINE  OFFLINE
ora.evmd
      1        OFFLINE OFFLINE
ora.gipcd
      1        ONLINE  ONLINE       rac1
ora.gpnpd
      1        ONLINE  ONLINE       rac1
ora.mdnsd
      1        ONLINE  ONLINE       rac1
[oracle@rac1 ~]$


[oracle@rac1 ~]$ SQL> select NAME, STATE, VOTING_FILES from v$asm_diskgroup;

NAME                           STATE       V
------------------------------ ----------- -
DATA1                          MOUNTED     N
DATA                           MOUNTED     N
                               <<<<< VOTE Diskgroup is missing in this output. 
SQL>

SQL> select NAME, PATH, STATE, VOTING_FILE from v$asm_disk where PATH='/dev/oracleasm/disks/DISK5';

no rows selected  << no output SQL>

[oracle@rac1 ~]$ crsctl query css votedisk
Located 0 voting disk(s). <<<
[oracle@rac1 ~]$


C. Create New Diskgroup

Note: You don’t  have new disk right now, but want to resolve this issue, then use existing ASM diskgroup to restore Voting disk. In this case you can ignore this step “Create New Diskgroup”.

SQL> create diskgroup DATA2 external redundancy disk '/dev/oracleasm/disks/DISK6' attribute 'COMPATIBLE.ASM' = '11.2';

Diskgroup created.

SQL>


D. Restore/Move/Replace Votedisk.

Note: Voting Disk will be restore from OCR backup.

From Node 1 as GI HOME owner

[oracle@rac1 ~]$ crsctl replace votedisk +DATA2
Successful addition of voting disk 7ebe19bb115e4f51bfd96935eb1b92b7.
Successfully replaced voting disk group with +DATA2.
CRS-4266: Voting file(s) successfully replaced <<<
[oracle@rac1 ~]$
[oracle@rac1 ~]$ crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   7ebe19bb115e4f51bfd96935eb1b92b7 (/dev/oracleasm/disks/DISK6) [DATA2] <<<
Located 1 voting disk(s).
[oracle@rac1 ~]$


E. Stop CRS on Node 1

From RAC1
As root user

[root@rac1 ~]# crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac1'
CRS-2673: Attempting to stop 'ora.asm' on 'rac1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac1'
CRS-2677: Stop of 'ora.mdnsd' on 'rac1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac1'
CRS-2677: Stop of 'ora.ctssd' on 'rac1' succeeded
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac1'
CRS-2677: Stop of 'ora.cssd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rac1'
CRS-2677: Stop of 'ora.gipcd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac1'
CRS-2677: Stop of 'ora.gpnpd' on 'rac1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@rac1 ~]#


F. Start CRS on both nodes.

From RAC1
As root

[root@rac1 ~]# crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root@rac1 ~]#
[root@rac1 ~]#

From RAC2
As root

[root@rac2 ~]# crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root@rac2 ~]#


6. Check Cluster Status

[root@rac1 ~]# crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online  <<<
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
[root@rac1 ~]#

[root@rac1 ~]# crsctl stat res -t -init
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
      1        ONLINE  ONLINE       rac1                     Started
ora.cluster_interconnect.haip
      1        ONLINE  ONLINE       rac1
ora.crf
      1        ONLINE  ONLINE       rac1
ora.crsd
      1        ONLINE  ONLINE       rac1  <<<
ora.cssd
      1        ONLINE  ONLINE       rac1
ora.cssdmonitor
      1        ONLINE  ONLINE       rac1
ora.ctssd
      1        ONLINE  ONLINE       rac1                     ACTIVE:0
ora.diskmon
      1        OFFLINE OFFLINE
ora.drivers.acfs
      1        ONLINE  OFFLINE
ora.evmd
      1        ONLINE  ONLINE       rac1
ora.gipcd
      1        ONLINE  ONLINE       rac1
ora.gpnpd
      1        ONLINE  ONLINE       rac1
ora.mdnsd
      1        ONLINE  ONLINE       rac1
[root@rac1 ~]#

[root@rac2 ~]# crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online  <<<
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
[root@rac2 ~]#

[root@rac2 ~]# crsctl stat res -t -init
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
      1        ONLINE  ONLINE       rac2                     Started
ora.cluster_interconnect.haip
      1        ONLINE  ONLINE       rac2
ora.crf
      1        ONLINE  ONLINE       rac2
ora.crsd
      1        ONLINE  ONLINE       rac2  <<<
ora.cssd
      1        ONLINE  ONLINE       rac2
ora.cssdmonitor
      1        ONLINE  ONLINE       rac2
ora.ctssd
      1        ONLINE  ONLINE       rac2                     ACTIVE:0
ora.diskmon
      1        OFFLINE OFFLINE
ora.evmd
      1        ONLINE  ONLINE       rac2
ora.gipcd
      1        ONLINE  ONLINE       rac2
ora.gpnpd
      1        ONLINE  ONLINE       rac2
ora.mdnsd
      1        ONLINE  ONLINE       rac2
[root@rac2 ~]#

Caution: Your use of any information or materials on this website is entirely at your own risk. It is provided for educational purposes only. It has been tested internally, however, we do not guarantee that it will work for you. Ensure that you run it in your test environment before using.

Still page under construction !!! 🙂

Add Node Back which was DELETED without software

Add Node Back to Cluster which was deleted without removing GI and RDBMS binaries.

0. Environment
1. Backup OCR
2. Cluster Node Addition
3. Run root.sh on new node RAC2
4. Check cluster status
5. Pin Node
6. Add instance to OCR
7. Update Inventory


0. Environment

One node RAC 11.2.0.3 (Not RAC ONE), single node RAC. Earlier it was two node RAC setup, recently i have deleted 2nd node from cluster for testing.
Node name: RAC1
OS: RHEL 5
DATABASE: nike, Instance: nike1

Task: We are going to add node “RAC2″ to our existing cluster.

Current Status

[root@rac1 ~]# crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       rac1
ora.DATA1.dg
               ONLINE  ONLINE       rac1
ora.LISTENER.lsnr
               ONLINE  ONLINE       rac1
ora.asm
               ONLINE  ONLINE       rac1                     Started
ora.gsd
               OFFLINE OFFLINE      rac1
ora.net1.network
               ONLINE  ONLINE       rac1
ora.ons
               ONLINE  ONLINE       rac1
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       rac1
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       rac1
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       rac1
ora.cvu
      1        ONLINE  ONLINE       rac1
ora.nike.db
      1        ONLINE  ONLINE       rac1                     Open
ora.nike.nike_srv.svc
      1        ONLINE  ONLINE       rac1
ora.oc4j
      1        ONLINE  ONLINE       rac1
ora.rac1.vip
      1        ONLINE  ONLINE       rac1
ora.scan1.vip
      1        ONLINE  ONLINE       rac1
ora.scan2.vip
      1        ONLINE  ONLINE       rac1
ora.scan3.vip
      1        ONLINE  ONLINE       rac1
[root@rac1 ~]#


1. Backup OCR

[root@rac1 ~]# ocrconfig -manualbackup

rac1     2015/06/23 05:46:12     /u01/app/11.2.0/grid/cdata/rac-scan/backup_20150623_054612.ocr

rac1     2015/06/23 02:39:07     /u01/app/11.2.0/grid/cdata/rac-scan/backup_20150623_023907.ocr

rac1     2015/06/19 23:38:03     /u01/app/11.2.0/grid/cdata/rac-scan/backup_20150619_233803.ocr
[root@rac1 ~]#


2. Cluster Node Addition.

Note: Pre-node check failed. Hence need to set the environment variable IGNORE_PREADDNODE_CHECKS=Y before running addNode.sh otherwise, the silent node addition will fail without showing any errors to the console.

As GI Home owner
From active node RAC1

[oracle@rac1 ~]$ cd /u01/app/11.2.0/grid/oui/bin/
[oracle@rac1 bin]$ IGNORE_PREADDNODE_CHECKS=Y
[oracle@rac1 bin]$ export IGNORE_PREADDNODE_CHECKS
[oracle@rac1 bin]$ ./addNode.sh -silent -noCopy "CLUSTER_NEW_NODES={rac2}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={rac2-vip}"
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 5671 MB    Passed
Oracle Universal Installer, Version 11.2.0.3.0 Production
Copyright (C) 1999, 2011, Oracle. All rights reserved.


Performing tests to see whether nodes rac2 are available
............................................................... 100% Done.

.
-----------------------------------------------------------------------------
Cluster Node Addition Summary
Global Settings
   Source: /u01/app/11.2.0/grid
   New Nodes
Space Requirements
   New Nodes
      rac2
         /u01: Required 7.50GB : Available 5.54GB
Installed Products
   Product Names
      Oracle Grid Infrastructure 11.2.0.3.0
      Sun JDK 1.5.0.30.03
      Installer SDK Component 11.2.0.3.0
      Oracle One-Off Patch Installer 11.2.0.1.7
      Oracle Universal Installer 11.2.0.3.0
      Oracle USM Deconfiguration 11.2.0.3.0
      Oracle Configuration Manager Deconfiguration 10.3.1.0.0
      Enterprise Manager Common Core Files 10.2.0.4.4
      Oracle DBCA Deconfiguration 11.2.0.3.0
      Oracle RAC Deconfiguration 11.2.0.3.0
      Oracle Quality of Service Management (Server) 11.2.0.3.0
      Installation Plugin Files 11.2.0.3.0
      Universal Storage Manager Files 11.2.0.3.0
      Oracle Text Required Support Files 11.2.0.3.0
      Automatic Storage Management Assistant 11.2.0.3.0
      Oracle Database 11g Multimedia Files 11.2.0.3.0
      Oracle Multimedia Java Advanced Imaging 11.2.0.3.0
      Oracle Globalization Support 11.2.0.3.0
      Oracle Multimedia Locator RDBMS Files 11.2.0.3.0
      Oracle Core Required Support Files 11.2.0.3.0
      Bali Share 1.1.18.0.0
      Oracle Database Deconfiguration 11.2.0.3.0
      Oracle Quality of Service Management (Client) 11.2.0.3.0
      Expat libraries 2.0.1.0.1
      Oracle Containers for Java 11.2.0.3.0
      Perl Modules 5.10.0.0.1
      Secure Socket Layer 11.2.0.3.0
      Oracle JDBC/OCI Instant Client 11.2.0.3.0
      Oracle Multimedia Client Option 11.2.0.3.0
      LDAP Required Support Files 11.2.0.3.0
      Character Set Migration Utility 11.2.0.3.0
      Perl Interpreter 5.10.0.0.2
      PL/SQL Embedded Gateway 11.2.0.3.0
      OLAP SQL Scripts 11.2.0.3.0
      Database SQL Scripts 11.2.0.3.0
      Oracle Extended Windowing Toolkit 3.4.47.0.0
      SSL Required Support Files for InstantClient 11.2.0.3.0
      SQL*Plus Files for Instant Client 11.2.0.3.0
      Oracle Net Required Support Files 11.2.0.3.0
      Oracle Database User Interface 2.2.13.0.0
      RDBMS Required Support Files for Instant Client 11.2.0.3.0
      RDBMS Required Support Files Runtime 11.2.0.3.0
      XML Parser for Java 11.2.0.3.0
      Oracle Security Developer Tools 11.2.0.3.0
      Oracle Wallet Manager 11.2.0.3.0
      Enterprise Manager plugin Common Files 11.2.0.3.0
      Platform Required Support Files 11.2.0.3.0
      Oracle JFC Extended Windowing Toolkit 4.2.36.0.0
      RDBMS Required Support Files 11.2.0.3.0
      Oracle Ice Browser 5.2.3.6.0
      Oracle Help For Java 4.2.9.0.0
      Enterprise Manager Common Files 10.2.0.4.3
      Deinstallation Tool 11.2.0.3.0
      Oracle Java Client 11.2.0.3.0
      Cluster Verification Utility Files 11.2.0.3.0
      Oracle Notification Service (eONS) 11.2.0.3.0
      Oracle LDAP administration 11.2.0.3.0
      Cluster Verification Utility Common Files 11.2.0.3.0
      Oracle Clusterware RDBMS Files 11.2.0.3.0
      Oracle Locale Builder 11.2.0.3.0
      Oracle Globalization Support 11.2.0.3.0
      Buildtools Common Files 11.2.0.3.0
      Oracle RAC Required Support Files-HAS 11.2.0.3.0
      SQL*Plus Required Support Files 11.2.0.3.0
      XDK Required Support Files 11.2.0.3.0
      Agent Required Support Files 10.2.0.4.3
      Parser Generator Required Support Files 11.2.0.3.0
      Precompiler Required Support Files 11.2.0.3.0
      Installation Common Files 11.2.0.3.0
      Required Support Files 11.2.0.3.0
      Oracle JDBC/THIN Interfaces 11.2.0.3.0
      Oracle Multimedia Locator 11.2.0.3.0
      Oracle Multimedia 11.2.0.3.0
      HAS Common Files 11.2.0.3.0
      Assistant Common Files 11.2.0.3.0
      PL/SQL 11.2.0.3.0
      HAS Files for DB 11.2.0.3.0
      Oracle Recovery Manager 11.2.0.3.0
      Oracle Database Utilities 11.2.0.3.0
      Oracle Notification Service 11.2.0.3.0
      SQL*Plus 11.2.0.3.0
      Oracle Netca Client 11.2.0.3.0
      Oracle Net 11.2.0.3.0
      Oracle JVM 11.2.0.3.0
      Oracle Internet Directory Client 11.2.0.3.0
      Oracle Net Listener 11.2.0.3.0
      Cluster Ready Services Files 11.2.0.3.0
      Oracle Database 11g 11.2.0.3.0
-----------------------------------------------------------------------------


Instantiating scripts for add node (Tuesday, June 23, 2015 6:01:55 AM IST)
.                                                                 1% Done.
Instantiation of add node scripts complete

Saving inventory on nodes (Tuesday, June 23, 2015 6:03:32 AM IST)
.                                                               100% Done.
Save inventory complete
WARNING:
The following configuration scripts need to be executed as the "root" user in each new cluster node. Each script in the list below is followed by a list of nodes.
/u01/app/11.2.0/grid/root.sh #On nodes rac2
To execute the configuration scripts:
    1. Open a terminal window
    2. Log in as "root"
    3. Run the scripts in each cluster node

The Cluster Node Addition of /u01/app/11.2.0/grid was successful.
Please check '/tmp/silentInstall.log' for more details.
[oracle@rac1 bin]$

Note: set environment properly, better set ORACLE_HOME tag for above script like below

./addNode.sh -silent -noCopy ORACLE_HOME=/u01/app/11.2.0/grid “CLUSTER_NEW_NODES={rac2}” “CLUSTER_NEW_VIRTUAL_HOSTNAMES={rac2-vip}”


3. Run root.sh on new node RAC2

Form Node RAC2
As root

[root@rac2 ~]# /u01/app/11.2.0/grid/root.sh
Performing root user operation for Oracle 11g

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
User ignored Prerequisites during installation
OLR initialization - successful
Adding Clusterware entries to inittab
CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node rac1, number 1, and is terminating
An active cluster was found during exclusive startup, restarting to join the cluster
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 11g Release 2.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Preparing packages for installation...
cvuqdisk-1.0.9-1
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
[root@rac2 ~]#


4. Check cluster status

[oracle@rac1 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       rac1
               ONLINE  ONLINE       rac2
ora.DATA1.dg
               ONLINE  ONLINE       rac1
               ONLINE  ONLINE       rac2
ora.LISTENER.lsnr
               ONLINE  ONLINE       rac1
               ONLINE  ONLINE       rac2
ora.asm
               ONLINE  ONLINE       rac1                     Started
               ONLINE  ONLINE       rac2                     Started
ora.gsd
               OFFLINE OFFLINE      rac1
               OFFLINE OFFLINE      rac2
ora.net1.network
               ONLINE  ONLINE       rac1
               ONLINE  ONLINE       rac2
ora.ons
               ONLINE  ONLINE       rac1
               ONLINE  ONLINE       rac2
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       rac2
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       rac1
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       rac1
ora.cvu
      1        ONLINE  ONLINE       rac1
ora.nike.db
      1        ONLINE  ONLINE       rac1                     Open
ora.nike.nike_srv.svc
      1        ONLINE  ONLINE       rac1
ora.oc4j
      1        ONLINE  ONLINE       rac1
ora.rac1.vip
      1        ONLINE  ONLINE       rac1
ora.rac2.vip
      1        ONLINE  ONLINE       rac2
ora.scan1.vip
      1        ONLINE  ONLINE       rac2
ora.scan2.vip
      1        ONLINE  ONLINE       rac1
ora.scan3.vip
      1        ONLINE  ONLINE       rac1
[oracle@rac1 ~]$


5. Pin Node

As root from node RAC1

[root@rac1 ~]# olsnodes -s -t
rac1    Active  Pinned
rac2    Active  Unpinned
[root@rac1 ~]#
[root@rac1 ~]# crsctl pin css -n rac2
CRS-4664: Node rac2 successfully pinned.
[root@rac1 ~]#
[root@rac1 ~]# olsnodes -s -t
rac1    Active  Pinned
rac2    Active  Pinned <<<<
[root@rac1 ~]#


6. Add instance to OCR

As ORACLE HOME owner

[oracle@rac1 ~]$ which srvctl
/u01/app/oracle/product/11.2.0/db_1/bin/srvctl << you should run from ORACLE_HOME
[oracle@rac1 ~]$ 
[oracle@rac1 ~]$ srvctl status database -d nike -v
Instance nike1 is running on node rac1 with online services nike_srv. Instance status: Open.
[oracle@rac1 ~]$ 
[oracle@rac1 ~]$ srvctl add instance -d nike -i nike2 -n rac2
[oracle@rac1 ~]$
[oracle@rac1 ~]$ srvctl status database -d nike -v
Instance nike1 is running on node rac1 with online services nike_srv. Instance status: Open.
Instance nike2 is not running on node rac2 <<<< Instance added, we need start manually
[oracle@rac1 ~]$
[oracle@rac1 ~]$ srvctl start instance -d nike -i nike2
[oracle@rac1 ~]$
[oracle@rac1 ~]$ srvctl status database -d nike -v
Instance nike1 is running on node rac1 with online services nike_srv. Instance status: Open.
Instance nike2 is running on node rac2. Instance status: Open. <<< Now it is running
[oracle@rac1 ~]$


7. Update Inventory

Note: addNode script will automatically update NODE_LIST for GI Home.

<HOME NAME="Ora11g_gridinfrahome1" LOC="/u01/app/11.2.0/grid" TYPE="O" IDX="1" CRS="true">
   <NODE_LIST>
      <NODE NAME="rac1"/>
      <NODE NAME="rac2"/>  <<< automatically updated NODE_LIST by addnode script.
   </NODE_LIST>
</HOME>
<HOME NAME="OraDb11g_home1" LOC="/u01/app/oracle/product/11.2.0/db_1" TYPE="O" IDX="2">
   <NODE_LIST>
      <NODE NAME="rac1"/>
	                                      <<<< Need to update NODE_LIST manually for ORACLE_HOME
</NODE_LIST>           
</HOME>

[oracle@rac1 ~]$ cd /u01/app/oracle/product/11.2.0/db_1/oui/bin
[oracle@rac1 bin]$ ./runInstaller -updateNodeList ORACLE_HOME=/u01/app/oracle/product/11.2.0/db_1 "CLUSTER_NODES={rac1,rac2}"
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 5671 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /u01/app/oraInventory
'UpdateNodeList' was successful.
[oracle@rac1 bin]$

<HOME NAME="Ora11g_gridinfrahome1" LOC="/u01/app/11.2.0/grid" TYPE="O" IDX="1" CRS="true">
   <NODE_LIST>
      <NODE NAME="rac1"/>
      <NODE NAME="rac2"/>
   </NODE_LIST>
</HOME>
<HOME NAME="OraDb11g_home1" LOC="/u01/app/oracle/product/11.2.0/db_1" TYPE="O" IDX="2">
   <NODE_LIST>
      <NODE NAME="rac1"/>
      <NODE NAME="rac2"/>  << updated node list.
   </NODE_LIST>

Caution: Your use of any information or materials on this website is entirely at your own risk. It is provided for educational purposes only. It has been tested internally, however, we do not guarantee that it will work for you. Ensure that you run it in your test environment before using.
Still page under construction !!! 🙂

Delete Node without remove software

Delete Node without remove GI and RDBMS binaries.

0. Environment
1. Backup OCR
2. Check status of service
3. Shutdown instance 2
4. Unpin Node
5. Disable Oracle Clusterware
6. Delete Node from Clusterware Configuration
7. Backup Inventory
8. Update Inventory for ORACLE_HOME
9. Update Inventory for GI_HOME


0. Environment

Two node RAC, version 11.2.0.3
Node name: RAC1, RAC2
Database name: nike , instances: nike1, nik2 . Admin Managed Database.
Service name: nike_srv
OS: RHEL 5.7

Task: We are going to delete node RAC2 from cluster without removing GI and RDBMS binaries because i want to add the node back later.

Current Status
===============

[root@rac1 ~]# crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       rac1
               ONLINE  ONLINE       rac2
ora.DATA1.dg
               ONLINE  ONLINE       rac1
               ONLINE  ONLINE       rac2
ora.LISTENER.lsnr
               ONLINE  ONLINE       rac1
               ONLINE  ONLINE       rac2
ora.asm
               ONLINE  ONLINE       rac1                     Started
               ONLINE  ONLINE       rac2                     Started
ora.gsd
               OFFLINE OFFLINE      rac1
               OFFLINE OFFLINE      rac2
ora.net1.network
               ONLINE  ONLINE       rac1
               ONLINE  ONLINE       rac2
ora.ons
               ONLINE  ONLINE       rac1
               ONLINE  ONLINE       rac2
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       rac2
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       rac1
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       rac1
ora.cvu
      1        ONLINE  ONLINE       rac1
ora.nike.db
      1        ONLINE  ONLINE       rac1                     Open
      2        ONLINE  ONLINE       rac2                     Open
ora.nike.nike_srv.svc
      1        ONLINE  ONLINE       rac1
ora.oc4j
      1        ONLINE  OFFLINE
ora.rac1.vip
      1        ONLINE  ONLINE       rac1
ora.rac2.vip
      1        ONLINE  ONLINE       rac2
ora.scan1.vip
      1        ONLINE  ONLINE       rac2
ora.scan2.vip
      1        ONLINE  ONLINE       rac1
ora.scan3.vip
      1        ONLINE  ONLINE       rac1
[root@rac1 ~]#


1. Backup OCR

[root@rac1 ~]# ocrconfig -manualbackup

rac1     2015/06/23 02:39:07     /u01/app/11.2.0/grid/cdata/rac-scan/backup_20150623_023907.ocr

rac1     2015/06/19 23:38:03     /u01/app/11.2.0/grid/cdata/rac-scan/backup_20150619_233803.ocr
[root@rac1 ~]#


2. Check status of service

[oracle@rac2 ~]$ srvctl status service -d nike
Service nike_srv is running on instance(s) nike1
[oracle@rac2 ~]$

Note: Confirm where service is now running. If service running on instance 2 then manually failover the service.

srvctl relocate service -d <dbname> -s <service name> -i <old_inst> -t <new_inst>

Note that this does not disconnect any current sessions


3. Shutdown instance 2

[oracle@rac2 ~]$ srvctl stop instance -d nike -i nike2
[oracle@rac2 ~]$ srvctl status database -d nike
Instance nike1 is running on node rac1
Instance nike2 is not running on node rac2
[oracle@rac2 ~]$


4. Unpin Node

As root from node RAC1

[root@rac1 ~]# olsnodes -s -t
rac1    Active  Pinned
rac2    Active  Pinned
[root@rac1 ~]# crsctl unpin css -n rac2
CRS-4667: Node rac2 successfully unpinned.
[root@rac1 ~]# olsnodes -s -t
rac1    Active  Pinned
rac2    Active  Unpinned <<<<
[root@rac1 ~]#

Note: If Cluster Synchronization Services (CSS) is not running on the node you are deleting, then the crsctl unpin css command in this step fails.


5. Disable Oracle Clusterware

From node RAC2, which you want to delete
As user root.

[root@rac2 ~]# cd /u01/app/11.2.0/grid/crs/install/
[root@rac2 install]# ./rootcrs.pl -deconfig -force
Using configuration parameter file: ./crsconfig_params
Network exists: 1/192.168.2.0/255.255.255.0/eth0, type static
VIP exists: /rac1-vip/192.168.2.103/192.168.2.0/255.255.255.0/eth0, hosting node rac1
VIP exists: /rac2-vip/192.168.2.104/192.168.2.0/255.255.255.0/eth0, hosting node rac2
GSD exists
ONS exists: Local port 6100, remote port 6200, EM port 2016
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac2'
CRS-2673: Attempting to stop 'ora.crsd' on 'rac2'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac2'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rac2'
CRS-2673: Attempting to stop 'ora.DATA1.dg' on 'rac2'
CRS-2677: Stop of 'ora.DATA1.dg' on 'rac2' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rac2'
CRS-2677: Stop of 'ora.asm' on 'rac2' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rac2' has completed
CRS-2677: Stop of 'ora.crsd' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac2'
CRS-2673: Attempting to stop 'ora.evmd' on 'rac2'
CRS-2673: Attempting to stop 'ora.asm' on 'rac2'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac2'
CRS-2677: Stop of 'ora.evmd' on 'rac2' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rac2' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rac2' succeeded
CRS-2677: Stop of 'ora.asm' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac2'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac2'
CRS-2677: Stop of 'ora.cssd' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'rac2'
CRS-2677: Stop of 'ora.crf' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rac2'
CRS-2677: Stop of 'ora.gipcd' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac2'
CRS-2677: Stop of 'ora.gpnpd' on 'rac2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
Successfully deconfigured Oracle clusterware stack on this node
[root@rac2 install]#


6. Delete Node from Clusterware Configuration

From node RAC1
As root user

[root@rac1 ~]# crsctl delete node -n rac2
CRS-4661: Node rac2 successfully deleted.
[root@rac1 ~]#

[root@rac1 ~]# olsnodes -t -s
rac1    Active  Pinned
[root@rac1 ~]#

As ORACLE HOME owner, remove instance 2 from OCR

[oracle@rac1 ~]$ which srvctl
/u01/app/oracle/product/11.2.0/db_1/bin/srvctl << you should run from ORACLE_HOME
[oracle@rac1 ~]$
[oracle@rac1 ~]$ srvctl remove instance -d nike -i nike2 <<<
Remove instance from the database nike? (y/[n]) y
[oracle@rac1 ~]$

[oracle@rac1 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       rac1
ora.DATA1.dg
               ONLINE  ONLINE       rac1
ora.LISTENER.lsnr
               ONLINE  ONLINE       rac1
ora.asm
               ONLINE  ONLINE       rac1                     Started
ora.gsd
               OFFLINE OFFLINE      rac1
ora.net1.network
               ONLINE  ONLINE       rac1
ora.ons
               ONLINE  ONLINE       rac1
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       rac1
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       rac1
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       rac1
ora.cvu
      1        ONLINE  ONLINE       rac1
ora.nike.db
      1        ONLINE  ONLINE       rac1                     Open
ora.nike.nike_srv.svc
      1        ONLINE  ONLINE       rac1
ora.oc4j
      1        ONLINE  ONLINE       rac1
ora.rac1.vip
      1        ONLINE  ONLINE       rac1
ora.scan1.vip
      1        ONLINE  ONLINE       rac1
ora.scan2.vip
      1        ONLINE  ONLINE       rac1
ora.scan3.vip
      1        ONLINE  ONLINE       rac1
[oracle@rac1 ~]$

From node RAC2

[root@rac2 ~]# ps -ef | grep init
root         1     0  0 Jun22 ?        00:00:00 init [5]
root      9125  8531  0 03:30 pts/1    00:00:00 grep init
[root@rac2 ~]# ps -ef | grep d.bin
root      9127  8531  0 03:30 pts/1    00:00:00 grep d.bin
[root@rac2 ~]#


7. Backup Inventory

From node RAC1
As root.

[root@rac1 ~]# cat /etc/oraInst.loc
inventory_loc=/u01/app/oraInventory
inst_group=oinstall
[root@rac1 ~]#
[root@rac1 ~]# cp -rp /u01/app/oraInventory /u01/app/oraInventory_bkp


8. Update Inventory for ORACLE_HOME

From node RAC1
As ORACLE_HOME owner

[oracle@rac1 bin]$ cd /u01/app/oracle/product/11.2.0/db_1/oui/bin
[oracle@rac1 bin]$ ./runInstaller -updateNodeList ORACLE_HOME=/u01/app/oracle/product/11.2.0/db_1 "CLUSTER_NODES={rac1}"
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 5671 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /u01/app/oraInventory
'UpdateNodeList' was successful.
[oracle@rac1 bin]$


9. Update Inventory for GI_HOME

From node RAC1
As GRID_HOME owner

[oracle@rac1 bin]$ cd /u01/app/11.2.0/grid/oui/bin/
[oracle@rac1 bin]$ ./runInstaller -updateNodeList ORACLE_HOME=/u01/app/11.2.0/grid "CLUSTER_NODES={rac1}" CRS=TRUE
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 5671 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /u01/app/oraInventory
'UpdateNodeList' was successful.
[oracle@rac1 bin]$

Caution: Your use of any information or materials on this website is entirely at your own risk. It is provided for educational purposes only. It has been tested internally, however, we do not guarantee that it will work for you. Ensure that you run it in your test environment before using.
Still page under construction !!! 🙂

Add Node

Add Node to 11gR2 Oracle RAC Cluster (11.2.0.3)

0. Environment

1. Pre-installation tasks for GI for a cluster

0.i) Backup OCR
i) Install the cvuqdisk Package for Linux
ii) Verify New Node (HWOS)
iii) Verify Peer (REFNODE)
iv) Verify New Node (New Pre-Node)
v) Run fixup scripts

2. Cluster Node Addition for GI Home.

i) Run addnode.sh script
ii) Run orainstRoot.sh #On nodes rac2
iii) Run root.sh #On nodes rac2
iv) Check Clusterware Resources after ran root.sh
v) Run cluvfy post-addNode script
vi) Check Cluster Nodes
vii) Check TNS Listener
viii) Check ASM Status
ix) Check OCR
x) Check Vote disk

3. Cluster Node Addition for RDBMS Home.

i) Run addnode.sh script
ii) root.sh #On nodes rac2 from RDBMS home

4. Add Instance to Database through Command-Line or you can add via dbca.

i)Pre-task
ii) Add redo thread
iii) Add undo tablespace
iv) Add instance to OCR
v) Add service to new instance via srvctl or you can add via dbca
vi) Check the cluster stack

Let start !!!


0. Environment

One node RAC 11.2.0.3 (Not RAC ONE), single node RAC. Earlier it was two node RAC setup, recently i have deleted 2nd node from cluster for testing.
Node name: RAC1
OS: RHEL 5
DATABASE: nike, instance: nike1

Task: We are going to add node “RAC2” to our existing cluster.

Current status

[root@rac1 ~]# crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       rac1
ora.DATA1.dg
               ONLINE  ONLINE       rac1
ora.LISTENER.lsnr
               ONLINE  ONLINE       rac1
ora.asm
               ONLINE  ONLINE       rac1                     Started
ora.gsd
               OFFLINE OFFLINE      rac1
ora.net1.network
               ONLINE  ONLINE       rac1
ora.ons
               ONLINE  ONLINE       rac1
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       rac1
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       rac1
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       rac1
ora.cvu
      1        ONLINE  ONLINE       rac1
ora.nike.db
      1        ONLINE  ONLINE       rac1                     Open
ora.nike.nike_srv.svc
      1        ONLINE  ONLINE       rac1
ora.oc4j
      1        ONLINE  ONLINE       rac1
ora.rac1.vip
      1        ONLINE  ONLINE       rac1
ora.scan1.vip
      1        ONLINE  ONLINE       rac1
ora.scan2.vip
      1        ONLINE  ONLINE       rac1
ora.scan3.vip
      1        ONLINE  ONLINE       rac1
[root@rac1 ~]#


1. Pre-installation tasks for GI for a cluster

A) Install and Configure the Linux Operating System on the New Node     << This is step already done by SA
B) Configure Access to the Shared Storage     << This is step already done by SA
C) Install and Configure ASMLib     << This is step already done by SA
D) SSH configure    << This is step already done by SA


0.i) Backup OCR

From: Node RAC1
As root

[root@rac1 ~]# ocrconfig -manualbackup
rac1     2015/06/19 23:38:03     /u01/app/11.2.0/grid/cdata/rac-scan/backup_20150619_233803.ocr
[root@rac1 ~]#


i) Install the cvuqdisk Package for Linux

[root@rac2 oracle]# rpm -ivh cvuqdisk-1.0.9-1.rpm
Preparing...                ########################################### [100%]
Using default group oinstall to install package
   1:cvuqdisk               ########################################### [100%]
[root@rac2 oracle]#

Note: Without cvuqdisk package, CVU cannot discover shared disks and you will receive the error message "Package cvuqdisk not installed" when the CVU is run. 

Example below:
==============
Checking shared storage accessibility...

WARNING:
rac2:PRVF-7017 : Package cvuqdisk not installed
        rac2
No shared storage found
Shared storage check failed on nodes "rac2"


ii) Verify New Node (HWOS)

As GI Home owner
From active node RAC1.

[oracle@rac1 ~]$ cluvfy stage -post hwos -n rac2

Performing post-checks for hardware and operating system setup

Checking node reachability...
Node reachability check passed from node "rac1"


Checking user equivalence...
User equivalence check passed for user "oracle"

Checking node connectivity...

Checking hosts config file...

Verification of the hosts config file successful

Node connectivity passed for subnet "192.168.2.0" with node(s) rac2
TCP connectivity check passed for subnet "192.168.2.0"

Node connectivity passed for subnet "192.168.0.0" with node(s) rac2
TCP connectivity check passed for subnet "192.168.0.0"

Node connectivity passed for subnet "10.0.4.0" with node(s) rac2
TCP connectivity check passed for subnet "10.0.4.0"


Interfaces found on subnet "10.0.4.0" that are likely candidates for VIP are:
rac2 eth2:10.0.4.15

Interfaces found on subnet "192.168.2.0" that are likely candidates for a private interconnect are:
rac2 eth0:192.168.2.102

Interfaces found on subnet "192.168.0.0" that are likely candidates for a private interconnect are:
rac2 eth1:192.168.0.102

Node connectivity check passed

Checking multicast communication...

Checking subnet "192.168.2.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "192.168.2.0" for multicast communication with multicast group "230.0.1.0" passed.

Checking subnet "192.168.0.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "192.168.0.0" for multicast communication with multicast group "230.0.1.0" passed.

Checking subnet "10.0.4.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "10.0.4.0" for multicast communication with multicast group "230.0.1.0" passed.

Check of multicast communication passed.
Check for multiple users with UID value 0 passed
Time zone consistency check passed

Checking shared storage accessibility...

  Disk                                  Sharing Nodes (1 in count)
  ------------------------------------  ------------------------
  /dev/sda                              rac2

  Disk                                  Sharing Nodes (1 in count)
  ------------------------------------  ------------------------
  /dev/sdb                              rac2


Shared storage check was successful on nodes "rac2"

Post-check for hardware and operating system setup was successful.
[oracle@rac1 ~]$


iii) Verify Peer (REFNODE)

From active node RAC1
As GI Home owner

[oracle@rac1 ~]$ cluvfy comp peer -refnode rac1 -n rac2 -orainv oinstall -osdba dba -verbose

Verifying peer compatibility

Checking peer compatibility...

Compatibility check: Physical memory [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          3.8633GB (4050940.0KB)    3.8633GB (4050940.0KB)    matched
Physical memory check passed

Compatibility check: Available memory [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          3.6215GB (3797424.0KB)    2.6165GB (2743628.0KB)    mismatched
Available memory check failed

Compatibility check: Swap space [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          1.9994GB (2096472.0KB)    5.5385GB (5807488.0KB)    mismatched
Swap space check failed

Compatibility check: Free disk space for "/u01/app/11.2.0/grid" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          17.165GB (1.7998848E7KB)  21.2725GB (2.2305792E7KB)  mismatched
Free disk space check failed

Compatibility check: Free disk space for "/tmp" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          3.707GB (3887104.0KB)     4.4717GB (4688896.0KB)    mismatched
Free disk space check failed

Compatibility check: User existence for "oracle" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          oracle(1100)              oracle(1100)              matched
User existence for "oracle" check passed

Compatibility check: Group existence for "oinstall" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          oinstall(1000)            oinstall(1000)            matched
Group existence for "oinstall" check passed

Compatibility check: Group existence for "dba" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          dba(1200)                 dba(1200)                 matched
Group existence for "dba" check passed

Compatibility check: Group membership for "oracle" in "oinstall (Primary)" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          yes                       yes                       matched
Group membership for "oracle" in "oinstall (Primary)" check passed

Compatibility check: Group membership for "oracle" in "dba" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          yes                       yes                       matched
Group membership for "oracle" in "dba" check passed

Compatibility check: Run level [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          5                         5                         matched
Run level check passed

Compatibility check: System architecture [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          x86_64                    x86_64                    matched
System architecture check passed

Compatibility check: Kernel version [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          2.6.32-200.13.1.el5uek    2.6.32-200.13.1.el5uek    matched
Kernel version check passed

Compatibility check: Kernel param "semmsl" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          250                       250                       matched
Kernel param "semmsl" check passed

Compatibility check: Kernel param "semmns" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          32000                     32000                     matched
Kernel param "semmns" check passed

Compatibility check: Kernel param "semopm" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          100                       100                       matched
Kernel param "semopm" check passed

Compatibility check: Kernel param "semmni" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          128                       128                       matched
Kernel param "semmni" check passed

Compatibility check: Kernel param "shmmax" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          2074081280                1054504960                mismatched
Kernel param "shmmax" check failed

Compatibility check: Kernel param "shmmni" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          4096                      4096                      matched
Kernel param "shmmni" check passed

Compatibility check: Kernel param "shmall" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          2097152                   2097152                   matched
Kernel param "shmall" check passed

Compatibility check: Kernel param "file-max" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          6815744                   6815744                   matched
Kernel param "file-max" check passed

Compatibility check: Kernel param "ip_local_port_range" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          between 9000.0 & 65500.0  between 9000.0 & 65500.0  matched
Kernel param "ip_local_port_range" check passed

Compatibility check: Kernel param "rmem_default" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          262144                    262144                    matched
Kernel param "rmem_default" check passed

Compatibility check: Kernel param "rmem_max" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          4194304                   4194304                   matched
Kernel param "rmem_max" check passed

Compatibility check: Kernel param "wmem_default" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          262144                    262144                    matched
Kernel param "wmem_default" check passed

Compatibility check: Kernel param "wmem_max" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          1048586                   1048586                   matched
Kernel param "wmem_max" check passed

Compatibility check: Kernel param "aio-max-nr" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          1048576                   1048576                   matched
Kernel param "aio-max-nr" check passed

Compatibility check: Package existence for "make" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          make-3.81-3.el5           make-3.81-3.el5           matched
Package existence for "make" check passed

Compatibility check: Package existence for "binutils" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          binutils-2.17.50.0.6-14.el5  binutils-2.17.50.0.6-14.el5  matched
Package existence for "binutils" check passed

Compatibility check: Package existence for "gcc (x86_64)" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          gcc-4.1.2-51.el5 (x86_64)  gcc-4.1.2-51.el5 (x86_64)  matched
Package existence for "gcc (x86_64)" check passed

Compatibility check: Package existence for "libaio (x86_64)" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          libaio-0.3.106-5 (x86_64),libaio-0.3.106-5 (i386)  libaio-0.3.106-5 (x86_64),libaio-0.3.106-5 (i386)  matched
Package existence for "libaio (x86_64)" check passed

Compatibility check: Package existence for "glibc (x86_64)" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          glibc-2.5-65 (x86_64),glibc-2.5-65 (i686)  glibc-2.5-65 (x86_64),glibc-2.5-65 (i686)  matched
Package existence for "glibc (x86_64)" check passed

Compatibility check: Package existence for "compat-libstdc++-33 (x86_64)" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          compat-libstdc++-33-3.2.3-61 (x86_64),compat-libstdc++-33-3.2.3-61 (i386)  compat-libstdc++-33-3.2.3-61 (x86_64),compat-libstdc++-33-3.2.3-61 (i386)  matched
Package existence for "compat-libstdc++-33 (x86_64)" check passed

Compatibility check: Package existence for "elfutils-libelf (x86_64)" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          elfutils-libelf-0.137-3.el5 (x86_64),elfutils-libelf-0.137-3.el5 (i386)  elfutils-libelf-0.137-3.el5 (x86_64),elfutils-libelf-0.137-3.el5 (i386)  matched
Package existence for "elfutils-libelf (x86_64)" check passed

Compatibility check: Package existence for "elfutils-libelf-devel" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          elfutils-libelf-devel-0.137-3.el5  elfutils-libelf-devel-0.137-3.el5  matched
Package existence for "elfutils-libelf-devel" check passed

Compatibility check: Package existence for "glibc-common" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          glibc-common-2.5-65       glibc-common-2.5-65       matched
Package existence for "glibc-common" check passed

Compatibility check: Package existence for "glibc-devel (x86_64)" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          glibc-devel-2.5-65 (x86_64),glibc-devel-2.5-65 (i386)  glibc-devel-2.5-65 (x86_64),glibc-devel-2.5-65 (i386)  matched
Package existence for "glibc-devel (x86_64)" check passed

Compatibility check: Package existence for "glibc-headers" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          glibc-headers-2.5-65      glibc-headers-2.5-65      matched
Package existence for "glibc-headers" check passed

Compatibility check: Package existence for "gcc-c++ (x86_64)" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          gcc-c++-4.1.2-51.el5 (x86_64)  gcc-c++-4.1.2-51.el5 (x86_64)  matched
Package existence for "gcc-c++ (x86_64)" check passed

Compatibility check: Package existence for "libaio-devel (x86_64)" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          libaio-devel-0.3.106-5 (i386),libaio-devel-0.3.106-5 (x86_64)  libaio-devel-0.3.106-5 (i386),libaio-devel-0.3.106-5 (x86_64)  matched
Package existence for "libaio-devel (x86_64)" check passed

Compatibility check: Package existence for "libgcc (x86_64)" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          libgcc-4.1.2-51.el5 (x86_64),libgcc-4.1.2-51.el5 (i386)  libgcc-4.1.2-51.el5 (x86_64),libgcc-4.1.2-51.el5 (i386)  matched
Package existence for "libgcc (x86_64)" check passed

Compatibility check: Package existence for "libstdc++ (x86_64)" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          libstdc++-4.1.2-51.el5 (x86_64),libstdc++-4.1.2-51.el5 (i386)  libstdc++-4.1.2-51.el5 (x86_64),libstdc++-4.1.2-51.el5 (i386)  matched
Package existence for "libstdc++ (x86_64)" check passed

Compatibility check: Package existence for "libstdc++-devel (x86_64)" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          libstdc++-devel-4.1.2-51.el5 (x86_64)  libstdc++-devel-4.1.2-51.el5 (x86_64)  matched
Package existence for "libstdc++-devel (x86_64)" check passed

Compatibility check: Package existence for "sysstat" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          sysstat-7.0.2-11.el5      sysstat-7.0.2-11.el5      matched
Package existence for "sysstat" check passed

Compatibility check: Package existence for "ksh" [reference node: rac1]
  Node Name     Status                    Ref. node status          Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          ksh-20100202-1.el5_6.6    ksh-20100202-1.el5_6.6    matched
Package existence for "ksh" check passed

Verification of peer compatibility was unsuccessful.
Checks did not pass for the following node(s):
        rac2
[oracle@rac1 ~]$


iv) Verify New Node (New Pre-Node)

As GI Home owner
From Node RAC1

[oracle@rac1 ~]$ cluvfy stage -pre nodeadd -n rac2 -fixup -verbose

Performing pre-checks for node addition

Checking node reachability...

Check: Node reachability from node "rac1"
  Destination Node                      Reachable?
  ------------------------------------  ------------------------
  rac2                                  yes
Result: Node reachability check passed from node "rac1"


Checking user equivalence...

Check: User equivalence for user "oracle"
  Node Name                             Status
  ------------------------------------  ------------------------
  rac2                                  passed
Result: User equivalence check passed for user "oracle"

Checking node connectivity...

Checking hosts config file...
  Node Name                             Status
  ------------------------------------  ------------------------
  rac1                                  passed
  rac2                                  passed

Verification of the hosts config file successful


Interface information for node "rac1"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.2.101   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:02:46:97 1500
 eth0   192.168.2.106   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:02:46:97 1500
 eth0   192.168.2.107   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:02:46:97 1500
 eth0   192.168.2.103   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:02:46:97 1500
 eth0   192.168.2.105   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:02:46:97 1500
 eth1   192.168.0.101   192.168.0.0     0.0.0.0         10.0.4.2        08:00:27:9A:66:6A 1500
 eth1   169.254.6.127   169.254.0.0     0.0.0.0         10.0.4.2        08:00:27:9A:66:6A 1500
 eth2   10.0.4.15       10.0.4.0        0.0.0.0         10.0.4.2        08:00:27:D3:B8:9F 1500


Interface information for node "rac2"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.2.102   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:93:7F:00 1500
 eth1   192.168.0.102   192.168.0.0     0.0.0.0         10.0.4.2        08:00:27:63:80:76 1500
 eth2   10.0.4.15       10.0.4.0        0.0.0.0         10.0.4.2        08:00:27:4C:B3:01 1500


Check: Node connectivity for interface "eth0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rac1[192.168.2.101]             rac1[192.168.2.106]             yes
  rac1[192.168.2.101]             rac1[192.168.2.107]             yes
  rac1[192.168.2.101]             rac1[192.168.2.103]             yes
  rac1[192.168.2.101]             rac1[192.168.2.105]             yes
  rac1[192.168.2.101]             rac2[192.168.2.102]             yes
  rac1[192.168.2.106]             rac1[192.168.2.107]             yes
  rac1[192.168.2.106]             rac1[192.168.2.103]             yes
  rac1[192.168.2.106]             rac1[192.168.2.105]             yes
  rac1[192.168.2.106]             rac2[192.168.2.102]             yes
  rac1[192.168.2.107]             rac1[192.168.2.103]             yes
  rac1[192.168.2.107]             rac1[192.168.2.105]             yes
  rac1[192.168.2.107]             rac2[192.168.2.102]             yes
  rac1[192.168.2.103]             rac1[192.168.2.105]             yes
  rac1[192.168.2.103]             rac2[192.168.2.102]             yes
  rac1[192.168.2.105]             rac2[192.168.2.102]             yes
Result: Node connectivity passed for interface "eth0"


Check: TCP connectivity of subnet "192.168.2.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rac1:192.168.2.101              rac1:192.168.2.106              passed
  rac1:192.168.2.101              rac1:192.168.2.107              passed
  rac1:192.168.2.101              rac1:192.168.2.103              passed
  rac1:192.168.2.101              rac1:192.168.2.105              passed
  rac1:192.168.2.101              rac2:192.168.2.102              passed
Result: TCP connectivity check passed for subnet "192.168.2.0"

Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "192.168.2.0".
Subnet mask consistency check passed.

Result: Node connectivity check passed

Checking multicast communication...

Checking subnet "192.168.2.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "192.168.2.0" for multicast communication with multicast group "230.0.1.0" passed.

Check of multicast communication passed.

Checking CRS integrity...

Clusterware version consistency passed
The Oracle Clusterware is healthy on node "rac1"

CRS integrity check passed

Checking shared resources...

Checking CRS home location...
"/u01/app/11.2.0/grid" is shared
Result: Shared resources check for node addition passed


Checking node connectivity...

Checking hosts config file...
  Node Name                             Status
  ------------------------------------  ------------------------
  rac1                                  passed
  rac2                                  passed

Verification of the hosts config file successful


Interface information for node "rac1"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.2.101   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:02:46:97 1500
 eth0   192.168.2.106   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:02:46:97 1500
 eth0   192.168.2.107   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:02:46:97 1500
 eth0   192.168.2.103   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:02:46:97 1500
 eth0   192.168.2.105   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:02:46:97 1500
 eth1   192.168.0.101   192.168.0.0     0.0.0.0         10.0.4.2        08:00:27:9A:66:6A 1500
 eth1   169.254.6.127   169.254.0.0     0.0.0.0         10.0.4.2        08:00:27:9A:66:6A 1500
 eth2   10.0.4.15       10.0.4.0        0.0.0.0         10.0.4.2        08:00:27:D3:B8:9F 1500


Interface information for node "rac2"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.2.102   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:93:7F:00 1500
 eth1   192.168.0.102   192.168.0.0     0.0.0.0         10.0.4.2        08:00:27:63:80:76 1500
 eth2   10.0.4.15       10.0.4.0        0.0.0.0         10.0.4.2        08:00:27:4C:B3:01 1500


Check: Node connectivity for interface "eth0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rac1[192.168.2.101]             rac1[192.168.2.106]             yes
  rac1[192.168.2.101]             rac1[192.168.2.107]             yes
  rac1[192.168.2.101]             rac1[192.168.2.103]             yes
  rac1[192.168.2.101]             rac1[192.168.2.105]             yes
  rac1[192.168.2.101]             rac2[192.168.2.102]             yes
  rac1[192.168.2.106]             rac1[192.168.2.107]             yes
  rac1[192.168.2.106]             rac1[192.168.2.103]             yes
  rac1[192.168.2.106]             rac1[192.168.2.105]             yes
  rac1[192.168.2.106]             rac2[192.168.2.102]             yes
  rac1[192.168.2.107]             rac1[192.168.2.103]             yes
  rac1[192.168.2.107]             rac1[192.168.2.105]             yes
  rac1[192.168.2.107]             rac2[192.168.2.102]             yes
  rac1[192.168.2.103]             rac1[192.168.2.105]             yes
  rac1[192.168.2.103]             rac2[192.168.2.102]             yes
  rac1[192.168.2.105]             rac2[192.168.2.102]             yes
Result: Node connectivity passed for interface "eth0"


Check: TCP connectivity of subnet "192.168.2.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rac1:192.168.2.101              rac1:192.168.2.106              passed
  rac1:192.168.2.101              rac1:192.168.2.107              passed
  rac1:192.168.2.101              rac1:192.168.2.103              passed
  rac1:192.168.2.101              rac1:192.168.2.105              passed
  rac1:192.168.2.101              rac2:192.168.2.102              passed
Result: TCP connectivity check passed for subnet "192.168.2.0"


Check: Node connectivity for interface "eth1"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rac1[192.168.0.101]             rac2[192.168.0.102]             yes
Result: Node connectivity passed for interface "eth1"


Check: TCP connectivity of subnet "192.168.0.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rac1:192.168.0.101              rac2:192.168.0.102              passed
Result: TCP connectivity check passed for subnet "192.168.0.0"

Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "192.168.2.0".
Subnet mask consistency check passed for subnet "192.168.0.0".
Subnet mask consistency check passed.

Result: Node connectivity check passed

Checking multicast communication...

Checking subnet "192.168.2.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "192.168.2.0" for multicast communication with multicast group "230.0.1.0" passed.

Checking subnet "192.168.0.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "192.168.0.0" for multicast communication with multicast group "230.0.1.0" passed.

Check of multicast communication passed.

Check: Total memory
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          3.8633GB (4050940.0KB)    1.5GB (1572864.0KB)       passed
  rac1          3.8633GB (4050940.0KB)    1.5GB (1572864.0KB)       passed
Result: Total memory check passed

Check: Available memory
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          3.6216GB (3797500.0KB)    50MB (51200.0KB)          passed
  rac1          2.4853GB (2606016.0KB)    50MB (51200.0KB)          passed
Result: Available memory check passed

Check: Swap space
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          1.9994GB (2096472.0KB)    3.8633GB (4050940.0KB)    failed  <<<<<
  rac1          5.5385GB (5807488.0KB)    3.8633GB (4050940.0KB)    passed
Result: Swap space check failed

Check: Free disk space for "rac2:/u01/app/11.2.0/grid"
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /u01/app/11.2.0/grid  rac2          /u01          17.165GB      5.5GB         passed
Result: Free disk space check passed for "rac2:/u01/app/11.2.0/grid"

Check: Free disk space for "rac1:/u01/app/11.2.0/grid"
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /u01/app/11.2.0/grid  rac1          /u01          21.2705GB     5.5GB         passed
Result: Free disk space check passed for "rac1:/u01/app/11.2.0/grid"

Check: Free disk space for "rac2:/tmp"
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /tmp              rac2          /             3.707GB       1GB           passed
Result: Free disk space check passed for "rac2:/tmp"

Check: Free disk space for "rac1:/tmp"
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /tmp              rac1          /             4.4717GB      1GB           passed
Result: Free disk space check passed for "rac1:/tmp"

Check: User existence for "oracle"
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rac2          passed                    exists(1100)
  rac1          passed                    exists(1100)

Checking for multiple users with UID value 1100
Result: Check for multiple users with UID value 1100 passed
Result: User existence check passed for "oracle"

Check: Run level
  Node Name     run level                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          5                         3,5                       passed
  rac1          5                         3,5                       passed
Result: Run level check passed

Check: Hard limits for "maximum open file descriptors"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rac1              hard          65536         65536         passed
  rac2              hard          65536         65536         passed
Result: Hard limits check passed for "maximum open file descriptors"

Check: Soft limits for "maximum open file descriptors"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rac1              soft          1024          1024          passed
  rac2              soft          1024          1024          passed
Result: Soft limits check passed for "maximum open file descriptors"

Check: Hard limits for "maximum user processes"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rac1              hard          16384         16384         passed
  rac2              hard          16384         16384         passed
Result: Hard limits check passed for "maximum user processes"

Check: Soft limits for "maximum user processes"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rac1              soft          2047          2047          passed
  rac2              soft          2047          2047          passed
Result: Soft limits check passed for "maximum user processes"

Check: System architecture
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          x86_64                    x86_64                    passed
  rac1          x86_64                    x86_64                    passed
Result: System architecture check passed

Check: Kernel version
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          2.6.32-200.13.1.el5uek    2.6.18                    passed
  rac1          2.6.32-200.13.1.el5uek    2.6.18                    passed
Result: Kernel version check passed

Check: Kernel parameter for "semmsl"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac1              250           250           250           passed
  rac2              250           250           250           passed
Result: Kernel parameter check passed for "semmsl"

Check: Kernel parameter for "semmns"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac1              32000         32000         32000         passed
  rac2              32000         32000         32000         passed
Result: Kernel parameter check passed for "semmns"

Check: Kernel parameter for "semopm"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac1              100           100           100           passed
  rac2              100           100           100           passed
Result: Kernel parameter check passed for "semopm"

Check: Kernel parameter for "semmni"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac1              128           128           128           passed
  rac2              128           128           128           passed
Result: Kernel parameter check passed for "semmni"

Check: Kernel parameter for "shmmax"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac1              1054504960    1054504960    2074081280    failed        Current value too low. Configured value too low.  <<< 
  rac2              2074081280    1054504960    2074081280    failed        Configured value too low.  <<<
Result: Kernel parameter check failed for "shmmax"

Check: Kernel parameter for "shmmni"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac1              4096          4096          4096          passed
  rac2              4096          4096          4096          passed
Result: Kernel parameter check passed for "shmmni"

Check: Kernel parameter for "shmall"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac1              2097152       2097152       2097152       passed
  rac2              2097152       2097152       2097152       passed
Result: Kernel parameter check passed for "shmall"

Check: Kernel parameter for "file-max"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac1              6815744       6815744       6815744       passed
  rac2              6815744       6815744       6815744       passed
Result: Kernel parameter check passed for "file-max"

Check: Kernel parameter for "ip_local_port_range"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac1              between 9000.0 & 65500.0  between 9000.0 & 65500.0  between 9000.0 & 65500.0  passed
  rac2              between 9000.0 & 65500.0  between 9000.0 & 65500.0  between 9000.0 & 65500.0  passed
Result: Kernel parameter check passed for "ip_local_port_range"

Check: Kernel parameter for "rmem_default"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac1              262144        262144        262144        passed
  rac2              262144        262144        262144        passed
Result: Kernel parameter check passed for "rmem_default"

Check: Kernel parameter for "rmem_max"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac1              4194304       4194304       4194304       passed
  rac2              4194304       4194304       4194304       passed
Result: Kernel parameter check passed for "rmem_max"

Check: Kernel parameter for "wmem_default"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac1              262144        262144        262144        passed
  rac2              262144        262144        262144        passed
Result: Kernel parameter check passed for "wmem_default"

Check: Kernel parameter for "wmem_max"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac1              1048586       1048586       1048576       passed
  rac2              1048586       1048586       1048576       passed
Result: Kernel parameter check passed for "wmem_max"

Check: Kernel parameter for "aio-max-nr"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac1              1048576       1048576       1048576       passed
  rac2              1048576       1048576       1048576       passed
Result: Kernel parameter check passed for "aio-max-nr"

Check: Package existence for "make"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          make-3.81-3.el5           make-3.81                 passed
  rac1          make-3.81-3.el5           make-3.81                 passed
Result: Package existence check passed for "make"

Check: Package existence for "binutils"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          binutils-2.17.50.0.6-14.el5  binutils-2.17.50.0.6      passed
  rac1          binutils-2.17.50.0.6-14.el5  binutils-2.17.50.0.6      passed
Result: Package existence check passed for "binutils"

Check: Package existence for "gcc(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          gcc(x86_64)-4.1.2-51.el5  gcc(x86_64)-4.1.2         passed
  rac1          gcc(x86_64)-4.1.2-51.el5  gcc(x86_64)-4.1.2         passed
Result: Package existence check passed for "gcc(x86_64)"

Check: Package existence for "libaio(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          libaio(x86_64)-0.3.106-5  libaio(x86_64)-0.3.106    passed
  rac1          libaio(x86_64)-0.3.106-5  libaio(x86_64)-0.3.106    passed
Result: Package existence check passed for "libaio(x86_64)"

Check: Package existence for "glibc(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          glibc(x86_64)-2.5-65      glibc(x86_64)-2.5-24      passed
  rac1          glibc(x86_64)-2.5-65      glibc(x86_64)-2.5-24      passed
Result: Package existence check passed for "glibc(x86_64)"

Check: Package existence for "compat-libstdc++-33(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          compat-libstdc++-33(x86_64)-3.2.3-61  compat-libstdc++-33(x86_64)-3.2.3  passed
  rac1          compat-libstdc++-33(x86_64)-3.2.3-61  compat-libstdc++-33(x86_64)-3.2.3  passed
Result: Package existence check passed for "compat-libstdc++-33(x86_64)"

Check: Package existence for "elfutils-libelf(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          elfutils-libelf(x86_64)-0.137-3.el5  elfutils-libelf(x86_64)-0.125  passed
  rac1          elfutils-libelf(x86_64)-0.137-3.el5  elfutils-libelf(x86_64)-0.125  passed
Result: Package existence check passed for "elfutils-libelf(x86_64)"

Check: Package existence for "elfutils-libelf-devel"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          elfutils-libelf-devel-0.137-3.el5  elfutils-libelf-devel-0.125  passed
  rac1          elfutils-libelf-devel-0.137-3.el5  elfutils-libelf-devel-0.125  passed
Result: Package existence check passed for "elfutils-libelf-devel"

Check: Package existence for "glibc-common"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          glibc-common-2.5-65       glibc-common-2.5          passed
  rac1          glibc-common-2.5-65       glibc-common-2.5          passed
Result: Package existence check passed for "glibc-common"

Check: Package existence for "glibc-devel(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          glibc-devel(x86_64)-2.5-65  glibc-devel(x86_64)-2.5   passed
  rac1          glibc-devel(x86_64)-2.5-65  glibc-devel(x86_64)-2.5   passed
Result: Package existence check passed for "glibc-devel(x86_64)"

Check: Package existence for "glibc-headers"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          glibc-headers-2.5-65      glibc-headers-2.5         passed
  rac1          glibc-headers-2.5-65      glibc-headers-2.5         passed
Result: Package existence check passed for "glibc-headers"

Check: Package existence for "gcc-c++(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          gcc-c++(x86_64)-4.1.2-51.el5  gcc-c++(x86_64)-4.1.2     passed
  rac1          gcc-c++(x86_64)-4.1.2-51.el5  gcc-c++(x86_64)-4.1.2     passed
Result: Package existence check passed for "gcc-c++(x86_64)"

Check: Package existence for "libaio-devel(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          libaio-devel(x86_64)-0.3.106-5  libaio-devel(x86_64)-0.3.106  passed
  rac1          libaio-devel(x86_64)-0.3.106-5  libaio-devel(x86_64)-0.3.106  passed
Result: Package existence check passed for "libaio-devel(x86_64)"

Check: Package existence for "libgcc(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          libgcc(x86_64)-4.1.2-51.el5  libgcc(x86_64)-4.1.2      passed
  rac1          libgcc(x86_64)-4.1.2-51.el5  libgcc(x86_64)-4.1.2      passed
Result: Package existence check passed for "libgcc(x86_64)"

Check: Package existence for "libstdc++(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          libstdc++(x86_64)-4.1.2-51.el5  libstdc++(x86_64)-4.1.2   passed
  rac1          libstdc++(x86_64)-4.1.2-51.el5  libstdc++(x86_64)-4.1.2   passed
Result: Package existence check passed for "libstdc++(x86_64)"

Check: Package existence for "libstdc++-devel(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          libstdc++-devel(x86_64)-4.1.2-51.el5  libstdc++-devel(x86_64)-4.1.2  passed
  rac1          libstdc++-devel(x86_64)-4.1.2-51.el5  libstdc++-devel(x86_64)-4.1.2  passed
Result: Package existence check passed for "libstdc++-devel(x86_64)"

Check: Package existence for "sysstat"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          sysstat-7.0.2-11.el5      sysstat-7.0.2             passed
  rac1          sysstat-7.0.2-11.el5      sysstat-7.0.2             passed
Result: Package existence check passed for "sysstat"

Check: Package existence for "ksh"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac2          ksh-20100202-1.el5_6.6    ksh-20060214              passed
  rac1          ksh-20100202-1.el5_6.6    ksh-20060214              passed
Result: Package existence check passed for "ksh"

Checking for multiple users with UID value 0
Result: Check for multiple users with UID value 0 passed

Check: Current group ID
Result: Current group ID check passed

Starting check for consistency of primary group of root user
  Node Name                             Status
  ------------------------------------  ------------------------
  rac2                                  passed
  rac1                                  passed

Check for consistency of root user's primary group passed

Checking OCR integrity...

OCR integrity check passed

Checking Oracle Cluster Voting Disk configuration...

Oracle Cluster Voting Disk configuration check passed
Check: Time zone consistency
Result: Time zone consistency check passed

Starting Clock synchronization checks using Network Time Protocol(NTP)...

NTP Configuration file check started...
Network Time Protocol(NTP) configuration file not found on any of the nodes. Oracle Cluster Time Synchronization Service(CTSS) can be used instead of NTP for time synchronization on the cluster nodes
No NTP Daemons or Services were found to be running

Result: Clock synchronization check using Network Time Protocol(NTP) passed


Checking to make sure user "oracle" is not in "root" group
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rac2          passed                    does not exist
  rac1          passed                    does not exist
Result: User "oracle" is not part of "root" group. Check passed
Checking consistency of file "/etc/resolv.conf" across nodes

Checking the file "/etc/resolv.conf" to make sure only one of domain and search entries is defined
File "/etc/resolv.conf" does not have both domain and search entries defined
Checking if domain entry in file "/etc/resolv.conf" is consistent across the nodes...
domain entry in file "/etc/resolv.conf" is consistent across nodes
Checking if search entry in file "/etc/resolv.conf" is consistent across the nodes...
search entry in file "/etc/resolv.conf" is consistent across nodes
Checking file "/etc/resolv.conf" to make sure that only one search entry is defined
All nodes have one search entry defined in file "/etc/resolv.conf"
Checking all nodes to make sure that search entry is "rajasekhar.com" as found on node "rac1"
All nodes of the cluster have same value for 'search'
Checking DNS response time for an unreachable node
  Node Name                             Status
  ------------------------------------  ------------------------
  rac1                                  failed
  rac2                                  failed
PRVF-5636 : The DNS response time for an unreachable node exceeded "15000" ms on following nodes: rac1,rac2

File "/etc/resolv.conf" is not consistent across nodes

Fixup information has been generated for following node(s):
rac2,rac1
Please run the following script on each node as "root" user to execute the fixups:
'/tmp/CVU_11.2.0.3.0_oracle/runfixup.sh'

Pre-check for node addition was unsuccessful on all the nodes.
[oracle@rac1 ~]$


v) Run fixup scripts

On RAC 1, as root user

[root@rac1 ~]# /tmp/CVU_11.2.0.3.0_oracle/runfixup.sh
Response file being used is :/tmp/CVU_11.2.0.3.0_oracle/fixup.response
Enable file being used is :/tmp/CVU_11.2.0.3.0_oracle/fixup.enable
Log file location: /tmp/CVU_11.2.0.3.0_oracle/orarun.log
Setting Kernel Parameters...
kernel.shmmax = 68719476736
kernel.shmmax = 1054504960
/tmp/CVU_11.2.0.3.0_oracle/orarun.sh: line 230: [: 68719476736kernel.shmmax: integer expression expected
The value for shmmax in response file is not greater than value for shmmax in /etc/sysctl.conf file. Hence not changing it.
kernel.shmmax = 2074081280
[root@rac1 ~]#

On RAC2, as root user

[root@rac2 ~]# /tmp/CVU_11.2.0.3.0_oracle/runfixup.sh
Response file being used is :/tmp/CVU_11.2.0.3.0_oracle/fixup.response
Enable file being used is :/tmp/CVU_11.2.0.3.0_oracle/fixup.enable
Log file location: /tmp/CVU_11.2.0.3.0_oracle/orarun.log
Setting Kernel Parameters...
kernel.shmmax = 68719476736
kernel.shmmax = 1054504960
/tmp/CVU_11.2.0.3.0_oracle/orarun.sh: line 230: [: 68719476736kernel.shmmax: integer expression expected
The value for shmmax in response file is not greater than value for shmmax in /etc/sysctl.conf file. Hence not changing it.
The value for shmmax in response file is not greater than value of shmmax for current session. Hence not changing it.
[root@rac2 ~]#


2. Cluster Node Addition for GI Home.

Note: Pre-node check failed. Hence need to set the environment variable IGNORE_PREADDNODE_CHECKS=Y before running addNode.sh otherwise, the silent node addition will fail without showing any errors to the console.


i) Run addnode.sh script

As GI Home owner
From active node RAC1

[oracle@rac1 ~]$ cd /u01/app/11.2.0/grid/oui/bin/
[oracle@rac1 bin]$ IGNORE_PREADDNODE_CHECKS=Y
[oracle@rac1 bin]$ export IGNORE_PREADDNODE_CHECKS
[oracle@rac1 bin]$ ./addNode.sh -silent "CLUSTER_NEW_NODES={rac2}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={rac2-vip}"
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 5671 MB    Passed
Oracle Universal Installer, Version 11.2.0.3.0 Production
Copyright (C) 1999, 2011, Oracle. All rights reserved.


Performing tests to see whether nodes rac2 are available
............................................................... 100% Done.

.
-----------------------------------------------------------------------------
Cluster Node Addition Summary
Global Settings
   Source: /u01/app/11.2.0/grid
   New Nodes
Space Requirements
   New Nodes
      rac2
         /u01: Required 6.99GB : Available 15.98GB
Installed Products
   Product Names
      Oracle Grid Infrastructure 11.2.0.3.0
      Sun JDK 1.5.0.30.03
      Installer SDK Component 11.2.0.3.0
      Oracle One-Off Patch Installer 11.2.0.1.7
      Oracle Universal Installer 11.2.0.3.0
      Oracle USM Deconfiguration 11.2.0.3.0
      Oracle Configuration Manager Deconfiguration 10.3.1.0.0
      Enterprise Manager Common Core Files 10.2.0.4.4
      Oracle DBCA Deconfiguration 11.2.0.3.0
      Oracle RAC Deconfiguration 11.2.0.3.0
      Oracle Quality of Service Management (Server) 11.2.0.3.0
      Installation Plugin Files 11.2.0.3.0
      Universal Storage Manager Files 11.2.0.3.0
      Oracle Text Required Support Files 11.2.0.3.0
      Automatic Storage Management Assistant 11.2.0.3.0
      Oracle Database 11g Multimedia Files 11.2.0.3.0
      Oracle Multimedia Java Advanced Imaging 11.2.0.3.0
      Oracle Globalization Support 11.2.0.3.0
      Oracle Multimedia Locator RDBMS Files 11.2.0.3.0
      Oracle Core Required Support Files 11.2.0.3.0
      Bali Share 1.1.18.0.0
      Oracle Database Deconfiguration 11.2.0.3.0
      Oracle Quality of Service Management (Client) 11.2.0.3.0
      Expat libraries 2.0.1.0.1
      Oracle Containers for Java 11.2.0.3.0
      Perl Modules 5.10.0.0.1
      Secure Socket Layer 11.2.0.3.0
      Oracle JDBC/OCI Instant Client 11.2.0.3.0
      Oracle Multimedia Client Option 11.2.0.3.0
      LDAP Required Support Files 11.2.0.3.0
      Character Set Migration Utility 11.2.0.3.0
      Perl Interpreter 5.10.0.0.2
      PL/SQL Embedded Gateway 11.2.0.3.0
      OLAP SQL Scripts 11.2.0.3.0
      Database SQL Scripts 11.2.0.3.0
      Oracle Extended Windowing Toolkit 3.4.47.0.0
      SSL Required Support Files for InstantClient 11.2.0.3.0
      SQL*Plus Files for Instant Client 11.2.0.3.0
      Oracle Net Required Support Files 11.2.0.3.0
      Oracle Database User Interface 2.2.13.0.0
      RDBMS Required Support Files for Instant Client 11.2.0.3.0
      RDBMS Required Support Files Runtime 11.2.0.3.0
      XML Parser for Java 11.2.0.3.0
      Oracle Security Developer Tools 11.2.0.3.0
      Oracle Wallet Manager 11.2.0.3.0
      Enterprise Manager plugin Common Files 11.2.0.3.0
      Platform Required Support Files 11.2.0.3.0
      Oracle JFC Extended Windowing Toolkit 4.2.36.0.0
      RDBMS Required Support Files 11.2.0.3.0
      Oracle Ice Browser 5.2.3.6.0
      Oracle Help For Java 4.2.9.0.0
      Enterprise Manager Common Files 10.2.0.4.3
      Deinstallation Tool 11.2.0.3.0
      Oracle Java Client 11.2.0.3.0
      Cluster Verification Utility Files 11.2.0.3.0
      Oracle Notification Service (eONS) 11.2.0.3.0
      Oracle LDAP administration 11.2.0.3.0
      Cluster Verification Utility Common Files 11.2.0.3.0
      Oracle Clusterware RDBMS Files 11.2.0.3.0
      Oracle Locale Builder 11.2.0.3.0
      Oracle Globalization Support 11.2.0.3.0
      Buildtools Common Files 11.2.0.3.0
      Oracle RAC Required Support Files-HAS 11.2.0.3.0
      SQL*Plus Required Support Files 11.2.0.3.0
      XDK Required Support Files 11.2.0.3.0
      Agent Required Support Files 10.2.0.4.3
      Parser Generator Required Support Files 11.2.0.3.0
      Precompiler Required Support Files 11.2.0.3.0
      Installation Common Files 11.2.0.3.0
      Required Support Files 11.2.0.3.0
      Oracle JDBC/THIN Interfaces 11.2.0.3.0
      Oracle Multimedia Locator 11.2.0.3.0
      Oracle Multimedia 11.2.0.3.0
      HAS Common Files 11.2.0.3.0
      Assistant Common Files 11.2.0.3.0
      PL/SQL 11.2.0.3.0
      HAS Files for DB 11.2.0.3.0
      Oracle Recovery Manager 11.2.0.3.0
      Oracle Database Utilities 11.2.0.3.0
      Oracle Notification Service 11.2.0.3.0
      SQL*Plus 11.2.0.3.0
      Oracle Netca Client 11.2.0.3.0
      Oracle Net 11.2.0.3.0
      Oracle JVM 11.2.0.3.0
      Oracle Internet Directory Client 11.2.0.3.0
      Oracle Net Listener 11.2.0.3.0
      Cluster Ready Services Files 11.2.0.3.0
      Oracle Database 11g 11.2.0.3.0
-----------------------------------------------------------------------------


Instantiating scripts for add node (Sunday, June 21, 2015 12:35:19 PM IST)
.                                                                 1% Done.
Instantiation of add node scripts complete

Copying to remote nodes (Sunday, June 21, 2015 12:35:22 PM IST)
...............................................................................................                                 96% Done.
Home copied to new nodes

Saving inventory on nodes (Sunday, June 21, 2015 12:45:29 PM IST)
.                                                               100% Done.
Save inventory complete
WARNING:A new inventory has been created on one or more nodes in this session. However, it has not yet been registered as the central inventory of this system.
To register the new inventory please run the script at '/u01/app/oraInventory/orainstRoot.sh' with root privileges on nodes 'rac2'.
If you do not register the inventory, you may not be able to update or patch the products you installed.
The following configuration scripts need to be executed as the "root" user in each new cluster node. Each script in the list below is followed by a list of nodes.
/u01/app/oraInventory/orainstRoot.sh #On nodes rac2
/u01/app/11.2.0/grid/root.sh #On nodes rac2
To execute the configuration scripts:
    1. Open a terminal window
    2. Log in as "root"
    3. Run the scripts in each cluster node

The Cluster Node Addition of /u01/app/11.2.0/grid was successful.
Please check '/tmp/silentInstall.log' for more details.
[oracle@rac1 bin]$

Note: If you are using GNS:
cd $GI_HOME/oui/bin
./addNode.sh -silent “CLUSTER_NEW_NODES={rac2}”


ii) Run orainstRoot.sh #On nodes rac2

On node RAC2
As root

[root@rac2 ~]# /u01/app/oraInventory/orainstRoot.sh
Creating the Oracle inventory pointer file (/etc/oraInst.loc)
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.
[root@rac2 ~]#


iii) Run root.sh #On nodes rac2

On node RAC2
As root

[root@rac2 ~]# /u01/app/11.2.0/grid/root.sh
Performing root user operation for Oracle 11g

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
User ignored Prerequisites during installation
OLR initialization - successful
Adding Clusterware entries to inittab
CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node rac1, number 1, and is terminating
An active cluster was found during exclusive startup, restarting to join the cluster
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 11g Release 2.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
[root@rac2 ~]#

Note: The root.sh script will configuring Grid Infrastructure on the new node and includes adding High Availability Services to the /etc/inittab so that CRS starts up when the machine starts.


iv) Check Clusterware Resources after ran root.sh

[root@rac2 ~]# crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       rac1
               ONLINE  ONLINE       rac2
ora.DATA1.dg
               ONLINE  ONLINE       rac1
               ONLINE  ONLINE       rac2
ora.LISTENER.lsnr
               ONLINE  ONLINE       rac1
               ONLINE  ONLINE       rac2
ora.asm
               ONLINE  ONLINE       rac1                     Started
               ONLINE  ONLINE       rac2                     Started
ora.gsd
               OFFLINE OFFLINE      rac1
               OFFLINE OFFLINE      rac2
ora.net1.network
               ONLINE  ONLINE       rac1
               ONLINE  ONLINE       rac2
ora.ons
               ONLINE  ONLINE       rac1
               ONLINE  ONLINE       rac2
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       rac2
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       rac1
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       rac1
ora.cvu
      1        ONLINE  ONLINE       rac1
ora.nike.db
      1        ONLINE  ONLINE       rac1                     Open
ora.nike.nike_srv.svc
      1        ONLINE  ONLINE       rac1
ora.oc4j
      1        ONLINE  ONLINE       rac1
ora.rac1.vip
      1        ONLINE  ONLINE       rac1
ora.rac2.vip
      1        ONLINE  ONLINE       rac2
ora.scan1.vip
      1        ONLINE  ONLINE       rac2
ora.scan2.vip
      1        ONLINE  ONLINE       rac1
ora.scan3.vip
      1        ONLINE  ONLINE       rac1
[root@rac2 ~]#

[root@rac2 ~]# crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
[root@rac2 ~]#


v) Run cluvfy post-node add script

As GI Home owner
From node RAC1 as best practice, because initially we ran pre-node cluvfy from RAC 1 only

[oracle@rac1 ~]$ cluvfy stage -post nodeadd -n rac2 -verbose

Performing post-checks for node addition

Checking node reachability...

Check: Node reachability from node "rac1"
  Destination Node                      Reachable?
  ------------------------------------  ------------------------
  rac2                                  yes
Result: Node reachability check passed from node "rac1"


Checking user equivalence...

Check: User equivalence for user "oracle"
  Node Name                             Status
  ------------------------------------  ------------------------
  rac2                                  passed
Result: User equivalence check passed for user "oracle"

Checking node connectivity...

Checking hosts config file...
  Node Name                             Status
  ------------------------------------  ------------------------
  rac2                                  passed
  rac1                                  passed

Verification of the hosts config file successful


Interface information for node "rac2"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.2.102   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:93:7F:00 1500
 eth0   192.168.2.107   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:93:7F:00 1500
 eth0   192.168.2.104   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:93:7F:00 1500
 eth1   192.168.0.102   192.168.0.0     0.0.0.0         10.0.4.2        08:00:27:63:80:76 1500
 eth1   169.254.215.111 169.254.0.0     0.0.0.0         10.0.4.2        08:00:27:63:80:76 1500
 eth2   10.0.4.15       10.0.4.0        0.0.0.0         10.0.4.2        08:00:27:4C:B3:01 1500


Interface information for node "rac1"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.2.101   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:02:46:97 1500
 eth0   192.168.2.106   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:02:46:97 1500
 eth0   192.168.2.103   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:02:46:97 1500
 eth0   192.168.2.105   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:02:46:97 1500
 eth1   192.168.0.101   192.168.0.0     0.0.0.0         10.0.4.2        08:00:27:9A:66:6A 1500
 eth1   169.254.6.127   169.254.0.0     0.0.0.0         10.0.4.2        08:00:27:9A:66:6A 1500
 eth2   10.0.4.15       10.0.4.0        0.0.0.0         10.0.4.2        08:00:27:D3:B8:9F 1500


Check: Node connectivity for interface "eth0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rac2[192.168.2.102]             rac2[192.168.2.107]             yes
  rac2[192.168.2.102]             rac2[192.168.2.104]             yes
  rac2[192.168.2.102]             rac1[192.168.2.101]             yes
  rac2[192.168.2.102]             rac1[192.168.2.106]             yes
  rac2[192.168.2.102]             rac1[192.168.2.103]             yes
  rac2[192.168.2.102]             rac1[192.168.2.105]             yes
  rac2[192.168.2.107]             rac2[192.168.2.104]             yes
  rac2[192.168.2.107]             rac1[192.168.2.101]             yes
  rac2[192.168.2.107]             rac1[192.168.2.106]             yes
  rac2[192.168.2.107]             rac1[192.168.2.103]             yes
  rac2[192.168.2.107]             rac1[192.168.2.105]             yes
  rac2[192.168.2.104]             rac1[192.168.2.101]             yes
  rac2[192.168.2.104]             rac1[192.168.2.106]             yes
  rac2[192.168.2.104]             rac1[192.168.2.103]             yes
  rac2[192.168.2.104]             rac1[192.168.2.105]             yes
  rac1[192.168.2.101]             rac1[192.168.2.106]             yes
  rac1[192.168.2.101]             rac1[192.168.2.103]             yes
  rac1[192.168.2.101]             rac1[192.168.2.105]             yes
  rac1[192.168.2.106]             rac1[192.168.2.103]             yes
  rac1[192.168.2.106]             rac1[192.168.2.105]             yes
  rac1[192.168.2.103]             rac1[192.168.2.105]             yes
Result: Node connectivity passed for interface "eth0"


Check: TCP connectivity of subnet "192.168.2.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rac1:192.168.2.101              rac2:192.168.2.102              passed
  rac1:192.168.2.101              rac2:192.168.2.107              passed
  rac1:192.168.2.101              rac2:192.168.2.104              passed
  rac1:192.168.2.101              rac1:192.168.2.106              passed
  rac1:192.168.2.101              rac1:192.168.2.103              passed
  rac1:192.168.2.101              rac1:192.168.2.105              passed
Result: TCP connectivity check passed for subnet "192.168.2.0"

Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "192.168.2.0".
Subnet mask consistency check passed.

Result: Node connectivity check passed

Checking multicast communication...

Checking subnet "192.168.2.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "192.168.2.0" for multicast communication with multicast group "230.0.1.0" passed.

Check of multicast communication passed.

Checking cluster integrity...

  Node Name
  ------------------------------------
  rac1
  rac2

Cluster integrity check passed


Checking CRS integrity...

Clusterware version consistency passed
The Oracle Clusterware is healthy on node "rac2"
The Oracle Clusterware is healthy on node "rac1"

CRS integrity check passed

Checking shared resources...

Checking CRS home location...
"/u01/app/11.2.0/grid" is not shared
Result: Shared resources check for node addition passed


Checking node connectivity...

Checking hosts config file...
  Node Name                             Status
  ------------------------------------  ------------------------
  rac2                                  passed
  rac1                                  passed

Verification of the hosts config file successful


Interface information for node "rac2"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.2.102   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:93:7F:00 1500
 eth0   192.168.2.107   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:93:7F:00 1500
 eth0   192.168.2.104   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:93:7F:00 1500
 eth1   192.168.0.102   192.168.0.0     0.0.0.0         10.0.4.2        08:00:27:63:80:76 1500
 eth1   169.254.215.111 169.254.0.0     0.0.0.0         10.0.4.2        08:00:27:63:80:76 1500
 eth2   10.0.4.15       10.0.4.0        0.0.0.0         10.0.4.2        08:00:27:4C:B3:01 1500


Interface information for node "rac1"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.2.101   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:02:46:97 1500
 eth0   192.168.2.106   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:02:46:97 1500
 eth0   192.168.2.103   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:02:46:97 1500
 eth0   192.168.2.105   192.168.2.0     0.0.0.0         10.0.4.2        08:00:27:02:46:97 1500
 eth1   192.168.0.101   192.168.0.0     0.0.0.0         10.0.4.2        08:00:27:9A:66:6A 1500
 eth1   169.254.6.127   169.254.0.0     0.0.0.0         10.0.4.2        08:00:27:9A:66:6A 1500
 eth2   10.0.4.15       10.0.4.0        0.0.0.0         10.0.4.2        08:00:27:D3:B8:9F 1500


Check: Node connectivity for interface "eth0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rac2[192.168.2.102]             rac2[192.168.2.107]             yes
  rac2[192.168.2.102]             rac2[192.168.2.104]             yes
  rac2[192.168.2.102]             rac1[192.168.2.101]             yes
  rac2[192.168.2.102]             rac1[192.168.2.106]             yes
  rac2[192.168.2.102]             rac1[192.168.2.103]             yes
  rac2[192.168.2.102]             rac1[192.168.2.105]             yes
  rac2[192.168.2.107]             rac2[192.168.2.104]             yes
  rac2[192.168.2.107]             rac1[192.168.2.101]             yes
  rac2[192.168.2.107]             rac1[192.168.2.106]             yes
  rac2[192.168.2.107]             rac1[192.168.2.103]             yes
  rac2[192.168.2.107]             rac1[192.168.2.105]             yes
  rac2[192.168.2.104]             rac1[192.168.2.101]             yes
  rac2[192.168.2.104]             rac1[192.168.2.106]             yes
  rac2[192.168.2.104]             rac1[192.168.2.103]             yes
  rac2[192.168.2.104]             rac1[192.168.2.105]             yes
  rac1[192.168.2.101]             rac1[192.168.2.106]             yes
  rac1[192.168.2.101]             rac1[192.168.2.103]             yes
  rac1[192.168.2.101]             rac1[192.168.2.105]             yes
  rac1[192.168.2.106]             rac1[192.168.2.103]             yes
  rac1[192.168.2.106]             rac1[192.168.2.105]             yes
  rac1[192.168.2.103]             rac1[192.168.2.105]             yes
Result: Node connectivity passed for interface "eth0"


Check: TCP connectivity of subnet "192.168.2.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rac1:192.168.2.101              rac2:192.168.2.102              passed
  rac1:192.168.2.101              rac2:192.168.2.107              passed
  rac1:192.168.2.101              rac2:192.168.2.104              passed
  rac1:192.168.2.101              rac1:192.168.2.106              passed
  rac1:192.168.2.101              rac1:192.168.2.103              passed
  rac1:192.168.2.101              rac1:192.168.2.105              passed
Result: TCP connectivity check passed for subnet "192.168.2.0"


Check: Node connectivity for interface "eth1"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rac2[192.168.0.102]             rac1[192.168.0.101]             yes
Result: Node connectivity passed for interface "eth1"


Check: TCP connectivity of subnet "192.168.0.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rac1:192.168.0.101              rac2:192.168.0.102              passed
Result: TCP connectivity check passed for subnet "192.168.0.0"

Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "192.168.2.0".
Subnet mask consistency check passed for subnet "192.168.0.0".
Subnet mask consistency check passed.

Result: Node connectivity check passed

Checking multicast communication...

Checking subnet "192.168.2.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "192.168.2.0" for multicast communication with multicast group "230.0.1.0" passed.

Checking subnet "192.168.0.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "192.168.0.0" for multicast communication with multicast group "230.0.1.0" passed.

Check of multicast communication passed.

Checking node application existence...

Checking existence of VIP node application (required)
  Node Name     Required                  Running?                  Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          yes                       yes                       passed
  rac1          yes                       yes                       passed
VIP node application check passed

Checking existence of NETWORK node application (required)
  Node Name     Required                  Running?                  Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          yes                       yes                       passed
  rac1          yes                       yes                       passed
NETWORK node application check passed

Checking existence of GSD node application (optional)
  Node Name     Required                  Running?                  Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          no                        no                        exists
  rac1          no                        no                        exists
GSD node application is offline on nodes "rac2,rac1"

Checking existence of ONS node application (optional)
  Node Name     Required                  Running?                  Comment
  ------------  ------------------------  ------------------------  ----------
  rac2          no                        yes                       passed
  rac1          no                        yes                       passed
ONS node application check passed


Checking Single Client Access Name (SCAN)...
  SCAN Name         Node          Running?      ListenerName  Port          Running?
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac-scan.rajasekhar.com  rac2          true          LISTENER_SCAN1  1521          true
  rac-scan.rajasekhar.com  rac1          true          LISTENER_SCAN2  1521          true
  rac-scan.rajasekhar.com  rac1          true          LISTENER_SCAN3  1521          true

Checking TCP connectivity to SCAN Listeners...
  Node          ListenerName              TCP connectivity?
  ------------  ------------------------  ------------------------
  rac1          LISTENER_SCAN1            yes
  rac1          LISTENER_SCAN2            yes
  rac1          LISTENER_SCAN3            yes
TCP connectivity to SCAN Listeners exists on all cluster nodes

Checking name resolution setup for "rac-scan.rajasekhar.com"...
  SCAN Name     IP Address                Status                    Comment
  ------------  ------------------------  ------------------------  ----------
  rac-scan.rajasekhar.com  192.168.2.107             passed
  rac-scan.rajasekhar.com  192.168.2.105             passed
  rac-scan.rajasekhar.com  192.168.2.106             passed

Verification of SCAN VIP and Listener setup passed

Checking to make sure user "oracle" is not in "root" group
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rac2          passed                    does not exist
Result: User "oracle" is not part of "root" group. Check passed

Checking if Clusterware is installed on all nodes...
Check of Clusterware install passed

Checking if CTSS Resource is running on all nodes...
Check: CTSS Resource running on all nodes
  Node Name                             Status
  ------------------------------------  ------------------------
  rac2                                  passed
Result: CTSS resource check passed


Querying CTSS for time offset on all nodes...
Result: Query of CTSS for time offset passed

Check CTSS state started...
Check: CTSS state
  Node Name                             State
  ------------------------------------  ------------------------
  rac2                                  Active
CTSS is in Active state. Proceeding with check of clock time offsets on all nodes...
Reference Time Offset Limit: 1000.0 msecs
Check: Reference Time Offset
  Node Name     Time Offset               Status
  ------------  ------------------------  ------------------------
  rac2          0.0                       passed

Time offset is within the specified limits on the following set of nodes:
"[rac2]"
Result: Check of clock time offsets passed


Oracle Cluster Time Synchronization Services check passed

Post-check for node addition was successful. <<<
[oracle@rac1 ~]$


vi) Check Cluster Nodes

[oracle@rac2 ~]$ olsnodes -n
rac1    1
rac2    2
[oracle@rac2 ~]$


vii) Check TNS Listener

On node RAC2

[oracle@rac2 ~]$ ps -ef | grep tns
root        13     2  0 Jun20 ?        00:00:00 [netns]
oracle   24168     1  0 12:57 ?        00:00:00 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER_SCAN1 -inherit
oracle   24639     1  0 12:57 ?        00:00:00 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER -inherit
oracle   28355 28292  0 13:19 pts/1    00:00:00 grep tns
[oracle@rac2 ~]$


viii) Check ASM Status

On node RAC2

[oracle@rac2 ~]$ srvctl status asm -a
ASM is running on rac2,rac1
ASM is enabled.
[oracle@rac2 ~]$


ix) Check OCR

On node RAC2

[oracle@rac2 ~]$ ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       4076
         Available space (kbytes) :     258044
         ID                       : 1037097601
         Device/File Name         :      +DATA
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check bypassed due to non-privileged user

[oracle@rac2 ~]$


x) Check Vote disk

On node RAC2

[oracle@rac2 ~]$ crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   a785661b81264f8ebfa8538128d4e1fe (/dev/oracleasm/disks/DISK1) [DATA]
 2. ONLINE   7a41922f13254f61bf4cee3f53b9aa74 (/dev/oracleasm/disks/DISK2) [DATA]
 3. ONLINE   492244b5021f4fc7bf7d75b74cfe841a (/dev/oracleasm/disks/DISK3) [DATA]
Located 3 voting disk(s).
[oracle@rac2 ~]$


3. Cluster Node Addition for RDBMS Home.


i) Run addnode.sh script

From Node 1 RAC1
As RDBMS HOME owner

[oracle@rac1 ~]$ cd /u01/app/oracle/product/11.2.0/db_1/oui/bin/    <<< This is RDBMS home

[oracle@rac1 bin]$ ./addNode.sh -silent "CLUSTER_NEW_NODES={rac2}"
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 5670 MB    Passed
Oracle Universal Installer, Version 11.2.0.3.0 Production
Copyright (C) 1999, 2011, Oracle. All rights reserved.


Performing tests to see whether nodes rac2 are available
............................................................... 100% Done.

........
-----------------------------------------------------------------------------
Cluster Node Addition Summary
Global Settings
   Source: /u01/app/oracle/product/11.2.0/db_1
   New Nodes
Space Requirements
   New Nodes
      rac2
         /u01: Required 5.04GB : Available 10.57GB
Installed Products
   Product Names
      Oracle Database 11g 11.2.0.3.0
      Sun JDK 1.5.0.30.03
      Installer SDK Component 11.2.0.3.0
      Oracle One-Off Patch Installer 11.2.0.1.7
      Oracle Universal Installer 11.2.0.3.0
      Oracle USM Deconfiguration 11.2.0.3.0
      Oracle Configuration Manager Deconfiguration 10.3.1.0.0
      Oracle DBCA Deconfiguration 11.2.0.3.0
      Oracle RAC Deconfiguration 11.2.0.3.0
      Oracle Database Deconfiguration 11.2.0.3.0
      Oracle Configuration Manager Client 10.3.2.1.0
      Oracle Configuration Manager 10.3.5.0.1
      Oracle ODBC Driverfor Instant Client 11.2.0.3.0
      LDAP Required Support Files 11.2.0.3.0
      SSL Required Support Files for InstantClient 11.2.0.3.0
      Bali Share 1.1.18.0.0
      Oracle Extended Windowing Toolkit 3.4.47.0.0
      Oracle JFC Extended Windowing Toolkit 4.2.36.0.0
      Oracle Real Application Testing 11.2.0.3.0
      Oracle Database Vault J2EE Application 11.2.0.3.0
      Oracle Label Security 11.2.0.3.0
      Oracle Data Mining RDBMS Files 11.2.0.3.0
      Oracle OLAP RDBMS Files 11.2.0.3.0
      Oracle OLAP API 11.2.0.3.0
      Platform Required Support Files 11.2.0.3.0
      Oracle Database Vault option 11.2.0.3.0
      Oracle RAC Required Support Files-HAS 11.2.0.3.0
      SQL*Plus Required Support Files 11.2.0.3.0
      Oracle Display Fonts 9.0.2.0.0
      Oracle Ice Browser 5.2.3.6.0
      Oracle JDBC Server Support Package 11.2.0.3.0
      Oracle SQL Developer 11.2.0.3.0
      Oracle Application Express 11.2.0.3.0
      XDK Required Support Files 11.2.0.3.0
      RDBMS Required Support Files for Instant Client 11.2.0.3.0
      SQLJ Runtime 11.2.0.3.0
      Database Workspace Manager 11.2.0.3.0
      RDBMS Required Support Files Runtime 11.2.0.3.0
      Oracle Globalization Support 11.2.0.3.0
      Exadata Storage Server 11.2.0.1.0
      Provisioning Advisor Framework 10.2.0.4.3
      Enterprise Manager Database Plugin -- Repository Support 11.2.0.3.0
      Enterprise Manager Repository Core Files 10.2.0.4.4
      Enterprise Manager Database Plugin -- Agent Support 11.2.0.3.0
      Enterprise Manager Grid Control Core Files 10.2.0.4.4
      Enterprise Manager Common Core Files 10.2.0.4.4
      Enterprise Manager Agent Core Files 10.2.0.4.4
      RDBMS Required Support Files 11.2.0.3.0
      regexp 2.1.9.0.0
      Agent Required Support Files 10.2.0.4.3
      Oracle 11g Warehouse Builder Required Files 11.2.0.3.0
      Oracle Notification Service (eONS) 11.2.0.3.0
      Oracle Text Required Support Files 11.2.0.3.0
      Parser Generator Required Support Files 11.2.0.3.0
      Oracle Database 11g Multimedia Files 11.2.0.3.0
      Oracle Multimedia Java Advanced Imaging 11.2.0.3.0
      Oracle Multimedia Annotator 11.2.0.3.0
      Oracle JDBC/OCI Instant Client 11.2.0.3.0
      Oracle Multimedia Locator RDBMS Files 11.2.0.3.0
      Precompiler Required Support Files 11.2.0.3.0
      Oracle Core Required Support Files 11.2.0.3.0
      Sample Schema Data 11.2.0.3.0
      Oracle Starter Database 11.2.0.3.0
      Oracle Message Gateway Common Files 11.2.0.3.0
      Oracle XML Query 11.2.0.3.0
      XML Parser for Oracle JVM 11.2.0.3.0
      Oracle Help For Java 4.2.9.0.0
      Installation Plugin Files 11.2.0.3.0
      Enterprise Manager Common Files 10.2.0.4.3
      Expat libraries 2.0.1.0.1
      Deinstallation Tool 11.2.0.3.0
      Oracle Quality of Service Management (Client) 11.2.0.3.0
      Perl Modules 5.10.0.0.1
      JAccelerator (COMPANION) 11.2.0.3.0
      Oracle Containers for Java 11.2.0.3.0
      Perl Interpreter 5.10.0.0.2
      Oracle Net Required Support Files 11.2.0.3.0
      Secure Socket Layer 11.2.0.3.0
      Oracle Universal Connection Pool 11.2.0.3.0
      Oracle JDBC/THIN Interfaces 11.2.0.3.0
      Oracle Multimedia Client Option 11.2.0.3.0
      Oracle Java Client 11.2.0.3.0
      Character Set Migration Utility 11.2.0.3.0
      Oracle Code Editor 1.2.1.0.0I
      PL/SQL Embedded Gateway 11.2.0.3.0
      OLAP SQL Scripts 11.2.0.3.0
      Database SQL Scripts 11.2.0.3.0
      Oracle Locale Builder 11.2.0.3.0
      Oracle Globalization Support 11.2.0.3.0
      SQL*Plus Files for Instant Client 11.2.0.3.0
      Required Support Files 11.2.0.3.0
      Oracle Database User Interface 2.2.13.0.0
      Oracle ODBC Driver 11.2.0.3.0
      Oracle Notification Service 11.2.0.3.0
      XML Parser for Java 11.2.0.3.0
      Oracle Security Developer Tools 11.2.0.3.0
      Oracle Wallet Manager 11.2.0.3.0
      Cluster Verification Utility Common Files 11.2.0.3.0
      Oracle Clusterware RDBMS Files 11.2.0.3.0
      Oracle UIX 2.2.24.6.0
      Enterprise Manager plugin Common Files 11.2.0.3.0
      HAS Common Files 11.2.0.3.0
      Precompiler Common Files 11.2.0.3.0
      Installation Common Files 11.2.0.3.0
      Oracle Help for the  Web 2.0.14.0.0
      Oracle LDAP administration 11.2.0.3.0
      Buildtools Common Files 11.2.0.3.0
      Assistant Common Files 11.2.0.3.0
      Oracle Recovery Manager 11.2.0.3.0
      PL/SQL 11.2.0.3.0
      Generic Connectivity Common Files 11.2.0.3.0
      Oracle Database Gateway for ODBC 11.2.0.3.0
      Oracle Programmer 11.2.0.3.0
      Oracle Database Utilities 11.2.0.3.0
      Enterprise Manager Agent 10.2.0.4.3
      SQL*Plus 11.2.0.3.0
      Oracle Netca Client 11.2.0.3.0
      Oracle Multimedia Locator 11.2.0.3.0
      Oracle Call Interface (OCI) 11.2.0.3.0
      Oracle Multimedia 11.2.0.3.0
      Oracle Net 11.2.0.3.0
      Oracle XML Development Kit 11.2.0.3.0
      Database Configuration and Upgrade Assistants 11.2.0.3.0
      Oracle JVM 11.2.0.3.0
      Oracle Advanced Security 11.2.0.3.0
      Oracle Internet Directory Client 11.2.0.3.0
      Oracle Enterprise Manager Console DB 11.2.0.3.0
      HAS Files for DB 11.2.0.3.0
      Oracle Net Listener 11.2.0.3.0
      Oracle Text 11.2.0.3.0
      Oracle Net Services 11.2.0.3.0
      Oracle Database 11g 11.2.0.3.0
      Oracle OLAP 11.2.0.3.0
      Oracle Spatial 11.2.0.3.0
      Oracle Partitioning 11.2.0.3.0
      Enterprise Edition Options 11.2.0.3.0
-----------------------------------------------------------------------------


Instantiating scripts for add node (Sunday, June 21, 2015 2:06:51 PM IST)
.                                                                 1% Done.
Instantiation of add node scripts complete

Copying to remote nodes (Sunday, June 21, 2015 2:08:47 PM IST)
...............................................................................................                                 96% Done.
Home copied to new nodes

Saving inventory on nodes (Sunday, June 21, 2015 8:10:31 PM IST)
.                                                               100% Done.
Save inventory complete
WARNING:
The following configuration scripts need to be executed as the "root" user in each new cluster node. Each script in the list below is followed by a list of nodes.
/u01/app/oracle/product/11.2.0/db_1/root.sh #On nodes rac2
To execute the configuration scripts:
    1. Open a terminal window
    2. Log in as "root"
    3. Run the scripts in each cluster node

The Cluster Node Addition of /u01/app/oracle/product/11.2.0/db_1 was successful.
Please check '/tmp/silentInstall.log' for more details.
[oracle@rac1 bin]$


ii) root.sh #On nodes rac2 from RDBMS home

[root@rac2 ~]# /u01/app/oracle/product/11.2.0/db_1/root.sh
Performing root user operation for Oracle 11g

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/oracle/product/11.2.0/db_1

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Finished product-specific root actions.
[root@rac2 ~]#


4. Add Instance to Database through Command-Line or you can add via dbca.


i) Pre-task

On RAC2
As RDBMS Home owner

[oracle@rac2 ~]$ cd /u01/app/oracle/product/11.2.0/db_1/dbs
[oracle@rac2 dbs]$ ls -ltr
total 28
-rw-r----- 1 oracle oinstall 1536 Jun 21 19:56 orapwnike1 <<
-rw-r----- 1 oracle oinstall  161 Jun 21 19:56 initDBUA5216639.ora
-rw-r----- 1 oracle oinstall   36 Jun 21 19:56 initnike1.ora << 
-rw-rw---- 1 oracle oinstall 1544 Jun 21 19:56 hc_DBUA5216639.dat 
-rw-rw---- 1 oracle oinstall 1544 Jun 21 19:56 hc_nike1.dat 
-rw-r--r-- 1 oracle oinstall 2851 Jun 21 19:56 init.ora 
-rw-r----- 1 oracle oinstall 1536 Jun 21 19:56 orapwDBUA5216639 [oracle@rac2 dbs]$ 
[oracle@rac2 dbs]$ mv initnike1.ora initnike2.ora
[oracle@rac2 dbs]$ mv orapwnike1 orapwnike2 
[oracle@rac2 dbs]$ ls -ltr 
total 28 
-rw-r----- 1 oracle oinstall 1536 Jun 21 19:56 orapwnike2 
-rw-r----- 1 oracle oinstall  161 Jun 21 19:56 initDBUA5216639.ora 
-rw-r----- 1 oracle oinstall   36 Jun 21 19:56 initnike2.ora 
-rw-rw---- 1 oracle oinstall 1544 Jun 21 19:56 hc_DBUA5216639.dat 
-rw-rw---- 1 oracle oinstall 1544 Jun 21 19:56 hc_nike1.dat 
-rw-r--r-- 1 oracle oinstall 2851 Jun 21 19:56 init.ora 
-rw-r----- 1 oracle oinstall 1536 Jun 21 19:56 orapwDBUA5216639 [oracle@rac2 dbs]$ 
[oracle@rac2 dbs]$ cat initnike2.ora 
SPFILE='+DATA1/nike/spfilenike.ora' 
[oracle@rac2 dbs]$ 
[oracle@rac2 dbs]$ echo "nike2:/u01/app/oracle/product/11.2.0/db_1:N" >> /etc/oratab
[oracle@rac2 dbs]$ echo "nike:/u01/app/oracle/product/11.2.0/db_1:N" >> /etc/oratab

cat /etc/oratab
..
#
+ASM2:/u01/app/11.2.0/grid:N            # line added by Agent
nike2:/u01/app/oracle/product/11.2.0/db_1:N
nike:/u01/app/oracle/product/11.2.0/db_1:N


[oracle@rac2 ~]$ mkdir -p /u01/app/oracle/admin/nike/adump
[oracle@rac2 ~]$ mkdir -p /u01/app/oracle/admin/nike/dpdump
[oracle@rac2 ~]$ mkdir -p /u01/app/oracle/admin/nike/hdump
[oracle@rac2 ~]$ mkdir -p /u01/app/oracle/admin/nike/pfile


ii) Add redo thread

On RAC1, As ORACLE HOME owner

SQL> set lines 180
SQL> col MEMBER for a60
SQL> select b.thread#, a.group#, a.member, b.bytes FROM v$logfile a, v$log b WHERE a.group# = b.group#;

   THREAD#     GROUP# MEMBER                                                            BYTES
---------- ---------- ------------------------------------------------------------ ----------
         1          2 +DATA1/nike/onlinelog/group_2.290.847761035                    52428800
         1          1 +DATA1/nike/onlinelog/group_1.289.847761031                    52428800

SQL>

SQL> alter database add logfile thread 2 group 3 ('+DATA1') size 52428800, group 4 ('+DATA1') size 52428800;

Database altered.

SQL>

SQL> select b.thread#, a.group#, a.member, b.bytes FROM v$logfile a, v$log b WHERE a.group# = b.group#;

   THREAD#     GROUP# MEMBER                                                            BYTES
---------- ---------- ------------------------------------------------------------ ----------
         1          2 +DATA1/nike/onlinelog/group_2.290.847761035                    52428800
         1          1 +DATA1/nike/onlinelog/group_1.289.847761031                    52428800
         2          3 +DATA1/nike/onlinelog/group_3.271.883012175                    52428800
         2          4 +DATA1/nike/onlinelog/group_4.270.883012181                    52428800

SQL>

SQL> alter database enable public thread 2;

Database altered.

SQL>


iii) Add undo tablespace

On node RAC1

SQL> set pages 0
SQL> set long 9999999
SQL> select dbms_metadata.get_ddl('TABLESPACE','UNDOTBS1') from dual;

  CREATE UNDO TABLESPACE "UNDOTBS1" DATAFILE
  SIZE 26214400
  AUTOEXTEND ON NEXT 5242880 MAXSIZE 32767M
  BLOCKSIZE 8192
  EXTENT MANAGEMENT LOCAL AUTOALLOCATE
   ALTER DATABASE DATAFILE
  '+DATA1/nike/datafile/undotbs1.357.847760839' RESIZE 41943040


SQL> create undo tablespace undotbs2 datafile '+DATA1' size 25M autoextend on next 5m maxsize 40M;

Tablespace created.

SQL>

SQL> alter system set undo_tablespace=undotbs2 scope=spfile sid='nike2';

System altered.

SQL> alter system set instance_number=2 scope=spfile sid='nike2';

System altered.

SQL> alter system set thread=2 scope=spfile sid='nike2';

System altered.

SQL>

SQL> alter system set cluster_database_instances=2 scope=spfile sid='*';

System altered.

SQL>

SQL> select inst_id,name,value from gv$parameter where name like 'undo_table%';

INST_ID NAME                 VALUE
------- -------------------- ---------------
      2 undo_tablespace      UNDOTBS2
      1 undo_tablespace      UNDOTBS1

SQL>


iv) Add instance to OCR

From node RAC1 as ORACLE_HOME owner

[oracle@rac1 ~]$ which srvctl
/u01/app/oracle/product/11.2.0/db_1/bin/srvctl   <<< You should run from RDBMS Home
[oracle@rac1 ~]$
[oracle@rac1 ~]$ srvctl add instance -d nike -i nike2 -n rac2
[oracle@rac1 ~]$
[oracle@rac1 ~]$ srvctl status database -d nike -v
Instance nike1 is running on node rac1 with online services nike_srv. Instance status: Open.
Instance nike2 is not running on node rac2 <<<
[oracle@rac1 ~]$ 
[oracle@rac1 ~]$ srvctl start instance -d nike -i nike2
[oracle@rac1 ~]$
[oracle@rac1 ~]$ srvctl status database -d nike -v
Instance nike1 is running on node rac1 with online services nike_srv. Instance status: Open.
Instance nike2 is running on node rac2. Instance status: Open. <<<
[oracle@rac1 ~]$
[oracle@rac1 ~]$ srvctl config database -d nike
Database unique name: nike
Database name: nike
Oracle home: /u01/app/oracle/product/11.2.0/db_1 <<<
Oracle user: oracle
Spfile: +DATA1/nike/spfilenike.ora
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: nike
Database instances: nike1,nike2  <<<<
Disk Groups: DATA1
Mount point paths:
Services: nike_srv <<
Type: RAC
Database is administrator managed <<< [oracle@rac1 ~]$ SQL> col host_name format a22
SQL> set lines 180
SQL> select host_name, inst_id, instance_name, status, to_char(startup_time, 'DD-MON-YYYY HH24:MI:SS') as "START_TIME" from gv$instance order by inst_id;

HOST_NAME                 INST_ID INSTANCE_NAME    STATUS       START_TIME
---------------------- ---------- ---------------- ------------ --------------------
rac1.rajasekhar.com             1 nike1            OPEN         21-JUN-2015 11:38:48
rac2.rajasekhar.com             2 nike2            OPEN         22-JUN-2015 01:25:13

SQL>


v) Add New Instance to service via srvctl or you can add via dbca

From node RAC1, as ORACLE_HOME owner

[oracle@rac1 ~]$ which srvctl
/u01/app/oracle/product/11.2.0/db_1/bin/srvctl   <<< You should run from RDBMS Home
[oracle@rac1 ~]$
[oracle@rac1 ~]$ srvctl add service -d nike -s nike_srv -a nike2 -u  <<<< -a means available instance.
[oracle@rac1 ~]$
[oracle@rac1 ~]$ srvctl config service -d nike -s nike_srv
Service name: nike_srv <<<
Service is enabled
Server pool: nike_nike_srv
Cardinality: 1
Disconnect: false
Service role: PRIMARY
Management policy: AUTOMATIC
DTP transaction: false
AQ HA notifications: false
Failover type: NONE
Failover method: NONE
TAF failover retries: 0
TAF failover delay: 0
Connection Load Balancing Goal: LONG
Runtime Load Balancing Goal: NONE
TAF policy specification: NONE
Edition:
Preferred instances: nike1
Available instances: nike2 <<<<
[oracle@rac1 ~]$

Note: If you want add instance as preferred then follow below

[oracle@rac1 ~]$ which srvctl
/u01/app/oracle/product/11.2.0/db_1/bin/srvctl   <<< You should run from RDBMS Home
[oracle@rac1 ~]$
[oracle@rac1 ~]$ srvctl add service -d nike -s nike_srv -r nike2 -u
[oracle@rac1 ~]$ srvctl config service -d nike -s nike_srv
Service name: nike_srv
Service is enabled
Server pool: nike_nike_srv
Cardinality: 2
Disconnect: false
Service role: PRIMARY
Management policy: AUTOMATIC
DTP transaction: false
AQ HA notifications: false
Failover type: NONE
Failover method: NONE
TAF failover retries: 0
TAF failover delay: 0
Connection Load Balancing Goal: LONG
Runtime Load Balancing Goal: NONE
TAF policy specification: NONE
Edition:
Preferred instances: nike2,nike1 <<<
Available instances:
[oracle@rac1 ~]$ srvctl status service -d nike
Service nike_srv is running on instance(s) nike1 <<<< 
[oracle@rac1 ~]$ srvctl start service -d nike
[oracle@rac1 ~]$ srvctl status service -d nike -v
Service nike_srv is running on instance(s) nike1,nike2 <<<<
[oracle@rac1 ~]$

Note: Modify tnsnames.ora file if required.


vi) Check the cluster stack

[oracle@rac1 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       rac1
               ONLINE  ONLINE       rac2
ora.DATA1.dg
               ONLINE  ONLINE       rac1
               ONLINE  ONLINE       rac2
ora.LISTENER.lsnr
               ONLINE  ONLINE       rac1
               ONLINE  ONLINE       rac2
ora.asm
               ONLINE  ONLINE       rac1                     Started
               ONLINE  ONLINE       rac2                     Started
ora.gsd
               OFFLINE OFFLINE      rac1
               OFFLINE OFFLINE      rac2
ora.net1.network
               ONLINE  ONLINE       rac1
               ONLINE  ONLINE       rac2
ora.ons
               ONLINE  ONLINE       rac1
               ONLINE  ONLINE       rac2
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       rac2
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       rac1
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       rac1
ora.cvu
      1        ONLINE  ONLINE       rac1
ora.nike.db
      1        ONLINE  ONLINE       rac1                     Open
      2        ONLINE  ONLINE       rac2                     Open
ora.nike.nike_srv.svc
      1        ONLINE  ONLINE       rac1
ora.oc4j
      1        ONLINE  ONLINE       rac1
ora.rac1.vip
      1        ONLINE  ONLINE       rac1
ora.rac2.vip
      1        ONLINE  ONLINE       rac2
ora.scan1.vip
      1        ONLINE  ONLINE       rac2
ora.scan2.vip
      1        ONLINE  ONLINE       rac1
ora.scan3.vip
      1        ONLINE  ONLINE       rac1
[oracle@rac1 ~]$

Caution: Your use of any information or materials on this website is entirely at your own risk. It is provided for educational purposes only. It has been tested internally, however, we do not guarantee that it will work for you. Ensure that you run it in your test environment before using.

Still page under construction !!! 🙂