Tag Archives: after server reboot

CSSD won’t start automatically

Cause

After server reboot, CSSD and ASM are not came up automatically.

Data Collection

[oracle@Linux4 ~]$ ps -ef | grep d.bin
oracle    4233     1  0 09:22 ?        00:00:03 /oracle11g/app/11.2.0/oracle/grid/bin/ohasd.bin reboot
oracle    4794     1  0 09:23 ?        00:00:01 /oracle11g/app/11.2.0/oracle/grid/bin/oraagent.bin
oracle    4810     1  0 09:23 ?        00:00:00 /oracle11g/app/11.2.0/oracle/grid/bin/evmd.bin
oracle    4812     1  0 09:23 ?        00:00:00 /oracle11g/app/11.2.0/oracle/grid/bin/tnslsnr LISTENER -inherit
oracle    4855  4810  0 09:23 ?        00:00:00 /oracle11g/app/11.2.0/oracle/grid/bin/evmlogger.bin -o 

/oracle11g/app/11.2.0/oracle/grid/evm/log/evmlogger.info -l /oracle11g/app/11.2.0/oracle/grid/evm/log/evmlogger.log
oracle    4942  4748  0 09:38 pts/1    00:00:00 grep d.bin
[oracle@Linux4 ~]$

[oracle@Linux4 ~]$ crsctl stat res -t -init
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               OFFLINE OFFLINE      linux4
ora.LISTENER.lsnr
               ONLINE  ONLINE       linux4
ora.asm
               OFFLINE OFFLINE      linux4                   Instance Shutdown <----
ora.ons
               OFFLINE OFFLINE      linux4
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
      1        ONLINE  OFFLINE <-----
ora.diskmon
      1        OFFLINE OFFLINE
ora.evmd
      1        ONLINE  ONLINE       linux4
[oracle@Linux4 ~]$

[oracle@Linux4 ~]$ crsctl stat res ora.cssd -p
NAME=ora.cssd
TYPE=ora.cssd.type
ACL=owner:oracle:rwx,pgrp:oinstall:rwx,other::r--
ACTION_FAILURE_TEMPLATE=
ACTION_SCRIPT=
ACTIVE_PLACEMENT=0
AGENT_FILENAME=%CRS_HOME%/bin/cssdagent%CRS_EXE_SUFFIX%
AGENT_HB_INTERVAL=0
AGENT_HB_MISCOUNT=10
AUTO_START=never <----
CARDINALITY=1
CHECK_ARGS=
CHECK_COMMAND=
CHECK_INTERVAL=30
CLEAN_ARGS=abort
CLEAN_COMMAND=
CSSD_MODE=
CSSD_PATH=%CRS_HOME%/bin/ocssd%CRS_EXE_SUFFIX%
CSS_USER=oracle
DAEMON_LOGGING_LEVELS=CSSD=2,GIPCNM=2,GIPCGM=2,GIPCCM=2,CLSF=0,SKGFD=0,GPNP=1,OLR=0
DAEMON_TRACING_LEVELS=CSSD=0,GIPCNM=0,GIPCGM=0,GIPCCM=0,CLSF=0,SKGFD=0,GPNP=0,OLR=0
DEFAULT_TEMPLATE=
DEGREE=1
DESCRIPTION="Resource type for CSSD"
DETACHED=true
ENABLED=1
ENV_OPTS=
FAILOVER_DELAY=0
FAILURE_INTERVAL=3
FAILURE_THRESHOLD=5
HOSTING_MEMBERS=
INTERVAL_FS=3
LOAD=1
LOGGING_LEVEL=1
NOT_RESTARTING_TEMPLATE=
OFFLINE_CHECK_INTERVAL=0
OMON_INITRATE=1000
OMON_POLLRATE=500
ORA_OPROCD_MODE=
ORA_VERSION=11.2.0.4.0
PID_FILE=
PLACEMENT=balanced
PROCD_TIMEOUT=1000
PROCESS_TO_MONITOR=
PROFILE_CHANGE_TEMPLATE=
REBOOT_OPTS=
RESTART_ATTEMPTS=5
SCRIPT_TIMEOUT=600
SERVER_POOLS=
START_ARGS=
START_COMMAND=
START_DEPENDENCIES=weak(concurrent:ora.diskmon)
START_TIMEOUT=600
STATE_CHANGE_TEMPLATE=
STOP_ARGS=
STOP_COMMAND=
STOP_DEPENDENCIES=hard(shutdown:ora.diskmon)
STOP_TIMEOUT=900
TIMEOUT_FS=0
UNRESPONSIVE_TIMEOUT=180
UPTIME_THRESHOLD=1m
USR_ORA_ENV=
VMON_INITLIMIT=16
VMON_INITRATE=500
VMON_POLLRATE=500

[oracle@Linux4 ~]$

[oracle@Linux4 ~]$ crsctl stat res ora.asm -p
NAME=ora.asm
TYPE=ora.asm.type
ACL=owner:oracle:rwx,pgrp:oinstall:rwx,other::r--
ACTION_FAILURE_TEMPLATE=
ACTION_SCRIPT=
AGENT_FILENAME=%CRS_HOME%/bin/oraagent%CRS_EXE_SUFFIX%
ALIAS_NAME=
ASM_DISKSTRING=/dev/oracleasm/disks/*
AUTO_START=restore <----
CHECK_INTERVAL=1
CHECK_TIMEOUT=30
DEFAULT_TEMPLATE=PROPERTY(RESOURCE_CLASS=asm) ELEMENT(INSTANCE_NAME= %GEN_USR_ORA_INST_NAME%)
DEGREE=1
DESCRIPTION=Oracle ASM resource
ENABLED=1
GEN_USR_ORA_INST_NAME=+ASM
LOAD=1
LOGGING_LEVEL=1
NLS_LANG=
NOT_RESTARTING_TEMPLATE=
OFFLINE_CHECK_INTERVAL=0
PROFILE_CHANGE_TEMPLATE=
RESTART_ATTEMPTS=5
SCRIPT_TIMEOUT=60
SPFILE=+DATA/asm/asmparameterfile/registry.253.894413045
START_DEPENDENCIES=hard(ora.cssd) weak(ora.LISTENER.lsnr)
START_TIMEOUT=900
STATE_CHANGE_TEMPLATE=
STOP_DEPENDENCIES=hard(ora.cssd)
STOP_TIMEOUT=600
TYPE_VERSION=1.2
UPTIME_THRESHOLD=1d
USR_ORA_ENV=
USR_ORA_INST_NAME=+ASM
USR_ORA_OPEN_MODE=mount
USR_ORA_OPI=false
USR_ORA_STOP_MODE=immediate
VERSION=11.2.0.4.0

[oracle@Linux4 ~]$

Action Plan

Start the resources manually

[oracle@Linux4 ~]$ crsctl start res ora.cssd -init
CRS-2672: Attempting to start 'ora.cssd' on 'linux4'
CRS-2672: Attempting to start 'ora.diskmon' on 'linux4'
CRS-2676: Start of 'ora.diskmon' on 'linux4' succeeded
CRS-2676: Start of 'ora.cssd' on 'linux4' succeeded
[oracle@Linux4 ~]$
[oracle@Linux4 ~]$ ps -ef | grep cssd.bin
oracle    5184     1  0 10:19 ?        00:00:00 /oracle11g/app/11.2.0/oracle/grid/bin/ocssd.bin
oracle    5207  4748  0 10:20 pts/1    00:00:00 grep cssd.bin
[oracle@Linux4 ~]$
[oracle@Linux4 ~]$ srvctl start asm
[oracle@Linux4 ~]$
[oracle@Linux4 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       linux4
ora.LISTENER.lsnr
               ONLINE  ONLINE       linux4
ora.asm
               ONLINE  ONLINE       linux4                   Started
ora.ons
               OFFLINE OFFLINE      linux4
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
      1        ONLINE  ONLINE       linux4
ora.diskmon
      1        OFFLINE OFFLINE
ora.evmd
      1        ONLINE  ONLINE       linux4
[oracle@Linux4 ~]$

We need to change the auto_start attribute of a resource to value ‘always’ or ‘1’ to start resource automatically after server reboot.

Run below commands as GI home owner.

[oracle@Linux4 ~]$ crsctl modify resource "ora.cssd" -attr "AUTO_START=always"
[oracle@Linux4 ~]$ crsctl modify resource "ora.asm" -attr "AUTO_START=always"
[oracle@Linux4 ~]$

[oracle@Linux4 ~]$ crsctl stat res ora.asm -p | grep "AUTO_START"
AUTO_START=always
[oracle@Linux4 ~]$ crsctl stat res ora.cssd -p | grep "AUTO_START"
AUTO_START=always
[oracle@Linux4 ~]$

Additional Information:

always (1)
 Causes the resource to restart when the node restarts regardless of the resource’s state when the node stopped.

restore (0)
 Does not start the resource at restart time if it was in an offline state, such as STATE=OFFLINE, TARGET=OFFLINE, when the node stopped. The 

resource is restored to its state when the node went down. The resource is started only if it was online before and not otherwise.

never (2)
 Oracle Clusterware never restarts the resource regardless of the resource’s state when the node is stopped.

Caution: Your use of any information or materials on this website is entirely at your own risk. It is provided for educational purposes only. It has been tested internally, however, we do not guarantee that it will work for you. Ensure that you run it in your test environment before using.