Cause
After server reboot, CSSD and ASM are not came up automatically.
Data Collection
[oracle@Linux4 ~]$ ps -ef | grep d.bin oracle 4233 1 0 09:22 ? 00:00:03 /oracle11g/app/11.2.0/oracle/grid/bin/ohasd.bin reboot oracle 4794 1 0 09:23 ? 00:00:01 /oracle11g/app/11.2.0/oracle/grid/bin/oraagent.bin oracle 4810 1 0 09:23 ? 00:00:00 /oracle11g/app/11.2.0/oracle/grid/bin/evmd.bin oracle 4812 1 0 09:23 ? 00:00:00 /oracle11g/app/11.2.0/oracle/grid/bin/tnslsnr LISTENER -inherit oracle 4855 4810 0 09:23 ? 00:00:00 /oracle11g/app/11.2.0/oracle/grid/bin/evmlogger.bin -o /oracle11g/app/11.2.0/oracle/grid/evm/log/evmlogger.info -l /oracle11g/app/11.2.0/oracle/grid/evm/log/evmlogger.log oracle 4942 4748 0 09:38 pts/1 00:00:00 grep d.bin [oracle@Linux4 ~]$ [oracle@Linux4 ~]$ crsctl stat res -t -init -------------------------------------------------------------------------------- NAME TARGET STATE SERVER STATE_DETAILS -------------------------------------------------------------------------------- Local Resources -------------------------------------------------------------------------------- ora.DATA.dg OFFLINE OFFLINE linux4 ora.LISTENER.lsnr ONLINE ONLINE linux4 ora.asm OFFLINE OFFLINE linux4 Instance Shutdown <---- ora.ons OFFLINE OFFLINE linux4 -------------------------------------------------------------------------------- Cluster Resources -------------------------------------------------------------------------------- ora.cssd 1 ONLINE OFFLINE <----- ora.diskmon 1 OFFLINE OFFLINE ora.evmd 1 ONLINE ONLINE linux4 [oracle@Linux4 ~]$ [oracle@Linux4 ~]$ crsctl stat res ora.cssd -p NAME=ora.cssd TYPE=ora.cssd.type ACL=owner:oracle:rwx,pgrp:oinstall:rwx,other::r-- ACTION_FAILURE_TEMPLATE= ACTION_SCRIPT= ACTIVE_PLACEMENT=0 AGENT_FILENAME=%CRS_HOME%/bin/cssdagent%CRS_EXE_SUFFIX% AGENT_HB_INTERVAL=0 AGENT_HB_MISCOUNT=10 AUTO_START=never <---- CARDINALITY=1 CHECK_ARGS= CHECK_COMMAND= CHECK_INTERVAL=30 CLEAN_ARGS=abort CLEAN_COMMAND= CSSD_MODE= CSSD_PATH=%CRS_HOME%/bin/ocssd%CRS_EXE_SUFFIX% CSS_USER=oracle DAEMON_LOGGING_LEVELS=CSSD=2,GIPCNM=2,GIPCGM=2,GIPCCM=2,CLSF=0,SKGFD=0,GPNP=1,OLR=0 DAEMON_TRACING_LEVELS=CSSD=0,GIPCNM=0,GIPCGM=0,GIPCCM=0,CLSF=0,SKGFD=0,GPNP=0,OLR=0 DEFAULT_TEMPLATE= DEGREE=1 DESCRIPTION="Resource type for CSSD" DETACHED=true ENABLED=1 ENV_OPTS= FAILOVER_DELAY=0 FAILURE_INTERVAL=3 FAILURE_THRESHOLD=5 HOSTING_MEMBERS= INTERVAL_FS=3 LOAD=1 LOGGING_LEVEL=1 NOT_RESTARTING_TEMPLATE= OFFLINE_CHECK_INTERVAL=0 OMON_INITRATE=1000 OMON_POLLRATE=500 ORA_OPROCD_MODE= ORA_VERSION=11.2.0.4.0 PID_FILE= PLACEMENT=balanced PROCD_TIMEOUT=1000 PROCESS_TO_MONITOR= PROFILE_CHANGE_TEMPLATE= REBOOT_OPTS= RESTART_ATTEMPTS=5 SCRIPT_TIMEOUT=600 SERVER_POOLS= START_ARGS= START_COMMAND= START_DEPENDENCIES=weak(concurrent:ora.diskmon) START_TIMEOUT=600 STATE_CHANGE_TEMPLATE= STOP_ARGS= STOP_COMMAND= STOP_DEPENDENCIES=hard(shutdown:ora.diskmon) STOP_TIMEOUT=900 TIMEOUT_FS=0 UNRESPONSIVE_TIMEOUT=180 UPTIME_THRESHOLD=1m USR_ORA_ENV= VMON_INITLIMIT=16 VMON_INITRATE=500 VMON_POLLRATE=500 [oracle@Linux4 ~]$ [oracle@Linux4 ~]$ crsctl stat res ora.asm -p NAME=ora.asm TYPE=ora.asm.type ACL=owner:oracle:rwx,pgrp:oinstall:rwx,other::r-- ACTION_FAILURE_TEMPLATE= ACTION_SCRIPT= AGENT_FILENAME=%CRS_HOME%/bin/oraagent%CRS_EXE_SUFFIX% ALIAS_NAME= ASM_DISKSTRING=/dev/oracleasm/disks/* AUTO_START=restore <---- CHECK_INTERVAL=1 CHECK_TIMEOUT=30 DEFAULT_TEMPLATE=PROPERTY(RESOURCE_CLASS=asm) ELEMENT(INSTANCE_NAME= %GEN_USR_ORA_INST_NAME%) DEGREE=1 DESCRIPTION=Oracle ASM resource ENABLED=1 GEN_USR_ORA_INST_NAME=+ASM LOAD=1 LOGGING_LEVEL=1 NLS_LANG= NOT_RESTARTING_TEMPLATE= OFFLINE_CHECK_INTERVAL=0 PROFILE_CHANGE_TEMPLATE= RESTART_ATTEMPTS=5 SCRIPT_TIMEOUT=60 SPFILE=+DATA/asm/asmparameterfile/registry.253.894413045 START_DEPENDENCIES=hard(ora.cssd) weak(ora.LISTENER.lsnr) START_TIMEOUT=900 STATE_CHANGE_TEMPLATE= STOP_DEPENDENCIES=hard(ora.cssd) STOP_TIMEOUT=600 TYPE_VERSION=1.2 UPTIME_THRESHOLD=1d USR_ORA_ENV= USR_ORA_INST_NAME=+ASM USR_ORA_OPEN_MODE=mount USR_ORA_OPI=false USR_ORA_STOP_MODE=immediate VERSION=11.2.0.4.0 [oracle@Linux4 ~]$
Action Plan
Start the resources manually
[oracle@Linux4 ~]$ crsctl start res ora.cssd -init CRS-2672: Attempting to start 'ora.cssd' on 'linux4' CRS-2672: Attempting to start 'ora.diskmon' on 'linux4' CRS-2676: Start of 'ora.diskmon' on 'linux4' succeeded CRS-2676: Start of 'ora.cssd' on 'linux4' succeeded [oracle@Linux4 ~]$ [oracle@Linux4 ~]$ ps -ef | grep cssd.bin oracle 5184 1 0 10:19 ? 00:00:00 /oracle11g/app/11.2.0/oracle/grid/bin/ocssd.bin oracle 5207 4748 0 10:20 pts/1 00:00:00 grep cssd.bin [oracle@Linux4 ~]$ [oracle@Linux4 ~]$ srvctl start asm [oracle@Linux4 ~]$ [oracle@Linux4 ~]$ crsctl stat res -t -------------------------------------------------------------------------------- NAME TARGET STATE SERVER STATE_DETAILS -------------------------------------------------------------------------------- Local Resources -------------------------------------------------------------------------------- ora.DATA.dg ONLINE ONLINE linux4 ora.LISTENER.lsnr ONLINE ONLINE linux4 ora.asm ONLINE ONLINE linux4 Started ora.ons OFFLINE OFFLINE linux4 -------------------------------------------------------------------------------- Cluster Resources -------------------------------------------------------------------------------- ora.cssd 1 ONLINE ONLINE linux4 ora.diskmon 1 OFFLINE OFFLINE ora.evmd 1 ONLINE ONLINE linux4 [oracle@Linux4 ~]$
We need to change the auto_start attribute of a resource to value ‘always’ or ‘1’ to start resource automatically after server reboot.
Run below commands as GI home owner.
[oracle@Linux4 ~]$ crsctl modify resource "ora.cssd" -attr "AUTO_START=always" [oracle@Linux4 ~]$ crsctl modify resource "ora.asm" -attr "AUTO_START=always" [oracle@Linux4 ~]$ [oracle@Linux4 ~]$ crsctl stat res ora.asm -p | grep "AUTO_START" AUTO_START=always [oracle@Linux4 ~]$ crsctl stat res ora.cssd -p | grep "AUTO_START" AUTO_START=always [oracle@Linux4 ~]$
Additional Information:
always (1) Causes the resource to restart when the node restarts regardless of the resource’s state when the node stopped. restore (0) Does not start the resource at restart time if it was in an offline state, such as STATE=OFFLINE, TARGET=OFFLINE, when the node stopped. The resource is restored to its state when the node went down. The resource is started only if it was online before and not otherwise. never (2) Oracle Clusterware never restarts the resource regardless of the resource’s state when the node is stopped.
Caution: Your use of any information or materials on this website is entirely at your own risk. It is provided for educational purposes only. It has been tested internally, however, we do not guarantee that it will work for you. Ensure that you run it in your test environment before using.