Troubleshooting RAC Services Startup


Hi guys, I would like to share a very beautful article by Mr. Srikrishna Murthy Annam ( https://learnwithme11g.wordpress.com)

This article gives you the troubleshooting steps for this particular issue. These steps may also help you to troubleshoot if you have issues starting RAC services.

STEP 1: Reboot the rac servers and Check services after re-booting the servers. The services are still down.

STEP 2: Verify storage on both rac servers

[grid@askmdbrac01 ~]$ /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[grid@askmdbrac01 ~]$ /etc/init.d/oracleasm listdisks
CRSVOL1
DISK1
FRADISK1
[grid@askmdbrac01 ~]$ multipath -ll
-bash: multipath: command not found
[grid@askmdbrac01 ~]$ exit
logout
[oracle@askmdbrac01 ~]$ sudo su –
[root@askmdbrac01 ~]# multipath -ll
vpath2 (360050768019280fb280000000000055e) dm-10 IBM,2145
[size=200G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=50][active]
\_ 2:0:1:2 sdf        8:80  [active][ready]
\_ 3:0:1:2 sdl        8:176 [active][ready]
\_ round-robin 0 [prio=10][enabled]
\_ 2:0:0:2 sdc        8:32  [active][ready]
\_ 3:0:0:2 sdi        8:128 [active][ready]
vpath1 (360050768019280fb280000000000055d) dm-12 IBM,2145
[size=200G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=50][active]
\_ 3:0:0:1 sdh        8:112 [active][ready]
\_ round-robin 0 [prio=10][enabled]
\_ 2:0:1:1 sde        8:64  [active][ready]
\_ 3:0:1:1 sdk        8:160 [active][ready]
vpath0 (360050768019280fb280000000000055c) dm-11 IBM,2145
[size=10G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=50][active]
\_ 2:0:1:0 sdd        8:48  [active][ready]
\_ 3:0:1:0 sdj        8:144 [active][ready]
\_ round-robin 0 [prio=10][enabled]
\_ 3:0:0:0 sdg        8:96  [active][ready]
[root@askmdbrac01 ~]# exit[grid@askmdbrac02 ~]$ /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[grid@askmdbrac02 ~]$ /etc/init.d/oracleasm listdisks
CRSVOL1
DISK1
FRADISK1
[grid@askmdbrac02 ~]$ multipath -ll
-bash: multipath: command not found
[grid@askmdbrac02 ~]$ exit
logout
[oracle@askmdbrac02 ~]$ sudo su –
[root@askmdbrac02 ~]# multipath -ll
vpath2 (360050768019280fb280000000000055e) dm-10 IBM,2145
[size=200G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=50][active]
\_ 2:0:1:2 sdf        8:80  [active][ready]
\_ 3:0:0:2 sdi        8:128 [active][ready]
\_ round-robin 0 [prio=10][enabled]
\_ 2:0:0:2 sdc        8:32  [active][ready]
\_ 3:0:1:2 sdl        8:176 [active][ready]
vpath1 (360050768019280fb280000000000055d) dm-12 IBM,2145
[size=200G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=50][active]
\_ 3:0:1:1 sdk        8:160 [active][ready]
\_ round-robin 0 [prio=10][enabled]
\_ 2:0:1:1 sde        8:64  [active][ready]
\_ 3:0:0:1 sdh        8:112 [active][ready]
vpath0 (360050768019280fb280000000000055c) dm-11 IBM,2145
[size=10G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=50][active]
\_ 2:0:1:0 sdd        8:48  [active][ready]
\_ 3:0:0:0 sdg        8:96  [active][ready]
\_ round-robin 0 [prio=10][enabled]
\_ 3:0:1:0 sdj        8:144 [active][ready]
[root@askmdbrac02 ~]#

Observation : Storage is fine and i am able to see ASM disks from both the nodes. Storage is configured with multipathing. I also verified multipathing and status shows fine.

STEP 3: Check high availability services and cluster ready services on both nodes with the following commands

[grid@askmdbrac01 ~]$ crsctl check has
CRS-4638: Oracle High Availability Services is online
[grid@askmdbrac01 ~]$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
[grid@askmdbrac01 ~]$
[grid@askmdbrac01 ~]$ crsctl check cluster
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
[grid@askmdbrac01 ~]$[grid@askmdbrac02 ~]$ crsctl check has
CRS-4638: Oracle High Availability Services is online
[grid@askmdbrac02 ~]$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
[grid@askmdbrac02 ~]$
[grid@askmdbrac02 ~]$ crsctl check cluster
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
[grid@askmdbrac02 ~]$

Observation : The high availability services are up , but crs services are down.

STEP 4: Do ping and nslookup for other rac nodes from each node to make sure that each node in cluster is accessible from every other node. Check this for private ips, virtual ips and for scan ips.

Observation : The ping and nslookup are working fine and all the nodes are reachable from every other node.

STEP 5 : Then verify the log files from grid infrastructure and see if you can find any error messages from log files. The cluster log files are located at $ORACLE_HOME/log/askmdbrac01 and $ORACLE_HOME/log/askmdbrac02.

[cssd(11175)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /o001/home/11.2.0.2/grid/log/askmdbrac01/cssd/ocssd.log
2013-04-02 19:38:27.425
[ohasd(6876)]CRS-2765:Resource ‘ora.diskmon’ has failed on server ‘askmdbrac01′.
2013-04-02 19:38:39.988
[cssd(11175)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /o001/home/11.2.0.2/grid/log/askmdbrac01/cssd/ocssd.log
2013-04-02 19:38:55.007
[cssd(11175)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /o001/home/11.2.0.2/grid/log/askmdbrac01/cssd/ocssd.log
2013-04-02 19:39:10.025
…..
…..
[cssd(11175)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /o001/home/11.2.0.2/grid/log/askmdbrac01/cssd/ocssd.log
2013-04-02 19:48:23.836
[/o001/home/11.2.0.2/grid/bin/cssdagent(11163)]CRS-5818:Aborted command ‘start for resource: ora.cssd 1 1′ for resource ‘ora.cssd’. Details at (:CRSAGF00113:) {0:27:4} in /o001/home/11.2.0.2/grid/log/askmdbrac01/agent/ohasd/oracssdagent_root/oracssdagent_root.log.
2013-04-02 19:48:23.837
[cssd(11175)]CRS-1656:The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in /o001/home/11.2.0.2/grid/log/askmdbrac01/cssd/ocssd.log
2013-04-02 19:48:23.838
[cssd(11175)]CRS-1603:CSSD on node askmdbrac01 shutdown by user.
2013-04-02 19:48:25.730
[cssd(11175)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /o001/home/11.2.0.2/grid/log/askmdbrac01/cssd/ocssd.log
2013-04-02 19:48:29.863
[ohasd(6876)]CRS-2765:Resource ‘ora.cssdmonitor’ has failed on server ‘askmdbrac01′.
2013-04-02 19:48:30.126
[ohasd(6876)]CRS-2771:Maximum restart attempts reached for resource ‘ora.cssd’; will not restart.
2013-04-02 19:48:35.833
[ohasd(6876)]CRS-2765:Resource ‘ora.diskmon’ has failed on server ‘askmdbrac01′.

Observation : Found from the cluster alert log file (alertaskmdbrac01.log) that the cluster services can’t find voting disk. So it could be due the in-accessible asm storage or due to currupted voting disks.

Then i tried to see if i can see voting disks through command.

SETP 6 :  Check the voting disks availability ( 11gR2 has voting disks in asm. I don’t have asm instance up now. So i will get error if i try to query the voting disks on asm)

[root@askmdbrac01 ~]# /o001/home/11.2.0.2/grid/bin/crsctl query css votedisk
Unable to communicate with the Cluster Synchronization Services daemon.
[root@askmdbrac01 ~]#[grid@askmdbrac02 ~]$ /o001/home/11.2.0.2/grid/bin/crsctl query css votedisk
Unable to communicate with the Cluster Synchronization Services daemon.
[grid@askmdbrac02 ~]$

Observation : I am not able to query the voting disks. Now i have only one option to look at the asm storage accessibility.

STEP 7 :  Tried re-enabling the asmlibs as follows on both the nodes …

[root@askmdbrac01 ~]# /etc/init.d/oracleasm restart
Dropping Oracle ASMLib disks:                              [  OK  ]
Shutting down the Oracle ASMLib driver:                    [  OK  ]
[root@askmdbrac01 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: no
Checking if /dev/oracleasm is mounted: no
[root@askmdbrac01 ~]# /etc/init.d/oracleasm start
[root@askmdbrac01 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: no
Checking if /dev/oracleasm is mounted: no
[root@askmdbrac01 ~]# /etc/init.d/oracleasm enable
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@askmdbrac01 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[root@askmdbrac01 ~]# /etc/init.d/oracleasm listdisks
CRSVOL1
DISK1
FRADISK1
[root@askmdbrac01 ~]#[root@askmdbrac02 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[root@askmdbrac02 ~]# /etc/init.d/oracleasm restart
Dropping Oracle ASMLib disks:                              [  OK  ]
Shutting down the Oracle ASMLib driver:                    [  OK  ]
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@askmdbrac02 ~]#  /etc/init.d/oracleasm listdisks
CRSVOL1
DISK1
FRADISK1
[root@askmdbrac02 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[root@askmdbrac02 ~]#

Observation : Not able to start services even after restarting the asmlib.

STEP 8 : Tried to re-configure the asmlib as below …..

[root@askmdbrac01 ~]# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.This will configure the on-boot properties of the Oracle ASM library
driver.  The following questions will determine whether the driver is
loaded on boot and what permissions it will have.  The current values
will be shown in brackets (‘[]’).  Hitting <ENTER> without typing an
answer will keep that current value.  Ctrl-C will abort.Default user to own the driver interface []: grid
Default group to own the driver interface []: asmadmin
Start Oracle ASM library driver on boot (y/n) [y]:
Scan for Oracle ASM disks on boot (y/n) [y]:
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@askmdbrac01 ~]#
[root@askmdbrac01 ~]# /etc/init.d/oracleasm stop
Dropping Oracle ASMLib disks:                              [  OK  ]
Shutting down the Oracle ASMLib driver:                    [FAILED]
[root@askmdbrac01 ~]# /etc/init.d/oracleasm start
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@askmdbrac01 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[root@askmdbrac01 ~]#[root@askmdbrac02 ~]# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library
driver.  The following questions will determine whether the driver is
loaded on boot and what permissions it will have.  The current values
will be shown in brackets (‘[]’).  Hitting <ENTER> without typing an
answer will keep that current value.  Ctrl-C will abort.

Default user to own the driver interface []: grid
Default group to own the driver interface []: asmadmin
Start Oracle ASM library driver on boot (y/n) [y]:
Scan for Oracle ASM disks on boot (y/n) [y]:
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@askmdbrac02 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[root@askmdbrac02 ~]# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library
driver.  The following questions will determine whether the driver is
loaded on boot and what permissions it will have.  The current values
will be shown in brackets (‘[]’).  Hitting <ENTER> without typing an
answer will keep that current value.  Ctrl-C will abort.

Default user to own the driver interface [grid]:
Default group to own the driver interface [asmadmin]:
Start Oracle ASM library driver on boot (y/n) [y]:
Scan for Oracle ASM disks on boot (y/n) [y]:
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@askmdbrac02 ~]#
[root@askmdbrac02 ~]# /etc/init.d/oracleasm stop
Dropping Oracle ASMLib disks:                              [  OK  ]
Shutting down the Oracle ASMLib driver:                    [FAILED]
[root@askmdbrac02 ~]# /etc/init.d/oracleasm start
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@askmdbrac02 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[root@askmdbrac02 ~]#

Observation : Identified that the asm library drivers configuration is wrong and i re-configured with correct user and group. See the difference in the lines when it is asking for prompt on second node. I executed this command twice to show you the difference when it is properly configured and when it is not properly configured.

STEP 9 : Stop and Start cluster services on both the nodes.

[root@askmdbrac01 ~]#  /o001/home/11.2.0.2/grid/bin/crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on ‘askmdbrac01′
CRS-2673: Attempting to stop ‘ora.crsd’ on ‘askmdbrac01′
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on ‘askmdbrac01′
CRS-2673: Attempting to stop ‘ora.LISTENER.lsnr’ on ‘askmdbrac01′
CRS-2673: Attempting to stop ‘ora.racdb.racdb_service.svc’ on ‘askmdbrac01′
CRS-2673: Attempting to stop ‘ora.LISTENER_SCAN1.lsnr’ on ‘askmdbrac01′
CRS-2677: Stop of ‘ora.LISTENER.lsnr’ on ‘askmdbrac01′ succeeded
CRS-2677: Stop of ‘ora.LISTENER_SCAN1.lsnr’ on ‘askmdbrac01′ succeeded
CRS-2673: Attempting to stop ‘ora.scan1.vip’ on ‘askmdbrac01′
CRS-2677: Stop of ‘ora.scan1.vip’ on ‘askmdbrac01′ succeeded
CRS-2672: Attempting to start ‘ora.scan1.vip’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.racdb.racdb_service.svc’ on ‘askmdbrac01′ succeeded
CRS-2673: Attempting to stop ‘ora.askmdbrac01.vip’ on ‘askmdbrac01′
CRS-2673: Attempting to stop ‘ora.CRS.dg’ on ‘askmdbrac01′
CRS-2673: Attempting to stop ‘ora.racdb.db’ on ‘askmdbrac01′
CRS-2677: Stop of ‘ora.askmdbrac01.vip’ on ‘askmdbrac01′ succeeded
CRS-2672: Attempting to start ‘ora.askmdbrac01.vip’ on ‘askmdbrac02′
CRS-2676: Start of ‘ora.scan1.vip’ on ‘askmdbrac02′ succeeded
CRS-2672: Attempting to start ‘ora.LISTENER_SCAN1.lsnr’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.racdb.db’ on ‘askmdbrac01′ succeeded
CRS-2673: Attempting to stop ‘ora.FRA.dg’ on ‘askmdbrac01′
CRS-2673: Attempting to stop ‘ora.PRDSHR_DATA.dg’ on ‘askmdbrac01′
CRS-2676: Start of ‘ora.askmdbrac01.vip’ on ‘askmdbrac02′ succeeded
CRS-2677: Stop of ‘ora.PRDSHR_DATA.dg’ on ‘askmdbrac01′ succeeded
CRS-2676: Start of ‘ora.LISTENER_SCAN1.lsnr’ on ‘askmdbrac02′ succeeded
CRS-2677: Stop of ‘ora.FRA.dg’ on ‘askmdbrac01′ succeeded
CRS-2677: Stop of ‘ora.CRS.dg’ on ‘askmdbrac01′ succeeded
CRS-2673: Attempting to stop ‘ora.asm’ on ‘askmdbrac01′
CRS-2677: Stop of ‘ora.asm’ on ‘askmdbrac01′ succeeded
CRS-2673: Attempting to stop ‘ora.ons’ on ‘askmdbrac01′
CRS-2677: Stop of ‘ora.ons’ on ‘askmdbrac01′ succeeded
CRS-2673: Attempting to stop ‘ora.net1.network’ on ‘askmdbrac01′
CRS-2677: Stop of ‘ora.net1.network’ on ‘askmdbrac01′ succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on ‘askmdbrac01′ has completed
CRS-2677: Stop of ‘ora.crsd’ on ‘askmdbrac01′ succeeded
CRS-2673: Attempting to stop ‘ora.mdnsd’ on ‘askmdbrac01′
CRS-2673: Attempting to stop ‘ora.crf’ on ‘askmdbrac01′
CRS-2673: Attempting to stop ‘ora.ctssd’ on ‘askmdbrac01′
CRS-2673: Attempting to stop ‘ora.evmd’ on ‘askmdbrac01′
CRS-2673: Attempting to stop ‘ora.asm’ on ‘askmdbrac01′
CRS-2677: Stop of ‘ora.mdnsd’ on ‘askmdbrac01′ succeeded
CRS-2677: Stop of ‘ora.crf’ on ‘askmdbrac01′ succeeded
CRS-2677: Stop of ‘ora.evmd’ on ‘askmdbrac01′ succeeded
CRS-2677: Stop of ‘ora.asm’ on ‘askmdbrac01′ succeeded
CRS-2673: Attempting to stop ‘ora.cluster_interconnect.haip’ on ‘askmdbrac01′
CRS-2677: Stop of ‘ora.cluster_interconnect.haip’ on ‘askmdbrac01′ succeeded
CRS-2677: Stop of ‘ora.ctssd’ on ‘askmdbrac01′ succeeded
CRS-2673: Attempting to stop ‘ora.cssd’ on ‘askmdbrac01′
CRS-2677: Stop of ‘ora.cssd’ on ‘askmdbrac01′ succeeded
CRS-2673: Attempting to stop ‘ora.gipcd’ on ‘askmdbrac01′
CRS-2673: Attempting to stop ‘ora.diskmon’ on ‘askmdbrac01′
CRS-2677: Stop of ‘ora.gipcd’ on ‘askmdbrac01′ succeeded
CRS-2673: Attempting to stop ‘ora.gpnpd’ on ‘askmdbrac01′
CRS-2677: Stop of ‘ora.gpnpd’ on ‘askmdbrac01′ succeeded
CRS-2677: Stop of ‘ora.diskmon’ on ‘askmdbrac01′ succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on ‘askmdbrac01′ has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@askmdbrac01 ~]# /o001/home/11.2.0.2/grid/bin/crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root@askmdbrac01 ~]#[root@askmdbrac02 ~]# /o001/home/11.2.0.2/grid/bin/crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on ‘askmdbrac02′
CRS-2673: Attempting to stop ‘ora.crsd’ on ‘askmdbrac02′
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on ‘askmdbrac02′
CRS-2673: Attempting to stop ‘ora.LISTENER_SCAN1.lsnr’ on ‘askmdbrac02′
CRS-2673: Attempting to stop ‘ora.askmdbrac02.vip’ on ‘askmdbrac02′
CRS-2673: Attempting to stop ‘ora.oc4j’ on ‘askmdbrac02′
CRS-2673: Attempting to stop ‘ora.LISTENER.lsnr’ on ‘askmdbrac02′
CRS-2673: Attempting to stop ‘ora.LISTENER_SCAN3.lsnr’ on ‘askmdbrac02′
CRS-2673: Attempting to stop ‘ora.cvu’ on ‘askmdbrac02′
CRS-2673: Attempting to stop ‘ora.LISTENER_SCAN2.lsnr’ on ‘askmdbrac02′
CRS-2673: Attempting to stop ‘ora.CRS.dg’ on ‘askmdbrac02′
CRS-2673: Attempting to stop ‘ora.racdb.racdb_service.svc’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.askmdbrac02.vip’ on ‘askmdbrac02′ succeeded
CRS-2677: Stop of ‘ora.racdb.racdb_service.svc’ on ‘askmdbrac02′ succeeded
CRS-2673: Attempting to stop ‘ora.racdb.db’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.LISTENER_SCAN1.lsnr’ on ‘askmdbrac02′ succeeded
CRS-2673: Attempting to stop ‘ora.scan1.vip’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.LISTENER.lsnr’ on ‘askmdbrac02′ succeeded
CRS-2673: Attempting to stop ‘ora.askmdbrac01.vip’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.scan1.vip’ on ‘askmdbrac02′ succeeded
CRS-2677: Stop of ‘ora.LISTENER_SCAN2.lsnr’ on ‘askmdbrac02′ succeeded
CRS-2673: Attempting to stop ‘ora.scan2.vip’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.askmdbrac01.vip’ on ‘askmdbrac02′ succeeded
CRS-2677: Stop of ‘ora.LISTENER_SCAN3.lsnr’ on ‘askmdbrac02′ succeeded
CRS-2673: Attempting to stop ‘ora.scan3.vip’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.scan2.vip’ on ‘askmdbrac02′ succeeded
CRS-2677: Stop of ‘ora.scan3.vip’ on ‘askmdbrac02′ succeeded
CRS-2677: Stop of ‘ora.racdb.db’ on ‘askmdbrac02′ succeeded
CRS-2673: Attempting to stop ‘ora.FRA.dg’ on ‘askmdbrac02′
CRS-2673: Attempting to stop ‘ora.PRDSHR_DATA.dg’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.FRA.dg’ on ‘askmdbrac02′ succeeded
CRS-2677: Stop of ‘ora.PRDSHR_DATA.dg’ on ‘askmdbrac02′ succeeded
CRS-2677: Stop of ‘ora.oc4j’ on ‘askmdbrac02′ succeeded
CRS-2677: Stop of ‘ora.cvu’ on ‘askmdbrac02′ succeeded
CRS-2677: Stop of ‘ora.CRS.dg’ on ‘askmdbrac02′ succeeded
CRS-2673: Attempting to stop ‘ora.asm’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.asm’ on ‘askmdbrac02′ succeeded
CRS-2673: Attempting to stop ‘ora.ons’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.ons’ on ‘askmdbrac02′ succeeded
CRS-2673: Attempting to stop ‘ora.net1.network’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.net1.network’ on ‘askmdbrac02′ succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on ‘askmdbrac02′ has completed
CRS-2677: Stop of ‘ora.crsd’ on ‘askmdbrac02′ succeeded
CRS-2673: Attempting to stop ‘ora.ctssd’ on ‘askmdbrac02′
CRS-2673: Attempting to stop ‘ora.evmd’ on ‘askmdbrac02′
CRS-2673: Attempting to stop ‘ora.asm’ on ‘askmdbrac02′
CRS-2673: Attempting to stop ‘ora.mdnsd’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.evmd’ on ‘askmdbrac02′ succeeded
CRS-2677: Stop of ‘ora.mdnsd’ on ‘askmdbrac02′ succeeded
CRS-2677: Stop of ‘ora.asm’ on ‘askmdbrac02′ succeeded
CRS-2673: Attempting to stop ‘ora.cluster_interconnect.haip’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.cluster_interconnect.haip’ on ‘askmdbrac02′ succeeded
CRS-2677: Stop of ‘ora.ctssd’ on ‘askmdbrac02′ succeeded
CRS-2673: Attempting to stop ‘ora.cssd’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.cssd’ on ‘askmdbrac02′ succeeded
CRS-2673: Attempting to stop ‘ora.diskmon’ on ‘askmdbrac02′
CRS-2673: Attempting to stop ‘ora.crf’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.crf’ on ‘askmdbrac02′ succeeded
CRS-2673: Attempting to stop ‘ora.gipcd’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.diskmon’ on ‘askmdbrac02′ succeeded
CRS-2677: Stop of ‘ora.gipcd’ on ‘askmdbrac02′ succeeded
CRS-2673: Attempting to stop ‘ora.gpnpd’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.gpnpd’ on ‘askmdbrac02′ succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on ‘askmdbrac02′ has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@askmdbrac02 ~]# /o001/home/11.2.0.2/grid/bin/crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root@askmdbrac02 ~]#

Observation : Cluster ready services started without any issues. Now i need to verify the services status.

STEP 10 :  Verify the status.

[root@askmdbrac01 ~]# /o001/home/11.2.0.2/grid/bin/crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
[root@askmdbrac01 ~]# /o001/home/11.2.0.2/grid/bin/crs_stat -t
Name           Type           Target    State     Host
————————————————————
ora.CRS.dg     ora….up.type ONLINE    ONLINE    askmdbrac01
ora.FRA.dg     ora….up.type ONLINE    ONLINE    askmdbrac01
ora….ER.lsnr ora….er.type ONLINE    ONLINE    askmdbrac01
ora….N1.lsnr ora….er.type ONLINE    OFFLINE
ora….N2.lsnr ora….er.type ONLINE    ONLINE    askmdbrac01
ora….N3.lsnr ora….er.type ONLINE    ONLINE    askmdbrac01
ora….DATA.dg ora….up.type ONLINE    ONLINE    askmdbrac01
ora.asm        ora.asm.type   ONLINE    ONLINE    askmdbrac01
ora.cvu        ora.cvu.type   ONLINE    ONLINE    askmdbrac01
ora.gsd        ora.gsd.type   OFFLINE   OFFLINE
ora….network ora….rk.type ONLINE    ONLINE    askmdbrac01
ora.oc4j       ora.oc4j.type  ONLINE    ONLINE    askmdbrac01
ora.ons        ora.ons.type   ONLINE    ONLINE    askmdbrac01
ora….SM1.asm application    ONLINE    ONLINE    askmdbrac01
ora….01.lsnr application    ONLINE    ONLINE    askmdbrac01
ora….l01.gsd application    OFFLINE   OFFLINE
ora….l01.ons application    ONLINE    ONLINE    askmdbrac01
ora….l01.vip ora….t1.type ONLINE    ONLINE    askmdbrac01
ora….SM2.asm application    ONLINE    ONLINE    askmdbrac02
ora….02.lsnr application    ONLINE    OFFLINE
ora….l02.gsd application    OFFLINE   OFFLINE
ora….l02.ons application    ONLINE    OFFLINE
ora….l02.vip ora….t1.type ONLINE    OFFLINE
ora….ry.acfs ora….fs.type ONLINE    OFFLINE
ora.scan1.vip  ora….ip.type ONLINE    OFFLINE
ora.scan2.vip  ora….ip.type ONLINE    ONLINE    askmdbrac01
ora.scan3.vip  ora….ip.type ONLINE    ONLINE    askmdbrac01
ora.racdb.db ora….se.type ONLINE    ONLINE    askmdbrac01
ora….ice.svc ora….ce.type ONLINE    ONLINE    askmdbrac01
[root@askmdbrac01 ~]# sudo su – grid
[grid@askmdbrac01 ~]$ . ./profile.grid
[grid@askmdbrac01 ~]$ Check_CRS.txt
HA Resource                                        Target     State
———–                                        ——     —–
ora.CRS.dg                                         ONLINE     ONLINE on askmdbrac01
ora.FRA.dg                                         ONLINE     ONLINE on askmdbrac01
ora.LISTENER.lsnr                                  ONLINE     ONLINE on askmdbrac01
ora.LISTENER_SCAN1.lsnr                            ONLINE     ONLINE on askmdbrac02
ora.LISTENER_SCAN2.lsnr                            ONLINE     ONLINE on askmdbrac01
ora.LISTENER_SCAN3.lsnr                            ONLINE     ONLINE on askmdbrac01
ora.PRDSHR_DATA.dg                                 ONLINE     ONLINE on askmdbrac01
ora.asm                                            ONLINE     ONLINE on askmdbrac01
ora.cvu                                            ONLINE     ONLINE on askmdbrac01
ora.gsd                                            OFFLINE    OFFLINE
ora.net1.network                                   ONLINE     ONLINE on askmdbrac01
ora.oc4j                                           ONLINE     ONLINE on askmdbrac01
ora.ons                                            ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.ASM1.asm                           ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.LISTENER_askmdbrac01.lsnr          ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.gsd                                OFFLINE    OFFLINE
ora.askmdbrac01.ons                                ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.vip                                ONLINE     ONLINE on askmdbrac01
ora.askmdbrac02.ASM2.asm                           ONLINE     ONLINE on askmdbrac02
ora.askmdbrac02.LISTENER_askmdbrac02.lsnr          ONLINE     ONLINE on askmdbrac02
ora.askmdbrac02.gsd                                OFFLINE    OFFLINE
ora.askmdbrac02.ons                                ONLINE     ONLINE on askmdbrac02
ora.askmdbrac02.vip                                ONLINE     ONLINE on askmdbrac02
ora.registry.acfs                                  ONLINE     OFFLINE
ora.scan1.vip                                      ONLINE     ONLINE on askmdbrac02
ora.scan2.vip                                      ONLINE     ONLINE on askmdbrac01
ora.scan3.vip                                      ONLINE     ONLINE on askmdbrac01
ora.racdb.db                                     ONLINE     ONLINE on askmdbrac01
ora.racdb.racdb_service.svc                    ONLINE     ONLINE on askmdbrac02
[grid@askmdbrac01 ~]$ Check_CRS.txt
HA Resource                                        Target     State
———–                                        ——     —–
ora.CRS.dg                                         ONLINE     ONLINE on askmdbrac01
ora.FRA.dg                                         ONLINE     ONLINE on askmdbrac01
ora.LISTENER.lsnr                                  ONLINE     ONLINE on askmdbrac01
ora.LISTENER_SCAN1.lsnr                            ONLINE     ONLINE on askmdbrac02
ora.LISTENER_SCAN2.lsnr                            ONLINE     ONLINE on askmdbrac01
ora.LISTENER_SCAN3.lsnr                            ONLINE     ONLINE on askmdbrac01
ora.PRDSHR_DATA.dg                                 ONLINE     ONLINE on askmdbrac01
ora.asm                                            ONLINE     ONLINE on askmdbrac01
ora.cvu                                            ONLINE     ONLINE on askmdbrac01
ora.gsd                                            OFFLINE    OFFLINE
ora.net1.network                                   ONLINE     ONLINE on askmdbrac01
ora.oc4j                                           ONLINE     ONLINE on askmdbrac01
ora.ons                                            ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.ASM1.asm                           ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.LISTENER_askmdbrac01.lsnr          ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.gsd                                OFFLINE    OFFLINE
ora.askmdbrac01.ons                                ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.vip                                ONLINE     ONLINE on askmdbrac01
ora.askmdbrac02.ASM2.asm                           ONLINE     ONLINE on askmdbrac02
ora.askmdbrac02.LISTENER_askmdbrac02.lsnr          ONLINE     ONLINE on askmdbrac02
ora.askmdbrac02.gsd                                OFFLINE    OFFLINE
ora.askmdbrac02.ons                                ONLINE     ONLINE on askmdbrac02
ora.askmdbrac02.vip                                ONLINE     ONLINE on askmdbrac02
ora.registry.acfs                                  ONLINE     OFFLINE
ora.scan1.vip                                      ONLINE     ONLINE on askmdbrac02
ora.scan2.vip                                      ONLINE     ONLINE on askmdbrac01
ora.scan3.vip                                      ONLINE     ONLINE on askmdbrac01
ora.racdb.db                                     ONLINE     ONLINE on askmdbrac01
ora.racdb.racdb_service.svc                    ONLINE     ONLINE on askmdbrac02
[grid@askmdbrac01 ~]$ srvctl status database -d racdb
Instance racdb1 is running on node askmdbrac01
Instance racdb2 is running on node askmdbrac02
[grid@askmdbrac01 ~]$[root@askmdbrac02 ~]# /o001/home/11.2.0.2/grid/bin/crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
[root@askmdbrac02 ~]# /o001/home/11.2.0.2/grid/bin/crs_stat -t
Name           Type           Target    State     Host
————————————————————
ora.CRS.dg     ora….up.type ONLINE    ONLINE    askmdbrac01
ora.FRA.dg     ora….up.type ONLINE    ONLINE    askmdbrac01
ora….ER.lsnr ora….er.type ONLINE    ONLINE    askmdbrac01
ora….N1.lsnr ora….er.type ONLINE    ONLINE    askmdbrac02
ora….N2.lsnr ora….er.type ONLINE    ONLINE    askmdbrac01
ora….N3.lsnr ora….er.type ONLINE    ONLINE    askmdbrac01
ora….DATA.dg ora….up.type ONLINE    ONLINE    askmdbrac01
ora.asm        ora.asm.type   ONLINE    ONLINE    askmdbrac01
ora.cvu        ora.cvu.type   ONLINE    ONLINE    askmdbrac01
ora.gsd        ora.gsd.type   OFFLINE   OFFLINE
ora….network ora….rk.type ONLINE    ONLINE    askmdbrac01
ora.oc4j       ora.oc4j.type  ONLINE    ONLINE    askmdbrac01
ora.ons        ora.ons.type   ONLINE    ONLINE    askmdbrac01
ora….SM1.asm application    ONLINE    ONLINE    askmdbrac01
ora….01.lsnr application    ONLINE    ONLINE    askmdbrac01
ora….l01.gsd application    OFFLINE   OFFLINE
ora….l01.ons application    ONLINE    ONLINE    askmdbrac01
ora….l01.vip ora….t1.type ONLINE    ONLINE    askmdbrac01
ora….SM2.asm application    ONLINE    ONLINE    askmdbrac02
ora….02.lsnr application    ONLINE    ONLINE    askmdbrac02
ora….l02.gsd application    OFFLINE   OFFLINE
ora….l02.ons application    ONLINE    ONLINE    askmdbrac02
ora….l02.vip ora….t1.type ONLINE    ONLINE    askmdbrac02
ora….ry.acfs ora….fs.type ONLINE    OFFLINE
ora.scan1.vip  ora….ip.type ONLINE    ONLINE    askmdbrac02
ora.scan2.vip  ora….ip.type ONLINE    ONLINE    askmdbrac01
ora.scan3.vip  ora….ip.type ONLINE    ONLINE    askmdbrac01
ora.racdb.db ora….se.type ONLINE    ONLINE    askmdbrac01
ora….ice.svc ora….ce.type ONLINE    ONLINE    askmdbrac01
[root@askmdbrac02 ~]# sudo su – grid
[grid@askmdbrac02 ~]$ . ./profile.grid
[grid@askmdbrac02 ~]$ Check_CRS.txt
HA Resource                                        Target     State
———–                                        ——     —–
ora.CRS.dg                                         ONLINE     ONLINE on askmdbrac01
ora.FRA.dg                                         ONLINE     ONLINE on askmdbrac01
ora.LISTENER.lsnr                                  ONLINE     ONLINE on askmdbrac01
ora.LISTENER_SCAN1.lsnr                            ONLINE     ONLINE on askmdbrac02
ora.LISTENER_SCAN2.lsnr                            ONLINE     ONLINE on askmdbrac01
ora.LISTENER_SCAN3.lsnr                            ONLINE     ONLINE on askmdbrac01
ora.PRDSHR_DATA.dg                                 ONLINE     ONLINE on askmdbrac01
ora.asm                                            ONLINE     ONLINE on askmdbrac01
ora.cvu                                            ONLINE     ONLINE on askmdbrac01
ora.gsd                                            OFFLINE    OFFLINE
ora.net1.network                                   ONLINE     ONLINE on askmdbrac01
ora.oc4j                                           ONLINE     ONLINE on askmdbrac01
ora.ons                                            ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.ASM1.asm                           ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.LISTENER_askmdbrac01.lsnr          ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.gsd                                OFFLINE    OFFLINE
ora.askmdbrac01.ons                                ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.vip                                ONLINE     ONLINE on askmdbrac01
ora.askmdbrac02.ASM2.asm                           ONLINE     ONLINE on askmdbrac02
ora.askmdbrac02.LISTENER_askmdbrac02.lsnr          ONLINE     ONLINE on askmdbrac02
ora.askmdbrac02.gsd                                OFFLINE    OFFLINE
ora.askmdbrac02.ons                                ONLINE     ONLINE on askmdbrac02
ora.askmdbrac02.vip                                ONLINE     ONLINE on askmdbrac02
ora.registry.acfs                                  ONLINE     OFFLINE
ora.scan1.vip                                      ONLINE     ONLINE on askmdbrac02
ora.scan2.vip                                      ONLINE     ONLINE on askmdbrac01
ora.scan3.vip                                      ONLINE     ONLINE on askmdbrac01
ora.racdb.db                                     ONLINE     ONLINE on askmdbrac01
ora.racdb.racdb_service.svc                    ONLINE     ONLINE on askmdbrac02
[grid@askmdbrac02 ~]$ srvctl status database -d racdb
Instance racdb1 is running on node askmdbrac01
Instance racdb2 is running on node askmdbrac02
[grid@askmdbrac02 ~]$

This concludes the article on troubleshooting RAC services startup issue.

Thanks

This article gives you the troubleshooting steps for this particular issue. These steps may also help you to troubleshoot if you have issues starting RAC services.

STEP 1: Reboot the rac servers and Check services after re-booting the servers. The services are still down.

STEP 2: Verify storage on both rac servers

[grid@askmdbrac01 ~]$ /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[grid@askmdbrac01 ~]$ /etc/init.d/oracleasm listdisks
CRSVOL1
DISK1
FRADISK1
[grid@askmdbrac01 ~]$ multipath -ll
-bash: multipath: command not found
[grid@askmdbrac01 ~]$ exit
logout
[oracle@askmdbrac01 ~]$ sudo su –
[root@askmdbrac01 ~]# multipath -ll
vpath2 (360050768019280fb280000000000055e) dm-10 IBM,2145
[size=200G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=50][active]
\_ 2:0:1:2 sdf        8:80  [active][ready]
\_ 3:0:1:2 sdl        8:176 [active][ready]
\_ round-robin 0 [prio=10][enabled]
\_ 2:0:0:2 sdc        8:32  [active][ready]
\_ 3:0:0:2 sdi        8:128 [active][ready]
vpath1 (360050768019280fb280000000000055d) dm-12 IBM,2145
[size=200G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=50][active]
\_ 3:0:0:1 sdh        8:112 [active][ready]
\_ round-robin 0 [prio=10][enabled]
\_ 2:0:1:1 sde        8:64  [active][ready]
\_ 3:0:1:1 sdk        8:160 [active][ready]
vpath0 (360050768019280fb280000000000055c) dm-11 IBM,2145
[size=10G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=50][active]
\_ 2:0:1:0 sdd        8:48  [active][ready]
\_ 3:0:1:0 sdj        8:144 [active][ready]
\_ round-robin 0 [prio=10][enabled]
\_ 3:0:0:0 sdg        8:96  [active][ready]
[root@askmdbrac01 ~]# exit[grid@askmdbrac02 ~]$ /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[grid@askmdbrac02 ~]$ /etc/init.d/oracleasm listdisks
CRSVOL1
DISK1
FRADISK1
[grid@askmdbrac02 ~]$ multipath -ll
-bash: multipath: command not found
[grid@askmdbrac02 ~]$ exit
logout
[oracle@askmdbrac02 ~]$ sudo su –
[root@askmdbrac02 ~]# multipath -ll
vpath2 (360050768019280fb280000000000055e) dm-10 IBM,2145
[size=200G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=50][active]
\_ 2:0:1:2 sdf        8:80  [active][ready]
\_ 3:0:0:2 sdi        8:128 [active][ready]
\_ round-robin 0 [prio=10][enabled]
\_ 2:0:0:2 sdc        8:32  [active][ready]
\_ 3:0:1:2 sdl        8:176 [active][ready]
vpath1 (360050768019280fb280000000000055d) dm-12 IBM,2145
[size=200G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=50][active]
\_ 3:0:1:1 sdk        8:160 [active][ready]
\_ round-robin 0 [prio=10][enabled]
\_ 2:0:1:1 sde        8:64  [active][ready]
\_ 3:0:0:1 sdh        8:112 [active][ready]
vpath0 (360050768019280fb280000000000055c) dm-11 IBM,2145
[size=10G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=50][active]
\_ 2:0:1:0 sdd        8:48  [active][ready]
\_ 3:0:0:0 sdg        8:96  [active][ready]
\_ round-robin 0 [prio=10][enabled]
\_ 3:0:1:0 sdj        8:144 [active][ready]
[root@askmdbrac02 ~]#

Observation : Storage is fine and i am able to see ASM disks from both the nodes. Storage is configured with multipathing. I also verified multipathing and status shows fine.

STEP 3: Check high availability services and cluster ready services on both nodes with the following commands

[grid@askmdbrac01 ~]$ crsctl check has
CRS-4638: Oracle High Availability Services is online
[grid@askmdbrac01 ~]$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
[grid@askmdbrac01 ~]$
[grid@askmdbrac01 ~]$ crsctl check cluster
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
[grid@askmdbrac01 ~]$[grid@askmdbrac02 ~]$ crsctl check has
CRS-4638: Oracle High Availability Services is online
[grid@askmdbrac02 ~]$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
[grid@askmdbrac02 ~]$
[grid@askmdbrac02 ~]$ crsctl check cluster
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
[grid@askmdbrac02 ~]$

Observation : The high availability services are up , but crs services are down.

STEP 4: Do ping and nslookup for other rac nodes from each node to make sure that each node in cluster is accessible from every other node. Check this for private ips, virtual ips and for scan ips.

Observation : The ping and nslookup are working fine and all the nodes are reachable from every other node.

STEP 5 : Then verify the log files from grid infrastructure and see if you can find any error messages from log files. The cluster log files are located at $ORACLE_HOME/log/askmdbrac01 and $ORACLE_HOME/log/askmdbrac02.

[cssd(11175)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /o001/home/11.2.0.2/grid/log/askmdbrac01/cssd/ocssd.log
2013-04-02 19:38:27.425
[ohasd(6876)]CRS-2765:Resource ‘ora.diskmon’ has failed on server ‘askmdbrac01′.
2013-04-02 19:38:39.988
[cssd(11175)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /o001/home/11.2.0.2/grid/log/askmdbrac01/cssd/ocssd.log
2013-04-02 19:38:55.007
[cssd(11175)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /o001/home/11.2.0.2/grid/log/askmdbrac01/cssd/ocssd.log
2013-04-02 19:39:10.025
…..
…..
[cssd(11175)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /o001/home/11.2.0.2/grid/log/askmdbrac01/cssd/ocssd.log
2013-04-02 19:48:23.836
[/o001/home/11.2.0.2/grid/bin/cssdagent(11163)]CRS-5818:Aborted command ‘start for resource: ora.cssd 1 1′ for resource ‘ora.cssd’. Details at (:CRSAGF00113:) {0:27:4} in /o001/home/11.2.0.2/grid/log/askmdbrac01/agent/ohasd/oracssdagent_root/oracssdagent_root.log.
2013-04-02 19:48:23.837
[cssd(11175)]CRS-1656:The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in /o001/home/11.2.0.2/grid/log/askmdbrac01/cssd/ocssd.log
2013-04-02 19:48:23.838
[cssd(11175)]CRS-1603:CSSD on node askmdbrac01 shutdown by user.
2013-04-02 19:48:25.730
[cssd(11175)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /o001/home/11.2.0.2/grid/log/askmdbrac01/cssd/ocssd.log
2013-04-02 19:48:29.863
[ohasd(6876)]CRS-2765:Resource ‘ora.cssdmonitor’ has failed on server ‘askmdbrac01′.
2013-04-02 19:48:30.126
[ohasd(6876)]CRS-2771:Maximum restart attempts reached for resource ‘ora.cssd’; will not restart.
2013-04-02 19:48:35.833
[ohasd(6876)]CRS-2765:Resource ‘ora.diskmon’ has failed on server ‘askmdbrac01′.

Observation : Found from the cluster alert log file (alertaskmdbrac01.log) that the cluster services can’t find voting disk. So it could be due the in-accessible asm storage or due to currupted voting disks.

Then i tried to see if i can see voting disks through command.

SETP 6 :  Check the voting disks availability ( 11gR2 has voting disks in asm. I don’t have asm instance up now. So i will get error if i try to query the voting disks on asm)

[root@askmdbrac01 ~]# /o001/home/11.2.0.2/grid/bin/crsctl query css votedisk
Unable to communicate with the Cluster Synchronization Services daemon.
[root@askmdbrac01 ~]#[grid@askmdbrac02 ~]$ /o001/home/11.2.0.2/grid/bin/crsctl query css votedisk
Unable to communicate with the Cluster Synchronization Services daemon.
[grid@askmdbrac02 ~]$

Observation : I am not able to query the voting disks. Now i have only one option to look at the asm storage accessibility.

STEP 7 :  Tried re-enabling the asmlibs as follows on both the nodes …

[root@askmdbrac01 ~]# /etc/init.d/oracleasm restart
Dropping Oracle ASMLib disks:                              [  OK  ]
Shutting down the Oracle ASMLib driver:                    [  OK  ]
[root@askmdbrac01 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: no
Checking if /dev/oracleasm is mounted: no
[root@askmdbrac01 ~]# /etc/init.d/oracleasm start
[root@askmdbrac01 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: no
Checking if /dev/oracleasm is mounted: no
[root@askmdbrac01 ~]# /etc/init.d/oracleasm enable
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@askmdbrac01 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[root@askmdbrac01 ~]# /etc/init.d/oracleasm listdisks
CRSVOL1
DISK1
FRADISK1
[root@askmdbrac01 ~]#[root@askmdbrac02 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[root@askmdbrac02 ~]# /etc/init.d/oracleasm restart
Dropping Oracle ASMLib disks:                              [  OK  ]
Shutting down the Oracle ASMLib driver:                    [  OK  ]
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@askmdbrac02 ~]#  /etc/init.d/oracleasm listdisks
CRSVOL1
DISK1
FRADISK1
[root@askmdbrac02 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[root@askmdbrac02 ~]#

Observation : Not able to start services even after restarting the asmlib.

STEP 8 : Tried to re-configure the asmlib as below …..

[root@askmdbrac01 ~]# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.This will configure the on-boot properties of the Oracle ASM library
driver.  The following questions will determine whether the driver is
loaded on boot and what permissions it will have.  The current values
will be shown in brackets (‘[]’).  Hitting <ENTER> without typing an
answer will keep that current value.  Ctrl-C will abort.Default user to own the driver interface []: grid
Default group to own the driver interface []: asmadmin
Start Oracle ASM library driver on boot (y/n) [y]:
Scan for Oracle ASM disks on boot (y/n) [y]:
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@askmdbrac01 ~]#
[root@askmdbrac01 ~]# /etc/init.d/oracleasm stop
Dropping Oracle ASMLib disks:                              [  OK  ]
Shutting down the Oracle ASMLib driver:                    [FAILED]
[root@askmdbrac01 ~]# /etc/init.d/oracleasm start
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@askmdbrac01 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[root@askmdbrac01 ~]#[root@askmdbrac02 ~]# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library
driver.  The following questions will determine whether the driver is
loaded on boot and what permissions it will have.  The current values
will be shown in brackets (‘[]’).  Hitting <ENTER> without typing an
answer will keep that current value.  Ctrl-C will abort.

Default user to own the driver interface []: grid
Default group to own the driver interface []: asmadmin
Start Oracle ASM library driver on boot (y/n) [y]:
Scan for Oracle ASM disks on boot (y/n) [y]:
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@askmdbrac02 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[root@askmdbrac02 ~]# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library
driver.  The following questions will determine whether the driver is
loaded on boot and what permissions it will have.  The current values
will be shown in brackets (‘[]’).  Hitting <ENTER> without typing an
answer will keep that current value.  Ctrl-C will abort.

Default user to own the driver interface [grid]:
Default group to own the driver interface [asmadmin]:
Start Oracle ASM library driver on boot (y/n) [y]:
Scan for Oracle ASM disks on boot (y/n) [y]:
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@askmdbrac02 ~]#
[root@askmdbrac02 ~]# /etc/init.d/oracleasm stop
Dropping Oracle ASMLib disks:                              [  OK  ]
Shutting down the Oracle ASMLib driver:                    [FAILED]
[root@askmdbrac02 ~]# /etc/init.d/oracleasm start
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@askmdbrac02 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[root@askmdbrac02 ~]#

Observation : Identified that the asm library drivers configuration is wrong and i re-configured with correct user and group. See the difference in the lines when it is asking for prompt on second node. I executed this command twice to show you the difference when it is properly configured and when it is not properly configured.

STEP 9 : Stop and Start cluster services on both the nodes.

[root@askmdbrac01 ~]#  /o001/home/11.2.0.2/grid/bin/crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on ‘askmdbrac01′
CRS-2673: Attempting to stop ‘ora.crsd’ on ‘askmdbrac01′
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on ‘askmdbrac01′
CRS-2673: Attempting to stop ‘ora.LISTENER.lsnr’ on ‘askmdbrac01′
CRS-2673: Attempting to stop ‘ora.racdb.racdb_service.svc’ on ‘askmdbrac01′
CRS-2673: Attempting to stop ‘ora.LISTENER_SCAN1.lsnr’ on ‘askmdbrac01′
CRS-2677: Stop of ‘ora.LISTENER.lsnr’ on ‘askmdbrac01′ succeeded
CRS-2677: Stop of ‘ora.LISTENER_SCAN1.lsnr’ on ‘askmdbrac01′ succeeded
CRS-2673: Attempting to stop ‘ora.scan1.vip’ on ‘askmdbrac01′
CRS-2677: Stop of ‘ora.scan1.vip’ on ‘askmdbrac01′ succeeded
CRS-2672: Attempting to start ‘ora.scan1.vip’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.racdb.racdb_service.svc’ on ‘askmdbrac01′ succeeded
CRS-2673: Attempting to stop ‘ora.askmdbrac01.vip’ on ‘askmdbrac01′
CRS-2673: Attempting to stop ‘ora.CRS.dg’ on ‘askmdbrac01′
CRS-2673: Attempting to stop ‘ora.racdb.db’ on ‘askmdbrac01′
CRS-2677: Stop of ‘ora.askmdbrac01.vip’ on ‘askmdbrac01′ succeeded
CRS-2672: Attempting to start ‘ora.askmdbrac01.vip’ on ‘askmdbrac02′
CRS-2676: Start of ‘ora.scan1.vip’ on ‘askmdbrac02′ succeeded
CRS-2672: Attempting to start ‘ora.LISTENER_SCAN1.lsnr’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.racdb.db’ on ‘askmdbrac01′ succeeded
CRS-2673: Attempting to stop ‘ora.FRA.dg’ on ‘askmdbrac01′
CRS-2673: Attempting to stop ‘ora.PRDSHR_DATA.dg’ on ‘askmdbrac01′
CRS-2676: Start of ‘ora.askmdbrac01.vip’ on ‘askmdbrac02′ succeeded
CRS-2677: Stop of ‘ora.PRDSHR_DATA.dg’ on ‘askmdbrac01′ succeeded
CRS-2676: Start of ‘ora.LISTENER_SCAN1.lsnr’ on ‘askmdbrac02′ succeeded
CRS-2677: Stop of ‘ora.FRA.dg’ on ‘askmdbrac01′ succeeded
CRS-2677: Stop of ‘ora.CRS.dg’ on ‘askmdbrac01′ succeeded
CRS-2673: Attempting to stop ‘ora.asm’ on ‘askmdbrac01′
CRS-2677: Stop of ‘ora.asm’ on ‘askmdbrac01′ succeeded
CRS-2673: Attempting to stop ‘ora.ons’ on ‘askmdbrac01′
CRS-2677: Stop of ‘ora.ons’ on ‘askmdbrac01′ succeeded
CRS-2673: Attempting to stop ‘ora.net1.network’ on ‘askmdbrac01′
CRS-2677: Stop of ‘ora.net1.network’ on ‘askmdbrac01′ succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on ‘askmdbrac01′ has completed
CRS-2677: Stop of ‘ora.crsd’ on ‘askmdbrac01′ succeeded
CRS-2673: Attempting to stop ‘ora.mdnsd’ on ‘askmdbrac01′
CRS-2673: Attempting to stop ‘ora.crf’ on ‘askmdbrac01′
CRS-2673: Attempting to stop ‘ora.ctssd’ on ‘askmdbrac01′
CRS-2673: Attempting to stop ‘ora.evmd’ on ‘askmdbrac01′
CRS-2673: Attempting to stop ‘ora.asm’ on ‘askmdbrac01′
CRS-2677: Stop of ‘ora.mdnsd’ on ‘askmdbrac01′ succeeded
CRS-2677: Stop of ‘ora.crf’ on ‘askmdbrac01′ succeeded
CRS-2677: Stop of ‘ora.evmd’ on ‘askmdbrac01′ succeeded
CRS-2677: Stop of ‘ora.asm’ on ‘askmdbrac01′ succeeded
CRS-2673: Attempting to stop ‘ora.cluster_interconnect.haip’ on ‘askmdbrac01′
CRS-2677: Stop of ‘ora.cluster_interconnect.haip’ on ‘askmdbrac01′ succeeded
CRS-2677: Stop of ‘ora.ctssd’ on ‘askmdbrac01′ succeeded
CRS-2673: Attempting to stop ‘ora.cssd’ on ‘askmdbrac01′
CRS-2677: Stop of ‘ora.cssd’ on ‘askmdbrac01′ succeeded
CRS-2673: Attempting to stop ‘ora.gipcd’ on ‘askmdbrac01′
CRS-2673: Attempting to stop ‘ora.diskmon’ on ‘askmdbrac01′
CRS-2677: Stop of ‘ora.gipcd’ on ‘askmdbrac01′ succeeded
CRS-2673: Attempting to stop ‘ora.gpnpd’ on ‘askmdbrac01′
CRS-2677: Stop of ‘ora.gpnpd’ on ‘askmdbrac01′ succeeded
CRS-2677: Stop of ‘ora.diskmon’ on ‘askmdbrac01′ succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on ‘askmdbrac01′ has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@askmdbrac01 ~]# /o001/home/11.2.0.2/grid/bin/crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root@askmdbrac01 ~]#[root@askmdbrac02 ~]# /o001/home/11.2.0.2/grid/bin/crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on ‘askmdbrac02′
CRS-2673: Attempting to stop ‘ora.crsd’ on ‘askmdbrac02′
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on ‘askmdbrac02′
CRS-2673: Attempting to stop ‘ora.LISTENER_SCAN1.lsnr’ on ‘askmdbrac02′
CRS-2673: Attempting to stop ‘ora.askmdbrac02.vip’ on ‘askmdbrac02′
CRS-2673: Attempting to stop ‘ora.oc4j’ on ‘askmdbrac02′
CRS-2673: Attempting to stop ‘ora.LISTENER.lsnr’ on ‘askmdbrac02′
CRS-2673: Attempting to stop ‘ora.LISTENER_SCAN3.lsnr’ on ‘askmdbrac02′
CRS-2673: Attempting to stop ‘ora.cvu’ on ‘askmdbrac02′
CRS-2673: Attempting to stop ‘ora.LISTENER_SCAN2.lsnr’ on ‘askmdbrac02′
CRS-2673: Attempting to stop ‘ora.CRS.dg’ on ‘askmdbrac02′
CRS-2673: Attempting to stop ‘ora.racdb.racdb_service.svc’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.askmdbrac02.vip’ on ‘askmdbrac02′ succeeded
CRS-2677: Stop of ‘ora.racdb.racdb_service.svc’ on ‘askmdbrac02′ succeeded
CRS-2673: Attempting to stop ‘ora.racdb.db’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.LISTENER_SCAN1.lsnr’ on ‘askmdbrac02′ succeeded
CRS-2673: Attempting to stop ‘ora.scan1.vip’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.LISTENER.lsnr’ on ‘askmdbrac02′ succeeded
CRS-2673: Attempting to stop ‘ora.askmdbrac01.vip’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.scan1.vip’ on ‘askmdbrac02′ succeeded
CRS-2677: Stop of ‘ora.LISTENER_SCAN2.lsnr’ on ‘askmdbrac02′ succeeded
CRS-2673: Attempting to stop ‘ora.scan2.vip’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.askmdbrac01.vip’ on ‘askmdbrac02′ succeeded
CRS-2677: Stop of ‘ora.LISTENER_SCAN3.lsnr’ on ‘askmdbrac02′ succeeded
CRS-2673: Attempting to stop ‘ora.scan3.vip’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.scan2.vip’ on ‘askmdbrac02′ succeeded
CRS-2677: Stop of ‘ora.scan3.vip’ on ‘askmdbrac02′ succeeded
CRS-2677: Stop of ‘ora.racdb.db’ on ‘askmdbrac02′ succeeded
CRS-2673: Attempting to stop ‘ora.FRA.dg’ on ‘askmdbrac02′
CRS-2673: Attempting to stop ‘ora.PRDSHR_DATA.dg’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.FRA.dg’ on ‘askmdbrac02′ succeeded
CRS-2677: Stop of ‘ora.PRDSHR_DATA.dg’ on ‘askmdbrac02′ succeeded
CRS-2677: Stop of ‘ora.oc4j’ on ‘askmdbrac02′ succeeded
CRS-2677: Stop of ‘ora.cvu’ on ‘askmdbrac02′ succeeded
CRS-2677: Stop of ‘ora.CRS.dg’ on ‘askmdbrac02′ succeeded
CRS-2673: Attempting to stop ‘ora.asm’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.asm’ on ‘askmdbrac02′ succeeded
CRS-2673: Attempting to stop ‘ora.ons’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.ons’ on ‘askmdbrac02′ succeeded
CRS-2673: Attempting to stop ‘ora.net1.network’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.net1.network’ on ‘askmdbrac02′ succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on ‘askmdbrac02′ has completed
CRS-2677: Stop of ‘ora.crsd’ on ‘askmdbrac02′ succeeded
CRS-2673: Attempting to stop ‘ora.ctssd’ on ‘askmdbrac02′
CRS-2673: Attempting to stop ‘ora.evmd’ on ‘askmdbrac02′
CRS-2673: Attempting to stop ‘ora.asm’ on ‘askmdbrac02′
CRS-2673: Attempting to stop ‘ora.mdnsd’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.evmd’ on ‘askmdbrac02′ succeeded
CRS-2677: Stop of ‘ora.mdnsd’ on ‘askmdbrac02′ succeeded
CRS-2677: Stop of ‘ora.asm’ on ‘askmdbrac02′ succeeded
CRS-2673: Attempting to stop ‘ora.cluster_interconnect.haip’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.cluster_interconnect.haip’ on ‘askmdbrac02′ succeeded
CRS-2677: Stop of ‘ora.ctssd’ on ‘askmdbrac02′ succeeded
CRS-2673: Attempting to stop ‘ora.cssd’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.cssd’ on ‘askmdbrac02′ succeeded
CRS-2673: Attempting to stop ‘ora.diskmon’ on ‘askmdbrac02′
CRS-2673: Attempting to stop ‘ora.crf’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.crf’ on ‘askmdbrac02′ succeeded
CRS-2673: Attempting to stop ‘ora.gipcd’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.diskmon’ on ‘askmdbrac02′ succeeded
CRS-2677: Stop of ‘ora.gipcd’ on ‘askmdbrac02′ succeeded
CRS-2673: Attempting to stop ‘ora.gpnpd’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.gpnpd’ on ‘askmdbrac02′ succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on ‘askmdbrac02′ has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@askmdbrac02 ~]# /o001/home/11.2.0.2/grid/bin/crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root@askmdbrac02 ~]#

Observation : Cluster ready services started without any issues. Now i need to verify the services status.

STEP 10 :  Verify the status.

[root@askmdbrac01 ~]# /o001/home/11.2.0.2/grid/bin/crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
[root@askmdbrac01 ~]# /o001/home/11.2.0.2/grid/bin/crs_stat -t
Name           Type           Target    State     Host
————————————————————
ora.CRS.dg     ora….up.type ONLINE    ONLINE    askmdbrac01
ora.FRA.dg     ora….up.type ONLINE    ONLINE    askmdbrac01
ora….ER.lsnr ora….er.type ONLINE    ONLINE    askmdbrac01
ora….N1.lsnr ora….er.type ONLINE    OFFLINE
ora….N2.lsnr ora….er.type ONLINE    ONLINE    askmdbrac01
ora….N3.lsnr ora….er.type ONLINE    ONLINE    askmdbrac01
ora….DATA.dg ora….up.type ONLINE    ONLINE    askmdbrac01
ora.asm        ora.asm.type   ONLINE    ONLINE    askmdbrac01
ora.cvu        ora.cvu.type   ONLINE    ONLINE    askmdbrac01
ora.gsd        ora.gsd.type   OFFLINE   OFFLINE
ora….network ora….rk.type ONLINE    ONLINE    askmdbrac01
ora.oc4j       ora.oc4j.type  ONLINE    ONLINE    askmdbrac01
ora.ons        ora.ons.type   ONLINE    ONLINE    askmdbrac01
ora….SM1.asm application    ONLINE    ONLINE    askmdbrac01
ora….01.lsnr application    ONLINE    ONLINE    askmdbrac01
ora….l01.gsd application    OFFLINE   OFFLINE
ora….l01.ons application    ONLINE    ONLINE    askmdbrac01
ora….l01.vip ora….t1.type ONLINE    ONLINE    askmdbrac01
ora….SM2.asm application    ONLINE    ONLINE    askmdbrac02
ora….02.lsnr application    ONLINE    OFFLINE
ora….l02.gsd application    OFFLINE   OFFLINE
ora….l02.ons application    ONLINE    OFFLINE
ora….l02.vip ora….t1.type ONLINE    OFFLINE
ora….ry.acfs ora….fs.type ONLINE    OFFLINE
ora.scan1.vip  ora….ip.type ONLINE    OFFLINE
ora.scan2.vip  ora….ip.type ONLINE    ONLINE    askmdbrac01
ora.scan3.vip  ora….ip.type ONLINE    ONLINE    askmdbrac01
ora.racdb.db ora….se.type ONLINE    ONLINE    askmdbrac01
ora….ice.svc ora….ce.type ONLINE    ONLINE    askmdbrac01
[root@askmdbrac01 ~]# sudo su – grid
[grid@askmdbrac01 ~]$ . ./profile.grid
[grid@askmdbrac01 ~]$ Check_CRS.txt
HA Resource                                        Target     State
———–                                        ——     —–
ora.CRS.dg                                         ONLINE     ONLINE on askmdbrac01
ora.FRA.dg                                         ONLINE     ONLINE on askmdbrac01
ora.LISTENER.lsnr                                  ONLINE     ONLINE on askmdbrac01
ora.LISTENER_SCAN1.lsnr                            ONLINE     ONLINE on askmdbrac02
ora.LISTENER_SCAN2.lsnr                            ONLINE     ONLINE on askmdbrac01
ora.LISTENER_SCAN3.lsnr                            ONLINE     ONLINE on askmdbrac01
ora.PRDSHR_DATA.dg                                 ONLINE     ONLINE on askmdbrac01
ora.asm                                            ONLINE     ONLINE on askmdbrac01
ora.cvu                                            ONLINE     ONLINE on askmdbrac01
ora.gsd                                            OFFLINE    OFFLINE
ora.net1.network                                   ONLINE     ONLINE on askmdbrac01
ora.oc4j                                           ONLINE     ONLINE on askmdbrac01
ora.ons                                            ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.ASM1.asm                           ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.LISTENER_askmdbrac01.lsnr          ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.gsd                                OFFLINE    OFFLINE
ora.askmdbrac01.ons                                ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.vip                                ONLINE     ONLINE on askmdbrac01
ora.askmdbrac02.ASM2.asm                           ONLINE     ONLINE on askmdbrac02
ora.askmdbrac02.LISTENER_askmdbrac02.lsnr          ONLINE     ONLINE on askmdbrac02
ora.askmdbrac02.gsd                                OFFLINE    OFFLINE
ora.askmdbrac02.ons                                ONLINE     ONLINE on askmdbrac02
ora.askmdbrac02.vip                                ONLINE     ONLINE on askmdbrac02
ora.registry.acfs                                  ONLINE     OFFLINE
ora.scan1.vip                                      ONLINE     ONLINE on askmdbrac02
ora.scan2.vip                                      ONLINE     ONLINE on askmdbrac01
ora.scan3.vip                                      ONLINE     ONLINE on askmdbrac01
ora.racdb.db                                     ONLINE     ONLINE on askmdbrac01
ora.racdb.racdb_service.svc                    ONLINE     ONLINE on askmdbrac02
[grid@askmdbrac01 ~]$ Check_CRS.txt
HA Resource                                        Target     State
———–                                        ——     —–
ora.CRS.dg                                         ONLINE     ONLINE on askmdbrac01
ora.FRA.dg                                         ONLINE     ONLINE on askmdbrac01
ora.LISTENER.lsnr                                  ONLINE     ONLINE on askmdbrac01
ora.LISTENER_SCAN1.lsnr                            ONLINE     ONLINE on askmdbrac02
ora.LISTENER_SCAN2.lsnr                            ONLINE     ONLINE on askmdbrac01
ora.LISTENER_SCAN3.lsnr                            ONLINE     ONLINE on askmdbrac01
ora.PRDSHR_DATA.dg                                 ONLINE     ONLINE on askmdbrac01
ora.asm                                            ONLINE     ONLINE on askmdbrac01
ora.cvu                                            ONLINE     ONLINE on askmdbrac01
ora.gsd                                            OFFLINE    OFFLINE
ora.net1.network                                   ONLINE     ONLINE on askmdbrac01
ora.oc4j                                           ONLINE     ONLINE on askmdbrac01
ora.ons                                            ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.ASM1.asm                           ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.LISTENER_askmdbrac01.lsnr          ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.gsd                                OFFLINE    OFFLINE
ora.askmdbrac01.ons                                ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.vip                                ONLINE     ONLINE on askmdbrac01
ora.askmdbrac02.ASM2.asm                           ONLINE     ONLINE on askmdbrac02
ora.askmdbrac02.LISTENER_askmdbrac02.lsnr          ONLINE     ONLINE on askmdbrac02
ora.askmdbrac02.gsd                                OFFLINE    OFFLINE
ora.askmdbrac02.ons                                ONLINE     ONLINE on askmdbrac02
ora.askmdbrac02.vip                                ONLINE     ONLINE on askmdbrac02
ora.registry.acfs                                  ONLINE     OFFLINE
ora.scan1.vip                                      ONLINE     ONLINE on askmdbrac02
ora.scan2.vip                                      ONLINE     ONLINE on askmdbrac01
ora.scan3.vip                                      ONLINE     ONLINE on askmdbrac01
ora.racdb.db                                     ONLINE     ONLINE on askmdbrac01
ora.racdb.racdb_service.svc                    ONLINE     ONLINE on askmdbrac02
[grid@askmdbrac01 ~]$ srvctl status database -d racdb
Instance racdb1 is running on node askmdbrac01
Instance racdb2 is running on node askmdbrac02
[grid@askmdbrac01 ~]$[root@askmdbrac02 ~]# /o001/home/11.2.0.2/grid/bin/crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
[root@askmdbrac02 ~]# /o001/home/11.2.0.2/grid/bin/crs_stat -t
Name           Type           Target    State     Host
————————————————————
ora.CRS.dg     ora….up.type ONLINE    ONLINE    askmdbrac01
ora.FRA.dg     ora….up.type ONLINE    ONLINE    askmdbrac01
ora….ER.lsnr ora….er.type ONLINE    ONLINE    askmdbrac01
ora….N1.lsnr ora….er.type ONLINE    ONLINE    askmdbrac02
ora….N2.lsnr ora….er.type ONLINE    ONLINE    askmdbrac01
ora….N3.lsnr ora….er.type ONLINE    ONLINE    askmdbrac01
ora….DATA.dg ora….up.type ONLINE    ONLINE    askmdbrac01
ora.asm        ora.asm.type   ONLINE    ONLINE    askmdbrac01
ora.cvu        ora.cvu.type   ONLINE    ONLINE    askmdbrac01
ora.gsd        ora.gsd.type   OFFLINE   OFFLINE
ora….network ora….rk.type ONLINE    ONLINE    askmdbrac01
ora.oc4j       ora.oc4j.type  ONLINE    ONLINE    askmdbrac01
ora.ons        ora.ons.type   ONLINE    ONLINE    askmdbrac01
ora….SM1.asm application    ONLINE    ONLINE    askmdbrac01
ora….01.lsnr application    ONLINE    ONLINE    askmdbrac01
ora….l01.gsd application    OFFLINE   OFFLINE
ora….l01.ons application    ONLINE    ONLINE    askmdbrac01
ora….l01.vip ora….t1.type ONLINE    ONLINE    askmdbrac01
ora….SM2.asm application    ONLINE    ONLINE    askmdbrac02
ora….02.lsnr application    ONLINE    ONLINE    askmdbrac02
ora….l02.gsd application    OFFLINE   OFFLINE
ora….l02.ons application    ONLINE    ONLINE    askmdbrac02
ora….l02.vip ora….t1.type ONLINE    ONLINE    askmdbrac02
ora….ry.acfs ora….fs.type ONLINE    OFFLINE
ora.scan1.vip  ora….ip.type ONLINE    ONLINE    askmdbrac02
ora.scan2.vip  ora….ip.type ONLINE    ONLINE    askmdbrac01
ora.scan3.vip  ora….ip.type ONLINE    ONLINE    askmdbrac01
ora.racdb.db ora….se.type ONLINE    ONLINE    askmdbrac01
ora….ice.svc ora….ce.type ONLINE    ONLINE    askmdbrac01
[root@askmdbrac02 ~]# sudo su – grid
[grid@askmdbrac02 ~]$ . ./profile.grid
[grid@askmdbrac02 ~]$ Check_CRS.txt
HA Resource                                        Target     State
———–                                        ——     —–
ora.CRS.dg                                         ONLINE     ONLINE on askmdbrac01
ora.FRA.dg                                         ONLINE     ONLINE on askmdbrac01
ora.LISTENER.lsnr                                  ONLINE     ONLINE on askmdbrac01
ora.LISTENER_SCAN1.lsnr                            ONLINE     ONLINE on askmdbrac02
ora.LISTENER_SCAN2.lsnr                            ONLINE     ONLINE on askmdbrac01
ora.LISTENER_SCAN3.lsnr                            ONLINE     ONLINE on askmdbrac01
ora.PRDSHR_DATA.dg                                 ONLINE     ONLINE on askmdbrac01
ora.asm                                            ONLINE     ONLINE on askmdbrac01
ora.cvu                                            ONLINE     ONLINE on askmdbrac01
ora.gsd                                            OFFLINE    OFFLINE
ora.net1.network                                   ONLINE     ONLINE on askmdbrac01
ora.oc4j                                           ONLINE     ONLINE on askmdbrac01
ora.ons                                            ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.ASM1.asm                           ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.LISTENER_askmdbrac01.lsnr          ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.gsd                                OFFLINE    OFFLINE
ora.askmdbrac01.ons                                ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.vip                                ONLINE     ONLINE on askmdbrac01
ora.askmdbrac02.ASM2.asm                           ONLINE     ONLINE on askmdbrac02
ora.askmdbrac02.LISTENER_askmdbrac02.lsnr          ONLINE     ONLINE on askmdbrac02
ora.askmdbrac02.gsd                                OFFLINE    OFFLINE
ora.askmdbrac02.ons                                ONLINE     ONLINE on askmdbrac02
ora.askmdbrac02.vip                                ONLINE     ONLINE on askmdbrac02
ora.registry.acfs                                  ONLINE     OFFLINE
ora.scan1.vip                                      ONLINE     ONLINE on askmdbrac02
ora.scan2.vip                                      ONLINE     ONLINE on askmdbrac01
ora.scan3.vip                                      ONLINE     ONLINE on askmdbrac01
ora.racdb.db                                     ONLINE     ONLINE on askmdbrac01
ora.racdb.racdb_service.svc                    ONLINE     ONLINE on askmdbrac02
[grid@askmdbrac02 ~]$ srvctl status database -d racdb
Instance racdb1 is running on node askmdbrac01
Instance racdb2 is running on node askmdbrac02
[grid@askmdbrac02 ~]$

This concludes the article on troubleshooting RAC services startup issue.

Thanks

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s