How to find out if you’re accessing a RAC database


How can you tell if you’re accessing a RAC database? Simple!

(  there is nothing in v$database, You need to check v$instance

PARALLEL VARCHAR2(3) Indicates whether the instance is mounted in cluster database mode (YES) or not (NO)

)
You can tell if it is a cluster database by looking to see if the cluster database parameter is set:-

SQL> select name, value from v$parameter where name=’cluster_database’;

NAME VALUE
——————— ———————
cluster_database TRUE
or
set serveroutput on
BEGIN
IF dbms_utility.is_cluster_database THEN
dbms_output.put_line(‘Running in SHARED/RAC mode.’);
ELSE
dbms_output.put_line(‘Running in EXCLUSIVE mode.’);
END IF;
END;
/

Question. I think we have some data in GV$ views if we are using more then one instance?

ANS: All v$ views are built on top of gv$ views. In single instance mode gv$ views will return the same data as the v$ views.

Advertisements

Oracle Local Registry (OLR) – 11gR2


From 11gR2 Oracle introduced “Oracle Local Registry(OLR)”, something new as part of Oracle Clusterware.OLR is node’s local repository,similar to OCR (but local) and is managed by OHASD.It pertains data of local node only and is not shared among other nodes.On a linux environment the configuration file is placed in /etc/oracle location.

$ pwd
/etc/oracle
$ ls -lrt olr*
-rw-r–r– 1 root dba 96 Feb 24 07:34 olr.loc
The olr.loc file consist of the OLR config file location and the crs_home location.

$ cat olr.loc
olrconfig_loc=/u01/app/oracle/grid/11.2.0/cdata/<hostname>.olr
crs_home=/u01/app/oracle/grid/11.2.0
So, by default the location is , olrconfig_loc =<GRID_HOME>/cdata/<hostname>.olr.

To check the location using command (as oracle user):-

$ ocrcheck -local -config
Oracle Local Registry configuration is :
Device/File Name : /u01/app/oracle/grid/11.2.0/cdata/<hostname>.olr
To get more information on it, you need to use “ocrcheck -local”, which must be run as root user.If run as Oracle user, we get [Permission denied] error

[root@nn4040 bin]# ./ocrcheck -local
Status of Oracle Local Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 2552
Available space (kbytes) : 259568
ID : 1311683887
Device/File Name : /u01/app/oracle/grid/11.2.0/cdata/<hostname>.olr
Device/File integrity check succeeded

Local registry integrity check succeeded

Logical corruption check succeeded
The OLR can be dumped in a similar way the OCR is dumped

$ ocrdump -h
Name:
ocrdump – Dump contents of Oracle Cluster/Local Registry to a file.

Synopsis:
ocrdump [-local] [|-stdout] [-backupfile ] [-keyname ] [-xml] [-noheader]

$ ocrdump -local olr.lst –> Normal Text Format
$ ocrdump -local -xml olr_xml.lst –> XML format
The OLR dump shows that it contains data about ORA_CRS_HOME,localhost verion, activeversion,GPnP details,OCR latest backup time and location,information about OCR daily, weekly backup location and node name etc.This formation stored in the OLR is needed by OHASD to start or join a cluster.When the crs is started using crsctl start crs command , the alert<hostname>.log in GRID_HOME/log/<hostname>/ shows –

[ohasd(10303)]CRS-2112:The OLR service started on node nn4040.
2011-03-10 02:01:48.443
[ohasd(10303)]CRS-1301:Oracle High Availability Service started on node nn4040.
2011-03-10 02:01:48.458
………..
………..
Few other administrative commands –

To import a specified file to OLR:

# ocrconfig –local –import file_name

To manually back up OLR:

# ocrconfig –local –manualbackup

To view the contents of the OLR backup file:

ocrdump -local -backupfile olr_backup_file_name

To change the OLR backup location:

ocrconfig -local -backuploc new_olr_backup_path

To restore OLR:

# crsctl stop crs
# ocrconfig -local -restore file_name
# ocrcheck -local
# crsctl start crs
$ cluvfy comp olr
Reference – http://download.oracle.com/docs/cd/E11882_01/rac.112/e16794/votocr.htm#CWADD70700

how to add/delete OCR disks/Voting disks in 11g R2 database


Please see below command output where we’ve ocr and ocr mirror in ASM DG and raw device respectively.
[oracle@RAC2 ~]$ ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 1039908
Used space (kbytes) : 4012
Available space (kbytes) : 1035896
ID : 2141770140
Device/File Name : +DATADG
Device/File integrity check succeeded
Device/File Name : /dev/raw/raw1
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Now we’ll delete ocr mirror device using below command and please note that while deleting OCR was online in my case.
[root@RAC2 bin]# ./ocrconfig -delete /dev/raw/raw1 (CRS was online)
Verify OCR locations again and now ocr mirror is removed from the output.
[root@RAC2 bin]# ./ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 1039908
Used space (kbytes) : 4012
Available space (kbytes) : 1035896
ID : 2141770140
Device/File Name : +DATADG
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
Also note that ocr.loc is updated accordingly.
[root@RAC2 bin]# cat /etc/oracle/ocr.loc
#Device/file /dev/raw/raw1 being deleted
ocrconfig_loc=+DATADG
Adding RAW OCR device again (CRS was online) : Now we’ll add ocr mirror device again using raw device using below command.
=============================
local_only=false[root@RAC2 bin]# ./ocrconfig -add /dev/raw/raw1
Please note that ocr.loc is again updated mentioning about mirrorconfig location.
[root@RAC2 bin]# cat /etc/oracle/ocr.loc
#Device/file getting replaced by device /dev/raw/raw1
ocrconfig_loc=+DATADG
ocrmirrorconfig_loc=/dev/raw/raw1
local_only=false[cd /u01/app/11.2.0/grid/
[root@RAC2 grid]# cd bin
[root@RAC2 bin]# ./ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 1039908
Used space (kbytes) : 4012
Available space (kbytes) : 1035896
ID : 2141770140
Device/File Name : +DATADG
Device/File integrity check succeeded
Device/File Name : /dev/raw/raw1
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
The same is not the case with Voting device. Once you move voting devices from raw devices to ASM Diskgroup, it’s not possible to add or delete voting files from ASM diskgroup as shown below.
[root@RAC2 bin]# ./crsctl add css votedisk /dev/raw/raw2
CRS-4258: Addition and deletion of voting files are not allowed because the voting files are on ASM

Mount ASM diskgroups with new ASM instance


Imagine you have 11gR2 Oracle Restart configuration with database files located in ASM.

After server crash you realized that local disks are corrupted and with local disks you lost all Oracle installations. Even though this is important system you don’t have database backup (always take backups!).

But you managed to save all ASM disks as they were located on separate storage.

This will be small beginner guide on how to help yourself in such situation.


As old server crashed you must create new server configuration, identical as old configuration. Nice thing about ASM is that it keeps it’s metadata in disk header. If disks are intact and headers are not damaged you should be able to mount diskgroups with new ASM instance. But this new instance must be compatible with your diskgroups.

Grid Infrastrcuture and database software were 11.2.0.1 version and this version I will install on new server.

To keep this post short enough steps like creating users, installing ASMLib and other packages, configuring kernel parameters,… are excluded.

List Oracle ASM disks mounted to new server.
With “scandisks” command I will find devices which have been labeled as ASM disks.

# oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
# oracleasm listdisks
DISK1
DISK2
DISK3
DISK4
DISK5
FRA1
Install “Oracle Grid Infrastructure software only” option to avoid automatic Oracle Restart and ASM configuration. This configuration will be performed later manually.

After installation finished run noted perl script as root to configure Grid Infrastructure for a Stand-Alone server.
For my configuration script looks like this:

To configure Grid Infrastructure for a Stand-Alone Server run the following command as the root user:
/u01/app/11.2.0.1/grid/perl/bin/perl -I/u01/app/11.2.0.1/grid/perl/lib -I/u01/app/11.2.0.1/grid/crs/install /u01/app/11.2.0.1/grid/crs/install/roothas.pl
Start cssd if it’s not running.
# ./crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
      1        OFFLINE OFFLINE
ora.diskmon
      1        OFFLINE OFFLINE
# ./crs_start ora.cssd
Attempting to start `ora.cssd` on member `asterix`
Attempting to stop `ora.diskmon` on member `asterix`
Stop of `ora.diskmon` on member `asterix` succeeded.
Attempting to start `ora.diskmon` on member `asterix`
Start of `ora.diskmon` on member `asterix` succeeded.
Start of `ora.cssd` on member `asterix` succeeded.
Create parameter file for ASM instance in $ORACLE_HOME/dbs directory of Grid Infrastructure.
init+ASM.ora
*.asm_diskstring='/dev/oracleasm/disks'
*.asm_power_limit=1
*.diagnostic_dest='/u01/app/grid'
*.instance_type='asm'
*.large_pool_size=12M
*.remote_login_passwordfile='EXCLUSIVE'
Register and start ASM instance.

1
2
3
4
5
6
7
$ export ORACLE_SID=+ASM
$ export ORACLE_HOME=/u01/app/11.2.0.1/grid
$ srvctl add asm -p $ORACLE_HOME/dbs/init+ASM.ora
$ srvctl start asm
$ srvctl status asm
ASM is running on asterix

Now notice what I see when I start ASM configuration assistant

$ ./asmca

These are diskgroups with my database and recovery files.
Click “Mount all” to mount them all.

Install Oracle database software and create parameter file in “$ORACLE_HOME/dbs” to start database.

$ export ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1
$ export ORACLE_SID=ora11gr2
$ cd $ORACLE_HOME/dbs
$ cat initora11gr2.ora
*.spfile='+DATA1/ora11gr2/spfileora11gr2.ora'
$ sqlplus / as sysdba
SQL*Plus: Release 11.2.0.1.0 Production on Wed Oct 29 14:29:37 2014
Copyright (c) 1982, 2009, Oracle.  All rights reserved.
Connected to an idle instance.
SQL> startup
ORACLE instance started.
Total System Global Area  668082176 bytes
Fixed Size                  2216344 bytes
Variable Size             222301800 bytes
Database Buffers          436207616 bytes
Redo Buffers                7356416 bytes
Database mounted.
Database opened.
SQL>
SQL>
SQL> select name from v$datafile;
NAME
--------------------------------------------------------------------------------
+DATA1/ora11gr2/datafile/system.297.844627929
+DATA1/ora11gr2/datafile/sysaux.265.844627967
+DATA1/ora11gr2/datafile/undotbs1.266.844627991
+DATA1/ora11gr2/datafile/users.267.844628031
+DATA2/ora11gr2/datafile/marko.261.859213577

Database is successfully opened and you can register instance using SRVCTL command.

$ srvctl add database -d $ORACLE_SID -o $ORACLE_HOME -p $ORACLE_HOME/dbs/initora11gr2.ora
$ srvctl start database -d $ORACLE_SID
Final status.
$ ./crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA1.dg
               ONLINE  ONLINE       asterix
ora.DATA2.dg
               ONLINE  ONLINE       asterix
ora.FRA1.dg
               ONLINE  ONLINE       asterix
ora.asm
               ONLINE  ONLINE       asterix                  Started
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
      1        ONLINE  ONLINE       asterix
ora.diskmon
      1        ONLINE  ONLINE       asterix
ora.ora11gr2.db
      1        ONLINE  ONLINE       asterix                  Open

 

 

How to add a new SCAN listener in Oracle 11gR2 RAC setup


As an Oracle DBA each one of us might see such activity in the projects with RAC setup. It is not quite very easy to add a SCAN listener when your environment doesn’t use DNS server.

Environment:

2 Node RAC 11gR2 setup
2 SCAN listeners already configured
DNS server is not configured
SCAN was configured using /etc/hosts on the node
Challenge:

When your environment is configured with SCAN with /etc/hosts file on the node, it is not possible to directly add a new SCAN listener using the same /etc/hosts file to the RAC setup.

Steps to add SCAN listener:

Check the current configuration of SCAN and SCAN_LISTENER on both the nodes.
[oracle@rac_node1 ~]# srvctl config scanSCAN name: racscan, Network: 1/192.168.0.0/255.255.255.0/bond0

SCAN VIP name: scan1, IP: /scan-rac/192.168.0.10

SCAN name: racscan, Network: 1/192.168.0.0/255.255.255.0/bond0

SCAN VIP name: scan2, IP: /scan-rac/192.168.0.11

[oracle@rac_node1 ~]# srvctl config scan_listener

SCAN Listener LISTENER_SCAN1 exists. Port: TCP:1521

SCAN Listener LISTENER_SCAN2 exists. Port: TCP:1521

Check the current status of SCAN and SCAN_LISTENER
[oracle@rac_node1 ~]# srvctl status scanSCAN VIP scan1 is enabled

SCAN VIP scan1 is running on node rac_node2

SCAN VIP scan2 is enabled

SCAN VIP scan2 is running on node rac_node1

[oracle@rac_node1 ~]# srvctl status scan_listenerSCAN Listener LISTENER_SCAN1 is enabled

SCAN listener LISTENER_SCAN1 is running on node rac_node2

SCAN Listener LISTENER_SCAN2 is enabled

SCAN listener LISTENER_SCAN2 is running on node rac_node1

I have added a new IP of scan in /etc/hosts of both the nodes as below
192.168.0.10 racscan192.168.0.11 racscan

192.168.0.12 racscan

Stopped scan_listener and scan then modified the existing scan with ROOT privilege.
[oracle@rac_node1 ~]$ srvctl stop scan_listener[oracle@rac_node1 ~]$ srvctl stop scan

[root@rac_node1 ~]# srvctl modify scan -n racscan

Checked the configuration of scan to identify if new IP address of scan is registered.
[root@rac_node1 ~]# srvctl config scanSCAN name: rac-scan, Network: 1/192.168.0.0/255.255.255.0/eth0

SCAN VIP name: scan1, IP: /rac-scan.localdomain/192.168.0.10

SCAN VIP name: scan2, IP: /rac-scan.localdomain/192.168.0.11

There is the PROBLEM the new IP address is not registered in the scan configuration. We will not be able to add a new SCAN using /etc/hosts file on the server.
I have now removed the entries of SCAN in /etc/hosts file and then configured DNS for SCAN IP addresses and VIP’s as well.
Then modified the SCAN configuration.
[root@rac_node1 ~]# srvctl modify scan -n racscan[root@rac_node1 ~]# srvctl config scan

SCAN name: rac-scan, Network: 1/192.168.0.0/255.255.255.0/eth0

SCAN VIP name: scan1, IP: /rac-scan.localdomain/192.168.0.10

SCAN VIP name: scan2, IP: /rac-scan.localdomain/192.168.0.11

SCAN VIP name: scan3, IP: /rac-scan.localdomain/192.168.0.11

Here is the ANSWER, the third IP has been now configured to scan. Let us now start SCAN and SCAN_LISTENER to complete this activity successfully.
[oracle@rac_node1 ~]$ srvctl start scan_listener[oracle@rac_node1 ~]$ srvctl start scan

[oracle@rac_node1 ~]# srvctl status scan_listener

SCAN Listener LISTENER_SCAN1 is enabled

SCAN listener LISTENER_SCAN1 is running on node rac_node2

SCAN Listener LISTENER_SCAN2 is enabled

SCAN listener LISTENER_SCAN2 is running on node rac_node1

SCAN Listener LISTENER_SCAN3 is enabled

SCAN listener LISTENER_SCAN3 is running on node rac_node1

Conclusion:

Adding a new SCAN listener through /etc/hosts is quite a challenge and clusterware software seems not having such intelligence to identify a new IP from /etc/hosts. So the solution that would work fine if you had to add new SCAN listener is through DNS server.

It was very interesting when I have received an activity to ADD a SCAN listener to the existing RAC 11gR2 setup. As an Oracle DBA each one of us might see such activity in the projects with RAC setup. It is not quite very easy to add a SCAN listener when your environment doesn’t use DNS server.

Environment:

2 Node RAC 11gR2 setup
2 SCAN listeners already configured
DNS server is not configured
SCAN was configured using /etc/hosts on the node
Challenge:

When your environment is configured with SCAN with /etc/hosts file on the node, it is not possible to directly add a new SCAN listener using the same /etc/hosts file to the RAC setup.

Steps to add SCAN listener:

Check the current configuration of SCAN and SCAN_LISTENER on both the nodes.
[oracle@rac_node1 ~]# srvctl config scanSCAN name: racscan, Network: 1/192.168.0.0/255.255.255.0/bond0

SCAN VIP name: scan1, IP: /scan-rac/192.168.0.10

SCAN name: racscan, Network: 1/192.168.0.0/255.255.255.0/bond0

SCAN VIP name: scan2, IP: /scan-rac/192.168.0.11

[oracle@rac_node1 ~]# srvctl config scan_listener

SCAN Listener LISTENER_SCAN1 exists. Port: TCP:1521

SCAN Listener LISTENER_SCAN2 exists. Port: TCP:1521

Check the current status of SCAN and SCAN_LISTENER
[oracle@rac_node1 ~]# srvctl status scanSCAN VIP scan1 is enabled

SCAN VIP scan1 is running on node rac_node2

SCAN VIP scan2 is enabled

SCAN VIP scan2 is running on node rac_node1

[oracle@rac_node1 ~]# srvctl status scan_listenerSCAN Listener LISTENER_SCAN1 is enabled

SCAN listener LISTENER_SCAN1 is running on node rac_node2

SCAN Listener LISTENER_SCAN2 is enabled

SCAN listener LISTENER_SCAN2 is running on node rac_node1

I have added a new IP of scan in /etc/hosts of both the nodes as below
192.168.0.10 racscan192.168.0.11 racscan

192.168.0.12 racscan

Stopped scan_listener and scan then modified the existing scan with ROOT privilege.
[oracle@rac_node1 ~]$ srvctl stop scan_listener[oracle@rac_node1 ~]$ srvctl stop scan

[root@rac_node1 ~]# srvctl modify scan -n racscan

Checked the configuration of scan to identify if new IP address of scan is registered.
[root@rac_node1 ~]# srvctl config scanSCAN name: rac-scan, Network: 1/192.168.0.0/255.255.255.0/eth0

SCAN VIP name: scan1, IP: /rac-scan.localdomain/192.168.0.10

SCAN VIP name: scan2, IP: /rac-scan.localdomain/192.168.0.11

There is the PROBLEM the new IP address is not registered in the scan configuration. We will not be able to add a new SCAN using /etc/hosts file on the server.
I have now removed the entries of SCAN in /etc/hosts file and then configured DNS for SCAN IP addresses and VIP’s as well.
Then modified the SCAN configuration.
[root@rac_node1 ~]# srvctl modify scan -n racscan[root@rac_node1 ~]# srvctl config scan

SCAN name: rac-scan, Network: 1/192.168.0.0/255.255.255.0/eth0

SCAN VIP name: scan1, IP: /rac-scan.localdomain/192.168.0.10

SCAN VIP name: scan2, IP: /rac-scan.localdomain/192.168.0.11

SCAN VIP name: scan3, IP: /rac-scan.localdomain/192.168.0.11

Here is the ANSWER, the third IP has been now configured to scan. Let us now start SCAN and SCAN_LISTENER to complete this activity successfully.
[oracle@rac_node1 ~]$ srvctl start scan_listener[oracle@rac_node1 ~]$ srvctl start scan

[oracle@rac_node1 ~]# srvctl status scan_listener

SCAN Listener LISTENER_SCAN1 is enabled

SCAN listener LISTENER_SCAN1 is running on node rac_node2

SCAN Listener LISTENER_SCAN2 is enabled

SCAN listener LISTENER_SCAN2 is running on node rac_node1

SCAN Listener LISTENER_SCAN3 is enabled

SCAN listener LISTENER_SCAN3 is running on node rac_node1

Conclusion:

Adding a new SCAN listener through /etc/hosts is quite a challenge and clusterware software seems not having such intelligence to identify a new IP from /etc/hosts. So the solution that would work fine if you had to add new SCAN listener is through DNS server.

Troubleshooting RAC Services Startup


Hi guys, I would like to share a very beautful article by Mr. Srikrishna Murthy Annam ( https://learnwithme11g.wordpress.com)

This article gives you the troubleshooting steps for this particular issue. These steps may also help you to troubleshoot if you have issues starting RAC services.

STEP 1: Reboot the rac servers and Check services after re-booting the servers. The services are still down.

STEP 2: Verify storage on both rac servers

[grid@askmdbrac01 ~]$ /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[grid@askmdbrac01 ~]$ /etc/init.d/oracleasm listdisks
CRSVOL1
DISK1
FRADISK1
[grid@askmdbrac01 ~]$ multipath -ll
-bash: multipath: command not found
[grid@askmdbrac01 ~]$ exit
logout
[oracle@askmdbrac01 ~]$ sudo su –
[root@askmdbrac01 ~]# multipath -ll
vpath2 (360050768019280fb280000000000055e) dm-10 IBM,2145
[size=200G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=50][active]
\_ 2:0:1:2 sdf        8:80  [active][ready]
\_ 3:0:1:2 sdl        8:176 [active][ready]
\_ round-robin 0 [prio=10][enabled]
\_ 2:0:0:2 sdc        8:32  [active][ready]
\_ 3:0:0:2 sdi        8:128 [active][ready]
vpath1 (360050768019280fb280000000000055d) dm-12 IBM,2145
[size=200G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=50][active]
\_ 3:0:0:1 sdh        8:112 [active][ready]
\_ round-robin 0 [prio=10][enabled]
\_ 2:0:1:1 sde        8:64  [active][ready]
\_ 3:0:1:1 sdk        8:160 [active][ready]
vpath0 (360050768019280fb280000000000055c) dm-11 IBM,2145
[size=10G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=50][active]
\_ 2:0:1:0 sdd        8:48  [active][ready]
\_ 3:0:1:0 sdj        8:144 [active][ready]
\_ round-robin 0 [prio=10][enabled]
\_ 3:0:0:0 sdg        8:96  [active][ready]
[root@askmdbrac01 ~]# exit[grid@askmdbrac02 ~]$ /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[grid@askmdbrac02 ~]$ /etc/init.d/oracleasm listdisks
CRSVOL1
DISK1
FRADISK1
[grid@askmdbrac02 ~]$ multipath -ll
-bash: multipath: command not found
[grid@askmdbrac02 ~]$ exit
logout
[oracle@askmdbrac02 ~]$ sudo su –
[root@askmdbrac02 ~]# multipath -ll
vpath2 (360050768019280fb280000000000055e) dm-10 IBM,2145
[size=200G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=50][active]
\_ 2:0:1:2 sdf        8:80  [active][ready]
\_ 3:0:0:2 sdi        8:128 [active][ready]
\_ round-robin 0 [prio=10][enabled]
\_ 2:0:0:2 sdc        8:32  [active][ready]
\_ 3:0:1:2 sdl        8:176 [active][ready]
vpath1 (360050768019280fb280000000000055d) dm-12 IBM,2145
[size=200G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=50][active]
\_ 3:0:1:1 sdk        8:160 [active][ready]
\_ round-robin 0 [prio=10][enabled]
\_ 2:0:1:1 sde        8:64  [active][ready]
\_ 3:0:0:1 sdh        8:112 [active][ready]
vpath0 (360050768019280fb280000000000055c) dm-11 IBM,2145
[size=10G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=50][active]
\_ 2:0:1:0 sdd        8:48  [active][ready]
\_ 3:0:0:0 sdg        8:96  [active][ready]
\_ round-robin 0 [prio=10][enabled]
\_ 3:0:1:0 sdj        8:144 [active][ready]
[root@askmdbrac02 ~]#

Observation : Storage is fine and i am able to see ASM disks from both the nodes. Storage is configured with multipathing. I also verified multipathing and status shows fine.

STEP 3: Check high availability services and cluster ready services on both nodes with the following commands

[grid@askmdbrac01 ~]$ crsctl check has
CRS-4638: Oracle High Availability Services is online
[grid@askmdbrac01 ~]$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
[grid@askmdbrac01 ~]$
[grid@askmdbrac01 ~]$ crsctl check cluster
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
[grid@askmdbrac01 ~]$[grid@askmdbrac02 ~]$ crsctl check has
CRS-4638: Oracle High Availability Services is online
[grid@askmdbrac02 ~]$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
[grid@askmdbrac02 ~]$
[grid@askmdbrac02 ~]$ crsctl check cluster
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
[grid@askmdbrac02 ~]$

Observation : The high availability services are up , but crs services are down.

STEP 4: Do ping and nslookup for other rac nodes from each node to make sure that each node in cluster is accessible from every other node. Check this for private ips, virtual ips and for scan ips.

Observation : The ping and nslookup are working fine and all the nodes are reachable from every other node.

STEP 5 : Then verify the log files from grid infrastructure and see if you can find any error messages from log files. The cluster log files are located at $ORACLE_HOME/log/askmdbrac01 and $ORACLE_HOME/log/askmdbrac02.

[cssd(11175)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /o001/home/11.2.0.2/grid/log/askmdbrac01/cssd/ocssd.log
2013-04-02 19:38:27.425
[ohasd(6876)]CRS-2765:Resource ‘ora.diskmon’ has failed on server ‘askmdbrac01′.
2013-04-02 19:38:39.988
[cssd(11175)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /o001/home/11.2.0.2/grid/log/askmdbrac01/cssd/ocssd.log
2013-04-02 19:38:55.007
[cssd(11175)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /o001/home/11.2.0.2/grid/log/askmdbrac01/cssd/ocssd.log
2013-04-02 19:39:10.025
…..
…..
[cssd(11175)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /o001/home/11.2.0.2/grid/log/askmdbrac01/cssd/ocssd.log
2013-04-02 19:48:23.836
[/o001/home/11.2.0.2/grid/bin/cssdagent(11163)]CRS-5818:Aborted command ‘start for resource: ora.cssd 1 1′ for resource ‘ora.cssd’. Details at (:CRSAGF00113:) {0:27:4} in /o001/home/11.2.0.2/grid/log/askmdbrac01/agent/ohasd/oracssdagent_root/oracssdagent_root.log.
2013-04-02 19:48:23.837
[cssd(11175)]CRS-1656:The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in /o001/home/11.2.0.2/grid/log/askmdbrac01/cssd/ocssd.log
2013-04-02 19:48:23.838
[cssd(11175)]CRS-1603:CSSD on node askmdbrac01 shutdown by user.
2013-04-02 19:48:25.730
[cssd(11175)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /o001/home/11.2.0.2/grid/log/askmdbrac01/cssd/ocssd.log
2013-04-02 19:48:29.863
[ohasd(6876)]CRS-2765:Resource ‘ora.cssdmonitor’ has failed on server ‘askmdbrac01′.
2013-04-02 19:48:30.126
[ohasd(6876)]CRS-2771:Maximum restart attempts reached for resource ‘ora.cssd’; will not restart.
2013-04-02 19:48:35.833
[ohasd(6876)]CRS-2765:Resource ‘ora.diskmon’ has failed on server ‘askmdbrac01′.

Observation : Found from the cluster alert log file (alertaskmdbrac01.log) that the cluster services can’t find voting disk. So it could be due the in-accessible asm storage or due to currupted voting disks.

Then i tried to see if i can see voting disks through command.

SETP 6 :  Check the voting disks availability ( 11gR2 has voting disks in asm. I don’t have asm instance up now. So i will get error if i try to query the voting disks on asm)

[root@askmdbrac01 ~]# /o001/home/11.2.0.2/grid/bin/crsctl query css votedisk
Unable to communicate with the Cluster Synchronization Services daemon.
[root@askmdbrac01 ~]#[grid@askmdbrac02 ~]$ /o001/home/11.2.0.2/grid/bin/crsctl query css votedisk
Unable to communicate with the Cluster Synchronization Services daemon.
[grid@askmdbrac02 ~]$

Observation : I am not able to query the voting disks. Now i have only one option to look at the asm storage accessibility.

STEP 7 :  Tried re-enabling the asmlibs as follows on both the nodes …

[root@askmdbrac01 ~]# /etc/init.d/oracleasm restart
Dropping Oracle ASMLib disks:                              [  OK  ]
Shutting down the Oracle ASMLib driver:                    [  OK  ]
[root@askmdbrac01 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: no
Checking if /dev/oracleasm is mounted: no
[root@askmdbrac01 ~]# /etc/init.d/oracleasm start
[root@askmdbrac01 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: no
Checking if /dev/oracleasm is mounted: no
[root@askmdbrac01 ~]# /etc/init.d/oracleasm enable
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@askmdbrac01 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[root@askmdbrac01 ~]# /etc/init.d/oracleasm listdisks
CRSVOL1
DISK1
FRADISK1
[root@askmdbrac01 ~]#[root@askmdbrac02 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[root@askmdbrac02 ~]# /etc/init.d/oracleasm restart
Dropping Oracle ASMLib disks:                              [  OK  ]
Shutting down the Oracle ASMLib driver:                    [  OK  ]
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@askmdbrac02 ~]#  /etc/init.d/oracleasm listdisks
CRSVOL1
DISK1
FRADISK1
[root@askmdbrac02 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[root@askmdbrac02 ~]#

Observation : Not able to start services even after restarting the asmlib.

STEP 8 : Tried to re-configure the asmlib as below …..

[root@askmdbrac01 ~]# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.This will configure the on-boot properties of the Oracle ASM library
driver.  The following questions will determine whether the driver is
loaded on boot and what permissions it will have.  The current values
will be shown in brackets (‘[]’).  Hitting <ENTER> without typing an
answer will keep that current value.  Ctrl-C will abort.Default user to own the driver interface []: grid
Default group to own the driver interface []: asmadmin
Start Oracle ASM library driver on boot (y/n) [y]:
Scan for Oracle ASM disks on boot (y/n) [y]:
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@askmdbrac01 ~]#
[root@askmdbrac01 ~]# /etc/init.d/oracleasm stop
Dropping Oracle ASMLib disks:                              [  OK  ]
Shutting down the Oracle ASMLib driver:                    [FAILED]
[root@askmdbrac01 ~]# /etc/init.d/oracleasm start
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@askmdbrac01 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[root@askmdbrac01 ~]#[root@askmdbrac02 ~]# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library
driver.  The following questions will determine whether the driver is
loaded on boot and what permissions it will have.  The current values
will be shown in brackets (‘[]’).  Hitting <ENTER> without typing an
answer will keep that current value.  Ctrl-C will abort.

Default user to own the driver interface []: grid
Default group to own the driver interface []: asmadmin
Start Oracle ASM library driver on boot (y/n) [y]:
Scan for Oracle ASM disks on boot (y/n) [y]:
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@askmdbrac02 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[root@askmdbrac02 ~]# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library
driver.  The following questions will determine whether the driver is
loaded on boot and what permissions it will have.  The current values
will be shown in brackets (‘[]’).  Hitting <ENTER> without typing an
answer will keep that current value.  Ctrl-C will abort.

Default user to own the driver interface [grid]:
Default group to own the driver interface [asmadmin]:
Start Oracle ASM library driver on boot (y/n) [y]:
Scan for Oracle ASM disks on boot (y/n) [y]:
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@askmdbrac02 ~]#
[root@askmdbrac02 ~]# /etc/init.d/oracleasm stop
Dropping Oracle ASMLib disks:                              [  OK  ]
Shutting down the Oracle ASMLib driver:                    [FAILED]
[root@askmdbrac02 ~]# /etc/init.d/oracleasm start
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@askmdbrac02 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[root@askmdbrac02 ~]#

Observation : Identified that the asm library drivers configuration is wrong and i re-configured with correct user and group. See the difference in the lines when it is asking for prompt on second node. I executed this command twice to show you the difference when it is properly configured and when it is not properly configured.

STEP 9 : Stop and Start cluster services on both the nodes.

[root@askmdbrac01 ~]#  /o001/home/11.2.0.2/grid/bin/crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on ‘askmdbrac01′
CRS-2673: Attempting to stop ‘ora.crsd’ on ‘askmdbrac01′
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on ‘askmdbrac01′
CRS-2673: Attempting to stop ‘ora.LISTENER.lsnr’ on ‘askmdbrac01′
CRS-2673: Attempting to stop ‘ora.racdb.racdb_service.svc’ on ‘askmdbrac01′
CRS-2673: Attempting to stop ‘ora.LISTENER_SCAN1.lsnr’ on ‘askmdbrac01′
CRS-2677: Stop of ‘ora.LISTENER.lsnr’ on ‘askmdbrac01′ succeeded
CRS-2677: Stop of ‘ora.LISTENER_SCAN1.lsnr’ on ‘askmdbrac01′ succeeded
CRS-2673: Attempting to stop ‘ora.scan1.vip’ on ‘askmdbrac01′
CRS-2677: Stop of ‘ora.scan1.vip’ on ‘askmdbrac01′ succeeded
CRS-2672: Attempting to start ‘ora.scan1.vip’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.racdb.racdb_service.svc’ on ‘askmdbrac01′ succeeded
CRS-2673: Attempting to stop ‘ora.askmdbrac01.vip’ on ‘askmdbrac01′
CRS-2673: Attempting to stop ‘ora.CRS.dg’ on ‘askmdbrac01′
CRS-2673: Attempting to stop ‘ora.racdb.db’ on ‘askmdbrac01′
CRS-2677: Stop of ‘ora.askmdbrac01.vip’ on ‘askmdbrac01′ succeeded
CRS-2672: Attempting to start ‘ora.askmdbrac01.vip’ on ‘askmdbrac02′
CRS-2676: Start of ‘ora.scan1.vip’ on ‘askmdbrac02′ succeeded
CRS-2672: Attempting to start ‘ora.LISTENER_SCAN1.lsnr’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.racdb.db’ on ‘askmdbrac01′ succeeded
CRS-2673: Attempting to stop ‘ora.FRA.dg’ on ‘askmdbrac01′
CRS-2673: Attempting to stop ‘ora.PRDSHR_DATA.dg’ on ‘askmdbrac01′
CRS-2676: Start of ‘ora.askmdbrac01.vip’ on ‘askmdbrac02′ succeeded
CRS-2677: Stop of ‘ora.PRDSHR_DATA.dg’ on ‘askmdbrac01′ succeeded
CRS-2676: Start of ‘ora.LISTENER_SCAN1.lsnr’ on ‘askmdbrac02′ succeeded
CRS-2677: Stop of ‘ora.FRA.dg’ on ‘askmdbrac01′ succeeded
CRS-2677: Stop of ‘ora.CRS.dg’ on ‘askmdbrac01′ succeeded
CRS-2673: Attempting to stop ‘ora.asm’ on ‘askmdbrac01′
CRS-2677: Stop of ‘ora.asm’ on ‘askmdbrac01′ succeeded
CRS-2673: Attempting to stop ‘ora.ons’ on ‘askmdbrac01′
CRS-2677: Stop of ‘ora.ons’ on ‘askmdbrac01′ succeeded
CRS-2673: Attempting to stop ‘ora.net1.network’ on ‘askmdbrac01′
CRS-2677: Stop of ‘ora.net1.network’ on ‘askmdbrac01′ succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on ‘askmdbrac01′ has completed
CRS-2677: Stop of ‘ora.crsd’ on ‘askmdbrac01′ succeeded
CRS-2673: Attempting to stop ‘ora.mdnsd’ on ‘askmdbrac01′
CRS-2673: Attempting to stop ‘ora.crf’ on ‘askmdbrac01′
CRS-2673: Attempting to stop ‘ora.ctssd’ on ‘askmdbrac01′
CRS-2673: Attempting to stop ‘ora.evmd’ on ‘askmdbrac01′
CRS-2673: Attempting to stop ‘ora.asm’ on ‘askmdbrac01′
CRS-2677: Stop of ‘ora.mdnsd’ on ‘askmdbrac01′ succeeded
CRS-2677: Stop of ‘ora.crf’ on ‘askmdbrac01′ succeeded
CRS-2677: Stop of ‘ora.evmd’ on ‘askmdbrac01′ succeeded
CRS-2677: Stop of ‘ora.asm’ on ‘askmdbrac01′ succeeded
CRS-2673: Attempting to stop ‘ora.cluster_interconnect.haip’ on ‘askmdbrac01′
CRS-2677: Stop of ‘ora.cluster_interconnect.haip’ on ‘askmdbrac01′ succeeded
CRS-2677: Stop of ‘ora.ctssd’ on ‘askmdbrac01′ succeeded
CRS-2673: Attempting to stop ‘ora.cssd’ on ‘askmdbrac01′
CRS-2677: Stop of ‘ora.cssd’ on ‘askmdbrac01′ succeeded
CRS-2673: Attempting to stop ‘ora.gipcd’ on ‘askmdbrac01′
CRS-2673: Attempting to stop ‘ora.diskmon’ on ‘askmdbrac01′
CRS-2677: Stop of ‘ora.gipcd’ on ‘askmdbrac01′ succeeded
CRS-2673: Attempting to stop ‘ora.gpnpd’ on ‘askmdbrac01′
CRS-2677: Stop of ‘ora.gpnpd’ on ‘askmdbrac01′ succeeded
CRS-2677: Stop of ‘ora.diskmon’ on ‘askmdbrac01′ succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on ‘askmdbrac01′ has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@askmdbrac01 ~]# /o001/home/11.2.0.2/grid/bin/crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root@askmdbrac01 ~]#[root@askmdbrac02 ~]# /o001/home/11.2.0.2/grid/bin/crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on ‘askmdbrac02′
CRS-2673: Attempting to stop ‘ora.crsd’ on ‘askmdbrac02′
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on ‘askmdbrac02′
CRS-2673: Attempting to stop ‘ora.LISTENER_SCAN1.lsnr’ on ‘askmdbrac02′
CRS-2673: Attempting to stop ‘ora.askmdbrac02.vip’ on ‘askmdbrac02′
CRS-2673: Attempting to stop ‘ora.oc4j’ on ‘askmdbrac02′
CRS-2673: Attempting to stop ‘ora.LISTENER.lsnr’ on ‘askmdbrac02′
CRS-2673: Attempting to stop ‘ora.LISTENER_SCAN3.lsnr’ on ‘askmdbrac02′
CRS-2673: Attempting to stop ‘ora.cvu’ on ‘askmdbrac02′
CRS-2673: Attempting to stop ‘ora.LISTENER_SCAN2.lsnr’ on ‘askmdbrac02′
CRS-2673: Attempting to stop ‘ora.CRS.dg’ on ‘askmdbrac02′
CRS-2673: Attempting to stop ‘ora.racdb.racdb_service.svc’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.askmdbrac02.vip’ on ‘askmdbrac02′ succeeded
CRS-2677: Stop of ‘ora.racdb.racdb_service.svc’ on ‘askmdbrac02′ succeeded
CRS-2673: Attempting to stop ‘ora.racdb.db’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.LISTENER_SCAN1.lsnr’ on ‘askmdbrac02′ succeeded
CRS-2673: Attempting to stop ‘ora.scan1.vip’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.LISTENER.lsnr’ on ‘askmdbrac02′ succeeded
CRS-2673: Attempting to stop ‘ora.askmdbrac01.vip’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.scan1.vip’ on ‘askmdbrac02′ succeeded
CRS-2677: Stop of ‘ora.LISTENER_SCAN2.lsnr’ on ‘askmdbrac02′ succeeded
CRS-2673: Attempting to stop ‘ora.scan2.vip’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.askmdbrac01.vip’ on ‘askmdbrac02′ succeeded
CRS-2677: Stop of ‘ora.LISTENER_SCAN3.lsnr’ on ‘askmdbrac02′ succeeded
CRS-2673: Attempting to stop ‘ora.scan3.vip’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.scan2.vip’ on ‘askmdbrac02′ succeeded
CRS-2677: Stop of ‘ora.scan3.vip’ on ‘askmdbrac02′ succeeded
CRS-2677: Stop of ‘ora.racdb.db’ on ‘askmdbrac02′ succeeded
CRS-2673: Attempting to stop ‘ora.FRA.dg’ on ‘askmdbrac02′
CRS-2673: Attempting to stop ‘ora.PRDSHR_DATA.dg’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.FRA.dg’ on ‘askmdbrac02′ succeeded
CRS-2677: Stop of ‘ora.PRDSHR_DATA.dg’ on ‘askmdbrac02′ succeeded
CRS-2677: Stop of ‘ora.oc4j’ on ‘askmdbrac02′ succeeded
CRS-2677: Stop of ‘ora.cvu’ on ‘askmdbrac02′ succeeded
CRS-2677: Stop of ‘ora.CRS.dg’ on ‘askmdbrac02′ succeeded
CRS-2673: Attempting to stop ‘ora.asm’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.asm’ on ‘askmdbrac02′ succeeded
CRS-2673: Attempting to stop ‘ora.ons’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.ons’ on ‘askmdbrac02′ succeeded
CRS-2673: Attempting to stop ‘ora.net1.network’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.net1.network’ on ‘askmdbrac02′ succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on ‘askmdbrac02′ has completed
CRS-2677: Stop of ‘ora.crsd’ on ‘askmdbrac02′ succeeded
CRS-2673: Attempting to stop ‘ora.ctssd’ on ‘askmdbrac02′
CRS-2673: Attempting to stop ‘ora.evmd’ on ‘askmdbrac02′
CRS-2673: Attempting to stop ‘ora.asm’ on ‘askmdbrac02′
CRS-2673: Attempting to stop ‘ora.mdnsd’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.evmd’ on ‘askmdbrac02′ succeeded
CRS-2677: Stop of ‘ora.mdnsd’ on ‘askmdbrac02′ succeeded
CRS-2677: Stop of ‘ora.asm’ on ‘askmdbrac02′ succeeded
CRS-2673: Attempting to stop ‘ora.cluster_interconnect.haip’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.cluster_interconnect.haip’ on ‘askmdbrac02′ succeeded
CRS-2677: Stop of ‘ora.ctssd’ on ‘askmdbrac02′ succeeded
CRS-2673: Attempting to stop ‘ora.cssd’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.cssd’ on ‘askmdbrac02′ succeeded
CRS-2673: Attempting to stop ‘ora.diskmon’ on ‘askmdbrac02′
CRS-2673: Attempting to stop ‘ora.crf’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.crf’ on ‘askmdbrac02′ succeeded
CRS-2673: Attempting to stop ‘ora.gipcd’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.diskmon’ on ‘askmdbrac02′ succeeded
CRS-2677: Stop of ‘ora.gipcd’ on ‘askmdbrac02′ succeeded
CRS-2673: Attempting to stop ‘ora.gpnpd’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.gpnpd’ on ‘askmdbrac02′ succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on ‘askmdbrac02′ has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@askmdbrac02 ~]# /o001/home/11.2.0.2/grid/bin/crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root@askmdbrac02 ~]#

Observation : Cluster ready services started without any issues. Now i need to verify the services status.

STEP 10 :  Verify the status.

[root@askmdbrac01 ~]# /o001/home/11.2.0.2/grid/bin/crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
[root@askmdbrac01 ~]# /o001/home/11.2.0.2/grid/bin/crs_stat -t
Name           Type           Target    State     Host
————————————————————
ora.CRS.dg     ora….up.type ONLINE    ONLINE    askmdbrac01
ora.FRA.dg     ora….up.type ONLINE    ONLINE    askmdbrac01
ora….ER.lsnr ora….er.type ONLINE    ONLINE    askmdbrac01
ora….N1.lsnr ora….er.type ONLINE    OFFLINE
ora….N2.lsnr ora….er.type ONLINE    ONLINE    askmdbrac01
ora….N3.lsnr ora….er.type ONLINE    ONLINE    askmdbrac01
ora….DATA.dg ora….up.type ONLINE    ONLINE    askmdbrac01
ora.asm        ora.asm.type   ONLINE    ONLINE    askmdbrac01
ora.cvu        ora.cvu.type   ONLINE    ONLINE    askmdbrac01
ora.gsd        ora.gsd.type   OFFLINE   OFFLINE
ora….network ora….rk.type ONLINE    ONLINE    askmdbrac01
ora.oc4j       ora.oc4j.type  ONLINE    ONLINE    askmdbrac01
ora.ons        ora.ons.type   ONLINE    ONLINE    askmdbrac01
ora….SM1.asm application    ONLINE    ONLINE    askmdbrac01
ora….01.lsnr application    ONLINE    ONLINE    askmdbrac01
ora….l01.gsd application    OFFLINE   OFFLINE
ora….l01.ons application    ONLINE    ONLINE    askmdbrac01
ora….l01.vip ora….t1.type ONLINE    ONLINE    askmdbrac01
ora….SM2.asm application    ONLINE    ONLINE    askmdbrac02
ora….02.lsnr application    ONLINE    OFFLINE
ora….l02.gsd application    OFFLINE   OFFLINE
ora….l02.ons application    ONLINE    OFFLINE
ora….l02.vip ora….t1.type ONLINE    OFFLINE
ora….ry.acfs ora….fs.type ONLINE    OFFLINE
ora.scan1.vip  ora….ip.type ONLINE    OFFLINE
ora.scan2.vip  ora….ip.type ONLINE    ONLINE    askmdbrac01
ora.scan3.vip  ora….ip.type ONLINE    ONLINE    askmdbrac01
ora.racdb.db ora….se.type ONLINE    ONLINE    askmdbrac01
ora….ice.svc ora….ce.type ONLINE    ONLINE    askmdbrac01
[root@askmdbrac01 ~]# sudo su – grid
[grid@askmdbrac01 ~]$ . ./profile.grid
[grid@askmdbrac01 ~]$ Check_CRS.txt
HA Resource                                        Target     State
———–                                        ——     —–
ora.CRS.dg                                         ONLINE     ONLINE on askmdbrac01
ora.FRA.dg                                         ONLINE     ONLINE on askmdbrac01
ora.LISTENER.lsnr                                  ONLINE     ONLINE on askmdbrac01
ora.LISTENER_SCAN1.lsnr                            ONLINE     ONLINE on askmdbrac02
ora.LISTENER_SCAN2.lsnr                            ONLINE     ONLINE on askmdbrac01
ora.LISTENER_SCAN3.lsnr                            ONLINE     ONLINE on askmdbrac01
ora.PRDSHR_DATA.dg                                 ONLINE     ONLINE on askmdbrac01
ora.asm                                            ONLINE     ONLINE on askmdbrac01
ora.cvu                                            ONLINE     ONLINE on askmdbrac01
ora.gsd                                            OFFLINE    OFFLINE
ora.net1.network                                   ONLINE     ONLINE on askmdbrac01
ora.oc4j                                           ONLINE     ONLINE on askmdbrac01
ora.ons                                            ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.ASM1.asm                           ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.LISTENER_askmdbrac01.lsnr          ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.gsd                                OFFLINE    OFFLINE
ora.askmdbrac01.ons                                ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.vip                                ONLINE     ONLINE on askmdbrac01
ora.askmdbrac02.ASM2.asm                           ONLINE     ONLINE on askmdbrac02
ora.askmdbrac02.LISTENER_askmdbrac02.lsnr          ONLINE     ONLINE on askmdbrac02
ora.askmdbrac02.gsd                                OFFLINE    OFFLINE
ora.askmdbrac02.ons                                ONLINE     ONLINE on askmdbrac02
ora.askmdbrac02.vip                                ONLINE     ONLINE on askmdbrac02
ora.registry.acfs                                  ONLINE     OFFLINE
ora.scan1.vip                                      ONLINE     ONLINE on askmdbrac02
ora.scan2.vip                                      ONLINE     ONLINE on askmdbrac01
ora.scan3.vip                                      ONLINE     ONLINE on askmdbrac01
ora.racdb.db                                     ONLINE     ONLINE on askmdbrac01
ora.racdb.racdb_service.svc                    ONLINE     ONLINE on askmdbrac02
[grid@askmdbrac01 ~]$ Check_CRS.txt
HA Resource                                        Target     State
———–                                        ——     —–
ora.CRS.dg                                         ONLINE     ONLINE on askmdbrac01
ora.FRA.dg                                         ONLINE     ONLINE on askmdbrac01
ora.LISTENER.lsnr                                  ONLINE     ONLINE on askmdbrac01
ora.LISTENER_SCAN1.lsnr                            ONLINE     ONLINE on askmdbrac02
ora.LISTENER_SCAN2.lsnr                            ONLINE     ONLINE on askmdbrac01
ora.LISTENER_SCAN3.lsnr                            ONLINE     ONLINE on askmdbrac01
ora.PRDSHR_DATA.dg                                 ONLINE     ONLINE on askmdbrac01
ora.asm                                            ONLINE     ONLINE on askmdbrac01
ora.cvu                                            ONLINE     ONLINE on askmdbrac01
ora.gsd                                            OFFLINE    OFFLINE
ora.net1.network                                   ONLINE     ONLINE on askmdbrac01
ora.oc4j                                           ONLINE     ONLINE on askmdbrac01
ora.ons                                            ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.ASM1.asm                           ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.LISTENER_askmdbrac01.lsnr          ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.gsd                                OFFLINE    OFFLINE
ora.askmdbrac01.ons                                ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.vip                                ONLINE     ONLINE on askmdbrac01
ora.askmdbrac02.ASM2.asm                           ONLINE     ONLINE on askmdbrac02
ora.askmdbrac02.LISTENER_askmdbrac02.lsnr          ONLINE     ONLINE on askmdbrac02
ora.askmdbrac02.gsd                                OFFLINE    OFFLINE
ora.askmdbrac02.ons                                ONLINE     ONLINE on askmdbrac02
ora.askmdbrac02.vip                                ONLINE     ONLINE on askmdbrac02
ora.registry.acfs                                  ONLINE     OFFLINE
ora.scan1.vip                                      ONLINE     ONLINE on askmdbrac02
ora.scan2.vip                                      ONLINE     ONLINE on askmdbrac01
ora.scan3.vip                                      ONLINE     ONLINE on askmdbrac01
ora.racdb.db                                     ONLINE     ONLINE on askmdbrac01
ora.racdb.racdb_service.svc                    ONLINE     ONLINE on askmdbrac02
[grid@askmdbrac01 ~]$ srvctl status database -d racdb
Instance racdb1 is running on node askmdbrac01
Instance racdb2 is running on node askmdbrac02
[grid@askmdbrac01 ~]$[root@askmdbrac02 ~]# /o001/home/11.2.0.2/grid/bin/crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
[root@askmdbrac02 ~]# /o001/home/11.2.0.2/grid/bin/crs_stat -t
Name           Type           Target    State     Host
————————————————————
ora.CRS.dg     ora….up.type ONLINE    ONLINE    askmdbrac01
ora.FRA.dg     ora….up.type ONLINE    ONLINE    askmdbrac01
ora….ER.lsnr ora….er.type ONLINE    ONLINE    askmdbrac01
ora….N1.lsnr ora….er.type ONLINE    ONLINE    askmdbrac02
ora….N2.lsnr ora….er.type ONLINE    ONLINE    askmdbrac01
ora….N3.lsnr ora….er.type ONLINE    ONLINE    askmdbrac01
ora….DATA.dg ora….up.type ONLINE    ONLINE    askmdbrac01
ora.asm        ora.asm.type   ONLINE    ONLINE    askmdbrac01
ora.cvu        ora.cvu.type   ONLINE    ONLINE    askmdbrac01
ora.gsd        ora.gsd.type   OFFLINE   OFFLINE
ora….network ora….rk.type ONLINE    ONLINE    askmdbrac01
ora.oc4j       ora.oc4j.type  ONLINE    ONLINE    askmdbrac01
ora.ons        ora.ons.type   ONLINE    ONLINE    askmdbrac01
ora….SM1.asm application    ONLINE    ONLINE    askmdbrac01
ora….01.lsnr application    ONLINE    ONLINE    askmdbrac01
ora….l01.gsd application    OFFLINE   OFFLINE
ora….l01.ons application    ONLINE    ONLINE    askmdbrac01
ora….l01.vip ora….t1.type ONLINE    ONLINE    askmdbrac01
ora….SM2.asm application    ONLINE    ONLINE    askmdbrac02
ora….02.lsnr application    ONLINE    ONLINE    askmdbrac02
ora….l02.gsd application    OFFLINE   OFFLINE
ora….l02.ons application    ONLINE    ONLINE    askmdbrac02
ora….l02.vip ora….t1.type ONLINE    ONLINE    askmdbrac02
ora….ry.acfs ora….fs.type ONLINE    OFFLINE
ora.scan1.vip  ora….ip.type ONLINE    ONLINE    askmdbrac02
ora.scan2.vip  ora….ip.type ONLINE    ONLINE    askmdbrac01
ora.scan3.vip  ora….ip.type ONLINE    ONLINE    askmdbrac01
ora.racdb.db ora….se.type ONLINE    ONLINE    askmdbrac01
ora….ice.svc ora….ce.type ONLINE    ONLINE    askmdbrac01
[root@askmdbrac02 ~]# sudo su – grid
[grid@askmdbrac02 ~]$ . ./profile.grid
[grid@askmdbrac02 ~]$ Check_CRS.txt
HA Resource                                        Target     State
———–                                        ——     —–
ora.CRS.dg                                         ONLINE     ONLINE on askmdbrac01
ora.FRA.dg                                         ONLINE     ONLINE on askmdbrac01
ora.LISTENER.lsnr                                  ONLINE     ONLINE on askmdbrac01
ora.LISTENER_SCAN1.lsnr                            ONLINE     ONLINE on askmdbrac02
ora.LISTENER_SCAN2.lsnr                            ONLINE     ONLINE on askmdbrac01
ora.LISTENER_SCAN3.lsnr                            ONLINE     ONLINE on askmdbrac01
ora.PRDSHR_DATA.dg                                 ONLINE     ONLINE on askmdbrac01
ora.asm                                            ONLINE     ONLINE on askmdbrac01
ora.cvu                                            ONLINE     ONLINE on askmdbrac01
ora.gsd                                            OFFLINE    OFFLINE
ora.net1.network                                   ONLINE     ONLINE on askmdbrac01
ora.oc4j                                           ONLINE     ONLINE on askmdbrac01
ora.ons                                            ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.ASM1.asm                           ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.LISTENER_askmdbrac01.lsnr          ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.gsd                                OFFLINE    OFFLINE
ora.askmdbrac01.ons                                ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.vip                                ONLINE     ONLINE on askmdbrac01
ora.askmdbrac02.ASM2.asm                           ONLINE     ONLINE on askmdbrac02
ora.askmdbrac02.LISTENER_askmdbrac02.lsnr          ONLINE     ONLINE on askmdbrac02
ora.askmdbrac02.gsd                                OFFLINE    OFFLINE
ora.askmdbrac02.ons                                ONLINE     ONLINE on askmdbrac02
ora.askmdbrac02.vip                                ONLINE     ONLINE on askmdbrac02
ora.registry.acfs                                  ONLINE     OFFLINE
ora.scan1.vip                                      ONLINE     ONLINE on askmdbrac02
ora.scan2.vip                                      ONLINE     ONLINE on askmdbrac01
ora.scan3.vip                                      ONLINE     ONLINE on askmdbrac01
ora.racdb.db                                     ONLINE     ONLINE on askmdbrac01
ora.racdb.racdb_service.svc                    ONLINE     ONLINE on askmdbrac02
[grid@askmdbrac02 ~]$ srvctl status database -d racdb
Instance racdb1 is running on node askmdbrac01
Instance racdb2 is running on node askmdbrac02
[grid@askmdbrac02 ~]$

This concludes the article on troubleshooting RAC services startup issue.

Thanks

This article gives you the troubleshooting steps for this particular issue. These steps may also help you to troubleshoot if you have issues starting RAC services.

STEP 1: Reboot the rac servers and Check services after re-booting the servers. The services are still down.

STEP 2: Verify storage on both rac servers

[grid@askmdbrac01 ~]$ /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[grid@askmdbrac01 ~]$ /etc/init.d/oracleasm listdisks
CRSVOL1
DISK1
FRADISK1
[grid@askmdbrac01 ~]$ multipath -ll
-bash: multipath: command not found
[grid@askmdbrac01 ~]$ exit
logout
[oracle@askmdbrac01 ~]$ sudo su –
[root@askmdbrac01 ~]# multipath -ll
vpath2 (360050768019280fb280000000000055e) dm-10 IBM,2145
[size=200G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=50][active]
\_ 2:0:1:2 sdf        8:80  [active][ready]
\_ 3:0:1:2 sdl        8:176 [active][ready]
\_ round-robin 0 [prio=10][enabled]
\_ 2:0:0:2 sdc        8:32  [active][ready]
\_ 3:0:0:2 sdi        8:128 [active][ready]
vpath1 (360050768019280fb280000000000055d) dm-12 IBM,2145
[size=200G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=50][active]
\_ 3:0:0:1 sdh        8:112 [active][ready]
\_ round-robin 0 [prio=10][enabled]
\_ 2:0:1:1 sde        8:64  [active][ready]
\_ 3:0:1:1 sdk        8:160 [active][ready]
vpath0 (360050768019280fb280000000000055c) dm-11 IBM,2145
[size=10G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=50][active]
\_ 2:0:1:0 sdd        8:48  [active][ready]
\_ 3:0:1:0 sdj        8:144 [active][ready]
\_ round-robin 0 [prio=10][enabled]
\_ 3:0:0:0 sdg        8:96  [active][ready]
[root@askmdbrac01 ~]# exit[grid@askmdbrac02 ~]$ /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[grid@askmdbrac02 ~]$ /etc/init.d/oracleasm listdisks
CRSVOL1
DISK1
FRADISK1
[grid@askmdbrac02 ~]$ multipath -ll
-bash: multipath: command not found
[grid@askmdbrac02 ~]$ exit
logout
[oracle@askmdbrac02 ~]$ sudo su –
[root@askmdbrac02 ~]# multipath -ll
vpath2 (360050768019280fb280000000000055e) dm-10 IBM,2145
[size=200G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=50][active]
\_ 2:0:1:2 sdf        8:80  [active][ready]
\_ 3:0:0:2 sdi        8:128 [active][ready]
\_ round-robin 0 [prio=10][enabled]
\_ 2:0:0:2 sdc        8:32  [active][ready]
\_ 3:0:1:2 sdl        8:176 [active][ready]
vpath1 (360050768019280fb280000000000055d) dm-12 IBM,2145
[size=200G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=50][active]
\_ 3:0:1:1 sdk        8:160 [active][ready]
\_ round-robin 0 [prio=10][enabled]
\_ 2:0:1:1 sde        8:64  [active][ready]
\_ 3:0:0:1 sdh        8:112 [active][ready]
vpath0 (360050768019280fb280000000000055c) dm-11 IBM,2145
[size=10G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=50][active]
\_ 2:0:1:0 sdd        8:48  [active][ready]
\_ 3:0:0:0 sdg        8:96  [active][ready]
\_ round-robin 0 [prio=10][enabled]
\_ 3:0:1:0 sdj        8:144 [active][ready]
[root@askmdbrac02 ~]#

Observation : Storage is fine and i am able to see ASM disks from both the nodes. Storage is configured with multipathing. I also verified multipathing and status shows fine.

STEP 3: Check high availability services and cluster ready services on both nodes with the following commands

[grid@askmdbrac01 ~]$ crsctl check has
CRS-4638: Oracle High Availability Services is online
[grid@askmdbrac01 ~]$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
[grid@askmdbrac01 ~]$
[grid@askmdbrac01 ~]$ crsctl check cluster
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
[grid@askmdbrac01 ~]$[grid@askmdbrac02 ~]$ crsctl check has
CRS-4638: Oracle High Availability Services is online
[grid@askmdbrac02 ~]$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
[grid@askmdbrac02 ~]$
[grid@askmdbrac02 ~]$ crsctl check cluster
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
[grid@askmdbrac02 ~]$

Observation : The high availability services are up , but crs services are down.

STEP 4: Do ping and nslookup for other rac nodes from each node to make sure that each node in cluster is accessible from every other node. Check this for private ips, virtual ips and for scan ips.

Observation : The ping and nslookup are working fine and all the nodes are reachable from every other node.

STEP 5 : Then verify the log files from grid infrastructure and see if you can find any error messages from log files. The cluster log files are located at $ORACLE_HOME/log/askmdbrac01 and $ORACLE_HOME/log/askmdbrac02.

[cssd(11175)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /o001/home/11.2.0.2/grid/log/askmdbrac01/cssd/ocssd.log
2013-04-02 19:38:27.425
[ohasd(6876)]CRS-2765:Resource ‘ora.diskmon’ has failed on server ‘askmdbrac01′.
2013-04-02 19:38:39.988
[cssd(11175)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /o001/home/11.2.0.2/grid/log/askmdbrac01/cssd/ocssd.log
2013-04-02 19:38:55.007
[cssd(11175)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /o001/home/11.2.0.2/grid/log/askmdbrac01/cssd/ocssd.log
2013-04-02 19:39:10.025
…..
…..
[cssd(11175)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /o001/home/11.2.0.2/grid/log/askmdbrac01/cssd/ocssd.log
2013-04-02 19:48:23.836
[/o001/home/11.2.0.2/grid/bin/cssdagent(11163)]CRS-5818:Aborted command ‘start for resource: ora.cssd 1 1′ for resource ‘ora.cssd’. Details at (:CRSAGF00113:) {0:27:4} in /o001/home/11.2.0.2/grid/log/askmdbrac01/agent/ohasd/oracssdagent_root/oracssdagent_root.log.
2013-04-02 19:48:23.837
[cssd(11175)]CRS-1656:The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in /o001/home/11.2.0.2/grid/log/askmdbrac01/cssd/ocssd.log
2013-04-02 19:48:23.838
[cssd(11175)]CRS-1603:CSSD on node askmdbrac01 shutdown by user.
2013-04-02 19:48:25.730
[cssd(11175)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /o001/home/11.2.0.2/grid/log/askmdbrac01/cssd/ocssd.log
2013-04-02 19:48:29.863
[ohasd(6876)]CRS-2765:Resource ‘ora.cssdmonitor’ has failed on server ‘askmdbrac01′.
2013-04-02 19:48:30.126
[ohasd(6876)]CRS-2771:Maximum restart attempts reached for resource ‘ora.cssd’; will not restart.
2013-04-02 19:48:35.833
[ohasd(6876)]CRS-2765:Resource ‘ora.diskmon’ has failed on server ‘askmdbrac01′.

Observation : Found from the cluster alert log file (alertaskmdbrac01.log) that the cluster services can’t find voting disk. So it could be due the in-accessible asm storage or due to currupted voting disks.

Then i tried to see if i can see voting disks through command.

SETP 6 :  Check the voting disks availability ( 11gR2 has voting disks in asm. I don’t have asm instance up now. So i will get error if i try to query the voting disks on asm)

[root@askmdbrac01 ~]# /o001/home/11.2.0.2/grid/bin/crsctl query css votedisk
Unable to communicate with the Cluster Synchronization Services daemon.
[root@askmdbrac01 ~]#[grid@askmdbrac02 ~]$ /o001/home/11.2.0.2/grid/bin/crsctl query css votedisk
Unable to communicate with the Cluster Synchronization Services daemon.
[grid@askmdbrac02 ~]$

Observation : I am not able to query the voting disks. Now i have only one option to look at the asm storage accessibility.

STEP 7 :  Tried re-enabling the asmlibs as follows on both the nodes …

[root@askmdbrac01 ~]# /etc/init.d/oracleasm restart
Dropping Oracle ASMLib disks:                              [  OK  ]
Shutting down the Oracle ASMLib driver:                    [  OK  ]
[root@askmdbrac01 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: no
Checking if /dev/oracleasm is mounted: no
[root@askmdbrac01 ~]# /etc/init.d/oracleasm start
[root@askmdbrac01 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: no
Checking if /dev/oracleasm is mounted: no
[root@askmdbrac01 ~]# /etc/init.d/oracleasm enable
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@askmdbrac01 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[root@askmdbrac01 ~]# /etc/init.d/oracleasm listdisks
CRSVOL1
DISK1
FRADISK1
[root@askmdbrac01 ~]#[root@askmdbrac02 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[root@askmdbrac02 ~]# /etc/init.d/oracleasm restart
Dropping Oracle ASMLib disks:                              [  OK  ]
Shutting down the Oracle ASMLib driver:                    [  OK  ]
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@askmdbrac02 ~]#  /etc/init.d/oracleasm listdisks
CRSVOL1
DISK1
FRADISK1
[root@askmdbrac02 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[root@askmdbrac02 ~]#

Observation : Not able to start services even after restarting the asmlib.

STEP 8 : Tried to re-configure the asmlib as below …..

[root@askmdbrac01 ~]# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.This will configure the on-boot properties of the Oracle ASM library
driver.  The following questions will determine whether the driver is
loaded on boot and what permissions it will have.  The current values
will be shown in brackets (‘[]’).  Hitting <ENTER> without typing an
answer will keep that current value.  Ctrl-C will abort.Default user to own the driver interface []: grid
Default group to own the driver interface []: asmadmin
Start Oracle ASM library driver on boot (y/n) [y]:
Scan for Oracle ASM disks on boot (y/n) [y]:
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@askmdbrac01 ~]#
[root@askmdbrac01 ~]# /etc/init.d/oracleasm stop
Dropping Oracle ASMLib disks:                              [  OK  ]
Shutting down the Oracle ASMLib driver:                    [FAILED]
[root@askmdbrac01 ~]# /etc/init.d/oracleasm start
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@askmdbrac01 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[root@askmdbrac01 ~]#[root@askmdbrac02 ~]# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library
driver.  The following questions will determine whether the driver is
loaded on boot and what permissions it will have.  The current values
will be shown in brackets (‘[]’).  Hitting <ENTER> without typing an
answer will keep that current value.  Ctrl-C will abort.

Default user to own the driver interface []: grid
Default group to own the driver interface []: asmadmin
Start Oracle ASM library driver on boot (y/n) [y]:
Scan for Oracle ASM disks on boot (y/n) [y]:
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@askmdbrac02 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[root@askmdbrac02 ~]# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library
driver.  The following questions will determine whether the driver is
loaded on boot and what permissions it will have.  The current values
will be shown in brackets (‘[]’).  Hitting <ENTER> without typing an
answer will keep that current value.  Ctrl-C will abort.

Default user to own the driver interface [grid]:
Default group to own the driver interface [asmadmin]:
Start Oracle ASM library driver on boot (y/n) [y]:
Scan for Oracle ASM disks on boot (y/n) [y]:
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@askmdbrac02 ~]#
[root@askmdbrac02 ~]# /etc/init.d/oracleasm stop
Dropping Oracle ASMLib disks:                              [  OK  ]
Shutting down the Oracle ASMLib driver:                    [FAILED]
[root@askmdbrac02 ~]# /etc/init.d/oracleasm start
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@askmdbrac02 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[root@askmdbrac02 ~]#

Observation : Identified that the asm library drivers configuration is wrong and i re-configured with correct user and group. See the difference in the lines when it is asking for prompt on second node. I executed this command twice to show you the difference when it is properly configured and when it is not properly configured.

STEP 9 : Stop and Start cluster services on both the nodes.

[root@askmdbrac01 ~]#  /o001/home/11.2.0.2/grid/bin/crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on ‘askmdbrac01′
CRS-2673: Attempting to stop ‘ora.crsd’ on ‘askmdbrac01′
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on ‘askmdbrac01′
CRS-2673: Attempting to stop ‘ora.LISTENER.lsnr’ on ‘askmdbrac01′
CRS-2673: Attempting to stop ‘ora.racdb.racdb_service.svc’ on ‘askmdbrac01′
CRS-2673: Attempting to stop ‘ora.LISTENER_SCAN1.lsnr’ on ‘askmdbrac01′
CRS-2677: Stop of ‘ora.LISTENER.lsnr’ on ‘askmdbrac01′ succeeded
CRS-2677: Stop of ‘ora.LISTENER_SCAN1.lsnr’ on ‘askmdbrac01′ succeeded
CRS-2673: Attempting to stop ‘ora.scan1.vip’ on ‘askmdbrac01′
CRS-2677: Stop of ‘ora.scan1.vip’ on ‘askmdbrac01′ succeeded
CRS-2672: Attempting to start ‘ora.scan1.vip’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.racdb.racdb_service.svc’ on ‘askmdbrac01′ succeeded
CRS-2673: Attempting to stop ‘ora.askmdbrac01.vip’ on ‘askmdbrac01′
CRS-2673: Attempting to stop ‘ora.CRS.dg’ on ‘askmdbrac01′
CRS-2673: Attempting to stop ‘ora.racdb.db’ on ‘askmdbrac01′
CRS-2677: Stop of ‘ora.askmdbrac01.vip’ on ‘askmdbrac01′ succeeded
CRS-2672: Attempting to start ‘ora.askmdbrac01.vip’ on ‘askmdbrac02′
CRS-2676: Start of ‘ora.scan1.vip’ on ‘askmdbrac02′ succeeded
CRS-2672: Attempting to start ‘ora.LISTENER_SCAN1.lsnr’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.racdb.db’ on ‘askmdbrac01′ succeeded
CRS-2673: Attempting to stop ‘ora.FRA.dg’ on ‘askmdbrac01′
CRS-2673: Attempting to stop ‘ora.PRDSHR_DATA.dg’ on ‘askmdbrac01′
CRS-2676: Start of ‘ora.askmdbrac01.vip’ on ‘askmdbrac02′ succeeded
CRS-2677: Stop of ‘ora.PRDSHR_DATA.dg’ on ‘askmdbrac01′ succeeded
CRS-2676: Start of ‘ora.LISTENER_SCAN1.lsnr’ on ‘askmdbrac02′ succeeded
CRS-2677: Stop of ‘ora.FRA.dg’ on ‘askmdbrac01′ succeeded
CRS-2677: Stop of ‘ora.CRS.dg’ on ‘askmdbrac01′ succeeded
CRS-2673: Attempting to stop ‘ora.asm’ on ‘askmdbrac01′
CRS-2677: Stop of ‘ora.asm’ on ‘askmdbrac01′ succeeded
CRS-2673: Attempting to stop ‘ora.ons’ on ‘askmdbrac01′
CRS-2677: Stop of ‘ora.ons’ on ‘askmdbrac01′ succeeded
CRS-2673: Attempting to stop ‘ora.net1.network’ on ‘askmdbrac01′
CRS-2677: Stop of ‘ora.net1.network’ on ‘askmdbrac01′ succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on ‘askmdbrac01′ has completed
CRS-2677: Stop of ‘ora.crsd’ on ‘askmdbrac01′ succeeded
CRS-2673: Attempting to stop ‘ora.mdnsd’ on ‘askmdbrac01′
CRS-2673: Attempting to stop ‘ora.crf’ on ‘askmdbrac01′
CRS-2673: Attempting to stop ‘ora.ctssd’ on ‘askmdbrac01′
CRS-2673: Attempting to stop ‘ora.evmd’ on ‘askmdbrac01′
CRS-2673: Attempting to stop ‘ora.asm’ on ‘askmdbrac01′
CRS-2677: Stop of ‘ora.mdnsd’ on ‘askmdbrac01′ succeeded
CRS-2677: Stop of ‘ora.crf’ on ‘askmdbrac01′ succeeded
CRS-2677: Stop of ‘ora.evmd’ on ‘askmdbrac01′ succeeded
CRS-2677: Stop of ‘ora.asm’ on ‘askmdbrac01′ succeeded
CRS-2673: Attempting to stop ‘ora.cluster_interconnect.haip’ on ‘askmdbrac01′
CRS-2677: Stop of ‘ora.cluster_interconnect.haip’ on ‘askmdbrac01′ succeeded
CRS-2677: Stop of ‘ora.ctssd’ on ‘askmdbrac01′ succeeded
CRS-2673: Attempting to stop ‘ora.cssd’ on ‘askmdbrac01′
CRS-2677: Stop of ‘ora.cssd’ on ‘askmdbrac01′ succeeded
CRS-2673: Attempting to stop ‘ora.gipcd’ on ‘askmdbrac01′
CRS-2673: Attempting to stop ‘ora.diskmon’ on ‘askmdbrac01′
CRS-2677: Stop of ‘ora.gipcd’ on ‘askmdbrac01′ succeeded
CRS-2673: Attempting to stop ‘ora.gpnpd’ on ‘askmdbrac01′
CRS-2677: Stop of ‘ora.gpnpd’ on ‘askmdbrac01′ succeeded
CRS-2677: Stop of ‘ora.diskmon’ on ‘askmdbrac01′ succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on ‘askmdbrac01′ has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@askmdbrac01 ~]# /o001/home/11.2.0.2/grid/bin/crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root@askmdbrac01 ~]#[root@askmdbrac02 ~]# /o001/home/11.2.0.2/grid/bin/crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on ‘askmdbrac02′
CRS-2673: Attempting to stop ‘ora.crsd’ on ‘askmdbrac02′
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on ‘askmdbrac02′
CRS-2673: Attempting to stop ‘ora.LISTENER_SCAN1.lsnr’ on ‘askmdbrac02′
CRS-2673: Attempting to stop ‘ora.askmdbrac02.vip’ on ‘askmdbrac02′
CRS-2673: Attempting to stop ‘ora.oc4j’ on ‘askmdbrac02′
CRS-2673: Attempting to stop ‘ora.LISTENER.lsnr’ on ‘askmdbrac02′
CRS-2673: Attempting to stop ‘ora.LISTENER_SCAN3.lsnr’ on ‘askmdbrac02′
CRS-2673: Attempting to stop ‘ora.cvu’ on ‘askmdbrac02′
CRS-2673: Attempting to stop ‘ora.LISTENER_SCAN2.lsnr’ on ‘askmdbrac02′
CRS-2673: Attempting to stop ‘ora.CRS.dg’ on ‘askmdbrac02′
CRS-2673: Attempting to stop ‘ora.racdb.racdb_service.svc’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.askmdbrac02.vip’ on ‘askmdbrac02′ succeeded
CRS-2677: Stop of ‘ora.racdb.racdb_service.svc’ on ‘askmdbrac02′ succeeded
CRS-2673: Attempting to stop ‘ora.racdb.db’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.LISTENER_SCAN1.lsnr’ on ‘askmdbrac02′ succeeded
CRS-2673: Attempting to stop ‘ora.scan1.vip’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.LISTENER.lsnr’ on ‘askmdbrac02′ succeeded
CRS-2673: Attempting to stop ‘ora.askmdbrac01.vip’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.scan1.vip’ on ‘askmdbrac02′ succeeded
CRS-2677: Stop of ‘ora.LISTENER_SCAN2.lsnr’ on ‘askmdbrac02′ succeeded
CRS-2673: Attempting to stop ‘ora.scan2.vip’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.askmdbrac01.vip’ on ‘askmdbrac02′ succeeded
CRS-2677: Stop of ‘ora.LISTENER_SCAN3.lsnr’ on ‘askmdbrac02′ succeeded
CRS-2673: Attempting to stop ‘ora.scan3.vip’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.scan2.vip’ on ‘askmdbrac02′ succeeded
CRS-2677: Stop of ‘ora.scan3.vip’ on ‘askmdbrac02′ succeeded
CRS-2677: Stop of ‘ora.racdb.db’ on ‘askmdbrac02′ succeeded
CRS-2673: Attempting to stop ‘ora.FRA.dg’ on ‘askmdbrac02′
CRS-2673: Attempting to stop ‘ora.PRDSHR_DATA.dg’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.FRA.dg’ on ‘askmdbrac02′ succeeded
CRS-2677: Stop of ‘ora.PRDSHR_DATA.dg’ on ‘askmdbrac02′ succeeded
CRS-2677: Stop of ‘ora.oc4j’ on ‘askmdbrac02′ succeeded
CRS-2677: Stop of ‘ora.cvu’ on ‘askmdbrac02′ succeeded
CRS-2677: Stop of ‘ora.CRS.dg’ on ‘askmdbrac02′ succeeded
CRS-2673: Attempting to stop ‘ora.asm’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.asm’ on ‘askmdbrac02′ succeeded
CRS-2673: Attempting to stop ‘ora.ons’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.ons’ on ‘askmdbrac02′ succeeded
CRS-2673: Attempting to stop ‘ora.net1.network’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.net1.network’ on ‘askmdbrac02′ succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on ‘askmdbrac02′ has completed
CRS-2677: Stop of ‘ora.crsd’ on ‘askmdbrac02′ succeeded
CRS-2673: Attempting to stop ‘ora.ctssd’ on ‘askmdbrac02′
CRS-2673: Attempting to stop ‘ora.evmd’ on ‘askmdbrac02′
CRS-2673: Attempting to stop ‘ora.asm’ on ‘askmdbrac02′
CRS-2673: Attempting to stop ‘ora.mdnsd’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.evmd’ on ‘askmdbrac02′ succeeded
CRS-2677: Stop of ‘ora.mdnsd’ on ‘askmdbrac02′ succeeded
CRS-2677: Stop of ‘ora.asm’ on ‘askmdbrac02′ succeeded
CRS-2673: Attempting to stop ‘ora.cluster_interconnect.haip’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.cluster_interconnect.haip’ on ‘askmdbrac02′ succeeded
CRS-2677: Stop of ‘ora.ctssd’ on ‘askmdbrac02′ succeeded
CRS-2673: Attempting to stop ‘ora.cssd’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.cssd’ on ‘askmdbrac02′ succeeded
CRS-2673: Attempting to stop ‘ora.diskmon’ on ‘askmdbrac02′
CRS-2673: Attempting to stop ‘ora.crf’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.crf’ on ‘askmdbrac02′ succeeded
CRS-2673: Attempting to stop ‘ora.gipcd’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.diskmon’ on ‘askmdbrac02′ succeeded
CRS-2677: Stop of ‘ora.gipcd’ on ‘askmdbrac02′ succeeded
CRS-2673: Attempting to stop ‘ora.gpnpd’ on ‘askmdbrac02′
CRS-2677: Stop of ‘ora.gpnpd’ on ‘askmdbrac02′ succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on ‘askmdbrac02′ has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@askmdbrac02 ~]# /o001/home/11.2.0.2/grid/bin/crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root@askmdbrac02 ~]#

Observation : Cluster ready services started without any issues. Now i need to verify the services status.

STEP 10 :  Verify the status.

[root@askmdbrac01 ~]# /o001/home/11.2.0.2/grid/bin/crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
[root@askmdbrac01 ~]# /o001/home/11.2.0.2/grid/bin/crs_stat -t
Name           Type           Target    State     Host
————————————————————
ora.CRS.dg     ora….up.type ONLINE    ONLINE    askmdbrac01
ora.FRA.dg     ora….up.type ONLINE    ONLINE    askmdbrac01
ora….ER.lsnr ora….er.type ONLINE    ONLINE    askmdbrac01
ora….N1.lsnr ora….er.type ONLINE    OFFLINE
ora….N2.lsnr ora….er.type ONLINE    ONLINE    askmdbrac01
ora….N3.lsnr ora….er.type ONLINE    ONLINE    askmdbrac01
ora….DATA.dg ora….up.type ONLINE    ONLINE    askmdbrac01
ora.asm        ora.asm.type   ONLINE    ONLINE    askmdbrac01
ora.cvu        ora.cvu.type   ONLINE    ONLINE    askmdbrac01
ora.gsd        ora.gsd.type   OFFLINE   OFFLINE
ora….network ora….rk.type ONLINE    ONLINE    askmdbrac01
ora.oc4j       ora.oc4j.type  ONLINE    ONLINE    askmdbrac01
ora.ons        ora.ons.type   ONLINE    ONLINE    askmdbrac01
ora….SM1.asm application    ONLINE    ONLINE    askmdbrac01
ora….01.lsnr application    ONLINE    ONLINE    askmdbrac01
ora….l01.gsd application    OFFLINE   OFFLINE
ora….l01.ons application    ONLINE    ONLINE    askmdbrac01
ora….l01.vip ora….t1.type ONLINE    ONLINE    askmdbrac01
ora….SM2.asm application    ONLINE    ONLINE    askmdbrac02
ora….02.lsnr application    ONLINE    OFFLINE
ora….l02.gsd application    OFFLINE   OFFLINE
ora….l02.ons application    ONLINE    OFFLINE
ora….l02.vip ora….t1.type ONLINE    OFFLINE
ora….ry.acfs ora….fs.type ONLINE    OFFLINE
ora.scan1.vip  ora….ip.type ONLINE    OFFLINE
ora.scan2.vip  ora….ip.type ONLINE    ONLINE    askmdbrac01
ora.scan3.vip  ora….ip.type ONLINE    ONLINE    askmdbrac01
ora.racdb.db ora….se.type ONLINE    ONLINE    askmdbrac01
ora….ice.svc ora….ce.type ONLINE    ONLINE    askmdbrac01
[root@askmdbrac01 ~]# sudo su – grid
[grid@askmdbrac01 ~]$ . ./profile.grid
[grid@askmdbrac01 ~]$ Check_CRS.txt
HA Resource                                        Target     State
———–                                        ——     —–
ora.CRS.dg                                         ONLINE     ONLINE on askmdbrac01
ora.FRA.dg                                         ONLINE     ONLINE on askmdbrac01
ora.LISTENER.lsnr                                  ONLINE     ONLINE on askmdbrac01
ora.LISTENER_SCAN1.lsnr                            ONLINE     ONLINE on askmdbrac02
ora.LISTENER_SCAN2.lsnr                            ONLINE     ONLINE on askmdbrac01
ora.LISTENER_SCAN3.lsnr                            ONLINE     ONLINE on askmdbrac01
ora.PRDSHR_DATA.dg                                 ONLINE     ONLINE on askmdbrac01
ora.asm                                            ONLINE     ONLINE on askmdbrac01
ora.cvu                                            ONLINE     ONLINE on askmdbrac01
ora.gsd                                            OFFLINE    OFFLINE
ora.net1.network                                   ONLINE     ONLINE on askmdbrac01
ora.oc4j                                           ONLINE     ONLINE on askmdbrac01
ora.ons                                            ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.ASM1.asm                           ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.LISTENER_askmdbrac01.lsnr          ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.gsd                                OFFLINE    OFFLINE
ora.askmdbrac01.ons                                ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.vip                                ONLINE     ONLINE on askmdbrac01
ora.askmdbrac02.ASM2.asm                           ONLINE     ONLINE on askmdbrac02
ora.askmdbrac02.LISTENER_askmdbrac02.lsnr          ONLINE     ONLINE on askmdbrac02
ora.askmdbrac02.gsd                                OFFLINE    OFFLINE
ora.askmdbrac02.ons                                ONLINE     ONLINE on askmdbrac02
ora.askmdbrac02.vip                                ONLINE     ONLINE on askmdbrac02
ora.registry.acfs                                  ONLINE     OFFLINE
ora.scan1.vip                                      ONLINE     ONLINE on askmdbrac02
ora.scan2.vip                                      ONLINE     ONLINE on askmdbrac01
ora.scan3.vip                                      ONLINE     ONLINE on askmdbrac01
ora.racdb.db                                     ONLINE     ONLINE on askmdbrac01
ora.racdb.racdb_service.svc                    ONLINE     ONLINE on askmdbrac02
[grid@askmdbrac01 ~]$ Check_CRS.txt
HA Resource                                        Target     State
———–                                        ——     —–
ora.CRS.dg                                         ONLINE     ONLINE on askmdbrac01
ora.FRA.dg                                         ONLINE     ONLINE on askmdbrac01
ora.LISTENER.lsnr                                  ONLINE     ONLINE on askmdbrac01
ora.LISTENER_SCAN1.lsnr                            ONLINE     ONLINE on askmdbrac02
ora.LISTENER_SCAN2.lsnr                            ONLINE     ONLINE on askmdbrac01
ora.LISTENER_SCAN3.lsnr                            ONLINE     ONLINE on askmdbrac01
ora.PRDSHR_DATA.dg                                 ONLINE     ONLINE on askmdbrac01
ora.asm                                            ONLINE     ONLINE on askmdbrac01
ora.cvu                                            ONLINE     ONLINE on askmdbrac01
ora.gsd                                            OFFLINE    OFFLINE
ora.net1.network                                   ONLINE     ONLINE on askmdbrac01
ora.oc4j                                           ONLINE     ONLINE on askmdbrac01
ora.ons                                            ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.ASM1.asm                           ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.LISTENER_askmdbrac01.lsnr          ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.gsd                                OFFLINE    OFFLINE
ora.askmdbrac01.ons                                ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.vip                                ONLINE     ONLINE on askmdbrac01
ora.askmdbrac02.ASM2.asm                           ONLINE     ONLINE on askmdbrac02
ora.askmdbrac02.LISTENER_askmdbrac02.lsnr          ONLINE     ONLINE on askmdbrac02
ora.askmdbrac02.gsd                                OFFLINE    OFFLINE
ora.askmdbrac02.ons                                ONLINE     ONLINE on askmdbrac02
ora.askmdbrac02.vip                                ONLINE     ONLINE on askmdbrac02
ora.registry.acfs                                  ONLINE     OFFLINE
ora.scan1.vip                                      ONLINE     ONLINE on askmdbrac02
ora.scan2.vip                                      ONLINE     ONLINE on askmdbrac01
ora.scan3.vip                                      ONLINE     ONLINE on askmdbrac01
ora.racdb.db                                     ONLINE     ONLINE on askmdbrac01
ora.racdb.racdb_service.svc                    ONLINE     ONLINE on askmdbrac02
[grid@askmdbrac01 ~]$ srvctl status database -d racdb
Instance racdb1 is running on node askmdbrac01
Instance racdb2 is running on node askmdbrac02
[grid@askmdbrac01 ~]$[root@askmdbrac02 ~]# /o001/home/11.2.0.2/grid/bin/crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
[root@askmdbrac02 ~]# /o001/home/11.2.0.2/grid/bin/crs_stat -t
Name           Type           Target    State     Host
————————————————————
ora.CRS.dg     ora….up.type ONLINE    ONLINE    askmdbrac01
ora.FRA.dg     ora….up.type ONLINE    ONLINE    askmdbrac01
ora….ER.lsnr ora….er.type ONLINE    ONLINE    askmdbrac01
ora….N1.lsnr ora….er.type ONLINE    ONLINE    askmdbrac02
ora….N2.lsnr ora….er.type ONLINE    ONLINE    askmdbrac01
ora….N3.lsnr ora….er.type ONLINE    ONLINE    askmdbrac01
ora….DATA.dg ora….up.type ONLINE    ONLINE    askmdbrac01
ora.asm        ora.asm.type   ONLINE    ONLINE    askmdbrac01
ora.cvu        ora.cvu.type   ONLINE    ONLINE    askmdbrac01
ora.gsd        ora.gsd.type   OFFLINE   OFFLINE
ora….network ora….rk.type ONLINE    ONLINE    askmdbrac01
ora.oc4j       ora.oc4j.type  ONLINE    ONLINE    askmdbrac01
ora.ons        ora.ons.type   ONLINE    ONLINE    askmdbrac01
ora….SM1.asm application    ONLINE    ONLINE    askmdbrac01
ora….01.lsnr application    ONLINE    ONLINE    askmdbrac01
ora….l01.gsd application    OFFLINE   OFFLINE
ora….l01.ons application    ONLINE    ONLINE    askmdbrac01
ora….l01.vip ora….t1.type ONLINE    ONLINE    askmdbrac01
ora….SM2.asm application    ONLINE    ONLINE    askmdbrac02
ora….02.lsnr application    ONLINE    ONLINE    askmdbrac02
ora….l02.gsd application    OFFLINE   OFFLINE
ora….l02.ons application    ONLINE    ONLINE    askmdbrac02
ora….l02.vip ora….t1.type ONLINE    ONLINE    askmdbrac02
ora….ry.acfs ora….fs.type ONLINE    OFFLINE
ora.scan1.vip  ora….ip.type ONLINE    ONLINE    askmdbrac02
ora.scan2.vip  ora….ip.type ONLINE    ONLINE    askmdbrac01
ora.scan3.vip  ora….ip.type ONLINE    ONLINE    askmdbrac01
ora.racdb.db ora….se.type ONLINE    ONLINE    askmdbrac01
ora….ice.svc ora….ce.type ONLINE    ONLINE    askmdbrac01
[root@askmdbrac02 ~]# sudo su – grid
[grid@askmdbrac02 ~]$ . ./profile.grid
[grid@askmdbrac02 ~]$ Check_CRS.txt
HA Resource                                        Target     State
———–                                        ——     —–
ora.CRS.dg                                         ONLINE     ONLINE on askmdbrac01
ora.FRA.dg                                         ONLINE     ONLINE on askmdbrac01
ora.LISTENER.lsnr                                  ONLINE     ONLINE on askmdbrac01
ora.LISTENER_SCAN1.lsnr                            ONLINE     ONLINE on askmdbrac02
ora.LISTENER_SCAN2.lsnr                            ONLINE     ONLINE on askmdbrac01
ora.LISTENER_SCAN3.lsnr                            ONLINE     ONLINE on askmdbrac01
ora.PRDSHR_DATA.dg                                 ONLINE     ONLINE on askmdbrac01
ora.asm                                            ONLINE     ONLINE on askmdbrac01
ora.cvu                                            ONLINE     ONLINE on askmdbrac01
ora.gsd                                            OFFLINE    OFFLINE
ora.net1.network                                   ONLINE     ONLINE on askmdbrac01
ora.oc4j                                           ONLINE     ONLINE on askmdbrac01
ora.ons                                            ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.ASM1.asm                           ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.LISTENER_askmdbrac01.lsnr          ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.gsd                                OFFLINE    OFFLINE
ora.askmdbrac01.ons                                ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.vip                                ONLINE     ONLINE on askmdbrac01
ora.askmdbrac02.ASM2.asm                           ONLINE     ONLINE on askmdbrac02
ora.askmdbrac02.LISTENER_askmdbrac02.lsnr          ONLINE     ONLINE on askmdbrac02
ora.askmdbrac02.gsd                                OFFLINE    OFFLINE
ora.askmdbrac02.ons                                ONLINE     ONLINE on askmdbrac02
ora.askmdbrac02.vip                                ONLINE     ONLINE on askmdbrac02
ora.registry.acfs                                  ONLINE     OFFLINE
ora.scan1.vip                                      ONLINE     ONLINE on askmdbrac02
ora.scan2.vip                                      ONLINE     ONLINE on askmdbrac01
ora.scan3.vip                                      ONLINE     ONLINE on askmdbrac01
ora.racdb.db                                     ONLINE     ONLINE on askmdbrac01
ora.racdb.racdb_service.svc                    ONLINE     ONLINE on askmdbrac02
[grid@askmdbrac02 ~]$ srvctl status database -d racdb
Instance racdb1 is running on node askmdbrac01
Instance racdb2 is running on node askmdbrac02
[grid@askmdbrac02 ~]$

This concludes the article on troubleshooting RAC services startup issue.

Thanks

Multiplex Control Files In 11gR2 RAC/ASM


1) Add location to the control_files parameter.

[oracle@rac1 ~]$ export ORACLE_SID=orcl1
[oracle@rac1 ~]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.1.0 Production on Sun Aug 18 11:22:46 2013

Copyright (c) 1982, 2009, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 – Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
SQL> show parameter control_files;

NAME TYPE VALUE
———————————— ———– ——————————
control_files string +DATA/orcl/controlfile/current
.260.823159575, +FRA/orcl/cont
rolfile/current.256.823159577

SQL> alter system set control_files=’+DATA/orcl/controlfile/current.260.823159575′,
‘+FRA/orcl/controlfile/current.256.823159577’, ‘+DATA’, ‘+FRA’ scope=spfile;

2) Stop the database and start it in nomount mode.

[oracle@rac1 ~]$ srvctl stop database -d orcl
[oracle@rac1 ~]$ srvctl start database -d orcl -o nomount
3) Create the control file using RMAN by pointing to an actual existing controlfile.

[oracle@rac1 ~]$ rman target /

Recovery Manager: Release 11.2.0.1.0 – Production on Sun Aug 18 11:38:56 2013

Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved.

connected to target database: ORCL (not mounted)

RMAN> restore controlfile from ‘+DATA/orcl/controlfile/current.260.823159575’;

Starting restore at 18-AUG-2013 11:39:31
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=1 instance=orcl1 device type=DISK

channel ORA_DISK_1: copied control file copy
output file name=+DATA/orcl/controlfile/current.260.823159575
output file name=+FRA/orcl/controlfile/current.256.823159577
output file name=+DATA/orcl/controlfile/current.270.823779575
Finished restore at 18-AUG-2013 11:39:39
4) Stop the database and start it in normal mode.

[oracle@rac1 ~]$ srvctl stop database -d orcl
[oracle@rac1 ~]$ srvctl start database -d orcl
SQL> select open_mode,name from v$database;

OPEN_MODE NAME
——————– ———
READ WRITE ORCL
+DATA/orcl/controlfile/current.260.823159575
+FRA/orcl/controlfile/current.256.823159577
+DATA/orcl/controlfile/current.269.823780197
+FRA/orcl/controlfile/current.271.823780199

reference: http://practical-tech.blogspot.in/