Recover lost voting and OCR Disks from backup


From 11gr2 onwards, you must know that

1) Voting disk also will be backed up automatically to ocr backup. (Many of us does not know this)

2) Clusterware can be startup in exclusive mode i.e no crs (as you know crsd starts later along with other cluster resources)

3) OCR and voting disks are part of ASM, means you can recreate the diskgroups without CRS by manually starting the ASM and assign that diskgroup to CRS as OCR/Voting Disk

Okay, now see how this works, the situational tasks

Assuming you have not started the cluster stack and only hasd is running.

1) Start CRS in exclusive mode in any of the node.

root@rac2 ~]# crsctl start crs -excl -nocrs
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start ‘ora.gipcd’ on ‘rac2′
CRS-2672: Attempting to start ‘ora.mdnsd’ on ‘rac2′
CRS-2676: Start of ‘ora.gipcd’ on ‘rac2′ succeeded
CRS-2676: Start of ‘ora.mdnsd’ on ‘rac2′ succeeded
CRS-2672: Attempting to start ‘ora.gpnpd’ on ‘rac2′
CRS-2676: Start of ‘ora.gpnpd’ on ‘rac2′ succeeded
CRS-2672: Attempting to start ‘ora.cssdmonitor’ on ‘rac2′
CRS-2676: Start of ‘ora.cssdmonitor’ on ‘rac2′ succeeded
CRS-2672: Attempting to start ‘ora.cssd’ on ‘rac2′
CRS-2679: Attempting to clean ‘ora.diskmon’ on ‘rac2′
CRS-2681: Clean of ‘ora.diskmon’ on ‘rac2′ succeeded
CRS-2672: Attempting to start ‘ora.diskmon’ on ‘rac2′
CRS-2676: Start of ‘ora.diskmon’ on ‘rac2′ succeeded
CRS-2676: Start of ‘ora.cssd’ on ‘rac2′ succeeded
CRS-2672: Attempting to start ‘ora.ctssd’ on ‘rac2′
CRS-2672: Attempting to start ‘ora.drivers.acfs’ on ‘rac2′
CRS-2676: Start of ‘ora.drivers.acfs’ on ‘rac2′ succeeded
CRS-2676: Start of ‘ora.ctssd’ on ‘rac2′ succeeded
CRS-2672: Attempting to start ‘ora.asm’ on ‘rac2′
CRS-2676: Start of ‘ora.asm’ on ‘rac2′ succeeded
CRS-2672: Attempting to start ‘ora.crsd’ on ‘rac2′
CRS-2676: Start of ‘ora.crsd’ on ‘rac2′ succeeded
2) Add the new disks to asm diskgroup

oracle@rac2 ~]$ sqlplus / as sysasm
SQL*Plus: Release 11.2.0.1.0 Production on Wed Jun 01 17:23:56 2013
Copyright (c) 1982, 2009, Oracle.  All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 – 64bit Production
With the Real Application Clusters and Automatic Storage Management options
SQL> create diskgroup DATA external redundancy disk ‘ORCL:DATA’ attribute ‘COMPATIBLE.ASM’ = ‘11.2’;
Diskgroup Created.
3) Identify the latest backup

Identify the latest OCR backup.
root@oracle ~]# /u01/app/11.2.0/grid/bin/ocrconfig -showbackup
oracle      2013/06/01 09:00:30     /u01/app/11.2.0/grid/cdata/oracle/backup00.ocr
oracle      2013/06/06 05:37:29     /u01/app/11.2.0/grid/cdata/oracle/backup01.ocr
oracle      2013/06/06 01:37:27     /u01/app/11.2.0/grid/cdata/oracle/backup02.ocr
oracle      2013/05/31 01:37:21     /u01/app/11.2.0/grid/cdata/oracle/day.ocr
oracle         2013/05/24 13:37:19     /u01/app/11.2.0/grid/cdata/oracle/week.ocr

4) Restore the OCR from automatic backup

[root@rac2 ~]# ocrconfig -restore /u01/app/11.2.0/grid/cdata/oracle/backup00.ocr

5) Start the CRS in exclusive mode

crsctl start res ora.crsd -init
CRS-2672: Attempting to start ‘ora.crsd’ on ‘rac2′
CRS-2676: Start of ‘ora.crsd’ on ‘rac2′ succeeded
6) Replace the voting disk from automatic backup

[root@rac2 ~]# crsctl replace votedisk +DATA
Successful addition of voting disk ahshuehki8489020msjsjsj
Successfully replaced voting disk group with +DATA.
CRS-4266: Voting file(s) successfully replaced
7) Stop the crs and restart

$CRS_HOME/bin/crsctl stop crs -f
$CRS_HOME/bin/crsctl start crs
8) Start the crs in other nodes

$CRS_HOME/bin/crsctl start crs
9) Verify the cluster

# $GRID_HOME/bin/crsctl check cluster -all
rac1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
rac2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
To note, just in case,

If your SPFILE of ASM is in ocr/votingdisk diskgroup, you need to create temporary parameters for asm and then start asm and then finally add this spfile to Datagroup.
<pre>
Sample Parameters
*.asm_power_limit=1
*.diagnostic_dest=’/u01/app/oragrid’
*.instance_type=’asm’
*.large_pool_size=12M
*.remote_login_passwordfile=’EXCLUSIVE’
SQL> create spfile=’+CRS’ from pfile=’/tmp/asm_pfile.ora’;
Hope this helps

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s