ITPUX技术网

交流 . 资讯 . 分享
Make progress together!
Linux操作系统下Oracle11g R2 RAC 安装配置教程
Oracle数据库备份恢复高级培训视频(案例、实战、深入、全面)
Oracle数据库DBA高级工程师培训视频(集群容灾,核心深入,备份恢复)

Oracle 11gR2 RAC GRID模拟OCR和Votedisk损坏及恢复过程

内容发布:风哥| 发布时间:2014-4-14 10:22:03
Oracle 11gR2 RAC GRID模拟OCRVotedisk损坏及恢复过程

1破坏前的ocrvotedisk信息检查

检查ocr自动备份
[root@rac1 ~]# ocrconfig -showbackup
rac2 2013/10/13 09:45:30 /u01/grid/product/11.2.0/cdata/rac-cluster/backup00.ocr
rac2 2013/10/13 05:45:29 /u01/grid/product/11.2.0/cdata/rac-cluster/backup01.ocr
rac2 2013/10/13 01:45:28 /u01/grid/product/11.2.0/cdata/rac-cluster/backup02.ocr
rac2 2013/10/12 01:45:26 /u01/grid/product/11.2.0/cdata/rac-cluster/day.ocr
rac2 2013/09/28 02:55:56 /u01/grid/product/11.2.0/cdata/rac-cluster/week.ocr
PROT-25: Manual backups for the Oracle Cluster Registry are not available

这里有一个PROT-25的提示信息,手工备份时无效,所以破坏后直接用自动备份恢复

检查ocr磁盘信息

[root@rac1 ~]# ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 2720
Available space (kbytes) : 259400
ID : 2026562699
Device/File Name : +OCRDATA
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded


查看votedisk磁盘信息
[root@rac1 ~]# crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 5c190e6ab4c04facbfdd4ca0e836a798 (ORCL:OCR1) [OCRDATA]
2. ONLINE abcc18afe6214fbcbfa02fad1c41b21b (ORCL:OCR2) [OCRDATA]
3. ONLINE 31e0a9df91514f73bf50a4e0a344af3d (ORCL:OCR3) [OCRDATA]
Located 3 voting disk(s).


可以看到OCR磁盘和votedisk都通过ASM OCRDATA磁盘组管理
查看ASM磁盘组 OCRDATA信息,我的OCRDATA磁盘组是有OCR1-3组成所以直接通过下列命令查询
[root@rac1 ~]# /etc/init.d/oracleasm querydisk -d OCR1
Disk "OCR1" is a valid ASM disk on device /dev/sda1[8,1]
[root@rac1 ~]# /etc/init.d/oracleasm querydisk -d OCR2
Disk "OCR2" is a valid ASM disk on device /dev/sdh1[8,113]
[root@rac1 ~]# /etc/init.d/oracleasm querydisk -d OCR3
Disk "OCR3" is a valid ASM disk on device /dev/sdb1[8,17]


由于ASM参数文件保存在OCRDATA下,所以对备份一份spfile
SQL> show parameter spfile;
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
spfile string +OCRDATA/rac-cluster/asmparame
terfile/registry.253.825083547

SQL> create pfile='/tmp/asmbak.ora' from spfile; --备份到/tmp/asmbak.ora
File created.


2模拟损坏

使用dd命令破坏这几个磁盘,
[root@rac1 ~]# dd if=/dev/zero of=/dev/sda1 bs=1M count=10
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.005454 seconds, 1.9 GB/s
[root@rac1 ~]# dd if=/dev/zero of=/dev/sdh1 bs=1M count=10
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.00603 seconds, 1.7 GB/s


破坏后执行下面命令,发现各节点服务一切正常
[root@rac1 ~]# crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.DATA.dg ora....up.type ONLINE ONLINE rac1
ora.FRA.dg ora....up.type ONLINE ONLINE rac1
ora....ER.lsnr ora....er.type ONLINE ONLINE rac1
ora....N1.lsnr ora....er.type ONLINE ONLINE rac1
ora.OCRDATA.dg ora....up.type ONLINE ONLINE rac1
ora.asm ora.asm.type ONLINE ONLINE rac1
ora.eons ora.eons.type ONLINE ONLINE rac1
ora.gsd ora.gsd.type OFFLINE OFFLINE
ora....network ora....rk.type ONLINE ONLINE rac1
ora.oc4j ora.oc4j.type OFFLINE OFFLINE
ora.ons ora.ons.type ONLINE ONLINE rac1
ora....SM1.asm application ONLINE ONLINE rac1
ora....C1.lsnr application ONLINE ONLINE rac1
ora.rac1.gsd application OFFLINE OFFLINE
ora.rac1.ons application ONLINE ONLINE rac1
ora.rac1.vip ora....t1.type ONLINE ONLINE rac1
ora....SM2.asm application ONLINE ONLINE rac2
ora....C2.lsnr application ONLINE ONLINE rac2
ora.rac2.gsd application OFFLINE OFFLINE
ora.rac2.ons application ONLINE ONLINE rac2
ora.rac2.vip ora....t1.type ONLINE ONLINE rac2
ora.ractest.db ora....se.type ONLINE ONLINE rac1
ora....ry.acfs ora....fs.type ONLINE ONLINE rac1
ora.scan1.vip ora....ip.type ONLINE ONLINE rac1

[root@rac1 ~]# ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 2720
Available space (kbytes) : 259400
ID : 2026562699
Device/File Name : +OCRDATA
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
[root@rac1 ~]# crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 5c190e6ab4c04facbfdd4ca0e836a798 (ORCL:OCR1) [OCRDATA]
2. ONLINE abcc18afe6214fbcbfa02fad1c41b21b (ORCL:OCR2) [OCRDATA]
3. ONLINE 31e0a9df91514f73bf50a4e0a344af3d (ORCL:OCR3) [OCRDATA]
Located 3 voting disk(s)
.
停止RAC1 crs服务
[root@rac1 ~]# crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac1'
CRS-2673: Attempting to stop 'ora.crsd' on 'rac1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rac1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rac1'
CRS-2673: Attempting to stop 'ora.OCRDATA.dg' on 'rac1'
CRS-2673: Attempting to stop 'ora.ractest.db' on 'rac1'
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'rac1'
CRS-2677: Stop of 'ora.registry.acfs' on 'rac1' succeeded
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.rac1.vip' on 'rac1'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rac1'
CRS-2677: Stop of 'ora.rac1.vip' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.rac1.vip' on 'rac2'
CRS-2677: Stop of 'ora.scan1.vip' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'rac2'
CRS-2676: Start of 'ora.rac1.vip' on 'rac2' succeeded
CRS-2676: Start of 'ora.scan1.vip' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'rac2'
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'rac2' succeeded
CRS-2677: Stop of 'ora.OCRDATA.dg' on 'rac1' succeeded
CRS-2677: Stop of 'ora.ractest.db' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rac1'
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'rac1'
CRS-2677: Stop of 'ora.FRA.dg' on 'rac1' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rac1'
CRS-2677: Stop of 'ora.asm' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rac1'
CRS-2673: Attempting to stop 'ora.eons' on 'rac1'
CRS-2677: Stop of 'ora.ons' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rac1'
CRS-2677: Stop of 'ora.net1.network' on 'rac1' succeeded
CRS-2677: Stop of 'ora.eons' on 'rac1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rac1' has completed
CRS-2677: Stop of 'ora.crsd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'rac1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac1'
CRS-2673: Attempting to stop 'ora.evmd' on 'rac1'
CRS-2673: Attempting to stop 'ora.asm' on 'rac1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rac1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac1'
CRS-2677: Stop of 'ora.cssdmonitor' on 'rac1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rac1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rac1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rac1' succeeded
CRS-2677: Stop of 'ora.drivers.acfs' on 'rac1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac1'
CRS-2677: Stop of 'ora.cssd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac1'
CRS-2673: Attempting to stop 'ora.diskmon' on 'rac1'
CRS-2677: Stop of 'ora.gpnpd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rac1'
CRS-2677: Stop of 'ora.gipcd' on 'rac1' succeeded
CRS-2677: Stop of 'ora.diskmon' on 'rac1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac1' has completed
CRS-4133: Oracle High Availability Services has been stopped.


3 故障定位

在此启动发现已经无法正常启动了,跟踪alter日志,(由于系统日志没有发现什么重要信息,这里就没贴出来)
[root@rac1 ~]# tail -f /u01/grid/product/11.2.0/log/rac1/alertrac1.log
........
[ohasd(28327)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server 'rac1'.
2013-10-13 11:22:46.094
[cssd(28791)]CRS-1713:CSSD daemon is started in clustered mode
2013-10-13 11:22:46.178
[cssd(28791)]CRS-1637:Unable to locate configured voting file with ID 5c190e6a-b4c04fac-bfdd4ca0-e836a798; details at (:CSSNM00020:) in /u01/grid/product/11.2.0/log/rac1/cssd/ocssd.log
2013-10-13 11:22:46.179
[cssd(28791)]CRS-1637:Unable to locate configured voting file with ID abcc18af-e6214fbc-bfa02fad-1c41b21b; details at (:CSSNM00020:) in /u01/grid/product/11.2.0/log/rac1/cssd/ocssd.log
2013-10-13 11:22:46.179
[cssd(28791)]CRS-1705:Found 1 configured voting files but 2 voting files are required, terminating to ensure data integrity; details at (:CSSNM00021:) in /u01/grid/product/11.2.0/log/rac1/cssd/ocssd.log
2013-10-13 11:22:46.179
[cssd(28791)]CRS-1603:CSSD on node rac1 shutdown by user.
2013-10-13 11:22:52.768
[ohasd(28327)]CRS-2765:Resource 'ora.diskmon' has failed on server 'rac1'.


发现服务都无法启动,根据上面的提示检查ocssd.log日志文件
2013-10-13 11:43:47.201: [ SKGFD][1084574016]Lib :ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so: closing handle 0x1dc67d90 for disk :ORCL:OCR3:
2013-10-13 11:43:47.201: [ CSSD][1084574016]clssnmvDiskVerify: file is not a voting file, cannot recognize on-disk signature for a voting
2013-10-13 11:43:47.201: [ CSSD][1084574016]clssnmvDiskVerify: file is not a voting file, cannot recognize on-disk signature for a voting
2013-10-13 11:43:47.201: [ CSSD][1084574016]clssnmvDiskVerify: file is not a voting file, cannot recognize on-disk signature for a voting
2013-10-13 11:43:47.201: [ CSSD][1084574016]clssnmvDiskVerify: file is not a voting file, cannot recognize on-disk signature for a voting
2013-10-13 11:43:47.201: [ CSSD][1084574016]clssnmvDiskVerify: file is not a voting file, cannot recognize on-disk signature for a voting
2013-10-13 11:43:47.201: [ CSSD][1084574016]clssnmvDiskVerify: file is not a voting file, cannot recognize on-disk signature for a voting
2013-10-13 11:43:47.201: [ CSSD][1084574016]clssnmvDiskVerify: file is not a voting file, cannot recognize on-disk signature for a voting


上面错误写的非常清楚无法识别表决磁盘,继续查看下面的log
2013-10-13 11:43:47.201: [ SKGFD][1084574016]Handle 0x1dc61820 from lib :ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so: for disk :ORCL:OCR3:
2013-10-13 11:43:47.201: [ CLSF][1084574016]Opened hdl:0x1dc17dc0 for dev:ORCL:OCR3:
2013-10-13 11:43:47.213: [ CSSD][1084574016]clssnmvDiskVerify: Successful discovery for disk ORCL:OCR3, UID 31e0a9df-91514f73-bf50a4e0-a344af3d, Pending CIN 0:1378101162:0, Committed CIN 0:1378101162:0
2013-10-13 11:43:47.213: [ CLSF][1084574016]Closing handle:0x1dc17dc0
2013-10-13 11:43:47.213: [ SKGFD][1084574016]Lib :ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so: closing handle 0x1dc61820 for disk :ORCL:OCR3:
2013-10-13 11:43:47.213: [ CSSD][1084574016]clssnmvDiskVerify: Successful discovery of 1 disks


发先只有OCR3磁盘可以用
2013-10-13 11:43:47.213: [ CSSD][1084574016]clssnmvVerifyCommittedConfigVFs: Insufficient voting files found, found 1 of 3 configured, needed 2 voting files
2013-10-13 11:43:47.213: [ CSSD][1084574016](:CSSNM00020:)clssnmvVerifyCommittedConfigVFs: voting file 0, id 5c190e6a-b4c04fac-bfdd4ca0-e836a798 not found
2013-10-13 11:43:47.213: [ CSSD][1084574016](:CSSNM00020:)clssnmvVerifyCommittedConfigVFs: voting file 1, id abcc18af-e6214fbc-bfa02fad-1c41b21b not found

上面log写的很清楚,在3快磁盘中只有1块可用,voting file 0voting file 1 notfound,根据上面的log找到OCR3是可以用的,通过oracleasmdisk命令可以找到我们损坏的磁盘除了OCR3剩下
的两块就是OCR1 OCR2

[root@rac1 ~]# /etc/init.d/oracleasm listdisks
DATA1
DATA2
DATA3
FRA1
FRA2
OCR1
OCR2
OCR3


4实施ocrvotedisk恢复

游客,如果您要查看本帖隐藏内容请回复

节点一已经恢复了,节点2重启下crs服务就可以了
至此CORVotedisk损坏恢复完成,在这个实验中我犯了一个小错误,由于我的环境是通过Linux下的虚拟化搭建的,通过iscsi-target做的共享磁盘,我的rac没有修改udev设备,



上一篇:Oracle tnslsnr设置监听口令
下一篇:oracle中的dual表
专业提供Oracle数据库服务、主机、存储、备份、中间件等相关技术支持服务,QQ号:176140749
关注ITPUX技术网微信公众号itpux_com  ,了解本站最新技术资料的分享.

欢迎加QQ群,提供超多高质量Oracle/Unix/Linux技术文档与视频教程的下载。
Oracle/Linux/Unix群1:199479729    Oracle/Linux/Unix群5: 150201289
Oracle/Linux/Unix群2:158277749    Oracle/Linux/Unix群6: 244609803
Oracle/Linux/Unix群3:336282998    Oracle/Linux/Unix群7: 522261684
Oracle/Linux/Unix群4:189070296    Oracle/Linux/Unix群8: 522651731
备注:请勿重复加群,另请注明 from itpux

加群分享视频教程部分如下:
1.Oracle11gR2/Oracle12c RAC实施维护视频;   6.WebSphere四节点集群实施与应用布署;     
2.Oracle11gRAC/12c DataGuard容灾实施维护; 7.存储备份容灾技术与虚拟存储配置;
3.Oracle11g+Linux+Udev+ASM技术实施;        8.Linux平台多路径软件的配置与Oracle应用;
4.Win/Linux上Oracle11g/12c数据库实施;        9.WebLogic11g/12c多节点集群实施;
5.Oracle底层技术/数据块解析/BBED深入解析     10.Oracle数据库备份恢复高级培训/大量案例;  
回复

使用道具 举报

内容发布:wahucc| 发布时间:2014-4-17 13:54:17
学习一下了
回复 支持 反对

使用道具 举报

内容发布:wahucc| 发布时间:2014-4-17 14:03:50
学习一下了
回复 支持 反对

使用道具 举报

内容发布:edwin| 发布时间:2014-6-9 16:04:52
学习了。。。。。
回复

使用道具 举报

内容发布:pettyass| 发布时间:2014-6-14 16:45:57
又来晚了啊,呵呵以前都是在CTO里混,终于又找到新的组织了
回复 支持 反对

使用道具 举报

内容发布:MARCO| 发布时间:2014-6-17 16:06:38
:lol:lol:lol:lol:lol:lol:lol:lol
回复

使用道具 举报

内容发布:贝壳shell| 发布时间:2014-7-22 17:13:49
疯哥威武~~~~~~~~~~~
爱是你我!
回复

使用道具 举报

内容发布:xjcydf909| 发布时间:2014-10-28 13:23:56
学习学习,谢分享
回复 支持 反对

使用道具 举报

内容发布:zty1919| 发布时间:2016-6-3 08:36:28
多谢分享!!!!!!!!!!!!
回复

使用道具 举报

内容发布:hlr| 发布时间:2016-6-8 13:15:39
多次额分享 认真学习学习
回复 支持 反对

使用道具 举报

1框架
您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

快速回复 返回顶部 返回列表