ORA-15042

ORA-15042 ORA-15040 ORA-15032 ASM add disk加盘

2015/01/29 by admin Leave a Comment

存在这种可能性，即ORACLE ASM在add disk扩盘时add disk操作正常完成，disk group的rebalance其实还没有开始，但是由于新加入的disk存在硬件故障，导致add disk后写入到disk header的所有metadata元数据全部丢失，且由于diskgroup是外部冗余即EXTERNAL REdundancy所以该diskgroup由于已经加入了一个DISK，而该DISK上的metadata全部丢失的缘故，所以该diskgroup 将无法正常MOUNT。

且由于新加入的disk上的所有metadata都丢失了，而不仅仅是丢失了disk header的KFBTYP_DISKHEAD，所以还不能是仅仅将KFBTYP_DISKHEAD的信息通过kfed merge其他的disk信息并做修改来还原，其需要通过特殊的手工处理才能绕过该问题。

如下面的例子:

SUCCESS: diskgroup TESTDG03 was created
NOTE: cache deleting context for group TESTDG03 1/0x86485c30
NOTE: cache registered group TESTDG03 number=1 incarn=0xab385c36
NOTE: cache began mount (first) of group TESTDG03 number=1 incarn=0xab385c36
NOTE: Assigning number (1,3) to disk (/oracleasm/asm-disk04)
NOTE: Assigning number (1,2) to disk (/oracleasm/asm-disk03)
NOTE: Assigning number (1,1) to disk (/oracleasm/asm-disk02)
NOTE: Assigning number (1,0) to disk (/oracleasm/asm-disk01)
Thu Jan 29 08:21:07 2015
NOTE: GMON heartbeating for grp 1
GMON querying group 1 at 92 for pid 20, osid 20176
Thu Jan 29 08:21:07 2015
NOTE: cache opening disk 0 of grp 1: TESTDG03_0000 path:/oracleasm/asm-disk01
NOTE: F1X0 found on disk 0 au 2 fcn 0.0
NOTE: cache opening disk 1 of grp 1: TESTDG03_0001 path:/oracleasm/asm-disk02
NOTE: cache opening disk 2 of grp 1: TESTDG03_0002 path:/oracleasm/asm-disk03
NOTE: cache opening disk 3 of grp 1: TESTDG03_0003 path:/oracleasm/asm-disk04
NOTE: cache mounting (first) external redundancy group 1/0xAB385C36 (TESTDG03)
NOTE: cache recovered group 1 to fcn 0.0
NOTE: redo buffer size is 256 blocks (1053184 bytes)
Thu Jan 29 08:21:07 2015
NOTE: LGWR attempting to mount thread 1 for diskgroup 1 (TESTDG03)
NOTE: LGWR found thread 1 closed at ABA 0.10750
NOTE: LGWR mounted thread 1 for diskgroup 1 (TESTDG03)
NOTE: LGWR opening thread 1 at fcn 0.0 ABA 2.0
NOTE: setting 11.2 start ABA for group TESTDG03 thread 1 to 2.0
NOTE: cache mounting group 1/0xAB385C36 (TESTDG03) succeeded
NOTE: cache ending mount (success) of group TESTDG03 number=1 incarn=0xab385c36
GMON querying group 1 at 93 for pid 13, osid 4612
Thu Jan 29 08:21:07 2015
NOTE: Instance updated compatible.asm to 11.2.0.0.0 for grp 1
SUCCESS: diskgroup TESTDG03 was mounted
SUCCESS: CREATE DISKGROUP TESTDG03 EXTERNAL REDUNDANCY  DISK '/oracleasm/asm-disk01' SIZE 129500M ,
'/oracleasm/asm-disk02' SIZE 128800M ,
'/oracleasm/asm-disk03' SIZE 129200M ,
'/oracleasm/asm-disk04' SIZE 128800M  ATTRIBUTE 'compatible.asm'='11.2.0.0.0','au_size'='1M' /* ASMCA */
Thu Jan 29 08:21:07 2015
NOTE: diskgroup resource ora.TESTDG03.dg is online
NOTE: diskgroup resource ora.TESTDG03.dg is updated
Thu Jan 29 08:21:23 2015
SQL> alter diskgroup testdg03  add disk '/oracleasm/asm-disk06' 
ORA-15032: not all alterations performed
ORA-15260: permission denied on ASM disk group
ERROR: alter diskgroup testdg03  add disk '/oracleasm/asm-disk06'
Thu Jan 29 08:21:31 2015
SQL> alter diskgroup testdg03  add disk '/oracleasm/asm-disk06' 
NOTE: Assigning number (1,4) to disk (/oracleasm/asm-disk06)
NOTE: requesting all-instance membership refresh for group=1
NOTE: initializing header on grp 1 disk TESTDG03_0004
NOTE: requesting all-instance disk validation for group=1
Thu Jan 29 08:21:32 2015
NOTE: skipping rediscovery for group 1/0xab385c36 (TESTDG03) on local instance.
NOTE: requesting all-instance disk validation for group=1
NOTE: skipping rediscovery for group 1/0xab385c36 (TESTDG03) on local instance.
NOTE: initiating PST update: grp = 1
Thu Jan 29 08:21:32 2015
GMON updating group 1 at 94 for pid 21, osid 22706
NOTE: PST update grp = 1 completed successfully 
NOTE: membership refresh pending for group 1/0xab385c36 (TESTDG03)
GMON querying group 1 at 95 for pid 13, osid 4612
NOTE: cache opening disk 4 of grp 1: TESTDG03_0004 path:/oracleasm/asm-disk06
GMON querying group 1 at 96 for pid 13, osid 4612
SUCCESS: refreshed membership for 1/0xab385c36 (TESTDG03)
SUCCESS: alter diskgroup testdg03  add disk '/oracleasm/asm-disk06'
NOTE: Attempting voting file refresh on diskgroup TESTDG03
Thu Jan 29 08:22:09 2015
SQL> alter diskgroup testdg03 dismount 
NOTE: cache dismounting (clean) group 1/0xAB385C36 (TESTDG03) 
NOTE: messaging CKPT to quiesce pins Unix process pid: 22730, image: oracle@mlab2.oracle.com (TNS V1-V3)
Thu Jan 29 08:22:10 2015
NOTE: LGWR doing clean dismount of group 1 (TESTDG03)
NOTE: LGWR closing thread 1 of diskgroup 1 (TESTDG03) at ABA 2.15
NOTE: cache dismounted group 1/0xAB385C36 (TESTDG03) 
Thu Jan 29 08:22:10 2015
GMON dismounting group 1 at 97 for pid 21, osid 22730
NOTE: Disk  in mode 0x8 marked for de-assignment
NOTE: Disk  in mode 0x8 marked for de-assignment
NOTE: Disk  in mode 0x8 marked for de-assignment
NOTE: Disk  in mode 0x8 marked for de-assignment
NOTE: Disk  in mode 0x8 marked for de-assignment
SUCCESS: diskgroup TESTDG03 was dismounted
NOTE: cache deleting context for group TESTDG03 1/0xab385c36
Thu Jan 29 08:22:10 2015
NOTE: diskgroup resource ora.TESTDG03.dg is offline
SUCCESS: alter diskgroup testdg03 dismount
NOTE: diskgroup resource ora.TESTDG03.dg is updated
SQL> alter diskgroup testdg03 mount 
NOTE: cache registered group TESTDG03 number=1 incarn=0x83f85c5f
NOTE: cache began mount (first) of group TESTDG03 number=1 incarn=0x83f85c5f
NOTE: Assigning number (1,3) to disk (/oracleasm/asm-disk04)
NOTE: Assigning number (1,2) to disk (/oracleasm/asm-disk03)
NOTE: Assigning number (1,1) to disk (/oracleasm/asm-disk02)
NOTE: Assigning number (1,0) to disk (/oracleasm/asm-disk01)
Thu Jan 29 08:22:22 2015
NOTE: GMON heartbeating for grp 1
GMON querying group 1 at 100 for pid 21, osid 22730
Thu Jan 29 08:22:22 2015
NOTE: Assigning number (1,4) to disk ()
GMON querying group 1 at 101 for pid 21, osid 22730
NOTE: cache dismounting (clean) group 1/0x83F85C5F (TESTDG03) 
NOTE: messaging CKPT to quiesce pins Unix process pid: 22730, image: oracle@mlab2.oracle.com (TNS V1-V3)
NOTE: dbwr not being msg'd to dismount
NOTE: lgwr not being msg'd to dismount
NOTE: cache dismounted group 1/0x83F85C5F (TESTDG03) 
NOTE: cache ending mount (fail) of group TESTDG03 number=1 incarn=0x83f85c5f
NOTE: cache deleting context for group TESTDG03 1/0x83f85c5f
GMON dismounting group 1 at 102 for pid 21, osid 22730
NOTE: Disk  in mode 0x8 marked for de-assignment
NOTE: Disk  in mode 0x8 marked for de-assignment
NOTE: Disk  in mode 0x8 marked for de-assignment
NOTE: Disk  in mode 0x8 marked for de-assignment
NOTE: Disk  in mode 0x8 marked for de-assignment
ERROR: diskgroup TESTDG03 was not mounted
ORA-15032: not all alterations performed
ORA-15040: diskgroup is incomplete
ORA-15042: ASM disk "4" is missing from group number "1" 
ERROR: alter diskgroup testdg03 mount
Thu Jan 29 08:27:37 2015
SQL> alter diskgroup testdg03 mount 
NOTE: cache registered group TESTDG03 number=1 incarn=0x56985c64
NOTE: cache began mount (first) of group TESTDG03 number=1 incarn=0x56985c64
NOTE: Assigning number (1,3) to disk (/oracleasm/asm-disk04)
NOTE: Assigning number (1,2) to disk (/oracleasm/asm-disk03)
NOTE: Assigning number (1,1) to disk (/oracleasm/asm-disk02)
NOTE: Assigning number (1,0) to disk (/oracleasm/asm-disk01)
Thu Jan 29 08:27:43 2015
NOTE: GMON heartbeating for grp 1
GMON querying group 1 at 105 for pid 21, osid 23017
NOTE: cache opening disk 0 of grp 1: TESTDG03_0000 path:/oracleasm/asm-disk01
NOTE: F1X0 found on disk 0 au 2 fcn 0.0
NOTE: cache opening disk 1 of grp 1: TESTDG03_0001 path:/oracleasm/asm-disk02
NOTE: cache opening disk 2 of grp 1: TESTDG03_0002 path:/oracleasm/asm-disk03
NOTE: cache opening disk 3 of grp 1: TESTDG03_0003 path:/oracleasm/asm-disk04
NOTE: cache mounting (first) external redundancy group 1/0x56985C64 (TESTDG03)
NOTE: cache recovered group 1 to fcn 0.609
NOTE: redo buffer size is 256 blocks (1053184 bytes)
Thu Jan 29 08:27:43 2015
NOTE: LGWR attempting to mount thread 1 for diskgroup 1 (TESTDG03)
NOTE: LGWR found thread 1 closed at ABA 2.15
NOTE: LGWR mounted thread 1 for diskgroup 1 (TESTDG03)
NOTE: LGWR opening thread 1 at fcn 0.609 ABA 3.16
NOTE: cache mounting group 1/0x56985C64 (TESTDG03) succeeded
NOTE: cache ending mount (success) of group TESTDG03 number=1 incarn=0x56985c64
GMON querying group 1 at 106 for pid 13, osid 4612
Thu Jan 29 08:27:43 2015
NOTE: Instance updated compatible.asm to 11.2.0.0.0 for grp 1
SUCCESS: diskgroup TESTDG03 was mounted
SUCCESS: alter diskgroup testdg03 mount
Thu Jan 29 08:27:43 2015
NOTE: diskgroup resource ora.TESTDG03.dg is online
NOTE: diskgroup resource ora.TESTDG03.dg is updated
Thu Jan 29 08:33:52 2015
SQL> alter diskgroup testdg03 check all norepair 
NOTE: starting check of diskgroup TESTDG03
Thu Jan 29 08:33:52 2015
GMON checking disk 0 for group 1 at 107 for pid 21, osid 23017
GMON checking disk 1 for group 1 at 108 for pid 21, osid 23017
GMON checking disk 2 for group 1 at 109 for pid 21, osid 23017
GMON checking disk 3 for group 1 at 110 for pid 21, osid 23017
ERROR: no kfdsk for  (4)
ERROR: check of diskgroup TESTDG03 found 1 total errors
ORA-15049: diskgroup "TESTDG03" contains 1 error(s)
ORA-15032: not all alterations performed
ORA-15049: diskgroup "TESTDG03" contains 1 error(s)
ERROR: alter diskgroup testdg03 check all norepair
Thu Jan 29 08:34:07 2015
SQL> alter diskgroup testdg03 check all 
NOTE: starting check of diskgroup TESTDG03
Thu Jan 29 08:34:07 2015
GMON checking disk 0 for group 1 at 111 for pid 21, osid 23017
GMON checking disk 1 for group 1 at 112 for pid 21, osid 23017
GMON checking disk 2 for group 1 at 113 for pid 21, osid 23017
GMON checking disk 3 for group 1 at 114 for pid 21, osid 23017
ERROR: no kfdsk for  (4)
ERROR: check of diskgroup TESTDG03 found 1 total errors
ORA-15049: diskgroup "TESTDG03" contains 1 error(s)
ORA-15032: not all alterations performed
ORA-15049: diskgroup "TESTDG03" contains 1 error(s)
ERROR: alter diskgroup testdg03 check all
SQL> alter diskgroup testdg03 check all repair 
NOTE: starting check of diskgroup TESTDG03
GMON checking disk 0 for group 1 at 115 for pid 21, osid 23017
GMON checking disk 1 for group 1 at 116 for pid 21, osid 23017
GMON checking disk 2 for group 1 at 117 for pid 21, osid 23017
GMON checking disk 3 for group 1 at 118 for pid 21, osid 23017
ERROR: no kfdsk for  (4)
ERROR: check of diskgroup TESTDG03 found 1 total errors
ORA-15049: diskgroup "TESTDG03" contains 1 error(s)
ORA-15032: not all alterations performed
ORA-15049: diskgroup "TESTDG03" contains 1 error(s)
ERROR: alter diskgroup testdg03 check all repair

[oracle@mlab2 oracleasm]$ oerr ora 15042
15042, 00000, “ASM disk \”%s\” is missing from group number \”%s\” ”
// *Cause: The specified disk, which is a necessary part of a diskgroup,
// could not be found on the system.
// *Action: Check the hardware configuration.
//

ORA-15042错误正是因为add disk的磁盘上的metadata全部丢失了，但搞笑的时候新加入的盘上可能因为还没有开始rebalance而没有一点真正有意义的数据，但因为ASM认为该disk已经add进来了，所以必须要该disk可用才能mount diskgroup。而且用户甚至无法强制DROP这个DISK，原因是需要DISKGROUP在MOUNT状态下才可以drop disk，这就变成了鸡生蛋蛋生鸡的死循环，要DROP这个disk必须MOUNT DISKGROUP，但要MOUNT DISKGROUP要先DROP该DISK。

对于此问题一般需要诗檀软件工程师手动修改ASM metadata来绕过问题，或者如果有之前的ASM metadata也可以采用。

如果自己搞不定可以找诗檀软件专业ORACLE数据库修复团队成员帮您恢复!

诗檀软件专业数据库修复团队

服务热线： 13764045638 QQ号:47079569 邮箱：service@parnassusdata.com

Filed Under: Oracle, Oracle ASM, Oracle数据恢复、数据库恢复、灾难恢复 Tagged With: ORA-15032, ORA-15040, ORA-15042

【Oracle ASM数据恢复】ORA-15042: ASM disk is missing after add disk took place错误解析

2013/02/11 by admin Leave a Comment

如果在10.2.0.4 以后版本当向ASM Diskgroup中加入新的磁盘后diskgroup被dismount，尝试mount该diskgroup时报错ORA-15042: ASM disk is missing after add disk took place，那么可以参考本帖。

Tue Feb 12 17:33:59 2013
NOTE: X->S down convert bast on F1B3 bastCount=2
Wed Feb 13 04:06:38 2013 < ALTER DISKGROUP DG1 ADD DISK
  '/dev/mapper/t1_asm03p1',
  '/dev/mapper/t1_asm04p1',
  '/dev/mapper/t1_asm05p1',
  '/dev/mapper/t1_asm06p1'
  rebalance power 4 
Wed Feb 13 04:06:38 2013
NOTE: reconfiguration of group 1/0x53bffa1 (DG1), full=1
Wed Feb 13 04:06:39 2013
NOTE: initializing header on grp 1 disk DG1_0026
NOTE: initializing header on grp 1 disk DG1_0027
NOTE: initializing header on grp 1 disk DG1_0028
NOTE: initializing header on grp 1 disk DG1_0029
NOTE: cache opening disk 26 of grp 1: DG1_0026 path:/dev/mapper/t1_asm03p1
NOTE: cache opening disk 27 of grp 1: DG1_0027 path:/dev/mapper/t1_asm04p1
NOTE: cache opening disk 28 of grp 1: DG1_0028 path:/dev/mapper/t1_asm05p1
NOTE: cache opening disk 29 of grp 1: DG1_0029 path:/dev/mapper/t1_asm06p1
NOTE: PST update: grp = 1
NOTE: requesting all-instance disk validation for group=1
Wed Feb 13 04:06:39 2013
NOTE: disk validation pending for group 1/0x53bffa1 (DG1)
Wed Feb 13 04:06:40 2013
NOTE: requesting all-instance membership refresh for group=1
Wed Feb 13 04:06:40 2013
NOTE: membership refresh pending for group 1/0x53bffa1 (DG1)
SUCCESS: validated disks for 1/0x53bffa1 (DG1)
SUCCESS: refreshed membership for 1/0x53bffa1 (DG1)
Wed Feb 13 04:07:11 2013 < ALTER DISKGROUP DG1 ADD DISK
  '/dev/mapper/t1_asm03p1',
  '/dev/mapper/t1_asm04p1',
  '/dev/mapper/t1_asm05p1',
  '/dev/mapper/t1_asm06p1'
  rebalance power 4 
NOTE: cache closing disk 26 of grp 1: DG1_0026 path:/dev/mapper/t1_asm03p1
NOTE: cache closing disk 26 of grp 1: DG1_0026 path:/dev/mapper/t1_asm03p1
NOTE: cache closing disk 27 of grp 1: DG1_0027 path:/dev/mapper/t1_asm04p1
NOTE: cache closing disk 27 of grp 1: DG1_0027 path:/dev/mapper/t1_asm04p1
NOTE: cache closing disk 28 of grp 1: DG1_0028 path:/dev/mapper/t1_asm05p1
NOTE: cache closing disk 28 of grp 1: DG1_0028 path:/dev/mapper/t1_asm05p1
NOTE: cache closing disk 29 of grp 1: DG1_0029 path:/dev/mapper/t1_asm06p1
NOTE: cache closing disk 29 of grp 1: DG1_0029 path:/dev/mapper/t1_asm06p1
Wed Feb 13 04:09:36 2013
SQL> ALTER DISKGROUP DG1 ADD DISK
  '/dev/mapper/t1_asm03p1',
  '/dev/mapper/t1_asm04p1',
  '/dev/mapper/t1_asm05p1',
  '/dev/mapper/t1_asm06p1'
  rebalance power 4 
Wed Feb 13 04:09:36 2013
NOTE: reconfiguration of group 1/0x53bffa1 (DG1), full=1
Wed Feb 13 04:09:36 2013
NOTE: requesting all-instance membership refresh for group=1
Wed Feb 13 04:09:36 2013
NOTE: membership refresh pending for group 1/0x53bffa1 (DG1)
SUCCESS: validated disks for 1/0x53bffa1 (DG1)
NOTE: PST update: grp = 1, dsk = 26, mode = 0x4
NOTE: PST update: grp = 1, dsk = 27, mode = 0x4
NOTE: PST update: grp = 1, dsk = 28, mode = 0x4
NOTE: PST update: grp = 1, dsk = 29, mode = 0x4
Wed Feb 13 04:09:42 2013
ERROR: too many offline disks in PST (grp 1)
Wed Feb 13 04:09:42 2013
SUCCESS: refreshed membership for 1/0x53bffa1 (DG1)
ERROR: ORA-15040 thrown in RBAL for group number 1
Wed Feb 13 04:09:42 2013
Errors in file /opt/oracle/product/10.2.0/asm/admin/+ASM/bdump/+asm1_rbal_30556.trc:
ORA-15040: diskgroup is incomplete
ORA-15066: offlining disk "" may result in a data loss
ORA-15042: ASM disk "29" is missing
ORA-15042: ASM disk "28" is missing
ORA-15042: ASM disk "27" is missing
ORA-15042: ASM disk "26" is missing
Wed Feb 13 04:09:43 2013
ERROR: PST-initiated MANDATORY DISMOUNT of group DG1
 Received dirty detach msg from node 3 for dom 1
Wed Feb 13 04:09:43 2013
Dirty detach reconfiguration started (old inc 12, new inc 12)

这时我们需要分析ASM的 DISK DIRECTORY和PST以及 DISK HEADER:
我们来看看：

kfddde[4].entry.incarn: 2 ; 0x724: A=0 NUMM=0x1
kfddde[4].entry.hash: 0 ; 0x728: 0x00000000
kfddde[4].entry.refer.number: 0 ; 0x72c: 0x00000000
kfddde[4].entry.refer.incarn: 0 ; 0x730: A=0 NUMM=0x0
kfddde[4].dsknum: 28 ; 0x734: 0x001c
kfddde[4].state: 8 ; 0x736: KFDSTA_ADDING <<<===============================
kfddde[4].ub1spare: 0 ; 0x737: 0x00
kfddde[4].dskname: DG1_0028 ; 0x738: length=8
kfddde[4].fgname: DG1_0028 ; 0x758: length=8
kfddde[4].crestmp.hi: 32983460 ; 0x778: HOUR=0x4 DAYS=0xd MNTH=0x2 YEAR=0x7dd
kfddde[4].crestmp.lo: 443710464 ; 0x77c: USEC=0x0 MSEC=0x9f SECS=0x27 MINS=0x6
kfddde[4].failstmp.hi: 0 ; 0x780: HOUR=0x0 DAYS=0x0 MNTH=0x0 YEAR=0x0
kfddde[4].failstmp.lo: 0 ; 0x784: USEC=0x0 MSEC=0x0 SECS=0x0 MINS=0x0
kfddde[4].timer: 0 ; 0x788: 0x00000000
kfddde[4].size: 307199 ; 0x78c: 0x0004afff

kfddde[3].entry.incarn: 2 ; 0x564: A=0 NUMM=0x1
kfddde[3].entry.hash: 0 ; 0x568: 0x00000000
kfddde[3].entry.refer.number: 0 ; 0x56c: 0x00000000
kfddde[3].entry.refer.incarn: 0 ; 0x570: A=0 NUMM=0x0
kfddde[3].dsknum: 27 ; 0x574: 0x001b
kfddde[3].state: 8 ; 0x576: KFDSTA_ADDING <<<===============================
kfddde[3].ub1spare: 0 ; 0x577: 0x00
kfddde[3].dskname: DG1_0027 ; 0x578: length=8
kfddde[3].fgname: DG1_0027 ; 0x598: length=8
kfddde[3].crestmp.hi: 32983460 ; 0x5b8: HOUR=0x4 DAYS=0xd MNTH=0x2 YEAR=0x7dd
kfddde[3].crestmp.lo: 443710464 ; 0x5bc: USEC=0x0 MSEC=0x9f SECS=0x27 MINS=0x6
kfddde[3].failstmp.hi: 0 ; 0x5c0: HOUR=0x0 DAYS=0x0 MNTH=0x0 YEAR=0x0
kfddde[3].failstmp.lo: 0 ; 0x5c4: USEC=0x0 MSEC=0x0 SECS=0x0 MINS=0x0

kfddde[4].entry.hash: 0 ; 0x728: 0x00000000
kfddde[4].entry.refer.number: 0 ; 0x72c: 0x00000000
kfddde[4].entry.refer.incarn: 0 ; 0x730: A=0 NUMM=0x0
kfddde[4].dsknum: 28 ; 0x734: 0x001c
kfddde[4].state: 8 ; 0x736: KFDSTA_ADDING <<<===============================
kfddde[4].ub1spare: 0 ; 0x737: 0x00
kfddde[4].dskname: DG1_0028 ; 0x738: length=8
kfddde[4].fgname: DG1_0028 ; 0x758: length=8
kfddde[4].crestmp.hi: 32983460 ; 0x778: HOUR=0x4 DAYS=0xd MNTH=0x2 YEAR=0x7dd
kfddde[4].crestmp.lo: 443710464 ; 0x77c: USEC=0x0 MSEC=0x9f SECS=0x27 MINS=0x6
kfddde[4].failstmp.hi: 0 ; 0x780: HOUR=0x0 DAYS=0x0 MNTH=0x0 YEAR=0x0
kfddde[4].failstmp.lo: 0 ; 0x784: USEC=0x0 MSEC=0x0 SECS=0x0 MINS=0x0
kfddde[4].timer: 0 ; 0x788: 0x00000000
kfddde[4].size: 307199 ; 0x78c: 0x0004afff

kfddde[5].entry.incarn: 2 ; 0x8e4: A=0 NUMM=0x1
kfddde[5].entry.hash: 0 ; 0x8e8: 0x00000000
kfddde[5].entry.refer.number: 0 ; 0x8ec: 0x00000000
kfddde[5].entry.refer.incarn: 0 ; 0x8f0: A=0 NUMM=0x0
kfddde[5].dsknum: 29 ; 0x8f4: 0x001d
kfddde[5].state: 8 ; 0x8f6: KFDSTA_ADDING <<<===============================
kfddde[5].ub1spare: 0 ; 0x8f7: 0x00
kfddde[5].dskname: DG1_0029 ; 0x8f8: length=8
kfddde[5].fgname: DG1_0029 ; 0x918: length=8
kfddde[5].crestmp.hi: 32983460 ; 0x938: HOUR=0x4 DAYS=0xd MNTH=0x2 YEAR=0x7dd
kfddde[5].crestmp.lo: 443710464 ; 0x93c: USEC=0x0 MSEC=0x9f SECS=0x27 MINS=0x6
kfddde[5].failstmp.hi: 0 ; 0x940: HOUR=0x0 DAYS=0x0 MNTH=0x0 YEAR=0x0
kfddde[5].failstmp.lo: 0 ; 0x944: USEC=0x0 MSEC=0x0 SECS=0x0 MINS=0x0
kfddde[5].timer: 0 ; 0x948: 0x00000000

File_name :: dg1_3.kfed

/dev/mapper/t1_asm05p1
kfbh.endian: 0 ; 0x000: 0x00
kfbh.hard: 0 ; 0x001: 0x00
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.datfmt: 0 ; 0x003: 0x00
kfbh.block.blk: 0 ; 0x004: T=0 NUMB=0x0
kfbh.block.obj: 0 ; 0x008: TYPE=0x0 NUMB=0x0
kfbh.check: 0 ; 0x00c: 0x00000000
kfbh.fcn.base: 0 ; 0x010: 0x00000000
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000

/dev/mapper/t1_asm06p1
kfbh.endian: 0 ; 0x000: 0x00
kfbh.hard: 0 ; 0x001: 0x00
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.datfmt: 0 ; 0x003: 0x00
kfbh.block.blk: 0 ; 0x004: T=0 NUMB=0x0
kfbh.block.obj: 0 ; 0x008: TYPE=0x0 NUMB=0x0
kfbh.check: 0 ; 0x00c: 0x00000000
kfbh.fcn.base: 0 ; 0x010: 0x00000000
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000

从上面的DISK DIRECTORY中的status可以看到KFDSTA_ADDING ，即新加入的磁盘仍在加入过程中，同时也没有完成rebalance。

查询PST的脚本如下：

vi kfed_pst.sh
-----
#! /bin/sh
rm /tmp/kfed_PST.out
for i in `ls *`
do
echo $i >> /tmp/kfed_PST.out
./kfed read $i aun=1 blkn=2 >> /tmp/kfed_PST.out
done
----

chmod u+x kfed_pst.sh

对于该问题需要手动Patch ASM metadata的方法来解决，否则无法让diskgroup重新mount起来。

如果自己搞不定可以找ASKMACLEAN专业ORACLE数据库修复团队成员帮您恢复!

Filed Under: Oracle, Oracle ASM Tagged With: ORA-15042, PST

【Oracle ASM数据恢复】ORA-15038: disk ‘XXXXXXX’ mismatch on ‘Time Stamp’ with Target Disk Group错误解析

2012/12/11 by admin 1 Comment

如果自己搞不定可以找ASKMACLEAN专业ORACLE数据库修复团队成员帮您恢复!

如果在mount diskgroup时遇到如下错误，则你有必要读一下这篇文章：

SQL> alter diskgroup DATA mount;
alter diskgroup DATA mount
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15040: diskgroup is incomplete
ORA-15042: ASM disk "17" is missing from group number "1"
ORA-15042: ASM disk "16" is missing from group number "1"
ORA-15042: ASM disk "15" is missing from group number "1"
ORA-15042: ASM disk "14" is missing from group number "1"
ORA-15042: ASM disk "13" is missing from group number "1"
ORA-15038: disk 'ORCL:DATA25' mismatch on 'Time Stamp' with target disk group [1861040353] [1861808156]
ORA-15038: disk 'ORCL:DATA24' mismatch on 'Time Stamp' with target disk group [1861040353] [1861808156]
ORA-15038: disk 'ORCL:DATA23' mismatch on 'Time Stamp' with target disk group [1861040353] [1861808156]
ORA-15038: disk 'ORCL:DATA22' mismatch on 'Time Stamp' with target disk group [1861040353] [1861808156]
ORA-15038: disk 'ORCL:DATA21' mismatch on 'Time Stamp' with target disk group [1861040353] [1861808156]

[oracle@mlab2 ~]$ oerr ora 15042
15042, 00000, "ASM disk \"%s\" is missing from group number \"%s\" "
// *Cause:  The specified disk, which is a necessary part of a diskgroup,
//          could not be found on the system.
// *Action: Check the hardware configuration.
//
[oracle@mlab2 ~]$ oerr ora 15038
15038, 00000, "disk '%s' mismatch on '%s' with target disk group [%s] [%s]"
// *Cause:  An attempt was made to mount into a disk group a disk whose
//          recorded allocation unit size, metadata block size, physical 
//          sector size, or creation time stamp was inconsistent with the other 
//          disk group members.
// *Action: Check if the system configuration has changed. Verify disk 
//          discovery string. 
//

该错误一般是由于OS级别的多路径配置存在问题，导致某个节点上出现大量重复的存储设备而引起的。

对于该类问题，首先从alert.log确认最近一次成功mount时那些磁盘对mount而言是必要的，例如：

NOTE: cache opening disk 11 of grp 1: DATA2 label:DATA2 <<<<<<<<<<<<<<<<<< 5 NOTE: cache opening disk 12 of grp 1: DATA20 label:DATA20 >
NOTE: cache opening disk 13 of grp 1: DATA3 label:DATA3 <<<<<<<<<<<<<<<<<<<<<< 6
NOTE: cache opening disk 14 of grp 1: DATA4 label:DATA4
NOTE: cache opening disk 15 of grp 1: DATA5 label:DATA5
NOTE: cache opening disk 16 of grp 1: DATA6 label:DATA6
NOTE: cache opening disk 17 of grp 1: DATA7 label:DATA7
NOTE: cache opening disk 18 of grp 1: DATA8 label:DATA8
NOTE: cache opening disk 19 of grp 1: DATA9 label:DATA9

接下来可以通过kfed来验证DISK HEADER，使用如下脚本：

#! /bin/sh
rm /tmp/kfed_DH.out /tmp/kfed_FS.out /tmp/kfed_BK.out /tmp/kfed_FD.out /tmp/kfed_DD.out /tmp/kfed_PST.out
for i in `ls /dev/asm-disk*`
do
echo $i >> /tmp/kfed_DH.out
kfed read $i >> /tmp/kfed_DH.out
echo $i >> /tmp/kfed_FS.out
kfed read $i blkn=1 >> /tmp/kfed_FS.out
echo $i >> /tmp/kfed_BK.out
kfed read $i aun=1 blkn=254 >> /tmp/kfed_BK.out
echo $i >> /tmp/kfed_FD.out
kfed read $i aun=2 blkn=1 >> /tmp/kfed_FD.out
echo $i >> /tmp/kfed_DD.out
kfed read $i aun=2 blkn=2 >> /tmp/kfed_DD.out
echo $i >> /tmp/kfed_PST.out
kfed read $i aun=1 blkn=2 >> /tmp/kfed_PST.out
done

Filed Under: Oracle, Oracle数据恢复、数据库恢复、灾难恢复 Tagged With: ORA-15038, ORA-15042