ORA-00600 [KCBZPB_1], [59033077], [4], [1], [] example

below is the 600 entry in the alertlog:

alert.log:
Hex dump of Absolute File 14, Block 312821 in trace file /u01/ORAHOME/app/oracle/admin/TIGERS7/bdump/tigers7_dbw0_10999.trc
***
Corrupt block relative dba: 0x0384c5f5 (file 14, block 312821)
Bad header found during preparing block for write
Data in bad block -
type: 6 format: 1 rdba: 0x00000384
last change scn: 0xf90b.c5f55f7c seq: 0x9 flg: 0x72
consistency value in tail: 0x0001f90b
check value in block header: 0x102, block checksum disabled
spare1: 0x6, spare2: 0x2, spare3: 0x0
***
Thu Apr 16 18:32:48 2009
Errors in file /u01/ORAHOME/app/oracle/admin/TIGERS7/bdump/tigers7_dbw0_10999.trc:
ORA-00600: internal error code, arguments: [kcbzpb_1], [59033077], [4], [1], [], [], [], []
Thu Apr 16 18:32:49 2009
Errors in file /u01/ORAHOME/app/oracle/admin/TIGERS7/bdump/tigers7_dbw0_10999.trc:
ORA-00600: internal error code, arguments: [kcbzpb_1], [59033077], [4], [1], [], [], [], []
DBW0: terminating instance due to error 600
Instance terminated by DBW0, pid = 10999
Thu Apr 16 19:04:58 2009

After that, We have executed dbverify against the identified file and it produced no errors:

DBVERIFY: Release 9.2.0.8.0 - Production on Thu Apr 16 19:31:32 2009

Copyright (c) 1982, 2002, Oracle Corporation. All rights reserved.


DBVERIFY - Verification starting : FILE = /u32/ORAINDX/oradata/TIGERS7/indx01.dbf

DBVERIFY - Verification complete

Total Pages Examined : 1280000
Total Pages Processed (Data) : 0
Total Pages Failing (Data) : 0
Total Pages Processed (Index): 1262823
Total Pages Failing (Index): 0
Total Pages Processed (Other): 8751
Total Pages Processed (Seg) : 0
Total Pages Failing (Seg) : 0
Total Pages Empty : 8426
Total Pages Marked Corrupt : 0
Total Pages Influx : 0
Highest block SCN : 10386833124905 (2418.1602203177)

we do open a sr ,and oracle support suggest to do below query:
ACTION PLAN
===========
1) please describe the sequence of events leading up to the problem

2) please upload the alert.log. ZIP if >2MB. Dot not use RAR.

3) please describe your backup strategy:
a) when was your last valid backup?
b) are you using RMAN to perform this backup?
c) do you have all archivelogs from the last backup to now?
d) was this a hot or cold backup?

4) even if you’re not using RMAN, run the following in RMAN:
$ rman target /
RMAN> backup validate check logical database;

5) Once RMAN validate is completed, run the following in SQL*Plus as SYSDBA:
SQL> select * from v$database_block_corruption;

6) Please run the following query in SQL*Plus as SYSDBA
— db must be in either MOUNT or OPEN mode
— Save the queries to a file, eg. rec_query1.sql, then run it in SQL*Plus
—————– start ——————
set echo on
set pagesize 2000 linesize 200 trimspool on
col name form a60
col status form a10
col dbname form a15
col member form a60
col inst_id form 999
col resetlogs_time form a25
col created form a25
col DB_UNIQUE_NAME form a15
col stat form 9999999999
col thr form 99999
col “Uptime” form a80

spool rec_query1.out
show user
alter session set nls_date_format=’DD-MM-RR hh24:mi:ss’;

select inst_id, instance_name, status,
to_char(STARTUP_TIME,’dd-Mon-yyyy hh24:mi’) || ‘ – ‘ ||
trunc(SYSDATE-(STARTUP_TIME) ) || ‘ day(s), ‘ ||
trunc(24*((SYSDATE-STARTUP_TIME) – trunc(SYSDATE-STARTUP_TIME)))||’ hour(s), ‘ ||
mod(trunc(1440*((SYSDATE-STARTUP_TIME) – trunc(SYSDATE-STARTUP_TIME))), 60) ||’ minute(s), ‘ ||
mod(trunc(86400*((SYSDATE-STARTUP_TIME) – trunc(SYSDATE-STARTUP_TIME))), 60) ||’ seconds’
“Uptime”
from gv$instance
order by inst_id
/

select dbid, name dbname, open_mode, database_role,
to_char(created,’dd-Mon-YYYY hh24:mi:ss’) created,
to_char(resetlogs_time,’dd-Mon-YYYY hh24:mi:ss’) resetlogs_time
from v$database;

archive log list;

select count(*) from v$backup where status = ‘ACTIVE’;

select * from v$log;
select * from v$logfile;
select * from v$recover_file order by 1;

select distinct(status)from v$datafile;
select FILE#,TS# , status, NAME from v$datafile
where status not in (‘SYSTEM’,’ONLINE’)
order by 1;

select fhsta, count(*)
from X$KCVFH group by fhsta;

select min(fhrba_Seq), max(fhrba_Seq)
from X$KCVFH;
select hxfil FILE#,fhsta STAT,fhscn SCN,
fhthr thr, fhrba_Seq SEQUENCE,fhtnm TABLESPACE
from x$kcvfh order by 1;

7) dump the block. Run the following as SYSDBA in SQL*Plus:
SQL> alter session set max_dump_file_size=unlimited;
SQL> oradebug setmypid;
SQL> alter system dump datafile ‘full pathname for file 14’ block 312821;
SQL> oradebug tracefile_name;

==> upload the said trace file

8) run dbv against datafile 14:
$ dbv file= logfile=

spool off
—————– end ——————

RESEARCH
===============
ORA-600 [4519] “Block Corruption Detected – Cache type wrong”
We found a corrupted block when trying to read a block using
consistent read. An invalid block type was found.
Possible Block Corruption in Memory.

ORA-600 [kcbzpb_1] A block has been read cleanly from disk and updated successfully by the
clients of the cache layer.
Before the cache layer writes the block back to disk it does a health
check on the cache header.
If requested to do so (default), it generates a checksum for the block.
The health check is failing.
MEMORY CORRUPTION

ORA-600 [kcbzpb_1] was raised because DBA 59033077 => 14,312821 was found corrupted when read in the cache before we writ eit to disk.
Alert.log shows same block as corrupted, BAD HEADER, meaning blocks was overwriten.
Now DBV doesn’t show any corruption in file 14.

ACTION PLAN
====================

Hi,

I reviewed the information and the crash was caused by in memory corruption.
If restarted your database should be fine.

RESEARCH
================
Db crashed with ORA-600 [KCBZPB_1] because of corrupted block in memory:

STACK: kcbbxsv kcbbwlru kcbbdrv ksbabs ksbrdp

Bug.5866883/5845232 (36) INSTANCE GOES DOWN DUE TO ORA-600 [KCBZPB_1] V9208:
Bug.5845843/5845232 (96) DATABASE CRASH BY ORA-00600 [2032] , ORA-00600 [KCBZPB_1]

Bug:5845232: Block corruption / errors from concurrent dequeue operations
Tags: AQ CORR/PHY DUMP OERI R9208 REGRESSION SUPERCEEDED
Details:
This problem is introduced in 9.2.0.8 by the fix for bug 4144683.
Concurrent dequeue operations can lead to block corruption
/ memory corruption with varying symptoms such as ORA-600 [6033],
ORA-600 [6101] and ORA-600 [kcoapl_blkchk] if DB_BLOCK_CHECKING is enabled.
The fix for this bug is Patch 6401576.

Bug:6401576 ORA-600 [KTBAIR1] / ORA-600 [KCBZPB_1] / CORRUPTION MESSAGES –> DB CRASH
Abstract: OERI[ktbair1] / ORA-600 [6101] index corruption possible
Fixed-Releases: WIN:9208P22
Tags: CORR/IND OERI
Details:
Note: This fix replaces the fix in bug 5845232.
Certain index operations can lead to block corruption
/ memory corruption with varying symptoms such as ORA-600 [6033],
ORA-600 [6101] , ORA-600 [ktbair1] , ORA-600 [kcbzpb_1],
ORA-600 [4519] and ORA-600 [kcoapl_blkchk] if DB_BLOCK_CHECKING is enabled.

ISSUE CLARIFICATION
====================
Db crashed with ORA-600 [KCBZPB_1]

ISSUE VERIFICATION
===================
alert.log and trace file

CAUSE DETERMINATION
======================
in memory corruption

CAUSE JUSTIFICATION
====================
Bug:6401576 ORA-600 [KTBAIR1] / ORA-600 [KCBZPB_1] / CORRUPTION MESSAGES –> DB CRASH

POTENTIAL SOLUTION(S)
======================
apply patch for Bug:6401576

POTENTIAL SOLUTION JUSTIFICATION(S)
====================================
to fi x the issue

SOLUTION / ACTION PLAN
=======================

Hi,

These errors looks very similar to Bug:6401576 ORA-600 [KTBAIR1] / ORA-600 [KCBZPB_1] / CORRUPTION MESSAGES –> DB CRASH

Please download and apply one-off patch for Bug:6401576 from

Metalink->Patches->patch#=6401576 ->Platform=Hp_UX

Thanks, Rodica

Comments

  1. Alert logfile shows…

    Sun Oct 3 23:32:45 2010
    Errors in file /orafs/admin/prodfcc/bdump/prodfcc_dbw0_1040460.trc:
    ORA-00600: internal error code, arguments: [kcbzpb_1], [42211701], [4], [1], [], [], [], []
    Sun Oct 3 23:32:45 2010
    Errors in file /orafs/admin/prodfcc/bdump/prodfcc_dbw0_1040460.trc:
    ORA-00600: internal error code, arguments: [kcbzpb_1], [42211701], [4], [1], [], [], [], []
    DBW0: terminating instance due to error 600
    Sun Oct 3 23:32:45 2010
    Errors in file /orafs/admin/prodfcc/bdump/prodfcc_ckpt_872652.trc:
    ORA-00600: internal error code, arguments: [], [], [], [], [], [], [], []
    Sun Oct 3 23:32:47 2010
    Errors in file /orafs/admin/prodfcc/bdump/prodfcc_lgwr_1241148.trc:
    ORA-00600: internal error code, arguments: [], [], [], [], [], [], [], []
    Sun Oct 3 23:32:47 2010
    Errors in file /orafs/admin/prodfcc/bdump/prodfcc_pmon_659484.trc:
    ORA-00600: internal error code, arguments: [], [], [], [], [], [], [], []
    Sun Oct 3 23:33:01 2010
    Instance terminated by DBW0, pid = 1040460
    Sun Oct 3 23:34:48 2010
    Starting ORACLE instance (normal)
    LICENSE_MAX_SESSION = 0
    LICENSE_SESSIONS_WARNING = 0
    SCN scheme 3
    Using log_archive_dest parameter default value
    LICENSE_MAX_USERS = 0
    SYS auditing is disabled
    Starting up ORACLE RDBMS Version: 9.2.0.1.0.
    System parameters with non-default values:
    processes = 750
    timed_statistics = TRUE
    shared_pool_size = 3221225472
    sga_max_size = 8490823192
    large_pool_size = 33554432
    java_pool_size = 33554432
    control_files = /fccmisc/control01.ctl, /fccmisc/control02.ctl, /fccmisc/control03.ctl
    db_block_size = 8192
    db_cache_size = 536870912
    compatible = 9.2.0.0.0

    Looking at the uploaded alertlog file, we found messages like: “Bad header found during buffer read”.
    And the block corruptions were reported in: files: #18, #17, #11 and #6.

    Action Plan:
    =========
    1. Refer to note: 352907.1
    Title: Script To Run DBV On All Datafiles Of the Database

    And run DBV on all the datafiles and provide the output. SO that we can see the extensiveness of DB-corruption and proceed further.
    Datafiles are opened in “read-only” mode by DBV, so it cannot corrupt the contents. (Cannot change the contents of the file being checked.)
    DBV can be run against datafiles which are currently opened by a database instance – there is no need to shutdown the database.

    After analyzing the trace file and the output of dbv it shows that there is no physical corruption.
    This seems to be due to memory corruptions at that point of time. This memory corruptions can happen due to various reasons
    including Oracle bugs, OS issues, memory /hardware issues, rogue processes updating Oracle Memory space etc.

    Please check the hardware and OS logs if there some error reported at the time when ORA-00600 [kcbzpb_1].

    There are many bugs which are fixed in 9.2.0.8 and 10g.

    Few Known Bugs:
    —————————

    NB Bug Fixed Description
    P* 6047085 Linux x64-64: SGA corruption / crash following any ORA-7445 See <>
    6401576 9.2.0.8.P22 OERI[ktbair1] / ORA-600 [6101] index corruption possible
    PI 5496862 10.2.0.3, 11.1.0.6 AIX: Mandatory patch to use Oracle with IBM Technology Level 5 (5300-5) See <>
    P 2747111 9.2.0.4, 10.1.0.2 Tru64: RAC instance may crash with OERI:[SKGXPIODONE17] / DBWR dies with OERI:KCBZPB_1
    2002197 9.0.1.3, 9.2.0.1 OERI:KCBZPB_1 possible in PQ slave

    This is for your information.
    Plan to come to latest oracle patchset versions ie 10g or 11g. As you know 9i database version that has moved to sustaining support .

    Advantages for being on the latest patchset release:

    + Latest patchset contains fix for all the known issues/Bugs raised in previous releases.
    + Patchsets doesn’t include any new functionality, it’s only for bug fixing
    + New bugs can only be logged against latest patchset release of supported versions
    + Backports can only be requested against latest patchset release of supported versions
    + Most stable release since a lot of bugs are already be fixed in the successive patchsets

Comment

*

沪ICP备14014813号-2

沪公网安备 31010802001379号