Upgrade 11.2.0.1 DB/RDBMS to 11.2.0.2 in Linux

<Upgrade 11.2.0.1 GI/CRS to 11.2.0.2 in Linux>一文中我们介绍了升级11.2.0.1 GI/CRS到11.2.0.2的详细步骤,因为GI/CRS的版本总是要求大于DB/RDBMS,所以这是我们升级RDBMS数据库软件的前提条件。

接下来我们将具体介绍升级11.2.0.1 DB/RDBMS到 11.2.0.2的详细步骤:

一、 下载补丁介质

11.2.0.2的patchset目前没有公开的下载地址,因为updates.oracle.com目前已经不再提供ftp下载模式,所以我们只能通过登录My Oracle Support后进入Patch栏目搜索Patchid并获得加密的下载链接。

11.2.0.2补丁集的全称是11.2.0.2.0 PATCH SET FOR ORACLE DATABASE SERVER (Patchset)(patchid:10098816),可以通过10098816这个id到Patch栏目搜索,并找出对应平台的介质zip包。如在Linux x86-64平台上:

Patch 10098816 11.2.0.2.0 PATCH SET FOR ORACLE DATABASE SERVER_download

 

以上p10098816_112020_Linux-x86-64_1of7.zip和p10098816_112020_Linux-x86-64_2of7.zip ,这2个zip包对应为Database/RDBMS软件的介质,我们不需要下载所有的7个zip包,有这2个升级数据库软件就已经足够了。

完成以上2个软件的下载后,分别解压zip包:

unzip p10098816_112020_Linux-x86-64_1of7.zip -d  $PATCHHOME
unzip p10098816_112020_Linux-x86-64_2of7.zip -d  $PATCHHOME

二、以out of place方式安装11.2.0.2 DB数据库软件

因为11.2.0.2的Patchset以后都是out of place的,所以我们可以不用像在11gr2以前那样必须在原有安装低版本软件的基础上才能升级软件,而可以选择在别的位置完全新安装。

注意该步骤不需要停止数据库实例,可以在前期工作中完成。

以DB/RDBMS数据库软件的拥有者身份(oracle用户)启动方才解压目录下的oui安装界面:

su - oracle

(oracle)$ unset ORACLE_HOME ORACLE_BASE ORACLE_SID
(oracle)$ export DISPLAY=:0
(oracle)$ cd $PATCHHOME
(oracle)$ ./runInstaller

在Oracle Universal Installer界面下的Select Installation Options Screen选择install database only.

upgrade_110202_DB_1

 

在Grid Installation Options下若是RAC 数据库则选择Oracle Real Application cluster database installation,注意如果在该屏幕下出现[FATAL] [INS-35354] The system on which you are attempting to install Oracle RAC is not part of a valid cluster则可能是在之前的安装Gird的过程中没有正确的Update Inventory更新信息库信息,见<11gr2 RAC安装INS-35354问题一例>

若是单节点数据库则选择Single instance database installation

 

upgrade_110202_DB_2

 

在Specify Installation Location Screen上一般OUI会帮你自动匹配一个$ORACLE_BASE变量下不同于原有数据库软件安装目录的新目录,确认这些目录下有足够的磁盘空间,保险起见空间应大于10GB。注意这里是out of place安装,所以千万不要填入原有的安装路径。

 

upgrade_110202_DB_3

 

以上安装完成后OUI会提示要在所有节点上以root身份执行root.sh脚本:

su - root
(root #) /s01/orabase/product/11.2.0/dbhome_2/root.sh

Running Oracle 11g root script...

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /s01/orabase/product/11.2.0/dbhome_2

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Finished product-specific root actions.

三、升级前的准备工作
以上我们完成了11.2.0.2 数据库软件的安装工作,但是还没有升级实例和数据字典。
在正式升级之前,极有必要完成一系列的备份和准备工作,这些准备工作可以详见拙作<Oracle数据库升级前必要的准备工作>

1.清理数据字典中的无用数据,包括审计和回收站,它们可能拉慢数据字典升级的速度:

TRUNCATE TABLE SYS.AUD$;
purge DBA_RECYCLEBIN;

 

2.如果条件允许的话,建议使用RMAN全量备份数据库,前提是数据库没有达到TB级别。

rman target / catalog rman/rman@cata

backup as compressed backupset incremental level 0 database ;

 

3. 收集数据字典的统计信息,若dictionary的统计信息不准备可能导致catupgrd.sql字典升级脚本运行过久:

SQL> set timing on;

SQL> EXECUTE dbms_stats.gather_dictionary_stats;

PL/SQL procedure successfully completed.

Elapsed: 00:00:27.81

 

4.运行dbupgdiag.sql升级信息收集脚本, 该脚本可以提供数据库的一些版本信息和组建信息,以下为该脚本的示例输出内容:

cat db_upg_diag_VPROD_07-Sep-2011_0737.log

                          *** Start of LogFile ***

  Oracle Database Upgrade Diagnostic Utility       09-07-2011 19:37:23

===============
Database Uptime
===============

19:32 07-SEP-11

=================
Database Wordsize
=================

This is a 64-bit database

================
Software Version
================

Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
PL/SQL Release 11.2.0.1.0 - Production
CORE    11.2.0.1.0      Production
TNS for Linux: Version 11.2.0.1.0 - Production
NLSRTL Version 11.2.0.1.0 - Production

=============
Compatibility
=============

Compatibility is set as 11.2.0.0.0

================
Component Status
================

Comp ID Component                          Status    Version        Org_Version    Prv_Version
------- ---------------------------------- --------- -------------- -------------- --------------
CATALOG Oracle Database Catalog Views      VALID     11.2.0.1.0
CATPROC Oracle Database Packages and Types VALID     11.2.0.1.0
OWM     Oracle Workspace Manager           VALID     11.2.0.1.0
RAC     Oracle Real Application Clusters   VALID     11.2.0.1.0

======================================================
List of Invalid Database Objects Owned by SYS / SYSTEM
======================================================

Number of Invalid Objects
------------------------------------------------------------------
There are no Invalid Objects

DOC>################################################################
DOC>
DOC> If there are no Invalid objects below will result in zero rows.
DOC>
DOC>################################################################
DOC>#

no rows selected

================================
List of Invalid Database Objects
================================

Number of Invalid Objects
------------------------------------------------------------------
There are no Invalid Objects

DOC>################################################################
DOC>
DOC> If there are no Invalid objects below will result in zero rows.
DOC>
DOC>################################################################
DOC>#

no rows selected

==============================================================
Identifying whether a database was created as 32-bit or 64-bit
==============================================================

DOC>###########################################################################
DOC>
DOC> Result referencing the string 'B023' ==> Database was created as 32-bit
DOC> Result referencing the string 'B047' ==> Database was created as 64-bit
DOC> When String results in 'B023' and when upgrading database to 10.2.0.3.0
DOC> (64-bit) , For known issue refer below articles
DOC>
DOC> Note 412271.1 ORA-600 [22635] and ORA-600 [KOKEIIX1] Reported While
DOC> Upgrading Or Patching Databases To 10.2.0.3
DOC> Note 579523.1 ORA-600 [22635], ORA-600 [KOKEIIX1], ORA-7445 [KOPESIZ] and
DOC> OCI-21500 [KOXSIHREAD1] Reported While Upgrading To 11.1.0.6
DOC>
DOC>###########################################################################
DOC>#

Metadata Initial DB Creation Info
-------- -----------------------------------
B047     Database was created as 64-bit

===================================================
Number of Duplicate Objects Owned by SYS and SYSTEM
===================================================

Counting duplicate objects ....

  COUNT(1)
----------
         4

=========================================
Duplicate Objects Owned by SYS and SYSTEM
=========================================

Querying duplicate objects ....

OBJECT_NAME                              OBJECT_TYPE
---------------------------------------- ----------------------------------------
AQ$_SCHEDULES                            TABLE
AQ$_SCHEDULES_PRIMARY                    INDEX
DBMS_REPCAT_AUTH                         PACKAGE BODY
DBMS_REPCAT_AUTH                         PACKAGE

DOC>
DOC>################################################################################
DOC>
DOC> If any objects found please follow below article.
DOC> Note 1030426.6 How to Clean Up Duplicate Objects Owned by SYS and SYSTEM schema
DOC> Read the Exceptions carefully before taking actions.
DOC>
DOC>################################################################################
DOC>#

================
JVM Verification
================

JAVAVM - NOT Installed. Below results can be ignored

================================================
Checking Existence of Java-Based Users and Roles
================================================

DOC>
DOC>################################################################################
DOC>
DOC> There should not be any Java Based users for database version 9.0.1 and above.
DOC> If any users found, it is faulty JVM.
DOC>
DOC>################################################################################
DOC>#

User Existence
---------------------------
No Java Based Users

DOC>
DOC>###############################################################
DOC>
DOC> Healthy JVM Should contain Six Roles.
DOC> If there are more or less than six role, JVM is inconsistent.
DOC>
DOC>###############################################################
DOC>#

Role
------------------------------
No JAVA related Roles

Roles

=========================================
List of Invalid Java Objects owned by SYS
=========================================

There are no SYS owned invalid JAVA objects

DOC>
DOC>#################################################################
DOC>
DOC> Check the status of the main JVM interface packages DBMS_JAVA
DOC> and INITJVMAUX and make sure it is VALID.
DOC> If there are no Invalid objects below will result in zero rows.
DOC>
DOC>#################################################################
DOC>#

no rows selected

INFO: Below query should succeed with 'foo' as result.
select dbms_java.longname('foo') "JAVAVM TESTING" from dual
       *
ERROR at line 1:
ORA-00904: "DBMS_JAVA"."LONGNAME": invalid identifier

                            *** End of LogFile ***

以上spool内容显示所要升级的数据库现有CATALOG、CATPROC、OWM和RAC组件,且没有安装JVM,升级JVM组建的数据字典将消耗较长的时间。

另外一个建议运行的脚本是utlu112i.sql,它位于新安装的$ORACLE_HOME/rdbms/admin目录下。

该脚本会给出一些升级前地建议,包括建议保证系统表空间和闪回区域有足够的空间,以及收集数据字典的统计信息,如以下输出:

SQL> @/s01/orabase/product/11.2.0/dbhome_2/rdbms/admin/utlu112i.sql
Oracle Database 11.2 Pre-Upgrade Information Tool 09-07-2011 20:02:30
Script Version: 11.2.0.2.0 Build: 001
.
**********************************************************************
Database:
**********************************************************************
--> name:          VPROD
--> version:       11.2.0.1.0
--> compatible:    11.2.0.0.0
--> blocksize:     8192
--> platform:      Linux x86 64-bit
--> timezone file: V11
.
**********************************************************************
Tablespaces: [make adjustments in the current environment]
**********************************************************************
--> SYSTEM tablespace is adequate for the upgrade.
.... minimum required size: 267 MB
--> SYSAUX tablespace is adequate for the upgrade.
.... minimum required size: 150 MB
--> UNDOTBS1 tablespace is adequate for the upgrade.
.... minimum required size: 253 MB
--> TEMP tablespace is adequate for the upgrade.
.... minimum required size: 61 MB
.
**********************************************************************
Flashback: ON
**********************************************************************
FlashbackInfo:
--> name:          +SYSTEMDG
--> limit:         4977 MB
--> used:          264 MB
--> size:          4977 MB
--> reclaim:       0 MB
--> files:         7
WARNING: --> Flashback Recovery Area Set.  Please ensure adequate disk space              in recover
y areas before performing an upgrade.
.
**********************************************************************
Update Parameters: [Update Oracle Database 11.2 init.ora or spfile]
Note: Pre-upgrade tool was run on a lower version 64-bit database.
**********************************************************************
--> If Target Oracle is 32-Bit, refer here for Update Parameters:
-- No update parameter changes are required.
.

--> If Target Oracle is 64-Bit, refer here for Update Parameters:
-- No update parameter changes are required.
.
**********************************************************************
Renamed Parameters: [Update Oracle Database 11.2 init.ora or spfile]
**********************************************************************
-- No renamed parameters found. No changes are required.
.
**********************************************************************
Obsolete/Deprecated Parameters: [Update Oracle Database 11.2 init.ora or spfile]
**********************************************************************
-- No obsolete parameters found. No changes are required
.

**********************************************************************
Components: [The following database components will be upgraded or installed]
**********************************************************************
--> Oracle Catalog Views         [upgrade]  VALID
--> Oracle Packages and Types    [upgrade]  VALID
--> Real Application Clusters    [upgrade]  VALID
--> Oracle Workspace Manager     [upgrade]  VALID
.
**********************************************************************
Miscellaneous Warnings
**********************************************************************
WARNING: --> The "cluster_database" parameter is currently "TRUE"
.... and must be set to "FALSE" prior to running a manual upgrade.
WARNING: --> Database is using a timezone file older than version 14.
.... After the release migration, it is recommended that DBMS_DST package
.... be used to upgrade the 11.2.0.1.0 database timezone version
.... to the latest version which comes with the new release.
WARNING: --> Your recycle bin is turned on and currently contains no objects.
.... Because it is REQUIRED that the recycle bin be empty prior to upgrading
.... and your recycle bin is turned on, you may need to execute the command:
        PURGE DBA_RECYCLEBIN
.... prior to executing your upgrade to confirm the recycle bin is empty.
.
**********************************************************************
Recommendations
**********************************************************************
Oracle recommends gathering dictionary statistics prior to
upgrading the database.
To gather dictionary statistics execute the following command
while connected as SYSDBA:

    EXECUTE dbms_stats.gather_dictionary_stats;

**********************************************************************
Oracle recommends removing all hidden parameters prior to upgrading.

To view existing hidden parameters execute the following command
while connected AS SYSDBA:

    SELECT name,description from SYS.V$PARAMETER WHERE name
        LIKE '\_%' ESCAPE '\'

Changes will need to be made in the init.ora or spfile.

**********************************************************************
Oracle recommends reviewing any defined events prior to upgrading.

To view existing non-default events execute the following commands
while connected AS SYSDBA:
  Events:
    SELECT (translate(value,chr(13)||chr(10),' ')) FROM sys.v$parameter2
      WHERE  UPPER(name) ='EVENT' AND  isdefault='FALSE'

  Trace Events:
    SELECT (translate(value,chr(13)||chr(10),' ')) from sys.v$parameter2
      WHERE UPPER(name) = '_TRACE_EVENTS' AND isdefault='FALSE'

Changes will need to be made in the init.ora or spfile.

**********************************************************************

 

5.如果数据库很大那么建议打开闪回数据库flashback database,并创建还原点,这样可以极大地缩短回退时间。

可以通过以下查询判断数据库是或否启用了flashback database功能:

 

SQL> select FLASHBACK_ON from v$database;

FLASHBACK_ON
------------------
NO

 

若显示NO则说明之前没有启用数据库闪回功能,若希望启用数据库闪回功能需要数据库短时间停机:

 

关闭所有的数据库实例

SQL> shutdown immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.

启动某一套实例到mount 状态

SQL> startup mount;
ORACLE instance started.

Total System Global Area 1252663296 bytes
Fixed Size                  2212936 bytes
Variable Size             603982776 bytes
Database Buffers          637534208 bytes
Redo Buffers                8933376 bytes
Database mounted.

SQL> alter database flashback on;

Database altered.

在本节点打开数据库,并启动所有节点

SQL> alter database open;

Database altered.

 

以上在数据库级别启用了闪回flashback功能。

接着我们需要停止应用程序,注意在这一步之前的准备工作都可以在线完成,但是本步骤将要求停止一切应用程序的链接,关闭数据库,并启动到restrict限制模式,以便创建restore point,方便可能的升级回退。,strict模式避免了普通用户的链接。

在所有节点上关闭数据库实例,并在唯一节点上启动数据库到restrict模式。

 

startup restrict;

ORACLE instance started.

Total System Global Area 1252663296 bytes
Fixed Size 2212936 bytes
Variable Size 603982776 bytes
Database Buffers 637534208 bytes
Redo Buffers 8933376 bytes
Database mounted.
Database opened.

SQL> conn maclean/maclean
ERROR:
ORA-01035: ORACLE only available to users with RESTRICTED SESSION privilege

Warning: You are no longer connected to ORACLE.

conn / as sysdba

SQL> create restore point maclean_rollback guarantee flashback database;

Restore point created.

SQL> select * from v$restore_point;

       SCN DATABASE_INCARNATION# GUA STORAGE_SIZE
---------- --------------------- --- ------------
TIME
---------------------------------------------------------------------------
RESTORE_POINT_TIME                                                          PRE
--------------------------------------------------------------------------- ---
NAME
--------------------------------------------------------------------------------
    601958                     1 YES     15941632
07-SEP-11 07.52.59.000000000 PM
                                                                            YES
MACLEAN_ROLLBACK

 

四、正式升级数据库实例和数据字典

1. 关闭所有数据库实例

2. 复制相关的pfile或spfile形式的参数到新的ORACLE_HOME下,这里我们假设使用ASM存储共享的spfile,那么只需要在所有节点上将init$SID.ora形式的文件拷贝即可:

 

(oracle $) cat $ORACLE_HOME/dbs/initVPROD1.ora
SPFILE='+SYSTEMDG/VPROD/spfileVPROD.ora'

(oracle $) cp $ORACLE_HOME/dbs/initVPROD1.ora /s01/orabase/product/11.2.0/dbhome_2/dbs

设置ORACLE_HOME和PATH变量指向新的11.2.0.2数据库软件

(oracle $) export ORACLE_HOME=/s01/orabase/product/11.2.0/dbhome_2
(oracle $) export PATH=/s01/orabase/product/11.2.0/dbhome_2/bin:$PATH

设置正确的ORACLE_SID

(oracle $) export ORACLE_SID=VPROD1
(oracle $) unset LD_LIBRARY_PATH

 

3. 启动实例到nomount状态,并修改cluster_database参数到spfile:

 

SQL> startup nomount;
ORACLE instance started.

Total System Global Area 1252663296 bytes
Fixed Size                  2226072 bytes
Variable Size             402655336 bytes
Database Buffers          838860800 bytes
Redo Buffers                8921088 bytes

SQL> alter system set cluster_database=false scope=spfile;

System altered.

 

4. 重启实例到upgrade模式,升级数据字典,运行$ORACLE_HOME/rdbms/admin/catupgrd.sql脚本:

 

SQL> shutdown immediate;
ORA-01507: database not mounted

ORACLE instance shut down.
SQL> startup upgrade;
ORACLE instance started.

Total System Global Area 1252663296 bytes
Fixed Size                  2226072 bytes
Variable Size             402655336 bytes
Database Buffers          838860800 bytes
Redo Buffers                8921088 bytes
Database mounted.
Database opened.

SQL> set echo on  

SQL> SPOOL /tmp/upgrade.log

SQL> set time on; 

20:40:40 SQL> @/s01/orabase/product/11.2.0/dbhome_2/rdbms/admin/catupgrd.sql 

在以上catupgrd.sql脚本运行过程中可以通过DBA_SERVER_REGISTRY视图了解组件字典升级的进度

SQL> select * from DBA_SERVER_REGISTRY;
select * from DBA_SERVER_REGISTRY
              *
ERROR at line 1:
ORA-04063: view "SYS.DBA_SERVER_REGISTRY" has errors
or
ERROR at line 1:
ORA-04063: package body "SYS.DBMS_REGISTRY" has errors

在一开始会提示该视图有错误,这不要紧,稍等一会。

SQL> select comp_name,status,version from dba_server_registry;

COMP_NAME                                          STATUS                           VERSION
-------------------------------------------------- --------------------------       ------------------------------
Oracle Workspace Manager                           UPGRADING                        11.2.0.1.0
Oracle Database Catalog Views                      VALID                            11.2.0.2.0
Oracle Database Packages and Types                 VALID                            11.2.0.2.0
Oracle Real Application Clusters                   VALID                            11.2.0.2.0

20:50:40 SQL>
20:50:40 SQL> Rem *********************************************************************
20:50:40 SQL> Rem END catupgrd.sql
20:50:40 SQL> Rem *********************************************************************
20:50:40 SQL> 

以上catupgrd.sql脚本运行了10分钟左右

重启实例,运行utlrp.sql脚本编译失效对象

sqlplus  / as sysdba
startup;

@?/rdbms/admin/utlrp

TIMESTAMP
--------------------------------------------------------------------------------
COMP_TIMESTAMP UTLRP_BGN  2011-09-07 20:53:38

该脚本会自动根据cpu数目选择并行度

DOC>   This script automatically chooses serial or parallel recompilation
DOC>   based on the number of CPUs available (parameter cpu_count) multiplied
DOC>   by the number of threads per CPU (parameter parallel_threads_per_cpu).
DOC>   On RAC, this number is added across all RAC nodes.

TIMESTAMP
--------------------------------------------------------------------------------
COMP_TIMESTAMP UTLRP_END  2011-09-07 20:55:09

该脚本耗时约2分钟

修改cluster_database参数为true,并重启所有节点实例

SQL> alter system set cluster_database=true scope=spfile;

System altered.

可以看到以上在数据库仅安装了CATALOG、CATPROC、OWM和RAC Cluster View 4种组件的情况下,catupgrd.sql字典升级脚本仅耗时10分钟左右。 而实际的生产库可能安装了更多的组件,如JVM等组件将耗时较多。

以下总结了各Oracle组件升级字典的平均耗时,是一张十分有用的升级时间参考表:

DB Sample Upgrade Time

较少组件情况下

Component HH:MM:SS
Oracle Server 00:16:17
JServer JAVA Virtual Machine 00:05:19
Oracle XDK 00:00:48
Oracle Text 00:00:58
Oracle XML Database 00:04:09
Oracle Database Java Packages 00:00:33
Gathering Statistics 00:02:43
Total Upgrade Time: 00:30:47

 

较多组件情况下

Component HH:MM:SS
Oracle Server 00:16:17
JServer JAVA Virtual Machine 00:05:19
Oracle Workspace Manager 00:01:01
Oracle Enterprise Manager 00:10:13
Oracle XDK 00:00:48
Oracle Text 00:00:58
Oracle XML Database 00:04:09
Oracle Database Java Packages 00:00:33
Oracle Multimedia 00:07:43
Oracle Expression Filter 00:00:18
Oracle Rule Manager 00:00:12
Gathering Statistics 00:04:53
Total Upgrade Time: 00:52:31

 

5.使用srvctl命令更新ocr中DBHOME相关信息:

 

su  - oracle

srvctl upgrade database -d VPROD -o $NEW_ORACLE_HOME

srvctl upgrade database -d VPROD -o /s01/orabase/product/11.2.0/dbhome_2

[oracle@vrh1 ~]$ srvctl config database -d VPROD
Database unique name: VPROD
Database name: VPROD
Oracle home: /s01/orabase/product/11.2.0/dbhome_2
Oracle user: oracle
Spfile: +SYSTEMDG/VPROD/spfileVPROD.ora
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: VPROD
Database instances: VPROD1,VPROD2
Disk Groups: SYSTEMDG
Mount point paths:
Services:
Type: RAC
Database is administrator managed

[oracle@vrh1 ~]$ srvctl stop database -d VPROD
PRCC-1016 : VPROD was already stopped
[oracle@vrh1 ~]$ srvctl start database -d VPROD  

[oracle@vrh1 ~]$ srvctl status  database -d VPROD
Instance VPROD1 is running on node vrh1
Instance VPROD2 is running on node vrh2

 

6.修改oracle用户的profile配置文件指中的变量:

 

cat .bash_profile 

# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi

# User specific environment and startup programs

ORACLE_HOME=/s01/orabase/product/11.2.0/dbhome_2
ORACLE_SID=VPROD1
ORACLE_BASE=/s01/orabase
PATH=$ORACLE_HOME/bin:$ORACLE_HOME/OPatch:$PATH:$HOME/bin

export PATH ORACLE_HOME ORACLE_SID ORACLE_BASE

SQL> select * from global_name;

GLOBAL_NAME
--------------------------------------------------------------------------------
www.askmac.cn

SQL> select * from v$version;

BANNER
--------------------------------------------------------------------------------
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
PL/SQL Release 11.2.0.2.0 - Production
CORE    11.2.0.2.0      Production
TNS for Linux: Version 11.2.0.2.0 - Production
NLSRTL Version 11.2.0.2.0 - Production

 

7. 数据库升级完成后进入一个pending area,建议在至少2个礼拜内,不要升级compatible参数和删除restore point。

在确认没有回退的必要后,修改compatible参数并删除restore point:

 

alter system set compatible=’11.2.0.2.0′ scope=spfile;

drop restore point  MACLEAN_ROLLBACK;

srvctl stop database -d VPROD 

srvctl start database -d VPROD

以上成功地将11.2.0.1的RAC数据库升级到了11.2.0.2。

 

五、回退升级操作(Database Downgrade)

我们可以选择2种回退办法:

  1. 通过restore point还原到11.2.0.1的数据库
  2. 执行catdwgrd.sql降级数据字典

针对第一种方法:

关闭所有节点实例

srvctl stop database -d VPROD

export ORACLE_HOME=$OLD_ORACLE_HOME
export PATH=$OLD_ORACLE_HOME/bin:$PATH
unset LD_LIBRARY_PATH

sqlplus  / as sysdba

SQL> select * from v$restore_point;

       SCN DATABASE_INCARNATION# GUA STORAGE_SIZE
---------- --------------------- --- ------------
TIME
---------------------------------------------------------------------------
RESTORE_POINT_TIME                                                          PRE
--------------------------------------------------------------------------- ---
NAME
--------------------------------------------------------------------------------
    601958                     1 YES    462307328
07-SEP-11 07.52.59.000000000 PM
                                                                            YES
MACLEAN_ROLLBACK

SQL> flashback database to restore point MACLEAN_ROLLBACK;

Flashback complete.

flashback database的速度 视乎flashback log多少而定,一般是很快的,在1分钟之内。

SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-01589: must use RESETLOGS or NORESETLOGS option for database open

SQL> alter database open resetlogs;

Database altered.

以上通过restore point的方法是我所推荐的,这种方法简单、省时省力、高效且问题少少,是一种绿色方案。同时不要忘记使用srvctl upgrade命令还原ocr中的DBHOME信息,以及还原profile文件。

针对第二种方法:
catdwgrd.sql的运行有诸多限制,其所消耗的时间可能要略长于catupgrd.sql。而且该脚本在运行过程中可能遇到各种错误,不推荐使用这种方法。

关于使用catdwgrd.sql脚本降级数据库11.2.0.2到11.2.0.1,可以参考MOS note <How To Downgrade From Database 11.2 To Previous Release (includes 11.2.0.2-11.2.0.1) [ID 883335.1]>

Applying 11G R2 GI PSU 11.2.0.2.3

GI PSU 11.2.0.2.3在最近的一次CPU July中被释出,该Patch Set Update包含了最新的CPU,且GI和Database PSU的都包含在其中,可以直接从<Patch 12419353: GI PSU 11.2.0.2.3 (INCLUDES DATABASE PSU 11.2.0.2.3)>页面下载到,该PSU所修复的Bug包括:

CPU molecules in GI PSU 11.2.0.2.3:

GI PSU 11.2.0.2.3 contains the following new CPU 11.2.0.2 molecules:

12586486 - DB-11.2.0.2-MOLECULE-004-CPUJUL2011

12586487 - DB-11.2.0.2-MOLECULE-005-CPUJUL2011

12586488 - DB-11.2.0.2-MOLECULE-006-CPUJUL2011

12586489 - DB-11.2.0.2-MOLECULE-007-CPUJUL2011

12586490 - DB-11.2.0.2-MOLECULE-008-CPUJUL2011

12586491 - DB-11.2.0.2-MOLECULE-009-CPUJUL2011

12586492 - DB-11.2.0.2-MOLECULE-010-CPUJUL2011

12586493 - DB-11.2.0.2-MOLECULE-011-CPUJUL2011

12586494 - DB-11.2.0.2-MOLECULE-012-CPUJUL2011

12586495 - DB-11.2.0.2-MOLECULE-013-CPUJUL2011

12586496 - DB-11.2.0.2-MOLECULE-014-CPUJUL2011

5.2 Bugs Fixed in GI PSU 11.2.0.2.3
GI PSU 11.2.0.2.3 contains all fixes previously released in GI PSU 11.2.0.2.2
(see Section 5.3 for a list of these bug fixes) and the following new fixes:

Note:

ACFS is not supported on HP and therefore the bug fixes for ACFS do not apply to the HP GI PSU 3. 

Automatic Storage Management

6892311 - PROVIDE REASON FOR MOUNT FORCE FAILURE WITHOUT REQUIRING PST DUMP

9078442 - ORA-19762 FROM ASMCMD CP COPYING FILE WITH DIFFERENT BYTE ORDER FROM FILESYSTEM

9572787 - LONG WAITS FOR ENQ: AM CONTENTION FOLLOWING CELL CRASH CAUSED CLUSTERWIDE OUTAGE

9953542 - TB_SOL_SP: HIT 7445 [KFKLCLOSE()+20] ERROR WHEN DG OFFLINE

10040921 - HUNG DATABASE WORKLOAD AND BACKGROUNDS AFTER INDUCING WRITE ERRORS ON AVD VOLUME

10155605 - 11201-OCE:DISABLE FC IN ONE NODE, ASM DISKGOUP FORCE DISMOUNTED IN OTHER NODES.

10278372 - TB:X:CONSISTENTLY PRINT "WARNING: ELAPSED TIME DID NOT ADVANCE" IN ASM ALERT LOG

10310299 - TB:X:LOST WRITES DUE TO RESYNC MISSING EXTENTS WHEN DISK GO OFFLINE DURING REBAL

10324294 - DBMV2: DBFS INSTANCE WAITS MUCH FOR "ASM METADATA FILE OPERATION"

10356782 - DBMV2+: ASM INSTANCE CRASH WITH ORA-600 : [KFCGET0_04], [25],

10367188 - TB:X:REBOOT 2 CELL NODES,ASM FOREGROUND PROCESS HIT ORA-600[KFNSMASTERWAIT01]

10621169 - FORCE DISMOUNT IN ASM RECOVERY MAY DROP REDO'S AND CAUSE METADATA CORRUPTIONS

11065646 - ASM MAY PICK INCORRECT PST WHEN MULTIPLE COPIES EXTANT

11664719 - 11203_ASM_X64:ARB0 STUCK IN DG REBALANCE

11695285 - ORA-15081 I/O WRITE ERROR OCCURED AFTER CELL NODE FAILURE TEST

11707302 - FOUND CORRUPTED ASM FILES AFTER CELL NODES FAILURE TESTING.

11707699 - DATABASE CANNOT MOUNT DUE TO ORA-00214: CONTROL FILE INCONSISTENCY

11800170 - ASM IN KSV WAIT AFTER APPLICATION OF 11.2.0.2 GRID PSU

11800854 - BUG TO TRACK LRG 5135625

12620422 - FAILED TO ONLINE DISKS BECAUSE OF A POSSIBLE RACING RESYNC

Buffer Cache Management

11674485 - LOST DISK WRITE INCORRECTLY SIGNALLED IN STANDBY DATABASE WHEN APPLYING REDO

Generic

9748749 - ORA-7445 [KOXSS2GPAGE]

10082277 - EXCESSIVE ALLOCATION IN PCUR OF "KKSCSADDCHILDNO" CAUSES ORA-4031 ERRORS

10126094 - ORA-600 [KGLLOCKOWNERSLISTDELETE] OR [KGLLOCKOWNERSLISTAPPEND-OVF]

10142788 - APPS 11I PL/SQL NCOMP:ORA-04030: OUT OF PROCESS MEMORY

10258337 - UNUSABLE INDEX SEGMENT NOT REMOVED FOR "ALTER TABLE MOVE"

10378005 - EXPDP RAISES ORA-00600[KOLRARFC: INVALID LOB TYPE], EXP IS SUCCESSFUL

10636231 - HIGH VERSION COUNT FOR INSERT STATEMENTS WITH REASON INST_DRTLD_MISMATCH

12431716 - UNEXPECTED CHANGE IN MUTEX WAIT BEHAVIOUR IN 11.2.0.2.2 PSU (HIGHER CPU POSSIBLE

High Availability

9869401 - REDO TRANSPORT COMPRESSION (RTC) MESSAGES APPEARING IN ALERT LOG

10157249 - CATALOG UPGRADE TO 11.2.0.2 FAILS WITH ORA-1

10193846 - RMAN DUPLICATE FAILS WITH ORA-19755 WHEN BCT FILE OF PRIMARY IS NOT ACCESSIBLE

10648873 - SR11.2.0.3TXN_REGRESS - TRC - KCRFW_REDO_WRITE

11664046 - STBH: WRONG SEQUENCE NUMBER GENERATED AFTER DB SWITCHOVER FROM STBY TO PRIMARY

Oracle Portable ClusterWare

8906163 - PE: NETWORK AND VIP RESOURCES FAIL TO START IN SOLARIS CONTAINERS

9593552 - GIPCCONNECT() IS NOT ASYNC 11.2.0.2GIBTWO

9897335 - TB-ASM: UNNECCESSARY OCR OPERATION LOG MESSAGES IN ASM ALERT LOG WITH ASM OCR

9902536 - LNX64-11202-MESSAGE: EXCESSIVE GNS LOGGING IN CRS ALERT FILE WHEN SELFCHECK FAIL

9916145 - LX64: INTERNAL ERROR IN CRSD.LOG, MISROUTED REQUEST, ASSERT IN CLSM2M.CPP

9916435 - ROOTCRS.PL FAILS TO CREATE NODEAPPS DURING ADD NODE OPERATION

9939306 - SERVICES NOT COMING UP AFTER SWITCHOVER USING SRVCTL START DATABASE

10012319 - ORA-600 [KFDVF_CSS], [19], [542] ON STARTUP OF ASM DURING ADDNODE

10019726 - MEMORY LEAK 1.2MB/HR IN CRSD.BIN ON NON-N NODE

10056713 - LNX64-11202-CSS: SPLIT BRAIN WHEN START CRS STACK IN PARALLEL WITH PRIV NIC DOWN

10103954 - INTERMITTENT "CANNOT COMMUNICATE WITH CRSD DAEMON" ERRORS

10104377 - GIPC ENSURE INITIAL MESSAGE IS NOT LOST DURING ESTABLISH PHASE

10115514 - SOL-X64-11202: CLIENT REGISTER IN GLOBAL GROUP MASTER#DISKMON#GROUP#MX NOT EXIT

10190153 - HPI-SG-11202 ORA.CTSSD AND ORA.CRSD GOES OFFLINE AFTER KILL GIPC ON CRS MASTER

10231906 - 11202-OCE-SYMANTEC:DOWN ONE OF PRIVAE LINKS ON NODE 3,OCSSD CRASHED ON NODE 3

10233811 - AFTER PATCHING GRID HOME, UNABLE TO START RESOURCES DBFS AND GOLDEN

10253630 - TB:X:HANG DETECTED,"WAITING FOR INSTANCE RECOVERY OF GROUP 2" FOR 45 MINUTES

10272615 - TB:X:SHUTDOWN SERVICE CELLD ON 2 CELL NODES,CSSD ABORT IN CLSSNMRCFGMGRTHREAD

10280665 - TB:X:STOP CELLD ON 2 CELL NODES,CSSD ABORT IN CLSSNMVVERIFYPENDINGCONFIGVFS

10299006 - AFTER 11.2.0.2 UPGRADE, ORAAGENT.BIN CONNECTS TO DATABASE WITH TOO MANY SESSIONS

10322157 - 11202_GIBONE: PERM OF FILES UNDER $CH/CRS/SBS CHANGED AFTER PATCHED

10324594 - STATIC ENDPOINT IN THE LEASE BLOCKS OVERWRITTEN DURING UPGRADE

10331452 - SOL-11202-UD: 10205->11202 NETWORK RES USR_ORA_IF VALUE MISSED AFTER UPGRADE

10357258 - SOL-11202-UD: 10205->11202 [IPMP] HUNDREDS OF DUP IP AFTER INTRA-NODE FAILOVER

10361177 - LNX64-11203-GNS: MANY GNS SELF CHECK FAILURE ALERT MESSAGES

10385838 - TB:X:CSS CORE DUMP AT GIPCHAINTERNALSEND

10397652 - AIX-11202-GIPC:DISABLE SWITCH PORT FOR ONE PRIVATE NIC,HAIP DID NOT FAILOVER

10398810 - DOUBLE FREE IN SETUPWORK DUE TO TIMING

10419987 - PEER LISTENER IS ACCESSING A GROCK THAT IS ALREADY DELETED

10621175 - TB_RAC_X64:X: CLSSSCEXIT: CSSD SIGNAL 11 IN THREAD GMDEATHCHECK

10622973 - LOSS OF LEGACY FEATURES IN 11.2

10631693 - TB:X:CLSSNMHANDLEVFDISCOVERACK: NO PENDINGCONFIGURATION TO COMPLETE. CSS ABORT

10637483 - TB:X:REBOOT ONE CELL NODE, CSS ABORT AT CLSSNMVDDISCTHREAD

10637741 - HARD STOP DEPENDENCY CAN CAUSE WRONG FAIL-OVER ORDER

10638381 - 11202-OCE-SYMANTEC: HAIP FAIL TO START WHEN PRIVATE IP IS PLUMBED ON VIRTUAL NIC

11069614 - RDBMS INSTANCE CRASH DUE TO SLOW REAP OF GIPC MESSAGES ON CMT SYSTEMS

11071429 - PORT 11GR2 CRS TO EL6

11654726 - SCAN LISTENER STARTUP FAILS IF /VAR/OPT/ORACLE/LISTENER.ORA EXISTS.

11663339 - DBMV2:SHARED PROCESS SPINNING CAUSES DELAY IN PRIMARY MEMBER CLEANUP

11682409 - RE-USING OCI MEMORY ACROSS CONNECTIONS CAUSES A MEMORY CORRUPTION

11698552 - SRVCTL REPORT WRONG STATUS FOR DATABASE INSTANCE.

11741224 - INCORRECT ACTIVE VERSION CHECK WHILE ENABLING THE BATCH FUNCTIONALITY

11744313 - LNX64-11203-RACG: UNEXPECTED CRSD RESTART DURING PARALLEL STACK START

11775080 - ORA-29701/29702 OCCURS WHEN WORKLOAD TEST RUNNING FOR A LONG TIME AND IS RESTART

11781515 - EVMD/CRSD FAIL TO START AFTER REBOOT, EVEN AFTER CRSCTL START CLUSTERWARE

11807012 - LNX64-11203-RACG: DB SERVICE RUNS INTO "UNKNOWN" STATE AFTER STACK START

11812615 - LNX64-11203-DIT: INCONSISTENT PERMISSION BEFORE/AFTER ROOTCRS.PL -UNLOCK/-PATCH

11828633 - DATABASE SERVICE DID NOT FAIL OVER AND COULD NOT BE STARTED AFTER NODE FAILURE

11840629 - KERNEL CRASH DUMP AND REBOOT FAIL INSIDE SOLARIS CONTAINER

11866171 - ENABLE CRASHDUMP WHEN REBOOTING THE MACHINE (LINUX)

11877079 - HUNDREDS OF ORAAGENT.BIN@HOSTNAME SESSSIONS IN 11.2.0.2 DATABASE

11899801 - 11202_GIBTWO_HPI:AFTER KILL ASM PMON, POLICY AND ADMIN DB RUNNING ON SAME SERVER

11904778 - LNX64-OEL6-11202: CRS STACK CAN'T BE START AFTER RESTART

11933693 - 11.1.0.7 DATABASE INSTANCE TERMINATED BY 11.2.0.2 CRS AGENT

11936945 - CVU NOT RECOGNIZING THE OEL6 ON LINUX

12332919 - ORAAGENT KEEPS EXITING

12340501 - SRVCTL SHOWS INSTANCE AS DOWN AFTER RELOCATION

12340700 - EVMD CONF FILES CAN HAVE WRONG PERMISSIONS AFTER INSTALL

12349848 - LNX64-11203: VIPS FELL OFFLINE WHEN BRING DOWN 3/4 PUBLIC NICS ONE BY ONE

12378938 - THE LISTENER STOPS WHEN THE ORA.NET1.NETWORK'S STATE IS CHANGED TO UNKNOWN

12380213 - 11203_110415:ERROR EXCEPTION WHILE INSTALLATION 11202 DB WITH DATAFILES ON 11203

12399977 - TYPO IN SUB PERFORM_START_SERVICE RETURNS ZERO (SUCCESS) EVEN WHEN FAILED

12677816 - SCAN LISTENER FAILD TO STARTUP IF /VAR/OPT/ORACLE/LISTENER.ORA EXIST

Oracle Space Management

8223165 - ORA-00600 [KTSXTFFS2] AFTER DATABASE STARTUP

9443361 - WRONG RESULTS (ROWDATA) FOR SELECT IN SERIAL FROM COMPRESSED TABLE

10061015 - LNX64-11202:HIT MANY ORA-600 ARGUMENTS: [KTFBHGET:CLSVIOL_KCBGCUR_9] DURING DBCA

10132870 - INDEX BLOCK CORRUPTION - ORA-600 [KCBZPBUF_2], [6401] ON RECOVER

10324526 - ORA-600 [KDDUMMY_BLKCHK] [6106] WHEN UPDATE SUBPARTITION OF TABLE IN TTS

Oracle Transaction Management

10053725 - TS11.2.0.3V3 - TRC - K2GUPDATEGLOBALPREPARECOUNT

10233732 - ORA-600 [K2GUGPC: PTCNT >= TCNT] OCCURS IN A DATABASE LINK TRANSACTION

Oracle Universal Storage Management

9867867 - SUSE10-LNX64-11202:NODE REBOOT HANG WHILE ORACLE_HOME LOCATED ON ACFS

9936659 - LNX64-11202-CRS: ORACLE HOME PUT ON ACFS, DB INST FAILS TO RESTART AFTER CRASH

9942881 - TIGHTEN UP KILL SEMANTICS FOR 'CLEAN' ACTION.

10113899 - AIX KSPRINTTOBUFFER TIMESTAMPS NEEDS TIME SINCE BOOT AND WALL_CLOCK TIMES

10266447 - ROOTUPGRADE.SH FAILS: 'FATAL: MODULE ORACLEOKS NOT FOUND' , ACFS-9121, ACFS-9310

11789566 - ACFS RECOVERY PHASE 2

11804097 - GBM LOCK TAKEN WHEN DETERMINING WHETHER THE FILE SYSTEM IS MOUNTED AND ONLINE

11846686 - ACFSROOT FAILS ON ORACLELINUX-RELEASE-5-6.0.1 RUNNUNG A 2.6.18 KERNEL

12318560 - ALLOW IOS TO RESTART WHEN WRITE ERROR MESG RETURNS SUCCESS

12326246 - ASM TO RETURN DIFF VALUES WHEN OFFLINE MESG UNSUCCESSFUL

12378675 - AIX-11203-HA-ACFS: HIT INVALID ASM BLOCK HEADER WHEN CONFIGURE DG USING AIX LVS

12398567 - ACFS FILE SYSTEM NOT ACCESSIBLE

12545060 - CHOWN OR RM CMD TO LOST+FOUND DIR IN ACFS FAILS ON LINUX

Oracle Utilities

9735282 - GETTING ORA-31693, ORA-2354, ORA-1426 WHEN IMPORTING PARTITIONED TABLE

Oracle Virtual Operating System Services

10317487 - RMAN CONTROLFILE BACKUP FAILS WITH ODM ERROR ORA-17500 AND ORA-245

11651810 - STBH: HIGH HARD PARSING DUE TO FILEOPENBLOCK EATING UP SHARED POOL MEMORY

XML Database

10368698 - PERF ISSUE WITH UPDATE RESOURCE_VIEW DURING AND AFTER UPGRADING TO 11.2.0.2

5.3 Bugs Fixed in GI PSU 11.2.0.2.2
This section describes bugs fixed in the GI PSU 11.2.0.2.2 release.

ACFS

10015603 - KERNEL PANIC IN OKS DRIVER WHEN SHUTDOWING CRS STACK

10178670 - ACFS VOLUMES ARE NOT MOUNTING ONCE RESTARTED THE SERVER

10019796 - FAIL TO GET ENCRYPTION STATUS OF FILES UNTIL DOING ENCR OP FIRST

10029794 - THE DIR CAN'T READ EVEN IF THE DIR IS NOT IN ANY REALM

10056808 - MOUNT ACFS FS FAILED WHEN FS IS FULL

10061534 - DB INSTANCE TERMINATED DUE TO ORA-445 WHEN START INSTANCE ON ALL NODES

10069698 - THE EXISTING FILE COULD CORRUPT IF INPUT INCORRECT PKCS PASSOWRD

10070563 - MULTIPLE WRITES TO THE SAME BLOCK WITH REPLICATION ON CAN GO OUT OF ORDER

10087118 - UNMOUNT PANICS IF ANOTHER USER IS SITTING IN A SNAPSHOT ROOT DIRECTORY

10216878 - REPLI-RELATED RESOURCE FAILED TO FAILOVER WHEN DG DISMOUNTED

10228079 - MOUTING DG ORA-15196 [KFC.C:25316] [ENDIAN_KFBH] AFTER NODE REBOOT

10241696 - FAILED TO MOUNT ACFS FS TO DIRECTORY CREATED ON ANOTHER ACFS FS

10252497 - ADVM/ACFS FAILS TO INSTALL ON SLES10

9861790 - LX64: ADVM DRIVERS HANGING OS DURING ACFS START ATTEMPTS

9906432 - KERNEL PANIC WHILE DISMOUNT ACFS DG FORCE

9975343 - FAIL TO PREPARE SECURITY IF SET ENCRYPTION FIRST ON THE OTHER NODE

10283549 - FIX AIX PANIC AND REMOVE -DAIX_PERF

10283596 - ACFS:KERNEL PANIC DURING USM LABEL PATCHING - ON AIX

10326548 - WRITE-PROTETED ACFS FILES SHOULD NOT BE DELETED BY NON-ROOT USER

ADVM

10045316 - RAC DB INSTALL ON SHARED ACFS HANGS AT LINKING PHASE

10283167 - ASM INSTANCE CANNOT STARTUP DUE TO EXISTENCE OF VMBX PROCESS

10268642 - NODE PANIC FOR BAD TRAP IN "ORACLEADVM" FOR NULL POINTER

10150020 - LINUX HANGS IN ADVM MIRROR RECOVERY, AFTER ASM EVICTIONS

Automatic Storage Management

9788588 - STALENESS REGISTRY MAY GET CLEARED PREMATURELY

10022980 - DISK NOT EXPELLED WHEN COMPACT DISABLED

10040531 - ORA-600 [KFRHTADD01] TRYING TO MOUNT RECO DISKGROUP

10209232 - STBH: DB STUCK WITH A STALE EXTENT MAP AND RESULTS IN DATA CORRUPTIONS

10073683 - ORA-600 [KFCBINITSLOT40] ON ASM ON DBMV2 WITH BP5

9715581 - DBMV2: EXADATA AUTO MANAGEMENT FAILED TO BRING UP DISKS ONLINE

10019218 - ASM DROPPED DISKS BEFORE DISK_REPAIR_TIME EXPIRED

10084145 - DBMV2: ORA-600 [1427] MOUNTING DISKGROUP AFTER ALL CELLS RESTARTED

11067567 - KEPT GENERATING "ELAPSED TIME DID NOT ADVANCE " IN ASM ALERT LOG

10356513 - DISK OFFLINED WITH NON ZERO TIMEOUT EXPELLED IMMEDIATELY

10332589 - TB:X:MOUNT NORMAL REDUNDANCY DG, FAILED WITH ORA-00600:[KFCINITRQ20]

10329146 - MARKING DIFFERENT SR BITS FROM MULTIPLE DBWS CAN CAUSE A LOST WRITE

10299224 - TB:X:PIVOTING AN EXTENT ON AN OFFLINE DISK CAN CAUSE STALE XMAPS IN RDBMS

10245086 - ORA-01210 DURING CREATE TABLESPACE

10230571 - TB:X:REBOOT ONE CELL NODE, RBAL HIT ORA-600[17183]

10228151 - ASM DISKGROUPS NOT GETTING MOUNTED

10227288 - DG FORCIBLY DISMOUNTED AFTER ONE FG LOST DUE TO "COULD NOT READ PST FOR GRP 4"

10222719 - ASM INSTANCE HANGS WITH RBAL PROCESS WAITS ON "NO FREE BUFFER"

10102506 - DISK RESYNC TAKES A LONG TIME EVEN WITH NO STALE EXTENTS

10094201 - DISK OFFLINE IS SLOW

10190642 - ORA-00600: [1433] FOLLOWED BY INSTANCE CRASH WITH ASM ON EXADATA

11067567 - 11202_gibtwo: kept generating "elapsed time did not advance " in asm alert log

Buffer Cache Management

9651350 - ora-00308 and ora-27037 when ora-8103 without event 10736 been set

10110863 - trace files is still generated after applying patch:9651350

10205230 - tb_x64: hit ora-00600: [kclwcrs_6]

10332111 - sql running long in active standby

CRS Group

CLEANUP

9949676 - GNSD.BIN CORE DUMP AFTER KILL ASM PMON ON ALL NODES AT SAME TIME

9975837 - GNS INCORRECTLY PROCESSES IPV6 LOOKUP REQUESTS

10007185 - GNS DUMPS CORE IN CLSKGOPANIC AT CLSKPDVA 717

10028343 - GNS CAN NOT BE RELOCATED AFTER PUBLIC RESTARTED

CRS

9876201 - OHASD AGENT CORE DUMP AT EONSHTTP.C:162

10011084 - 11202 STEP3 MODIFY BINARY AFTER INSTALLATION CANNOT EXCUTE SUCCESSFULLY

10028235 - 'CLSNVIPAGENT.CPP', LINE 1522: ERROR: FORMAL ARGUMENT TYPE OF ...

10045436 - 'ORA.LISTENER.LSNR' FAILED TO BE FENCED OFF DURING CRSD CLEANUP

10062301 - VALUE FOR FIELD 'CLUSTER_NAME' IS MISSING IN CRSCONFIG_PARAMS

10110969 - PORTABILITY ISSUES IN FUNCTION TOLOWER_HOST

10175855 - FAILED TO UGPRADE 11.2.0.1 + ARU 12900951 -> 11.2.0.2

9891341 - CRSD CORE DUMP IN PROATH_MASTER_EXIT_HELPER AT PROATH.C:1834

11655840 - RAC1 DB' STATE_DETAILS IS WRONG AFTER KILL GIPCD

10634513 - OHASD DUMPS CORE WHEN PLUG IN UNPLUGGED PRIVATE NETWORK NIC

10236074 - ASM INSTANCES CRASH SEVERAL TIMES DURING PARALLEL CRS STARTUP

10052529 - DB INST OFFLINE AFTER STOP/START CRS STACK ON ALL NODES IN PARALLEL

10065216 - VIRTUAL MEMORY USAGE OF ORAROOTAGENT IS BIG(1321MB) AND NOT DECREASING

10168006 - ORAAGENT PROCESS MEMORY GROWTH PERIODICALLY.

CSS

9907089 - CSS CORE DUMP DURING EXADATA ROLLING UPGRADE

9926027 - NODE REBOOTED AFTER CRS CLEAN-UP SUCCEEDED 11202 GI + 10205 RAC DB

10014392 - CRSCTL DELETE NODE FAILS WITH CRS-4662 & CRS-4000

10015460 - REMOVAL OF WRONG INCARNATION OF A NODE DUE TO MANUAL SHUTDOWN STATE

10040109 - PMON KILL LEAD TO OS REBOOT

10048027 - ASM UPGRADE FAILS

10052721 - 11201- 11202 NON-ROLLING,CRSCTL.BIN CORE AT CLSSNSQANUM, SIGNAL 11

10083789 - A NODE DOESNT INITIATE A RECONFIG DUE TO INCORRECT RECONFIG STATE

9944978 - FALSE CSS EVICTION AFTER PRIVATE NIC RESUME

9978195 - STOP DB ACTION TIMED OUT AND AGENT EXITS DUE TO FAILURE TO STOP EVENT BRIDGE

10248739 - AFTER APPLY THE PATCH, THE NODE EVICTED DURING START CRS STACK

CVU

9679401 - OUI PREREQ CHECKS FAILED FOR WRONG OWNSHIP OF RESOLV.CONF_`HOST`

9959110 - GNS INTEGRITY PREREQUISITE FAILED WITH PRVF-5213

9979706 - COMP OCR CHECK FAILS TO VERIFY SIZE OF OCR LOCATION

10029900 - CVU PRE NODEADD CHECK VD ERROR

10033106 - ADDNODE.SH SHOULD INDICATE WHAT HAPPENS WHEN ERROR OCCURRING

10075643 - UNABLE TO CONTINUE CONFIG.SH FOR CRS UPGRAD

10083009 - GIPCD FAILS TO RETRIEVE INFORMATION FROM PEERS DUE TO INVALID ENDPOINT

GIPC

9812956 - STATUS OF CRSD AND EVMD GOES INTERMEDIATE FOR EVER WHEN KILL GIPC

9915329 - ORA-600 [603] IN DB AND ORA-603 IN ASM AFTER DOWN INTER-CONNECT NIC

9944948 - START RESOUCE HAIP FAILED WHEN RUN ROOT.SH

9971646 - ORAROOTAGENT CORE DUMPED AT NETWORKHAMAINTHREAD::READROUTEDATA

9974223 - GRID INFRASTRUCTURE NEEDS MULTICAST COMMUNICATION ON 230.0.1.0 ADDRESSES WORKING

10053985 - ERROR IN NETWORK ADDRESS ON SOLARIS 11

10057680 - OHASD ORAROOTAGENT.BIN SPIN CPU AFTER SIMULATE ASM DISK ERROR

10078086 - ROOTUPGRADE.SH FAIL FOR 'CRSCTL STARTUPGRADE' FAIL,10205-> 11202

10260251 - GRID INSTALLATION FAILS TO START HAIP DUE TO CHANGE IN NETWORK INTERFACE NAME

10111010 - CRSD HANGS FOR THE HANAME OF PEER CRSD

11782423 - OHASD.BIN TAKES CPU ABOUT 95% ~ 100%

11077756 - STARTUP FAILURE OF HAIP CAUSES INSTALLATION FAILURE

10375649 - DISABLE HAIP ON PRIMECLUSTER

10284828 - INTERFACE UPDATES GET LOST DURING BOUNCE OF CRSD PROCESS

10284693 - AIX EPIPE FAILURE

10233159 - NEED 20 MINS TO STARTUP CRS WHEN 1/2 GIPC NICS DOWN

10128191 - LRGSRG9 AND LRGSRGE FAILURE

GNS

9864003 - NODE REBOOT DUE TO 'ORA.GNS' FAILED TO BE FENCED OFF DURING CRSD

GPNP

9336825 - GPNPD FLUSH PROFILE PUSH ERROR MESSAGES IN CRS ALERT LOG

10314123 - GPNPD MAY NOT UPDATE PROFILE TO LATEST ON START

10105195 - PROC-32 ACCESSING OCR; CRS DOES NOT COME UP ON NODE

10205290 - DBCA FAILED WITH ERROR ORA-00132

10376847 - [ORA.CRF] [START] ERROR = ERROR 9 ENCOUNTERED WHEN CONNECTING TO MOND

IPD-OS

9812970 - IPD DO NOT MARK TYPE OF DISKS USED FOR VOTING DISK CORRECTLY

10057296 - IPD SPLIT BRAIN AFTER CHANGE BDB LOCATION

10069541 - IPD SPLIT BRAIN AFTER STOPPING ORA.CRF ON MASTER NODE

10071992 - UNREASONABLE VALUES FOR DISK STATISTICS

10072474 - A NODE IS NOT MONITORED AFTER STOP AND START THE ORA.CRF ON IT

10073075 - INVALID DATA RECEIVED FROM THE CLUSTER LOGGER SERVI

10107380 - IPD NOT STARTED DUE TO SCRFOSM_GET_IDS FAILED

OCR

9978765 - ROOTUPGRADE.SH HANG AND CRSD CRASHED ON OTHER NODES,10205-> 11202

10016083 - 'OCRCONFIG -ADD' NEEDS HELPFUL MESSAGE FOR ERROR ORA-15221

OPSM

9918485 - EMCONFIG FAIL WITH NULLPOINTEREXCEPTION AT RACTRANSFERCORE.JAVA

10018215 - RACONE DOES NOT SHUTDOWN INSTANCE DURING RELOCATION

10042143 - ORECORE11 LWSFDSEV CAUSED SEGV IN SRVM NATIVE METHODS

OTHERS

9963327 - CHMOD.PL GETS CALLED INSTEAD OF CHMOD.EXE

10008467 - FAILS DUE TO WRONG VERSION OF PERL USED:

10015210 - OCTSSD LEAK MEMORY 1.7M HR ON PE MASTER DURING 23 HOURS RUNNI

10027079 - CRS_SHUTDOWN_SYNCH EVENT NOT SENT IN SIHA

10028637 - SCLS.C COMPILE ERRORS ON AIX UNDECLARED IDENTIFIERS

10029119 - 11201-11202 CRS UPGRADE OUI ASKS TO RUN ROOTUPGRADE.SH

10036834 - PATCHES NOT FOUND ERROR WHILE UPGRADING GRID FROM 11201 TO 11202

10038791 - HAS SRG SRV GETTING MANY DIFS FOR AIX ON LABEL 100810 AND LATER

10040647 - LNX64-112022-UD; AQ AND RLB DO NOT WORK AFTER UPGRADING FROM 11201

10044622 - EVMD FAILED TO START AFTER KILL OHASD.BIN

10048487 - DIAGCOLLECTION CANNOT RETRIEVE IPD REPORTS

10073372 - DEINSTALL FAILED TO DELETE CRS_HOME ON REMOTE NODE IF OCR VD ON NFS

10089120 - WRONG PROMPT MESSAGE BY DEINSTALL COMMAND WHILE DELETING CRS HOME

10124517 - CRS STACK DOES NOT START AUTOMATICALLY AFTER NODE REBOOT

10157622 - 11.2.0.2 GI BUNDLE 1 HAS-CRS TRACKING BUG

RACG

10036193 - STANDBY NIC DOESN'T WORK IF DOWN PUBLIC NIC

10146768 - NETWORK RESOURCE FAILS TO START WITH IPMP ON SOLARIS 11

USM Miscellaneous

10146744 - ORA.REGISTRY.ACFS BECOME UNKOWN AND ACFS FS DISMOUNT

10283058 - RESOURCES ACFS NEEDS AN OPTION TO DISALLOW THE MOUNTING OF FILE SYSTEMS ON RESOURCE START

10193581 - ROOT.SH CRS-2674: START OF 'ORA.REGISTRY.ACFS' FAIL

10244210 - FAIL TO INSTALL ADVM/ACFS ON SOLARIS CONTAINER

10311856 - APPLY ASSERTION FAILURE:PBOARDENTRY>USRGBOARDRECENTRY_RECORD

Generic

9591812 - incorrect wait events in 11.2 ("cursor: mutex s" instead of "cursor: mutex x")

9905049 - ebr: ora-00600: internal error code, arguments: [kqlhdlod-bad-base-objn]

10052141 - exadata database crash with ora-7445 [_wordcopy_bwd_dest_aligned] and ora-600 [2

10052956 - ora-7445 [kjtdq()+176]

10157402 - lob segment has null data after long to lob conversion in parallel mode

10187168 - obsolete parent cursors if version count exceeds a threshold

10217802 - alter user rename raises ora-4030

10229719 - qrmp:12.2:ora-07445 while performing complete database import on solaris sparc

10264680 - incorrect version_number reported after patch for 10187168 applied

10411618 - add different wait schemes for mutex waits

11069199 - ora-600 [kksobsoletecursor:invalid stub] quering pq when pq is disabled

11818335 - additional changes when wait schemes for mutex waits is disabled

High Availability

10018789 - dbmv2-bigbh:spin in kgllock caused db hung and high library cache lock

10129643 - appsst gsi11g m9000: ksim generic wait event

10170431 - ctwr consuming lots of cpu cycles

Oracle Space Management

6523037 - et11.1dl: ora-600 [kddummy_blkchk] [6110] on update

9724970 - pdml fails with ora-600 [4511]. ora-600 [kdblkcheckerror] by block check

10218814 - dbmv2: ora-00600:[3020] data block corruption on standby

10219576 - ora-600 [ktsl_allocate_disp-fragment]

Oracle Transaction Management

10358019 - invalid results from flashback_transaction_query after applying patch:10322043

Oracle Utilities

10373381 - ora-600 [kkpo_rcinfo_defstg:objnotfound] after rerunning catupgrd.sql

Oracle Virtual Operating System Services

10127360 - dg4msql size increasing to 1.5gb after procedure executed 250 times

Server Manageability

11699057 - ora-00001: unique constraint (sys.wri$_sqlset_plans_tocap_pk) violated

针对Grid Infrastructure的大致升级步骤如下,具体见该PSU的readme文档,见(6 Appendix A: Manual Steps for Apply/Rollback Patch)
部分:

1.升级Opatch,Patch 12419353 requires OPatch version 11.2.0.1.4.
可以从这里下载到11.2上最新的opatch工具,解压后直接覆盖原Opatch目录即可

2.关闭相关的RAC数据库  

<ORACLE_HOME>/bin/srvctl stop database –d <db-unique-name> 

3.Umount所有的ACFS文件系统

4. 切换到oracle用户,停止所有节点上的相关资源
su  - oracle
$<ORACLE_HOME>/bin/srvctl stop home -o <ORACLE_HOME> -s <status file location> -n <node name>

5.在所有节点上运行rootcrs.pl脚本停止GI相关资源
su - root
<GI_HOME>/crs/install/rootcrs.pl -unlock
Using configuration parameter file: /g01/11.2.0/grid/crs/install/crsconfig_params
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'vrh1'
CRS-2673: Attempting to stop 'ora.crsd' on 'vrh1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'vrh1'
CRS-2673: Attempting to stop 'ora.asm' on 'vrh1'
CRS-2677: Stop of 'ora.asm' on 'vrh1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'vrh1' has completed
CRS-2677: Stop of 'ora.crsd' on 'vrh1' succeeded
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'vrh1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'vrh1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'vrh1'
CRS-2673: Attempting to stop 'ora.evmd' on 'vrh1'
CRS-2673: Attempting to stop 'ora.asm' on 'vrh1'
CRS-2677: Stop of 'ora.asm' on 'vrh1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'vrh1'
CRS-2677: Stop of 'ora.evmd' on 'vrh1' succeeded
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'vrh1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'vrh1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'vrh1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'vrh1'
CRS-2677: Stop of 'ora.cssd' on 'vrh1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'vrh1'
CRS-2673: Attempting to stop 'ora.diskmon' on 'vrh1'
CRS-2677: Stop of 'ora.diskmon' on 'vrh1' succeeded
CRS-2677: Stop of 'ora.gipcd' on 'vrh1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'vrh1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'vrh1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'vrh1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'vrh1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
Successfully unlock /g01/11.2.0/grid

6.解压下载的PSU补丁包
unzip p12419353_112020_Linux-x86-64.zip 

7.
在所有节点上实施对GI的Patch

 <GI_HOME>/OPatch/opatch napply -oh <GI_HOME> -local /tmp/12419353

Invoking OPatch 11.2.0.1.6

Oracle Interim Patch Installer version 11.2.0.1.6
Copyright (c) 2011, Oracle Corporation.  All rights reserved.

UTIL session

Oracle Home       : /g01/11.2.0/grid
Central Inventory : /g01/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.6
OUI version       : 11.2.0.2.0
Log file location : /g01/11.2.0/grid/cfgtoollogs/opatch/opatch2011-08-12_00-07-39AM.log

Verifying environment and performing prerequisite checks...
OPatch continues with these patches:   12419353  

Do you want to proceed? [y|n]

Backing up files...
Applying interim patch '12419353' to OH '/g01/11.2.0/grid'

Patching component oracle.crs, 11.2.0.2.0...
Copying file to "/g01/11.2.0/grid/crs/install/crsconfig_lib.pm"
Copying file to "/g01/11.2.0/grid/crs/install/crspatch.pm"
Copying file to "/g01/11.2.0/grid/crs/install/s_crsconfig_lib.pm"

Patching component oracle.usm, 11.2.0.2.0...
Patches 12419353 successfully applied.
Log file location: /g01/11.2.0/grid/cfgtoollogs/opatch/opatch2011-08-12_00-07-39AM.log

OPatch succeeded.

<GI_HOME>/OPatch/opatch napply -oh <GI_HOME> -local /tmp/12419331

Invoking OPatch 11.2.0.1.6

Oracle Interim Patch Installer version 11.2.0.1.6
Copyright (c) 2011, Oracle Corporation.  All rights reserved.

UTIL session

Oracle Home       : /g01/11.2.0/grid
Central Inventory : /g01/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.6
OUI version       : 11.2.0.2.0
Log file location : /g01/11.2.0/grid/cfgtoollogs/opatch/opatch2011-08-12_00-10-46AM.log

Verifying environment and performing prerequisite checks...
OPatch continues with these patches:   12419331  

Backing up files...
Applying interim patch '12419331' to OH '/g01/11.2.0/grid'
ApplySession: Optional component(s) [ oracle.sysman.console.db, 11.2.0.2.0 ] ,
[ oracle.sysman.oms.core, 10.2.0.4.3 ] , [ oracle.rdbms.dv, 11.2.0.2.0 ] ,
[ oracle.sysman.plugin.db.main.repository, 11.2.0.2.0 ]  not present in the Oracle Home or a higher version is found.

Patching component oracle.rdbms.rsf, 11.2.0.2.0...

Patching component oracle.rdbms, 11.2.0.2.0...
Copying file to "/g01/11.2.0/grid/psu/11.2.0.2.3/catpsu.sql"
Copying file to "/g01/11.2.0/grid/psu/11.2.0.2.3/catpsu_rollback.sql"
Copying file to "/g01/11.2.0/grid/cpu/scripts/patch_8837510.sql"
Copying file to "/g01/11.2.0/grid/cpu/scripts/emdb_recomp_invalids.sql"

Patching component oracle.ldap.rsf, 11.2.0.2.0...

Patching component oracle.rdbms.dbscripts, 11.2.0.2.0...

Patching component oracle.rdbms.rman, 11.2.0.2.0...
Patches 12419331 successfully applied.
Log file location: /g01/11.2.0/grid/cfgtoollogs/opatch/opatch2011-08-12_00-10-46AM.log

OPatch succeeded.

后续对DB HOME升级的步骤可以参照以下文档摘录 

8.Run the pre script for DB component of the patch.

As the database home owner execute:

$<UNZIPPED_PATCH_LOCATION>/12419353/custom/server/12419353/custom/scripts/prepatch.sh -dbhome <ORACLE_HOME>

9.Apply the DB patch.

As the database home owner execute:

$<ORACLE_HOME>/OPatch/opatch napply -oh <ORACLE_HOME> -local <UNZIPPED_PATCH_LOCATION>/12419353/custom/server/12419353
$<ORACLE_HOME>/OPatch/opatch napply -oh <ORACLE_HOME> -local <UNZIPPED_PATCH_LOCATION>/12419331

10.Run the post script for DB component of the patch.

As the database home owner execute:

$<UNZIPPED_PATCH_LOCATION>/12419353/custom/server/12419353/custom/scripts/postpatch.sh -dbhome <ORACLE_HOME>

11.Run the post script.

As the root user execute:

#<GI_HOME>/rdbms/install/rootadd_rdbms.sh
If this is a GI Home, as the root user execute:

#<GI_HOME>/crs/install/rootcrs.pl -patch
If this is an Oracle Restart Home, as the root user execute:

#<GI_HOME>/crs/install/roothas.pl -patch

12.Start the CRS managed resources that were earlier running from DB homes.

If this is a GI Home environment, as the database home owner execute:

$<ORACLE_HOME>/bin/srvctl start home -o <ORACLE_HOME> -s <status file location> -n <node name>
If this is an Oracle Restart Home environment, as the database home owner execute:

$<ORACLE_HOME>/bin/srvctl start home -o <ORACLE_HOME> -s <status file location> 

OPatch工具相关的环境变量

  • ORACLE_HOME – This is the Oracle home location.
  • OPATCH_DEBUG – This is the log level that specifies the amount of logging OPatch should perform.
  • OPATCH_PLATFORM_ID – This is the unique platform ID.
  • PATH – This is the path information.
  • OPatch.SKIP_VERIFY_PLATFORM – Allows us to skip verification phase. Using this will not restore Oracle Home if patching fails at verification phase.

Critical Patch Update July 2011 Released

2011年7月的CPU在19日发布了,Database相关的PSU/CPU包括:

11.2.0.2:CPU Patch 12419321, or DB PSU Patch 12419331, or GI PSU Patch 12419353, or Exadata BP9 Patch 12681774

11.2.0.1:CPU Patch 12419278, or DB PSU Patch 12419378, or Exadata BP11 Patch 12608545

11.1.0.7: CPU Patch 12419265, or PSU Patch 12419384

10.2.0.5:CPU Patch 12419258, or PSU Patch 12419392

10.2.0.4:CPU Patch 12419249, or PSU Patch 12419397

Oracle Database Patch Set Update (PSU)

Oracle Database UNIX Advisory Number Comments
11.2.0.2.3 Database PSU Patch 12419331 See Section 3.1.3.2, “Oracle Database 11.2.0.2”
11.2.0.2.3 Grid Infrastructure PSU Patch 12419353 See Section 3.1.3.2, “Oracle Database 11.2.0.2” Includes CPUJul2011 and 11.2.0.2.3 Database PSU
11.2.0.2 BP9 for Exadata Patch 12681774 See Section 3.1.3.2, “Oracle Database 11.2.0.2” Includes CPUJul2011 and 11.2.0.2.3 Database and Grid Infrastructure PSU fixes for Exadata
11.2.0.1.6 Database PSU Patch 12419378 See Section 3.1.3.3, “Oracle Database 11.2.0.1”
11.2.0.1 BP11 for Exadata Patch 12608545 See Section 3.1.3.3, “Oracle Database 11.2.0.1” Includes CPUJul2011 and 11.2.0.1.6 PSU fixes for Exadata
11.2.0.1.2 Grid Infrastructure Patch 9655006 Released July 2010
11.1.0.7.8 Database PSU Patch 12419384 See Section 3.1.3.4, “Oracle Database 11.1.0.7”
11.1.0.7.7 CRS PSU Patch 11724953 Released April 2011
10.2.0.5.4 Database PSU Patch 12419392 See Section 3.1.3.5, “Oracle Database 10.2.0.5”
10.2.0.5.2 CRS PSU Patch 9952245 Released January 2011
10.2.0.4.9 Database PSU Patch 12419397 See Section 3.1.3.6, “Oracle Database 10.2.0.4” Overlay PSU
10.2.0.4.4 Database PSU Patch 9352164 Released April 2010 Base PSU for 10.2.0.4.9
10.2.0.4.4 CRS PSU Patch 9294403 Released April 2010

具体见<Patch Set Update and Critical Patch Update July 2011 Availability Document [ID 1323616.1]>

Waiting For KKSFBC CHILD COMPLETION?

客户有一套AIX 5.3上的10.2.0.4.5生产库系统,最近频繁出现”KKSFBC CHILD COMPLETION”等待,同时导致session不断spin消耗CPU并hang住,从表象看这似乎是由bug引起的。以KKSFBC CHILD COMPLETION为关键字到MOS查询可以找到<Bug 6795880 – Session spins / OERI after ‘kksfbc child completion’ wait – superceded [ID 6795880.8]>,该Bug的症状为进程不断spin且hang住、出现’KKSFBC CHILD COMPLETION’等待事件、还可能伴有’Waits for “cursor: pin S”‘等待事件,直接影响的版本有11.1.0.6、10.2.0.3和10.2.0.4。

对于该Bug的描述是在发生’kksfbc child completion’等待事件后会话陷入无休止的自旋(spins)中,这种自旋(spins)发生在由堆栈调用(stack call)kksSearchChildList->kkshgnc陷入对kksSearchChildList函数的无限循环中。

就当前用户提供的版本号及等待事件信息仍不足以定位到该Bug,我们需要更详细的stack call。所幸的是这个trouble是可以重现的(reproduceable),在之后的一次案发现场我们得到了必要的信息:

Name            PID  CPU%  PgSp Owner
oracle      3723390  10.0   7.0 oracle

SQL> oradebug setospid 3723390
Oracle pid: 155, Unix process pid: 3723390, image: oracle@lmsdb1

SQL> oradebug unlimit;
Statement processed.

SQL> oradebug short_stack;
ksdxfstk+002c<-ksdxcb+04e4<-sspuser+0074<-000044F0
<-kksfbc+0b9c<-kkspsc0+0ffc<-kksParseCursor+00d4
<-opiosq0+0ae0<-kpooprx+0168<-kpoal8+0400<-opiodr+0ae0<-ttcpip+1020<-opitsk+1124
<-opiino+0990<-opiodr+0ae0<-opidrv+0484<-sou2o+0090<-opimai_real+01bc<-main+0098<-__start+0098

SQL> oradebug dump processstate 10;
Statement processed.

SQL> oradebug dump systemstate 266;
Statement processed.

=========================process state dump=========================

    SO: 7000003f72e3378, type: 4, owner: 7000003f225c798, flag: INIT/-/-/0x00
    (session) sid: 270 trans: 0, creator: 7000003f225c798, flag: (41) USR/- BSY/-/-/-/-/-
              DID: 0001-009B-0000017F, short-term DID: 0000-0000-00000000
              txn branch: 0
              oct: 0, prv: 0, sql: 700000243ef9540, psql: 7000002fe76d9c0, user: 28/OLSUSER
    service name: SYS$USERS
    O/S info: user: newprepay, term: unknown, ospid: 1234, machine: newprepayC
              program: JDBC Thin Client
    application name: JDBC Thin Client, hash value=2546894660
    last wait for 'kksfbc child completion' blocking sess=0x0 seq=4397 wait_time=48840 seconds since wait started=144918
                =0, =0, =0
    Dumping Session Wait History
     for 'kksfbc child completion' count=1 wait_time=48840
                =0, =0, =0

可以从以上trace中看到会话确实曾长时间处于’kksfbc child completion’等待中,之后陷入无限自旋(spins)中消耗了大量CPU时间。但这里实际的表现又存有差异,引发无限循环的函数是kksfbc而不是kksSearchChildList(常规的调用序列是:kksParseCursor->kkspsc0->kksfbc ->kksSearchChildList->kkshgnc)。

kksfbc意为K[Kernel]K[Kompile]S[Shared]F[Find]B[Best]C[Child]该函数用以在软解析时找寻合适的子游标,在10.2.0.2以后引入了mutex互斥体来取代原有的Cursor Pin机制,Mutex较Latch更为轻量级。著名的Tanel Poder指出虽然mutex的引入改变了众多cursor pin的内部机制,但kksfbc仍需要持有library cache latches才能扫描library cache hash chains(However the traversing of library cache hash chains (the right child cursor lookup using kksfbc()) was still protected by library cache latches)。另一方面当kksfbc函数针对某个parent cursor找到合适child cursor后,可能使用KKSCHLPINx方法将该child cursor pin住,这个时候就需要exclusive地持有该child cursor相应的mutex,如:

SQL>  select mutex_addr,requesting_session rs,blocking_session bs,location from x$mutex_sleep_history;

MUTEX_ADDR               RS         BS LOCATION
---------------- ---------- ---------- ----------------------------------------
00000000A3CF8928        159        148 kksfbc [KKSCHLPIN1]
00000000A3CEA8A8        159          0 kksfbc [KKSCHLPIN1]
00000000A3CF5508        159          0 kksfbc [KKSCHLPIN1]
00000000A3CF10E8        148        159 kksLockDelete [KKSCHLPIN6]

当有一个进程执行kksfbc,而其他进程可能需要陷入’kksfbc child completion’等待中(更多的是cursor:pin S等待事件),但这种等待一般是十分轻微的,你很难从某个”仪表”上观察到这一事件。因为一系列kksfbc相关的bug仅发生在10.2.0.2以后,可以猜测是由于mutex的引入引起的。

打破’kksfbc child completion’异常等待的一种行之有效的workaround方法是设置隐藏参数_use_kks_mutex_pin为false,即使用传统的由latch保护的Cursor pin;设置该hidden parameter需要重启实例,具体的设置方法如下:

alter system set "_kks_use_mutex_pin"=false scope=spfile;

restart instance...........

但这种方法也仅仅是行之有效,而非万试万灵。从消极的角度来说,因为放弃了mutex保护cursor pin的机制,在解析频繁的系统中library cache latch的争用将白热化,有可能成为Top 5等待事件。

另一种值得推荐的workaround是减少硬编码的SQL语句,使用session_cached_cursor减少实际的软解析操作也有助于避免引发该Bug;当然这要求我们修改代码,但从长远来看这种改良是值得的。

最后Oracle在10.2.0.4上提供了该Bug的one-off Patch 8575528,其在10.2.0.4 psu4以后的等价补丁为(Equivalent patch)为merge patch 9696904:

8557428 9696904 7527908 Both fixes are needed. 6795880 superceded by 8575528 in 9696904 which includes extra files so may cause new conflicts

但merge patch 9696904目前仅有Linux x86/64平台上的版本,而问题数据库所在平台为IBM AIX on POWER Systems (64-bit)。如果要通过补丁来fix这个问题的话,AIX平台的用户可以要求Oracle development开发部门build一个Power版的9696904 patch,也可以升级到10.2.0.5上去

注意Patch 8575528: MISSING ENTRIES IN V$MUTEX_SLEEP.LOCATION目前有IBM AIX on POWER Systems (64-bit)平台上10.2.0.4.4的版本,经过验证该one-off patch可以在10.2.0.4 PSU4以后的版本上实施(包括10.2.0.4.5/10.2.0.4.6等),不需要如以上描述地去apply 9696904这个merge patch。当然升级到10.2.0.5/11.1.0.7或以上版本依然有效。

此外apply以上8575528:后需要修改隐藏参数”_cursor_features_enabled”并重启才能使fix生效,具体设置防范如下:

ALTER SYSTEM SET "_cursor_features_enabled"=10 scope=spfile;
restart instance.............

注意以上参数仅在使用one-off patch时需要设置,而当通过升级到10.2.0.5/11.1.0.7来修复问题时无需设置上述”_cursor_features_enabled”参数。

使用OPATCH_DEBUG环境变量调试Opatch工具

我们在使用opatch数据库补丁管理工具时可能碰到这样那样的问题,那么opatch工具是否存在一些调试选项以便我们了解更多的信息呢?同OUI工具一样,opatch存在一个debug选项,使用该选项我们仅需要设置OPATCH_DEBUG环境变量为TRUE就可以了,如:

[oracle@rh2 OPatch]$ export OPATCH_DEBUG=TRUE

[oracle@rh2 OPatch]$ ./opatch lsinventory
ORACLE_HOME is set at OPatch invocation
Machine Info: Linux rh2.oracle.com 2.6.32-100.0.19.el5 #1 
SMP Fri Sep 17 17:51:41 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux

_osArch is i386
_javaVMSpecVersion is 1.0
_javaVMSpecVendor is Sun Microsystems Inc.
_javaVMSpecName is Java Virtual Machine Specification
_javaVMVendor is Sun Microsystems Inc.
_javaJRESpecVersion is 1.4
_javaJRESpecVendor is Sun Microsystems Inc.
_javaJRESpecName is Java Platform API Specification
_javaSupportedClassVersion is 48.0
OPatch compiled with major version: 0, minor version: 0
_osArch (from OCM API) is i386
/s01/db_1/jdk/bin/java  -mx96m -cp ./ocm/lib/emocmutl.jar:./ocm/lib/emocmclnt.jar:/s01/db_1/oui/jlib/OraInstaller.jar:
/s01/db_1/oui/jlib/OraPrereq.jar:/s01/db_1/oui/jlib/share.jar:/s01/db_1/oui/jlib/srvm.jar:
/s01/db_1/oui/jlib/orai18n-mapping.jar:/s01/db_1/oui/jlib/xmlparserv2.jar:./jlib/opatch.jar:./jlib/opatchutil.jar:
./jlib/opatchprereq.jar:./jlib/opatchactions.jar:./jlib/opatchext.jar:
./jlib/opatchfmw.jar: -DOPatch.ORACLE_HOME=/s01/db_1 -DOPatch.DEBUG=true 
-DOPatch.RUNNING_DIR=. -DOPatch.MW_HOME= -DOPatch.WL_HOME= 
-DOPatch.COMMON_COMPONENTS_HOME=   oracle/opatch/OPatch lsinventory

/* 以上debug信息可以说明opatch工具需要用到Oracle软件自带的JRE */

Invoking OPatch 11.2.0.1.3

Oracle Interim Patch Installer version 11.2.0.1.3
Copyright (c) 2010, Oracle Corporation.  All rights reserved.

OPatchSession::parse() on "lsinventory",

Argument is "lsinventory"
 Add commands for Help
 add command "apply"
 add command "napply"
 add command "rollback"
 add command "nrollback"
 add command "lsinventory"
 add command "lsinv"
 add command "query"
 add command "util"
 add command "prereq"
 add command "version"
 add command "-help"
 add command "-help -fmw"
 Add supported commands for validation
 add command "apply"
 add command "rollback"
 add command "lsinv"
 add command "lsinventory"
 add command "query"
 add command "util"
 add command "prereq"
 add command "version"
 add command "napply"
 add command "nrollback"
 add command "-fmw"
 Not a command that can be mapped to Util Session.
CmdLineParser::initRuntimeOptions()
 Checking on class oracle.opatch.opatchutil.CmdLineOptions$StringArguments
 Get list of fields defined in the class oracle.opatch.opatchutil.CmdLineOptions$StringArguments
 There are 7 fields defined in this class.
   adding option "fp"
   adding option "dp"
   adding option "fr"
   adding option "dr"
   adding option "mp"
   adding option "phbasedir"
   adding option "phbasefile"
 Checking on class oracle.opatch.opatchutil.CmdLineOptions$BooleanArguments
 Get list of fields defined in the class oracle.opatch.opatchutil.CmdLineOptions$BooleanArguments
 There are 2 fields defined in this class.
   adding option "delay_link"
   adding option "cmd_end"
 Checking on class oracle.opatch.opatchutil.CmdLineOptions$IntegerArguments
 Get list of fields defined in the class oracle.opatch.opatchutil.CmdLineOptions$IntegerArguments
 There are 2 fields defined in this class.
   adding option "integerarg1"
   adding option "integerarg2"
 Checking on class oracle.opatch.opatchutil.CmdLineOptions$StringtegerArguments
 Get list of fields defined in the class oracle.opatch.opatchutil.CmdLineOptions$StringtegerArguments
 There are 5 fields defined in this class.
   adding option "stringtegerarg1"
   adding option "stringtegerarg2"
   adding option "ps"
   adding option "mp"
   adding option "xmlinput"
 Checking on class oracle.opatch.opatchutil.CmdLineOptions$DoubleArguments
 Get list of fields defined in the class oracle.opatch.opatchutil.CmdLineOptions$DoubleArguments
 There are 2 fields defined in this class.
   adding option "doublearg1"
   adding option "doublearg2"
 Checking on class oracle.opatch.opatchutil.CmdLineOptions$RawStringArguments
 Get list of fields defined in the class oracle.opatch.opatchutil.CmdLineOptions$RawStringArguments
 There are 1 fields defined in this class.
   adding option "cmd"
CmdLineHelper::loadRuntimeOption() for Class "oracle.opatch.opatchutil.OUSession"
  initializing String option 0, fp
  initializing String option 1, dp
  initializing String option 2, fr
  initializing String option 3, dr
  initializing String option 4, mp
  initializing String option 5, phbasedir
  initializing String option 6, phbasefile
 done init. String arg.
  initializing Boolean option 0, delay_link
  initializing Boolean option 1, cmd_end
 done init. Boolean arg.
  initializing Integer option 0, integerarg1
  initializing Integer option 1, integerarg2
 done init. Integer arg.
  initializing StringTeger option 0, stringtegerarg1
  initializing StringTeger option 1, stringtegerarg2
  initializing StringTeger option 2, ps
  initializing StringTeger option 3, mp
  initializing StringTeger option 4, xmlinput
 done init. SringTeger arg.
  initializing Double option 0, doublearg1
  initializing Double option 1, doublearg2
 done init. Double arg.
  initializing RawString option 0, cmd
 done init. RawString arg.
CmdLineHelper::loadRuntimeOption() for Class "oracle.opatch.opatchutil.OUSession", done.
CmdLineParser::initRuntimeOptions()
 Checking on class oracle.opatch.opatchprereq.CmdLineOptions$StringArguments
 Get list of fields defined in the class oracle.opatch.opatchprereq.CmdLineOptions$StringArguments
 There are 3 fields defined in this class.
   adding option "phbasedir"
   adding option "patchids"
   adding option "phbasefile"
 Checking on class oracle.opatch.opatchprereq.CmdLineOptions$BooleanArguments
 Get list of fields defined in the class oracle.opatch.opatchprereq.CmdLineOptions$BooleanArguments
 There are 2 fields defined in this class.
   adding option "booleanarg1"
   adding option "booleanarg2"
 Checking on class oracle.opatch.opatchprereq.CmdLineOptions$IntegerArguments
 Get list of fields defined in the class oracle.opatch.opatchprereq.CmdLineOptions$IntegerArguments
 There are 2 fields defined in this class.
   adding option "integerarg1"
   adding option "integerarg2"
 Checking on class oracle.opatch.opatchprereq.CmdLineOptions$StringtegerArguments
 Get list of fields defined in the class oracle.opatch.opatchprereq.CmdLineOptions$StringtegerArguments
 There are 2 fields defined in this class.
   adding option "stringtegerarg1"
   adding option "stringtegerarg2"
 Checking on class oracle.opatch.opatchprereq.CmdLineOptions$DoubleArguments
 Get list of fields defined in the class oracle.opatch.opatchprereq.CmdLineOptions$DoubleArguments
 There are 2 fields defined in this class.
   adding option "doublearg1"
   adding option "doublearg2"
CmdLineHelper::loadRuntimeOption() for Class "oracle.opatch.opatchprereq.PQSession"
  initializing String option 0, phbasedir
  initializing String option 1, patchids
  initializing String option 2, phbasefile
 done init. String arg.
  initializing Boolean option 0, booleanarg1
  initializing Boolean option 1, booleanarg2
 done init. Boolean arg.
  initializing Integer option 0, integerarg1
  initializing Integer option 1, integerarg2
 done init. Integer arg.
  initializing StringTeger option 0, stringtegerarg1
  initializing StringTeger option 1, stringtegerarg2
 done init. SringTeger arg.
  initializing Double option 0, doublearg1
  initializing Double option 1, doublearg2
 done init. Double arg.
CmdLineHelper::loadRuntimeOption() for Class "oracle.opatch.opatchprereq.PQSession", done.
 reqVer For using getEnv() = 10.2.0.4.0
 curVer = 10.2.0.4.0
 Current Ver later than required? :false
 Current Ver equals required? :true
Checking EMDROOT using OUI's API...
CmdLineParser.processOPatchProperties() begins
CmdLineParser.processOPatchProperties() ends
OUIReplacer::runEnvScript() called
SystemCall:RuntimeExec(cmds, runDir): GOING to start thread to read Input Stream
SystemCall:RuntimeExec(cmds, runDir): Started thread to read Input Stream
SystemCall:RuntimeExec(cmds, runDir): GOING to start thread to read Error Stream
ReaderThread::run(): Stream InputStream about to be read
SystemCall:RuntimeExec(cmds, runDir): Started thread to read Error Stream
SystemCall:RuntimeExec(cmds, runDir): GOING into process.waitFor()
ReaderThread::run(): Stream ErrorStream about to be read
ReaderThread::run(): Stream ErrorStream reading completed
ReaderThread::run(): Stream InputStream reading completed
SystemCall:RuntimeExec(cmds, runDir): process.waitFor() is OVER
SystemCall:RuntimeExec(cmds, runDir): Error stream thread joined successfully
SystemCall:RuntimeExec(cmds, runDir): Input stream thread joined successfully
OUIReplacer::setKeyValue() called
OPatchSession::main()
Environment:
   OPatch.ORACLE_HOME=/s01/db_1
   oracle.installer.invPtrLoc=/etc/oraInst.loc
   oracle.installer.oui_loc=/s01/db_1/oui
   oracle.installer.library_loc=/s01/db_1/oui/lib/linux
   oracle.installer.startup_location=/s01/db_1/oui
   OPatch.PLATFORM_ID=
   os.name=Linux
   OPatch.NO_FUSER=
   OPatch.SKIP_VERIFY=null
   OPatch.SKIP_VERIFY_SPACE=null
   oracle.installer.clusterEnabled=false
   TRACING.ENABLED=TRUE
   TRACING.LEVEL=2
   OPatch.DEBUG=true
   OPATCH_VERSION=11.2.0.1.3
   Bundled OPatch Property File=properties
   Minimum OUI version: 10.2
   OPatch.PATH=/s01/db_1/bin:/bin:/usr/bin:/sbin:/usr/sbin
Stand-Alone home : false
   OPatch.MW_HOME=
   OPatch.WL_HOME=
   OPatch.COMMON_COMPONENTS_HOME=

Environment:
   OPatch.ORACLE_HOME=/s01/db_1
   oracle.installer.invPtrLoc=/etc/oraInst.loc
   oracle.installer.oui_loc=/s01/db_1/oui
   oracle.installer.library_loc=/s01/db_1/oui/lib/linux
   oracle.installer.startup_location=/s01/db_1/oui
   OPatch.PLATFORM_ID=
   os.name=Linux
   OPatch.NO_FUSER=
   OPatch.SKIP_VERIFY=null
   OPatch.SKIP_VERIFY_SPACE=null
   oracle.installer.clusterEnabled=false
   TRACING.ENABLED=TRUE
   TRACING.LEVEL=2
   OPatch.DEBUG=true
   OPATCH_VERSION=11.2.0.1.3
   Bundled OPatch Property File=properties
   Minimum OUI version: 10.2
   OPatch.PATH=/s01/db_1/bin:/bin:/usr/bin:/sbin:/usr/sbin
Stand-Alone home : false
   OPatch.MW_HOME=
   OPatch.WL_HOME=
   OPatch.COMMON_COMPONENTS_HOME=

Oracle Home       : /s01/db_1
Central Inventory : /s01/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.3
OUI version       : 10.2.0.4.0
OUI location      : /s01/db_1/oui
Log file location : /s01/db_1/cfgtoollogs/opatch/opatch2011-05-06_23-38-48PM.log

Patch history file: /s01/db_1/cfgtoollogs/opatch/opatch_history.txt

OUISessionManager::instantiate()
lockCentralInventory(): OUISessionManager::lockCentralInventory() will retry 0 times with 120-second interval to get an Inventory lock.
OUISessionManager::lockCentralInventory() try round # 1
OUISessionManager::setupSession()
OUISessionManager::setupSession() instantiates a OUIInventorySession obj.
OUISessionManager::setupSession() init. the session
OUISessionManager::setupSession() sets up READ-ONLY session
OUISessionManager::setupSession() done
OUISessionManager::lockCentralInventory() set up session OK
 reqVer = 10.2
 curVer = 10.2.0.4.0
 Current Ver later than required? :true
 Current Ver equals required? :false
OracleHomeInventory::createInventoryObj()
OracleHomeInventory::createInventoryObj() gets OUIInventorySession object
Locker::lock()
   calling lockCentralInventory()
OUISessionManager::getInventorySession()
Caller Details: 
        Caller Name : OPatch    Caller Version : 11.2.0.1.3     Requested Read-only access : true       Oracle Home : /s01/db_1
OUISessionManager::register()
Registering the caller : OPatch
OracleHomeInventory::createInventoryObj() gets OUIInstallAreaControl object
OracleHomeInventory::createInventoryObj() gets OUIInstallInventory object
OracleHomeInventory::createInventoryObj() gets OUIOracleHomeInfo object
OracleHomeInventory::createInventoryObj() construction done
LsInventory::loadAndPrintInventory()
Retrieving inventory from Oracle Home...
OracleHomeInventory::load()
OracleHomeInventory::load() gets a Vector of all product entries
OracleHomeInventory::load() gets a Vector of all one-off entries
OracleHomeInventory::load() begins to process raw data from OUI to build up OPatch primitive classes
PatchObject(oracleHomePath,patchID)
PatchObject::PatchObject looks for the actions file /s01/db_1/inventory/oneoffs/9352164/etc/config/actions.xml
PatchObject::PatchObject looks for the inventory file /s01/db_1/inventory/oneoffs/9352164/etc/config/inventory.xml
PatchObject::PatchObject looks for the deploy file /s01/db_1/inventory/oneoffs/9352164/etc/config/deploy.xml
PatchObject::PatchObject looks for the interview file /s01/db_1/inventory/oneoffs/9352164/etc/config/sequence.xml
PatchObject::createPatchObject() deploy actions file /s01/db_1/inventory/oneoffs/9352164/etc/config/deploy.xml does not exist
PatchObject::createPatchObject() deploy actions file /s01/db_1/inventory/oneoffs/9352164/etc/config/deploy.xml unreadable
PatchObject::createPatchObject() interview file /s01/db_1/inventory/oneoffs/9352164/etc/config/sequence.xml does not exist or is not readable
PatchObject::createPatchObject() deploy actions file /s01/db_1/inventory/oneoffs/9352164/etc/config/sequence.xml unreadable
parserXMLFile : start
parserXMLFile : start
JarAction::processJarActionPaths()
JarAction::processJarActionPaths(): return true for successful parsing Jar path
JarAction::processJarActionPaths()
JarAction::processJarActionPaths(): return true for successful parsing Jar path
JarAction::getCompleteFileListPath() returns :ORACLE_HOME/rdbms/jlib/qsma.jar
JarAction::processJarActionPaths()
JarAction::processJarActionPaths(): return true for successful parsing Jar path
JarAction::processJarActionPaths()
JarAction::processJarActionPaths(): return true for successful parsing Jar path
JarAction::getCompleteFileListPath() returns :ORACLE_HOME/rdbms/jlib/qsma.jar
JarAction::processJarActionPaths()
JarAction::processJarActionPaths(): return true for successful parsing Jar path
JarAction::processJarActionPaths()
JarAction::processJarActionPaths(): return true for successful parsing Jar path
JarAction::getCompleteFileListPath() returns :ORACLE_HOME/rdbms/jlib/qsma.jar
JarAction::processJarActionPaths()
JarAction::processJarActionPaths(): return true for successful parsing Jar path
JarAction::processJarActionPaths()
JarAction::processJarActionPaths(): return true for successful parsing Jar path
JarAction::getCompleteFileListPath() returns :ORACLE_HOME/rdbms/jlib/CDC.jar
JarAction::processJarActionPaths()
JarAction::processJarActionPaths(): return true for successful parsing Jar path
JarAction::processJarActionPaths()
JarAction::processJarActionPaths(): return true for successful parsing Jar path
JarAction::getCompleteFileListPath() returns :ORACLE_HOME/sysman/jlib/emDB.jar
JarAction::processJarActionPaths()
JarAction::processJarActionPaths(): return true for successful parsing Jar path
JarAction::processJarActionPaths()
JarAction::processJarActionPaths(): return true for successful parsing Jar path
JarAction::getCompleteFileListPath() returns :ORACLE_HOME/sysman/jlib/emDB.jar
JarAction::processJarActionPaths()
JarAction::processJarActionPaths(): return true for successful parsing Jar path
JarAction::processJarActionPaths()
JarAction::processJarActionPaths(): return true for successful parsing Jar path
JarAction::getCompleteFileListPath() returns :ORACLE_HOME/sysman/jlib/emDB.jar
JarAction::processJarActionPaths()
JarAction::processJarActionPaths(): return true for successful parsing Jar path
JarAction::processJarActionPaths()
JarAction::processJarActionPaths(): return true for successful parsing Jar path
JarAction::getCompleteFileListPath() returns :ORACLE_HOME/sysman/jlib/emDB.jar
JarAction::processJarActionPaths()
JarAction::processJarActionPaths(): return true for successful parsing Jar path
JarAction::processJarActionPaths()
JarAction::processJarActionPaths(): return true for successful parsing Jar path
JarAction::getCompleteFileListPath() returns :ORACLE_HOME/sysman/jlib/emDB.jar
JarAction::processJarActionPaths()
JarAction::processJarActionPaths(): return true for successful parsing Jar path
JarAction::processJarActionPaths()
JarAction::processJarActionPaths(): return true for successful parsing Jar path
JarAction::getCompleteFileListPath() returns :ORACLE_HOME/sysman/jlib/emDB.jar
JarAction::processJarActionPaths()
JarAction::processJarActionPaths(): return true for successful parsing Jar path
JarAction::processJarActionPaths()
JarAction::processJarActionPaths(): return true for successful parsing Jar path
JarAction::getCompleteFileListPath() returns :ORACLE_HOME/sysman/jlib/emDB.jar
JarAction::processJarActionPaths()
JarAction::processJarActionPaths(): return true for successful parsing Jar path
JarAction::processJarActionPaths()
JarAction::processJarActionPaths(): return true for successful parsing Jar path
JarAction::getCompleteFileListPath() returns :ORACLE_HOME/sysman/jlib/emjsp.jar
JarAction::processJarActionPaths()
JarAction::processJarActionPaths(): return true for successful parsing Jar path
JarAction::processJarActionPaths()
JarAction::processJarActionPaths(): return true for successful parsing Jar path
JarAction::getCompleteFileListPath() returns :ORACLE_HOME/sysman/jlib/emjsp.jar
JarAction::processJarActionPaths()
JarAction::processJarActionPaths(): return true for successful parsing Jar path
JarAction::processJarActionPaths()
JarAction::processJarActionPaths(): return true for successful parsing Jar path
JarAction::getCompleteFileListPath() returns :ORACLE_HOME/sysman/jlib/emjsp.jar
JarAction::processJarActionPaths()
JarAction::processJarActionPaths(): return true for successful parsing Jar path
JarAction::processJarActionPaths()
JarAction::processJarActionPaths(): return true for successful parsing Jar path
JarAction::getCompleteFileListPath() returns :ORACLE_HOME/sysman/jlib/emjsp.jar
PatchObject(oracleHomePath,patchID)
PatchObject::PatchObject looks for the actions file /s01/db_1/inventory/oneoffs/9654991/etc/config/actions.xml
PatchObject::PatchObject looks for the inventory file /s01/db_1/inventory/oneoffs/9654991/etc/config/inventory.xml
PatchObject::PatchObject looks for the deploy file /s01/db_1/inventory/oneoffs/9654991/etc/config/deploy.xml
PatchObject::PatchObject looks for the interview file /s01/db_1/inventory/oneoffs/9654991/etc/config/sequence.xml
PatchObject::createPatchObject() deploy actions file /s01/db_1/inventory/oneoffs/9654991/etc/config/deploy.xml does not exist
PatchObject::createPatchObject() deploy actions file /s01/db_1/inventory/oneoffs/9654991/etc/config/deploy.xml unreadable
PatchObject::createPatchObject() interview file /s01/db_1/inventory/oneoffs/9654991/etc/config/sequence.xml does not exist or is not readable
PatchObject::createPatchObject() deploy actions file /s01/db_1/inventory/oneoffs/9654991/etc/config/sequence.xml unreadable
parserXMLFile : start
parserXMLFile : start
PatchObject(oracleHomePath,patchID)
PatchObject::PatchObject looks for the actions file /s01/db_1/inventory/oneoffs/7527908/etc/config/actions.xml
PatchObject::PatchObject looks for the inventory file /s01/db_1/inventory/oneoffs/7527908/etc/config/inventory.xml
PatchObject::PatchObject looks for the deploy file /s01/db_1/inventory/oneoffs/7527908/etc/config/deploy.xml
PatchObject::PatchObject looks for the interview file /s01/db_1/inventory/oneoffs/7527908/etc/config/sequence.xml
PatchObject::createPatchObject() deploy actions file /s01/db_1/inventory/oneoffs/7527908/etc/config/deploy.xml does not exist
PatchObject::createPatchObject() deploy actions file /s01/db_1/inventory/oneoffs/7527908/etc/config/deploy.xml unreadable
PatchObject::createPatchObject() interview file /s01/db_1/inventory/oneoffs/7527908/etc/config/sequence.xml does not exist or is not readable
PatchObject::createPatchObject() deploy actions file /s01/db_1/inventory/oneoffs/7527908/etc/config/sequence.xml unreadable
parserXMLFile : start
parserXMLFile : start
PatchObject(oracleHomePath,patchID)
PatchObject::PatchObject looks for the actions file /s01/db_1/inventory/oneoffs/9696904/etc/config/actions.xml
PatchObject::PatchObject looks for the inventory file /s01/db_1/inventory/oneoffs/9696904/etc/config/inventory.xml
PatchObject::PatchObject looks for the deploy file /s01/db_1/inventory/oneoffs/9696904/etc/config/deploy.xml
PatchObject::PatchObject looks for the interview file /s01/db_1/inventory/oneoffs/9696904/etc/config/sequence.xml
PatchObject::createPatchObject() deploy actions file /s01/db_1/inventory/oneoffs/9696904/etc/config/deploy.xml does not exist
PatchObject::createPatchObject() deploy actions file /s01/db_1/inventory/oneoffs/9696904/etc/config/deploy.xml unreadable
PatchObject::createPatchObject() interview file /s01/db_1/inventory/oneoffs/9696904/etc/config/sequence.xml does not exist or is not readable
PatchObject::createPatchObject() deploy actions file /s01/db_1/inventory/oneoffs/9696904/etc/config/sequence.xml unreadable
parserXMLFile : start
parserXMLFile : start
LsInventory::init()
LsInventorySession::process(): creates an output file /s01/db_1/cfgtoollogs/opatch/lsinv/lsinventory2011-05-06_23-38-48PM.txt
LsInventory::init() returns
Lsinventory Output file location : /s01/db_1/cfgtoollogs/opatch/lsinv/lsinventory2011-05-06_23-38-48PM.txt

--------------------------------------------------------------------------------
[Rules: should lock OH and Central Inv.= true, read from OUI= true, write to OUI= true, read from system= true, write to system= true, -all= false, -xml= false, -match= false, -patch= false, -detail= false, print patch = true, print all prods = false, print top-level prod = true, print detail = false, print by-bugs fixed = false ]
Installed Top-level Products (2): 

Oracle Database 10g                                                  10.2.0.1.0
Oracle Database 10g Release 2 Patch Set 3                            10.2.0.4.0
There are 2 products installed in this Oracle Home.

Interim patches (4) :

OneOffEntry::getBugIDsFixed()
OneOffEntry::getBugIDsFixed() returns 3 bugs.
Patch  9696904      : applied on Fri May 06 22:26:37 CST 2011
Unique Patch ID:  12575150
   Created on 10 May 2010, 23:22:02 hrs PST8PDT
   Bugs fixed:
     7025450, 8575528, 6904068
   This patch overlays patches:
     9352164
   This patch needs patches:
     9352164
   as prerequisites

OneOffEntry::getBugIDsFixed()
OneOffEntry::getBugIDsFixed() returns 1 bugs.
Patch  7527908      : applied on Fri May 06 22:25:33 CST 2011
Unique Patch ID:  12546933
   Created on 30 Apr 2010, 12:48:09 hrs PST8PDT
   Bugs fixed:
     7527908
   This patch overlays patches:
     9352164
   This patch needs patches:
     9352164
   as prerequisites

OneOffEntry::getBugIDsFixed()
OneOffEntry::getBugIDsFixed() returns 10 bugs.
Patch  9654991      : applied on Fri May 06 21:33:38 CST 2011
Unique Patch ID:  12816418
   Created on 30 Jun 2010, 18:30:40 hrs PST8PDT
   Bugs fixed:
     9714832, 9713537, 9678695, 9655017, 9173248, 8309642, 9678697, 9678690
     9573054, 9654991
   This patch overlays patches:
     9352164
   This patch needs patches:
     9352164
   as prerequisites

OneOffEntry::getBugIDsFixed()
OneOffEntry::getBugIDsFixed() returns 414 bugs.
Patch  9352164      : applied on Fri May 06 21:29:20 CST 2011
Unique Patch ID:  12307477
   Created on 1 Apr 2010, 11:33:14 hrs PST8PDT
   Bugs fixed:
     9377578, 6418420, 7835247, 7207654, 7592346, 6724797, 7936993, 7331867
     9093300, 7535152, 7552067, 5879114, 8866013, 5457450, 8344348, 7272297
     7136866, 7196894, 6258480, 7013124, 6512622, 6355586, 7196532, 8568395
     8309587, 7557226, 5399699, 6509115, 8568397, 8568398, 7127618, 5701695
     6052226, 7424804, 6817593, 7553884, 6741425, 7513673, 6783812, 8437213
     6452766, 6469211, 7527650, 8309592, 8479537, 5991038, 5686407, 6945157
     7639602, 9119226, 6403091, 7589862, 7552082, 6711853, 8304589, 6052169
     8199266, 6327692, 5756769, 9352191, 7460818, 6268409, 8232056, 6687381
     6972843, 8230457, 6800507, 7027551, 6778714, 6200820, 6645719, 7393804
     6775231, 3934160, 6683178, 6650256, 5085288, 7528105, 7329252, 6378112
     6151380, 6844866, 4723109, 8544896, 5126719, 5890312, 5934363, 7036453
     7610362, 8426816, 8433026, 7270434, 7172531, 6451626, 8247855, 5497338
     6324944, 6874522, 7175513, 6960489, 7341598, 8576156, 6797677, 8342923
     5895190, 6756739, 7150470, 7593835, 7356443, 7044551, 8227106, 4695511
     7298688, 5747462, 8556340, 7197445, 5348308, 7937113, 8341623, 7569205
     8409848, 6053134, 6163771, 6851669, 6181488, 6375150, 6825866, 7210921
     7295780, 6345573, 7033630, 6954722, 6824129, 7523475, 7457766, 7309458
     8324577, 6840740, 6804746, 7375611, 8268054, 6981690, 6512811, 6988017
     7375613, 8344399, 7340448, 8362683, 7375617, 8251247, 5933656, 6005347
     9145204, 6599920, 7238230, 6379441, 6452375, 6352003, 6833965, 7136489
     6610218, 7612639, 6392076, 7225204, 9119194, 5476236, 9442328, 7609057
     7609058, 6605106, 6374297, 6193945, 4693355, 8217795, 7039896, 7432514
     7330909, 6952701, 7190270, 8287155, 7587008, 7207932, 6802650, 7189447
     8481935, 4598439, 9442331, 6615740, 7155655, 6749617, 9442335, 7159505
     5868257, 5727166, 7173005, 6917874, 9442339, 7013768, 7691766, 7385253
     7291739, 7225720, 7257770, 7363767, 7244238, 6941717, 8267348, 7710551
     8354686, 7247217, 8328954, 7299153, 8909984, 6681695, 8702276, 9119284
     8217011, 7661251, 6265559, 6823287, 6991626, 6954829, 5259835, 6500033
     5923486, 7432601, 7022234, 8534387, 5147386, 7697802, 6653934, 7375644
     6490140, 7662491, 8331466, 5623467, 6070225, 6635214, 7396409, 6638558
     7038750, 6714608, 6838714, 6870937, 7219752, 7263842, 7278117, 6882739
     5404871, 8836667, 8373286, 7393292, 6678845, 6903051, 7936793, 6600051
     7155248, 4966512, 7155249, 7197637, 8836308, 8568402, 8568404, 8568405
     8431487, 5704108, 6343150, 7280764, 6923450, 7643632, 6145177, 8836671
     8310931, 6640411, 8347704, 8836675, 7155250, 7155251, 8836677, 7155252
     8836678, 7155253, 8292378, 7155254, 6219529, 7411865, 8227091, 8340379
     7276960, 6145687, 7659217, 5863926, 7022905, 6852598, 7123643, 6596564
     6413089, 6851438, 8836681, 8836683, 8836684, 8836686, 7579469, 7494333
     7315642, 8340383, 6786022, 8340387, 6926448, 7600026, 7462072, 6679303
     8815639, 7197583, 7172752, 7326645, 7008262, 9173244, 9173248, 7573151
     8490879, 7477934, 6725634, 6733655, 6799205, 6980597, 7499353, 6084232
     6014513, 7140204, 7254987, 8833280, 6647480, 6120004, 7693128, 6760697
     6051177, 8247215, 6858062, 7189645, 6844739, 6768251, 7196863, 5630796
     7378661, 7378735, 5970301, 6705822, 8290506, 6658484, 7599944, 9173253
     8309623, 7125408, 7257461, 6987790, 7568556, 6919819, 8886674, 5883691
     6955744, 7801939, 6074620, 7149004, 6857917, 8283650, 6445948, 5929055
     6110752, 7552042, 8210889, 8287504, 6506617, 7306915, 6271590, 5386204
     6976005, 8330783, 7606362, 5377242, 7043989, 8309632, 7575925, 6870047
     8309637, 5902053, 8309639, 7028176, 6827260, 7588384, 4726401, 6720712
     5910650, 6752765, 6971433, 6024730, 8315482, 6628122, 8239142, 9352164
     5695562, 4637902, 7345904, 8309642, 6994160, 8556586, 6404447, 8220734
     6919764, 7597354, 7523787, 6029179, 5231155, 6455659

probeAndPrintRACInfo()
Rac::getInstance()
 initializing racType
Rac::getClusterNodes()
 calling OiiOracleHomeInfo::getNodeList()
 OiiiOracleHomeInfo::getNodeList() returned 0 elements.
RAC::getClusterNodes() returned a list of 0 elements.
 User did not use -no_inventory, so why clusterNodes are empty?
 clusterNodes is null or empty, set racType to NO_RAC
Not a RAC system or shouldn't be treated like one
Rac::shouldPropagate()
Rac::getInstance()
 racType has been initialized, just return the cached object
[Rac: [RacType: typeDesc= NO_RAC, detail= Not a RAC system or shouldn't be treated like one ], should propagate file= false, local node="", remote nodes=", racTypeDetail="Not a RAC system or shouldn't be treated like one", racDiagMsg="
OPatch detected non-cluster Oracle Home from the inventory and will patch the local system only.
 ]
probeAndPrintRACInfo() done
--------------------------------------------------------------------------------
Locker::release()
OUISessionManager::unRegister()
Un-Registering the caller : OPatch
LsInventory::getInstance() returns
Cleaning up the directory : "/s01/db_1/.patch_storage/patch_unzip"...

OPatch succeeded.

记以录之!

Pre-check while you are applying one-off patch

Oracle Database的补丁除去我们最常见的Patch set大补丁集(就是那种动辄1、2个G的补丁包)外,还包括Critical Patch Update(CPU)和Patch Set Update(PSU),one-off Patch(一次性补丁);在数据库使用过程中出现某些问题后(最常见的是出现ORA-00600、ORA-07445错误),通过DBA与MOS的协作将问题定位到某个大版本上的具体Bug,如果该Bug在该版本上现有one-off patch的话,我们可以通过one-off patch的实施来解决问题,one-off patch具有如下特点:

  1. one-off patch的patch包一般都很小,因为它是为了修正一个或多个bug所特制,它一般只包含必要的inventory xml信息,以及和bug相关的共享库文件(一般是.o或者.a文件)
  2. one-off patch绝大多数是随需随制,一般是Oracle内部人员发现Bug或者客户在生产过程中遇到Bug后,向Oracle development开发部门提出patch building的Request ,开发人员接到building或backport的要求后会为客户系统指定的数据库版本和操作系统平台特制一个补丁(如果还没有的话);举例来说可能一个bug在Linux平台的10.2.0.4版本中已有one-off patch,但客户在Solaris Sparc 平台上10.2.0.3上碰到同样的bug,那么这个时候就可以提交SR要求Oracle开发部门backport这个patch
  3. 绝大多数one-off patch只需要用Opatch工具apply即可,不需要升级/修正数据字典,这一点和Patch set/CPU/PSU是不同的
  4. one-off patch是针对某个特殊版本的,如10.2.0.4上的one-off patch只能针对10.2.0.4实施,而在10.2.0.4.8(10.2.0.4的PSU 8)上该one-off patch可能是一个conflicting Bug即有冲突的一次性bug补丁;针对这种情况Oracle一般会给出当前版本上的替代one-off patch,例如6996030″MERGE LABEL REQUEST ON TOP OF 10.2.0.4 FOR AUDIT TRAIL CLEAN-UP”是10.2.0.4上的一个one-off patch,但在10.2.0.4.4上6996030是一个conflicting bug存在冲突,这样我们就需要一个替代品、一个10.2.0.4.4上的Equivalent patch等价补丁,在我们的例子中这个patch是9650152 “MERGE REQUEST ON TOP OF 10.2.0.4.4 FOR BUGS 9589005 6805009 6991606”
  5. 实施one-off patch意味着你的数据库软件将与其他主流版本的数据库在行为方式上有所区别,另一方面你需要考虑到如果发现了新的Bug并考虑打上另一个one-off patch时可能存在Conflicting冲突。大多数情况下one-off patch可以迅速帮我们解决Bug,但反过来说实施任意一个one-off patch时我们可能冒着今后花费更多时间去维护数据库补丁的风险;就这一点来看,应用one-off patch有着它明显的缺点
  6. 在11g中提出了hot patch的概念,这里的patch就是指的one-off patch;就目前看来online-patching只是一个噱头,因为真正具备在线实施能力的patch少之又少,你很难从MOS上的patch专栏中找到一个可用的online-patch
  7. 一个我个人的观点:Oracle GCS会通过stack call、trace、dump等信息将问题定位到bug,最为常见的是ORA-00600、ORA-07445等内部错误,这些错误因为有着明确的error argument、stack call等信息可以让GCS精确地定位故障,而对于另一些问题例如process spin、异常等待事件等不产生明确argument、stack、dump的问题时,定位就要困难一些。我们经常可以看到一些bug note,在客户apply one-off patch并设置特定event后one-off patch并不起作用,导致这种情况的原因多种多样,一种可能是patch在开发过程中存在问题(补丁代码没有被正确触发),另一种则是bug没有被准确定位。

我们目前的问题在于虽然one-off patch存在这样那样的问题,但很多时候又不得不实施one-off patch(大多数原因是停机时间限制);以及在一个已经实施了诸多Patch set、PSU、CPU和one-off patch的环境中新增one-off patch,为了更好地完成这项工作,我们有必要在具体实施补丁前完成以下的预备工作:

1.在定位到具体Bug后我们可以获得一个Bug No#,使用该No#到My Oracle Support上的Patches & Updates专栏中搜索该Bug相关的patch信息,这里存在三种情况:

  • 最好的情况是指定的数据库版本和操作系统平台上存在相关的patch
  • 指定数据库版本上有patch,注意这里的指定数据库版本需要精确到具体的PSU,但没有指定操作系统平台上的,这个时候可以提交SR要求在指定平台上build一个
  • 指定数据库版本上没有patch,注意这里的指定数据库版本需要精确到具体的PSU,例如10.2.0.4上的patch是不能直接用在10.2.0.4.x上的;针对这种情况我们可以首先查询Metalink Note“Patch Set Updates – One-off Patch Conflict Resolution”,可以从这个Note中查到指定one-off patch是否被包含在某个psu中,或者在某个psu上有特定的等价补丁(Equivalent on top),一部分one-off patch在不同的psu之间还存在通用性,如10.2.0.4.4上的部分patch可以在10.2.0.4.5上实施,具体是否通用可以参考下文中冲突检查的方法来检验。如果都没有等价补丁也没有通用补丁那么需要提交SR让Oracle开发部门去merge一个

2.更复杂的情况是在现有版本上已经应用了某些特定的patch(psu、cpu、或者其他one-off patch),我们要保证需要新增的one-off patch之间不存在conflict关系,也不会和系统中已有的patch发生conflict,这时候我们可以使用opatch prereq命令来做冲突检查并得到相关建议,如:

[oracle@rh2 tmp]$ /s01/db_1/OPatch/opatch lsinventory
Invoking OPatch 11.2.0.1.3
Oracle Interim Patch Installer version 11.2.0.1.3
Copyright (c) 2010, Oracle Corporation.  All rights reserved.

Oracle Home       : /s01/db_1
Central Inventory : /s01/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.3
OUI version       : 10.2.0.4.0
OUI location      : /s01/db_1/oui
Log file location : /s01/db_1/cfgtoollogs/opatch/opatch2011-05-06_21-41-39PM.log

Patch history file: /s01/db_1/cfgtoollogs/opatch/opatch_history.txt

Lsinventory Output file location : /s01/db_1/cfgtoollogs/opatch/lsinv/lsinventory2011-05-06_21-41-39PM.txt

--------------------------------------------------------------------------------
Installed Top-level Products (2): 

Oracle Database 10g                                                  10.2.0.1.0
Oracle Database 10g Release 2 Patch Set 3                            10.2.0.4.0
There are 2 products installed in this Oracle Home.

Interim patches (2) :

Patch  9654991      : applied on Fri May 06 21:33:38 CST 2011
Unique Patch ID:  12816418
   Created on 30 Jun 2010, 18:30:40 hrs PST8PDT
   Bugs fixed:
     9714832, 9713537, 9678695, 9655017, 9173248, 8309642, 9678697, 9678690
     9573054, 9654991
   This patch overlays patches:
     9352164
   This patch needs patches:
     9352164
   as prerequisites

Patch  9352164      : applied on Fri May 06 21:29:20 CST 2011
Unique Patch ID:  12307477
   Created on 1 Apr 2010, 11:33:14 hrs PST8PDT
   Bugs fixed:

/* 以上示例环境中实施了10.2.0.4上的psu4(9352164)和psu5(9654991) */

/* 在示例环境中我们试图实施补丁7527908和9696904,这2个merge patch是针对psu4(10.2.0.4.4)版本的,
    但在psu5(10.2.0.4.5)上没有相应的替代补丁,我们需要检测这2个patch是否和psu5发生冲突  */

[oracle@rh2 ~]$ mkdir $ORACLE_HOME/patches

/* 在Oracle HOME下创建存放patch的目录 */

[oracle@rh2 ~]$ cd $ORACLE_HOME/patches

[oracle@rh2 patches]$ unzip /home/oracle/patch/p7527908_102044_Linux-x86-64.zip 

Archive:  /home/oracle/patch/p7527908_102044_Linux-x86-64.zip
   creating: 7527908/
   creating: 7527908/files/
   creating: 7527908/files/lib/
   creating: 7527908/files/lib/libserver10.a/
  inflating: 7527908/files/lib/libserver10.a/ktm.o
  inflating: 7527908/files/lib/libserver10.a/ktu.o
  inflating: 7527908/files/lib/libserver10.a/ktusm.o
   creating: 7527908/etc/
   creating: 7527908/etc/config/
  inflating: 7527908/etc/config/inventory
  inflating: 7527908/etc/config/actions
   creating: 7527908/etc/xml/
  inflating: 7527908/etc/xml/GenericActions.xml
  inflating: 7527908/etc/xml/ShiphomeDirectoryStructure.xml
  inflating: 7527908/README.txt      

[oracle@rh2 patches]$ unzip ~/patch/p9696904_102044_Linux-x86-64.zip
Archive:  /home/oracle/patch/p9696904_102044_Linux-x86-64.zip
   creating: 9696904/
   creating: 9696904/files/
   creating: 9696904/files/lib/
   creating: 9696904/files/lib/libserver10.a/
  inflating: 9696904/files/lib/libserver10.a/kks1.o
  inflating: 9696904/files/lib/libserver10.a/kksc.o
  inflating: 9696904/files/lib/libserver10.a/kksh.o
  inflating: 9696904/files/lib/libserver10.a/ksmp.o
  inflating: 9696904/files/lib/libserver10.a/kspt.o
   creating: 9696904/etc/
   creating: 9696904/etc/config/
  inflating: 9696904/etc/config/inventory
  inflating: 9696904/etc/config/actions
   creating: 9696904/etc/xml/
  inflating: 9696904/etc/xml/GenericActions.xml
  inflating: 9696904/etc/xml/ShiphomeDirectoryStructure.xml
  inflating: 9696904/README.txt

接下来我们首先检查以上2个patch之间是否存在conflict冲突,这里我们会用到opatch prereq CheckConflictAmongPatchesWithDetail命令:

[oracle@rh2 db_1]$ /s01/db_1/OPatch/opatch prereq CheckConflictAmongPatchesWithDetail -phBaseDir $ORACLE_HOME/patches

Invoking OPatch 11.2.0.1.3
Oracle Interim Patch Installer version 11.2.0.1.3
Copyright (c) 2010, Oracle Corporation.  All rights reserved.

PREREQ session
Oracle Home       : /s01/db_1
Central Inventory : /s01/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.3
OUI version       : 10.2.0.4.0
OUI location      : /s01/db_1/oui
Log file location : /s01/db_1/cfgtoollogs/opatch/opatch2011-05-06_22-07-18PM.log

Patch history file: /s01/db_1/cfgtoollogs/opatch/opatch_history.txt
Invoking prereq "checkconflictamongpatcheswithdetail"

Prereq "checkConflictAmongPatchesWithDetail" passed.

OPatch succeeded.

/* 可以看到以上补丁间冲突检查发现需要实施的补丁间不存在冲突 */

接着我们需要为将要实施的补丁与系统中现有的补丁检查是否存在冲突,这里我们要用到opatch prereq CheckConflictAgainstOHWithDetail命令,如:

[oracle@rh2 db_1]$ /s01/db_1/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /s01/db_1/patches/9696904
Invoking OPatch 11.2.0.1.3

Oracle Interim Patch Installer version 11.2.0.1.3
Copyright (c) 2010, Oracle Corporation.  All rights reserved.

PREREQ session

Oracle Home       : /s01/db_1
Central Inventory : /s01/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.3
OUI version       : 10.2.0.4.0
OUI location      : /s01/db_1/oui
Log file location : /s01/db_1/cfgtoollogs/opatch/opatch2011-05-06_22-08-44PM.log

Patch history file: /s01/db_1/cfgtoollogs/opatch/opatch_history.txt

Invoking prereq "checkconflictagainstohwithdetail"

Prereq "checkConflictAgainstOHWithDetail" passed.

OPatch succeeded.

[oracle@rh2 db_1]$ /s01/db_1/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /s01/db_1/patches/7527908
Invoking OPatch 11.2.0.1.3

Oracle Interim Patch Installer version 11.2.0.1.3
Copyright (c) 2010, Oracle Corporation.  All rights reserved.

PREREQ session

Oracle Home       : /s01/db_1
Central Inventory : /s01/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.3
OUI version       : 10.2.0.4.0
OUI location      : /s01/db_1/oui
Log file location : /s01/db_1/cfgtoollogs/opatch/opatch2011-05-06_22-08-57PM.log

Patch history file: /s01/db_1/cfgtoollogs/opatch/opatch_history.txt

Invoking prereq "checkconflictagainstohwithdetail"

Prereq "checkConflictAgainstOHWithDetail" passed.

OPatch succeeded.

/* 在这里另外举一个检查后发现冲突的例子 */

[oracle@rh2 tmp]$ /s01/db_1/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir 8557428
Invoking OPatch 11.2.0.1.3

Oracle Interim Patch Installer version 11.2.0.1.3
Copyright (c) 2010, Oracle Corporation.  All rights reserved.

PREREQ session

Oracle Home       : /s01/db_1
Central Inventory : /s01/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.3
OUI version       : 10.2.0.4.0
OUI location      : /s01/db_1/oui
Log file location : /s01/db_1/cfgtoollogs/opatch/opatch2011-05-06_22-38-25PM.log

Patch history file: /s01/db_1/cfgtoollogs/opatch/opatch_history.txt

Invoking prereq "checkconflictagainstohwithdetail"

ZOP-47: The patch(es) has supersets with other patches installed in the Oracle Home (or) among themselves.

ZOP-40: The patch(es) has conflicts with other patches installed in the Oracle Home (or) among themselves.

Prereq "checkConflictAgainstOHWithDetail" failed.

Summary of Conflict Analysis:

There are no patches that can be applied now.

Following patches have conflicts. Please contact Oracle Support and get the merged patch of the patches :
9352164

Following patches are not required, as they are subset of the patches in Oracle Home or
subset of the patches in the given list : 

Following patches will be rolled back from Oracle Home on application of the patches in the given list :
9352164
Conflicts/Supersets for each patch are:

Patch : 8557428
        Bug Conflict with 9352164
        Conflicting bugs are:
        8479537,  7462072,  8210889,  6404447

OPatch succeeded.

/* 可以看到以上发现了8557428 one-off patch与9352164(PSU4)之间存在冲突,
    引发冲突的Bug是8479537,  7462072,  8210889,  6404447
    opatch建议用户提交合并8557428和9352164补丁的服务要求  */

通过以上检查可以发现虽然是针对psu4(10.2.0.4.4)发布的one-off patch,但在对psu5(10.2.0.4.5)的ORACLE HOME做checkConflict时未发现冲突,接着我们opatch apply -report命令来预演补丁实施,使用-report选项让opatch仅报告其所会做的操作,而不真正去实施这些操作:

[oracle@rh2 db_1]$ cd patches/7527908/
[oracle@rh2 7527908]$ ls
etc  files  README.txt
[oracle@rh2 7527908]$ /s01/db_1/OPatch/opatch apply -report
Invoking OPatch 11.2.0.1.3

Oracle Interim Patch Installer version 11.2.0.1.3
Copyright (c) 2010, Oracle Corporation.  All rights reserved.

Oracle Home       : /s01/db_1
Central Inventory : /s01/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.3
OUI version       : 10.2.0.4.0
OUI location      : /s01/db_1/oui
Log file location : /s01/db_1/cfgtoollogs/opatch/opatch2011-05-06_22-19-25PM.log

Patch history file: /s01/db_1/cfgtoollogs/opatch/opatch_history.txt

ApplySession applying interim patch '7527908' to OH '/s01/db_1'

Running prerequisite checks...
[Report: skip "rm -rf /s01/db_1/ccr"]
[Report: skip unzipping "/s01/db_1/OPatch/ocm/ocm.zip"]
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name: 

You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]:  Yes
[Report: Skip calling "/bin/sh /s01/db_1/ccr/bin/setupCCR -R
/s01/db_1/.patch_storage/ocmRespFile -S OPatch -V 11.2.0.1.3" under "/s01/db_1/ccr/bin"]

OPatch detected non-cluster Oracle Home from the inventory and will patch the local system only.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/s01/db_1')

Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files and inventory (not for auto-rollback) for the Oracle Home
Users request no RAC file generation.  Do not create MP files.

Skip patching component oracle.rdbms, 10.2.0.4.0 and its actions.
The actions are reported here, but are not performed.

Updating archive file "/s01/db_1/lib/libserver10.a"  with "lib/libserver10.a/ktm.o"
Updating archive file "/s01/db_1/lib/libserver10.a"  with "lib/libserver10.a/ktu.o"
Updating archive file "/s01/db_1/lib/libserver10.a"  with "lib/libserver10.a/ktusm.o"
Running make for target ioracle
ApplySession skipping inventory update.

Verifying the update...
Inventory and System verification is performed here.

The local system has been patched and can be restarted.

OPatch succeeded.

[oracle@rh2 patches]$ cd 9696904/
[oracle@rh2 9696904]$ /s01/db_1/OPatch/opatch apply -report
Invoking OPatch 11.2.0.1.3

Oracle Interim Patch Installer version 11.2.0.1.3
Copyright (c) 2010, Oracle Corporation.  All rights reserved.

Oracle Home       : /s01/db_1
Central Inventory : /s01/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.3
OUI version       : 10.2.0.4.0
OUI location      : /s01/db_1/oui
Log file location : /s01/db_1/cfgtoollogs/opatch/opatch2011-05-06_22-22-11PM.log

Patch history file: /s01/db_1/cfgtoollogs/opatch/opatch_history.txt

ApplySession applying interim patch '9696904' to OH '/s01/db_1'

Running prerequisite checks...
[Report: skip "rm -rf /s01/db_1/ccr"]
[Report: skip unzipping "/s01/db_1/OPatch/ocm/ocm.zip"]
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name: 

You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]:  Yes
[Report: Skip calling "/bin/sh /s01/db_1/ccr/bin/setupCCR -R /s01/db_1/.patch_storage/ocmRespFile -S 
OPatch -V 11.2.0.1.3" under "/s01/db_1/ccr/bin"]

OPatch detected non-cluster Oracle Home from the inventory and will patch the local system only.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/s01/db_1')

Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files and inventory (not for auto-rollback) for the Oracle Home
Users request no RAC file generation.  Do not create MP files.

Skip patching component oracle.rdbms, 10.2.0.4.0 and its actions.
The actions are reported here, but are not performed.

Updating archive file "/s01/db_1/lib/libserver10.a"  with "lib/libserver10.a/kks1.o"
Updating archive file "/s01/db_1/lib/libserver10.a"  with "lib/libserver10.a/kksc.o"
Updating archive file "/s01/db_1/lib/libserver10.a"  with "lib/libserver10.a/kksh.o"
Updating archive file "/s01/db_1/lib/libserver10.a"  with "lib/libserver10.a/ksmp.o"
Updating archive file "/s01/db_1/lib/libserver10.a"  with "lib/libserver10.a/kspt.o"
Running make for target ioracle
ApplySession skipping inventory update.

Verifying the update...
Inventory and System verification is performed here.

The local system has been patched and can be restarted.

OPatch succeeded.

以上使用opatch apply -report进一步确认了实施以上补丁不会引起冲突或其他失败,我们可以放心地正式应用补丁了!

[oracle@rh2 patches]$ cd 7527908/
[oracle@rh2 7527908]$ /s01/db_1/OPatch/opatch apply
Invoking OPatch 11.2.0.1.3

Oracle Interim Patch Installer version 11.2.0.1.3
Copyright (c) 2010, Oracle Corporation.  All rights reserved.

Oracle Home       : /s01/db_1
Central Inventory : /s01/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.3
OUI version       : 10.2.0.4.0
OUI location      : /s01/db_1/oui
Log file location : /s01/db_1/cfgtoollogs/opatch/opatch2011-05-06_22-24-46PM.log

Patch history file: /s01/db_1/cfgtoollogs/opatch/opatch_history.txt

ApplySession applying interim patch '7527908' to OH '/s01/db_1'

Running prerequisite checks...
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name: 

You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]:  Yes

OPatch detected non-cluster Oracle Home from the inventory and will patch the local system only.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/s01/db_1')

Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files and inventory (not for auto-rollback) for the Oracle Home
Backing up files affected by the patch '7527908' for restore. This might take a while...
Backing up files affected by the patch '7527908' for rollback. This might take a while...

Patching component oracle.rdbms, 10.2.0.4.0...
Updating archive file "/s01/db_1/lib/libserver10.a"  with "lib/libserver10.a/ktm.o"
Updating archive file "/s01/db_1/lib/libserver10.a"  with "lib/libserver10.a/ktu.o"
Updating archive file "/s01/db_1/lib/libserver10.a"  with "lib/libserver10.a/ktusm.o"
Running make for target ioracle
ApplySession adding interim patch '7527908' to inventory

Verifying the update...
Inventory check OK: Patch ID 7527908 is registered in Oracle Home inventory with proper meta-data.
Files check OK: Files from Patch ID 7527908 are present in Oracle Home.

The local system has been patched and can be restarted.

OPatch succeeded.

[oracle@rh2 7527908]$ cd ..
[oracle@rh2 patches]$ cd 9696904/
[oracle@rh2 9696904]$ /s01/db_1/OPatch/opatch apply
Invoking OPatch 11.2.0.1.3

Oracle Interim Patch Installer version 11.2.0.1.3
Copyright (c) 2010, Oracle Corporation.  All rights reserved.

Oracle Home       : /s01/db_1
Central Inventory : /s01/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.3
OUI version       : 10.2.0.4.0
OUI location      : /s01/db_1/oui
Log file location : /s01/db_1/cfgtoollogs/opatch/opatch2011-05-06_22-25-58PM.log

Patch history file: /s01/db_1/cfgtoollogs/opatch/opatch_history.txt

ApplySession applying interim patch '9696904' to OH '/s01/db_1'

Running prerequisite checks...
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name: 

You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]:  Yes

OPatch detected non-cluster Oracle Home from the inventory and will patch the local system only.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/s01/db_1')

Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files and inventory (not for auto-rollback) for the Oracle Home
Backing up files affected by the patch '9696904' for restore. This might take a while...
Backing up files affected by the patch '9696904' for rollback. This might take a while...

Patching component oracle.rdbms, 10.2.0.4.0...
Updating archive file "/s01/db_1/lib/libserver10.a"  with "lib/libserver10.a/kks1.o"
Updating archive file "/s01/db_1/lib/libserver10.a"  with "lib/libserver10.a/kksc.o"
Updating archive file "/s01/db_1/lib/libserver10.a"  with "lib/libserver10.a/kksh.o"
Updating archive file "/s01/db_1/lib/libserver10.a"  with "lib/libserver10.a/ksmp.o"
Updating archive file "/s01/db_1/lib/libserver10.a"  with "lib/libserver10.a/kspt.o"
Running make for target ioracle
ApplySession adding interim patch '9696904' to inventory

Verifying the update...
Inventory check OK: Patch ID 9696904 is registered in Oracle Home inventory with proper meta-data.
Files check OK: Files from Patch ID 9696904 are present in Oracle Home.

The local system has been patched and can be restarted.

OPatch succeeded.

[oracle@rh2 9696904]$ /s01/db_1/OPatch/opatch lsinventory
Invoking OPatch 11.2.0.1.3

Oracle Interim Patch Installer version 11.2.0.1.3
Copyright (c) 2010, Oracle Corporation.  All rights reserved.

Oracle Home       : /s01/db_1
Central Inventory : /s01/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.3
OUI version       : 10.2.0.4.0
OUI location      : /s01/db_1/oui
Log file location : /s01/db_1/cfgtoollogs/opatch/opatch2011-05-06_22-28-56PM.log

Patch history file: /s01/db_1/cfgtoollogs/opatch/opatch_history.txt

Lsinventory Output file location : /s01/db_1/cfgtoollogs/opatch/lsinv/lsinventory2011-05-06_22-28-56PM.txt

--------------------------------------------------------------------------------
Installed Top-level Products (2): 

Oracle Database 10g                                                  10.2.0.1.0
Oracle Database 10g Release 2 Patch Set 3                            10.2.0.4.0
There are 2 products installed in this Oracle Home.

Interim patches (4) :

Patch  9696904      : applied on Fri May 06 22:26:37 CST 2011
Unique Patch ID:  12575150
   Created on 10 May 2010, 23:22:02 hrs PST8PDT
   Bugs fixed:
     7025450, 8575528, 6904068
   This patch overlays patches:
     9352164
   This patch needs patches:
     9352164
   as prerequisites

Patch  7527908      : applied on Fri May 06 22:25:33 CST 2011
Unique Patch ID:  12546933
   Created on 30 Apr 2010, 12:48:09 hrs PST8PDT
   Bugs fixed:
     7527908
   This patch overlays patches:
     9352164
   This patch needs patches:
     9352164
   as prerequisites

Patch  9654991      : applied on Fri May 06 21:33:38 CST 2011
Unique Patch ID:  12816418
   Created on 30 Jun 2010, 18:30:40 hrs PST8PDT
   Bugs fixed:
     9714832, 9713537, 9678695, 9655017, 9173248, 8309642, 9678697, 9678690
     9573054, 9654991
   This patch overlays patches:
     9352164
   This patch needs patches:
     9352164
   as prerequisites

Patch  9352164      : applied on Fri May 06 21:29:20 CST 2011
Unique Patch ID:  12307477
   Created on 1 Apr 2010, 11:33:14 hrs PST8PDT
   Bugs fixed:
.............................................................

/* Applied one-off patch successfully, That's great! */

Oracle Patch Set Update and Critical Patch Update April 2011 Released

2011 April的Oracle Patch set Update与Critical Patch Update发布了,本次发布包括了对Oracle Database Server, Oracle Fusion Middleware, Oracle Enterprise Manager Grid Control, Oracle E-Business Suite and Supply Chain Products Suite, Oracle PeopleSoft Enterprise, and Oracle JDEdwards EntepriseOne, Oracle Siebel CRM, Oracle Industry Applications, Oracle Sun products suite, and Oracle OpenOffice Suite等多个产品的补丁更新。

其中与数据库相关的主要有补丁集更新和紧急补丁更新如下:

  1. Database 11.2.0.2上的CPU Patch 11724984, or PSU Patch 11724916
  2. Database 11.2.0.1上的CPU Patch 11724991, or PSU Patch 11724930
  3. Database 11.1.0.7上的CPU Patch 11724999, or PSU Patch 11724936
  4. Database 10.2.0.5上的CPU Patch 11725006, or PSU Patch 11724962
  5. Database 10.2.0.4上的CPU Patch 11725015, or PSU Patch 11724977

opatch java.lang.OutOfMemoryError:Java heap space错误一例

一套11.1.0.6的RAC系统在在使用opatch工具时出现了OutOfMemoryError: Java heap space错误,其错误日志如下:

opatch lsinventory
Invoking OPatch 11.1.0.6.2

Oracle Interim Patch Installer version 11.1.0.6.2
Copyright (c) 2007, Oracle Corporation.  All rights reserved.

Oracle Home       : /u01/oracle/db/11.1.0
Central Inventory : /u01/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.1.0.6.2
OUI version       : 11.1.0.7.0
OUI location      : /u01/oracle/db/11.1.0/oui
Log file location : /u01/oracle/db/11.1.0/cfgtoollogs/opatch/opatch2011-03-11_03-05-22AM.log

Lsinventory Output file location : /u01/oracle/db/11.1.0/cfgtoollogs/opatch/lsinv/lsinventory2011-03-11_03-05-22AM.txt

--------------------------------------------------------------------------------
Installed Top-level Products (2):

Oracle Database 11g                                                  11.1.0.6.0
Oracle Database 11g Patch Set 1                                      11.1.0.7.0

Interim patches (3) :

Patch  8496830      : applied on Wed Mar 30 07:11:26 CDT 2011
   Created on 27 Aug 2009, 01:59:52 hrs PST8PDT
   Bugs fixed:
     8496830

Patch  7327166      : applied on Wed Oct 20 11:25:00 CDT 2010
   Created on 31 Dec 2008, 10:49:56 hrs PST8PDT
   Bugs fixed:
     7327166

Patch  7253531      : applied on Wed Jul 28 14:45:44 CDT 2010
   Created on 24 Nov 2008, 03:52:49 hrs PST8PDT
   Bugs fixed:
     7253531
Exception in thread "Thread-7" Exception in thread "Thread-3" java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space

事后发现是由于opatch期间1号节点无法使用ssh服务登陆到其他节点上引起的,修复该故障后问题消失了。

另外提一句我们建议使用最新版本的opatch工具来管理数据库补丁,最新版本的opatch工具可以从<OPatch – Where Can I Find the Latest Version of OPatch?>下载到!

Applying online patch on 11gr2

在Oracle 11g中提出了online patch(也叫hot patch)的特性;Hot patching允许我们在实例始终在线的情况下安装,启用或禁用一个修复补丁或者诊断补丁。针对7*24在线的业务系统,hot patch为我们提供了一条既能避免当机时间而又可以实施补丁的途径。在Oracle 11g中我们可以使用Opatch命令行工具针对任意数据库实施在线补丁(前提是该补丁是一个hot patch)。一般来说在线补丁(hot patches)只能是那些代码修改范围小且复杂度很低的补丁,举例来说它们往往是一些诊断补丁(diagnostic patches)或者小bug的修复(small bug fixes)。值得注意的是hot patching将需要消耗额外的内存,决定其消耗内存数量的因素是:1.补丁本身的大小,2.实例中的进程总数;举例来说某个补丁的大小正好为一个OS page的大小(一般为4kB),那么当实例中运行的进程总数为1000时,则该hot patching所额外消耗的内存总数为4kB*1000=4MB。

hot patches与常规Conventional patches对比具有可在线实施和安装快的特性,如下图:
online patching

在实际生产环境中,相信没有多少朋友实施过hot patching,一来国内目前还没有普及11g的使用,二来hot patching的数量在所有interim patch中只占极少数;一直以来都想写这样一篇关于hot patching的博文,唯一妨碍我写作的问题是在11.2.0.1下找不到可实施的online interim patch;以MOS->patches&upgrade目前的分类我们很难找出某个base release下可用的hot patch,当然这并不妨碍补丁专栏的使用。为了这个令人郁闷的问题,我特意去提交了一个Service Request,得到的回复:

I have tried to find the patches which support online patching on 11.2.0.1 version,
but I also can not find them because there are too many patches and there is no catalog for the patches
which support online patching, and I can only check the patch readme to confirm whether that patch supports online patching.

I found one patch which supports online patching, but this patch is for 11.2.0.2 version.
The patch no. is 10188727.

Sorry for the inconvenience brought to you. Hope the above update can help you.
If the above patch is not what you want, then please update the SR and I will continue for your issue.

这其中提到的patch 10188727,可以从Note<RDBMS Online Patching Aka Hot Patching [ID 761111.1]>中找到,另外一个可找到的hot patch是11.1.0.6上的6198642<DUMMY PATCH FOR TESTING DB11 PATCHING>,不过很可惜该补丁只有Linux x86一个平台版本的。所以我不得不先将11.2.0.1的测试库升级到了11.2.0.2上:

SQL> select * from v$version;

BANNER
--------------------------------------------------------------------------------
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
PL/SQL Release 11.2.0.2.0 - Production
CORE    11.2.0.2.0      Production
TNS for Linux: Version 11.2.0.2.0 - Production
NLSRTL Version 11.2.0.2.0 - Production

[maclean@rh2 OPatch]$ ps -ef|grep pmon|grep -v grep
maclean  22481     1  0 19:19 ?        00:00:00 ora_pmon_PROD

[maclean@rh2 OPatch]$ pmap -d 22481
22481:   ora_pmon_PROD
Address           Kbytes Mode  Offset           Device    Mapping
0000000000400000  180232 r-x-- 0000000000000000 008:00002 oracle
000000000b602000    1820 rwx-- 000000000b002000 008:00002 oracle
000000000b7c9000     300 rwx-- 000000000b7c9000 000:00000   [ anon ]
000000000dbef000     436 rwx-- 000000000dbef000 000:00000   [ anon ]
0000000060000000 2050048 rwxs- 0000000000000000 000:00009   [ shmid=0x550001 ]
0000003e09a00000     112 r-x-- 0000000000000000 008:00001 ld-2.5.so
0000003e09c1b000       4 r-x-- 000000000001b000 008:00001 ld-2.5.so
0000003e09c1c000       4 rwx-- 000000000001c000 008:00001 ld-2.5.so
0000003e09e00000    1336 r-x-- 0000000000000000 008:00001 libc-2.5.so
0000003e09f4e000    2044 ----- 000000000014e000 008:00001 libc-2.5.so
0000003e0a14d000      16 r-x-- 000000000014d000 008:00001 libc-2.5.so
0000003e0a151000       4 rwx-- 0000000000151000 008:00001 libc-2.5.so
0000003e0a152000      20 rwx-- 0000003e0a152000 000:00000   [ anon ]
0000003e0a200000     520 r-x-- 0000000000000000 008:00001 libm-2.5.so
0000003e0a282000    2044 ----- 0000000000082000 008:00001 libm-2.5.so
0000003e0a481000       4 r-x-- 0000000000081000 008:00001 libm-2.5.so
0000003e0a482000       4 rwx-- 0000000000082000 008:00001 libm-2.5.so
0000003e0a600000       8 r-x-- 0000000000000000 008:00001 libdl-2.5.so
0000003e0a602000    2048 ----- 0000000000002000 008:00001 libdl-2.5.so
0000003e0a802000       4 r-x-- 0000000000002000 008:00001 libdl-2.5.so
0000003e0a803000       4 rwx-- 0000000000003000 008:00001 libdl-2.5.so
0000003e0aa00000      88 r-x-- 0000000000000000 008:00001 libpthread-2.5.so
0000003e0aa16000    2044 ----- 0000000000016000 008:00001 libpthread-2.5.so
0000003e0ac15000       4 r-x-- 0000000000015000 008:00001 libpthread-2.5.so
0000003e0ac16000       4 rwx-- 0000000000016000 008:00001 libpthread-2.5.so
0000003e0ac17000      16 rwx-- 0000003e0ac17000 000:00000   [ anon ]
0000003e0ae00000      28 r-x-- 0000000000000000 008:00001 librt-2.5.so
0000003e0ae07000    2048 ----- 0000000000007000 008:00001 librt-2.5.so
0000003e0b007000       4 r-x-- 0000000000007000 008:00001 librt-2.5.so
0000003e0b008000       4 rwx-- 0000000000008000 008:00001 librt-2.5.so
0000003e0da00000      84 r-x-- 0000000000000000 008:00001 libnsl-2.5.so
0000003e0da15000    2044 ----- 0000000000015000 008:00001 libnsl-2.5.so
0000003e0dc14000       4 r-x-- 0000000000014000 008:00001 libnsl-2.5.so
0000003e0dc15000       4 rwx-- 0000000000015000 008:00001 libnsl-2.5.so
0000003e0dc16000       8 rwx-- 0000003e0dc16000 000:00000   [ anon ]
00002abec920e000       8 rwx-- 00002abec920e000 000:00000   [ anon ]
00002abec9210000       4 r-x-- 0000000000000000 008:00002 libodmd11.so
00002abec9211000    1024 ----- 0000000000001000 008:00002 libodmd11.so
00002abec9311000       4 rwx-- 0000000000001000 008:00002 libodmd11.so
00002abec9312000     360 r-x-- 0000000000000000 008:00002 libcell11.so
00002abec936c000    1020 ----- 000000000005a000 008:00002 libcell11.so
00002abec946b000      36 rwx-- 0000000000059000 008:00002 libcell11.so
00002abec9474000       4 rwx-- 00002abec9474000 000:00000   [ anon ]
00002abec9475000     848 r-x-- 0000000000000000 008:00002 libskgxp11.so
00002abec9549000    1024 ----- 00000000000d4000 008:00002 libskgxp11.so
00002abec9649000       8 rwx-- 00000000000d4000 008:00002 libskgxp11.so
00002abec9665000       4 rwx-- 00002abec9665000 000:00000   [ anon ]
00002abec9666000    2580 r-x-- 0000000000000000 008:00002 libnnz11.so
00002abec98eb000    1020 ----- 0000000000285000 008:00002 libnnz11.so
00002abec99ea000     264 rwx-- 0000000000284000 008:00002 libnnz11.so
00002abec9a2c000       8 rwx-- 00002abec9a2c000 000:00000   [ anon ]
00002abec9a2e000      96 r-x-- 0000000000000000 008:00002 libclsra11.so
00002abec9a46000    1020 ----- 0000000000018000 008:00002 libclsra11.so
00002abec9b45000       4 rwx-- 0000000000017000 008:00002 libclsra11.so
00002abec9b46000       4 rwx-- 00002abec9b46000 000:00000   [ anon ]
00002abec9b47000     136 r-x-- 0000000000000000 008:00002 libdbcfg11.so
00002abec9b69000    1020 ----- 0000000000022000 008:00002 libdbcfg11.so
00002abec9c68000       8 rwx-- 0000000000021000 008:00002 libdbcfg11.so
00002abec9c6a000    6832 r-x-- 0000000000000000 008:00002 libhasgen11.so
00002abeca316000    1020 ----- 00000000006ac000 008:00002 libhasgen11.so
00002abeca415000     136 rwx-- 00000000006ab000 008:00002 libhasgen11.so
00002abeca437000      24 rwx-- 00002abeca437000 000:00000   [ anon ]
00002abeca43d000       8 r-x-- 0000000000000000 008:00002 libskgxn2.so
00002abeca43f000    1020 ----- 0000000000002000 008:00002 libskgxn2.so
00002abeca53e000       4 rwx-- 0000000000001000 008:00002 libskgxn2.so
00002abeca53f000       4 rwx-- 00002abeca53f000 000:00000   [ anon ]
00002abeca540000     656 r-x-- 0000000000000000 008:00002 libocr11.so
00002abeca5e4000    1020 ----- 00000000000a4000 008:00002 libocr11.so
00002abeca6e3000      12 rwx-- 00000000000a3000 008:00002 libocr11.so
00002abeca6e6000     628 r-x-- 0000000000000000 008:00002 libocrb11.so
00002abeca783000    1024 ----- 000000000009d000 008:00002 libocrb11.so
00002abeca883000       8 rwx-- 000000000009d000 008:00002 libocrb11.so
00002abeca885000      44 r-x-- 0000000000000000 008:00002 libocrutl11.so
00002abeca890000    1020 ----- 000000000000b000 008:00002 libocrutl11.so
00002abeca98f000       4 rwx-- 000000000000a000 008:00002 libocrutl11.so
00002abeca990000       4 rwx-- 00002abeca990000 000:00000   [ anon ]
00002abeca991000       4 r-x-- 0000000000000000 008:00001 libaio.so.1.0.1
00002abeca992000    2044 ----- 0000000000001000 008:00001 libaio.so.1.0.1
00002abecab91000       4 rwx-- 0000000000000000 008:00001 libaio.so.1.0.1
00002abecab92000      16 rwx-- 00002abecab92000 000:00000   [ anon ]
00002abecab96000      20 r-x-- 0000000000000000 008:00001 libnuma.so.1
00002abecab9b000    2044 ----- 0000000000005000 008:00001 libnuma.so.1
00002abecad9a000       4 rwx-- 0000000000004000 008:00001 libnuma.so.1
00002abecad9b000    1280 rwx-- 00002abecad9b000 000:00000   [ anon ]
00002abecaef5000      40 r-x-- 0000000000000000 008:00001 libnss_files-2.5.so
00002abecaeff000    2044 ----- 000000000000a000 008:00001 libnss_files-2.5.so
00002abecb0fe000       4 r-x-- 0000000000009000 008:00001 libnss_files-2.5.so
00002abecb0ff000       4 rwx-- 000000000000a000 008:00001 libnss_files-2.5.so
00002abecb100000    1700 rwx-- 00002abecb100000 000:00000   [ anon ]
00002abecb2a9000      28 rwx-- 0000000000000000 000:00011 zero
00002abecb2b0000      64 rwx-- 0000000000000000 000:00011 zero
00002abecb2c0000      64 rwx-- 0000000000000000 000:00011 zero
00002abecb2d0000      64 rwx-- 0000000000000000 000:00011 zero
00002abecb2e0000      64 rwx-- 0000000000000000 000:00011 zero
00002abecb2f0000      64 rwx-- 0000000000000000 000:00011 zero
00002abecb300000     164 rwx-- 0000000000057000 000:00011 zero
00002abecb329000       8 rwx-- 00002abecb329000 000:00000   [ anon ]
00002abecb32b000       4 rwxs- 0000000000000000 008:00002 hc_PROD.dat
00002abecb32c000      40 r-x-- 0000000000000000 008:00002 libnque11.so
00002abecb336000    1020 ----- 000000000000a000 008:00002 libnque11.so
00002abecb435000       4 rwx-- 0000000000009000 008:00002 libnque11.so
00002abecb436000    1048 rwx-- 00002abecb436000 000:00000   [ anon ]
00007fff3342d000      84 rwx-- 00007ffffffea000 000:00000   [ stack ]
ffffffffff600000    8192 ----- 0000000000000000 000:00000   [ anon ]
mapped: 2291488K    writeable/private: 7840K    shared: 2050052K

[maclean@rh2 ~]$  cd $ORACLE_HOME/OPatch
[maclean@rh2 OPatch]$ unzip p10188727_112020_Linux-x86-64.zip
Archive:  p10188727_112020_Linux-x86-64.zip
   creating: 10188727/
   creating: 10188727/files/
   creating: 10188727/files/lib/
   creating: 10188727/files/lib/libserver11.a/
  inflating: 10188727/files/lib/libserver11.a/kkopq.o
   creating: 10188727/etc/
   creating: 10188727/etc/config/
  inflating: 10188727/etc/config/inventory.xml
  inflating: 10188727/etc/config/actions.xml
  inflating: 10188727/etc/config/deploy.xml
   creating: 10188727/etc/xml/
  inflating: 10188727/etc/xml/GenericActions.xml
  inflating: 10188727/etc/xml/ShiphomeDirectoryStructure.xml
  inflating: 10188727/README.txt
   creating: 10188727/online/
   creating: 10188727/online/files/
   creating: 10188727/online/files/hpatch/
  inflating: 10188727/online/files/hpatch/bug10188727.pch
   creating: 10188727/online/etc/
   creating: 10188727/online/etc/config/
  inflating: 10188727/online/etc/config/inventory.xml
  inflating: 10188727/online/etc/config/actions.xml
  inflating: 10188727/online/etc/config/deploy.xml
   creating: 10188727/online/etc/xml/
  inflating: 10188727/online/etc/xml/GenericActions.xml
  inflating: 10188727/online/etc/xml/ShiphomeDirectoryStructure.xml 

[maclean@rh2 OPatch]$ opatch query 10188727 -all
Invoking OPatch 11.2.0.1.1

Oracle Interim Patch Installer version 11.2.0.1.1
Copyright (c) 2009, Oracle Corporation.  All rights reserved.

Oracle Home       : /s01/product/11.2.0/dbhome_2
Central Inventory : /s01/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.1
OUI version       : 11.2.0.2.0
OUI location      : /s01/product/11.2.0/dbhome_2/oui
Log file location : /s01/product/11.2.0/dbhome_2/cfgtoollogs/opatch/opatch2011-02-17_20-05-21PM.log

Patch history file: /s01/product/11.2.0/dbhome_2/cfgtoollogs/opatch/opatch_history.txt

--------------------------------------------------------------------------------
 Patch created on 2 Dec 2010, 01:44:13 hrs PST8PDT
 Need to shutdown Oracle instances: true
 Patch is roll-backable: true
 Patch is a "Patchset Update": false
 Patch is a rolling patch: true
 Patch has sql related actions: false
 Patch is an online patch: false
 Patch is a portal patch: false
 Patch is an "auto-enabled" patch: false

 List of platforms supported:
   226: Linux x86-64

 List of bugs to be fixed:
   10188727: AFTER UPGRADING TO 11.2.0.2 SOME SQLS FAIL WITH ORA-7445 [KKEIDC()+180] ERROR

 This patch is a "singleton" patch.

 This patch belongs to the "db" product family 

 List of executables affected:
   ORACLE_HOME/bin/oracle

 List of optional components:
   oracle.rdbms:  11.2.0.2.0

 List of optional actions:
   Update /s01/product/11.2.0/dbhome_2/lib/libserver11.a with /kkopq.o
   cd /s01/product/11.2.0/dbhome_2/rdbms/lib
     ; make -f ins_rdbms.mk ioracle ORACLE_HOME=/s01/product/11.2.0/dbhome_2

  Possible XML representation of the patch:

     10188727

--------------------------------------------------------------------------------
OPatch succeeded.

[maclean@rh2 OPatch]$ ./opatch query -is_online_patch 10188727
Invoking OPatch 11.2.0.1.1

Oracle Interim Patch Installer version 11.2.0.1.1
Copyright (c) 2009, Oracle Corporation.  All rights reserved.

Oracle Home       : /s01/product/11.2.0/dbhome_2
Central Inventory : /s01/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.1
OUI version       : 11.2.0.2.0
OUI location      : /s01/product/11.2.0/dbhome_2/oui
Log file location : /s01/product/11.2.0/dbhome_2/cfgtoollogs/opatch/opatch2011-02-17_19-45-33PM.log

Patch history file: /s01/product/11.2.0/dbhome_2/cfgtoollogs/opatch/opatch_history.txt

--------------------------------------------------------------------------------
 Patch is an online patch: false

OPatch succeeded.

/* 虽然Opatch返回"online patch: false",但实际上这是一个online patch,
    造成以上问题的原因可能是Opatch版本低 */

[maclean@rh2 OPatch]$ tree 10188727
10188727
|-- README.txt
|-- etc
|   |-- config
|   |   |-- actions.xml
|   |   |-- deploy.xml
|   |   `-- inventory.xml
|   `-- xml
|       |-- GenericActions.xml
|       `-- ShiphomeDirectoryStructure.xml
|-- files
|   `-- lib
|       `-- libserver11.a
|           `-- kkopq.o
`-- online
    |-- etc
    |   |-- config
    |   |   |-- actions.xml
    |   |   |-- deploy.xml
    |   |   `-- inventory.xml
    |   `-- xml
    |       |-- GenericActions.xml
    |       `-- ShiphomeDirectoryStructure.xml
    `-- files
        `-- hpatch
            `-- bug10188727.pch

/* 可以从以上目录结构中看到包含了online子目录,我们可以直接观察其inventory.xml信息文件 */

[maclean@rh2 OPatch]$ cat 10188727/online/etc/config/inventory.xml |grep instance
 <instance_shutdown>false</instance_shutdown>
 <instance_shutdown_message></instance_shutdown_message>

/* 以上instance_shutdown为false说明其可以作为online patch实施 */

[maclean@rh2 OPatch]$ cd 10188727

/* opatch online patching的具体语法如下 */
opatch apply online -connectString  <SID>:<USERNAME>:<PASSWORD>:<NODE1>, \
<SID2>:<USERNAME>:<PASSWORD>:<NODE2>,... 

[maclean@rh2 10188727]$ opatch apply online -connectString PROD:sys:password
Invoking OPatch 11.2.0.1.1

Oracle Interim Patch Installer version 11.2.0.1.1
Copyright (c) 2009, Oracle Corporation.  All rights reserved.

Oracle Home       : /s01/product/11.2.0/dbhome_2
Central Inventory : /s01/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.1
OUI version       : 11.2.0.2.0
OUI location      : /s01/product/11.2.0/dbhome_2/oui
Log file location : /s01/product/11.2.0/dbhome_2/cfgtoollogs/opatch/opatch2011-02-17_19-49-44PM.log

Patch history file: /s01/product/11.2.0/dbhome_2/cfgtoollogs/opatch/opatch_history.txt

The patch should be applied/rolled back in '-all_nodes' mode only.
Converting the RAC mode to '-all_nodes' mode.
ApplySession applying interim patch '10188727' to OH '/s01/product/11.2.0/dbhome_2'

Running prerequisite checks...

OPatch detected non-cluster Oracle Home from the inventory and will patch the local system only.

Backing up files and inventory (not for auto-rollback) for the Oracle Home
Backing up files affected by the patch '10188727' for restore. This might take a while...
Backing up files affected by the patch '10188727' for rollback. This might take a while...

Patching component oracle.rdbms, 11.2.0.2.0...
The patch will be installed on active database instances.
Installing and enabling the online patch 'bug10188727.pch', on database 'PROD'.

ApplySession adding interim patch '10188727' to inventory

Verifying the update...
Inventory check OK: Patch ID 10188727 is registered in Oracle Home inventory with proper meta-data.
Files check OK: Files from Patch ID 10188727 are present in Oracle Home.
OPatch succeeded.

[maclean@rh2 OPatch]$ opatch lsinventory -detail|tail -21
Interim patches (1) :

Patch (online) 10188727: applied on Thu Feb 17 19:49:52 CST 2011
Unique Patch ID:  13202318
   Created on 2 Dec 2010, 01:44:15 hrs PST8PDT
   Bugs fixed:
     10188727
   Files Touched:
     bug10188727.pch --> ORACLE_HOME/hpatch/bug10188727.pch
   Instances Patched:
     PROD
   Patch Location in Inventory:
     /s01/product/11.2.0/dbhome_2/inventory/oneoffs/10188727
   Patch Location in Storage area:
     /s01/product/11.2.0/dbhome_2/.patch_storage/10188727_Dec_2_2010_01_44_15

告警日志alert.log中的信息:
Patch file bug10188727.pch is out of sync with oracle binary; performing fixup
Patch file bug10188727.pch has been synced with oracle binary
Patch bug10188727.pch Installed - Update #1
Patch bug10188727.pch Enabled - Update #2

[maclean@rh2 OPatch]$ ps -ef|grep pmon|grep -v grep
maclean  22481     1  0 19:19 ?        00:00:00 ora_pmon_PROD

[maclean@rh2 OPatch]$ pmap -d 22481|tail -10
00002abecb435000       4 rwx-- 0000000000009000 008:00002 libnque11.so
00002abecb436000    1048 rwx-- 00002abecb436000 000:00000   [ anon ]
00002abecb53c000       8 r-x-- 000000000c64e000 008:00002 oracle
00002abecb53e000    5052 r-x-- 00000000000bd000 008:00002 oracle
00002abecba2d000     140 r-x-- 0000000000000000 008:00002 bug10188727.so
00002abecba50000    1024 ----- 0000000000023000 008:00002 bug10188727.so
00002abecbb50000       8 rwx-- 0000000000023000 008:00002 bug10188727.so
00007fff3342d000      84 rwx-- 00007ffffffea000 000:00000   [ stack ]
ffffffffff600000    8192 ----- 0000000000000000 000:00000   [ anon ]
mapped: 2297720K    writeable/private: 7848K    shared: 2050052K

/* 再次观察pmon进程的内存信息,可以看到pmap输出中多出了2个小的oracle正文镜像和
    名为bug10188727.so的共享库文件,而该后台进程的private memory由原来的7840k上升到7848k,
    实际增幅为8k */

/* 此外可以通过oradebug命令将该online patch禁用,虽然并不推荐这样做 */

SQL> oradebug patch disable  bug10188727.pch;
Statement processed.

SQL>  oradebug patch disable bug10188727.pch;
Patch file already disabled

[maclean@rh2 hpatch]$ pmap -d 22481|tail -8
00002abecb53c000       8 r-x-- 000000000c64e000 008:00002 oracle
00002abecb53e000    5052 r-x-- 00000000000bd000 008:00002 oracle
00002abecba2d000     140 r-x-- 0000000000000000 008:00002 bug10188727.so
00002abecba50000    1024 ----- 0000000000023000 008:00002 bug10188727.so
00002abecbb50000       8 rwx-- 0000000000023000 008:00002 bug10188727.so
00007fff3342d000      84 rwx-- 00007ffffffea000 000:00000   [ stack ]
ffffffffff600000    8192 ----- 0000000000000000 000:00000   [ anon ]
mapped: 2297720K    writeable/private: 7848K    shared: 2050052K

/* 但disable掉online patch并不会导致在线补丁额外消耗的内存被回收 */

/* 当然我们还可以很方便地启用它 */

SQL> oradebug patch enable bug10188727.pch;
Statement processed.
SQL>  oradebug patch enable bug10188727.pch;
Patch file already enabled

[maclean@rh2 ~]$ cd $ORACLE_HOME/hpatch

[maclean@rh2 hpatch]$ ls -l
total 368
-rw-r--r-- 1 maclean oinstall 177874 Feb 17 19:49 bug10188727.pch
-rwx------ 1 maclean oinstall      1 Feb 17 19:49 bug10188727.pchPROD.fixup
-rwx------ 1 maclean oinstall 176850 Feb 17 19:49 bug10188727.so
-rw------- 1 maclean oinstall    712 Feb 17 20:13 orapatchPROD.cfg

/* 注意不要在实例启动时删除以上hpatch目录及目录下任何文件,这可能导致instance出现意外 */

[maclean@rh2 hpatch]$ cd $ORACLE_HOME/OPatch

/* 我们还能够将online patch rollback回滚掉,如以下语法 */
opatch rollback -id <patchID> -connectString  <SID>:<USERNAME>:<PASSWORD>:<NODE1>, \
<SID2>:<USERNAME>:<PASSWORD>:<NODE2>,  ...

[maclean@rh2 OPatch]$ opatch rollback -id 10188727 -connectString PROD:sys:password -invPtrLoc /s01/product/11.2.0/dbhome_2/oraInst.loc
Invoking OPatch 11.2.0.1.1

Oracle Interim Patch Installer version 11.2.0.1.1
Copyright (c) 2009, Oracle Corporation.  All rights reserved.

Oracle Home       : /s01/product/11.2.0/dbhome_2
Central Inventory : /s01/oraInventory
   from           : /s01/product/11.2.0/dbhome_2/oraInst.loc
OPatch version    : 11.2.0.1.1
OUI version       : 11.2.0.2.0
OUI location      : /s01/product/11.2.0/dbhome_2/oui
Log file location : /s01/product/11.2.0/dbhome_2/cfgtoollogs/opatch/opatch2011-02-17_20-18-00PM.log

Patch history file: /s01/product/11.2.0/dbhome_2/cfgtoollogs/opatch/opatch_history.txt

RollbackSession rolling back interim patch '10188727' from OH '/s01/product/11.2.0/dbhome_2'

The patch should be applied/rolled back in '-all_nodes' mode only.
Converting the RAC mode to '-all_nodes' mode.

Running prerequisite checks...

OPatch detected non-cluster Oracle Home from the inventory and will patch the local system only.

Backing up files affected by the patch '10188727' for restore. This might take a while...

Patching component oracle.rdbms, 11.2.0.2.0...
The patch will be removed from active database instances.
Disabling and removing online patch 'bug10188727.pch', on database 'PROD'

RollbackSession removing interim patch '10188727' from inventory

OPatch succeeded.

告警日志alert.log中的remove信息:
Patch bug10188727.pch Disabled - Update #5
Patch bug10188727.pch Removed - Update #6
Thu Feb 17 20:18:07 2011
Online patch bug10188727.pch has been disabled
Online patch bug10188727.pch has been removed

[maclean@rh2 trace]$ pmap -d 22481|tail -8
00002abecb53c000       8 r-x-- 000000000c64e000 008:00002 oracle
00002abecb53e000    5052 r-x-- 00000000000bd000 008:00002 oracle
00002abecba2d000     140 r-x-- 0000000000000000 008:00002 bug10188727.so
00002abecba50000    1024 ----- 0000000000023000 008:00002 bug10188727.so
00002abecbb50000       8 rwx-- 0000000000023000 008:00002 bug10188727.so
00007fff3342d000      84 rwx-- 00007ffffffea000 000:00000   [ stack ]
ffffffffff600000    8192 ----- 0000000000000000 000:00000   [ anon ]
mapped: 2297720K    writeable/private: 7848K    shared: 2050052K

/* 显然rollback回滚掉该online interim patch也不足以回收内存,唯一的方法是重启实例 */

SQL> startup force;

[maclean@rh2 trace]$ ps -ef|grep pmon|grep -v grep
maclean  25563     1  0 20:22 ?        00:00:00 ora_pmon_PROD

[maclean@rh2 trace]$ pmap -d 25563|tail -8
00002aaf2f324000       4 rwxs- 0000000000000000 008:00002 hc_PROD.dat
00002aaf2f325000      40 r-x-- 0000000000000000 008:00002 libnque11.so
00002aaf2f32f000    1020 ----- 000000000000a000 008:00002 libnque11.so
00002aaf2f42e000       4 rwx-- 0000000000009000 008:00002 libnque11.so
00002aaf2f42f000    1048 rwx-- 00002aaf2f42f000 000:00000   [ anon ]
00007fffdfa84000      84 rwx-- 00007ffffffea000 000:00000   [ stack ]
ffffffffff600000    8192 ----- 0000000000000000 000:00000   [ anon ]
mapped: 2291488K    writeable/private: 7840K    shared: 2050052K

/* That's ok!
    ash to ash, dust to dust! */

References:<RDBMS Online Patching Aka Hot Patching [ID 761111.1]>

沪ICP备14014813号-2

沪公网安备 31010802001379号