Know more about RAC statistics and wait event

以下列出了RAC中的主要统计信息和等待事件:

 

1. Statistics:

1.1 V$SYSSTAT, V$SESSTAT (join to V$STATNAME)

  • gc cr blocks served
  • gc cr block build time
  • gc cr block flush time
  • gc cr block send time
  • gc current blocks served
  • gc current block pin time
  • gc current block flush time
  • gc current block send time
  • gc cr blocks received
  • gc cr block receive time
  • gc current blocks received
  • gc current block receive time
  • gc blocks lost
  • gc claim blocks lost
  • gc blocks corrupt
  • global enqueue gets sync
  • global enqueue gets async
  • global enqueue get time
  • global enqueue releases
  • gcs messages sent
  • ges messages sent
  • global enqueue CPU used by this session
  • gc CPU used by this session
  • IPC CPU used by this session
  • global undo segment hints helped (10.2)
  • global undo segment hints were stale (10.2)

1.2 V$SYSMETRIC

  • GC CR Block Received Per Second
  • GC CR Block Received Per Txn
  • GC Current Block Received Per Second
  • GC Current Block Received Per Txn
  • Global Cache Average CR Get Time
  • Global Cache Average Current Get Time
  • Global Cache Blocks Corrupted
  • Global Cache Blocks Lost

1.3 V$SEGMENT_STATISTICS, V$SEGSTAT

  • gc buffer busy
  • gc cr blocks received
  • gc current blocks received

1.4 DBA_HIST_SEG_STAT

  • GC_CR_BLOCKS_SERVED_TOTAL
  • GC_CR_BLOCKS_SERVED_DELTA
  • GC_CU_BLOCKS_SERVED_TOTAL
  • GC_CU_BLOCKS_SERVED_DELTA

1.5 V$GES_STATISTICS

  • acks for commit broadcast(actual)
  • acks for commit broadcast(logical)
  • broadcast msgs on commit(actual)
  • broadcast msgs on commit(logical)
  • broadcast msgs on commit(wasted)
  • dynamically allocated gcs resources
  • dynamically allocated gcs shadows
  • false posts waiting for scn acks
  • flow control messages received
  • flow control messages sent
  • gcs assume cvt
  • gcs assume no cvt
  • gcs ast xid
  • gcs blocked converts
  • gcs blocked cr converts
  • gcs compatible basts
  • gcs compatible cr basts (global)
  • gcs compatible cr basts (local)
  • gcs cr basts to PIs
  • gcs cr serve without current lock
  • gcs dbwr flush pi msgs
  • gcs dbwr write request msgs
  • gcs error msgs
  • gcs forward cr to pinged instance
  • gcs immediate (compatible) converts
  • gcs immediate (null) converts
  • gcs immediate cr (compatible) converts
  • gcs immediate cr (null) converts
  • gcs indirect ast
  • gcs lms flush pi msgs
  • gcs lms write request msgs
  • gcs msgs process time(ms)
  • gcs msgs received
  • gcs out-of-order msgs
  • gcs pings refused
  • gcs queued converts
  • gcs recovery claim msgs
  • gcs refuse xid
  • gcs regular cr
  • gcs retry convert request
  • gcs side channel msgs actual
  • gcs side channel msgs logical
  • gcs undo cr
  • gcs write notification msgs
  • gcs writes refused
  • ges msgs process time(ms)
  • ges msgs received
  • global posts dropped
  • global posts queue time
  • global posts queued
  • global posts requested
  • global posts sent
  • implicit batch messages received
  • implicit batch messages sent
  • lmd msg send time(ms)
  • lms(s) msg send time(ms)
  • messages flow controlled
  • messages queue sent actual
  • messages queue sent logical
  • messages received actual
  • messages received logical
  • messages sent directly
  • messages sent indirectly
  • messages sent not implicit batched
  • messages sent pbatched
  • msgs causing lmd to send msgs
  • msgs causing lms(s) to send msgs
  • msgs received queue time (ms)
  • msgs received queued
  • msgs sent queue time (ms)
  • msgs sent queue time on ksxp (ms)
  • msgs sent queued
  • msgs sent queued on ksxp
  • process batch messages received
  • process batch messages sent

2. Wait Events:

在10g中RAC等待事件可以分为3类,下面列出了主要的等待事件包括一些undocumented wait event.

2.1. Real time only :

Those wait events are only defined while the process is waiting; after the wait is over, they are reclassified according to the outcome of the global cache operation. They should appear only on the following views: V$SESSION_WAIT, V$ACTIVE_SESSION_HISTORY, V$EVENT_HISTOGRAM.

 

  • gc cr request
  • gc current request

2.2. Historical only:

These are events represent the outcome of a GC request (fixup events). They can appear in the following views: V$SESSION_EVENT, V$SYSTEM_EVENT, DBA_HIST_SYSTEM_EVENT, V$EVENTMETRIC.

  • gc cr block 2-way
  • gc cr block 3-way
  • gc cr block busy
  • gc cr block congested
  • gc cr block lost
  • gc cr block unknown
  • gc cr grant 2-way
  • gc cr grant busy
  • gc cr grant congested
  • gc cr grant unknown
  • gc current block 2-way
  • gc current block 3-way
  • gc current block busy
  • gc current block congested
  • gc current block lost
  • gc current block unknown
  • gc current grant 2-way
  • gc current grant busy
  • gc current grant congested
  • gc current grant unknown

2.3. Other events:

The remaining wait events may appear in any of the views listed before, namely: V$SESSION_WAIT, V$ACTIVE_SESSION_HISTORY, V$EVENT_HISTOGRAM, V$SESSION_EVENT, V$SYSTEM_EVENT, DBA_HIST_SYSTEM_EVENT, V$EVENTMETRIC.

  • LMON global data update
  • cr request retry
  • gc assume
  • gc block recovery request
  • gc buffer busy
  • gc claim
  • gc cr cancel
  • gc cr disk read
  • gc cr disk request
  • gc cr failure
  • gc cr multi block request
  • gc current cancel
  • gc current multi block request
  • gc current retry
  • gc current split
  • gc domain validation
  • gc freelist
  • gc prepare
  • gc quiesce wait
  • gc recovery free
  • gc recovery quiesce
  • gc remaster
  • gcs ddet enter server mode
  • gcs domain validation
  • gcs drm freeze begin
  • gcs drm freeze in enter server mode
  • gcs enter server mode
  • gcs log flush sync
  • gcs remastering wait for read latch
  • gcs remastering wait for write latch
  • gcs remote message
  • gcs resource directory to be unfrozen
  • gcs to be enabled
  • ges LMD suspend for testing event
  • ges LMD to inherit communication channels
  • ges LMD to shutdown
  • ges LMON for send queues
  • ges LMON to get to FTDONE
  • ges LMON to join CGS group
  • ges cached resource cleanup
  • ges cancel
  • ges cgs registration
  • ges enter server mode
  • ges generic event
  • ges global resource directory to be frozen
  • ges inquiry response
  • ges lmd and pmon to attach
  • ges lmd/lmses to freeze in rcfg – mrcvr
  • ges lmd/lmses to unfreeze in rcfg – mrcvr
  • ges master to get established for SCN op
  • ges performance test completion
  • ges pmon to exit
  • ges process with outstanding i/o
  • ges reconfiguration to start
  • ges remote message
  • ges resource cleanout during enqueue open
  • ges resource cleanout during enqueue open-cvt
  • ges resource directory to be unfrozen
  • ges retry query node
  • ges reusing os pid
  • ges user error
  • ges wait for lmon to be ready
  • ges1 LMON to wake up LMD – mrcvr
  • ges2 LMON to wake up LMD – mrcvr
  • ges2 LMON to wake up lms – mrcvr 2
  • ges2 LMON to wake up lms – mrcvr 3
  • ges2 proc latch in rm latch get 1
  • ges2 proc latch in rm latch get 2
  • global cache busy
  • global enqueue expand wait
  • latch: KCL gc element parent latch
  • latch: gcs resource hash
  • latch: ges resource hash list

了解Oracle RAC Brain Split Resolution

Slide:Upgrade 11.2.0.1 GI/CRS to 11.2.0.2 in Linux

Upgrade 11.2.0.1 DB/RDBMS to 11.2.0.2 in Linux

<Upgrade 11.2.0.1 GI/CRS to 11.2.0.2 in Linux>一文中我们介绍了升级11.2.0.1 GI/CRS到11.2.0.2的详细步骤,因为GI/CRS的版本总是要求大于DB/RDBMS,所以这是我们升级RDBMS数据库软件的前提条件。

接下来我们将具体介绍升级11.2.0.1 DB/RDBMS到 11.2.0.2的详细步骤:

一、 下载补丁介质

11.2.0.2的patchset目前没有公开的下载地址,因为updates.oracle.com目前已经不再提供ftp下载模式,所以我们只能通过登录My Oracle Support后进入Patch栏目搜索Patchid并获得加密的下载链接。

11.2.0.2补丁集的全称是11.2.0.2.0 PATCH SET FOR ORACLE DATABASE SERVER (Patchset)(patchid:10098816),可以通过10098816这个id到Patch栏目搜索,并找出对应平台的介质zip包。如在Linux x86-64平台上:

Patch 10098816 11.2.0.2.0 PATCH SET FOR ORACLE DATABASE SERVER_download

 

以上p10098816_112020_Linux-x86-64_1of7.zip和p10098816_112020_Linux-x86-64_2of7.zip ,这2个zip包对应为Database/RDBMS软件的介质,我们不需要下载所有的7个zip包,有这2个升级数据库软件就已经足够了。

完成以上2个软件的下载后,分别解压zip包:

unzip p10098816_112020_Linux-x86-64_1of7.zip -d  $PATCHHOME
unzip p10098816_112020_Linux-x86-64_2of7.zip -d  $PATCHHOME

二、以out of place方式安装11.2.0.2 DB数据库软件

因为11.2.0.2的Patchset以后都是out of place的,所以我们可以不用像在11gr2以前那样必须在原有安装低版本软件的基础上才能升级软件,而可以选择在别的位置完全新安装。

注意该步骤不需要停止数据库实例,可以在前期工作中完成。

以DB/RDBMS数据库软件的拥有者身份(oracle用户)启动方才解压目录下的oui安装界面:

su - oracle

(oracle)$ unset ORACLE_HOME ORACLE_BASE ORACLE_SID
(oracle)$ export DISPLAY=:0
(oracle)$ cd $PATCHHOME
(oracle)$ ./runInstaller

在Oracle Universal Installer界面下的Select Installation Options Screen选择install database only.

upgrade_110202_DB_1

 

在Grid Installation Options下若是RAC 数据库则选择Oracle Real Application cluster database installation,注意如果在该屏幕下出现[FATAL] [INS-35354] The system on which you are attempting to install Oracle RAC is not part of a valid cluster则可能是在之前的安装Gird的过程中没有正确的Update Inventory更新信息库信息,见<11gr2 RAC安装INS-35354问题一例>

若是单节点数据库则选择Single instance database installation

 

upgrade_110202_DB_2

 

在Specify Installation Location Screen上一般OUI会帮你自动匹配一个$ORACLE_BASE变量下不同于原有数据库软件安装目录的新目录,确认这些目录下有足够的磁盘空间,保险起见空间应大于10GB。注意这里是out of place安装,所以千万不要填入原有的安装路径。

 

upgrade_110202_DB_3

 

以上安装完成后OUI会提示要在所有节点上以root身份执行root.sh脚本:

su - root
(root #) /s01/orabase/product/11.2.0/dbhome_2/root.sh

Running Oracle 11g root script...

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /s01/orabase/product/11.2.0/dbhome_2

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Finished product-specific root actions.

三、升级前的准备工作
以上我们完成了11.2.0.2 数据库软件的安装工作,但是还没有升级实例和数据字典。
在正式升级之前,极有必要完成一系列的备份和准备工作,这些准备工作可以详见拙作<Oracle数据库升级前必要的准备工作>

1.清理数据字典中的无用数据,包括审计和回收站,它们可能拉慢数据字典升级的速度:

TRUNCATE TABLE SYS.AUD$;
purge DBA_RECYCLEBIN;

 

2.如果条件允许的话,建议使用RMAN全量备份数据库,前提是数据库没有达到TB级别。

rman target / catalog rman/rman@cata

backup as compressed backupset incremental level 0 database ;

 

3. 收集数据字典的统计信息,若dictionary的统计信息不准备可能导致catupgrd.sql字典升级脚本运行过久:

SQL> set timing on;

SQL> EXECUTE dbms_stats.gather_dictionary_stats;

PL/SQL procedure successfully completed.

Elapsed: 00:00:27.81

 

4.运行dbupgdiag.sql升级信息收集脚本, 该脚本可以提供数据库的一些版本信息和组建信息,以下为该脚本的示例输出内容:

cat db_upg_diag_VPROD_07-Sep-2011_0737.log

                          *** Start of LogFile ***

  Oracle Database Upgrade Diagnostic Utility       09-07-2011 19:37:23

===============
Database Uptime
===============

19:32 07-SEP-11

=================
Database Wordsize
=================

This is a 64-bit database

================
Software Version
================

Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
PL/SQL Release 11.2.0.1.0 - Production
CORE    11.2.0.1.0      Production
TNS for Linux: Version 11.2.0.1.0 - Production
NLSRTL Version 11.2.0.1.0 - Production

=============
Compatibility
=============

Compatibility is set as 11.2.0.0.0

================
Component Status
================

Comp ID Component                          Status    Version        Org_Version    Prv_Version
------- ---------------------------------- --------- -------------- -------------- --------------
CATALOG Oracle Database Catalog Views      VALID     11.2.0.1.0
CATPROC Oracle Database Packages and Types VALID     11.2.0.1.0
OWM     Oracle Workspace Manager           VALID     11.2.0.1.0
RAC     Oracle Real Application Clusters   VALID     11.2.0.1.0

======================================================
List of Invalid Database Objects Owned by SYS / SYSTEM
======================================================

Number of Invalid Objects
------------------------------------------------------------------
There are no Invalid Objects

DOC>################################################################
DOC>
DOC> If there are no Invalid objects below will result in zero rows.
DOC>
DOC>################################################################
DOC>#

no rows selected

================================
List of Invalid Database Objects
================================

Number of Invalid Objects
------------------------------------------------------------------
There are no Invalid Objects

DOC>################################################################
DOC>
DOC> If there are no Invalid objects below will result in zero rows.
DOC>
DOC>################################################################
DOC>#

no rows selected

==============================================================
Identifying whether a database was created as 32-bit or 64-bit
==============================================================

DOC>###########################################################################
DOC>
DOC> Result referencing the string 'B023' ==> Database was created as 32-bit
DOC> Result referencing the string 'B047' ==> Database was created as 64-bit
DOC> When String results in 'B023' and when upgrading database to 10.2.0.3.0
DOC> (64-bit) , For known issue refer below articles
DOC>
DOC> Note 412271.1 ORA-600 [22635] and ORA-600 [KOKEIIX1] Reported While
DOC> Upgrading Or Patching Databases To 10.2.0.3
DOC> Note 579523.1 ORA-600 [22635], ORA-600 [KOKEIIX1], ORA-7445 [KOPESIZ] and
DOC> OCI-21500 [KOXSIHREAD1] Reported While Upgrading To 11.1.0.6
DOC>
DOC>###########################################################################
DOC>#

Metadata Initial DB Creation Info
-------- -----------------------------------
B047     Database was created as 64-bit

===================================================
Number of Duplicate Objects Owned by SYS and SYSTEM
===================================================

Counting duplicate objects ....

  COUNT(1)
----------
         4

=========================================
Duplicate Objects Owned by SYS and SYSTEM
=========================================

Querying duplicate objects ....

OBJECT_NAME                              OBJECT_TYPE
---------------------------------------- ----------------------------------------
AQ$_SCHEDULES                            TABLE
AQ$_SCHEDULES_PRIMARY                    INDEX
DBMS_REPCAT_AUTH                         PACKAGE BODY
DBMS_REPCAT_AUTH                         PACKAGE

DOC>
DOC>################################################################################
DOC>
DOC> If any objects found please follow below article.
DOC> Note 1030426.6 How to Clean Up Duplicate Objects Owned by SYS and SYSTEM schema
DOC> Read the Exceptions carefully before taking actions.
DOC>
DOC>################################################################################
DOC>#

================
JVM Verification
================

JAVAVM - NOT Installed. Below results can be ignored

================================================
Checking Existence of Java-Based Users and Roles
================================================

DOC>
DOC>################################################################################
DOC>
DOC> There should not be any Java Based users for database version 9.0.1 and above.
DOC> If any users found, it is faulty JVM.
DOC>
DOC>################################################################################
DOC>#

User Existence
---------------------------
No Java Based Users

DOC>
DOC>###############################################################
DOC>
DOC> Healthy JVM Should contain Six Roles.
DOC> If there are more or less than six role, JVM is inconsistent.
DOC>
DOC>###############################################################
DOC>#

Role
------------------------------
No JAVA related Roles

Roles

=========================================
List of Invalid Java Objects owned by SYS
=========================================

There are no SYS owned invalid JAVA objects

DOC>
DOC>#################################################################
DOC>
DOC> Check the status of the main JVM interface packages DBMS_JAVA
DOC> and INITJVMAUX and make sure it is VALID.
DOC> If there are no Invalid objects below will result in zero rows.
DOC>
DOC>#################################################################
DOC>#

no rows selected

INFO: Below query should succeed with 'foo' as result.
select dbms_java.longname('foo') "JAVAVM TESTING" from dual
       *
ERROR at line 1:
ORA-00904: "DBMS_JAVA"."LONGNAME": invalid identifier

                            *** End of LogFile ***

以上spool内容显示所要升级的数据库现有CATALOG、CATPROC、OWM和RAC组件,且没有安装JVM,升级JVM组建的数据字典将消耗较长的时间。

另外一个建议运行的脚本是utlu112i.sql,它位于新安装的$ORACLE_HOME/rdbms/admin目录下。

该脚本会给出一些升级前地建议,包括建议保证系统表空间和闪回区域有足够的空间,以及收集数据字典的统计信息,如以下输出:

SQL> @/s01/orabase/product/11.2.0/dbhome_2/rdbms/admin/utlu112i.sql
Oracle Database 11.2 Pre-Upgrade Information Tool 09-07-2011 20:02:30
Script Version: 11.2.0.2.0 Build: 001
.
**********************************************************************
Database:
**********************************************************************
--> name:          VPROD
--> version:       11.2.0.1.0
--> compatible:    11.2.0.0.0
--> blocksize:     8192
--> platform:      Linux x86 64-bit
--> timezone file: V11
.
**********************************************************************
Tablespaces: [make adjustments in the current environment]
**********************************************************************
--> SYSTEM tablespace is adequate for the upgrade.
.... minimum required size: 267 MB
--> SYSAUX tablespace is adequate for the upgrade.
.... minimum required size: 150 MB
--> UNDOTBS1 tablespace is adequate for the upgrade.
.... minimum required size: 253 MB
--> TEMP tablespace is adequate for the upgrade.
.... minimum required size: 61 MB
.
**********************************************************************
Flashback: ON
**********************************************************************
FlashbackInfo:
--> name:          +SYSTEMDG
--> limit:         4977 MB
--> used:          264 MB
--> size:          4977 MB
--> reclaim:       0 MB
--> files:         7
WARNING: --> Flashback Recovery Area Set.  Please ensure adequate disk space              in recover
y areas before performing an upgrade.
.
**********************************************************************
Update Parameters: [Update Oracle Database 11.2 init.ora or spfile]
Note: Pre-upgrade tool was run on a lower version 64-bit database.
**********************************************************************
--> If Target Oracle is 32-Bit, refer here for Update Parameters:
-- No update parameter changes are required.
.

--> If Target Oracle is 64-Bit, refer here for Update Parameters:
-- No update parameter changes are required.
.
**********************************************************************
Renamed Parameters: [Update Oracle Database 11.2 init.ora or spfile]
**********************************************************************
-- No renamed parameters found. No changes are required.
.
**********************************************************************
Obsolete/Deprecated Parameters: [Update Oracle Database 11.2 init.ora or spfile]
**********************************************************************
-- No obsolete parameters found. No changes are required
.

**********************************************************************
Components: [The following database components will be upgraded or installed]
**********************************************************************
--> Oracle Catalog Views         [upgrade]  VALID
--> Oracle Packages and Types    [upgrade]  VALID
--> Real Application Clusters    [upgrade]  VALID
--> Oracle Workspace Manager     [upgrade]  VALID
.
**********************************************************************
Miscellaneous Warnings
**********************************************************************
WARNING: --> The "cluster_database" parameter is currently "TRUE"
.... and must be set to "FALSE" prior to running a manual upgrade.
WARNING: --> Database is using a timezone file older than version 14.
.... After the release migration, it is recommended that DBMS_DST package
.... be used to upgrade the 11.2.0.1.0 database timezone version
.... to the latest version which comes with the new release.
WARNING: --> Your recycle bin is turned on and currently contains no objects.
.... Because it is REQUIRED that the recycle bin be empty prior to upgrading
.... and your recycle bin is turned on, you may need to execute the command:
        PURGE DBA_RECYCLEBIN
.... prior to executing your upgrade to confirm the recycle bin is empty.
.
**********************************************************************
Recommendations
**********************************************************************
Oracle recommends gathering dictionary statistics prior to
upgrading the database.
To gather dictionary statistics execute the following command
while connected as SYSDBA:

    EXECUTE dbms_stats.gather_dictionary_stats;

**********************************************************************
Oracle recommends removing all hidden parameters prior to upgrading.

To view existing hidden parameters execute the following command
while connected AS SYSDBA:

    SELECT name,description from SYS.V$PARAMETER WHERE name
        LIKE '\_%' ESCAPE '\'

Changes will need to be made in the init.ora or spfile.

**********************************************************************
Oracle recommends reviewing any defined events prior to upgrading.

To view existing non-default events execute the following commands
while connected AS SYSDBA:
  Events:
    SELECT (translate(value,chr(13)||chr(10),' ')) FROM sys.v$parameter2
      WHERE  UPPER(name) ='EVENT' AND  isdefault='FALSE'

  Trace Events:
    SELECT (translate(value,chr(13)||chr(10),' ')) from sys.v$parameter2
      WHERE UPPER(name) = '_TRACE_EVENTS' AND isdefault='FALSE'

Changes will need to be made in the init.ora or spfile.

**********************************************************************

 

5.如果数据库很大那么建议打开闪回数据库flashback database,并创建还原点,这样可以极大地缩短回退时间。

可以通过以下查询判断数据库是或否启用了flashback database功能:

 

SQL> select FLASHBACK_ON from v$database;

FLASHBACK_ON
------------------
NO

 

若显示NO则说明之前没有启用数据库闪回功能,若希望启用数据库闪回功能需要数据库短时间停机:

 

关闭所有的数据库实例

SQL> shutdown immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.

启动某一套实例到mount 状态

SQL> startup mount;
ORACLE instance started.

Total System Global Area 1252663296 bytes
Fixed Size                  2212936 bytes
Variable Size             603982776 bytes
Database Buffers          637534208 bytes
Redo Buffers                8933376 bytes
Database mounted.

SQL> alter database flashback on;

Database altered.

在本节点打开数据库,并启动所有节点

SQL> alter database open;

Database altered.

 

以上在数据库级别启用了闪回flashback功能。

接着我们需要停止应用程序,注意在这一步之前的准备工作都可以在线完成,但是本步骤将要求停止一切应用程序的链接,关闭数据库,并启动到restrict限制模式,以便创建restore point,方便可能的升级回退。,strict模式避免了普通用户的链接。

在所有节点上关闭数据库实例,并在唯一节点上启动数据库到restrict模式。

 

startup restrict;

ORACLE instance started.

Total System Global Area 1252663296 bytes
Fixed Size 2212936 bytes
Variable Size 603982776 bytes
Database Buffers 637534208 bytes
Redo Buffers 8933376 bytes
Database mounted.
Database opened.

SQL> conn maclean/maclean
ERROR:
ORA-01035: ORACLE only available to users with RESTRICTED SESSION privilege

Warning: You are no longer connected to ORACLE.

conn / as sysdba

SQL> create restore point maclean_rollback guarantee flashback database;

Restore point created.

SQL> select * from v$restore_point;

       SCN DATABASE_INCARNATION# GUA STORAGE_SIZE
---------- --------------------- --- ------------
TIME
---------------------------------------------------------------------------
RESTORE_POINT_TIME                                                          PRE
--------------------------------------------------------------------------- ---
NAME
--------------------------------------------------------------------------------
    601958                     1 YES     15941632
07-SEP-11 07.52.59.000000000 PM
                                                                            YES
MACLEAN_ROLLBACK

 

四、正式升级数据库实例和数据字典

1. 关闭所有数据库实例

2. 复制相关的pfile或spfile形式的参数到新的ORACLE_HOME下,这里我们假设使用ASM存储共享的spfile,那么只需要在所有节点上将init$SID.ora形式的文件拷贝即可:

 

(oracle $) cat $ORACLE_HOME/dbs/initVPROD1.ora
SPFILE='+SYSTEMDG/VPROD/spfileVPROD.ora'

(oracle $) cp $ORACLE_HOME/dbs/initVPROD1.ora /s01/orabase/product/11.2.0/dbhome_2/dbs

设置ORACLE_HOME和PATH变量指向新的11.2.0.2数据库软件

(oracle $) export ORACLE_HOME=/s01/orabase/product/11.2.0/dbhome_2
(oracle $) export PATH=/s01/orabase/product/11.2.0/dbhome_2/bin:$PATH

设置正确的ORACLE_SID

(oracle $) export ORACLE_SID=VPROD1
(oracle $) unset LD_LIBRARY_PATH

 

3. 启动实例到nomount状态,并修改cluster_database参数到spfile:

 

SQL> startup nomount;
ORACLE instance started.

Total System Global Area 1252663296 bytes
Fixed Size                  2226072 bytes
Variable Size             402655336 bytes
Database Buffers          838860800 bytes
Redo Buffers                8921088 bytes

SQL> alter system set cluster_database=false scope=spfile;

System altered.

 

4. 重启实例到upgrade模式,升级数据字典,运行$ORACLE_HOME/rdbms/admin/catupgrd.sql脚本:

 

SQL> shutdown immediate;
ORA-01507: database not mounted

ORACLE instance shut down.
SQL> startup upgrade;
ORACLE instance started.

Total System Global Area 1252663296 bytes
Fixed Size                  2226072 bytes
Variable Size             402655336 bytes
Database Buffers          838860800 bytes
Redo Buffers                8921088 bytes
Database mounted.
Database opened.

SQL> set echo on  

SQL> SPOOL /tmp/upgrade.log

SQL> set time on; 

20:40:40 SQL> @/s01/orabase/product/11.2.0/dbhome_2/rdbms/admin/catupgrd.sql 

在以上catupgrd.sql脚本运行过程中可以通过DBA_SERVER_REGISTRY视图了解组件字典升级的进度

SQL> select * from DBA_SERVER_REGISTRY;
select * from DBA_SERVER_REGISTRY
              *
ERROR at line 1:
ORA-04063: view "SYS.DBA_SERVER_REGISTRY" has errors
or
ERROR at line 1:
ORA-04063: package body "SYS.DBMS_REGISTRY" has errors

在一开始会提示该视图有错误,这不要紧,稍等一会。

SQL> select comp_name,status,version from dba_server_registry;

COMP_NAME                                          STATUS                           VERSION
-------------------------------------------------- --------------------------       ------------------------------
Oracle Workspace Manager                           UPGRADING                        11.2.0.1.0
Oracle Database Catalog Views                      VALID                            11.2.0.2.0
Oracle Database Packages and Types                 VALID                            11.2.0.2.0
Oracle Real Application Clusters                   VALID                            11.2.0.2.0

20:50:40 SQL>
20:50:40 SQL> Rem *********************************************************************
20:50:40 SQL> Rem END catupgrd.sql
20:50:40 SQL> Rem *********************************************************************
20:50:40 SQL> 

以上catupgrd.sql脚本运行了10分钟左右

重启实例,运行utlrp.sql脚本编译失效对象

sqlplus  / as sysdba
startup;

@?/rdbms/admin/utlrp

TIMESTAMP
--------------------------------------------------------------------------------
COMP_TIMESTAMP UTLRP_BGN  2011-09-07 20:53:38

该脚本会自动根据cpu数目选择并行度

DOC>   This script automatically chooses serial or parallel recompilation
DOC>   based on the number of CPUs available (parameter cpu_count) multiplied
DOC>   by the number of threads per CPU (parameter parallel_threads_per_cpu).
DOC>   On RAC, this number is added across all RAC nodes.

TIMESTAMP
--------------------------------------------------------------------------------
COMP_TIMESTAMP UTLRP_END  2011-09-07 20:55:09

该脚本耗时约2分钟

修改cluster_database参数为true,并重启所有节点实例

SQL> alter system set cluster_database=true scope=spfile;

System altered.

可以看到以上在数据库仅安装了CATALOG、CATPROC、OWM和RAC Cluster View 4种组件的情况下,catupgrd.sql字典升级脚本仅耗时10分钟左右。 而实际的生产库可能安装了更多的组件,如JVM等组件将耗时较多。

以下总结了各Oracle组件升级字典的平均耗时,是一张十分有用的升级时间参考表:

DB Sample Upgrade Time

较少组件情况下

Component HH:MM:SS
Oracle Server 00:16:17
JServer JAVA Virtual Machine 00:05:19
Oracle XDK 00:00:48
Oracle Text 00:00:58
Oracle XML Database 00:04:09
Oracle Database Java Packages 00:00:33
Gathering Statistics 00:02:43
Total Upgrade Time: 00:30:47

 

较多组件情况下

Component HH:MM:SS
Oracle Server 00:16:17
JServer JAVA Virtual Machine 00:05:19
Oracle Workspace Manager 00:01:01
Oracle Enterprise Manager 00:10:13
Oracle XDK 00:00:48
Oracle Text 00:00:58
Oracle XML Database 00:04:09
Oracle Database Java Packages 00:00:33
Oracle Multimedia 00:07:43
Oracle Expression Filter 00:00:18
Oracle Rule Manager 00:00:12
Gathering Statistics 00:04:53
Total Upgrade Time: 00:52:31

 

5.使用srvctl命令更新ocr中DBHOME相关信息:

 

su  - oracle

srvctl upgrade database -d VPROD -o $NEW_ORACLE_HOME

srvctl upgrade database -d VPROD -o /s01/orabase/product/11.2.0/dbhome_2

[oracle@vrh1 ~]$ srvctl config database -d VPROD
Database unique name: VPROD
Database name: VPROD
Oracle home: /s01/orabase/product/11.2.0/dbhome_2
Oracle user: oracle
Spfile: +SYSTEMDG/VPROD/spfileVPROD.ora
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: VPROD
Database instances: VPROD1,VPROD2
Disk Groups: SYSTEMDG
Mount point paths:
Services:
Type: RAC
Database is administrator managed

[oracle@vrh1 ~]$ srvctl stop database -d VPROD
PRCC-1016 : VPROD was already stopped
[oracle@vrh1 ~]$ srvctl start database -d VPROD  

[oracle@vrh1 ~]$ srvctl status  database -d VPROD
Instance VPROD1 is running on node vrh1
Instance VPROD2 is running on node vrh2

 

6.修改oracle用户的profile配置文件指中的变量:

 

cat .bash_profile 

# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi

# User specific environment and startup programs

ORACLE_HOME=/s01/orabase/product/11.2.0/dbhome_2
ORACLE_SID=VPROD1
ORACLE_BASE=/s01/orabase
PATH=$ORACLE_HOME/bin:$ORACLE_HOME/OPatch:$PATH:$HOME/bin

export PATH ORACLE_HOME ORACLE_SID ORACLE_BASE

SQL> select * from global_name;

GLOBAL_NAME
--------------------------------------------------------------------------------
www.askmac.cn

SQL> select * from v$version;

BANNER
--------------------------------------------------------------------------------
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
PL/SQL Release 11.2.0.2.0 - Production
CORE    11.2.0.2.0      Production
TNS for Linux: Version 11.2.0.2.0 - Production
NLSRTL Version 11.2.0.2.0 - Production

 

7. 数据库升级完成后进入一个pending area,建议在至少2个礼拜内,不要升级compatible参数和删除restore point。

在确认没有回退的必要后,修改compatible参数并删除restore point:

 

alter system set compatible=’11.2.0.2.0′ scope=spfile;

drop restore point  MACLEAN_ROLLBACK;

srvctl stop database -d VPROD 

srvctl start database -d VPROD

以上成功地将11.2.0.1的RAC数据库升级到了11.2.0.2。

 

五、回退升级操作(Database Downgrade)

我们可以选择2种回退办法:

  1. 通过restore point还原到11.2.0.1的数据库
  2. 执行catdwgrd.sql降级数据字典

针对第一种方法:

关闭所有节点实例

srvctl stop database -d VPROD

export ORACLE_HOME=$OLD_ORACLE_HOME
export PATH=$OLD_ORACLE_HOME/bin:$PATH
unset LD_LIBRARY_PATH

sqlplus  / as sysdba

SQL> select * from v$restore_point;

       SCN DATABASE_INCARNATION# GUA STORAGE_SIZE
---------- --------------------- --- ------------
TIME
---------------------------------------------------------------------------
RESTORE_POINT_TIME                                                          PRE
--------------------------------------------------------------------------- ---
NAME
--------------------------------------------------------------------------------
    601958                     1 YES    462307328
07-SEP-11 07.52.59.000000000 PM
                                                                            YES
MACLEAN_ROLLBACK

SQL> flashback database to restore point MACLEAN_ROLLBACK;

Flashback complete.

flashback database的速度 视乎flashback log多少而定,一般是很快的,在1分钟之内。

SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-01589: must use RESETLOGS or NORESETLOGS option for database open

SQL> alter database open resetlogs;

Database altered.

以上通过restore point的方法是我所推荐的,这种方法简单、省时省力、高效且问题少少,是一种绿色方案。同时不要忘记使用srvctl upgrade命令还原ocr中的DBHOME信息,以及还原profile文件。

针对第二种方法:
catdwgrd.sql的运行有诸多限制,其所消耗的时间可能要略长于catupgrd.sql。而且该脚本在运行过程中可能遇到各种错误,不推荐使用这种方法。

关于使用catdwgrd.sql脚本降级数据库11.2.0.2到11.2.0.1,可以参考MOS note <How To Downgrade From Database 11.2 To Previous Release (includes 11.2.0.2-11.2.0.1) [ID 883335.1]>

Know GCS AND GES structure size in shared pool

RAC环境中共享池很大一部分被gcs和ges资源所占用,一般来说这些资源对象都是永久的(perm)的,所以我们无法期待LRU或flush shared_pool操作能够清理这些资源。

在使用大缓存(large buffer cache)的RAC实例环境中,查询v$sgastat内存动态性能视图时总是能发现’gcs resources’、’gcs shadows’、’ ges resource’、’ges enqueues ‘这些组件占用了共享池中的大量内存,为了避免shared pool出现著名的ORA-04031错误,Oracle推荐在RAC环境中设置较大的shared_pool_size初始化参数,此外显示地设置较大的GCS和GES资源结构的初始化分配数(INITIAL_ALLOCATION)也有利于避免ORA-4031。

这些控制GES和GCS资源结构初始化分配数量的参数主要包括:

  • _gcs_resources  number of gcs resources to be allocated GCS Resources Number of GCS resource structures determined by
    _gcs_resources parameter
    Stored in segmented array
    Externalized in X$KJBR
    Number of free GCS resource structures in X$KJBRFX
  • _gcs_shadow_locks number of pcm shadow locks to be allocated GCS Enqueues (Shadows/Clients) Number of GCS enqueue structures determined by  _gcs_shadow_locks parameter Stored in segmented array
    Externalized in X$KJBL
    Number of free GCS enqueue structures in X$KJBLFX
  • _lm_ress number of resources configured for cluster database LM_RESS controls the number of resources that can be locked by each lock manager instance. These resources include lock resources allocated for DML, DDL (data dictionary locks), data dictionary, and library cache locks plus the file and log management locks. Stored in heap
    Externalized in X$KJIRFT
  • _lm_locks number of enqueues configured for cluster database Stored in segmented array
    Externalized in X$KJILKFT

为了更好地在RAC环境中设置shared_pool_size共享池的大小(手动设置该参数并不会disable AMM or ASMM),我们很有必要评估大量初始化分配的全局资源本身将占用shared pool多大的空间。

我们可以通过v$resource_limit视图了解这些GES、GCS全局资源的分配情况:

SQL> select * from v$version;

BANNER
--------------------------------------------------------------------------------
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
PL/SQL Release 11.2.0.2.0 - Production
CORE    11.2.0.2.0      Production
TNS for Linux: Version 11.2.0.2.0 - Production
NLSRTL Version 11.2.0.2.0 - Production

SQL> select * from global_name;

GLOBAL_NAME
--------------------------------------------------------------------------------
www.askmac.cn



SQL> select * from v$resource_limit where resource_name in ('gcs_resources', 'gcs_shadows','ges_ress','ges_locks'); 

RESOURCE_NAME                  CURRENT_UTILIZATION MAX_UTILIZATION INITIAL_ALLOCATION        LIMIT_VALUE
------------------------------ ------------------- --------------- ------------------------- ------------------
ges_ress                                      7223            7486    1000000                 UNLIMITED
ges_locks                                     4944            5027    1000000                 UNLIMITED
gcs_resources                                 4021            4021     114466                    114466
gcs_shadows                                   3925            3925     114466                    114466

可以通过v$sgastat视图了解这些全局资源占用了多少空间:

select *
  from v$sgastat
 where name in
       ('ges resource ', 'ges enqueues', 'gcs resources', 'gcs shadows');

POOL         NAME                            BYTES
------------ -------------------------- ----------
shared pool  gcs resources                16483232
shared pool  gcs shadows                  11904560
shared pool  ges enqueues                 47809680
shared pool  ges resource                288405768

单个gcs_resources结构大约占用120 bytes
单个gcs_shadows 结构大约占用72 bytes
单个ges_resource 结构大约占用288 bytes

我们可以使用一下初步估算GES、GCS资源结构将至少占用多大的共享池资源:

‘gcs_resources’ = initial_allocation * 120 bytes = “_gcs_resources parameter” * 120 bytes
‘gcs_shadows’ = initial_allocation * 72 bytes = “_gcs_shadow_locks parameter” * 72 bytes
‘ges_resource’= initial_allocation * 288 bytes = “_lm_ress parameter ” * 288 bytes

注意这里计算出的仅仅是理论的最小值,实际值因为内存分配的机制所以必然会远大于计算值

如上例中 gcs resources = 114466 * 120 =13735920 << 实际值的16483232
gcs_shadows = 114466 * 72 = 8241552 << 实际值的11904560
ges_resource = 1000000 * 288 = 288000000 < 实际的288405768

一般来说我们将计算值 * 160% 后可以得出一个较为客观的估算值。

注意以上公式只是为我们在RAC环境中调优共享池的大小提供参考的依据。当我们观察v$resource_limit视图并认为需要提高GES、GSC资源的初始化分配数目时,可以参照上述方式估算出必要的shared_pool_size或sga_target大小。

Uninstall/Remove 11.2.0.2 Grid Infrastructure & Database in Linux

出于研究或者测试的目的我们可能已经在平台上安装了11gR2的Grid Infrastructure和RAC Database,因为GI部署的特殊性我们不能直接删除CRS_HOME和一些列脚本的方法来卸载GI和RAC Database软件,所幸在11gR2中Oracle提供了卸载软件的新特性:Deinstall,通过执行Deinstall脚本可以方便地删除Oracle软件产品在系统上的各类配置文件。

具体的卸载步骤如下:

1. 将平台上现有的数据库迁移走或者物理、逻辑地备份,如果该数据库已经没有任何价值的话使用DBCA删除该数据库及相关服务。

以oracle用户登录系统启动DBCA界面,并选择RAC database:

[oracle@vrh2 ~]$ dbca

deinstall_11gr2_rac_1

在step 1 of 2 :operations上选择删除数据库 delete a Database

deinstall_11gr2_rac_2

在 step 2 of 2 : List of cluster databases上选择所要删除的数据库

deinstall_11gr2_rac_3

逐一删除Cluster环境中所有的Database

2.
使用oracle用户登录任意节点并执行$ORACLE_HOME/deinstall目录下的deinstall脚本


SQL> select * from v$version;

BANNER
--------------------------------------------------------------------------------
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
PL/SQL Release 11.2.0.2.0 - Production
CORE    11.2.0.2.0      Production
TNS for Linux: Version 11.2.0.2.0 - Production
NLSRTL Version 11.2.0.2.0 - Production

SQL> select * from global_name;

GLOBAL_NAME
--------------------------------------------------------------------------------
www.askmac.cn


[root@vrh2 ~]# su - oracle

[oracle@vrh2 ~]$ cd $ORACLE_HOME/deinstall

[oracle@vrh2 deinstall]$ ./deinstall

Checking for required files and bootstrapping ...
Please wait ...
Location of logs /g01/oraInventory/logs/

############ ORACLE DEINSTALL & DECONFIG TOOL START ############

######################### CHECK OPERATION START #########################
Install check configuration START

Checking for existence of the Oracle home location /s01/orabase/product/11.2.0/dbhome_1
Oracle Home type selected for de-install is: RACDB
Oracle Base selected for de-install is: /s01/orabase
Checking for existence of central inventory location /g01/oraInventory
Checking for existence of the Oracle Grid Infrastructure home /g01/11.2.0/grid
The following nodes are part of this cluster: vrh1,vrh2

Install check configuration END

Skipping Windows and .NET products configuration check

Checking Windows and .NET products configuration END

Network Configuration check config START

Network de-configuration trace file location:
/g01/oraInventory/logs/netdc_check2011-08-31_11-19-25-PM.log

Specify all RAC listeners (do not include SCAN listener) that are to be de-configured [CRS_LISTENER]:

Network Configuration check config END

Database Check Configuration START

Database de-configuration trace file location: /g01/oraInventory/logs/databasedc_check2011-08-31_11-19-39-PM.log

Use comma as separator when specifying list of values as input

Specify the list of database names that are configured in this Oracle home []:
Database Check Configuration END

Enterprise Manager Configuration Assistant START

EMCA de-configuration trace file location: /g01/oraInventory/logs/emcadc_check2011-08-31_11-19-46-PM.log 

Enterprise Manager Configuration Assistant END
Oracle Configuration Manager check START
OCM check log file location : /g01/oraInventory/logs//ocm_check131.log
Oracle Configuration Manager check END

######################### CHECK OPERATION END #########################

####################### CHECK OPERATION SUMMARY #######################
Oracle Grid Infrastructure Home is: /g01/11.2.0/grid
The cluster node(s) on which the Oracle home de-installation will be performed are:vrh1,vrh2
Oracle Home selected for de-install is: /s01/orabase/product/11.2.0/dbhome_1
Inventory Location where the Oracle home registered is: /g01/oraInventory
Skipping Windows and .NET products configuration check
Following RAC listener(s) will be de-configured: CRS_LISTENER
No Enterprise Manager configuration to be updated for any database(s)
No Enterprise Manager ASM targets to update
No Enterprise Manager listener targets to migrate
Checking the config status for CCR
vrh1 : Oracle Home exists with CCR directory, but CCR is not configured
vrh2 : Oracle Home exists with CCR directory, but CCR is not configured
CCR check is finished
Do you want to continue (y - yes, n - no)? [n]: y
A log of this session will be written to: '/g01/oraInventory/logs/deinstall_deconfig2011-08-31_11-19-23-PM.out'
Any error messages from this session will be written to: '/g01/oraInventory/logs/deinstall_deconfig2011-08-31_11-19-23-PM.err'

######################## CLEAN OPERATION START ########################

Enterprise Manager Configuration Assistant START

EMCA de-configuration trace file location: /g01/oraInventory/logs/emcadc_clean2011-08-31_11-19-46-PM.log 

Updating Enterprise Manager ASM targets (if any)
Updating Enterprise Manager listener targets (if any)
Enterprise Manager Configuration Assistant END
Database de-configuration trace file location: /g01/oraInventory/logs/databasedc_clean2011-08-31_11-20-00-PM.log

Network Configuration clean config START

Network de-configuration trace file location: /g01/oraInventory/logs/netdc_clean2011-08-31_11-20-00-PM.log

De-configuring RAC listener(s): CRS_LISTENER

De-configuring listener: CRS_LISTENER
    Stopping listener: CRS_LISTENER
    Listener stopped successfully.
    Unregistering listener: CRS_LISTENER
    Listener unregistered successfully.
Listener de-configured successfully.

De-configuring Listener configuration file on all nodes...
Listener configuration file de-configured successfully.

De-configuring Naming Methods configuration file on all nodes...
Naming Methods configuration file de-configured successfully.

De-configuring Local Net Service Names configuration file on all nodes...
Local Net Service Names configuration file de-configured successfully.

De-configuring Directory Usage configuration file on all nodes...
Directory Usage configuration file de-configured successfully.

De-configuring backup files on all nodes...
Backup files de-configured successfully.

The network configuration has been cleaned up successfully.

Network Configuration clean config END

Oracle Configuration Manager clean START
OCM clean log file location : /g01/oraInventory/logs//ocm_clean131.log
Oracle Configuration Manager clean END
Removing Windows and .NET products configuration END
Oracle Universal Installer clean START

Detach Oracle home '/s01/orabase/product/11.2.0/dbhome_1' from the central inventory on the local node : Done

Delete directory '/s01/orabase/product/11.2.0/dbhome_1' on the local node : Done

Delete directory '/s01/orabase' on the local node : Done

Detach Oracle home '/s01/orabase/product/11.2.0/dbhome_1' from the central inventory on the remote nodes 'vrh1' : Done

Delete directory '/s01/orabase/product/11.2.0/dbhome_1' on the remote nodes 'vrh1' : Done

Delete directory '/s01/orabase' on the remote nodes 'vrh1' : Done

Oracle Universal Installer cleanup was successful.

Oracle Universal Installer clean END

Oracle install clean START

Clean install operation removing temporary directory '/tmp/deinstall2011-08-31_11-19-18PM' on node 'vrh2'
Clean install operation removing temporary directory '/tmp/deinstall2011-08-31_11-19-18PM' on node 'vrh1'

Oracle install clean END

######################### CLEAN OPERATION END #########################

####################### CLEAN OPERATION SUMMARY #######################
Following RAC listener(s) were de-configured successfully: CRS_LISTENER
Cleaning the config for CCR
As CCR is not configured, so skipping the cleaning of CCR configuration
CCR clean is finished
Skipping Windows and .NET products configuration clean
Successfully detached Oracle home '/s01/orabase/product/11.2.0/dbhome_1' from the central inventory on the local node.
Successfully deleted directory '/s01/orabase/product/11.2.0/dbhome_1' on the local node.
Successfully deleted directory '/s01/orabase' on the local node.
Successfully detached Oracle home '/s01/orabase/product/11.2.0/dbhome_1' from the central inventory on the remote nodes 'vrh1'.
Successfully deleted directory '/s01/orabase/product/11.2.0/dbhome_1' on the remote nodes 'vrh1'.
Successfully deleted directory '/s01/orabase' on the remote nodes 'vrh1'.
Oracle Universal Installer cleanup was successful.

Oracle deinstall tool successfully cleaned up temporary directories.
#######################################################################

############# ORACLE DEINSTALL & DECONFIG TOOL END #############

以上deinstall脚本会删除所有节点上的$ORACLE_HOME下的RDBMS软件,并从central inventory中将已经卸载的RDBMS软件注销,注意这种操作是不可逆的!

3.

使用root用户登录在所有节点上注意运行”$ORA_CRS_HOME/crs/install/rootcrs.pl -verbose -deconfig -force”的命令,注意在最后一个节点不要运行该命令。举例来说如果你有2个节点的话,就只要在一个节点上运行上述命令即可:

[root@vrh1 ~]# $ORA_CRS_HOME/crs/install/rootcrs.pl -verbose -deconfig -force

Using configuration parameter file: /g01/11.2.0/grid/crs/install/crsconfig_params
Network exists: 1/192.168.1.0/255.255.255.0/eth0, type static
VIP exists: /vrh1-vip/192.168.1.162/192.168.1.0/255.255.255.0/eth0, hosting node vrh1
VIP exists: /vrh2-vip/192.168.1.164/192.168.1.0/255.255.255.0/eth0, hosting node vrh2
VIP exists: /vrh3-vip/192.168.1.166/192.168.1.0/255.255.255.0/eth0, hosting node vrh3
GSD exists
ONS exists: Local port 6100, remote port 6200, EM port 2016
ACFS-9200: Supported
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'vrh1'
CRS-2677: Stop of 'ora.registry.acfs' on 'vrh1' succeeded
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'vrh1'
CRS-2673: Attempting to stop 'ora.crsd' on 'vrh1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'vrh1'
CRS-2673: Attempting to stop 'ora.oc4j' on 'vrh1'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'vrh1'
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'vrh1'
CRS-2673: Attempting to stop 'ora.SYSTEMDG.dg' on 'vrh1'
CRS-2677: Stop of 'ora.oc4j' on 'vrh1' succeeded
CRS-2672: Attempting to start 'ora.oc4j' on 'vrh2'
CRS-2676: Start of 'ora.oc4j' on 'vrh2' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'vrh1' succeeded
CRS-2677: Stop of 'ora.SYSTEMDG.dg' on 'vrh1' succeeded
CRS-2677: Stop of 'ora.FRA.dg' on 'vrh1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'vrh1'
CRS-2677: Stop of 'ora.asm' on 'vrh1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'vrh1' has completed
CRS-2677: Stop of 'ora.crsd' on 'vrh1' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'vrh1'
CRS-2673: Attempting to stop 'ora.evmd' on 'vrh1'
CRS-2673: Attempting to stop 'ora.asm' on 'vrh1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'vrh1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'vrh1'
CRS-2677: Stop of 'ora.asm' on 'vrh1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'vrh1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'vrh1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'vrh1' succeeded
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'vrh1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'vrh1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'vrh1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'vrh1'
CRS-2677: Stop of 'ora.cssd' on 'vrh1' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'vrh1'
CRS-2673: Attempting to stop 'ora.diskmon' on 'vrh1'
CRS-2677: Stop of 'ora.diskmon' on 'vrh1' succeeded
CRS-2677: Stop of 'ora.crf' on 'vrh1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'vrh1'
CRS-2677: Stop of 'ora.gipcd' on 'vrh1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'vrh1'
CRS-2677: Stop of 'ora.gpnpd' on 'vrh1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'vrh1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
Successfully deconfigured Oracle clusterware stack on this node

4.在最后的节点(last node)以root用户执行”$ORA_CRS_HOME/crs/install/rootcrs.pl -verbose -deconfig -force -lastnode”命令,该命令会清空OCR和Votedisk :

[root@vrh2 ~]# $ORA_CRS_HOME/crs/install/rootcrs.pl -verbose -deconfig -force -lastnode

Using configuration parameter file: /g01/11.2.0/grid/crs/install/crsconfig_params
CRS resources for listeners are still configured
Network exists: 1/192.168.1.0/255.255.255.0/eth0, type static
VIP exists: /vrh2-vip/192.168.1.164/192.168.1.0/255.255.255.0/eth0, hosting node vrh2
VIP exists: /vrh3-vip/192.168.1.166/192.168.1.0/255.255.255.0/eth0, hosting node vrh3
GSD exists
ONS exists: Local port 6100, remote port 6200, EM port 2016
ACFS-9200: Supported
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'vrh2'
CRS-2677: Stop of 'ora.registry.acfs' on 'vrh2' succeeded
CRS-2673: Attempting to stop 'ora.crsd' on 'vrh2'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'vrh2'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'vrh2'
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'vrh2'
CRS-2673: Attempting to stop 'ora.SYSTEMDG.dg' on 'vrh2'
CRS-2673: Attempting to stop 'ora.oc4j' on 'vrh2'
CRS-2677: Stop of 'ora.oc4j' on 'vrh2' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'vrh2' succeeded
CRS-2677: Stop of 'ora.SYSTEMDG.dg' on 'vrh2' succeeded
CRS-2677: Stop of 'ora.FRA.dg' on 'vrh2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'vrh2'
CRS-2677: Stop of 'ora.asm' on 'vrh2' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'vrh2' has completed
CRS-2677: Stop of 'ora.crsd' on 'vrh2' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'vrh2'
CRS-2673: Attempting to stop 'ora.evmd' on 'vrh2'
CRS-2673: Attempting to stop 'ora.asm' on 'vrh2'
CRS-2677: Stop of 'ora.asm' on 'vrh2' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'vrh2'
CRS-2677: Stop of 'ora.evmd' on 'vrh2' succeeded
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'vrh2' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'vrh2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'vrh2'
CRS-2677: Stop of 'ora.cssd' on 'vrh2' succeeded
CRS-2673: Attempting to stop 'ora.diskmon' on 'vrh2'
CRS-2677: Stop of 'ora.diskmon' on 'vrh2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'vrh2'
CRS-2676: Start of 'ora.cssdmonitor' on 'vrh2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'vrh2'
CRS-2672: Attempting to start 'ora.diskmon' on 'vrh2'
CRS-2676: Start of 'ora.diskmon' on 'vrh2' succeeded
CRS-2676: Start of 'ora.cssd' on 'vrh2' succeeded
CRS-4611: Successful deletion of voting disk +SYSTEMDG.
ASM de-configuration trace file location: /tmp/asmcadc_clean2011-08-31_11-55-52-PM.log
ASM Clean Configuration START
ASM Clean Configuration END

ASM with SID +ASM1 deleted successfully. Check /tmp/asmcadc_clean2011-08-31_11-55-52-PM.log for details.

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'vrh2'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'vrh2'
CRS-2673: Attempting to stop 'ora.ctssd' on 'vrh2'
CRS-2673: Attempting to stop 'ora.asm' on 'vrh2'
CRS-2677: Stop of 'ora.asm' on 'vrh2' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'vrh2' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'vrh2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'vrh2'
CRS-2677: Stop of 'ora.cssd' on 'vrh2' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'vrh2'
CRS-2673: Attempting to stop 'ora.diskmon' on 'vrh2'
CRS-2677: Stop of 'ora.gipcd' on 'vrh2' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'vrh2'
CRS-2677: Stop of 'ora.diskmon' on 'vrh2' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'vrh2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'vrh2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
Successfully deconfigured Oracle clusterware stack on this node

5.在任意节点以Grid Infrastructure拥有者用户执行”$ORA_CRS_HOME/deinstall/deinstall”脚本:

[root@vrh1 ~]# su - grid
[grid@vrh1 ~]$ cd $ORA_CRS_HOME
[grid@vrh1 grid]$ cd deinstall/

[grid@vrh1 deinstall]$ cat deinstall
#!/bin/sh
#
# $Header: install/utl/scripts/db/deinstall /main/3 2010/05/28 20:12:57 ssampath Exp $
#
# Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved.
#
#    NAME
#      deinstall - wrapper script that calls deinstall tool.
#
#    DESCRIPTION
#      This script will set all the necessary variables and call the tools
#      entry point.
#
#    NOTES
#
#
#    MODIFIED   (MM/DD/YY)
#    mwidjaja    04/29/10 - XbranchMerge mwidjaja_bug-9579184 from
#                           st_install_11.2.0.1.0
#    mwidjaja    04/15/10 - Added SHLIB_PATH for HP-PARISC
#    mwidjaja    01/14/10 - XbranchMerge mwidjaja_bug-9269768 from
#                           st_install_11.2.0.1.0
#    mwidjaja    01/14/10 - Fix help message for params
#    ssampath    12/24/09 - Fix for bug 9227535. Remove legacy version_check
#                           function
#    ssampath    12/01/09 - XbranchMerge ssampath_bug-9167533 from
#                           st_install_11.2.0.1.0
#    ssampath    11/30/09 - Set umask to 022.
#    prsubram    10/12/09 - XbranchMerge prsubram_bug-9005648 from main
#    prsubram    10/08/09 - Compute ARCHITECTURE_FLAG in the script
#    prsubram    09/15/09 - Setting LIBPATH for AIX
#    prsubram    09/10/09 - Add AIX specific code check java version
#    prsubram    09/10/09 - Change TOOL_DIR to BOOTSTRAP_DIR in java cmd
#                           invocation of bug#8874160
#    prsubram    09/08/09 - Change the default shell to /usr/xpg4/bin/sh on
#                           SunOS
#    prsubram    09/03/09 - Removing -d64 for client32 homes for the bug8859294
#    prsubram    06/22/09 - Resolve port specific id cmd issue
#    ssampath    06/02/09 - Fix for bug 8566942
#    ssampath    05/19/09 - Move removal of /tmp/deinstall to java
#                           code.
#    prsubram    04/30/09 - Fix for the bug#8474891
#    mwidjaja    04/29/09 - Added user check between the user running the
#                           script and inventory owner
#    ssampath    04/29/09 - Changes to make error message better when deinstall
#                           tool is invoked from inside ORACLE_HOME and -home
#                           is passed.
#    ssampath    04/15/09 - Fix for bug 8414555
#    prsubram    04/09/09 - LD_LIBRARY_PATH is ported for sol,hp-ux & aix
#    mwidjaja    03/26/09 - Disallow -home for running from OH
#    ssampath    03/24/09 - Fix for bug 8339519
#    wyou        02/25/09 - restructure the ohome check
#    wyou        02/25/09 - change the error msg for directory existance check
#    wyou        02/12/09 - add directory existance check
#    wyou        02/09/09 - add the check for the writablity for the oracle
#                           home passed-in
#    ssampath    01/21/09 - Add oui/lib to LD_LIBRARY_PATH
#    poosrini    01/07/09 - LOG related changes
#    ssampath    11/24/08 - Create /main/osds/unix branch
#    dchriste    10/30/08 - eliminate non-generic tools like 'cut'
#    ssampath    08/18/08 - Pickup srvm.jar from JLIB directory.
#    ssampath    07/30/08 - Add http_client.jar and OraCheckpoint.jar to
#                           CLASSPATH
#    ssampath    07/08/08 - assistantsCommon.jar and netca.jar location has
#                           changed.
#    ssampath    04/11/08 - If invoking the tool from installed home, JRE_HOME
#                           should be set to $OH/jdk/jre.
#    ssampath    04/09/08 - Add logic to instantiate ORA_CRS_HOME, JAVA_HOME
#                           etc.,
#    ssampath    04/03/08 - Pick up ldapjclnt11.jar
#    idai        04/03/08 - remove assistantsdc.jar and netcadc.jar
#    bktripat    02/23/07 -
#    khsingh     07/18/06 - add osdbagrp fix
#    khsingh     07/07/06 - fix regression
#    khsingh     06/20/06 - fix bug 5228203
#    bktripat    06/12/06 - Fix for bug 5246802
#    bktripat    05/08/06 -
#    khsingh     05/08/06 - fix tool to run from any parent directory
#    khsingh     05/08/06 - fix LD_LIBRARY_PATH to have abs. path
#    ssampath    05/01/06 - Fix for bug 5198219
#    bktripat    04/21/06 - Fix for bug 5074246
#    khsingh     04/11/06 - fix bug 5151658
#    khsingh     04/08/06 - Add WA for bugs 5006414 & 5093832
#    bktripat    02/08/06 - Fix for bug 5024086 & 5024061
#    bktripat    01/24/06 -
#    mstalin     01/23/06 - Add lib to pick libOsUtils.so
#    bktripat    01/19/06 - adding library changes
#    rahgupta    01/19/06 -
#    bktripat    01/19/06 -
#    mstalin     01/17/06 - Modify the assistants deconfig jar file name
#    rahgupta    01/17/06 - updating emcp classpath
#    khsingh     01/17/06 - export ORACLE_HOME
#    khsingh     01/17/06 - fix for CRS deconfig.
#    hying       01/17/06 - netcadc.jar
#    bktripat    01/16/06 -
#    ssampath    01/16/06 -
#    bktripat    01/11/06 -
#    clo         01/10/06 - add EMCP entries
#    hying       01/10/06 - netcaDeconfig.jar
#    mstalin     01/09/06 - Add OraPrereqChecks.jar
#    mstalin     01/09/06 -
#    khsingh     01/09/06 -
#    mstalin     01/09/06 - Add additional jars for assistants
#    ssampath    01/09/06 - removing parseOracleHome temporarily
#    ssampath    01/09/06 -
#    khsingh     01/08/06 - fix for CRS deconfig
#    ssampath    12/08/05 - added java version check
#    ssampath    12/08/05 - initial run,minor bugs fixed
#    ssampath    12/07/05 - Creation
#

#MACROS

if [ -z "$UNAME" ]; then UNAME="/bin/uname"; fi
if [ -z "$ECHO" ]; then ECHO="/bin/echo"; fi
if [ -z "$AWK" ]; then AWK="/bin/awk"; fi
if [ -z "$ID" ]; then ID="/usr/bin/id"; fi
if [ -z "$DIRNAME" ]; then DIRNAME="/usr/bin/dirname"; fi
if [ -z "$FILE" ]; then FILE="/usr/bin/file"; fi

if [ "`$UNAME`" = "SunOS" ]
then
    if [ -z "${_xpg4ShAvbl_deconfig}" ]
    then
        _xpg4ShAvbl_deconfig=1
        export _xpg4ShAvbl_deconfig
        /usr/xpg4/bin/sh $0 "$@"
        exit $?
    fi
        AWK="/usr/xpg4/bin/awk"
fi 

# Set umask to 022 always.
umask 022

INSTALLED_VERSION_FLAG=true
ARCHITECTURE_FLAG=64

TOOL_ARGS=$* # initialize this always.

# Since the OTN and the installed version of the tool is same, only way to
# differentiate is through the instantated variable ORA_CRS_HOME.  If it is
# NOT instantiated, then the tool is a downloaded version.
# Set HOME_VER to true based on the value of $INSTALLED_VERSION_FLAG
if [ x"$INSTALLED_VERSION_FLAG" = x"true" ]
then
   ORACLE_HOME=/g01/11.2.0/grid
   HOME_VER=1     # HOME_VER
   TOOL_ARGS="$ORACLE_HOME $TOOL_ARGS"
else
   HOME_VER=0
fi

# Save current working directory
CURR_DIR=`pwd`

# If CURR_DIR is different from TOOL_DIR get that location and cd into it.
TOOL_REL_PATH=`$DIRNAME $0`
cd $TOOL_REL_PATH

DOT=`$ECHO $TOOL_REL_PATH | $AWK -F'/' '{ print $1}'`

if [ "$DOT" = "." ];
then
  TOOL_DIR=$CURR_DIR/$TOOL_REL_PATH
elif [ `expr "$DOT" : '.*'` -gt 0 ];
then
  TOOL_DIR=$CURR_DIR/$TOOL_REL_PATH
else
  TOOL_DIR=$TOOL_REL_PATH
fi

# Check if this script is run as root.  If so, then error out.
# This is fix for bug 5024086.

RUID=`$ID|$AWK -F\( '{print $2}'|$AWK -F\) '{print $1}'`
if [ ${RUID} = "root" ];then
 $ECHO "You must not be logged in as root to run $0."
 $ECHO "Log in as Oracle user and rerun $0."
 exit $ROOT_USER
fi

# DEFINE FUNCTIONS BELOW
computeArchFlag() {
   TOOL_HOME=$1
   case `$UNAME` in
      HP-UX)
         if [ "`/usr/bin/file $TOOL_HOME/bin/kfod | $AWK -F\: '{print $2}' | $AWK -F\- '{print $2}' | $AWK '{print $1}'`" = "64" ];then
            ARCHITECTURE_FLAG="-d64"
         fi
      ;;
      AIX)
         if [ "`/usr/bin/file $TOOL_HOME/bin/kfod | $AWK -F\: '{print $2}' | $AWK '{print $1}' | $AWK -F\- '{print $1}'`" = "64" ];then
            ARCHITECTURE_FLAG="-d64"
         fi
      ;;
      *)
         if [ "`/usr/bin/file $TOOL_HOME/bin/kfod | $AWK -F\: '{print $2}' | $AWK '{print $2}' | $AWK -F\- '{print $1}'`" = "64" ];then
            ARCHITECTURE_FLAG="-d64"
         fi
      ;;
   esac
}

if [ $HOME_VER = 1 ];
then
   $ECHO "Checking for required files and bootstrapping ..."
   $ECHO "Please wait ..."
   TEMP_LOC=`$ORACLE_HOME/perl/bin/perl $ORACLE_HOME/deinstall/bootstrap.pl $HOME_VER $TOOL_ARGS`
   TOOL_DIR=$TEMP_LOC
else
   TEMP_LOC=`$TOOL_DIR/perl/bin/perl $TOOL_DIR/bootstrap.pl $HOME_VER $TOOL_ARGS`
fi

computeArchFlag $TOOL_DIR

$TOOL_DIR/perl/bin/perl $TOOL_DIR/deinstall.pl $HOME_VER $TEMP_LOC $TOOL_DIR $ARCHITECTURE_FLAG $TOOL_ARGS

[grid@vrh1 deinstall]$ ./deinstall

Checking for required files and bootstrapping ...
Please wait ...
Location of logs /tmp/deinstall2011-08-31_11-59-55PM/logs/

############ ORACLE DEINSTALL & DECONFIG TOOL START ############

######################### CHECK OPERATION START #########################
Install check configuration START

Checking for existence of the Oracle home location /g01/11.2.0/grid
Oracle Home type selected for de-install is: CRS
Oracle Base selected for de-install is: /g01/orabase
Checking for existence of central inventory location /g01/oraInventory
Checking for existence of the Oracle Grid Infrastructure home /g01/11.2.0/grid
The following nodes are part of this cluster: vrh1,vrh2,vrh3

Install check configuration END

Skipping Windows and .NET products configuration check

Checking Windows and .NET products configuration END

Traces log file: /tmp/deinstall2011-08-31_11-59-55PM/logs//crsdc.log
Enter an address or the name of the virtual IP used on node "vrh1"[vrh1-vip]
 > 

The following information can be collected by running "/sbin/ifconfig -a" on node "vrh1"
Enter the IP netmask of Virtual IP "192.168.1.162" on node "vrh1"[255.255.255.0]
 > 

Enter the network interface name on which the virtual IP address "192.168.1.162" is active
 > 

Enter an address or the name of the virtual IP used on node "vrh2"[vrh2-vip]
 > 

The following information can be collected by running "/sbin/ifconfig -a" on node "vrh2"
Enter the IP netmask of Virtual IP "192.168.1.164" on node "vrh2"[255.255.255.0]
 > 

Enter the network interface name on which the virtual IP address "192.168.1.164" is active
 > 

Enter an address or the name of the virtual IP used on node "vrh3"[vrh3-vip]
 > 

The following information can be collected by running "/sbin/ifconfig -a" on node "vrh3"
Enter the IP netmask of Virtual IP "192.168.1.166" on node "vrh3"[255.255.255.0]
 > 

Enter the network interface name on which the virtual IP address "192.168.1.166" is active
 > 

Enter an address or the name of the virtual IP[]
 > 

Network Configuration check config START

Network de-configuration trace file location: /tmp/deinstall2011-08-31_11-59-55PM/logs/
netdc_check2011-09-01_12-01-50-AM.log

Specify all RAC listeners (do not include SCAN listener) that are to be de-configured [LISTENER,LISTENER_SCAN1]:

Network Configuration check config END

Asm Check Configuration START

ASM de-configuration trace file location: /tmp/deinstall2011-08-31_11-59-55PM/logs/
asmcadc_check2011-09-01_12-01-51-AM.log

ASM configuration was not detected in this Oracle home. Was ASM configured in this Oracle home (y|n) [n]:
ASM was not detected in the Oracle Home

######################### CHECK OPERATION END #########################

####################### CHECK OPERATION SUMMARY #######################
Oracle Grid Infrastructure Home is: /g01/11.2.0/grid
The cluster node(s) on which the Oracle home de-installation will be performed are:vrh1,vrh2,vrh3
Oracle Home selected for de-install is: /g01/11.2.0/grid
Inventory Location where the Oracle home registered is: /g01/oraInventory
Skipping Windows and .NET products configuration check
Following RAC listener(s) will be de-configured: LISTENER,LISTENER_SCAN1
ASM was not detected in the Oracle Home
Do you want to continue (y - yes, n - no)? [n]: y
A log of this session will be written to: '/tmp/deinstall2011-08-31_11-59-55PM/logs/deinstall_deconfig2011-09-01_12-01-15-AM.out'
Any error messages from this session will be written to: '/tmp/deinstall2011-08-31_11-59-55PM/logs/deinstall_deconfig2011-09-01_12-01-15-AM.err'

######################## CLEAN OPERATION START ########################
ASM de-configuration trace file location: /tmp/deinstall2011-08-31_11-59-55PM/logs/asmcadc_clean2011-09-01_12-02-00-AM.log
ASM Clean Configuration END

Network Configuration clean config START

Network de-configuration trace file location: /tmp/deinstall2011-08-31_11-59-55PM/logs/netdc_clean2011-09-01_12-02-00-AM.log

De-configuring RAC listener(s): LISTENER,LISTENER_SCAN1

De-configuring listener: LISTENER
    Stopping listener: LISTENER
    Warning: Failed to stop listener. Listener may not be running.
Listener de-configured successfully.

De-configuring listener: LISTENER_SCAN1
    Stopping listener: LISTENER_SCAN1
    Warning: Failed to stop listener. Listener may not be running.
Listener de-configured successfully.

De-configuring Naming Methods configuration file on all nodes...
Naming Methods configuration file de-configured successfully.

De-configuring Local Net Service Names configuration file on all nodes...
Local Net Service Names configuration file de-configured successfully.

De-configuring Directory Usage configuration file on all nodes...
Directory Usage configuration file de-configured successfully.

De-configuring backup files on all nodes...
Backup files de-configured successfully.

The network configuration has been cleaned up successfully.

Network Configuration clean config END

---------------------------------------->

The deconfig command below can be executed in parallel on all the remote nodes.
Execute the command on  the local node after the execution completes on all the remote nodes.

Run the following command as the root user or the administrator on node "vrh3".

/tmp/deinstall2011-08-31_11-59-55PM/perl/bin/perl -I/tmp/deinstall2011-08-31_11-59-55PM/perl/lib
-I/tmp/deinstall2011-08-31_11-59-55PM/crs/install /tmp/deinstall2011-08-31_11-59-55PM/crs/install/rootcrs.pl
-force  -deconfig -paramfile "/tmp/deinstall2011-08-31_11-59-55PM/response/deinstall_Ora11g_gridinfrahome1.rsp"

Run the following command as the root user or the administrator on node "vrh2".

/tmp/deinstall2011-08-31_11-59-55PM/perl/bin/perl -I/tmp/deinstall2011-08-31_11-59-55PM/perl/lib
-I/tmp/deinstall2011-08-31_11-59-55PM/crs/install /tmp/deinstall2011-08-31_11-59-55PM/crs/install/rootcrs.pl -force
-deconfig -paramfile "/tmp/deinstall2011-08-31_11-59-55PM/response/deinstall_Ora11g_gridinfrahome1.rsp"

Run the following command as the root user or the administrator on node "vrh1".

/tmp/deinstall2011-08-31_11-59-55PM/perl/bin/perl -I/tmp/deinstall2011-08-31_11-59-55PM/perl/lib
-I/tmp/deinstall2011-08-31_11-59-55PM/crs/install /tmp/deinstall2011-08-31_11-59-55PM/crs/install/rootcrs.pl
-force  -deconfig -paramfile "/tmp/deinstall2011-08-31_11-59-55PM/response/deinstall_Ora11g_gridinfrahome1.rsp"
-lastnode

Press Enter after you finish running the above commands

执行deinstall过程中会要求以root用户在所有平台上执行相关命令

su - root

[root@vrh3 ~]# /tmp/deinstall2011-08-31_11-59-55PM/perl/bin/perl -I/tmp/deinstall2011-08-31_11-59-55PM/perl/lib
-I/tmp/deinstall2011-08-31_11-59-55PM/crs/install /tmp/deinstall2011-08-31_11-59-55PM/crs/install/rootcrs.pl -force
-deconfig -paramfile "/tmp/deinstall2011-08-31_11-59-55PM/response/deinstall_Ora11g_gridinfrahome1.rsp"
Using configuration parameter file: /tmp/deinstall2011-08-31_11-59-55PM/response/deinstall_Ora11g_gridinfrahome1.rsp
PRCR-1119 : Failed to look up CRS resources of ora.cluster_vip_net1.type type
PRCR-1068 : Failed to query resources
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.gsd is registered
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.ons is registered
Cannot communicate with crsd

ACFS-9200: Supported
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Stop failed, or completed with errors.
CRS-4544: Unable to connect to OHAS
CRS-4000: Command Stop failed, or completed with errors.
Successfully deconfigured Oracle clusterware stack on this node

[root@vrh2 ~]# /tmp/deinstall2011-08-31_11-59-55PM/perl/bin/perl -I/tmp/deinstall2011-08-31_11-59-55PM/perl/lib -I/tmp/deinstall2011-08-31_11-59-55PM/crs/install /tmp/deinstall2011-08-31_11-59-55PM/crs/install/rootcrs.pl -force  -deconfig -paramfile
"/tmp/deinstall2011-08-31_11-59-55PM/response/deinstall_Ora11g_gridinfrahome1.rsp"
Using configuration parameter file: /tmp/deinstall2011-08-31_11-59-55PM/response/deinstall_Ora11g_gridinfrahome1.rsp
Usage: srvctl [command] [object] []
    commands: enable|disable|start|stop|status|add|remove|modify|getenv|setenv|unsetenv|config
    objects: database|service|asm|diskgroup|listener|home|ons
For detailed help on each command and object and its options use:
  srvctl [command] -h or
  srvctl [command] [object] -h
PRKO-2012 : nodeapps object is not supported in Oracle Restart
ACFS-9200: Supported
CRS-4047: No Oracle Clusterware components configured.
CRS-4000: Command Stop failed, or completed with errors.
CRS-4047: No Oracle Clusterware components configured.
CRS-4000: Command Stop failed, or completed with errors.
You must kill crs processes or reboot the system to properly
cleanup the processes started by Oracle clusterware
ACFS-9313: No ADVM/ACFS installation detected.
Either /etc/oracle/olr.loc does not exist or is not readable
Make sure the file exists and it has read and execute access
Failure in execution (rc=-1, 256, No such file or directory) for command 1 /etc/init.d/ohasd deinstall
error: package cvuqdisk is not installed
Successfully deconfigured Oracle clusterware stack on this node

[root@vrh1 ~]# /tmp/deinstall2011-08-31_11-59-55PM/perl/bin/perl -I/tmp/deinstall2011-08-31_11-59-55PM/perl/lib
-I/tmp/deinstall2011-08-31_11-59-55PM/crs/install /tmp/deinstall2011-08-31_11-59-55PM/crs/install/rootcrs.pl -force
-deconfig -paramfile "/tmp/deinstall2011-08-31_11-59-55PM/response/deinstall_Ora11g_gridinfrahome1.rsp" -lastnode
Using configuration parameter file: /tmp/deinstall2011-08-31_11-59-55PM/response/deinstall_Ora11g_gridinfrahome1.rsp
Adding daemon to inittab
crsexcl failed to start
Failed to start the Clusterware. Last 20 lines of the alert log follow:
2011-08-31 23:36:55.813
[ctssd(4067)]CRS-2408:The clock on host vrh1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
2011-08-31 23:38:23.855
[ctssd(4067)]CRS-2408:The clock on host vrh1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
2011-08-31 23:39:03.873
[ctssd(4067)]CRS-2408:The clock on host vrh1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
2011-08-31 23:39:11.707
[/g01/11.2.0/grid/bin/orarootagent.bin(4559)]CRS-5822:Agent '/g01/11.2.0/grid/bin/orarootagent_root'
disconnected from server. Details at (:CRSAGF00117:) {0:2:27} in
/g01/11.2.0/grid/log/vrh1/agent/crsd/orarootagent_root/orarootagent_root.log.
2011-08-31 23:39:12.725
[ctssd(4067)]CRS-2405:The Cluster Time Synchronization Service on host vrh1 is shutdown by user
2011-08-31 23:39:12.764
[mdnsd(3868)]CRS-5602:mDNS service stopping by request.
2011-08-31 23:39:13.987
[/g01/11.2.0/grid/bin/orarootagent.bin(3892)]CRS-5016:Process "/g01/11.2.0/grid/bin/acfsload"
spawned by agent "/g01/11.2.0/grid/bin/orarootagent.bin" for action "check" failed:
details at "(:CLSN00010:)" in "/g01/11.2.0/grid/log/vrh1/agent/ohasd/orarootagent_root/orarootagent_root.log"
2011-08-31 23:39:27.121
[cssd(3968)]CRS-1603:CSSD on node vrh1 shutdown by user.
2011-08-31 23:39:27.130
[ohasd(3639)]CRS-2767:Resource state recovery not attempted for 'ora.cssdmonitor' as its target state is OFFLINE
2011-08-31 23:39:31.926
[gpnpd(3880)]CRS-2329:GPNPD on node vrh1 shutdown. 

Usage: srvctl [command] [object] []
    commands: enable|disable|start|stop|status|add|remove|modify|getenv|setenv|unsetenv|config
    objects: database|service|asm|diskgroup|listener|home|ons
For detailed help on each command and object and its options use:
  srvctl [command] -h or
  srvctl [command] [object] -h
PRKO-2012 : scan_listener object is not supported in Oracle Restart
Usage: srvctl [command] [object] []
    commands: enable|disable|start|stop|status|add|remove|modify|getenv|setenv|unsetenv|config
    objects: database|service|asm|diskgroup|listener|home|ons
For detailed help on each command and object and its options use:
  srvctl [command] -h or
  srvctl [command] [object] -h
PRKO-2012 : scan_listener object is not supported in Oracle Restart
Usage: srvctl [command] [object] []
    commands: enable|disable|start|stop|status|add|remove|modify|getenv|setenv|unsetenv|config
    objects: database|service|asm|diskgroup|listener|home|ons
For detailed help on each command and object and its options use:
  srvctl [command] -h or
  srvctl [command] [object] -h
PRKO-2012 : scan object is not supported in Oracle Restart
Usage: srvctl [command] [object] []
    commands: enable|disable|start|stop|status|add|remove|modify|getenv|setenv|unsetenv|config
    objects: database|service|asm|diskgroup|listener|home|ons
For detailed help on each command and object and its options use:
  srvctl [command] -h or
  srvctl [command] [object] -h
PRKO-2012 : scan object is not supported in Oracle Restart
Usage: srvctl [command] [object] []
    commands: enable|disable|start|stop|status|add|remove|modify|getenv|setenv|unsetenv|config
    objects: database|service|asm|diskgroup|listener|home|ons
For detailed help on each command and object and its options use:
  srvctl [command] -h or
  srvctl [command] [object] -h
PRKO-2012 : nodeapps object is not supported in Oracle Restart
ACFS-9200: Supported
CRS-4047: No Oracle Clusterware components configured.
CRS-4000: Command Stop failed, or completed with errors.
CRS-4047: No Oracle Clusterware components configured.
CRS-4000: Command Delete failed, or completed with errors.
CRS-4047: No Oracle Clusterware components configured.
CRS-4000: Command Stop failed, or completed with errors.
CRS-4047: No Oracle Clusterware components configured.
CRS-4000: Command Modify failed, or completed with errors.
Adding daemon to inittab
crsexcl failed to start
Failed to start the Clusterware. Last 20 lines of the alert log follow:
[ctssd(4067)]CRS-2408:The clock on host vrh1 has been updated by the Cluster Time
Synchronization Service to be synchronous with the mean cluster time.
2011-08-31 23:38:23.855
[ctssd(4067)]CRS-2408:The clock on host vrh1 has been updated by the Cluster Time
Synchronization Service to be synchronous with the mean cluster time.
2011-08-31 23:39:03.873
[ctssd(4067)]CRS-2408:The clock on host vrh1 has been updated by the Cluster Time
Synchronization Service to be synchronous with the mean cluster time.
2011-08-31 23:39:11.707
[/g01/11.2.0/grid/bin/orarootagent.bin(4559)]CRS-5822:Agent '/g01/11.2.0/grid/bin/orarootagent_root'
disconnected from server. Details at (:CRSAGF00117:) {0:2:27} in
/g01/11.2.0/grid/log/vrh1/agent/crsd/orarootagent_root/orarootagent_root.log.
2011-08-31 23:39:12.725
[ctssd(4067)]CRS-2405:The Cluster Time Synchronization Service on host vrh1 is shutdown by user
2011-08-31 23:39:12.764
[mdnsd(3868)]CRS-5602:mDNS service stopping by request.
2011-08-31 23:39:13.987
[/g01/11.2.0/grid/bin/orarootagent.bin(3892)]CRS-5016:Process
"/g01/11.2.0/grid/bin/acfsload" spawned by agent "/g01/11.2.0/grid/bin/orarootagent.bin" for action
"check" failed: details at "(:CLSN00010:)" in
"/g01/11.2.0/grid/log/vrh1/agent/ohasd/orarootagent_root/orarootagent_root.log"
2011-08-31 23:39:27.121
[cssd(3968)]CRS-1603:CSSD on node vrh1 shutdown by user.
2011-08-31 23:39:27.130
[ohasd(3639)]CRS-2767:Resource state recovery not attempted for 'ora.cssdmonitor' as its target state is OFFLINE
2011-08-31 23:39:31.926
[gpnpd(3880)]CRS-2329:GPNPD on node vrh1 shutdown.
[client(13099)]CRS-10001:01-Sep-11 00:11 ACFS-9200: Supported

CRS-4047: No Oracle Clusterware components configured.
CRS-4000: Command Delete failed, or completed with errors.
crsctl delete for vds in SYSTEMDG ... failed
CRS-4047: No Oracle Clusterware components configured.
CRS-4000: Command Delete failed, or completed with errors.
CRS-4047: No Oracle Clusterware components configured.
CRS-4000: Command Stop failed, or completed with errors.
ACFS-9313: No ADVM/ACFS installation detected.
Either /etc/oracle/olr.loc does not exist or is not readable
Make sure the file exists and it has read and execute access
Failure in execution (rc=-1, 256, No such file or directory) for command 1 /etc/init.d/ohasd deinstall
error: package cvuqdisk is not installed
Successfully deconfigured Oracle clusterware stack on this node

回到最初运行deintall的终端摁下回车

The deconfig command below can be executed in parallel on all the remote nodes.
Execute the command on  the local node after the execution completes on all the remote nodes.

Press Enter after you finish running the above commands

<----------------------------------------

Removing Windows and .NET products configuration END
Oracle Universal Installer clean START

Detach Oracle home '/g01/11.2.0/grid' from the central inventory on the local node : Done

Delete directory '/g01/11.2.0/grid' on the local node : Done

Delete directory '/g01/oraInventory' on the local node : Done

Delete directory '/g01/orabase' on the local node : Done

Detach Oracle home '/g01/11.2.0/grid' from the central inventory on the remote nodes 'vrh3,vrh2' : Done

Delete directory '/g01/11.2.0/grid' on the remote nodes 'vrh2,vrh3' : Done

Delete directory '/g01/oraInventory' on the remote nodes 'vrh3' : Done

Delete directory '/g01/oraInventory' on the remote nodes 'vrh2' : Failed <<<<

The directory '/g01/oraInventory' could not be deleted on the nodes 'vrh2'.
Delete directory '/g01/orabase' on the remote nodes 'vrh2' : Done

Delete directory '/g01/orabase' on the remote nodes 'vrh3' : Done

Oracle Universal Installer cleanup completed with errors.

Oracle Universal Installer clean END

Oracle install clean START

Clean install operation removing temporary directory '/tmp/deinstall2011-08-31_11-59-55PM' on node 'vrh1'
Clean install operation removing temporary directory '/tmp/deinstall2011-08-31_11-59-55PM' on node 'vrh2'
Clean install operation removing temporary directory '/tmp/deinstall2011-08-31_11-59-55PM' on node 'vrh3'

Oracle install clean END

######################### CLEAN OPERATION END #########################

####################### CLEAN OPERATION SUMMARY #######################
Following RAC listener(s) were de-configured successfully: LISTENER,LISTENER_SCAN1
Oracle Clusterware is stopped and successfully de-configured on node "vrh3"
Oracle Clusterware is stopped and successfully de-configured on node "vrh2"
Oracle Clusterware is stopped and successfully de-configured on node "vrh1"
Oracle Clusterware is stopped and de-configured successfully.
Skipping Windows and .NET products configuration clean
Successfully detached Oracle home '/g01/11.2.0/grid' from the central inventory on the local node.
Successfully deleted directory '/g01/11.2.0/grid' on the local node.
Successfully deleted directory '/g01/oraInventory' on the local node.
Successfully deleted directory '/g01/orabase' on the local node.
Successfully detached Oracle home '/g01/11.2.0/grid' from the central inventory on the remote nodes 'vrh3,vrh2'.
Successfully deleted directory '/g01/11.2.0/grid' on the remote nodes 'vrh2,vrh3'.
Successfully deleted directory '/g01/oraInventory' on the remote nodes 'vrh3'.
Failed to delete directory '/g01/oraInventory' on the remote nodes 'vrh2'.
Successfully deleted directory '/g01/orabase' on the remote nodes 'vrh2'.
Successfully deleted directory '/g01/orabase' on the remote nodes 'vrh3'.
Oracle Universal Installer cleanup completed with errors.

Run 'rm -rf /etc/oraInst.loc' as root on node(s) 'vrh1,vrh3' at the end of the session.

Run 'rm -rf /opt/ORCLfmap' as root on node(s) 'vrh1 vrh3 vrh2 ' at the end of the session.
Oracle deinstall tool successfully cleaned up temporary directories.
#######################################################################

############# ORACLE DEINSTALL & DECONFIG TOOL END #############

deintall运行完成后会提示让你在必要的节点上运行”rm -rf /etc/oraInst.loc”和”rm -rf /opt/ORCLfmap”,照做即可。
以上脚本运行完成后各节点上的GI已被删除,且/etc/inittab文件已还原为非GI版,/etc/init.d下的CRS相关脚本也已相应删除。

 

Reference:

<How to Proceed from Failed 11gR2 Grid Infrastructure (CRS) Installation [ID 942166.1]>

Script:RAC Failover检验脚本loop.sh

以下脚本可以用于验证RAC中FAILOVER的可用性:

loop.sh
  nohup sqlplus su/su@failover @verify.sql &
     sleep 1
  nohup sqlplus su/su@failover @verify.sql &
     sleep 1
  nohup sqlplus su/su@failover @verify.sql &
     sleep 1
  nohup sqlplus su/su@failover @verify.sql &
     sleep 1

verify.sql (检验SQL)
  REM  set pagesize 1000
  REM  the following query is for TAF connection verification
  col sid format 999
  col serial# format 9999999
  col failover_type format a13
  col failover_method format a15
  col failed_over format a11
  select sid, serial#, failover_type, failover_method, failed_over
    from v$session where username = 'SU';

  REM  the following query is for load balancing verification
  select instance_name from v$instance;
  exit

  REM you can also combine two queries:
  col inst_id format 999
  col sid format 999
  col serial# format 9999999
  col failover_type format a13
  col failover_method format a15
  col failed_over format a11

  select inst_id, sid, serial#, failover_type, failover_method,
         failed_over from gv$session where username = 'SU';

  REM  a simple select to see the distribution of users when testing 
  REM  connection load balancing

  select inst_id, count(*) from gv$session group by inst_id;

用法:
./loop.sh

Script:优化crs_stat命令的输出

在10g RAC中我们常用crs_stat命令查看CRS资源的状态,但是crs_stat命令的输出并不完整。可以通过以下脚本来优化crs_stat的输出:

--------------------------- Begin Shell Script -------------------------------

#!/usr/bin/ksh
#
# Sample 10g CRS resource status query script
#
# Description:
#    - Returns formatted version of crs_stat -t, in tabular
#      format, with the complete rsc names and filtering keywords
#   - The argument, $RSC_KEY, is optional and if passed to the script, will
#     limit the output to HA resources whose names match $RSC_KEY.
# Requirements:
#   - $ORA_CRS_HOME should be set in your environment 

RSC_KEY=$1
QSTAT=-u
AWK=/usr/xpg4/bin/awk    # if not available use /usr/bin/awk

# Table header:echo ""
$AWK \
  'BEGIN {printf "%-45s %-10s %-18s\n", "HA Resource", "Target", "State";
          printf "%-45s %-10s %-18s\n", "-----------", "------", "-----";}'

# Table body:
$ORA_CRS_HOME/bin/crs_stat $QSTAT | $AWK \
 'BEGIN { FS="="; state = 0; }
  $1~/NAME/ && $2~/'$RSC_KEY'/ {appname = $2; state=1};
  state == 0 {next;}
  $1~/TARGET/ && state == 1 {apptarget = $2; state=2;}
  $1~/STATE/ && state == 2 {appstate = $2; state=3;}
  state == 3 {printf "%-45s %-10s %-18s\n", appname, apptarget, appstate; state=0;}'

--------------------------- End Shell Script -------------------------------

利用Cluster Verify Utility工具体验RAC最佳实践

Cluster Verification Utilit(CVU)是Oracle所推荐的一种集群检验工具。该检验工具帮助用户在Cluter部署的各个阶段验证集群的重要组件,这些阶段包括硬件搭建、Clusterware的安装、RDBMS的安装、存储等等。我们既可以在Cluster安装之前使用CVU来帮我们检验所配置的环境正确可用,也可以在软件安装完成后使用CVU来做对集群的验收。

CVU提供了一种可扩展的框架,其所实施的常规检验活动是独立于具体的平台,并且向存储和网络的检验提供了厂商接口(Vendor Interface)。
CVU工具不依赖于其他Oracle软件,仅使用命令cluvfy,如cluvfy stage -pre crsinst -n vrh1,vrh2。

cluvfy的部署十分简单,在本地节点安装后,该工具在运行过程中会自动部署到远程主机上。具体的自动部署流程如下:

  1. 用户在本地节点安装CVU
  2. 用户针对多个节点实施Verify检验命令
  3. CVU工具将拷贝自身必要的文件到远程节点
  4. CVU会在所有节点执行检验任务并生成报告

CVU工具可以为我们提供以下功能:

  1. 验证Cluster集群是否规范配置以便后续的RAC安装、配置和操作顺利
  2. 全类型的验证
  3. 非破坏性的验证
  4. 提供了易于使用的接口
  5. 支持各种平台和配置的RAC,明确完善的统一行为方式

注意不要误解cluvfy的作用,它仅仅是一个检验者,而不负责实际的配置或修复工作:

  1. cluvfy不支持任何类型的cluster或RAC操作
  2. 在检验到问题或失败后,cluvfy不会采取任何修正行为
  3. cluvfy不是性能调优或监控工具
  4. cluvfy不会尝试帮助你验证RAC数据库的内部结构

RAC的实际部署可以被逻辑地区分为多个操作阶段,这些阶段被称作是”stage”,在实际的部署过程中每一个stage由一系列的操作组成。每一个stage的都有自身的预检查(pre-check)和验收检查(post-check),如图:

cluvfy_stage_list

 

我们会在CRS和RAC数据库的安装过程中具体使用Cluster Verify Utility的不同”stage”,各种不同的stage可以使用cluvfy stage -list命令列出:

cluvfy stage -list

USAGE:
cluvfy stage {-pre|-post} <stage-name> <stage-specific options>  [-verbose]

Valid Stages are:
      -pre cfs        : pre-check for CFS setup
      -pre crsinst    : pre-check for CRS installation
      -pre acfscfg    : pre-check for ACFS Configuration.
      -pre dbinst     : pre-check for database installation
      -pre dbcfg      : pre-check for database configuration
      -pre hacfg      : pre-check for HA configuration
      -pre nodeadd    : pre-check for node addition.
      -post hwos      : post-check for hardware and operating system
      -post cfs       : post-check for CFS setup
      -post crsinst   : post-check for CRS installation
      -post acfscfg   : post-check for ACFS Configuration.
      -post hacfg     : post-check for HA configuration
      -post nodeadd   : post-check for node addition.
      -post nodedel   : post-check for node deletion.

在RAC Cluster中独立的子系统或者模块被称作组件(component),集群组件的可用性、完整性、稳定性以及其他一些表现均可以使用CVU来验证。简单如某个存储设备、复杂如包含CRSD、EVMD、CSSD、OCR等多个子组件的CRS stack都可以被认为是一个组件。

在CRS运行过程中为了检验Cluster中的某个特定组件(component)或者为了独立诊断某个Cluster集群子系统,需要用到合适的组件检查命令;各种不同组件的检验可以使用cluvfy comp -list命令列出:

 cluvfy comp -list

USAGE:
cluvfy comp     [-verbose]

Valid Components are:
      nodereach       : checks reachability between nodes
      nodecon         : checks node connectivity
      cfs             : checks CFS integrity
      ssa             : checks shared storage accessibility
      space           : checks space availability
      sys             : checks minimum system requirements
      clu             : checks cluster integrity
      clumgr          : checks cluster manager integrity
      ocr             : checks OCR integrity
      olr             : checks OLR integrity
      ha              : checks HA integrity
      crs             : checks CRS integrity
      nodeapp         : checks node applications existence
      admprv          : checks administrative privileges
      peer            : compares properties with peers
      software        : checks software distribution
      acfs            : checks ACFS integrity
      asm             : checks ASM integrity
      gpnp            : checks GPnP integrity
      gns             : checks GNS integrity
      scan            : checks SCAN configuration
      ohasd           : checks OHASD integrity
      clocksync       : checks Clock Synchronization
      vdisk           : checks Voting Disk configuration and UDEV settings
      dhcp            : Checks DHCP configuration
      dns             : Checks DNS configuration

我们可以从OTN的CVU专栏内下载到最新版本的cluvfy,如果没有特殊的要求那么我们总是推荐使用最新版本。一般在完成RAC安装后也可以从以下2个路径找到cluvfy:

Clusterware Home
<crs_home>/bin/cluvfy

Oracle Home
$ORACLE_HOME/bin/cluvfy

使用最为频繁的几个cluvfy命令如下:

Verify the hardware and operating system:检验操作系统和硬件的配置

cluvfy stage -post hwos -n vrh1,vrh2

cluvfy stage -post hwos -n vrh1,vrh2

Performing post-checks for hardware and operating system setup 

Checking node reachability...
Node reachability check passed from node "vrh1"

Checking user equivalence...
User equivalence check passed for user "grid"

Checking node connectivity...

Checking hosts config file...

Verification of the hosts config file successful

Node connectivity passed for subnet "192.168.1.0" with node(s) vrh2,vrh1
TCP connectivity check passed for subnet "192.168.1.0"

Node connectivity passed for subnet "192.168.2.0" with node(s) vrh2,vrh1
TCP connectivity check passed for subnet "192.168.2.0"

Node connectivity passed for subnet "169.254.0.0" with node(s) vrh2,vrh1
TCP connectivity check passed for subnet "169.254.0.0"

Interfaces found on subnet "192.168.1.0" that are likely candidates for VIP are:
vrh2 eth0:192.168.1.163 eth0:192.168.1.164 eth0:192.168.1.166
vrh1 eth0:192.168.1.161 eth0:192.168.1.190 eth0:192.168.1.162

Interfaces found on subnet "169.254.0.0" that are likely candidates for VIP are:
vrh2 eth1:169.254.8.92
vrh1 eth1:169.254.175.195

Interfaces found on subnet "192.168.2.0" that are likely candidates for a private interconnect are:
vrh2 eth1:192.168.2.19
vrh1 eth1:192.168.2.18

Node connectivity check passed

Check for multiple users with UID value 0 passed
Time zone consistency check passed

Checking shared storage accessibility...

  Disk                                  Sharing Nodes (2 in count)
  ------------------------------------  ------------------------
  /dev/sdb                              vrh2 vrh1               

  Disk                                  Sharing Nodes (2 in count)
  ------------------------------------  ------------------------
  /dev/sdc                              vrh2 vrh1               

  Disk                                  Sharing Nodes (2 in count)
  ------------------------------------  ------------------------
  /dev/sdd                              vrh2 vrh1               

  Disk                                  Sharing Nodes (2 in count)
  ------------------------------------  ------------------------
  /dev/sde                              vrh2 vrh1               

  Disk                                  Sharing Nodes (2 in count)
  ------------------------------------  ------------------------
  /dev/sdf                              vrh2 vrh1               

  Disk                                  Sharing Nodes (2 in count)
  ------------------------------------  ------------------------
  /dev/sdg                              vrh2 vrh1               

  Disk                                  Sharing Nodes (2 in count)
  ------------------------------------  ------------------------
  /dev/sdh                              vrh2 vrh1               

  Disk                                  Sharing Nodes (2 in count)
  ------------------------------------  ------------------------
  /dev/sdi                              vrh2 vrh1               

  Disk                                  Sharing Nodes (2 in count)
  ------------------------------------  ------------------------
  /dev/sdj                              vrh2 vrh1               

  Disk                                  Sharing Nodes (2 in count)
  ------------------------------------  ------------------------
  /dev/sdk                              vrh2 vrh1               

  Disk                                  Sharing Nodes (2 in count)
  ------------------------------------  ------------------------
  /dev/sdl                              vrh2 vrh1               

  Disk                                  Sharing Nodes (2 in count)
  ------------------------------------  ------------------------
  /dev/sdm                              vrh2 vrh1               

  Disk                                  Sharing Nodes (2 in count)
  ------------------------------------  ------------------------
  /dev/sdn                              vrh2 vrh1               

  Disk                                  Sharing Nodes (2 in count)
  ------------------------------------  ------------------------
  /dev/sdo                              vrh2 vrh1               

  Disk                                  Sharing Nodes (2 in count)
  ------------------------------------  ------------------------
  /dev/sdp                              vrh2 vrh1               

  Disk                                  Sharing Nodes (2 in count)
  ------------------------------------  ------------------------
  /dev/sdq                              vrh2 vrh1               

  Disk                                  Sharing Nodes (2 in count)
  ------------------------------------  ------------------------
  /dev/sdr                              vrh2 vrh1               

  Disk                                  Sharing Nodes (2 in count)
  ------------------------------------  ------------------------
  /dev/sds                              vrh2 vrh1               

  Disk                                  Sharing Nodes (2 in count)
  ------------------------------------  ------------------------
  /dev/sdt                              vrh2 vrh1               
Shared storage check was successful on nodes "vrh2,vrh1"
Post-check for hardware and operating system setup was successful.

Cluster Installation Ready check on all nodes:安装Clusterware前执行以下命令

 cluvfy stage -pre crsinst -n vrh1,vrh2

cluvfy stage -pre crsinst -n vrh1,vrh2
Performing pre-checks for cluster services setup
Checking node reachability...
Node reachability check passed from node "vrh1"
Checking user equivalence...
User equivalence check passed for user "grid"
Checking node connectivity...
Checking hosts config file...
Verification of the hosts config file successful
Node connectivity passed for subnet "192.168.1.0" with node(s) vrh2,vrh1
TCP connectivity check passed for subnet "192.168.1.0"

Node connectivity passed for subnet "192.168.2.0" with node(s) vrh2,vrh1
TCP connectivity check passed for subnet "192.168.2.0"

Node connectivity passed for subnet "169.254.0.0" with node(s) vrh2,vrh1
TCP connectivity check passed for subnet "169.254.0.0"

Interfaces found on subnet "192.168.1.0" that are likely candidates for VIP are:
vrh2 eth0:192.168.1.163 eth0:192.168.1.164 eth0:192.168.1.166
vrh1 eth0:192.168.1.161 eth0:192.168.1.190 eth0:192.168.1.162

Interfaces found on subnet "169.254.0.0" that are likely candidates for VIP are:
vrh2 eth1:169.254.8.92
vrh1 eth1:169.254.175.195

Interfaces found on subnet "192.168.2.0" that are likely candidates for a private interconnect are:
vrh2 eth1:192.168.2.19
vrh1 eth1:192.168.2.18

Node connectivity check passed

Checking ASMLib configuration.
Check for ASMLib configuration passed.
Total memory check passed
Available memory check passed
Swap space check passed
Free disk space check passed for "vrh2:/tmp"
Free disk space check passed for "vrh1:/tmp"
Check for multiple users with UID value 54322 passed
User existence check passed for "grid"
Group existence check passed for "oinstall"
Group existence check passed for "dba"
Membership check for user "grid" in group "oinstall" [as Primary] failed
Check failed on nodes:
        vrh2,vrh1
Membership check for user "grid" in group "dba" failed
Check failed on nodes:
        vrh2,vrh1
Run level check passed
Hard limits check passed for "maximum open file descriptors"
Soft limits check passed for "maximum open file descriptors"
Hard limits check passed for "maximum user processes"
Soft limits check passed for "maximum user processes"
System architecture check passed
Kernel version check passed
Kernel parameter check passed for "semmsl"
Kernel parameter check passed for "semmns"
Kernel parameter check passed for "semopm"
Kernel parameter check passed for "semmni"
Kernel parameter check passed for "shmmax"
Kernel parameter check passed for "shmmni"
Kernel parameter check passed for "shmall"
Kernel parameter check passed for "file-max"
Kernel parameter check passed for "ip_local_port_range"
Kernel parameter check passed for "rmem_default"
Kernel parameter check passed for "rmem_max"
Kernel parameter check passed for "wmem_default"
Kernel parameter check passed for "wmem_max"
Kernel parameter check passed for "aio-max-nr"
Package existence check passed for "make-3.81( x86_64)"
Package existence check passed for "binutils-2.17.50.0.6( x86_64)"
Package existence check passed for "gcc-4.1.2 (x86_64)( x86_64)"
Package existence check passed for "libaio-0.3.106 (x86_64)( x86_64)"
Package existence check passed for "glibc-2.5-24 (x86_64)( x86_64)"
Package existence check passed for "compat-libstdc++-33-3.2.3 (x86_64)( x86_64)"
Package existence check passed for "elfutils-libelf-0.125 (x86_64)( x86_64)"
Package existence check passed for "elfutils-libelf-devel-0.125( x86_64)"
Package existence check passed for "glibc-common-2.5( x86_64)"
Package existence check passed for "glibc-devel-2.5 (x86_64)( x86_64)"
Package existence check passed for "glibc-headers-2.5( x86_64)"
Package existence check passed for "gcc-c++-4.1.2 (x86_64)( x86_64)"
Package existence check passed for "libaio-devel-0.3.106 (x86_64)( x86_64)"
Package existence check passed for "libgcc-4.1.2 (x86_64)( x86_64)"
Package existence check passed for "libstdc++-4.1.2 (x86_64)( x86_64)"
Package existence check passed for "libstdc++-devel-4.1.2 (x86_64)( x86_64)"
Package existence check passed for "sysstat-7.0.2( x86_64)"
Package existence check passed for "ksh-20060214( x86_64)"
Check for multiple users with UID value 0 passed
Current group ID check passed

Starting Clock synchronization checks using Network Time Protocol(NTP)...

NTP Configuration file check started...
No NTP Daemons or Services were found to be running

Clock synchronization check using Network Time Protocol(NTP) passed

Core file name pattern consistency check passed.

User "grid" is not part of "root" group. Check passed
Default user file creation mask check passed
Checking consistency of file "/etc/resolv.conf" across nodes

File "/etc/resolv.conf" does not have both domain and search entries defined
domain entry in file "/etc/resolv.conf" is consistent across nodes
search entry in file "/etc/resolv.conf" is consistent across nodes
All nodes have one search entry defined in file "/etc/resolv.conf"
The DNS response time for an unreachable node is within acceptable limit on all nodes

File "/etc/resolv.conf" is consistent across nodes

Time zone consistency check passed

Starting check for Huge Pages Existence ...

Check for Huge Pages Existence passed

Starting check for Hardware Clock synchronization at shutdown ...

Check for Hardware Clock synchronization at shutdown passed

Pre-check for cluster services setup was unsuccessful on all the nodes.

Database Installation Ready check on all nodes:安装RDBMS前执行以下命令
cluvfy stage -pre dbinst -n vrh1,vrh2

cluvfy stage -pre dbinst -n vrh1,vrh2 

Performing pre-checks for database installation 

Checking node reachability...
Node reachability check passed from node "vrh1"

Checking user equivalence...
User equivalence check passed for user "grid"

Checking node connectivity...

Checking hosts config file...

Verification of the hosts config file successful

Check: Node connectivity for interface "eth0"
Node connectivity passed for interface "eth0"

Check: Node connectivity for interface "eth1"
Node connectivity passed for interface "eth1"

Node connectivity check passed

Total memory check passed
Available memory check passed
Swap space check passed
Free disk space check passed for "vrh2:/tmp"
Free disk space check passed for "vrh1:/tmp"
Check for multiple users with UID value 54322 passed
User existence check passed for "grid"
Group existence check passed for "asmadmin"
Group existence check passed for "dba"
Membership check for user "grid" in group "asmadmin" [as Primary] passed
Membership check for user "grid" in group "dba" failed
Check failed on nodes:
        vrh2,vrh1
Run level check passed
Hard limits check passed for "maximum open file descriptors"
Soft limits check passed for "maximum open file descriptors"
Hard limits check passed for "maximum user processes"
Soft limits check passed for "maximum user processes"
System architecture check passed
Kernel version check passed
Kernel parameter check passed for "semmsl"
Kernel parameter check passed for "semmns"
Kernel parameter check passed for "semopm"
Kernel parameter check passed for "semmni"
Kernel parameter check passed for "shmmax"
Kernel parameter check passed for "shmmni"
Kernel parameter check passed for "shmall"
Kernel parameter check passed for "file-max"
Kernel parameter check passed for "ip_local_port_range"
Kernel parameter check passed for "rmem_default"
Kernel parameter check passed for "rmem_max"
Kernel parameter check passed for "wmem_default"
Kernel parameter check passed for "wmem_max"
Kernel parameter check passed for "aio-max-nr"
Package existence check passed for "make-3.81( x86_64)"
Package existence check passed for "binutils-2.17.50.0.6( x86_64)"
Package existence check passed for "gcc-4.1.2 (x86_64)( x86_64)"
Package existence check passed for "libaio-0.3.106 (x86_64)( x86_64)"
Package existence check passed for "glibc-2.5-24 (x86_64)( x86_64)"
Package existence check passed for "compat-libstdc++-33-3.2.3 (x86_64)( x86_64)"
Package existence check passed for "elfutils-libelf-0.125 (x86_64)( x86_64)"
Package existence check passed for "elfutils-libelf-devel-0.125( x86_64)"
Package existence check passed for "glibc-common-2.5( x86_64)"
Package existence check passed for "glibc-devel-2.5 (x86_64)( x86_64)"
Package existence check passed for "glibc-headers-2.5( x86_64)"
Package existence check passed for "gcc-c++-4.1.2 (x86_64)( x86_64)"
Package existence check passed for "libaio-devel-0.3.106 (x86_64)( x86_64)"
Package existence check passed for "libgcc-4.1.2 (x86_64)( x86_64)"
Package existence check passed for "libstdc++-4.1.2 (x86_64)( x86_64)"
Package existence check passed for "libstdc++-devel-4.1.2 (x86_64)( x86_64)"
Package existence check passed for "sysstat-7.0.2( x86_64)"
Package existence check passed for "ksh-20060214( x86_64)"
Check for multiple users with UID value 0 passed
Current group ID check passed
Default user file creation mask check passed

Checking CRS integrity...

CRS integrity check passed

Checking Cluster manager integrity... 

Checking CSS daemon...
Oracle Cluster Synchronization Services appear to be online.

Cluster manager integrity check passed

Checking node application existence...

Checking existence of VIP node application (required)
VIP node application check passed

Checking existence of NETWORK node application (required)
NETWORK node application check passed

Checking existence of GSD node application (optional)
GSD node application is offline on nodes "vrh2,vrh1"

Checking existence of ONS node application (optional)
ONS node application check passed

Checking if Clusterware is installed on all nodes...
Check of Clusterware install passed

Checking if CTSS Resource is running on all nodes...
CTSS resource check passed

Querying CTSS for time offset on all nodes...
Query of CTSS for time offset passed

Check CTSS state started...
CTSS is in Active state. Proceeding with check of clock time offsets on all nodes...
Check of clock time offsets passed

Oracle Cluster Time Synchronization Services check passed
Checking consistency of file "/etc/resolv.conf" across nodes

File "/etc/resolv.conf" does not have both domain and search entries defined
domain entry in file "/etc/resolv.conf" is consistent across nodes
search entry in file "/etc/resolv.conf" is consistent across nodes
All nodes have one search entry defined in file "/etc/resolv.conf"
The DNS response time for an unreachable node is within acceptable limit on all nodes

File "/etc/resolv.conf" is consistent across nodes

Time zone consistency check passed
Checking VIP configuration.
Checking VIP Subnet configuration.
Check for VIP Subnet configuration passed.
Checking VIP reachability
Check for VIP reachability passed.

Pre-check for database installation was unsuccessful on all the nodes.

利用cluvfy检验ocr组件:
cluvfy comp ocr

cluvfy comp ocr
Verifying OCR integrity

Checking OCR integrity...

Checking the absence of a non-clustered configuration...
All nodes free of non-clustered, local-only configurations

ASM Running check passed. ASM is running on all specified nodes

Checking OCR config file "/etc/oracle/ocr.loc"...

OCR config file "/etc/oracle/ocr.loc" check successful

Disk group for ocr location "+SYSTEMDG" available on all the nodes

Disk group for ocr location "+FRA" available on all the nodes

Disk group for ocr location "+DATA" available on all the nodes

NOTE:
This check does not verify the integrity of the OCR contents.
Execute 'ocrcheck' as a privileged user to verify the contents of OCR.

OCR integrity check passed

Verification of OCR integrity was successful.

11gr2 RAC安装INS-35354问题一例

今天在安装一套11.2.0.2 RAC数据库时出现了INS-35354的问题:
11gR2-GI-INS-35354

因为之前已经成功安装了11.2.0.2的GI,而且Cluster的一切状态都正常,出现这错误都少有点意外:

[grid@vrh1 ~]$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online

去MOS搜了一圈,发现有可能是oraInventory中的inventory.xml更新不正确导致的:

Applies to:
Oracle Server - Enterprise Edition - Version: 11.2.0.1 to 11.2.0.2 - Release: 11.2 to 11.2
Information in this document applies to any platform.
Symptoms

Installing 11gR2 database software in a Grid Infrastrsucture environment fails with the error INS-35354:

The system on which you are attempting to install Oracle RAC is not part of a valid cluster.

Grid Infrastructure (Oracle Clusterware) is running on all nodes in the cluster which can be verified with:

crsctl check crs

Changes
This is a new install.
Cause
As per 11gR2 documentation the error description is:

INS-35354: The system on which you are attempting to install Oracle RAC is not part of a valid cluster.

Cause: Prior to installing Oracle RAC, you must create a valid cluster. 
This is done by deploying Grid Infrastructure software, 
which will allow configuration of Oracle Clusterware and Automatic Storage Management.

However, the problem at hand may be that the central inventory is missing the "CRS=true" flag 
(for the Grid Infrastructure Home).
<inventory.xml>
-------------

<HOME_LIST>
<HOME NAME="Ora11g_gridinfrahome1" LOC="/u01/grid" TYPE="O" IDX="1">
<NODE_LIST>
<NODE NAME="node1"/>
<NODE NAME="node2"/>
</NODE_LIST>

 -------------

From the inventory.xml, we see that the HOME NAME line is missing the CRS="true" flag.

The error INS-35354 will occur when the central inventory entry for the Grid Infrastructure 
home is missing the flag that identifies it as CRS-type home.
Solution
Use the -updateNodeList option for the installer command to fix the the inventory.

The full syntax is:

./runInstaller -updateNodeList "CLUSTER_NODES={node1,node2}"
ORACLE_HOME="" ORACLE_HOME_NAME="" LOCAL_NODE="Node_Name" CRS=[true|false]

Execute the command on any node in the cluster.

Examples:

For a two-node RAC cluster on UNIX:

Node1:
cd /u01/grid/oui/bin
./runInstaller -updateNodeList "CLUSTER_NODES={node1,node2}" ORACLE_HOME="/u01/crs" 
ORACLE_HOME_NAME="GI_11201" LOCAL_NODE="node1" CRS=true

For a 2-node RAC cluster on Windows:

Node 1:
cd e:\app\11.2.0\grid\oui\bin
e:\app\11.2.0\grid\oui\bin\setup -updateNodeList "CLUSTER_NODES={RACNODE1,RACNODE2}" 
ORACLE_HOME="e:\app\11.2.0\grid" ORACLE_HOME_NAME="OraCrs11g_home1" LOCAL_NODE="RACNODE1" CRS=true

我环境中的inventory.xml内容如下:

[grid@vrh1 ContentsXML]$ cat inventory.xml 
<?xml version="1.0" standalone="yes" ?>
<!-- Copyright (c) 1999, 2010, Oracle. All rights reserved. -->
<!-- Do not modify the contents of this file by hand. -->
<INVENTORY>
<VERSION_INFO>
   <SAVED_WITH>11.2.0.2.0</SAVED_WITH>
   <MINIMUM_VER>2.1.0.6.0</MINIMUM_VER>
</VERSION_INFO>
<HOME_LIST>
<HOME NAME="Ora11g_gridinfrahome1" LOC="/g01/11.2.0/grid" TYPE="O" IDX="1" >
   <NODE_LIST>
      <NODE NAME="vrh1"/>
      <NODE NAME="vrh2"/>
   </NODE_LIST>
</HOME>
</HOME_LIST>
</INVENTORY>

显然是在<HOME NAME这里缺少了CRS=”true”的标志,导致OUI安装界面在检测时认为该节点没有安装GI。

解决方案其实很简单只要加入CRS=”true”在重启runInstaller即可,不需要如文档中介绍的那样使用runInstaller -updateNodeList的复杂命令组合。

[grid@vrh1 ContentsXML]$ cat /g01/oraInventory/ContentsXML/inventory.xml 
<?xml version="1.0" standalone="yes" ?>
<!-- Copyright (c) 1999, 2010, Oracle. All rights reserved. -->
<!-- Do not modify the contents of this file by hand. -->
<INVENTORY>
<VERSION_INFO>
   <SAVED_WITH>11.2.0.2.0</SAVED_WITH>
   <MINIMUM_VER>2.1.0.6.0</MINIMUM_VER>
</VERSION_INFO>
<HOME_LIST>
<HOME NAME="Ora11g_gridinfrahome1" LOC="/g01/11.2.0/grid" TYPE="O" IDX="1" CRS="true">
   <NODE_LIST>
      <NODE NAME="vrh1"/>
      <NODE NAME="vrh2"/>
   </NODE_LIST>
</HOME>
</HOME_LIST>
</INVENTORY>

如上修改后问题解决,安装界面正常:
11gr2-RAC-Installing-db-step-4-10

沪ICP备14014813号-2

沪公网安备 31010802001379号