OCRCONFIG

Usage of ocrconfig to administer Oracle Cluster Registry:
Prerequisites:
Requires administrative previlege on UNIX based systems or as a user with administrator previleges on windows based systems.

Usage:
ocrconfig -option

Options:
-backuploc: To change an ocr backup file location.
-downgrade: To downgrade ocr to an earlier version.
-export: To export the contents of an ocr to a target file.
-help: to display help for ocrconfig tool.
-import: To import the ocr contents from a previously exported ocr file.
-overwrite: To update an ocr configuration that is recorded on the ocr with the current ocr configuration information that is found on the node from which you are running this command.
-repair: To update an ocr configuration on the node from which you are running this command with the new configuration information specified by this command
-restore: To restore an ocr from an automatically created ocr backup file.
-showbackup: To display ocr backup information.
-upgrade: To upgrade an ocr to a latest version.

RAC Administration and Maintenance Tasks

RAC Administration and Maintenance Tasks using ocrconfig, ocrdump, olsnodes, srvctl, crsctl, oifcfg, ocrcheck

This topic covers the following RAC maintenance activities:

1. Checking CRS status
2. Viewing name of the cluster
3. Viewing nodes configuraion
4. Checking votedisk information
5. Checking OCR disk information
6. Timeout settings in cluster
7. Add/Remove OCR files
8. Add/Remove votedisk
9. Backing up OCR
10. Backing up votedisk
11. Restoring OCR Devices
12. Restoring voting disk devices
13. Changing Public IPs and Virtual IPs

Refer the below link to learn more about the above maintenance activities.
http://www.oracledba.org/11g/rac/11g_RAC_Admin_Maintenance_Tasks.html

The above link will also give beautiful information and usage of the below tools:

1. ocrconfig
2. ocrdump
3. olsnodes
4. ocrcheck
5. crsctl
6. oifcfg
7. srvctl

WebLogic Configuration for RAC

How-To Configure and Use Oracle Real Application Clusters (RAC) with Oracle WebLogic Server 10.3:
This demo shows how to configure Oracle WebLogic Server to work with Oracle Real Application Clusters (RAC) as well as how to test the connections to the backend Oracle RAC nodes using a web application.

Oracle WebLogic Server 10.3 integrates Oracle Real Application Clusters (RAC) features in Oracle Database 11g, minimizing database access time while allowing transparent access to rich pooling management functions that maximize both connection performance and availability.

There are multiple configuration options for Oracle RAC features within Oracle WebLogic Server. Oracle recommends using Oracle WebLogic Server JDBC multi data sources. This applies to scenarios with or without global transactions. Also you could configure Oracle WebLogic Server to use Oracle JDBC THIN driver’s connect-time failover as well as Fast Connection Failover from Oracle JDBC driver’s Implicit Connection Cache.

In this demo, we will configure and use Oracle WebLogic Server JDBC multi data sources for failover and load balancing.

Check the below URL for detailed steps:
http://www.oracle.com/technology/products/weblogic/howto/rac/index.html#

RAC Filesystem Options

Storage Options for RAC:
In this section we are going to discuss various file system options available to host Oracle Real application Cluster. Lets first understand what are all those file system and then we will see which file systems support Oracle RAC.

Following are the storage options to store Oracle database for RAC.

Raw: Raw Devices
ASM: Automatic Storage Management
CFS: Cluster File System
OCFS: Oracle Cluster File System
LVM: Logical Volume Manager
NFS: Network File System

Now lets find out what are all the physical stuffs that makes a Oracle Database:

1: Oracle Software (Called as Oracle Binaries)
2: Oracle Dumps (Known as bdump, cdump, udump, adump and spfile/pfile etc)
3: Oracle database files (Data files, Control Files and Redo log files )
4: Recovery Files (Oracle Archives and Oracle Flash back)
5: CRS Files (OCR (Oracle Cluster Registry) and Voting Disk. Applicable only for RAC database.)

As you know the above files are required to set up a database (Single Instance/RAC). Lets see which storage option is compatible to host these files:

RAW:
-------------------
Supported: Oracle Database Files and CRS
Not Supported: Oracle Software, Recovery files and Oracle Dumps
Platform: All Platforms

ASM:
------------------
Supported: Oracle Database Files and Recovery Files
Not Supported: Oracle Software, CRS and Oracle Dumps
Platform: All Platforms

CFS:
-----------------
Supported: All
Not Supported: None
Platform: AIX, HP Tru64 UNIX, SPARC Solaris

OCFS:
---------------
Supported: Oracle Database Files, CRS and Recovery Files
Not Supported: Oracle Software and Oracle Dumps
Platform: Windows, Linux

LVM:
--------------
Supported: All
Not Supported: None
Platform: HP-UX, HP Tru64 UNIX, SPARC Solaris

NFS:
-------------
Supported: All
Not Supported: None
Platform: Linux, SPARC Solaris

Convert Single Instance Database to RAC and ASM

Convert a Single Instance Database to Oracle RAC 10g on RHEL3:

Learn how to convert a single-instance database to Oracle Real Application Clusters (RAC) 10g on Red Hat Enterprise Linux 3 (RHEL3), step by step.
Contents
Overview
Step 1: Preliminary Installation
Step 2: Migrate Your Database to ASM
Step 3: Install Cluster Ready Services (CRS) Software
Step 4: Install Oracle RAC Software
Step 5: Post Installation
Step 6: Test Transparent Application Failover (TAF)Conclusion

Download for this article:
Oracle Database 10g Enterprise Edition and Clusterware for Linux x86
Oracle ASMlib Drivers

For those with limited hands-on experience, implementing Oracle RAC 10g can be an intimidating prospect. But it doesn't have to be that way.

In this guide, I'll provide the precise procedures for converting a single-instance Oracle 10g database to Oracle RAC on RHEL3. We'll use Oracle Automatic Storage Management (ASM), an integrated file system and a volume management solution built into the Oracle database kernel, for RAC database storage because it simplifies database storage management tasks while offering storage reliability and performance.

To summarize on a very high level, there are several ways to convert your database to RAC. If the database is small, you can consider installing a new RAC database on ASM and export/import your data from your current database to the RAC database. For a larger database, the preferred method would be to use RMAN to migrate the database files over to ASM.

The method demonstrated here is a two-phase approach:

first, migrating the data to ASM, and
second, converting the single-instance database to RAC.

If you are new to ASM, I would recommend taking this migration path to get familiar with ASM before leaping into ASM and RAC at the same time. (For more information about Oracle ASM, visit the Oracle ASM Product Center or refer to the documentation.)

This guide requires a basic understanding of RAC architecture and some familiarity with managing and administering Oracle Database and Red Hat Enterprise Linux. Refer to the documentation for details.

The detailed stepwise guidance is available at the below URL:
http://www.oracle.com/technology/pub/articles/chan_sing2rac_install.html

The above URL holds a beautiful document to migrate a single instance database from Cluster file system to ASM and to RAC.

Cluster Ready Services Logs

Log Directory Structure in Cluster Ready Services:
Each component in the CRS (Cluster Ready Services) stack has its respective directories created under the CRS home:
$ORA_CRS_HOME/crs/log Contains trace files for the CRS resources.

$ORA_CRS_HOME/crs/init Contains trace files of the CRS daemon during startup. Good place to start with any CRS login problems.

$ORA_CRS_HOME/css/log The Cluster Synchronization (CSS) logs indicate all actions such as reconfigurations, missed check-ins, connects, and disconnects from the client CSS listener. In some cases, the logger logs messages with the category of auth.crit for the reboots done by Oracle. This could be used for checking the exact time when the reboot occurred.

$ORA_CRS_HOME/css/init Contains core dumps from the Oracle Cluster Synchronization Service daemon (OCSSd) and the process ID (PID) for the CSS daemon whose death is treated as fatal. If abnormal restarts for CSS exist, the core files will have the format of core..

$ORA_CRS_HOME/evm/log Log files for the Event Volume Manager (EVM) and evmlogger daemons. Not used as often for debugging as the CRS and CSS directories.

$ORA_CRS_HOME/evm/init PID and lock files for EVM. Core files for EVM should also be written here.

$ORA_CRS_HOME/srvm/log Log files for Oracle Cluster Registry (OCR), which contains the details at the Oracle cluster level.

$ORA_CRS_HOME/log Log files for Oracle Clusterware (known as the cluster alert log), which contains diagnostic messages at the Oracle cluster level. This is available from Oracle database 10g R2.

CRS Administration (RAC)

CRS RESOURCE ADMINISTRATION:
----------------------------------------

You can use srvctl to manage these resources. Below are syntax and examples.

SRVCTL command line utility and examples are available at:

10G R2:
http://download.oracle.com/docs/cd/B19306_01/rac.102/b14197/srvctladmin.htm
Oracle Database Oracle Clusterware and Oracle Real Application Clusters Administration and Deployment Guide10g Release 2 (10.2)

9i R2:
http://download.oracle.com/docs/cd/B10500_01/rac.920/a96596/genadmin.htm#31583
Oracle9i Real Application Clusters AdministrationRelease 2 (9.2)

CRS RESOURCE STATUS:

srvctl status database -d [-f] [-v] [-S ]
srvctl status instance -d -i >[,]
[-f] [-v] [-S ]
srvctl status service -d -s [,]
[-f] [-v] [-S ]
srvctl status nodeapps [-n ]
srvctl status asm -n

EXAMPLES:

Status of the database, all instances and all services.
srvctl status database -d ORACLE -v
Status of named instances with their current services.
srvctl status instance -d ORACLE -i RAC01, RAC02 -v
Status of a named services.
srvctl status service -d ORACLE -s ERP -v
Status of all nodes supporting database applications.
srvctl status node

START CRS RESOURCES

srvctl start database -d [-o <>]
[-c -q]
srvctl start instance -d -i
[,] [-o ] [-c -q]
srvctl start service -d [-s [,]]
[-i ] [-o ] [-c -q]
srvctl start nodeapps -n
srvctl start asm -n [-i ] [-o ]

EXAMPLES:

Start the database with all enabled instances.
srvctl start database -d ORACLE
Start named instances.
srvctl start instance -d ORACLE -i RAC03, RAC04
Start named services. Dependent instances are started as needed.
srvctl start service -d ORACLE -s CRM
Start a service at the named instance.
srvctl start service -d ORACLE -s CRM -i RAC04
Start node applications.
srvctl start nodeapps -n myclust-4

STOP CRS RESOURCES

srvctl stop database -d [-o ]
[-c -q]
srvctl stop instance -d -i [,]
[-o ][-c -q]
srvctl stop service -d [-s [,]]
[-i ][-c -q] [-f]
srvctl stop nodeapps -n
srvctl stop asm -n [-i ] [-o ]

EXAMPLES:

Stop the database, all instances and all services.
srvctl stop database -d ORACLE
Stop named instances, first relocating all existing services.
srvctl stop instance -d ORACLE -i RAC03,RAC04
Stop the service.
srvctl stop service -d ORACLE -s CRM
Stop the service at the named instances.
srvctl stop service -d ORACLE -s CRM -i RAC04
Stop node applications. Note that instances and services also stop.
srvctl stop nodeapps -n myclust-4

ADD CRS RESOURCES

srvctl add database -d -o [-m ] [-p ]
[-A /netmask] [-r {PRIMARY PHYSICAL_STANDBY LOGICAL_STANDBY}]
[-s ] [-n ]
srvctl add instance -d -i -n
srvctl add service -d -s -r
[-a ] [-P ] [-u]
srvctl add nodeapps -n -o
[-A /netmask[/if1[if2...]]]
srvctl add asm -n -i -o

OPTIONS:

-A vip range, node, and database, address specification. The format of
address string is:
[]//[/] [,] []//
[/]
-a for services, list of available instances, this list cannot include
preferred instances
-m domain name with the format “us.mydomain.com”
-n node name that will support one or more instances
-o $ORACLE_HOME to locate Oracle binaries
-P for services, TAF preconnect policy - NONE, PRECONNECT
-r for services, list of preferred instances, this list cannot include
available instances.
-s spfile name
-u updates the preferred or available list for the service to support the
specified instance. Only one instance may be specified with the -u
switch. Instances that already support the service should not be
included.

EXAMPLES:

Add a new node:
srvctl add nodeapps -n myclust-1 -o $ORACLE_HOME –A
139.184.201.1/255.255.255.0/hme0
Add a new database.
srvctl add database -d ORACLE -o $ORACLE_HOME
Add named instances to an existing database.
srvctl add instance -d ORACLE -i RAC01 -n myclust-1
srvctl add instance -d ORACLE -i RAC02 -n myclust-2
srvctl add instance -d ORACLE -i RAC03 -n myclust-3
Add a service to an existing database with preferred instances (-r) and
available instances (-a). Use basic failover to the available instances.
srvctl add service -d ORACLE -s STD_BATCH -r RAC01,RAC02 -a RAC03,RAC04
Add a service to an existing database with preferred instances in list one and
available instances in list two. Use preconnect at the available instances.
srvctl add service -d ORACLE -s STD_BATCH -r RAC01,RAC02 -a RAC03,RAC04 -P PRECONNECT


REMOVE CRS RESOURCES

srvctl remove database -d
srvctl remove instance -d [-i ]
srvctl remove service -d -s [-i ]
srvctl remove nodeapps -n

EXAMPLES:

Remove the applications for a database.
srvctl remove database -d ORACLE
Remove the applications for named instances of an existing database.
srvctl remove instance -d ORACLE -i RAC03
srvctl remove instance -d ORACLE -i RAC04
Remove the service.
srvctl remove service -d ORACLE -s STD_BATCH
Remove the service from the instances.
srvctl remove service -d ORACLE -s STD_BATCH -i RAC03,RAC04
Remove all node applications from a node.
srvctl remove nodeapps -n myclust-4


MODIFY CRS RESOURCES

srvctl modify database -d [-n ] [-m ]
[-p ] [-r {PRIMARY PHYSICAL_STANDBY LOGICAL_STANDBY}]
[-s ]
srvctl modify instance -d -i -n
srvctl modify instance -d -i {-s -r}
srvctl modify service -d -s -i
-t [-f]
srvctl modify service -d -s -i
-r [-f]
srvctl modify nodeapps -n [-A ] [-x]

OPTIONS:

-i -t the instance name (-i) is replaced by the
instance name (-t)
-i -r the named instance is modified to be a preferred instance
-A address-list for VIP application, at node level
-s add or remove ASM dependency

EXAMPLES:

Modify an instance to execute on another node.
srvctl modify instance -d ORACLE -n myclust-4
Modify a service to execute on another node.
srvctl modify service -d ORACLE -s HOT_BATCH -i RAC01 -t RAC02
Modify an instance to be a preferred instance for a service.
srvctl modify service -d ORACLE -s HOT_BATCH -i RAC02 –r


RELOCATE SERVICES

srvctl relocate service -d -s [-i ]-t [-f]

EXAMPLES:

Relocate a service from one instance to another
srvctl relocate service -d ORACLE -s CRM -i RAC04 -t RAC01


ENABLE CRS RESOURCES (The resource may be up or down to use this function)

srvctl enable database -d
srvctl enable instance -d -i [,]
srvctl enable service -d -s ] [, ] [-i ]

EXAMPLES:

Enable the database.
srvctl enable database -d ORACLE
Enable the named instances.
srvctl enable instance -d ORACLE -i RAC01, RAC02
Enable the service.
srvctl enable service -d ORACLE -s ERP,CRM
Enable the service at the named instance.
srvctl enable service -d ORACLE -s CRM -i RAC03


DISABLE CRS RESOURCES (The resource must be down to use this function)

srvctl disable database -d
srvctl disable instance -d -i [,]
srvctl disable service -d -s ] [,] [-i ]

EXAMPLES:

Disable the database globally.
srvctl disable database -d ORACLE
Disable the named instances.
srvctl disable instance -d ORACLE -i RAC01, RAC02
Disable the service globally.
srvctl disable service -d ORACLE -s ERP,CRM
Disable the service at the named instance.
srvctl disable service -d ORACLE -s CRM -i RAC03,RAC04


Note: For more information on this see the Oracle10g Real Application Clusters
Administrator’s Guide - Appendix B

Cluster Ready Services (RAC)

CRS and 10g Real Application Clusters:

PURPOSE
----------
This document is to provide additional information on CRS (Cluster Ready Services)in 10g Real Application Clusters.

SCOPE & APPLICATION
--------------------------
This document is intended for RAC Database Administrators and Oracle support enginneers.

CRS and 10g REAL APPLICATION CLUSTERS
------------------------------------- -----------
CRS (Cluster Ready Services) is a new feature for 10g Real Application Clustersthat provides a standard cluster interface on all platforms and performs new high availability operations not available in previous versions.

CRS KEY FACTS
-----------------
Prior to installing CRS and 10g RAC, there are some key points to remember about CRS and 10g RAC:
- CRS is REQUIRED to be installed and running prior to installing 10g RAC.
- CRS can either run on top of the vendor clusterware (such as Sun Cluster, HP Serviceguard, IBM HACMP, TruCluster, Veritas Cluster, Fujitsu Primecluster, etc...) or can run without the vendor clusterware. The vendor clusterware was required in 9i RAC but is optional in 10g RAC.
- The CRS HOME and ORACLE_HOME must be installed in DIFFERENT locations.
- Shared Location(s) or devices for the Voting File and OCR (Oracle Configuration Repository) file must be available PRIOR to installing CRS. The voting file should be at least 20MB and the OCR file should be at least 100MB.
- CRS and RAC require that the following network interfaces be configured prior to installing CRS or RAC:
- Public Interface
- Private Interface
- Virtual (Public) Interface
For more information on this, see Note 264847.1 in metalink.oracle.com
- The root.sh script at the end of the CRS installation starts the CRS stack.
If your CRS stack does not start, see Note 240001.1 in metalink.oracle.com
- Only one set of CRS daemons can be running per RAC node.
- On Unix, the CRS stack is run from entries in /etc/inittab with "respawn".
- If there is a network split (nodes lose communication with each other). One or more nodes may reboot automatically to prevent data corruption.
- The supported method to start CRS is booting the machine. MANUAL STARTUP OF THE CRS STACK IS NOT SUPPORTED UNTIL 10.1.0.4 OR HIGHER.
- The supported method to stop is shutdown the machine or use "init.crs stop".
- Killing CRS daemons is not supported unless you are removing the CRS installation via Note 239998.1 in metalink.oracle.com because flag files can become mismatched.
- For maintenance, go to single user mode at the OS.Once the stack is started, you should be able to see all of the daemon processeswith a ps -ef command:
[rac1]/u01/home/beta> ps -ef grep crs
oracle 1363 999 0 11:23:21 ? 0:00 /u01/crs_home/bin/evmlogger.bin -o /u01
oracle 999 1 0 11:21:39 ? 0:01 /u01/crs_home/bin/evmd.bin
root 1003 1 0 11:21:39 ? 0:01 /u01/crs_home/bin/crsd.bin
oracle 1002 1 0 11:21:39 ? 0:01 /u01/crs_home/bin/ocssd.binCRS DAEMON

FUNCTIONALITY
------------------------
Here is a short description of each of the CRS daemon processes:
CRSD:
- Engine for HA operation
- Manages 'application resources'
- Starts, stops, and fails 'application resources' over
- Spawns separate 'actions' to start/stop/check application resources
- Maintains configuration profiles in the OCR (Oracle Configuration Repository)
- Stores current known state in the OCR.
- Runs as root
- Is restarted automatically on failure
OCSSD:
- OCSSD is part of RAC and Single Instance with ASM
- Provides access to node membership
- Provides group services
- Provides basic cluster locking
- Integrates with existing vendor clusteware, when present
- Can also runs without integration to vendor clustware
- Runs as Oracle.
- Failure exit causes machine reboot.
--- This is a feature to prevent data corruption in event of a split brain.
EVMD:
- Generates events when things happen
- Spawns a permanent child evmlogger
- Evmlogger, on demand, spawns children
- Scans callout directory and invokes callouts.
- Runs as Oracle.
- Restarted automatically on failure

CRS LOG DIRECTORIES
--------------------------
When troubleshooting CRS problems, it is important to review the directoriesunder the CRS Home.
$ORA_CRS_HOME/crs/log - This directory includes traces for CRS resources that arejoining, leaving, restarting, and relocating as identified by CRS.
$ORA_CRS_HOME/crs/init - Any core dumps for the crsd.bin daemon should be writtenhere. Note 1812.1 of metalink.oracle.com could be used to debug these.
$ORA_CRS_HOME/css/log - The css logs indicate all actions such as reconfigurations, missed checkins , connects, and disconnects from the clientCSS listener . In some cases the logger logs messages with the category of (auth.crit) for the reboots done by oracle. This could be used for checking the exact time when the reboot occured.
$ORA_CRS_HOME/css/init - Core dumps from the ocssd primarily and the pid for the css daemon whose death is treated as fatal are located here. If there are abnormal restarts for css then the core files will have the formats of core.. Note 1812.1 in metalink.oracle.com could be used to debug these.
$ORA_CRS_HOME/evm/log - Log files for the evm and evmlogger daemons. Not used as often for debugging as the CRS and CSS directories.
$ORA_CRS_HOME/evm/init - Pid and lock files for EVM. Core files for EVM shouldalso be written here. Note 1812.1 in metalink.oracle.com could be used to debug these.
$ORA_CRS_HOME/srvm/log - Log files for OCR.

STATUS FOR CRS RESOURCES
----------------------------------
After installing RAC and running the VIPCA (Virtual IP Configuration Assistant)launched with the RAC root.sh, you should be able to see all of your CRSresources with crs_stat. Example:
cd $ORA_CRS_HOME/bin ./crs_stat
NAME=ora.rac1.gsd
TYPE=application
TARGET=ONLINE
STATE=ONLINE

NAME=ora.rac1.oem
TYPE=application
TARGET=ONLINE
STATE=ONLINE

NAME=ora.rac1.ons
TYPE=application
TARGET=ONLINE
STATE=ONLINE

NAME=ora.rac1.vip
TYPE=application
TARGET=ONLINE
STATE=ONLINE

NAME=ora.rac2.gsd
TYPE=application
TARGET=ONLINE
STATE=ONLINE

NAME=ora.rac2.oem
TYPE=application
TARGET=ONLINE
STATE=ONLINE

NAME=ora.rac2.ons
TYPE=application
TARGET=ONLINE
STATE=ONLINE

NAME=ora.rac2.vip
TYPE=application
TARGET=ONLINE
STATE=ONLINE

There is also a script available to view CRS resources in a format that is easier to read. Just create a shell script with:
--------------------------- Begin Shell Script -------------------------------
#!/usr/bin/ksh
#
# Sample 10g CRS resource status query script
#
# Description:
# - Returns formatted version of crs_stat -t, in tabular
# format, with the complete rsc names and filtering keywords
# - The argument, $RSC_KEY, is optional and if passed to the script, will
# limit the output to HA resources whose names match $RSC_KEY.
# Requirements:
# - $ORA_CRS_HOME should be set in your environment
RSC_KEY=$1
QSTAT=-u
AWK=/usr/xpg4/bin/awk # if not available use /usr/bin/awk

# Table header:echo ""
$AWK \
'BEGIN {printf "%-45s %-10s %-18s\n", "HA Resource", "Target", "State";
printf "%-45s %-10s %-18s\n", "-----------", "------", "-----";}'

# Table body:
$ORA_CRS_HOME/bin/crs_stat $QSTAT $AWK \
'BEGIN { FS="="; state = 0; }
$1~/NAME/ && $2~/'$RSC_KEY'/ {appname = $2; state=1};
state == 0 {next;}
$1~/TARGET/ && state == 1 {apptarget = $2; state=2;}
$1~/STATE/ && state == 2 {appstate = $2; state=3;}
state == 3 {printf "%-45s %-10s %-18s\n", appname, apptarget, appstate; state=0;}'
--------------------------- End Shell Script -------------------------------

Note:For more information on CRS refer CRS Administration (RAC) link of this blog.