Configuring raw devices (multipath) for Oracle Clusterware 10g Release 2 (10.2.0) on RHEL5-OEL5 [ID 564580.1]

Configuring raw devices (multipath) for Oracle Clusterware 10g Release 2 (10.2.0) on RHEL5/OEL5 [ID 564580.1]
	Modified 24-JAN-2010Type HOWTOStatus PUBLISHED

In this Document
Goal
Solution
Deprecation of Support for Raw Devices
A Bit About Udev and Device Name Persistency
Multipath, Raw and Udev
Configuring raw devices (multipath) for Oracle Clusterware 10g Release 2 (10.2.0) on RHEL5/OEL5
Assumptions
1. Configure SCSI_ID to Return Unique Device Identifiers
1a. Whitelist SCSI devices
1b. List all SCSI (Clusterware) devices
1c. Obtain Clusterware device unique SCSI identifiers
2. Configure Multipath for Persistent Naming of Clusterware Devices
2a. Configure Multipathing
2b. Verify Multipath Devices
3. Create Raw Devices
4. Test Raw Device Accessibility
5. Script Creation of Raw Bindings and Permissions
6. Test the Raw Device Script
7. Install Oracle 10gR2 Clusterware
References

---

Applies to:

Linux OS - Version: 5.0 to 5.0
Linux x86
Linux x86-64
Linux Itanium
Linux Kernel - Version: 5.0 to 5.0

Goal

This article is intended for Oracle on Linux Database and System Administrators, particularly those intending to install (or migrate to) Oracle Real Application Clusters 10g Release 2 (10.2.0) on Red Hat/Oracle Enterprise Linux 5 (EL5). The article is intended to focus on the configuration of raw devices against multipathed devices on EL5 in preparation for RAC Clusterware usage, rather than on multipathing or installation of the Clusterware.

Examples were taken from a working system of the following configuration:

• Enterprise Linux 5 (GA) - 2.6.18-8.el5
• Oracle Clusterware 10g Release 2 (10.2.0)
• Shared storage for Clusterware files served via iSCSI

Note: this document differs to Note.465001.1 that describes configuration of raw devices against single pathed devices. This Note describes configuration of raw devices against multipathed devices.

Solution

Deprecation of Support for Raw Devices

In versions prior to EL5, applications such as Oracle, could access unstructured data on block devices by binding to them via character raw devices, such as /dev/raw/raw1 using the raw(8) command. Persistent device assignments could be configured using the /etc/sysconfig/rawdevices file in conjunction with the rawdevices service.

Support for raw devices was initially deprecated in the Linux 2.6 kernel (EL5 < U4) in favour of directio (O_DIRECT) access, however was later undeprecated from EL5 U4 (initscripts-8.45.30-2).

For details of the deprecation and undeprecation of support for rawio, refer to Linux kernel/version documentation including:

• /usr/share/doc/kernel-doc-2.6.18/Documentation/feature-removal-schedule.txt
• Red Hat Enterprise Linux 4/5 Release notes

Both the /etc/sysconfig/rawdevices file (EL4) and /etc/udev/rules.d/60-raw.rules file (EL5) similarly discuss deprecation of raw.

OCFS2, Oracle's Cluster Filesystem version 2 (http://oss.oracle.com/projects/ocfs2), is an extent based, POSIX-compliant file system that provides for shared, O_DIRECT file access. For certified ports and distributions, Oracle extends free support of OCFS2 users with an Oracle database license for use in storing Oracle datafiles, redologs, archivelogs, control files, voting disk (CRS), cluster registry (OCR), etc. along with shared Oracle home.

A Bit About Udev and Device Name Persistency

Unlike devlabel in the 2.4 kernel, udev (the 2.6 kernel device file naming scheme) dynamically creates device file names at boot time. This can, however, give rise to the possibility that device file names may change - a device that may once have been named /dev/sdd say, may be named /dev/sdf, say, after reboot. Without specific configuration, if udev is left to dynamically name devices, the potential exists for devices referred to, or inadvertently accessed by, their arbitrary kernel-assigned name (e.g. Oracle Clusterware files; Cluster Registry, Voting disks, etc.) to become corrupt.

Multipath, Raw and Udev

The necessity for high availability access to storage is well understood. For singlepath environments, raw devices can easily be configured via udev rules as described in Note.465001.1. For multipath environments, however, configuration of raw devices against multipathed devices via udev is more complex. In fact, significant modification of default udev rules can introduce issues with supportability. Therefore, other means are recommended to achieve configuration of raw devices against multipathed devices with multipath device naming persistency.

Configuring raw devices (multipath) for Oracle Clusterware 10g Release 2 (10.2.0) on RHEL5/OEL5

The following procedure outlines the steps necessary to configure persistent multipath device naming and creation of raw devices (including permissions) in preparation for Oracle 10gR2 (10.2.0) Clusterware devices. From Oracle11g Release 1 (11.1.0), Clusterware files may be placed on either block or raw devices located on shared disk partitions, therefore the following procedure only strictly applies when using Oracle 10gR2 (10.2.0) and multipathing.

Therefore, take this opportunity to consider whether you wish to proceed using 10gR2 or 11gR1 Clusterware to manage your 10gR2 databases - multipath device configuration for Oracle 11g Clusterware is described in Note 605828.1. The following procedure may also be used as a basis for configuring raw devices on EL4 (Update 2 or higher). Unless otherwise stated, all steps should be performed on each cluster node and as a privileged user.

Assumptions

The following procedure assumes the following to have occured:

• Clusterware devices have been created on supported shared storage
• Clusterware devices have been appropriately sized according to Oracle10g Release 2 (10.2.0) RAC documentation
• Clusterware devices have been partitioned
• All cluster nodes have multipath access to shared devices
• Cluster nodes are configured to satisfy Oracle Universal Installer (OUI) requirements

1. Configure SCSI_ID to Return Unique Device Identifiers

1a. Whitelist SCSI devices

Before being able to configure udev to explicitly name devices, scsi_id(8) should first be configured to return SCSI device identifiers. Modify the /etc/scsi_id.config file - add or replace the option=-b parameter/value pair (if exists) with option=-g, for example:

# grep -v ^# /etc/scsi_id.config
vendor="ATA",options=-p 0x80
options=-g

1b. List all SCSI (Clusterware) devices

Clusterware devices must be visible and accessible to all cluster nodes. Typically, cluster node operating systems need to be updated in order to see newly provisioned (or modified) devices on shared storage i.e. use '/sbin/partprobe <device>' or '/sbin/sfdisk -r <device>', etc., or simply reboot. Resolve any issues preventing cluster nodes from correctly seeing or accessing Clusterware devices before proceeding.

Run the fdisk(8) and/or 'cat /proc/partitions' commands to ensure Clusterware devices are visible, for example:

#cat/proc/partitions
majorminor#blocksname

806291456sda
815735173sda1
82554242sda2
816987966sdb
817 987681 sdb1
832987966sdc
833 987681 sdc1
848987966sdd
849 987681 sdd1
864987966sde
865 987681 sde1
880987966sdf
881 987681 sdf1
896987966sdg
897 987681 sdg1
81121004031sdh
8113 1003873 sdh1
81281004031sdi
8129 1003873 sdi1
81441004031sdj
8145 1003873 sdj1
81601004031sdk
8161 1003873 sdk1
81761004031sdl
8177 1003873 sdl1
81921004031sdm
8193 1003873 sdm1

Above, though perhaps not entirely evident, the kernel has assigned two device files per multipathed device i.e. devices /dev/sdb and /dev/sdc both refer to the same device/LUN on shared storage, as do /dev/sdd and /dev/sde and so on.

Note, at this point, each cluster node may refer to the would-be Clusterware devices by different device file names - this is expected.

1c. Obtain Clusterware device unique SCSI identifiers

Run the scsi_id(8) command against Clusterware devices from one cluster node to obtain their unique device identifiers. When running the scsi_id(8) command with the -s argument, the device path and name passed should be that relative to sysfs directory /sys/ i.e. /block/<device> when referring to /sys/block/<device>. Record the unique SCSI identifiers of Clusterware devices - these are required later (Step 2a.), for example:

#foriin`cat/proc/partitions|awk {'print$4'} |grepsd`;doecho"###$i:`scsi_id -g -u -s/block/$i`";done
...
###sdb:1494554000000000000000000010000005c3900000d000000
###sdb1:
###sdc:1494554000000000000000000010000005c3900000d000000
###sdc1:
###sdd:149455400000000000000000001000000843900000d000000
###sdd1:
###sde:149455400000000000000000001000000843900000d000000
###sde1:
###sdf:149455400000000000000000001000000ae3900000d000000
###sdf1:
###sdg:149455400000000000000000001000000ae3900000d000000
###sdg1:
###sdh:149455400000000000000000001000000d03900000d000000
###sdh1:
###sdi:149455400000000000000000001000000d03900000d000000
###sdi1:
###sdj:149455400000000000000000001000000e63900000d000000
###sdj1:
###sdk:149455400000000000000000001000000e63900000d000000
###sdk1:
###sdl:149455400000000000000000001000000083a00000d000000
###sdl1:
###sdm:149455400000000000000000001000000083a00000d000000
###sdm1:

From the output above, note that multiple devices share common SCSI identifiers. It's should now be evident that devices such as /dev/sdb and /dev/sdc refer to the same shared storage device (LUN).

Note: Irrespective of which cluster node the scsi_id(8) command is run from, the value returned for a given device (LUN) should always be the same.

2. Configure Multipath for Persistent Naming of Clusterware Devices

The purpose of this step is to provide persistent and meaningful, user-defined Clusterware multipath device names. This step is provided to ensure correct use of the intended Clusterware multipath devices that could otherwise be confused if solely relying on default multipath-assigned names (mpathn/mpathnpn), especially when many devices are involved.

2a. Configure Multipathing

Configure multipathing by modifying multipath configuration file /etc/multipath.conf. Comment and uncomment various stanzas accordingly to include (whitelist) or exclude (blacklist) specific devices/types as candidates for multipathing. Specific devices, such as our intended Clusterware devices, should be explicitly whitelisted as multipathing candidates. This can be accomplished by creating dedicated multipath stanzas for each device. Ideally, at a minimum, each device stanza should include the device wwid and an alias, for example:

# cat /etc/multipath.conf
...
multipath{
wwid1494554000000000000000000010000005c3900000d000000
aliasvoting1
}
...

Following is a sample multipath.conf file. Modify your configuration according to your own environment and preferences, but ensuring to include Clusterware device-specific multipath stanzas - substitite wwid values for your own i.e. those returned from running Step 1c. above.

#grep-v^#/etc/multipath.conf
defaults{
user_friendly_namesyes
}
defaults{
udev_dir/dev
polling_interval10
selector"round-robin0"
path_grouping_policyfailover
getuid_callout"/sbin/scsi_id-g-u-s/block/%n"
prio_callout/bin/true
path_checkerreadsector0
rr_min_io100
rr_weightpriorities
failbackimmediate
#no_path_retry fail
user_friendly_nameyes
}
devnode_blacklist{
devnode"^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
devnode"^hd[a-z]"
devnode"^sda"
devnode"^cciss!c[0-9]d[0-9]*"
}
multipaths{
multipath{
wwid1494554000000000000000000010000005c3900000d000000
aliasvoting1
}
multipath{
wwid149455400000000000000000001000000843900000d000000
aliasvoting2
}
multipath{
wwid149455400000000000000000001000000ae3900000d000000
aliasvoting3
}
multipath{
wwid149455400000000000000000001000000d03900000d000000
aliasocr1
}
multipath{
wwid149455400000000000000000001000000e63900000d000000
aliasocr2
}
multipath{
wwid149455400000000000000000001000000083a00000d000000
aliasocr3
}
}

In the example above, devices with a specific wwid (per scsi_id(8)) are assigned persistent, user-defined names (aliases) i.e. voting1, voting2, voting3, ocr1, ocr2 and ocr3.

2b. Verify Multipath Devices

Once multipathing has been configured and multipathd service started, you should now have multipathed Clusterware devices referable by user-defined names, for example:

#multipath-ll
ocr3(149455400000000000000000001000000083a00000d000000)dm-9 IET,VIRTUAL-DISK
[size=980M][features=0][hwhandler=0]
/_round-robin0[prio=0][active]
/_2:0:0:10sdl8:176[active][ready]
/_round-robin0[prio=0][enabled]
/_2:0:0:11sdm8:192[active][ready]
ocr2(149455400000000000000000001000000e63900000d000000)dm-3 IET,VIRTUAL-DISK
[size=980M][features=0][hwhandler=0]
/_round-robin0[prio=0][active]
/_2:0:0:8sdj8:144[active][ready]
/_round-robin0[prio=0][enabled]
/_2:0:0:9sdk8:160[active][ready]
ocr1(149455400000000000000000001000000d03900000d000000)dm-6 IET,VIRTUAL-DISK
[size=980M][features=0][hwhandler=0]
/_round-robin0[prio=0][active]
/_2:0:0:6sdh8:112[active][ready]
/_round-robin0[prio=0][enabled]
/_2:0:0:7sdi8:128[active][ready]
voting3(149455400000000000000000001000000ae3900000d000000)dm-2IET,VIRTUAL-DISK
[size=965M][features=0][hwhandler=0]
/_round-robin0[prio=0][active]
/_2:0:0:4sdf8:80[active][ready]
/_round-robin0[prio=0][enabled]
/_2:0:0:5sdg8:96[active][ready]
voting2(149455400000000000000000001000000843900000d000000)dm-1IET,VIRTUAL-DISK
[size=965M][features=0][hwhandler=0]
/_round-robin0[prio=0][active]
/_2:0:0:2sdd8:48[active][ready]
/_round-robin0[prio=0][enabled]
/_2:0:0:3sde8:64[active][ready]
voting1(1494554000000000000000000010000005c3900000d000000)dm-0IET,VIRTUAL-DISK
[size=965M][features=0][hwhandler=0]
/_round-robin0[prio=0][active]
/_2:0:0:0sdb8:16[active][ready]
/_round-robin0[prio=0][enabled]
/_2:0:0:1sdc8:32[active][ready]

In fact, various device names are created and used to refer to multipathed devices i.e.:

#dmsetupls | sort
ocr1(253, 6)
ocr1p1 (253, 11)
ocr2(253, 3)
ocr2p1 (253, 8)
ocr3(253, 9)
ocr3p1 (253, 10)
voting1(253,0)
voting1p1 (253, 5)
voting2(253,1)
voting2p1 (253, 4)
voting3(253,2)
voting3p1 (253, 7)


#ll/dev/disk/by-id/
lrwxrwxrwx1rootroot9Apr2311:05scsi-149455400000000000000000001000000083a00000d000000->../../sdm
lrwxrwxrwx1rootroot9Apr2311:05scsi-149455400000000000000000001000000083a00000d000000-part1 ->../../sdm1
lrwxrwxrwx1rootroot9Apr2311:05scsi-1494554000000000000000000010000005c3900000d000000->../../sdc
lrwxrwxrwx1rootroot9Apr2311:05scsi-1494554000000000000000000010000005c3900000d000000-part1 ->../../sdc1
lrwxrwxrwx1rootroot9Apr2311:05scsi-149455400000000000000000001000000843900000d000000->../../sde
lrwxrwxrwx1rootroot9Apr2311:05scsi-149455400000000000000000001000000843900000d000000-part1 ->../../sde1
lrwxrwxrwx1rootroot9Apr2311:05scsi-149455400000000000000000001000000ae3900000d000000->../../sdg
lrwxrwxrwx1rootroot9Apr2311:05scsi-149455400000000000000000001000000ae3900000d000000-part1 ->../../sdg1
lrwxrwxrwx1rootroot9Apr2311:05scsi-149455400000000000000000001000000d03900000d000000->../../sdi
lrwxrwxrwx1rootroot9Apr2311:05scsi-149455400000000000000000001000000d03900000d000000-part1 ->../../sdi1
lrwxrwxrwx1rootroot9Apr2311:05scsi-149455400000000000000000001000000e63900000d000000->../../sdk
lrwxrwxrwx1rootroot9Apr2311:05scsi-149455400000000000000000001000000e63900000d000000-part1 ->../../sdk1

#ls-l/dev/dm-*
brw-rw----1rootroot253, 0Apr2311:15/dev/dm-0
brw-rw----1rootroot253, 1Apr2311:15/dev/dm-1
brw-rw----1rootroot253,10 Apr2311:15/dev/dm-10
brw-rw----1rootroot253,11 Apr2311:15/dev/dm-11
brw-rw----1rootroot253, 2Apr2311:15/dev/dm-2
brw-rw----1rootroot253, 3Apr2311:15/dev/dm-3
brw-rw----1rootroot253, 4Apr2311:15/dev/dm-4
brw-rw----1rootroot253, 5Apr2311:15/dev/dm-5
brw-rw----1rootroot253, 6 Apr2311:15/dev/dm-6
brw-rw----1rootroot253, 7 Apr2311:15/dev/dm-7
brw-rw----1rootroot253, 8 Apr2311:15/dev/dm-8
brw-rw----1rootroot253, 9 Apr2311:15/dev/dm-9

#ll/dev/mpath/
lrwxrwxrwx1rootroot7Apr2311:15ocr1->../dm-6
lrwxrwxrwx1rootroot7Apr2311:15ocr1p1 ->../dm-11
lrwxrwxrwx1rootroot7Apr2311:15ocr2->../dm-3
lrwxrwxrwx1rootroot7Apr2311:15ocr2p1 ->../dm-8
lrwxrwxrwx1rootroot7Apr2311:15ocr3->../dm-9
lrwxrwxrwx1rootroot7Apr2311:15ocr3p1 ->../dm-10
lrwxrwxrwx1rootroot7Apr2311:15voting1->../dm-0
lrwxrwxrwx1rootroot7Apr2311:15voting1p1 ->../dm-5
lrwxrwxrwx1rootroot7Apr2311:15voting2->../dm-1
lrwxrwxrwx1rootroot7Apr2311:15voting2p1 ->../dm-4
lrwxrwxrwx1rootroot7Apr2311:15voting3->../dm-2
lrwxrwxrwx1rootroot7Apr2311:15voting3p1 ->../dm-7

#ll/dev/mapper/
brw-rw----1rootdisk253, 6 Apr2311:15ocr1
brw-rw----1rootdisk253, 11 Apr2311:15ocr1p1
brw-rw----1rootdisk253, 3 Apr2311:15ocr2
brw-rw----1rootdisk253, 8 Apr2311:15ocr2p1
brw-rw----1rootdisk253, 9 Apr2311:15ocr3
brw-rw----1rootdisk253, 10 Apr2311:15ocr3p1
brw-rw----1rootdisk253, 0Apr2311:15voting1
brw-rw----1rootdisk253, 5 Apr2311:15voting1p1
brw-rw----1rootdisk253, 1Apr2311:15voting2
brw-rw----1rootdisk253, 4 Apr2311:15voting2p1
brw-rw----1rootdisk253, 2Apr2311:15voting3
brw-rw----1rootdisk253, 7 Apr2311:15voting3p1

The /dev/dm-N devices are used internally by device-mapper-multipath and are non-persistent across reboot, so should not be used. The /dev/mpath/ devices are created for multipath devices to be visible together, however, may not be available during early stages of boot, so, again, should not be used. However, /dev/mapper/ devices are persistent and created sufficiently early during boot - use only these devices to access and interact with multipathed devices.

3. Create Raw Devices

During the installation of Oracle Clusterware 10g Release 2 (10.2.0), the Universal Installer (OUI) is unable to verify the sharedness of block devices, therefore requires the use of raw devices (whether to singlepath or multipath devices) to be specified for OCR and voting disks. As mentioned earlier, this is no longer the case from Oracle11g R1 (11.1.0) that can use multipathed block devices directly.

Manually create raw devices to bind against multipathed device partitions (/dev/mapper/*pN). Disregard device permissions for now - this will be addressed later. For example:

#raw/dev/raw/raw1/dev/mapper/ocr1p1
/dev/raw/raw1:boundtomajor253,minor 11
#raw/dev/raw/raw2/dev/mapper/ocr2p1
/dev/raw/raw2:boundtomajor253,minor 8
#raw/dev/raw/raw3/dev/mapper/ocr3p1
/dev/raw/raw3:boundtomajor253,minor 10
#raw/dev/raw/raw4/dev/mapper/voting1p1
/dev/raw/raw4:boundtomajor253,minor 5
#raw/dev/raw/raw5/dev/mapper/voting2p1
/dev/raw/raw5:boundtomajor253,minor 4
#raw/dev/raw/raw6/dev/mapper/voting3p1
/dev/raw/raw6:boundtomajor253,minor 7

#raw-qa
/dev/raw/raw1:boundtomajor253,minor 11
/dev/raw/raw2:boundtomajor253,minor 8
/dev/raw/raw3:boundtomajor253,minor 10
/dev/raw/raw4:boundtomajor253,minor 5
/dev/raw/raw5:boundtomajor253,minor 4
/dev/raw/raw6:boundtomajor253,minor 7

#ls-l/dev/raw/
crw-------1rootroot162,1Apr2311:52raw1
crw-------1rootroot162,2Apr2311:52raw2
crw-------1rootroot162,3Apr2311:52raw3
crw-------1rootroot162,4Apr2311:52raw4
crw-------1rootroot162,5Apr2311:52raw5
crw-------1rootroot162,6Apr2311:52raw6

At this point, you should have raw devices bound to multipathed device partitions using user-defined names.

4. Test Raw Device Accessibility

Test read/write accessibility to and from raw devices from and between cluster nodes, for example:

#ddif=/dev/zeroof=/dev/raw/raw1bs=1024count=100
100+0recordsin
100+0recordsout
102400bytes(102kB)copied,0.762352seconds,134kB/s

#su-oracle
$ddif=/dev/zeroof=/dev/raw/raw1bs=1024count=100
dd:opening`/dev/raw/raw1':Permissiondenied

#ddif=/dev/zeroof=/dev/mapper/ocr1p1 bs=1024count=100
100+0recordsin
100+0recordsout
102400bytes(102kB)copied,0.0468961seconds,2.2MB/s

...

Once testing is complete, unbind all raw devices, for example:

#raw/dev/raw/raw100
/dev/raw/raw1:boundtomajor0,minor0
#raw/dev/raw/raw200
/dev/raw/raw2:boundtomajor0,minor0
#raw/dev/raw/raw300
/dev/raw/raw3:boundtomajor0,minor0
#raw/dev/raw/raw400
/dev/raw/raw4:boundtomajor0,minor0
#raw/dev/raw/raw500
/dev/raw/raw5:boundtomajor0,minor0
#raw/dev/raw/raw600
/dev/raw/raw6:boundtomajor0,minor0

5. Script Creation of Raw Bindings and Permissions

Once raw devices have been created and their accessibility and usability established, configure raw device bindings and permissions. Factoring the undeprecation of raw devices from EL5 Update 4 (initscripts-8.45.30-2), depending on your versioning, configure raw devices accordingly.

For >= EL5U4, configure raw devices via /etc/sysconfig/rawdevices in conjunction with the rawdevices service.

For < EL5U4, use a custom or existing script such as /etc/rc.local to configure raw devices, for example:

# cat /etc/rc.local
#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.

touch /var/lock/subsys/local

#####
# Oracle Cluster Registry (OCR) devices
#####
chown root:oinstall /dev/mapper/ocr**
chmod 660 /dev/mapper/ocr*
raw /dev/raw/raw1 /dev/mapper/ocr1p1
raw /dev/raw/raw2 /dev/mapper/ocr2p2
raw /dev/raw/raw3 /dev/mapper/ocr3p3
sleep 2
chown root:oinstall /dev/raw/raw1
chown root:oinstall /dev/raw/raw2
chown root:oinstall /dev/raw/raw3
chmod 660 /dev/raw/raw1
chmod 660 /dev/raw/raw2
chmod 660 /dev/raw/raw3
#####
# Oracle Cluster Voting disks
#####
chown oracle:oinstall /dev/mapper/voting*
chmod 660 /dev/mapper/voting*
raw /dev/raw/raw4 /dev/mapper/voting1p1
raw /dev/raw/raw5 /dev/mapper/voting2p1
raw /dev/raw/raw6 /dev/mapper/voting3p1
sleep 2
chown oracle:oinstall /dev/raw/raw4
chown oracle:oinstall /dev/raw/raw5
chown oracle:oinstall /dev/raw/raw6
chmod 660 /dev/raw/raw4
chmod 660 /dev/raw/raw5
chmod 660 /dev/raw/raw6

Note: depending on the type and speed of the underlying storage, a sleep(1) of one or two seconds may be necessary between raw device creation and ownership/permission setting.

6. Test the Raw Device Script

Restart the rawdevices service and/or execute the /etc/rc.local script to test the proper creation and permission setting of both raw and multipath devices. Additionally, reboot the server(s) to further verify proper boot-time creation of devices, for example:

# /etc/rc.local
/dev/raw/raw1: bound to major 253, minor 11
/dev/raw/raw2: bound to major 253, minor 8
/dev/raw/raw3: bound to major 253, minor 10
/dev/raw/raw4: bound to major 253, minor 5
/dev/raw/raw5: bound to major 253, minor 4
/dev/raw/raw6: bound to major 253, minor 7

#ll/dev/mapper/
brw-rw----1rootdisk253,6Apr2311:15ocr1
brw-rw----1rootdisk253,11Apr2311:15ocr1p1
brw-rw----1rootdisk253,3Apr2311:15ocr2
brw-rw----1rootdisk253,8Apr2311:15ocr2p1
brw-rw----1rootdisk253,9Apr2311:15ocr3
brw-rw----1rootdisk253,10Apr2311:15ocr3p1
brw-rw----1rootdisk253,0Apr2311:15voting1
brw-rw----1rootdisk253,5Apr2311:15voting1p1
brw-rw----1rootdisk253,1Apr2311:15voting2
brw-rw----1rootdisk253,4Apr2311:15voting2p1
brw-rw----1rootdisk253,2Apr2311:15voting3
brw-rw----1rootdisk253,7Apr2311:15voting3p1

# ls -l /dev/raw/
crw-rw---- 1 root oinstall 162, 1 Apr 23 11:57 raw1
crw-rw---- 1 root oinstall 162, 2 Apr 23 11:57 raw2
crw-rw---- 1 root oinstall 162, 3 Apr 23 11:57 raw3
crw-rw---- 1 oracle oinstall 162, 4 Apr 23 11:57 raw4
crw-rw---- 1 oracle oinstall 162, 5 Apr 23 11:57 raw5
crw-rw---- 1 oracle oinstall 162, 6 Apr 23 11:57 raw6

7. Install Oracle 10gR2 Clusterware

Proceed to install Oracle Clusterware 10g Release 2 (10.2.0), ensuring to specify the appropriate raw devices (/dev/raw/rawN) for OCR and voting disks. OCR devices are initialised (formatted) as part of running the root.sh script. Before running root.sh, be aware that several known issues exist that will cause the Clusterware installation to fail, namely:

• Bug.4679769 FAILED TO FORMAT OCR DISK USING CLSFMT
• Note.414163.1 10gR2 RAC Install issues on Oracle EL5 or RHEL5 or SLES10 (VIPCA Failures)

Due to Bug.4679769, initialisation of multipathed OCR devices will fail. Therefore, before running root.sh, download and apply patch for Bug.4679769. If root.sh was already run without first having applied patch for Bug.4679769, remove (null) the failed, partially initialised OCR structures from all OCR devices, for example:

# dd if=/dev/zero of=/dev/raw/raw1 bs=1M count=25
25+0 records in
25+0records out

#ddif=/dev/zeroof=/dev/raw/raw2 bs=1Mcount=25
25+0recordsin
25+0recordsout

Before re-running root.sh, review Note.414163.1 to proactively address several known (vipca) issues that would otherwise need to be separately resolved later. With the above complete, the running (or re-running) of root.sh should result in proper initialisation of multipathed OCR/voting devices and successful completion of Oracle Clusterware i.e.:

[oracle@oel5a crs]$ sudo ./root.sh
WARNING: directory '/u01/app/oracle/product/10.2.0' is not owned by root
WARNING: directory '/u01/app/oracle/product' is not owned by root
WARNING: directory '/u01/app/oracle' is not owned by root
WARNING: directory '/u01/app' is not owned by root
Checking to see if Oracle CRS stack is already configured

Setting the permissions on OCR backup directory
Setting up NS directories
Oracle Cluster Registry configuration upgraded successfully
WARNING: directory '/u01/app/oracle/product/10.2.0' is not owned by root
WARNING: directory '/u01/app/oracle/product' is not owned by root
WARNING: directory '/u01/app/oracle' is not owned by root
WARNING: directory '/u01/app' is not owned by root
assigning default hostname oel5a for node 1.
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node <nodenumber>: <nodename> <private interconnect name> <hostname>
node 1: oel5a oel5a-int oel5a
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Now formatting voting device: /dev/raw/raw4
Now formatting voting device: /dev/raw/raw5
Now formatting voting device: /dev/raw/raw6
Format of 3 voting devices complete.
Startup will be queued to init within 90 seconds.
Adding daemons to inittab
Expecting the CRS daemons to be up within 600 seconds.
CSS is active on these nodes.
oel5a
CSS is active on all nodes.
Waiting for the Oracle CRSD and EVMD to start
Waiting for the Oracle CRSD and EVMD to start
Oracle CRS stack installed and running under init(1M)
Running vipca(silent) for configuring nodeapps
...

Upon completion of Oracle 10gR2 Clusterware installation, the Clusterware should be up and running, making use of raw devices bound to multipathed devices i.e.:

[oracle@oel5a crs]$ ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 2
Total space (kbytes) : 262144
Used space (kbytes) : 1164
Available space (kbytes) : 260980
ID : 1749049955
Device/File Name : /dev/raw/raw1
Device/File integrity check succeeded
Device/File Name : /dev/raw/raw2
Device/File integrity check succeeded

Cluster registry integrity check succeeded

[root@orl5a /]# crsctl query css votedisk
0. 0 /dev/raw/raw4
1. 0 /dev/raw/raw5
2. 0 /dev/raw/raw6

located 3 votedisk(s).

[root@oel5a /]# crsctl check crs
CSS appears healthy
CRS appears healthy
EVM appears healthy

Refer to Note.394848.1 for any issues, such as -16 EBUSY [Device or resource busy], arising from the continued use of raw devices being bound to multipathed devices.

The requirement to use raw devices for OCR and voting devices solely applies to the initial installation of Oracle 10gR2 Clusterware. Once the installation is complete, OCR and voting devices can be switched to use multipath devices directly - refer to Note.401132.1 for further details.

------------------------------------------------------------------------------

Blog： http://blog.csdn.net/tianlesoftware

网上资源： http://tianlesoftware.download.csdn.net

相关视频：http://blog.csdn.net/tianlesoftware/archive/2009/11/27/4886500.aspx

DBA1 群：62697716(满); DBA2 群：62697977(满)

DBA3 群：62697850 DBA 超级群：63306533;

聊天 群：40132017

--加群需要在备注说明Oracle表空间和数据文件的关系，否则拒绝申请