ASM Instance Changes

Here are some changes that we had to make after using DBCA to create our ASM instances:

First, we had to increase the large pool to 100M. Next, we had to increase the number of processes. The number of processes should be the default (40) times the number of nodes in the Oracle cluster (40 * # of nodes)

Two MetaLink Notes - Bookmark

Here are two more MetaLink notes to bookmark:

1. 368055.1 Deployment of very large databases (10TB to PB range) with Automatic Storage Management (ASM)
2. 139272.1 HP-UX: Asynchronous i/o

Both of these notes, I found useful during our recent VLDB 10g RAC HP-UX Itanium install. A friend of mine at work found the second note and told me about it. Thanks.

We Have An Oracle Cluster

Now, that we have an Oracle cluster and RAC database. Here is a quick high-level touchbase on the RAC environment from the Oracle documentation:

As a DBA, after installation your tasks are to administer your RAC environment at three levels:

• Instance Administration
• Database Administration
• Cluster Administration

For administering Real Application Clusters, use the following tools to perform administrative tasks in RAC:

• Cluster Verification Utility (CVU)—Install and use CVU before you install RAC to ensure that your configuration meets the minimum RAC installation requirements. Also use the CVU for on-going administrative tasks, such as node addition and node deletion.
• Enterprise Manager—Oracle recommends that you use Enterprise Manager to perform administrative tasks whenever feasible.
• Task-specific GUIs such as the Database Configuration Assistant (DBCA) and the Virtual Internet Protocol Configuration Assistant (VIPCA)
• Command-line tools such as SQL*Plus, Server Control (SRVCTL), the Oracle Clusterware command-line interface, and the Oracle Interface Configuration tool (OIFCFG)

I've got plenty of posts saved up and will enter them either tonight or soon. It has been crazy at work. Funny, how you learn so much more while in a storm than you do when things are calm. It is like that in life as well as at work. God has a way of tempering you in the storms that you face and bringing you thru them stronger and better than before. Man, I could preach on that especially with things going on in my personal life right now.

ORA-00600 [KGHALO4]

We got an ORA-00600 [KGHALO4] error today every time we tried to create the ASM instance with DBCA. We found that this is a bug in 10.2.0.1. See MetaLink Note: 340976.1.

Solutions:

Apply the one-off patch for bug 4414666 or set the _enable_NUMA_optimization=FALSE parameter in the init.ora as a workaround. For an ASM instance, two parameters are needed:
_enable_NUMA_optimization = FALSE _disable_instance_params_check = TRUE. Another solution is to apply the 10.2.0.2 patchset which fixes this problem.

Because this bug happens whenever you are trying to create the ASM Instance, it would be advisable to just apply the one-off patch for the bug and then install the 10.2.0.2 patchset as normal once everything is created and working.

OUI Trace

Here is how we traced the Oracle Universal Installer. Execute the following and redirect the output to a file:

./runInstaller -J-DTRACING.ENABLED=true -J-DTRACING.LEVEL=2

We used this method to review what the OUI was doing and determine that the OUI saw ServiceGuard (SG) and SG libraries on the cluster nodes. See my previous posts for more information.

HP Autopath and Oracle

We saw with the OCR and voting raw disks that we had to create a common device special filename (alias) that mapped to the multiple paths to the disks.

/dev/rdsk/ocr
/dev/rdsk/voting

It was all because of the HP Autopaths and virtual storage platform (EVA8000). Oracle Installer only allows entry of one path to the OCR and voting disks. Therefore, we had to create the special filenames in order to use OUI. mksf is the command. See HP portion in Note: 293819.1

We are now wondering about the creation of the ASM disk groups. Will HP autopathing cause us a problem? We should know soon.

Cleanup Failed CRS Install on HP-UX Itanium

Since we had to do this several times, the following is how to cleanup after a failed CRS Install on HP-UX Itanium. (Refer to MetaLink Note: 239998.1). There were problems with the rootdelete.sh and rootdeinstall.sh scripts so we always had to cleanup the failed install manually:

1. srvctl stop nodeapps -n (we didn't have to do this because our failed installs never got this far).
2. As root:
3. rm /sbin/init.d/init.cssd
4. rm /sbin/init.d/init.crs
5. rm /sbin/init.d/init.crsd
6. rm /sbin/init.d/evmd
7. rm /sbin/rc2.d/K001init.crs
8. rm /sbin/rc2.d/K960init.crs
9. rm /sbin/rc3.d/K001init.crs
10. rm /sbin/rc3.d/K960init.crs
11. rm /sbin/rc3.d/S960init.crs
12. rm -Rf /var/opt/oracle/scls_scr
13. rm -Rf /var/opt/oracle/oprocd
14. rm /etc/inittab.crs
15. cp /etc/inittab.orig /etc/inittab
16. If they are not already down, kill the EVM, CRS, and CSS processes.
17. rm -Rf /var/tmp/.oracle
18. rm -Rf /tmp/.oracle
19. remove the ocr.loc file
20. rm -Rf /* CRS Install Location */
21. De-install the CRS home in the OUI
22. Clean out the OCR and voting files with dd commands Example:
23. dd if=/dev/zero of=/dev/rdsk/voting bs=8192 count=2560
24. dd if=/dev/zero of=/dev/rdsk/ocr bs=8192 count=12800
25. rm -Rf /app/oracle/oraInventory

Once those are done, you can restart the OUI install of Clusterware at the very beginning. Oracle has also just released a new cleanup utility for failed CRS installs here is the link:

http://download-west.oracle.com/otndocs/products/clustering/deinstall/clusterdeconfig.zip

The new script didn't work for us either. Of course, I didn't try it after our system admins removed ServiceGuard so it may work now.

Clusterware Install Tips HP-UX Itanium

OK, we finally have a working Oracle Cluster. Praise God! I wanted to add some of the new things that have been discovered since the last post.

Make sure if you are not going to use HP ServiceGuard on your RAC cluster, that all of ServiceGuard has been stopped and uninstalled. Don't leave any libraries laying around. We found out this the hard way, after spending over a week trying to figure out why the Oracle Installer was acting so crazy and bizarre.

Here are the symptoms: First, if the OUI does not give you the option to add the other nodes and you have to use a configuration file, this is a red flag that the OUI thinks that you are using some vendor cluster software (in this case HP ServiceGuard) instead of using Oracle's. Secondly, if you have some variables that are not assigned (see previous post) in the rootconfig script this indicates that it is not really trying to install rather it is trying to upgrade/update the OCR.

If for some reason, you get the clusterware services running on one of the nodes but it doesn't start on the others and locks up. It probably means that your removal of ServiceGuard was incomplete and left a few SG libraries laying around.

We found all of this out the hard way because HP installed and started ServiceGuard when they installed the HP 9000 Superdome!

Finally, here is an undocumented procedure for HP-UX Itanium Clusterware 10gR2 installation: Shutdown the VIP interface BEFORE beginning the install. If you don't, an error message will appear saying that the VIP interface is being used by another system. Then you have to shut the VIPs down before continuing the install. This is weird because the VIP must be up in order for cluvfy nodecon to work.

CVU Shared Storage Accessibility Check

We are almost ready to reinstall Clusterware. However our cluvfy comp ssa check is returning the following in the cluvfy trace file:

ERROR>/tmp/9999//bin/lsnodes: cannot get local node number

Although Cluster Verify Utility is an excellent tool for prerequisites, there is room for improvement. Since we did not run cluvfy as thoroughly on our first attempt, we did not encounter Bug 4714708 - Cvu Cannot See Shared Drives.

It turns out that CVU currently does not work with devices other than SCSI devices.

Failed To Upgrade OCR Message

Now, we are getting a "Failed to upgrade Oracle Cluster Registry Configuration" message in the rootconfig shell script that is ran by root.sh at the end of the Clusterware installation. A friend of mine found most of the problem and described it as this:

"The rootconfig script has a few problems. The CRS_HOST_NAME_LIST and CRS_NODE_NAME_LIST are not populated. Script failed at this point:

if $CH/bin/ocrconfig -upgrade $CRS_ORACLE_OWNER $CRS_DBA_GROUP;
then$ECHO "Oracle Cluster Registry configuration upgraded successfully"
else$ECHO "Failed to upgrade Oracle Cluster Registry configuration"
exit 1
fi

libocr10.so is missing(running in debug mode)Commented this out and added environment variables and everything began to run. Script is trying to start init.crs but never starts the daemon."

So, once we talked to Oracle Support about this they suggested backing out the Clusterware install. Note 239998.1. Then re-running cluvfy and checking problems. In reviewing the CLUVFY output and installation documentation, we discovered missing OS patches and missing applications. HP C and C++ are not installed on all of the nodes.

We are canceling the re-install until all HP-UX Itanium patches (and those which are superseded) are installed. We are also waiting on a HP C and C++ install on the nodes which are missing them.

We are also asking Oracle Support to research whether or not GNU HP-UX Itanium C and C++ software can be substituted for the more costly HP licensed versions.

Clusterware Install Stopped at OCR Location

The Oracle Universal Installer (OUI) stopped while installing Clusterware with the following error :

"The location /dev/dsk/xxx, entered for the Oracle Cluster Registry (OCR) is not shared across all the nodes in the cluster. Specify a shared raw partition or cluster file system file that is visible by the same name on all nodes of the cluster."

This is caused because the /dev/dsk is a block device and OUI does not recognize or use block devices. So in order to get it to continue, we had to point to the /dev/rdsk/xxx raw character device instead.

You need to bind raw devices and use these raw devices within OUI. See MetaLink Note: 363995.1 for a little more detail.

We also had to do the same thing for the voting disk.

A Pattern is Developing...

I'm beginning to see a pattern. This matches what I've been told by other DBAs and Mike Ault during a RAC class. You spend days working on just the clusterware install before beginning the database install. Looks like Oracle would have these difficulties fixed especially by Release 2.

CLUVFY User Equivalence Failure

The runcluvfy.sh comp nodecon kept failing the User Equivalence Test. Turns out the cause was two different issues.

The first issue was with the path to ssh. To solve this issue we had to reference Oracle MetaLink Note 36598.1. Here is an excerpt from the Note:

"Generate a trace file using the cluvfy debugging environment variable SRVM_TRACE=TRUE as in Note 316817.1 and the resulting trace file will show something like:checkRemoteExecutionSetup:: Error checking user equivalence using Secured Shell'/usr/local/bin/ssh'; Fri May 12 15:38:18 CEST 2006. Cluvfy was looking in the wrong location for ssh. As it can be seen in the trace file, the location where the utility is searched is/usr/local/bin/, whilst it resides in /usr/bin.
To implement the solution, please execute the following steps:

1. add the symbolic links in /usr/local/bin pointing to the real ssh location, ie /usr/bin. Do the same for the scp command.
2. generate the ssh keys to all the cluster locations.
3. Run cluvfy again"

The second issue was with the banner. "ssh node date" would return a security banner in addition to the date. In order to get nodecon to work, I had to rename the /etc/issue.net (in HP-UX Itanium) to keep a banner from displaying. See MetaLink note: 338045.1. Of course, once we are finished with the install, I will rename it back.