Opatch with RAC

I got to use Opatch for RAC today to install some one-off patches. That is a subject for later. Man, what a lot of bugs in 10.2. Opatch is cluster aware and will propagate the changes to the other nodes and relink Oracle on the other nodes. There is an issue however. You cannot have UNIX banners being used with ssh. This is the same issue that we ran into on the Clusterware install. So, we had to rename the /etc/issue.net in order to get Opatch to propagate the changes to the other nodes. Opatch actually works really well.

ORA-29701 Cluster Manager

I had something strange occur today. A user query kept returning ORA-29701: unable to connect to Cluster Manager. Therefore, I searched MetaLink for this error. But, I couldn't find anything that applied to my situation (10gR2 RAC on HP-UX Itanium). This error only showed up on one of the nodes. The other nodes and instances were fine.

After stopping everything on the node, the ASM instance would not start and immediately barked at me with the same ORA-29701 error. At this point I asked someone else that has more experience with Oracle Clusterware than I do.

They checked it out and found that somehow the ASM /dev/rdsks and special files for the OCR and voting disks had changed ownership. Someone with root access must have run insf -e to reinstall the special files. Oh, great!

A sys admin had already created a shell script to change the ownership of the /dev/rdsks to oracle:dba and chmod them to 660. So, all we had to do was ask one of our sysadmins to run the script. They also had to manually chown root:oinstall /dev/voting and chown oracle:oinstall /dev/ocr.

So, if you get an ORA-29701 and can't figure it out, check the owner and permissions of your Oracle devices.

Database Control RMAN Wizard Bug

Here is a bug that I found in our 10gR1 systems. This bug has to do with Enterprise Manager (EM) Database Control 10.1.0.5 and scheduling backups with the Recovery Manager (RMAN) wizard. Whenever you do a custom RMAN backup and choose to only backup certain tablespaces, the UNDO tablespace is never given as an option.

Now, without the undo tablespace, database recovery is not possible. However, the EM RMAN wizard will not allow you the option of adding the undo tablespace to the list of tablespaces to be selected for backup.

So, here is the workaround:

• Modify the RMAN script that is created and manually include the undo tablespace in your list of tablespaces to be backed up. Then submit your job.

I will be creating some RMAN backups for a 10.2 database soon and will let you know if this bug is also in 10gR2.

Old Dog New Tricks

With all of the assistance I've had lately, this old dog has learned some new tricks. Now, everybody I'm sure already knows these; however, in my career I've had to learn things by the seat of my pants and on the job in production.

tnsnames.ora changes do not require restart of the listener. Kind of cool. And may favorite is "lsnrctl reload". The reload resets the listener without stopping and starting it.

HP EVA8000 Autopath Tuning

Here is some information that might be useful to you. Following our RAC cluster setup and install, the EVA 8000 performance was very disappointing. First, our Oracle performance was disturbing. Then we did some more I/O benchmarks using "dd" and found that I/O performance was actually worse than the first baseline benchmark. So, this indicated a problem with the underlying storage and not the Oracle setup.

The UNIX administrators, HP, and my peers began looking into the problem. I wasn't totally engaged but want to post the information anyway. Props to my colleages. This is more system administration territory; however, as DBAs we need to know the impact of EVA tuning.

HP checked the load balancing policy on the nodes. It was set to “No Load Balancing” which greatly impacts performance. Now, how it was changed to "No Load Balancing" after the first I/O benchmarks is still a mystery.

To see the LUNs, issue the command “autopath display” as root. This will list all the LUNs and show the HP Autopath load balancing policy.

root@hostname:/ # autopath display
...
Load Balancing Policy : No Load Balancing
...

So, HP and my peers recommended that the HP Autopath load balancing policy be set to round robin for all LUNs. The autopath set_lbpolicy command sets the load balancing policy for the specified device path.

autopath set_lbpolicy <{policy name} {path}>
description: sets load balancing policy
usage: autopath set_lbpolicy <{policy name} {path}>
Policy name: The load balancing policy to set
Valid policies are
RR : Round Robin.
SST : Shortest Service Time.
SQL : Shortest Queue Length.
NLB/OFF : No load Balancing.
Path : Device Special File e.g./dev/dsk/c#t#d#

Example:
# autopath set_lbpolicy RR /dev/dsk/c0t0d0
The example above sets the policy to Round Robin.

Here is a little more information about what the policies mean:

• RR : Round Robin. - I/O is routed through the paths of the active controller in round-robin order
• SST : Shortest Service Time. - is a measurement against the average service time of the IOs on a path
• SQL : Shortest Queue Length. - is a calculation on the device queue depth when it is on a certain path.
• NLB/OFF : No load Balancing. - No consideration given to service times or queue lengths, typically impacts performance.

ASM Instance Changes

Here are some changes that we had to make after using DBCA to create our ASM instances:

First, we had to increase the large pool to 100M. Next, we had to increase the number of processes. The number of processes should be the default (40) times the number of nodes in the Oracle cluster (40 * # of nodes)

Two MetaLink Notes - Bookmark

Here are two more MetaLink notes to bookmark:

1. 368055.1 Deployment of very large databases (10TB to PB range) with Automatic Storage Management (ASM)
2. 139272.1 HP-UX: Asynchronous i/o

Both of these notes, I found useful during our recent VLDB 10g RAC HP-UX Itanium install. A friend of mine at work found the second note and told me about it. Thanks.

We Have An Oracle Cluster

Now, that we have an Oracle cluster and RAC database. Here is a quick high-level touchbase on the RAC environment from the Oracle documentation:

As a DBA, after installation your tasks are to administer your RAC environment at three levels:

• Instance Administration
• Database Administration
• Cluster Administration

For administering Real Application Clusters, use the following tools to perform administrative tasks in RAC:

• Cluster Verification Utility (CVU)—Install and use CVU before you install RAC to ensure that your configuration meets the minimum RAC installation requirements. Also use the CVU for on-going administrative tasks, such as node addition and node deletion.
• Enterprise Manager—Oracle recommends that you use Enterprise Manager to perform administrative tasks whenever feasible.
• Task-specific GUIs such as the Database Configuration Assistant (DBCA) and the Virtual Internet Protocol Configuration Assistant (VIPCA)
• Command-line tools such as SQL*Plus, Server Control (SRVCTL), the Oracle Clusterware command-line interface, and the Oracle Interface Configuration tool (OIFCFG)

I've got plenty of posts saved up and will enter them either tonight or soon. It has been crazy at work. Funny, how you learn so much more while in a storm than you do when things are calm. It is like that in life as well as at work. God has a way of tempering you in the storms that you face and bringing you thru them stronger and better than before. Man, I could preach on that especially with things going on in my personal life right now.

ORA-00600 [KGHALO4]

We got an ORA-00600 [KGHALO4] error today every time we tried to create the ASM instance with DBCA. We found that this is a bug in 10.2.0.1. See MetaLink Note: 340976.1.

Solutions:

Apply the one-off patch for bug 4414666 or set the _enable_NUMA_optimization=FALSE parameter in the init.ora as a workaround. For an ASM instance, two parameters are needed:
_enable_NUMA_optimization = FALSE _disable_instance_params_check = TRUE. Another solution is to apply the 10.2.0.2 patchset which fixes this problem.

Because this bug happens whenever you are trying to create the ASM Instance, it would be advisable to just apply the one-off patch for the bug and then install the 10.2.0.2 patchset as normal once everything is created and working.

OUI Trace

Here is how we traced the Oracle Universal Installer. Execute the following and redirect the output to a file:

./runInstaller -J-DTRACING.ENABLED=true -J-DTRACING.LEVEL=2

We used this method to review what the OUI was doing and determine that the OUI saw ServiceGuard (SG) and SG libraries on the cluster nodes. See my previous posts for more information.

HP Autopath and Oracle

We saw with the OCR and voting raw disks that we had to create a common device special filename (alias) that mapped to the multiple paths to the disks.

/dev/rdsk/ocr
/dev/rdsk/voting

It was all because of the HP Autopaths and virtual storage platform (EVA8000). Oracle Installer only allows entry of one path to the OCR and voting disks. Therefore, we had to create the special filenames in order to use OUI. mksf is the command. See HP portion in Note: 293819.1

We are now wondering about the creation of the ASM disk groups. Will HP autopathing cause us a problem? We should know soon.

Cleanup Failed CRS Install on HP-UX Itanium

Since we had to do this several times, the following is how to cleanup after a failed CRS Install on HP-UX Itanium. (Refer to MetaLink Note: 239998.1). There were problems with the rootdelete.sh and rootdeinstall.sh scripts so we always had to cleanup the failed install manually:

1. srvctl stop nodeapps -n (we didn't have to do this because our failed installs never got this far).
2. As root:
3. rm /sbin/init.d/init.cssd
4. rm /sbin/init.d/init.crs
5. rm /sbin/init.d/init.crsd
6. rm /sbin/init.d/evmd
7. rm /sbin/rc2.d/K001init.crs
8. rm /sbin/rc2.d/K960init.crs
9. rm /sbin/rc3.d/K001init.crs
10. rm /sbin/rc3.d/K960init.crs
11. rm /sbin/rc3.d/S960init.crs
12. rm -Rf /var/opt/oracle/scls_scr
13. rm -Rf /var/opt/oracle/oprocd
14. rm /etc/inittab.crs
15. cp /etc/inittab.orig /etc/inittab
16. If they are not already down, kill the EVM, CRS, and CSS processes.
17. rm -Rf /var/tmp/.oracle
18. rm -Rf /tmp/.oracle
19. remove the ocr.loc file
20. rm -Rf /* CRS Install Location */
21. De-install the CRS home in the OUI
22. Clean out the OCR and voting files with dd commands Example:
23. dd if=/dev/zero of=/dev/rdsk/voting bs=8192 count=2560
24. dd if=/dev/zero of=/dev/rdsk/ocr bs=8192 count=12800
25. rm -Rf /app/oracle/oraInventory

Once those are done, you can restart the OUI install of Clusterware at the very beginning. Oracle has also just released a new cleanup utility for failed CRS installs here is the link:

http://download-west.oracle.com/otndocs/products/clustering/deinstall/clusterdeconfig.zip

The new script didn't work for us either. Of course, I didn't try it after our system admins removed ServiceGuard so it may work now.

Clusterware Install Tips HP-UX Itanium

OK, we finally have a working Oracle Cluster. Praise God! I wanted to add some of the new things that have been discovered since the last post.

Make sure if you are not going to use HP ServiceGuard on your RAC cluster, that all of ServiceGuard has been stopped and uninstalled. Don't leave any libraries laying around. We found out this the hard way, after spending over a week trying to figure out why the Oracle Installer was acting so crazy and bizarre.

Here are the symptoms: First, if the OUI does not give you the option to add the other nodes and you have to use a configuration file, this is a red flag that the OUI thinks that you are using some vendor cluster software (in this case HP ServiceGuard) instead of using Oracle's. Secondly, if you have some variables that are not assigned (see previous post) in the rootconfig script this indicates that it is not really trying to install rather it is trying to upgrade/update the OCR.

If for some reason, you get the clusterware services running on one of the nodes but it doesn't start on the others and locks up. It probably means that your removal of ServiceGuard was incomplete and left a few SG libraries laying around.

We found all of this out the hard way because HP installed and started ServiceGuard when they installed the HP 9000 Superdome!

Finally, here is an undocumented procedure for HP-UX Itanium Clusterware 10gR2 installation: Shutdown the VIP interface BEFORE beginning the install. If you don't, an error message will appear saying that the VIP interface is being used by another system. Then you have to shut the VIPs down before continuing the install. This is weird because the VIP must be up in order for cluvfy nodecon to work.

CVU Shared Storage Accessibility Check

We are almost ready to reinstall Clusterware. However our cluvfy comp ssa check is returning the following in the cluvfy trace file:

ERROR>/tmp/9999//bin/lsnodes: cannot get local node number

Although Cluster Verify Utility is an excellent tool for prerequisites, there is room for improvement. Since we did not run cluvfy as thoroughly on our first attempt, we did not encounter Bug 4714708 - Cvu Cannot See Shared Drives.

It turns out that CVU currently does not work with devices other than SCSI devices.

Failed To Upgrade OCR Message

Now, we are getting a "Failed to upgrade Oracle Cluster Registry Configuration" message in the rootconfig shell script that is ran by root.sh at the end of the Clusterware installation. A friend of mine found most of the problem and described it as this:

"The rootconfig script has a few problems. The CRS_HOST_NAME_LIST and CRS_NODE_NAME_LIST are not populated. Script failed at this point:

if $CH/bin/ocrconfig -upgrade $CRS_ORACLE_OWNER $CRS_DBA_GROUP;
then$ECHO "Oracle Cluster Registry configuration upgraded successfully"
else$ECHO "Failed to upgrade Oracle Cluster Registry configuration"
exit 1
fi

libocr10.so is missing(running in debug mode)Commented this out and added environment variables and everything began to run. Script is trying to start init.crs but never starts the daemon."

So, once we talked to Oracle Support about this they suggested backing out the Clusterware install. Note 239998.1. Then re-running cluvfy and checking problems. In reviewing the CLUVFY output and installation documentation, we discovered missing OS patches and missing applications. HP C and C++ are not installed on all of the nodes.

We are canceling the re-install until all HP-UX Itanium patches (and those which are superseded) are installed. We are also waiting on a HP C and C++ install on the nodes which are missing them.

We are also asking Oracle Support to research whether or not GNU HP-UX Itanium C and C++ software can be substituted for the more costly HP licensed versions.

Clusterware Install Stopped at OCR Location

The Oracle Universal Installer (OUI) stopped while installing Clusterware with the following error :

"The location /dev/dsk/xxx, entered for the Oracle Cluster Registry (OCR) is not shared across all the nodes in the cluster. Specify a shared raw partition or cluster file system file that is visible by the same name on all nodes of the cluster."

This is caused because the /dev/dsk is a block device and OUI does not recognize or use block devices. So in order to get it to continue, we had to point to the /dev/rdsk/xxx raw character device instead.

You need to bind raw devices and use these raw devices within OUI. See MetaLink Note: 363995.1 for a little more detail.

We also had to do the same thing for the voting disk.

A Pattern is Developing...

I'm beginning to see a pattern. This matches what I've been told by other DBAs and Mike Ault during a RAC class. You spend days working on just the clusterware install before beginning the database install. Looks like Oracle would have these difficulties fixed especially by Release 2.

CLUVFY User Equivalence Failure

The runcluvfy.sh comp nodecon kept failing the User Equivalence Test. Turns out the cause was two different issues.

The first issue was with the path to ssh. To solve this issue we had to reference Oracle MetaLink Note 36598.1. Here is an excerpt from the Note:

"Generate a trace file using the cluvfy debugging environment variable SRVM_TRACE=TRUE as in Note 316817.1 and the resulting trace file will show something like:checkRemoteExecutionSetup:: Error checking user equivalence using Secured Shell'/usr/local/bin/ssh'; Fri May 12 15:38:18 CEST 2006. Cluvfy was looking in the wrong location for ssh. As it can be seen in the trace file, the location where the utility is searched is/usr/local/bin/, whilst it resides in /usr/bin.
To implement the solution, please execute the following steps:

1. add the symbolic links in /usr/local/bin pointing to the real ssh location, ie /usr/bin. Do the same for the scp command.
2. generate the ssh keys to all the cluster locations.
3. Run cluvfy again"

The second issue was with the banner. "ssh node date" would return a security banner in addition to the date. In order to get nodecon to work, I had to rename the /etc/issue.net (in HP-UX Itanium) to keep a banner from displaying. See MetaLink note: 338045.1. Of course, once we are finished with the install, I will rename it back.

ASM File Locations

From Mike Ault's book Oracle 10g Grid & Real Application Cluster, the following shows the files locatable on the ASM instance and files that have to reside on the file system:

On the ASM Instance:

• Data Files
• Redo Files
• Control Files
• Archive Log Files

On Raw Partitions:

• Voting Disk
• OCR File

On the Local File System:

• Oracle Home Files
• Clusterware Home Files
• ASM Instance Home Files
• Alert Log, Trace Files
• Files for External Tables
• utl_file_dir location

X11 Forwarding From SUDO

X11 forwarding is not working using PuTTY and ssh for users once you "become" them using SUDO. Here is the error message:

$ Xlib: connection to "xxx.xx.xx.xxx" refused by server
Xlib: PuTTY X11 proxy: wrong authentication protocol attempted
Error: Can't open display: xxx.xx.xx.xxx:xx.x

Below are the steps used to successfully transfer xauth information to another user (Oracle in this case)

1. Enable X11 forwarding on your terminal application and login as you (as stated above)
   ‘chmod 644 .Xauthority’ (needs to be done every time service account needs access, it will reset when you log out)
2. ‘become service account x’
3. ‘xauth merge ~username/.Xauthority’ (needs to be done every time service account access is needed)

Once you get a copy of the .Xauthority file to /home/oracle it should work.