Wednesday, September 27, 2006

ORA-29701 Cluster Manager

I had something strange occur today. A user query kept returning ORA-29701: unable to connect to Cluster Manager. Therefore, I searched MetaLink for this error. But, I couldn't find anything that applied to my situation (10gR2 RAC on HP-UX Itanium). This error only showed up on one of the nodes. The other nodes and instances were fine.

After stopping everything on the node, the ASM instance would not start and immediately barked at me with the same ORA-29701 error. At this point I asked someone else that has more experience with Oracle Clusterware than I do.

They checked it out and found that somehow the ASM /dev/rdsks and special files for the OCR and voting disks had changed ownership. Someone with root access must have run insf -e to reinstall the special files. Oh, great!

A sys admin had already created a shell script to change the ownership of the /dev/rdsks to oracle:dba and chmod them to 660. So, all we had to do was ask one of our sysadmins to run the script. They also had to manually chown root:oinstall /dev/voting and chown oracle:oinstall /dev/ocr.

So, if you get an ORA-29701 and can't figure it out, check the owner and permissions of your Oracle devices.

8 comments:

Anonymous said...

Steven, Good Morning

could you tell me what was the oracle version you were using that day?

10.2.0.3 ?

Regards,
Fábio Sbano

sedwardba said...

Hi Fabio, it was 10.2.0.2 in 2006

Anonymous said...

1. All connections had the same problems?
2 Were you using some ConnectionPool?

Regards,
Thank you very much

sedwardba said...

Hi Fabio,

Yes, all connections had the same problems and we were not using a ConnectionPool.

Anonymous said...

I'm trying to reproduce this error 'ORA-29701' but so far not received the message on the client.
Do you have any idea?

sedwardba said...

I couldn't reproduce it either. It was one of those strange Oracle things that is not easily reproducible. Are you seeing this error? Or, are you trying to test for it?

Thanks,
SE

Anonymous said...

test only

sedwardba said...

Then you are probably OK and just haven't hit the same bug I did.