Thursday, February 02, 2017

Oracle Wait - enq: TM - contention

Oracle Waits.  Aren't they fun?  Here is an MOS document that has lots of good information regarding enq: TM - contention waits:  WAITEVENT: "enq: TM - contention" Reference Note (Doc ID 1980175.1)

Below are Copy and Pastes from the MOS doc:

Systemwide Waits:

"enq: TM - contention" waits are generally related to the application code being executed and do not indicate a problem with the DB itself. Possible things to check: Drill down into individual sessions showing high waits to understand which application code , which objects, and which SQL/s are involved in the waits. ASH reports for sessions showing high "enq: TM - contention" wait time may help.

Finding Blockers:


You can find current blockers by querying GV$LOCK like this:
SELECT distinct w.tm, w.p2 OBJECT_ID, l.inst_id, l.sid, l.lmode, l.request
 FROM 
  ( SELECT p2, p3, 'TM-'||substr(p2raw,-8)||'-'||lpad(p3,8,'0') TM
      FROM v$session_wait 
     WHERE event='enq: TM - contention'
       and state='WAITING'
  ) W, 
  gv$lock L
 WHERE l.type(+)='TM'
   and l.id1(+)=w.p2
   and l.id2(+)=w.p3
 ORDER BY tm, lmode desc, request desc
;

Troubleshooting


See the following documents for help troubleshooting issues relating to TM lock waits:
Document:1905174.1 Resolving Issues Where 'enq: TM - contention' Waits are Occurring
Document:33453.1 Locking and Referential Integrity

I did a search on this wait event. And, also found some blog posts about this wait happening whenever there are un-indexed foreign key constraints on a table.

But, in my humble opinion, I would use the Oracle MOS document as my guide first before looking for unindexed foreign key constraints.

To find all un-indexed foreign key constraints:

SELECT * FROM (
SELECT c.table_name, cc.column_name, cc.position column_position
FROM   dba_constraints c, dba_cons_columns cc
WHERE  c.constraint_name = cc.constraint_name
AND    c.constraint_type = 'R'
MINUS
SELECT i.table_name, ic.column_name, ic.column_position
FROM   dba_indexes i, dba_ind_columns ic
WHERE  i.index_name = ic.index_name
)
ORDER BY table_name, column_position;

Happy waiting ;-)

Wednesday, February 01, 2017

Oracle ASM Header Backup -- Update

This post will be an update to a post that I made on Thursday, December 09, 2010 about Oracle ASM Header Backups.  

There are multiple ways to skin this cat.  Another option for backing up your ASM file headers is to use "kfed".  Yes, I know, Oracle Support doesn't recommend that we use it.  However, if you ever need to recover the header, you will be very glad you did.  Below is a partial example of a backup using kfed:

/u01/app/oracle/product/12.1.0/grid_1/bin/kfed read ${ASM_DEV_DIR}/${i} text=${HDR_BACKUP_DIR}/${i}.${TODAY}.bkp

Utilizing "kfed read" will allow you to recover the header in case of corruption.  If you have header corruption and your other methods of header backup (like md_backup and dd's) do not work, then you can tell Oracle Support that you have a kfed file of the header from before the corruption.

As with any corruption, make sure you open an SR with Oracle Support before attempting any repair yourself.

For more information, review the information in My Oracle Support (MOS) regarding kfed.

Thursday, December 08, 2016

Wow, 6 Years Without A Post

Hard to believe that six years have passed since my last post.  I've been busy apparently.  Going to start making posts again after attending a PASS webinar entitled "What SQL Server Professionals Can Learn From Writing and Blogging" by Edwin Sarmiento.  The webinar was sponsored by the Professional Development PASS Virtual Chapter.

[http://www.EdwinMSarmiento.com]

So, what is a 19 year experienced Oracle DBA doing learning Microsoft SQL Server?   I'm Starting to work with different RDBMSs and NOSQL technologies.  Which is really a blast after being limited to only one vendor.  I'm also getting to work with DataStax [Cassandra].  Who said you can't teach an old dog new tricks?

In Edwin's presentation, he talked about the 3M's with blogging:

MASTERY

  • Writing helps CLARIFY your thinking
  • Writing forces you to VALIDATE your assumptions
  • Writing helps REINFORCE your knowledge

MARKETING

  • Writing helps ESTABLISH CREDIBILITY
  • Blogging builds your ONLINE REPUTATION (e.g. Brent Ozar, Pinal Dave, ...)
 
 MEETING

  • Writing builds CONNECTIONS with readers
  • Writing helps you see other's PERSPECTIVE
  • Writing develops valuable RELATIONSHIPS
     
That is all for now.  Edwin recommended scheduling 15 minutes everyday for blogging.  My time is up ;-)

Thursday, December 09, 2010

10g Impdp ORA-00959 $deleted$ Fix

Ran into a problem with an Import Datapump table restore. This is the error message:

Import: Release 10.2.0.4.0 - 64bit Production on Monday, 06 December, 2010 17:57:27 Copyright (c) 2003, 2007, Oracle. All rights reserved. ;;; Connected to: Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production With the Partitioning, Real Application Clusters, OLAP and Real Application Testing options Master table "SYSTEM"."SYS_IMPORT_FULL_01" successfully loaded/unloaded Starting "SYSTEM"."SYS_IMPORT_FULL_01": parfile=hm_camp_m010.par Processing object type TABLE_EXPORT/TABLE/TABLE ORA-39083: Object type TABLE failed to create with error: ORA-00959: tablespace '_$deleted$16$0' does not exist

Basically, bombed because the tablespace _$deleted$16$0 doesn't exist. Looked it up on MetaLink and found this document:

Import Partition Table Fails With Ora-00959: Tablespace '_$Deleted$0$0' [ID 798142.1]

There is an open unpublished bug where basic problem is that a tablespace is renamed to one that used to exist on the database. Here are the steps that will cause this problem.

create tablespace aa
create tablespace bb
drop tablespace aa
rename bb to aa

Then why it occurs is because when a tablespace is dropped, its ts$ entry is set to an invalid state, but it is not deleted. When a rename is done, if the target name already has a ts$ entry, then during the rename, because we cannot have two ts$ entries with the same name, the _$deleted$ construct is introduced. The change is not taken care of properly, and is flushed to other dictionary tables causing this problem.

The note didn't have good instructions to fix my issue. So, I'm recording what I did here:
  1. Run impdp with this parameter SQLFILE=dir:name to pull the DDL for the object.
  2. Modify and execute the SQL created in step 1.
  3. Run impdp with this parameter CONTENT=DATA_ONLY to bypass the Metadata portion of impdp steps.

In step 2 you take the create table SQL and modify it by creating the table in a tablespace that does exist.

Hopefully, this will help you.


Oracle ASM Header Backup

One of my DBA friends shared this information with me which I reviewed and researched. Kudos to my friend.

A situation popped up recently for one team where a drive went into predictive failure, and a controller hung, corrupting the ASM headers upon reboot. This resulted them having to completely rebuild the disk groups and restore the database from tape. If the team would have had a backup of their ASM headers, all of that wouldn’t have been necessary.

ASM metadata corruption is a clear and present danger that we as DBAs have to recognize and prepare for. It is nothing to be embarrassed about, and nothing to fear, as long as you are ready for it.

Oracle 11g

md_backup

md_backup is ran from the ASMCMD> prompt. It very simply creates a file that is a backup of your metadata, for all of your disk groups, or just a few if you want to specify. By default, it writes to the current directory and backs up all of the mounted disk groups header files.

Arguments: -b and –g

-b Specifies a location to store the backup file

-g Specifies the disk group to back up

Example: ASMCMD> md_backup –b /mydir/somebackupfile –g DG1 –g DG3

This will backup disk groups DG1 and DG3 to the file /mydir/somebackupfile.

md_restore

md_restore is a bit more involved than md_backup. As you may have guessed, it restores previously backed up metadata to disk groups you specifiy.

Arguments: -b –i –t –f –g –o

-b Specifies the backup file to read

-i Ignores errors. (Normally, if md_restore hits an error, it stops. This argument suppressed that.

-t Specifies the type of disk group to be created/restored. “full” creates a disk group and restores the metadata. “nodg” restores metadata only, and “newdg” creates a new disk group with a different name using in conjunction with the –o argument.

-o Renames the disk group

-f writes the SQL commands to be performed in the restore to a file instead of executing them (sql_script_file).

-g selects the disk groups to be restored. If none are specified, it restores all of them.

Example: ASMCMD> md_restore –t full –g DG1 –i backup_file

This will restore disk group DG1 using the specified file.

Example: ASMCMD> md_restore –t nodg –g DG1 –i backup_file

This will restore an existing disk group’s metadata.

Oracle 10g

When implementing ASM backups should not only be made of the data within ASM but the disk headers should also be backed up. Oracle 10g does not provide a way out of the box to do this but Oracle 11g provides two new commands in the asmcmd utility to backup disk headers. The header of a disk is comprised of the first 4096 bytes. While not supported it is possible to use the UNIX dd to read and write these bytes to a backup file.

dd if=device name of=unique header file bs=4096 count=1

That will capture the first 4k block of the ASM file. You would do this for each file, creating a separate backup for each. A simple dd will write it back in to the header, if you need to restore from your backup. Example:

dd if=/dev/rdisk/asmvol1 of=/u01/ora_backup/asmvo1_header.bak bs=4096 count=1

This would capture the first 4k of the given ASM file. Example:

dd if=/u01/ora_backup/asmvol1_header.bak of=/dev/rdisk/asmvol1

This would write in the backed up header back into the file it was backed up from.

It is always a good idea to keep backups of your disk group metadata, because no one can know when the corruption gremlin might strike.



Tuesday, May 12, 2009

VIP Address Fun

During a recent RAC install, we were given an IP address for our VIP to be created at install. However, the IP address we got from network was a "private" address and not a "public" address. During the Clusterware install the vipca failed complaining that the interface is not public. As per RFC 1918 networks in the following ranges are considered private. For more details please review http://tools.ietf.org/html/rfc1918

10.0.0.0 - 10.255.255.255 (10/8 prefix)
172.16.0.0 - 172.31.255.255 (172.16/12 prefix
192.168.0.0 - 192.168.255.255 (192.168/16 prefix

Here is another link: http://www.faqs.org/rfcs/rfc1918.html

When the vipca silent installation fails because of the address range, you have to run it manually or issue the command:

$ORA_CRS_HOME/bin/srvctl add nodeapps -n crmnode1 -o /ora/ora10 -A 1.2.3.4/255.255.255.0/lan0

See Note 316583.1 VIPCA FAILS COMPLAINING THAT INTERFACE IS NOT PUBLIC for more information.

Just follow the instructions in the note and you will be fine. However, when I first followed the instructions of the note, I couldn't get the add nodeapps for the VIP to work. Turns out the note did not have the correct syntax. I pointed it out to Oracle Support and the author of the note updated the syntax. So, it is always good to verify syntax from a separate source if you have problems with Oracle documents.

Also, the cluvfy will indicate the following:

WARNING:
Could not find a suitable set of interfaces for VIPs.
Result: Node connectivity check failed.

Even though everything is OK, you are just using a non-standard private VIP.

Sunday, May 03, 2009

Legacy Import Tricks

This blog entry is for those unfortunate to be stuck with the old legacy exp/imp instead of the expdp/impdp (Datapump).

We are importing a ton of data and here are some import tricks that are documented in Oracle MetaLink doc id 93763.1. The ones I'm listing here only apply to imports from direct exports. We had already done the other things in the note.

1. Increase the BUFFER size of the import
2. Create some huge redo log files to minimize the LGWR switches.

It was amazing. This MetaLink note actually works :-)

Cluvfy Open File Descriptors Failed

If your Cluvfy OS check fails for the following:

Check: Hard resource limit for "open file descriptors"
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
coronet 4096 65536 failed
vail 4096 65536 failed
Result: Hard resource limit check failed for "open file descriptors".

Increase these kernel parameters in HP-UX ia64:

nfile=131072
maxfiles_lim=65536

Use kctune nfile and kctune maxfiles_lim to verify the correct values after the UNIX administrator corrects them.

More HP-UX ia64 VIP Fun

Here is a tip which will save you a headache if you are installing Clusterware on a new server.
Before you begin a new Clusterware install, make sure when you receive your servers from the UNIX admins that the VIPs are not already configured and active. However, you should have the host and IP for the Virtual IP in the domain server and local hosts file.

Do netstat -inw at a UNIX prompt to see the interfaces. You should not see the VIP interface.

If it is configured or active, you can unplumb it if it is NOT aggregated (e.g. APA):

ifconfig lan0:1 unplumb
or if it is APA:
ifconfig lan900:1 0.0.0.0

By setting the IP address to 0.0.0.0 it disables the secondary interface.
/etc/rc.config.d/netconf file has the interfaces which are started at boot up. They can be commented out to remove an interface following a reboot.

Friday, April 24, 2009

HP-UX 11.31 IA64 Memory / Filecache

Following this last install, we noticed that our HP-UX 11.31 IA64 servers were using a lot of system memory. Over half of the available memory was being eaten up by sys memory and file cache. We determined that the default settings for file cache were way to high. So, we made some adjustments.

The filecache_min and filecache_max kernel parameters are the ones to check. These should be tuned to leave more memory for Oracle. You can also mount the file system with direct IO which bypasses file buffer cache. If you are using file system for your Oracle datafiles then you want those mount points mounted with direct IO.

So, we solved it by setting constant values for filecache_min and max instead of taking the default %. By lowering them significantly, we were able to give more memory back to Oracle and the other applications.

We still have quite a bit of sys memory being utilized. However, the filecache issue is solved. I'll update this post if we figure out why so much sys memory is utilized. Or, it could be "just the way it is".