Wednesday, May 30, 2007

Oracle inventory problems

Had an interesting problem happen today while applying cpu2007apr to our grid control infrastructure.

After applying the cpu to the em app server, I started work on the patches, and ran into a brick wall while patching a db server.

A simple opatch lsinventory would fail, showing one of the homes that wasn't the agent home, then spitting out this set of errors:

Inventory load failed... OPatch cannot load inventory for the given Oracle Home.
Possible causes are:
Oracle Home dir. path does not exist in Central Inventory
Oracle Home is a symbolic link
Oracle Home inventory is corrupted
LsInventorySession failed: OracleHomeInventory::load() gets null oracleHomeInfo

OPatch failed with error code 73

Oh no! The inventory is corrupted! Actually, it's not at all corrupted, I just did something stupid during the initial install of this box and haven't corrected it yet.

I've got two oraInventory's on this box. Which means I'm going to need to be very specific when identifying which oraInst.loc I'm using.

In this situation, I checked in the AGENT_HOME/oraInst.loc to locate the missing inventory, and then used that in the invPtrLoc command below:

[oracle@omsdb:/usr/users/oracle]# opatch lsinventory -detail \
-invPtrLoc /u01/app/oracle/product/agent10g/oraInst.loc

What the heck is ocssd.bin?

On a db system we were patching, we saw a process we didn't expect to be running.

[oracle@omsdb:/u01/app/oracle/stage/patch/cpu2007apr/5901877]# ps -ef | grep ora
oracle 486 1 0 Mar 21 ? 0:03 /u01/app/oracle/product/10.1.0/bin/ocssd.bin
jmorast 12755 12753 0 10:41:35 ? 0:00 /usr/local/sbin/sshd -R

A slight bit of research reveals the following from Frank Naude.

It appears this process is part of Oracle Cluster Synch Service Daemon, even though RAC and ASM aren't in use.

Solve the immediate issue by running this command from root:

/etc/init.d/init.cssd stop

A permanent fix is to comment out the following line in /etc/inittab and reboot the

h1:35:respawn:/etc/init.d/init.cssd run >/dev/null 2>&1

Sunday, May 27, 2007

Not your grandma's sqlplus

Andy Campbell has a great post about adding color to SQL*Plus. Now, I'm not one to go out and make changes like this to production without using them for 6 months or so... but...

Does SQL*Plus, much like vi (or 6, as the cool kids like to call it) become our bread and butter because of or rather in spite of it's stability (or perhaps one should look at it as lack of improvements) over time?

I made a point of becoming an expert with vi early in my career, and I've never regretted that choice. Much like the other basic components of living on a unix platform, it does it's job quickly and efficiently, with bells and whistles sacrificed in the name of speed and stability over time.

SQL*Plus... I suppose the biggest issue here is that new versions of Oracle rarely cause issue for scripts (can I get a shout out for svrmgrl?). Since that stability is a requirement (svrmgrl scripts were the ones that were broke, not ones involving sqlplus anyhow), change is going to arrive at a glacial pace.

There are some people attempting to implement newer feature sets. There are a few different replacements for sqlplus, I don't consider any of these ready for my production use.

So -- I'll go ahead and say it. Without a relatively large amount of customization, sqlplus is a pain in the ass. I guess vi is not a fair comparison for what I'm thinking. Sqlplus is a shell, vi isn't. Perhaps a better comparison is the bourne shell since both are evolving at about the same rate, both are about as irritating.

I suppose the point that I'm every so slowly arriving at, is that early in my career I shifted from bourne (and csh) to bash (I still hate you, HP/UX 10) and never looked back. Sure there's a script here or there that I've used bourne or ksh or csh with, but the majority of my daily work is in bash. Perhaps it's time to dig through the replacements I mentioned earlier and see if they're ready to be prime time players.

Or... do we keep tweaking sqlplus?

Friday, May 18, 2007

Lots of invalid objects

Here's a nice eyebrow raiser you'll occasionally see during installing CPU's or application upgrades. It probably shouldn't ever happen, right?

SQL> select count(*) from dba_objects where status <> 'INVALID';


So what's the problem?

Oracle: Session Blocking

From time to time I get a call from the user community at large of people not being able to do anything cause the app is hung up.

It's always been due to a session blocking everyone else. Too bad the application doesn't handle this better on it's own.

Here's a query I use from time to time to see who is blocking.

Select /*+ RULE */ username, v$session.SID, serial#, owner, object_name, object_type, 
FROM dba_objects, v$lock, v$session
WHERE object_id = v$lock.id1 AND v$lock.SID = v$session.SID
AND owner != 'SYS'
AND block = 1

Wednesday, May 16, 2007

Unix: tee

As defined from the man page for tee:
tee - replicate the standard output

tee [-ai] [file...]

The tee utility will copy standard input to standard output,
making a copy in zero or more files. tee will not buffer its
output. The options determine if the specified files are
overwritten or appended to.

The following options are supported.

-a Append the output to the files rather than overwriting

-i Ignore interrupts.

It's useful for when you want to capture a log of whatever command you are running, but still want to see the output. This was handy for me today when running some scripts from the vendor, which had the possibility to throw errors during the process which I would need to kill. I wanted to make sure to log everything, but was able to still see the problems as they occurred.

Example of usage.
> ./ | tee install.log


> ls

This will put the output of the script to the standard output (screen) as well as write that to the install.log file.

First Post!