Hi!
I have tried on Oracle forums but AFAIK all say this is pure VMware problem so now I'm comming here in a hope to get rid of big problem.
RAC 10.2, on RHEL 5.3 EE, in VmWare server 2.01 on Windows 2003 server EE. I'm trying to avoid
deprecated RHEL 4.X raw devices service and to test ASMLib for DB instance. I have two nedes.
Problem is that Oracle use clufvy to test sharednes of disks and this (in my case returns PARTIALY bad results). Here is brief explanation of the situation:
oracle@ishacrac2x ~$ ls -l /dev/sd*
brw-r----- 1 root disk 8, 0 Jun 15 2009 /dev/sda
brw-r----- 1 root disk 8, 1 Jun 14 22:16 /dev/sda1
brw-r----- 1 root disk 8, 2 Jun 15 2009 /dev/sda2
brw-rw---- 1 oracle dba 8, 16 Jun 15 2009 /dev/sdb
brw------- 1 oracle dba 8, 17 Jun 14 22:16 /dev/sdb1
brw-rw---- 1 oracle dba 8, 32 Jun 15 2009 /dev/sdc
brw------- 1 oracle dba 8, 33 Jun 14 22:16 /dev/sdc1
brw-rw---- 1 oracle dba 8, 48 Jun 15 2009 /dev/sdd
brw------- 1 oracle dba 8, 49 Jun 14 22:16 /dev/sdd1
brw-rw---- 1 oracle dba 8, 64 Jun 15 2009 /dev/sde
brw------- 1 oracle dba 8, 65 Jun 14 22:16 /dev/sde1
root@ishacrac2x ~# fdisk -l
Disk /dev/sda: 6442 MB, 6442450944 bytes
255 heads, 63 sectors/track, 783 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sda1 * 1 652 5237158+ 83 Linux
/dev/sda2 653 783 1052257+ 82 Linux swap / Solaris
Disk /dev/sdb: 4294 MB, 4294967296 bytes
255 heads, 63 sectors/track, 522 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdb1 1 522 4192933+ 83 Linux
Disk /dev/sdc: 4294 MB, 4294967296 bytes
255 heads, 63 sectors/track, 522 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdc1 1 522 4192933+ 83 Linux
Disk /dev/sdd: 42.9 GB, 42949672960 bytes
255 heads, 63 sectors/track, 5221 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdd1 1 5221 41937651 83 Linux
Disk /dev/sde: 42.9 GB, 42949672960 bytes
255 heads, 63 sectors/track, 5221 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sde1 1 5221 41937651 83 Linux
Lets make some check from Oracle point of view (one node at a time):
oracle@ishacrac2x ~$ export MYNODES=ishacrac1x
oracle@ishacrac2x ~$ export MYSHARED=/dev/sdb1,/dev/sdc1,/dev/sdd1,/dev/sde1
oracle@ishacrac2x ~$ /u01/install/clusterware/cluvfy/runcluvfy.sh comp ssa -n $MYNODES -s $MYSHARED
Verifying shared storage accessibility
Checking shared storage accessibility...
"/dev/sdb1" is shared.
"/dev/sdc1" is shared.
"/dev/sdd1" is shared.
"/dev/sde1" is shared.
Shared storage check was successful on nodes "ishacrac1x".
Checkin on node 2 node 2:
oracle@ishacrac2x ~$ export MYNODES=ishacrac2x
oracle@ishacrac2x ~$ /u01/install/clusterware/cluvfy/runcluvfy.sh comp ssa -n $MYNODES -s $MYSHARED
Verifying shared storage accessibility
Checking shared storage accessibility...
"/dev/sdb1" is shared.
"/dev/sdc1" is shared.
"/dev/sdd1" is shared.
"/dev/sde1" is shared.
Shared storage check was successful on nodes "ishacrac2x".
Verification of shared storage accessibility was successful.
All looks good, bu when you ty to check them together then verification failed:
oracle@ishacrac2x ~$ export MYNODES=ishacrac2x,ishacrac1x
oracle@ishacrac2x ~$ /u01/install/clusterware/cluvfy/runcluvfy.sh comp ssa -n $MYNODES -s $MYSHARED
Verifying shared storage accessibility
Checking shared storage accessibility...
ERROR: /dev/sdb1
Unable to determine the sharedness of /dev/sdb on nodes:
ishacrac2x,ishacrac1x
ERROR: /dev/sdc1
Unable to determine the sharedness of /dev/sdc on nodes:
ishacrac2x,ishacrac1x
ERROR: /dev/sdd1
Unable to determine the sharedness of /dev/sdd on nodes:
ishacrac2x,ishacrac1x
ERROR: /dev/sde1
Unable to determine the sharedness of /dev/sde on nodes:
ishacrac2x,ishacrac1x
Shared storage check failed on nodes "ishacrac2x,ishacrac1x".
Verification of shared storage accessibility was unsuccessful on all the nodes.
This all leads me to installer rerror shown on picture http://i44.tinypic.com/22zs7n.jpg.
I try to make a workaround and to install one node (in clusterware) and then another using addNode.sh AND THIS WORKED!
VOW!!!Clesterware is up and ruining without problems! sdb1 is for CRS and sdc1 is for voting disk-all fine! So far so good but not for a long time.
But then when I come to install ASM instance, I got the same problem Oracle doesn't see the shared disks (where spfile should be placed and seen by all ASM instances). The best fact is that Oracle doesn't see shared disks REGARDLESS fact that clusterware is up and running on sdb1 and sdc1 as shared disk for that very same cluster!!!
Obviously the way Oracle test this is "courios" and need some other part for success, regardless it doesn't measure the real situation.
Now I'm really tired of making one simple installation in several weeks terms.. so kindly ask what parameter in VMware can I established to anounce Oracle that I really have shared disks in my system? Any help comment ...anything...
THX,
Funky