Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Multipath mini how-to
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
jimb0
n00b
n00b


Joined: 13 Dec 2003
Posts: 5

PostPosted: Wed Dec 29, 2004 10:04 pm    Post subject: Multipath mini how-to Reply with quote

Key words are HBA, Fibrechannel QLA failover multipath

This is a mini how-to...

Hardware used:

2 Qlogic 2340
2 McData 4500
1 EMC DMX 800
1 IBM 345 8 gig mem, 2 3.2g cpu

This setup allows failover to a secondary path (hba) in-case of a fault in the primary path (hba).

Need to setup the SAN. Setup Meta/LUN on EMC box. Setup SDR on EMC.Setup vcm database EMC. Setup masking on the McData switches to same lun on the EMC to your HBA's

File need to do the auto failover is at http://sourceforge.net/project/showfiles.php?group_id=54695 (you will need to modify one file later in doc) Or you can look here for the file http://scsirastools.sourceforge.net/ get the tar.gz file and follow the read me file in the package. Now this software is for Software RAID-1 (software) not multipath, so I had to mod a file call "mdevt"in the package.

In gentoo make sure you:
emerge mdadm

On Gentoo 2.6.9-r9 complie like this

Into the kernel SCSI disk support, SCSI genertic support, Probe all LUN's on each SCSI device, Fibre Channel Driver support (under networking support), Mulitple devices driver support (RAID and LVM), IBM ServeRAID support (for local disk for OS) As modules. QLogic ISP2300, SCSI tansport Attributes -> FiberChannel Transport Attribues (under scsi device support), RAID support (under multi-device support), Multipath I/O Suport (under multi-device support)

modprobe qla2300
modproble md

GServer linux # lsmod
Module Size Used by
vmnet 26160 16
vmmon 45280 7
multipath 8072 1
md 39500 2 multipath
qla2300 123136 0
qla2xxx 105376 3 qla2300
scsi_transport_fc 4096 1 qla2xxx
GServer linux #

fdisk /dev/sdc


GServer root # cat /etc/raidtab
##need this for scsitools

raiddev /dev/md0
raid-level multipath
nr-raid-disks 2

device /dev/sdc1
raid-disk 0

device /dev/sde1
raid-disk 1


GServer root # cat /etc/mdadm.conf
##need this for mdadm
DEVICE /dev/hd*[0-9] /dev/sd*[0-9]
ARRAY /dev/md0 level=multipath num-devices=2 devices=/dev/sdc1,/dev/sde1

MAILADDR mccann.jim@gmail.com

GServer root #

Next do below

mdadm --create /dev/md0 --level multipath --raid-devices=2 /dev/sdc1 /dev/sde1 (answer yes) data well be lost.

mknod /dev/md0 b 9 0 (if needed)

mkfs whatever you like on /dev/md0


GServer tmp # cat /etc/conf.d/local.start
# /etc/conf.d/local.start:
# $Header: /home/cvsroot/gentoo-src/rc-scripts/etc/conf.d/local.start,v
1.4 2002/11/18 19:39:22 azarah Exp $

# This is a good place to load any misc.
# programs on startup ( 1>&2 )

## I use --build works well to restart after reboot. Remember
/dev/sdc1 and /dev/sde1 are the same disk. So mdadm seem funny about
it. Namely if you stop /dev/md0 and try to restart it doing --assemble. --build just works.

mdadm --build --level=multipath --raid-devices=2 /dev/md0 /dev/sdc1 /dev/sde1
mount /dev/md0 /vmware
sgraidmon -b
/etc/init.d/vmware start
/etc/init.d/httpd.vmware start

GServer tmp #


GServer tmp # cat /etc/conf.d/local.stop
# /etc/conf.d/local.stop:
# $Header: /home/cvsroot/gentoo-src/rc-scripts/etc/conf.d/local.stop,v
1.4 2002/11/18 19:39:22 azarah Exp $

# This is a good place to unload any misc.
# programs you started above.
# For example, if you are using OSS and have
# "/usr/local/bin/soundon" above, put
# "/usr/local/bin/soundoff" here.

/etc/init.d/vmware stop
/etc/init.d/httpd.vmware stop
umount /dev/md0
mdadm --stop --scan

GServer tmp #



Modify the mdevt (from scsitools) Or you can just cut and paste this text.

GServer root # cat /sbin/mdevt
#!/bin/sh
# mdevt - Handle md raid events from sgraidmon or "mdadm --follow"
#
# $1 = event ("Fail", "Insert", "Save", or "ActiveSpare")
# $2 = md dev (/dev/md0)
# $3 = disk (/dev/sdb6, for Fail/Insert)
# $4 = iteration (for sgraidmon Insert only)
# sgraidmon is the only tool that detects insertion
# mdadm was formerly known as mdctl.
log=/var/log/mdevents
pdir=/usr/share/scsirastools
pfile=$pdir/sdf.save
tmpf=/tmp/waitmd.sh
alarmcmd=/usr/sbin/alarms

echo "-------------------" >>$log
echo "mdevt: $* `date`" |tee -a >>$log
# write the event also to the syslog
logger -i "mdevt: $* `date`"
case "$1" in
#####
##### Fail/Remove a disk partition, called from sgraidmon
#####
"Fail")
cat /proc/mdstat >>$log
# make sure the drive is marked offline/faulty
echo "mdadm $2 -f $3" >>$log
mdadm $2 -f $3 >>$log 2>&1
# get the disk name from the partition device name
if [ -d /dev/scsi/host0 ] || [ -d /dev/scsi/host1 ]
then
idisk=`echo $3 |sed -e 's/part[0-9]/disc/'`
else
idisk=`echo $3 |sed -e 's/[0-9]//'`
fi
$alarmcmd >$tmpf 2>/dev/null
# remove the driver partition from the raid
echo "raidhotremove $2 $3" >>$log
raidhotremove $2 $3 >>$log 2>&1
;;
#####
##### Insert a new disk partition, called from sgraidmon
#####
"Insert")
echo "mdadm $2 -f $3" >>$log
mdadm $2 -f $3 >>$log 2>&1
echo "raidhotremove $2 $3" >>$log
raidhotremove $2 $3 >>$log 2>&1
#sync; sync
echo "raidhotadd $2 $3" >>$log
raidhotadd $2 $3 >>$log 2>&1
cat /proc/mdstat >>$log

;;
#####
##### Save partition layout - called at init time from sgraid
#####
"Save")
if [ $1 = "Save" ]
then
mkdir -p $pdir
echo "sfdisk -d $2 >$pfile" >>$log
sfdisk -d $2 >$pfile
# Some (like SuSE) have menu.lst but not grub.conf
if [ ! -f /etc/lilo.conf ] && [ -f /boot/grub/menu.lst ]
then
if [ ! -f /boot/grub/grub.conf ]
then
# Make a grub.conf for later reference
ln /boot/grub/menu.lst /boot/grub/grub.conf
fi
fi
if [ -f /boot/grub/grub.conf ]
then
# Make sure the second disk also has a valid mbr
# Note that this is a workaround for grub not handling raid
# devices, More would have to be done for >2 disks also.
# Write mbr & stage2 from hd $j to $i (1st partition)
i=1
j=0
tmpa=/tmp/grub.in
cat - <<EOF >$tmpa

install (hd$j,0)/grub/stage1 (hd$i) (hd$i,0)/grub/stage2 (hd$j,0)/grub/grub.conf
quit
EOF
echo "grub: writing mbr to hd$i" >>$log
grub <$tmpa >>$log 2>&1
echo "grub ret = $?" >>$log
fi
fi
;;
#####
##### Check if saved partition layout file is present
#####
"Check")
if [ -f $pfile ]
then
echo "OK, saved partition layout is there ($pfile)." | tee -a $log
exit 0
else
echo "Saved partition layout is missing ($pfile)." | tee -a $log
echo "Run '/etc/rc.d/init.d/sgraid start' to invoke mdevt
Save" | tee -a $log
exit -1
fi
;;
#####
##### Display Usage if invalid parameters
#####
*)
echo "Invalid function: $0 $1 $2 $3 $4" | tee -a $log
echo "Usage: mdevt <event> [md dev] [disk] [iteration]" | tee -a $log
echo " where parameters are:"
echo " 1 = event (Fail, Insert, Check, or Save)"
echo " 2 = md dev (e.g. /dev/md0 - for Fail/Insert only)"
echo " 3 = disk (e.g. /dev/sdb6 - for Fail,Insert,Save)"
echo " 4 = iteration (e.g. 1,2,3 - for Insert only)"
;;
esac
GServer root #



LOG files sgraidmon.log

Num Name bus:ch:id:lun Type Vendor Device_Model FRev Serial# Status
0 /dev/sg0 /dev/sda 0:0:0:0 Disk LSILOGIC 1030 IM 1000 LSI
INTERNAL notrdy
1 /dev/sg1 0:0:8:0 Proc IBM 32P0032a S320 1 1 000 ready
2 /dev/sg2 /dev/sdb 2:0:0:0 Disk EMC SYMMETRIX 5670 88000340a ready
3 /dev/sg3 /dev/sdc 2:0:0:33 Disk EMC SYMMETRIX 5670
880DB340Q online
4 /dev/sg4 /dev/sdd 3:0:0:0 Disk EMC SYMMETRIX 5670 88000470a ready
5 /dev/sg5 /dev/sde 3:0:0:33 Disk EMC SYMMETRIX 5670
880DB470Q online
monitoring 5 scsi devices ...
sense_err: end ret = 0, retry = 1, sense=5:20:0
get_defects: sense err 05:20:00
scandev: device 0 error=-2, errno=11

--- sgraidmon v1.25 log started at Fri Dec 17 16:21:41 2004

sgraidmon utility v1.25 for SCSI RAID monitoring
sense_err: end ret = 0, retry = 1, sense=5:20:0
get_defects: sense err 05:20:00
scandev: device 0 error=-2, errno=2
scandev: device 2 error=-6, errno=2
scandev: device 3 error=-6, errno=2
Num Name bus:ch:id:lun Type Vendor Device_Model FRev Serial# Status
0 /dev/sg0 /dev/sda 0:0:0:0 Disk LSILOGIC 1030 IM 1000 LSI
INTERNAL notrdy
1 /dev/sg1 0:0:8:0 Proc IBM 32P0032a S320 1 1 000 ready
2 /dev/sg2 /dev/sdb 2:0:0:0 Disk failed
3 /dev/sg3 /dev/sdc 2:0:0:33 Disk failed
4 /dev/sg4 /dev/sdd 3:0:0:0 Disk EMC SYMMETRIX 5670 88000470a ready
5 /dev/sg5 /dev/sde 3:0:0:33 Disk EMC SYMMETRIX 5670
880DB470Q online
monitoring 3 scsi devices ...

--- sgraidmon v1.25 log started at Fri Dec 17 16:22:16 2004


LoG file for mdevents

mdevt: Fail /dev/md0 /dev/sdc1 Fri Dec 17 16:32:32 EST 2004
Personalities : [multipath]
md0 : active multipath sde1[1] sdc1[2](F)
53538752 blocks [2/1] [_U]

unused devices: <none>
mdadm /dev/md0 -f /dev/sdc1
mdadm: set /dev/sdc1 faulty in /dev/md0
raidhotremove /dev/md0 /dev/sdc1
-------------------
mdevt: Insert /dev/md0 /dev/sdc1 1 Fri Dec 17 16:32:51 EST 2004
mdadm /dev/md0 -f /dev/sdc1
mdadm: set device faulty failed for /dev/sdc1: No such device
raidhotremove /dev/md0 /dev/sdc1
/dev/md0: can not hot-remove disk: disk not in array!
raidhotadd /dev/md0 /dev/sdc1
Personalities : [multipath]
md0 : active multipath sdc1[2] sde1[1]
53538752 blocks [2/2] [UU]

unused devices: <none>
-------------------
mdevt: Check Fri Dec 17 16:34:44 EST 2004
OK, saved partition layout is there (/usr/share/scsirastools/sdf.save).
-------------------



Why did I post this how-to? 3 Reasons. First I have used the Gentoo fourms to help me trouble shoot issues and this issue was not answered. And I know it is the best distro (hell, gentoo is gentration 2 right? :) ). I have used it for many servers. Second I used VWARE GSX with Gentoo. I think it works the best on Gentoo and it is fast and light. Third EMC does not make powerpath (EMC software $$$$) for genral linux distros. Yes they make it for RedHat and Suse (lazy emc not like you could use the software with out the hardware). But in a bin format. So I set out to do it with open source software.


Understand this. The system uses one path at a time. So it is NOT load balancing (EMC powerpath is but $$$$ and is not made for Gentoo). If the primary path fails the system will "hang" for about 5-10 Seconds. Then switch to the secondary path and countine on that path until the primary is back. But the biggest issue you will run into is if you mess with HBA and or cable/switch with Gentoo is to bring the DEV back online automagicly when the cable/switch get replaced. That is where scsitools comes into play.



Hope this helps someone.

System Admin, Email Admin, SAN Admin, Network Admin, OPEN System Project Leader
Jim McCann
Back to top
View user's profile Send private message
duderonomy
Guru
Guru


Joined: 20 Mar 2004
Posts: 349
Location: SF Bay Area

PostPosted: Wed Feb 02, 2005 6:12 am    Post subject: Re: Multipath mini how-to Reply with quote

jimb0 wrote:
Key words are HBA, Fibrechannel QLA failover multipath

This is a mini how-to...

Hardware used:

2 Qlogic 2340
2 McData 4500
1 EMC DMX 800
1 IBM 345 8 gig mem, 2 3.2g cpu

< snip>

Hope this helps someone.

System Admin, Email Admin, SAN Admin, Network Admin, OPEN System Project Leader
Jim McCann


Thank you for the contribution. I had wanted to configure a similar FC data host with gentoo
for use on a SAN. Thank you for writing all this. :)

-D
Back to top
View user's profile Send private message
fatalglitch
n00b
n00b


Joined: 25 Nov 2003
Posts: 54

PostPosted: Tue Aug 16, 2005 1:39 pm    Post subject: Reply with quote

If you are using Qlogic cards, there are drivers out there that have failover, and multipath support.

We're using HP Proliant DL385's with Dual Qlogic HBA's to a 12TB SAN, they are diskless nodes, and this method I've found to be the best.

Tom Callahan
Back to top
View user's profile Send private message
duderonomy
Guru
Guru


Joined: 20 Mar 2004
Posts: 349
Location: SF Bay Area

PostPosted: Wed Aug 17, 2005 5:34 am    Post subject: Reply with quote

fatalglitch wrote:
If you are using Qlogic cards, there are drivers out there that have failover, and multipath support.

Hello,

I continue to be interested in this topic.
When you say there are drivers that support
failover and multipath, are you referring to drivers
at the level of the hba or are you referring to upper
level MPP driver? Are there others besides that one?

Which QLogic model are you using?

Thx...
D
Back to top
View user's profile Send private message
drakkan
Apprentice
Apprentice


Joined: 21 Jun 2004
Posts: 232

PostPosted: Fri Aug 19, 2005 1:37 pm    Post subject: Reply with quote

what about linux multipath tools?

http://christophe.varoqui.free.fr/wiki/wakka.php?wiki=Home

it seems the future for multipath on linux, in gentoo,actually, there is a masked ebuild
Back to top
View user's profile Send private message
drakkan
Apprentice
Apprentice


Joined: 21 Jun 2004
Posts: 232

PostPosted: Sat Sep 17, 2005 11:47 am    Post subject: Reply with quote

The following link explain how I setup dm-multipath with EMC CX 300

https://www.redhat.com/archives/dm-devel/2005-September/msg00034.html

drakkan
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum