Linux/Multipath
Multipath
Turn off multipath:
chkconfig multipathd off
This is what is showing up in Knoppix as:
/dev/dm-1
Multipath I/O - Wikipedia, the free encyclopedia
Linux Multipathing, Linux Symposium 2005 p. 147
SCSI Rescan
See Linux/SCSI
dmsetup
See Linux/dmsetup
Show wwid
Show device WWID:
$ scsi_id -g -u -s /block/sda
Dump all WWID:
$ for dev in $( ls /sys/block/ | grep sd ) ; do scsi_id -g -u -s /block/$dev ; done | sort | uniq 360060e80100520e0052faebe00000001 360060e80100520e0052faebe00000002
Blacklisting
4.2. Configuration File Blacklist
By default, all devices are blacklisted, since the following lines appear in the initial configuration file.
blacklist {
devnode "*"
}
Blacklisting by WWID: (World-Wide IDentification)
blacklist {
wwid 26353900f02796769
}
"You can use the blacklist_exceptions section of the configuration file to enable multipathing on devices that have been blacklisted by default:"
blacklist_exceptions {
device {
vendor "IBM"
product "S/390.*"
}
}
"When specifying devices in the blacklist_exceptions section of the configuration file, you must specify the exceptions in the same way they were specified in the blacklist. For example, a WWID exception will not apply to devices specified by a devnode blacklist entry, even if the blacklisted device is associated with that WWID. Similarly, devnode exceptions apply only to devnode entries, and device exceptions apply only to device entries. "
Citrix Forums : Xen and multipath volumes on a ...
blacklist {
wwid "*"
devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
devnode "^hd[a-z]"
}
blacklist_exceptions {
wwid "36001ec9000dc4e2a000004f14871833e"
wwid "36001ec9000dc4e2a00000499486ddf24"
}
blacklist {
device {
vendor "*"
product "*"
}
devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
devnode "^hd[a-z]"
}
blacklist_exceptions {
device {
vendor "DGC"
product "*"
}
}
Red Hat Knowledgebase: How do I setup device-mapper multipathing in Red Hat Enterprise Linux 4?
In order to setup device-mapper-multipathing an updated system with the package device-mapper-multipath is recommended.
This functionality does not exist in releases of Red Hat Enterprise Linux 4 prior to Update 2.
To set up device-mapper multipathing, edit the /etc/multipath.conf file. Comment out the following lines at the top of the file:
devnode_blacklist {
devnode "*"
}
For example:
# devnode_blacklist {
# devnode "*"
# }
This section keeps device-mapper multipathing from scanning all of the devices. Uncomment the following section of the /etc/multipath.conf file:
defaults {
multipath_tool "/sbin/multipath -v0"
udev_dir /dev
polling_interval 10
default_selector "round-robin 0"
default_path_grouping_policy multibus
default_getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
default_prio_callout "/bin/true"
default_features "0"
rr_min_io 100
failback immediate
}
blacklist {
wwid 26353900f02796769
devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
devnode "^hd[a-z][[0-9]*]"
devnode "^cciss!c[0-9]d[0-9]*[p[0-9]*]"
}
This sets the default device-mapper behavior and filters out all the devices that are not normally multipathed, such as IDE hard drives and floppy drives. In order to use simple failover behaviour, the default_path_grouping_policy option in the defaults section should be changed to failover as illustrated by the sample below:
defaults {
multipath_tool "/sbin/multipath -v0"
udev_dir /dev
polling_interval 10
default_selector "round-robin 0"
default_path_grouping_policy failover
default_getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
default_prio_callout "/bin/true"
default_features "0"
rr_min_io 100
failback immediate
}
Save the configuration file and exit the editor. Execute the following commands:
modprobe dm-multipath modprobe dm-round-robin service multipathd start multipath -v2
The multipath -v2 command prints out multipathed paths that show which devices are multipathed. If the command does not print anything out, ensure that all SAN connections are set up properly and the system is multipathed. Execute this command to make sure that the multipath daemon starts on bootup:
chkconfig multipathd on
The devices will be created as /dev/mapper/mpath#, where # is the multipathed group. If /dev/sda is the only multipathed device, /dev/mapper/mpath0 will be the multipathed device for /dev/sda and /dev/sdb. Note that fdisk cannot be used with /dev/dm-# devices. Use fdisk on the underlying disks and execute the following command when device-mapper multipath maps the device to create a /dev/mapper/mpath# device for the partition:
kpartx -a /dev/mapper/mpath#
Note: dmsetup ls --target=multipath is a command that helps determine multipath devices assignments on a system.
If the hardware is not mentioned in the multipathing database, please view the article "How can I add more products into the mutipathing database?" in the Recommended FAQs section.
For an updated sample configuration file, see /usr/share/doc/device-mapper-multipath-<version>/multipath.conf.annotated
Devices
Physical: (fdisk -l)
/dev/sda /dev/sdb
Partitions: (fdisk -l)
/dev/sda1 /boot /dev/sda2 swap /dev/sda3 / /dev/sdb1 /boot /dev/sdb2 swap /dev/sdb3 /
Mapper: (mount, dmsetup -l)
/dev/mapper/mpath0 /dev/mapper/mpath0p1 /boot /dev/mapper/mpath0p2 swap /dev/mapper/mpath0p3 /
dm devices: (fdisk -l)
/dev/dm-0p1 /boot /dev/dm-0p2 swap /dev/dm-0p3 / #but these do not exist under /dev/, but these do: /dev/dm-0 /dev/dm-1 /dev/dm-2 /dev/dm-3
The wwid matches the Virtual Volume WWN with a prefixed '3'
cat /etc/multipath.conf
defaults {
user_friendly_names yes
}
blacklist {
devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
devnode "^(hd|xvd)[a-z]*"
wwid "*"
}
# Make sure our multipath devices are enabled.
blacklist_exceptions {
wwid "350002ac0350003f3"
}
multipath -ll:
mpath0 (350002ac0350003f3) dm-0 3PARdata,VV [size=12G][features=0][hwhandler=0] \_ round-robin 0 [prio=2][active] \_ 1:0:0:1 sda 8:0 [active][ready] \_ 2:0:0:1 sdc 8:32 [active][ready]
/proc/scsi/scsi:
Attached devices: Host: scsi1 Channel: 00 Id: 00 Lun: 01 Vendor: 3PARdata Model: VV Rev: 0000 Type: Direct-Access ANSI SCSI revision: 05 Host: scsi1 Channel: 00 Id: 00 Lun: 02 Vendor: 3PARdata Model: VV Rev: 0000 Type: Direct-Access ANSI SCSI revision: 05 Host: scsi2 Channel: 00 Id: 00 Lun: 01 Vendor: 3PARdata Model: VV Rev: 0000 Type: Direct-Access ANSI SCSI revision: 05 Host: scsi2 Channel: 00 Id: 00 Lun: 02 Vendor: 3PARdata Model: VV Rev: 0000 Type: Direct-Access ANSI SCSI revision: 05
cat /var/lib/multipath/bindings:
# Multipath bindings, Version : 1.0 # NOTE: this file is automatically maintained by the multipath program. # You should not need to edit this file in normal circumstances. # # This file was automatically generated by anaconda. # # Format: # alias wwid # mpath0 350002ac0350003f3 mpath1 350002ac0350103f3
- scsi_id -g -s /block/sdc
350002ac0350003f3
More Information
Gentoo Linux Documentation -- Multipathing for Gentoo
- "Multipathing services, generally deployed in enterprise environments, provide a means for high performance, load-balanced, and fault-tolerant data storage either locally or via a storage area network (SAN). Multipathing facilitates a single storage device to be transparently accessed across one or more paths. For example, if there are two connections from a server Host Bus Adapter (HBA) to two Fibre Channel switches and then to a SAN, when the HBA module loads and scans the bus, it will read four paths to the SAN: the paths from the server HBA to and from each Fibre Channel switch and at the storage device. Taking advantage of this situation, Multipath allows you to make use of each path simultaneously or independently to ensure a constant and reliable connection to the data in storage. Multipath serves as a failover for all connections points in the event of losing one path making critical data always available due to redundancy in the design and implementation. "
Linux SAN Multipathing « Thomas Vogt’s IT Blog
- " There are a lot of SAN multipathing solutions on Linux at the moment. Two of them are discussesed in this blog. The first one is device mapper multipathing that is a failover and load balancing solution with a lot of configuration options. The second one (mdadm multipathing) is just a failover solution with manuel re-anable of a failed path. The advantage of mdadm multiphating is that it is very easy to configure.
- Before using a multipathing solution for a production environment on Linux it is also important to determine if the used solution is supportet with the used Hardware. For example HP doesn’t support the Device Mapper Multipathing solution on their servers yet."
7.3. Configuring persistent storage in a Red Hat Enterprise Linux 5 environment
- "In an environment where external storage (for example, Fibre Channel or iSCSI) is used it is advised to configure persistent device names on your hosts. This will also aid in using Red Hat Virtualization's (live) migration feature to implement consistent device names across multiple systems."
# scsi_id -g -s /block/sdc 3600a0b800013275100000015427b625e
More Links
RHEL4 dm-multipath on root HOWTO - From the Recliner...
kpartx: Linux Create Device Maps From Partition Tables ( mpath devices for partitions ):
Q. How can I create mpath devices for partitions under CentOS Linux 5?
A. You need to use tool called kpartx. This is derived from util-linux partx package, which reads partition tables on specified device and create device maps over partitions segments detected. It is called from hotplug upon device maps creation and deletion. This is useful for setting multipath which is used to detect multiple paths to devices for fail-over or performance reasons and coalesces them.
To Setup your multipath type the following command at a shell prompt (login as the root user):
# for d in $(ls /dev/mapper/mpath*); do kpartx -a -v $d ; done
Multipath-usage.txt File for Red Hat Enterprise Linux 4 Update 3
Gentoo Linux Documentation -- Multipathing for Gentoo
Linux/VM: Enabling root-on-multipath for SLES9 on zSeries
Linux SAN Multipathing « Thomas Vogt’s IT Blog: (GOOD SOURCE)
Procedure for configuring the system with DM-Multipath:
- Install device-mapper-multipath rpm
- Edit the multipath.conf configuration file:
- comment out the default blacklist
- change any of the existing defaults as needed
- Start the multipath daemons
- Create the multipath device with the multipath
Right To Your Own Devices LG #114
- "Linux 2.4.x had the Logical Volume Manager (LVM) and other multi-disk/multi-partition block device constructs. These have been enhanced by the Device Mapper in Linux 2.6.x."
dmsetup(8): low level logical volume management - Linux man page
DMLoop - Device-mapper and LVM2 Wiki
How to detect a new disk with red hat advace server 3?? - LinuxQuestions.org
echo "scsi scan-new-devices" > /proc/scsi/scsi
- "You have a SCSI controller and the kernel does not see a newly added SCSI drive"
Symptom:
You have a SCSI controller and the kernel does not see a newly added SCSI drive.
Solution:
CAUTION: This is not for hotplugging your peripheral hardware. SCSI was not designed for this and you could damage your hardware! That said, it may be possible to switch on an already connected device or add a new drive recently hot-added into hot-plug capable enclosure. However, we make no guarantees, and using these commands could corrupt your data and/or thoroughly confuse your SCSI subsystem. Follow the instructions below at your own risk. Rebooting the system is still the only supported method for adding a new SCSI device to a running system.
Assuming that you have the necessary permissions for writing to /proc/scsi/scsi. The following are some individual commands available for viewing/manipulating scsi controllers and devices:
* echo "scsi add-single-device <H> <C> <I> <L>" > /proc/scsi/scsi where <H> <C> <I> <L> represents Host <H>, Channel <C>, Id <I>, and Lun <L>. Will add the device specified so that it is accessible to the system. * echo "scsi remove-single-device <H> <C> <I> <L>" > /proc/scsi/scsi where <H> <C> <I> <L> represents Host <H>, Channel <C>, Id <I>, and Lun <L>. Will remove the device specified so that it is no longer accessible to the system. * echo "scsi scan-new-devices" > /proc/scsi/scsi will scan all host adapters again to see if there are any new devices. * echo "scsi dump <#>" > /proc/scsi/scsi will dump the status of all current SCSI commands. <#> is the number specifing the level of detail for the dump, 0-9 are valid. * echo "scsi log <T> <#>" > /proc/scsi/scsi will change the detail level of SCSI logging represented by <T> of error, scan, mlqueue, mlcomplete, llqueue, llcomplete, hlqueue, or hlcomplete, to the level specified by the <#> number.
1. up2date device-mapper-multipath 2. Edit /etc/multipath.conf For detailed information, see: "SAN Persistent Binding and Multipathing in the 2.6 Kernel" 3. modprobe dm-multipath 4. modprobe dm-round-robin 5. service multipathd start 6. multipath -v2 Will show multipath luns and groups. Look for the multipath group number, this is the dm-# listed in /proc/partitions. The multipath lun is accessed via the /dev/dm-# device entry. 7. Format each SCSI DEVICE: * sfdisk /dev/sdX 8. (Optional) Create multipath devices for each partition: (not needed if using LVM, since you will just mount the logical volume device) * kpartx -a /dev/dm-# 9. Enable multipath to start on bootup: * chkconfig multipathd on
multipath -F
- Clear all multipath bindings. Useful when making changes to /etc/multipath.conf (clear multipath map, then run multipath to read the config file and build a new map).
multipath -v3 -ll
- List lots of information about all known disks, what multipath groups they belong to, settings, etc...
- (NOTE: the -ll also seems to force multipathd to pick up (rescan for) new devices that might have been added to the system but not recognized by mutlipathd yet.)
dmsetup ls --target=multipath
- Determine multipath device assignments on a system.
Multipath-usage.txt File for Red Hat Enterprise Linux 4 Update 3
Tools
lsmpath
#!/bin/sh
#
# Programmed by Kenneth Burgener <kenneth.burgener@applabs.com>
#
DV=$1
if [ "$DV" = "" ] ; then
DVS=$( ls /dev/mapper/mpath? | cut -d 'h' -f 2 )
for dv in $DVS ; do $0 $dv ; done
exit 0
fi
if [ ! -e /dev/mapper/mpath$DV ] ; then
echo "mpath$DV does not exist!"
exit 1
fi
#DM=$( dmsetup ls | grep "mpath$DV.(" | cut -d ' ' -f 2 | cut -d ')' -f 1 )
#DEV=$( ls /sys/block/dm-$DM/slaves/ | head -n 1 )
DEV=$( multipath -ll mpath$DV | tr -s " " | grep "sd[a-z] " | head -n 1 | cut -d " " -f 4 )
#VENDOR=$( cat /sys/block/$DEV/device/vendor )
#MODEL=$( cat /sys/block/$DEV/device/model )
#SMT=$( echo "$VENDOR - $MODEL" )
SMT=$( smartctl -a /dev/$DEV | grep "Device:" )
SIZE=$( fdisk -l /dev/mapper/mpath$DV 2>/dev/null | grep "Disk /dev/mapper/mpath$DV:" | cut -d " " -f 3-4 | cut -d "," -f 1 )
echo "mpath$DV ($SIZE) = $SMT"
lsdisks
#!/bin/sh # # Programmed by Kenneth Burgener <kenneth.burgener@applabs.com> # DVS=$( ls /sys/block/ | grep "sd[a-z]*$" ) for DEV in $DVS ; do VENDOR=$( cat /sys/block/$DEV/device/vendor ) MODEL=$( cat /sys/block/$DEV/device/model ) DEVINFO=$( echo "$VENDOR - $MODEL" ) SIZE=$( fdisk -l /dev/$DEV 2>/dev/null | grep "Disk /dev/$DEV:" | cut -d " " -f 3-4 | cut -d "," -f 1 ) echo "/dev/$DEV = ($SIZE) = $DEVINFO" done
SLES 11 Multipath
- "Ideally, you should configure multipathing for devices before you use them as components of a software RAID device. If you add multipathing after creating any software RAID devices, the DM-MP service might be starting after the multipath service on reboot, which makes multipathing appear not to be available for RAIDs. You can use the procedure in this section to get multipathing running for a previously existing software RAID."
keywords
linux multipath multi path multipathd kpartx dmsetup
[[Category::Linux]]