Linux/mdadm
mdadm
"mdadm is a Linux utility by Neil Brown that is used to manage RAID devices, previously known as mdctl. Besides managing, it can create, delete, or monitor Linux software RAIDs. Available under version 2 or later of the GNU General Public License, mdadm is free software. Mdadm derives its name from the “md” (multiple disk) device nodes it manages." [1]
Commands
Show all md devices status:
cat /proc/mdstat
Show md device details:
mdadm --detail /dev/md0
Create RAID 0 (stripe):
mdadm --create --verbose /dev/md0 --level=raid0 --raid-devices=3 /dev/hd[cde]1
Create RAID 5 (stripe with parity):
mdadm --create --verbose /dev/md0 --level=5 --raid-devices=3 /dev/sd[cde]1 mdadm --create --verbose /dev/md0 --level=5 --raid-devices=3 --spare-devices=1 /dev/sd[cdef]1
Add partition to md device:
mdadm /dev/md0 --add /dev/sdb1
Fail and remove partition of md device:
mdadm /dev/md0 --fail /dev/sdb1 mdadm /dev/md0 --remove /dev/sdb1
Start drive without resync:
mdadm -S /dev/md0 mdadm -A --assume-clean /dev/md0
Create /etc/mdadm.conf (high to low detail, any will work) [2]
mdadm --detail --scan --verbose > /etc/mdadm.conf mdadm --detail --scan > /etc/mdadm.conf mdadm --examine --scan > /etc/mdadm.conf
# ubuntu uses /etc/mdadm/mdadm.conf sudo mdadm --examine --scan --config=/etc/mdadm/mdadm.conf sudo mdadm --examine --scan --config=/etc/mdadm/mdadm.conf >> /etc/mdadm/mdadm.conf sudo mdadm -A /dev/md0
Stop array:
mdadm --stop /dev/md0
Start (assemble) array:
mdadm --assemble /dev/md0
Add disk to RAID 5:
unmount /dev/md1 mdadm --add /dev/md1 /dev/sdb3 mdadm --grow --raid-devices=4 /dev/md1 # wait for rebuild... watch -dc cat /proc/mdstat e2fsck -f /dev/md3 resize2fs /dev/md3
fdisk to 'Linux raid':
type 'fd'
Device Boot Start End Blocks Id System /dev/sdb1 2048 1953525167 976761560 fd Linux raid autodetect
Common mdadm commands
Generate mdadm.conf
cp /etc/mdadm/mdadm.conf /etc/mdadm/mdadm.conf.save /usr/share/mdadm/mkconf --generate > /etc/mdadm/mdadm.conf
Create RAID
mdadm --create /dev/md2 --raid-devices=3 --spare-devices=0 --level=5 --run /dev/sd[cde]1
Remove disk from RAID
mdadm --fail /dev/md0 /dev/sda1 mdadm --remove /dev/md0 /dev/sda1
Copy the partition structure (when replacing a failed drive)
sfdisk -d /dev/sda | sfdisk /dev/sdb mdadm --zero-superblock /dev/sdb
Add a disk to a RAID array (to replace a removed failed drive)
mdadm --add /dev/md0 /dev/sdf1
Check RAID status
cat /proc/mdstat mdadm --detail /dev/md0
Reassemble a group of RAID disks
#This works to move an assembly from one physical machine to another. mdadm --assemble /dev/md0 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1
Steps to emulate mdrun (which has been depreciated)
# haven't tested this. Use with care mdadm --examine --scan --config=partitions > /tmp/mdadm.conf mdadm --assemble --scan --config=/tmp/mdadm.conf
Add a disk to an existing RAID and resize the filesystem
mdadm --add /dev/md0 /dev/sdg1 mdadm --grow /dev/md0 -n 5 e2fsck -f /dev/md0 resize2fs /dev/md0 e2fsck -f /dev/md0
Replace all disks in an array with larger drives and resize
# For each drive in the existing array mdadm --fail /dev/md0 /dev/sda1 mdadm --remove /dev/md0 /dev/sda1 # physically replace the drive mdadm --add /dev/md0 /dev/sda1 # now, wait until md0 is rebuilt. # this can literally take days
# All drives have been replaced and sync'd, but they still use the original size. # Issue the following command to use all available space: mdadm --grow /dev/md0 --size=max
# Do not forget to resized the file system which sits on the raid set: # for ext2/3/4 e2fsck -f /dev/md0 && resize2fs /dev/md0 && e2fsck -f /dev/md0 # for lvm pv pvresize /dev/md0 # for ntfs ntfsresize /dev/md0 # note, most likely ntfs is NOT exported as a single partition. In the case # of a Xen hvm machine, it is a "disk device" so you will need to resize the # partition itself, then resize ntfs.
Stop and remove the RAID device
mdadm --stop /dev/md0 mdadm --remove /dev/md0
Destroy an existing array
mdadm --manage /dev/md2 --fail /dev/sd[cde]1 mdadm --manage /dev/md2 --remove /dev/sd[cde]1 mdadm --manage /dev/md2 --stop mdadm --zero-superblock /dev/sd[cde]1
Speed up a sync (after drive replacement)
cat /proc/sys/dev/raid/speed_limit_max 200000
Rename an existing array
mdadm --stop /dev/md127 mdadm -A /dev/md0 -m127 --update=super-minor /dev/sd[bcd]
Source: Linux Server Tech FAQ - Common mdadm commands - http://wiki.linuxservertech.com/index.php?action=artikel&cat=7&id=11&artlang=en
LVM
See LVM
Larger than 2TB RAID
To get larger than 2TB you need GPTs
parted /dev/sdb
mklabel gpt
print free
mkpart primary 1M 4001GB
p
set 1 raid on
align-check
optimal
1
q
mdadm --create --verbose /dev/md0 --level=5 --raid-devices=3 /dev/sd[bcd]1 mdadm --create --verbose /dev/md0 --level=5 --raid-devices=4 /dev/sd[bcde]1
mkfs.ext4 /dev/md0 -L /ci
cat /proc/mdstat
parted -a optimal /dev/sdf
Ref:
- server - How can I create a RAID array with >2TB disks? - Ask Ubuntu - https://askubuntu.com/questions/350266/how-can-i-create-a-raid-array-with-2tb-disks
- A guide to mdadm - Linux Raid Wiki - https://raid.wiki.kernel.org/index.php/A_guide_to_mdadm
- Using parted to create a RAID primary partition — Lucid Solutions - https://plone.lucidsolutions.co.nz/linux/io/using-parted-to-create-a-raid-primary-partition
Handle Drive Failure
Fail device:
mdadm /dev/md0 --fail /dev/sdb1 # -f mdadm: set /dev/sdb1 faulty in /dev/md0
Remove failed device:
mdadm /dev/md0 --remove /dev/sdb1
Verify failed device:
mdadm --detail /dev/md0
Add device to md:
mdadm /dev/md0 --add /dev/sdb1 # -a
Verify rebuild:
mdadm --detail /dev/md0 cat /proc/mdstat
References:
- Handling a Drive Failure - http://www.gagme.com/greg/linux/raid-lvm.php#failure
Force Partial Failure Recovery
inactive array (missing 2 devices due to io failure)
md4 : inactive sdf3[5] sde3[4] sdd3[3] sdc3[2]
5780957696 blocks
/etc/mdadm.conf
ARRAY /dev/md4 level=raid5 num-devices=6 UUID=f0ce1e02:2cd38a68:6ffb704c:fe6e32b0
Stop array:
mdadm --stop /dev/md4
Forcefully Rebuild Array:
mdadm -A --force /dev/md4 /dev/sd[acdef]3 # notice 'b' is missing as it is dead dead # but 'a' is only partially dead (random io errors)
References:
- linux - How to get an inactive RAID device working again? - Super User - http://superuser.com/questions/117824/how-to-get-an-inactive-raid-device-working-again
Bring online auto-read-only
cat /proc/mdstat Personalities : [raid1] md1 : active (auto-read-only) raid1 sde1[1] sdd1[0]
Bring online:
mdadm --readwrite /dev/md1
Ref:
- raid - New md array is auto-read-only and has resync=PENDING - Unix & Linux Stack Exchange - https://unix.stackexchange.com/questions/101072/new-md-array-is-auto-read-only-and-has-resync-pending
- linux - How do I reactivate my MDADM RAID5 array? - Super User - https://superuser.com/questions/603481/how-do-i-reactivate-my-mdadm-raid5-array
Pause Sync
checkarray -x --all
echo 0 > /proc/sys/dev/raid/speed_limit_max echo 0 > /proc/sys/dev/raid/speed_limit_min
echo frozen > /sys/block/md0/md/sync_action echo none > /sys/block/md0/md/resync_start echo idle > /sys/block/md0/md/sync_action
ref [3]
Notes
See mdadm/Notes