Linux/mdadm

From Omnia
Jump to navigation Jump to search
The printable version is no longer supported and may have rendering errors. Please update your browser bookmarks and please use the default browser print function instead.

mdadm

"mdadm is a Linux utility by Neil Brown that is used to manage RAID devices, previously known as mdctl. Besides managing, it can create, delete, or monitor Linux software RAIDs. Available under version 2 or later of the GNU General Public License, mdadm is free software. Mdadm derives its name from the “md” (multiple disk) device nodes it manages." [1]

Commands

Show all md devices status:

cat /proc/mdstat

Show md device details:

mdadm --detail /dev/md0

Create RAID 0 (stripe):

mdadm --create --verbose /dev/md0 --level=raid0 --raid-devices=3 /dev/hd[cde]1

Create RAID 5 (stripe with parity):

mdadm --create --verbose /dev/md0 --level=5 --raid-devices=3 /dev/sd[cde]1
mdadm --create --verbose /dev/md0 --level=5 --raid-devices=3 --spare-devices=1 /dev/sd[cdef]1

Add partition to md device:

mdadm /dev/md0 --add /dev/sdb1

Fail and remove partition of md device:

mdadm /dev/md0 --fail /dev/sdb1
mdadm /dev/md0 --remove /dev/sdb1

Start drive without resync:

mdadm -S /dev/md0
mdadm -A --assume-clean /dev/md0

Create /etc/mdadm.conf (high to low detail, any will work) [2]

mdadm --detail --scan --verbose > /etc/mdadm.conf
mdadm --detail --scan > /etc/mdadm.conf
mdadm --examine --scan > /etc/mdadm.conf
# ubuntu uses /etc/mdadm/mdadm.conf
sudo mdadm --examine --scan --config=/etc/mdadm/mdadm.conf
sudo mdadm --examine --scan --config=/etc/mdadm/mdadm.conf >> /etc/mdadm/mdadm.conf
sudo mdadm -A /dev/md0

Stop array:

mdadm --stop /dev/md0

Start (assemble) array:

mdadm --assemble /dev/md0

Add disk to RAID 5:

unmount /dev/md1
mdadm --add /dev/md1 /dev/sdb3
mdadm --grow --raid-devices=4 /dev/md1

# wait for rebuild...
watch -dc cat /proc/mdstat
e2fsck -f /dev/md3
resize2fs /dev/md3

fdisk to 'Linux raid':

type 'fd'
   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1            2048  1953525167   976761560   fd  Linux raid autodetect

Common mdadm commands

Generate mdadm.conf

cp /etc/mdadm/mdadm.conf /etc/mdadm/mdadm.conf.save
/usr/share/mdadm/mkconf --generate > /etc/mdadm/mdadm.conf

Create RAID

mdadm --create /dev/md2 --raid-devices=3 --spare-devices=0 --level=5 --run /dev/sd[cde]1

Remove disk from RAID

mdadm --fail /dev/md0 /dev/sda1
mdadm --remove /dev/md0 /dev/sda1

Copy the partition structure (when replacing a failed drive)

sfdisk -d /dev/sda | sfdisk /dev/sdb 
mdadm --zero-superblock /dev/sdb

Add a disk to a RAID array (to replace a removed failed drive)

mdadm --add /dev/md0 /dev/sdf1

Check RAID status

cat /proc/mdstat
mdadm --detail /dev/md0

Reassemble a group of RAID disks

#This works to move an assembly from one physical machine to another.
mdadm --assemble /dev/md0 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1

Steps to emulate mdrun (which has been depreciated)

# haven't tested this. Use with care
mdadm --examine --scan --config=partitions > /tmp/mdadm.conf
mdadm --assemble --scan --config=/tmp/mdadm.conf

Add a disk to an existing RAID and resize the filesystem

mdadm --add /dev/md0 /dev/sdg1
mdadm --grow /dev/md0 -n 5
e2fsck -f /dev/md0
resize2fs /dev/md0
e2fsck -f /dev/md0

Replace all disks in an array with larger drives and resize

# For each drive in the existing array
mdadm --fail /dev/md0 /dev/sda1
mdadm --remove /dev/md0 /dev/sda1
# physically replace the drive
mdadm --add /dev/md0 /dev/sda1
# now, wait until md0 is rebuilt.
# this can literally take days
# All drives have been replaced and sync'd, but they still use the original size.
# Issue the following command to use all available space:
mdadm --grow /dev/md0  --size=max
# Do not forget to resized the file system which sits on the raid set:
# for ext2/3/4
e2fsck -f /dev/md0 && resize2fs /dev/md0 && e2fsck -f /dev/md0
# for lvm pv
pvresize /dev/md0
# for ntfs
ntfsresize /dev/md0
# note, most likely ntfs is NOT exported as a single partition. In the case
# of a Xen hvm machine, it is a "disk device" so you will need to resize the
# partition itself, then resize ntfs.

Stop and remove the RAID device

mdadm --stop /dev/md0
mdadm --remove /dev/md0

Destroy an existing array

mdadm --manage /dev/md2 --fail /dev/sd[cde]1
mdadm --manage /dev/md2 --remove /dev/sd[cde]1
mdadm --manage /dev/md2 --stop
mdadm --zero-superblock /dev/sd[cde]1

Speed up a sync (after drive replacement)

cat /proc/sys/dev/raid/speed_limit_max
200000

Rename an existing array

mdadm --stop /dev/md127
mdadm -A /dev/md0 -m127 --update=super-minor /dev/sd[bcd]

Source: Linux Server Tech FAQ - Common mdadm commands - http://wiki.linuxservertech.com/index.php?action=artikel&cat=7&id=11&artlang=en

LVM

See LVM

Larger than 2TB RAID

To get larger than 2TB you need GPTs

parted /dev/sdb
 mklabel gpt
 print free
 mkpart primary 1M 4001GB
 p
 set 1 raid on
 align-check
    optimal
    1
 q
mdadm --create --verbose /dev/md0 --level=5 --raid-devices=3 /dev/sd[bcd]1
mdadm --create --verbose /dev/md0 --level=5 --raid-devices=4 /dev/sd[bcde]1
mkfs.ext4 /dev/md0 -L /ci
cat /proc/mdstat
parted -a optimal /dev/sdf


Ref:

Handle Drive Failure

Fail device:

mdadm /dev/md0 --fail /dev/sdb1    # -f
  mdadm: set /dev/sdb1 faulty in /dev/md0

Remove failed device:

mdadm /dev/md0 --remove /dev/sdb1

Verify failed device:

mdadm --detail /dev/md0

Add device to md:

mdadm /dev/md0 --add /dev/sdb1    # -a

Verify rebuild:

mdadm --detail /dev/md0
cat /proc/mdstat

References:

Force Partial Failure Recovery

inactive array (missing 2 devices due to io failure)

md4 : inactive sdf3[5] sde3[4] sdd3[3] sdc3[2]
      5780957696 blocks

/etc/mdadm.conf

ARRAY /dev/md4 level=raid5 num-devices=6 UUID=f0ce1e02:2cd38a68:6ffb704c:fe6e32b0

Stop array:

mdadm --stop /dev/md4

Forcefully Rebuild Array:

mdadm -A --force /dev/md4 /dev/sd[acdef]3
# notice 'b' is missing as it is dead dead
# but 'a' is only partially dead (random io errors)

References:

Bring online auto-read-only

cat /proc/mdstat 
Personalities : [raid1] 
md1 : active (auto-read-only) raid1 sde1[1] sdd1[0]

Bring online:

mdadm --readwrite /dev/md1

Ref:

Notes

See mdadm/Notes

keywords