RAID on Ubuntu
How To Create RAID Arrays with mdadm on Ubuntu 22.04 | DigitalOcean
How to install Ubuntu with software RAID-1
What is RAID?
The Redundant Array of Independent Disks, commonly known as RAID, is a technology used to combine multiple physical disk drives into a single logical unit for the purpose of data storage and performance improvement.
This blog will demonstrate to set up software RAID on Ubuntu(It should also work on other Linux).
To setup Intel VROC RAID on Ubuntu, go to see: Intel VROC RAID on Ubuntu.
The hierarchy of storage abstraction layers in Linux.
+------------------------+
| Application |
| |
|open("/mnt/media/1.png")|
| |
+------------------------+
+------------------------+
| Mount Point | Higher layer
| | ^
|/dev/sda1 -> /mnt/media | |
|/dev/sda2 -> /mnt/file | |
| | |
+------------------------+ |
|
+------------------------+ |
| Logical Volume | |
| | |
| /dev/mapper/vg1-lg1, | |
| /dev/mapper/vg1-lg2, | |
+------------------------+ |
|
+------------------------+ |
| File System | |
| | |
| ext4, Btrfs, etc | |
| | |
+------------------------+ |
|
+------------------------+ |
| Partition | |
| | |
| /dev/sda1, /dev/sda2 | |
| | |
+------------------------+ |
|
|
+------------------------+ |
| Physical Disk Drive | |
| | |
| HDD /dev/hda | Lower layer
| SSD /dev/sda |
+------------------------+
The RAID is treated as Logical Disk Drive above the Physical Disk Drive layer. So you should do partitioning on it after being created.
Background
Recently, there is a chance for me to install Ubuntu(server) OS on a Dell Precision xxx workstation which is includes 2 NVMe SSDs and 8 SATA hard drives(HDDs) and has in-box hardware-assisted RAID controller(Intel VROC). In the past, I just play cloud virtual machines or personal host with single disk.
Configuring storage is a critical part of setting up a reliable workstation. So firstly, how to organize these following disks to their roles?
- 2 SSDs hold the system to load quickly
- 8 HDDs store data persistently
In order to access these physical disks easily and reduce damages from data loss, I need to combine multiple disks to act as one, while keep data redundant and backup. After step-by-step research, there are some enterprise solutions present for me. These drive layer or file system layer approaches designed for specific purposes have their own advantages over others while they maybe achieve some same features.
Here are some benefits and shortcomings of them alongside common use cases:
-
RAID(Redundant Array of Independent Disks) Abstraction level: drive layer Concept: RAID uses multiple drives to act as one(logical drive). Benefits: improve data redundancy and data read/write performance
-
LVM(Logical Volume Management) Abstraction level: file system layer Concept: Manage a logical volume over multiple drives, each drive is a Physical Volume(PV). Benefits: combine multiple disks into one logical volume, extend the volume with new disk added, increase/decrease mounted folder in file system
-
ZFS(Z File System)
There are three types of raid, as Wiki saying:
- hardware RAID
- software RAID
- mdadm in Linux
- hardware-assisted software RAID, firmware RAID, fake RAID
- Intel VROC (Virtual RAID on CPU)
This document will introduce how to set up software RAID(RAID0, RAID1, RAID5, RAID 10) on already-installed Ubuntu.
To create a RAID to hold the Ubuntu OS when installing Ubuntu, see SoftwareRAID or How to install Ubuntu with software RAID-1
In addition, there are different challenges you will face when installing Ubuntu Server and Ubuntu Desktop.
Install Ubuntu Server on RAID: Ubuntu Server Image has inbox
mdadmutilities, so it is quite convenient to create the software RAID on multiple disks then install Ubuntu Server OS on the RAID in storage layer step during OS installation stage.
Install Ubuntu Desktop on RAID: Ubuntu Desk Image does not ship the
mdadmtool, so it is nearly impossible to create RAID and install Ubuntu Desktop OS on the RAID(however this one Install Ubuntu 20.04 desktop with RAID 1 and LVM on machine with UEFI BIOS from stackoverflow seems to be successful)
Set up RAID array
To create a RAID array ready to use in practice, there are always common steps:
- Create a RAID array(RAID 0, RAID 1, RAID 5 or RAID 10)
- Mount the RAID array
- Save the RAID array configuration for system boot
Create RAID array with mdadm
Create RAID 0 array using devices: /dev/sda and /dev/sdb
sudo mdadm --create --verbose /dev/md0 -l 0 -n 2 /dev/sda /dev/sdb
Mount RAID array for use
- Create a
ext4filesystem on the array
sudo mkfs.ext4 -F /dev/md0
- Mount the array
sudo mkdir -p /mnt/md0
sudo mount /dev/md0 /mnt/md0
Save RAID array configuration
Persist the RAID array configuration to make the system reassemble and mount the RAID array automatically after reboot.
Append the line to /etc/mdadm/mdadm.conf:
sudo mdadm --detail --scan | sudo tee -a /etc/mdadm/mdadm.conf
Make RAID array available in early boot stage:
sudo update-initramfs -u
Persist the mount point, edit /etc/fstab:
/dev/md0 /mnt/md0 ext4 defaults,nofail,discard 0 0
or persist the mount point by using UUID, get UUID of the disk drive,
$ blkid /dev/md124
/dev/md124: UUID="b7fa44f2-0f05-47a1-b4ef-e9ad306898de" BLOCK_SIZE="4096" TYPE="ext4"
then edit in /etc/fstab,
UUID=b7fa44f2-0f05-47a1-b4ef-e9ad306898de /volume ext4 defaults,nofail,discard 0 0
finally apply the new mount,
sudo mount -a
Delete RAID Array with mdadm
Make sure to remove what are using the RAID array,
[Optional] Umount the array from filesystem if mounted,
sudo umount /dev/md0
Stop RAID array,
sudo mdadm --stop /dev/md0
# Stop all arrays
sudo mdadm --stop --scan
Removes the RAID metadata and resets them to normal on the Drives,
sudo mdadm --zero-superblock /dev/sda
sudo mdadm --zero-superblock /dev/sd[a-h]
[Optional] Remove any persistent references to the array if exist. Edit the /etc/fstab:
sudo nano /etc/fstab
[Optional] Also, remove the array definition if exist, from the /etc/mdadm/mdadm.conf file:
sudo nano /etc/mdadm/mdadm.conf
Manage RAID Array with mdadm
Find the RAID arrays
$ cat /proc/mdstat
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10]
md126 : active raid1 nvme0n1[1] nvme1n1[0]
3800741888 blocks super external:/md127/0 [2/2] [UU]
md127 : inactive nvme0n1[1](S) nvme1n1[0](S)
10402 blocks super external:imsm
unused devices: <none>
Query information on RAID array
sudo mdadm --detail /dev/md0
sudo mdadm --query /dev/md0
Query information on individual physical devices
sudo mdadm --query /dev/sda
sudo mdadm --examine /dev/sda
Stop RAID array
sudo mdadm --stop /dev/md0
# Stop all arrays
sudo mdadm --stop --scan
Start an RAID array
# This works if the array is defined in the configuration `/etc/mdadm/mdadm.conf` file.
sudo mdadm --assemble --scan
sudo mdadm --assemble /dev/md0
# If the array is not persisted in `/etc/mdadm/mdadm.conf` file but keeping RAID metadata
sudo mdadm --assemble /dev/md0 /dev/sda /dev/sdb
Add a spare device to an RAID array
sudo mdadm /dev/md0 --add /dev/sde
Check block devices
$ lsblk -f
NAME FSTYPE FSVER LABEL UUID FSAVAIL FSUSE% MOUNTPOINTS
loop0 squashfs 4.0 0 100% /snap/core20/1974
loop1 squashfs 4.0 0 100% /snap/lxd/24322
loop2 squashfs 4.0 0 100% /snap/snapd/19457
sda isw_raid_member 1.3.00
sdb isw_raid_member 1.3.00
sdc isw_raid_member 1.3.00
sdd isw_raid_member 1.3.00
sde isw_raid_member 1.3.00
sdf isw_raid_member 1.3.00
sdg isw_raid_member 1.3.00
sdh isw_raid_member 1.3.00
nvme0n1 isw_raid_member 1.3.00
├─md126
│ ├─md126p1 vfat FAT32 292B-DB66 1G 1% /boot/efi
│ └─md126p2 ext4 1.0 0f58386c-334d-4877-8051-b855bae37fb0 3.3T 0% /
└─md127
nvme1n1 isw_raid_member 1.3.00
├─md126
│ ├─md126p1 vfat FAT32 292B-DB66 1G 1% /boot/efi
│ └─md126p2 ext4 1.0 0f58386c-334d-4877-8051-b855bae37fb0 3.3T 0% /
└─md127
List UUID of devices
$ sudo blkid
/dev/sdf: TYPE="isw_raid_member"
/dev/nvme0n1: TYPE="isw_raid_member"
/dev/sdd: TYPE="isw_raid_member"
/dev/sdb: TYPE="isw_raid_member"
/dev/sdg: TYPE="isw_raid_member"
/dev/sde: TYPE="isw_raid_member"
/dev/sdc: TYPE="isw_raid_member"
/dev/md126p2: UUID="ff1f3640-e590-486b-8570-c34dfd7bd1de" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="07473e4a-9324-435d-9238-cf358cd9a6a9"
/dev/md126p1: UUID="A636-3441" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="7eb27871-d9ad-4132-af06-7110948faf06"
/dev/nvme1n1: TYPE="isw_raid_member"
/dev/sda: TYPE="isw_raid_member"
/dev/md124: UUID="b7fa44f2-0f05-47a1-b4ef-e9ad306898de" BLOCK_SIZE="4096" TYPE="ext4"
/dev/sdh: TYPE="isw_raid_member"
/dev/loop1: TYPE="squashfs"
/dev/loop4: TYPE="squashfs"
/dev/loop2: TYPE="squashfs"
/dev/loop0: TYPE="squashfs"
/dev/loop3: TYPE="squashfs"
Partition a disk
sudo fdisk -l /dev/sda
Create filesystem on disk
sudo mkfs.ext4 -F /dev/sda
Delete partition and data in disk
sudo dd if=/dev/zero of=/dev/sda bs=512 count=1