Linux software RAID+LVM2 with root on RAID HOWTO


Note: This information is geared toward advanced users. Don’t try this at home.

This weekend it was time to perform some maintenance on my main server. Its single IDE hard drive had developed the metallic whine indicative of bearing failure, and the power supply began the constant whistling telling the world that it was doomed.

Original storage/power config:

  • Western Digital 200GB 8MB cache ATA-133
  • Onboard AMD-75xx IDE controller
  • 250W ATX PSU

(Yikes.)

Uprated specs:

  • 2x Seagate 250GB 16MB cache SATA/300
  • Promise TX2 SATA controller
  • 600W Enermax NoiseTaker II ATX12V 2.0 PSU

The typical setup for this is: create partitions of matching size on each SATA drive and pair these off, creating numerous RAID1 arrays. This has always struck me as inelegant. But, every HOWTO I could find insisted that at the very least, /boot and swap must be created this way. You can put / on LVM if you’re a cowboy, but you’re pushing your luck.

Knowing something of the way Linux RAID, LVM and LILO work, I decided this didn’t make any sense.

The resulting config is much more elegant than the method prescribed by conventional wisdom. A single RAID array hosts my logical volumes, everything including the filesystems, kernel image and swap are hosted within these, and the bootloader understands the whole thing.

Caveat: GRUB cannot understand LVM2, but LILO can. So much for GRUB being newer and more advanced – while most everyone now swears by GRUB, LILO has continued to add innovative features behind the scenes.

What I used:

  • Distribution: Gentoo Linux
  • Kernel: Linux 2.6.22
  • Bootloader: LILO 22.8
  • LVM: LVM2 2.02.10
  • MDADM: mdadm 2.6.2

And this initrd script.

I will shortly investigate how this would be done under a Red Hat based distribution. This is of particular interest to me, because Red Hat’s installer insists up and down that /boot must not be on anything more exotic than an MD RAID1 based partition, which is obviously not a requirement based on technical limitations of the underlying software.

Getting Started – RAID

I booted the server from a LiveCD (the Gentoo installcd in this case).

Both SATA disks were prepped for integration into a RAID array by using fdisk to create a single large partition on each disk and setting the type of the partition to ‘fd’, which is the type code for ‘Linux RAID Autodetect’.

mdadm -C -l1 -n2 /dev/md0 /dev/sda1 /dev/sdb1
Jigga WHAT?

LVM2

Time to set up LVM. If you’re not familiar with LVM, I highly recommend that you read some documentation before staking your reputation on it. The LVM2 Resource Page and LVM HOWTO on TLDP are great reading.

In short:

livecd:/ # pvcreate /dev/md0
livecd:/ # vgcreate TMA-1 /dev/md0
livecd:/ # lvcreate -L1G -nswap TMA-1
livecd:/ # lvcreate -L300M -nroot TMA-1
livecd:/ # lvcreate -L25G -nhome TMA-1
livecd:/ # lvcreate -L25G -nvar TMA-1
livecd:/ # lvcreate -L128M -ntmp TMA-1
livecd:/ # lvcreate -L181G -nusr TMA-1
livecd:/ # mkswap /dev/TMA-1/swap
livecd:/ # mke2fs -j /dev/TMA-1/root

You get the idea. I then copied the contents of the filesystem on the original, dying IDE drive to this fresh, new filesystem once I mounted all volumes.

Kernel and INITRD

Next step: Kernel has to be built with SATA controller driver, SCSI disk support, Device Mapper and RAID1 built in. Install the kernel and modules as normal.

Use the lvm2create_initrd script as follows:

sh lvm2create_initrd.sh -M gentoo -c /etc/lvm/lvm.conf -r /dev/md0 -R /etc/mdadm.conf 2.6.22-gentoo-r1

If you’re using Red Hat or Debian, specify ‘redhat’ or ‘debian’ as the -M option. Specify config file locations and kernel version string as needed.

Bootloader – LILO

I used version 22.8. I’m unsure whether earlier versions support the device mapper (the kernel component used to read LVM style partition boundaries). I will list here the deviations from a standard LILO config file. Please familiarize yourself with other aspects LILO configuration.

Two lilo.conf files need to be created, or you can modify the one. The reason is as follows:

boot = /dev/sda
boot = /dev/sdb

LILO needs to be written to the MBR of both (or all) RAID array members. Specifying boot = /dev/md0 will cause it to mutter racial epithets and quit. Also, you can’t use two boot parameters. So don’t paste the above verbatim into your config file. Silly goose.

image = /boot/vmlinuz-2.6.22
root = /dev/ram0
initrd = /boot/initrd-lvm2-2.6.22-gentoo-r1.gz
append = "lvm2root=/dev/TMA-1/root"
label = Gentoo

The important lines in the above config snippet are root =, initrd =, and append =. Specify the filename given to you by the lvm2create_initrd script in the initrd line.

Remember to edit your fstab file. Your device paths for logical volumes are now named according to the volume group you created. In my case, for example:

/dev/TMA-1/root / xfs noatime 0 0
/dev/TMA-1/swap none swap sw 0 0

Write LILO to your MBR’s, reboot and you’re set. Gentoo’s init scripts understand how to start and stop RAID and LVM, as do many other modern distributions.

There’s something terribly satisfying about a state-of-the-art configuration. 😉


4 responses to “Linux software RAID+LVM2 with root on RAID HOWTO”

  1. Awesome guide! I recently got interested in LVM and RAID and couldn’t any info about booting from both and thus was hesitant to use it in my system. This is just what I needed, since I’m also using Gentoo ;).
    I’m really curious of that initrd script (I never user a ram disk before) but understanding all the code is going to be a pain…
    BTW, I couldn’t believe that GRUB did not support LVM2. With a bit of surfing, I found out about GRUB2 (http://grub.enbug.org/) which does support both LVM and RAID, but it’s still WIP.

  2. Interesting note about LILO. I eventually switched to grub, grudgingly because all the places I worked insisted it was better (even though they never set the security password because it was too inconvenient ;). I’m going to have to look back into switching to LILO again.

    Great post!

    Thanks,

    Mike

  3. Great article, it will definitely help me to configure my server and all software on it. I tried to do it without any instructions but it was so hard. I hope your advices will help me to configure this god damn LVM and Raid. Once more time, thanks!