REMOVE DRIVE FROM EXISTING LVM

What do you do if you need to remove a drive from an existing LVM without adding a new disk? You can't use pvmove because it'll complain that there's no space available, even if you have enough free space. You then get a huge case of the WTF's ;)

Now I may be wrong (and feel free to correct me if I am), but my understanding is that pvmove needs free physical extents (PE). So that free space you can see is actually space on the filesystem, but the filesystem itself is taking up all the free PE. So basically what we need to do is shrink the filesystem so we have room for pvmove to move the data to the free PE.

To make things clearer, imagine this scenario:

We have 3 HDD's, 2 x 300Gb and 1 x 1TB. We want to remove one of the 300Gb drives, and our LVM has 500Gb free space. Our VG is called drives, and our LV is called storage.

    PV /dev/sdb1 VG drives lvm2 [298.00 GB / 0 free]
    PV /dev/sdc1 VG drives lvm2 [298.00 GB / 0 free]
    PV /dev/sdd1 VG drives lvm2 [931.00 GB / 0 free]

NOTE: Just because this worked for me doesn't guarantee success for you. Be very aware that the method below may make your data unrecoverable. I strongly advise a full backup before continuing.

First things first, unmount the logical volume.

    umount /mnt/storage

Resize the filesystem to make room for the data on the 300Gb drive (in this case ext3). I reduced the size to 800Gb to make sure there was enough room for the entire 300Gb disk (roughly 500Gb).

    sudo resize2fs -p /dev/drives/storage 800G

Reduce the logical volume. Make sure you don't reduce below the size of the filesystem. To be safe I've added 10Gb to the size.

    lvreduce -L 810G /dev/drives/storage

Now you can do your pvmove (This will take a looooong time. I left this running overnight)

    sudo pvmove -v /dev/sdb1
    sudo vgreduce drives /dev/sdb1
   
You probably want to extend your logical volume to the maximum available

    sudo vgdisplay  (Note the value for Total PE, mine was 433832)
    sudo lvextend -l 433832 /dev/drives/storage
   
Check you've used the maximum PE

    sudo vgdisplay (Free PE/Size should be 0/0)
   
At this point you may need to run e2fsck to verify

    sudo e2fsck -f /dev/drives/storage
   
Now extend the filesystem back to the maximum available

    sudo resize2fs -p /dev/drives/storage
   
If all goes well you should have a working filesystem with data intact.