Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
LVM fails to detect physical volume on RAID device
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware
View previous topic :: View next topic  
Author Message
orionbelt
Apprentice
Apprentice


Joined: 05 Apr 2006
Posts: 168
Location: Brussels, Belgium

PostPosted: Wed Apr 17, 2019 9:01 am    Post subject: LVM fails to detect physical volume on RAID device Reply with quote

I have the following 4 software RAID arrays on 4 disks ( /dev/sd[a-d] ):
Code:
% cat /proc/mdstat
Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [linear] [multipath]
md3 : active raid10 sda4[0] sdd4[3] sdc4[2] sdb4[1]
      951117824 blocks super 1.2 512K chunks 2 near-copies [4/4] [UUUU]
      bitmap: 0/8 pages [0KB], 65536KB chunk

md2 : active raid1 sdb6[4] sdd6[3] sda6[2] sdc6[5]
      9763768 blocks super 1.2 [4/4] [UUUU]
     
md1 : active raid1 sdb5[4] sdd5[3] sda5[2] sdc5[5]
      1950708 blocks super 1.2 [4/4] [UUUU]
     
md0 : active raid1 sdb1[0] sdd1[3] sda1[2] sdc1[1]
      30656 blocks [4/4] [UUUU]
     
unused devices: <none>

I was forced to do a hard reboot due to an X freeze. After the reboot, all 4 RAID arrays were missing the same 2 disks each, namely sdb and sdc. I was able to add these disks back to the 3 RAID1 arrays (md0, md1 and md2), e.g.:
Code:
mdadm --manage /dev/md0 --add /dev/sdb1 /dev/sdc1

and those arrays started recovering without a problem.

However, i was less lucky with md3, which was producing the error:
Code:
mdadm: /dev/md3 assembled from 2 drives - not enough to start the array.

I spent a lot of time looking for info on how to re-assemble the array, i tried the --force and --run options etc., but nothing worked. I won't bet my life that there was nothing else left to try, but i ended up with the nuclear option of zeroing the superblocks and re-creating the array with the --assume-clean option:
Code:
mdadm --zero-superblock /dev/sd[abcd]4
mdadm --create /dev/md3 --level=raid10 --raid-devices=4 --assume-clean /dev/sda4 /dev/sdb4 /dev/sdc4 /dev/sdd4

This "worked", producing the "cat /proc/mdstat" output that i am showing at the top.

But when i rebooted i got error messages on the console like this:
Code:
fsck.ext4: No such file or directory while trying to open /dev/vg/home

These are all LVM logical volumes residing on /dev/md3. So LVM can no longer locate the physical volume vg. Indeed:
Code:
% pvscan
  PV /dev/sde1   VG vg_tera         lvm2 [<1.82 TiB / 839.01 GiB free]
  Total: 1 [<1.82 TiB] / in use: 1 [<1.82 TiB] / in no VG: 0 [0   ]

vg_tera is a physical volume residing on a separate hard disk ( /dev/sde ), unrelated to the RAID arrays. Which seems to suggest that LVM works properly, and that "mdadm --create" failed to reconstruct /dev/md3, and the array is unrecoverable. Unfortunately, however, there is no partition on md3 that is not on LVM to try to mount it directly and see what happens. So i am not completely sure that all hope is lost...

So before restoring the backups, i would like to ask around whether there is anything else i could still do at this point to salvage the array...

In particular, i still have access to /etc/lvm/backup/vg config file. Assuming that "mdadm --create" did not destroy the array, could it be that something in the process destroyed just the LVM metadata, and that i might be able to restore them from /etc/lvm/backup/vg ?

Also, if the problem is indeed that the md3 array is dead, might there be something that could still be tried to recover it at this point?

Any and all help is thoroughly and in advance appreciated... :?
Back to top
View user's profile Send private message
PeGa!
Tux's lil' helper
Tux's lil' helper


Joined: 13 May 2005
Posts: 104
Location: Capital Federal, Buenos Aires, Argentina

PostPosted: Wed Apr 17, 2019 4:11 pm    Post subject: Reply with quote

Hello orionbelt,

/dev/md3 is a raid 10 volume, as per your /proc/mdstat.

A raid 10 volume is comprised by two raid 0 members and two raid 1 members. This means there are two ways of building the array: A raid 0 volume out of two raid 1 members, or a raid 1 volume out of two raid 0 members.

In any case, you can't mix one raid 1 member with one raid 0 member, because the raid 0 member can't be alone. Needs another drive for it to work. So given your error:

Code:
mdadm: /dev/md3 assembled from 2 drives - not enough to start the array.


Somehow two drives weren't recognized as valid members of the raid 10 volume.

In summary: Try to discover what was your original setup or to understand the actual topology of the four drives and start from there, analyzing the faulty disks.

Regards.
_________________
If it moves, it needs an ebuild.

Adopt an unanswered post.
Back to top
View user's profile Send private message
orionbelt
Apprentice
Apprentice


Joined: 05 Apr 2006
Posts: 168
Location: Brussels, Belgium

PostPosted: Wed Apr 17, 2019 5:14 pm    Post subject: Reply with quote

Hello PeGa!,

Thanks for your reply.

I just ran a script that tried all 24 permutations in the order of sda4, sdb4, sdc4 and sdd4, followed in each case by pvscan to see whether the LVM physical volume on md3 was actually detected, i.e. something like:
Code:
mdadm --stop /dev/md3
mdadm --create --assume-clean /dev/md3 --level=10 --raid-devices=4 /dev/sda4 /dev/sdb4 /dev/sdc4 /dev/sdd4
pvscan

mdadm --stop /dev/md3
mdadm --create --assume-clean /dev/md3 --level=10 --raid-devices=4 /dev/sda4 /dev/sdb4 /dev/sdd4 /dev/sdc4
pvscan

mdadm --stop /dev/md3
mdadm --create --assume-clean /dev/md3 --level=10 --raid-devices=4 /dev/sda4 /dev/sdc4 /dev/sdb4 /dev/sdd4
pvscan

......

So in essence i tried all possible ways to reorganize the array, which was your suggestion. Unfortunately, pvscan never detected the LVM volume on md3.

However, i am not certain that LVM would recognize the LVM partition on md3 automatically if md3 suddenly became a valid array. LVM can auto-detect a new disk connected to the system, but i am not sure that a suddenly activated and readable RAID array counts as such.

Is there perhaps some other way to force LVM to scan the newly-appeared md3 device for LVM partitions?
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum