Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
lvm cache broke boot
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Unsupported Software
View previous topic :: View next topic  
Author Message
Colt45
Tux's lil' helper
Tux's lil' helper


Joined: 05 Sep 2007
Posts: 102
Location: Central Washington

PostPosted: Mon Dec 26, 2016 12:25 pm    Post subject: lvm cache broke boot Reply with quote

I decided to get my Gentoo desktop running again this weekend and I managed to break it by adding a 128GB SSD and utilizing lvm cache function.
What I have it backing is three 1TB HDDs in mdadm RAID 5 as a pv for lvm
The problem is its not recognizing the cache or cache_pool type lvs during the initramfs so it refuses to start root.

Ive been making my initramfs via:
Code:
genkernel --lvm --mdadm initramfs




Im gonna try to chroot into it later; is there any way to have the initramfs built with this support?
_________________
"Hopps" - AMD Athlon X2 4450B, 4GB, 128GB Samsung SSD, (Gentoo Hardened amd64; gcc-4.9.3; 4.4.8-hardened-r1)
"Dash" - 2x Intel Xeon 5148, 12GB FB-DIMM, 3x Samsung 1TB (mdadm RAID 5), AMD HD6950, (Gentoo amd64-multilib; gcc-5.3.0; 4.4.0-pf4)
Back to top
View user's profile Send private message
Roman_Gruber
Advocate
Advocate


Joined: 03 Oct 2006
Posts: 3806
Location: Austro Bavaria

PostPosted: Thu Dec 29, 2016 9:19 pm    Post subject: Reply with quote

where is your boot partition?

i use an older intiramfs from genkernel or genkernel-next

You may need to adapt your hole initialization process.

Understand how it works
adapt it.

includes bootloader / init / kernel / initramfs.

And always the same question, are the needed data accessable by the software it needs at that stage

--

you sure have your reason for your setup.

I suggest you use the SSD for root.

Code:
/dev/mapper/root  117G   46G   66G  42% /


small boot partition + otehr partition for uefi on the SSD.

I only need 46gb for my /root. Thats on a SSD.
I also have junk on a 1TB HDD in my notebook.

those 46gb contains also my /home

--

i never used that mdadm thing.

when you already use lvm, why not only use lvm for that mirror thing? you just add an additional layer of complication when you use that.
plain lvm2 is easier to handle

i doubt that mdadm is a hardware raid? it has to be software raid. lvm2 already provides this. called stripped mirroring.
lvm2 is easier to deal with when it fails in my point of view, instead of dealing with 2 layers of software

--

You should check manually what the initramfs does anyway. I had to adapt it to get it working for my use case a year ago. I think genkernel a year ago was broken, but who knows
Back to top
View user's profile Send private message
MageSlayer
Apprentice
Apprentice


Joined: 26 Jul 2007
Posts: 250
Location: Ukraine

PostPosted: Fri Dec 30, 2016 9:37 am    Post subject: Re: lvm cache broke boot Reply with quote

Colt45 wrote:
I decided to get my Gentoo desktop running again this weekend and I managed to break it by adding a 128GB SSD and utilizing lvm cache function.
Im gonna try to chroot into it later; is there any way to have the initramfs built with this support?


You might be interested in https://forums.gentoo.org/viewtopic-t-1014256-highlight-.html
Specifically:
MageSlayer wrote:

Upon reboot, it appears that LVM cache uses cache_check utility from /sbin and it's a c++ application.
It requires libexpat.so* (from /usr/lib), libgcc_s.so* and libstdc++.so* (from /usr/lib/gcc/...).

And those libs were left under LVM and didn't get in initramfs!


Could anybody comment if that issue is fixed in recent genkernel versions?
Back to top
View user's profile Send private message
Colt45
Tux's lil' helper
Tux's lil' helper


Joined: 05 Sep 2007
Posts: 102
Location: Central Washington

PostPosted: Sat Dec 31, 2016 6:38 pm    Post subject: Reply with quote

Thank you very much mage! Thats the information I was looking for.
Currently Ive been working on building a custom initramfs, but its still crashing at the point where LVM tries to do its thing, so this explains it.
Ill see if I can manage to get it working.
_________________
"Hopps" - AMD Athlon X2 4450B, 4GB, 128GB Samsung SSD, (Gentoo Hardened amd64; gcc-4.9.3; 4.4.8-hardened-r1)
"Dash" - 2x Intel Xeon 5148, 12GB FB-DIMM, 3x Samsung 1TB (mdadm RAID 5), AMD HD6950, (Gentoo amd64-multilib; gcc-5.3.0; 4.4.0-pf4)
Back to top
View user's profile Send private message
Colt45
Tux's lil' helper
Tux's lil' helper


Joined: 05 Sep 2007
Posts: 102
Location: Central Washington

PostPosted: Sat Dec 31, 2016 11:19 pm    Post subject: Reply with quote

I wasnt able to get it working with cache_check utility, but I mightve missed something. So, is there anyway to get it to output more information to the console? Right now all Im getting is a kernel panic due to init exiting.
_________________
"Hopps" - AMD Athlon X2 4450B, 4GB, 128GB Samsung SSD, (Gentoo Hardened amd64; gcc-4.9.3; 4.4.8-hardened-r1)
"Dash" - 2x Intel Xeon 5148, 12GB FB-DIMM, 3x Samsung 1TB (mdadm RAID 5), AMD HD6950, (Gentoo amd64-multilib; gcc-5.3.0; 4.4.0-pf4)
Back to top
View user's profile Send private message
szatox
Veteran
Veteran


Joined: 27 Aug 2013
Posts: 1745

PostPosted: Sun Jan 01, 2017 2:42 pm    Post subject: Reply with quote

Append "debug" to your boot line. With genkernel's initramfs it will drop you to shell during early boot so you can have a look around
Back to top
View user's profile Send private message
Colt45
Tux's lil' helper
Tux's lil' helper


Joined: 05 Sep 2007
Posts: 102
Location: Central Washington

PostPosted: Mon Jan 02, 2017 11:56 pm    Post subject: Reply with quote

Im no longer using the genkernel initramfs and have moved my efforts to making my own initramfs.
Unfortunately I still haven't gotten it to work.

Code:

#!/bin/busybox sh

mount -t proc none /proc
mount -t sysfs none /sys

echo /sbin/mdev < /proc/sys/kernel/hotplug
mdev -s
wait
echo "Scanning MDADM"
( sleep 2
  mdadm --assemble --scan || rescue_shell
  sleep 2
) &
wait
echo "Scanning LVM"
lvm vgscan --mknodes || rescue_shell
lvm vgchange --sysinit -a ly vgdash/root || rescue_shell
lvm vgscan --mknodes || rescue_shell
wait
echo "Mounting root"
mount -o ro $(findfs UUID="4e6f31c7-a40f-460b-82b0-631c9be5c60b") /mnt/root || rescue_shell
echo "Clean up and switch to real root"
umount /dev
umount /proc
umount /sys

exec switch_root /mnt/root /sbin/init || rescue_shell

rescue_shell() {
    echo "Something went wrong. Dropping to a shell."
    exec sh
}


Thats my current init. I added the rescue block however its not dropping to it like it should and I don't get anything out of the boot process except "init died"
If anyone has any tips on how to get more info printed to the screen or why it isn't dropping to a shell would be very appreciated.
_________________
"Hopps" - AMD Athlon X2 4450B, 4GB, 128GB Samsung SSD, (Gentoo Hardened amd64; gcc-4.9.3; 4.4.8-hardened-r1)
"Dash" - 2x Intel Xeon 5148, 12GB FB-DIMM, 3x Samsung 1TB (mdadm RAID 5), AMD HD6950, (Gentoo amd64-multilib; gcc-5.3.0; 4.4.0-pf4)
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 13764

PostPosted: Tue Jan 03, 2017 12:18 am    Post subject: Reply with quote

Shell scripts execute in-order. You need to define rescue_shell() before you can call it. Your script defines it only after all possible uses, so any attempt to call it will instead try to run a freestanding command of that name. No such command is likely to exist in your initramfs, so the attempt fails. Also, at least in Bash, you cannot combine exec with ||:
Code:
$ (exec /foo || echo Die)
-bash: /foo: No such file or directory
$
I see no reason for your wait calls. Two of them run when no background jobs could exist. The middle one is used even though there was no reason to create a background job.

If you need more help, please capture as much output as you can and post it for us to review.
Back to top
View user's profile Send private message
szatox
Veteran
Veteran


Joined: 27 Aug 2013
Posts: 1745

PostPosted: Tue Jan 03, 2017 9:29 pm    Post subject: Reply with quote

I think busybox sh silently ignores wait anyway. Check your scripts with the interpreter that's going to actually run them.
Also, hardcoding things like this is usually a bad idea: mount -o ro $(findfs UUID="4e6f31c7-a40f-460b-82b0-631c9be5c60b") /mnt/root || rescue_shell
Back to top
View user's profile Send private message
khayyam
Watchman
Watchman


Joined: 07 Jun 2012
Posts: 6228
Location: Room 101

PostPosted: Tue Jan 03, 2017 11:04 pm    Post subject: Reply with quote

Colt45 wrote:
Code:
echo /sbin/mdev < /proc/sys/kernel/hotplug

Colt45 ... adding to the comments made by Hu, that would be enough to cause init to baulk ... you should have something like the following:

Code:
for d in /dev /mnt/root /sbin /proc /sys /etc /var/log /var/run ; do
  mkdir -m 700 -p "$d"
done
 
[ -e /dev/console ]  || mknod /dev/console c 5 1
[ -e /dev/null ]     || mknod /dev/null c 1 3
[ -e /dev/tty ]      || mknod /dev/tty c 5 0
[ -e /dev/urandom ]  || mknod /dev/urandom c 1 9
[ -e /dev/random ]   || mknod /dev/random c 1 8
[ -e /dev/zero ]     || mknod /dev/zero c 1 5

mount -t proc proc /proc
mount -t sysfs sysfs /sys
echo 0 > /proc/sys/kernel/printk
mount -t tmpfs -o nosuid,relatime,size=10240k,mode=755 dev /dev
touch /etc/mdev.conf
echo /sbin/mdev > /proc/sys/kernel/hotplug
mdev -s


I would probably have a more complex function but the following should work for rescue_shell()
Code:
rescue_shell() {
  echo "Something went wrong. Dropping to a shell."
  if [ -c '/dev/tty1' ]; then
    setsid sh -c 'exec sh --login </dev/tty1 >/dev/tty1 2>&1'
  else
    sh --login
  fi
}

Colt45 wrote:
Thats my current init. I added the rescue block however its not dropping to it like it should and I don't get anything out of the boot process except "init died"

I don't think it's getting that far due to the '<'

Colt45 wrote:
If anyone has any tips on how to get more info printed to the screen or why it isn't dropping to a shell would be very appreciated.

You need to create test conditions (and echo "foo failed") for info.

HTH & best ... khay
Back to top
View user's profile Send private message
Colt45
Tux's lil' helper
Tux's lil' helper


Joined: 05 Sep 2007
Posts: 102
Location: Central Washington

PostPosted: Wed Jan 04, 2017 2:04 am    Post subject: Reply with quote

I moved the rescue block to the top, but it still failed.
Ill try Khay's longer one and see if that works.

What I can tell you is that its getting to the point of mdadm assembling the RAID. I can see from the kernel messages that md127 was assembled from the three hdds.
Its dying right after persumably during lvm since that is the next step. However, none of my 'echo' commands are making it to the screen. They might as well not be there.

That mdev line is verbatim from the custom initramfs page on the gentoo wiki. I based my init off of that page plus the ones in the custom initramfs/examples page..
Basically just copied and pasted one together like pieces of a puzzle.

I agree on the hardcoded UUID, and I was gonna make it source it from the kernel command line, however I already have to use a mdadm.conf in the initramfs /etc to tell mdadm to assemble the array matching UUID suchandsuch
So I already have hardcoded stuff so I decided to just shortcut and hardcode the root UUID as well.
_________________
"Hopps" - AMD Athlon X2 4450B, 4GB, 128GB Samsung SSD, (Gentoo Hardened amd64; gcc-4.9.3; 4.4.8-hardened-r1)
"Dash" - 2x Intel Xeon 5148, 12GB FB-DIMM, 3x Samsung 1TB (mdadm RAID 5), AMD HD6950, (Gentoo amd64-multilib; gcc-5.3.0; 4.4.0-pf4)
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 13764

PostPosted: Wed Jan 04, 2017 2:35 am    Post subject: Reply with quote

khayyam wrote:
Colt45 wrote:
Code:
echo /sbin/mdev < /proc/sys/kernel/hotplug

Colt45 ... adding to the comments made by Hu, that would be enough to cause init to baulk ... you should have something like the following:
I should have caught that mistake, but that line is not immediately fatal. echo has no use for stdin, but does not object to a redirect of stdin. The line you cited will echo /sbin/mdev to the console after redirecting stdin from /proc/sys/kernel/hotplug, and the redirect will be ignored. This is clearly not what the user intended and may cause problems later when he has no configured hotplug - but the shell will not die on this line.
khayyam wrote:
Code:
for d in /dev /mnt/root /sbin /proc /sys /etc /var/log /var/run ; do
  mkdir -m 700 -p "$d"
done
 
[ -e /dev/console ]  || mknod /dev/console c 5 1
[ -e /dev/null ]     || mknod /dev/null c 1 3
[ -e /dev/tty ]      || mknod /dev/tty c 5 0
[ -e /dev/urandom ]  || mknod /dev/urandom c 1 9
[ -e /dev/random ]   || mknod /dev/random c 1 8
[ -e /dev/zero ]     || mknod /dev/zero c 1 5
I suggest instead telling the initramfs machinery to store those in the initramfs so that they do not need to be created explicitly.

Colt45 wrote:
That mdev line is verbatim from the custom initramfs page on the gentoo wiki.
In the spirit of Wiki maintenance: [Citation needed.] :) If the Wiki told you to use <, then the Wiki is wrong and needs to be fixed. A specific citation will guide the next Wiki editor who sees your message to the spot which needs correction.
Back to top
View user's profile Send private message
R0b0t1
Apprentice
Apprentice


Joined: 05 Jun 2008
Posts: 255

PostPosted: Wed Jan 04, 2017 2:57 am    Post subject: Reply with quote

Hello,

I hope this post isn't unwelcome - if you are making your own initramfs, I would suggest referring to the following:

http://unix.stackexchange.com/questions/120015/how-to-find-out-the-dynamic-libraries-executables-loads-when-run
http://stackoverflow.com/questions/6242761/how-do-i-find-the-direct-shared-object-dependencies-of-a-linux-elf-binary

The scripts which genkernel uses to build the initramfs may have a complete function which pulls in dependencies by parsing the output of `ldd`. I would suggest you search for it or look at the source of other parts of genkernel. At the very least the root it uses will be helpful - but you can also have busybox make much of your root for you, and populate /bin and /sbin.

Further, seeing as this is how genkernel handles dependency resolution (packages which would pull in lots of libraries tend to be statically compiled, but this isn't a requirement if I remember correctly) it is unlikely there is a problem with LVM. A few genkernel versions ago it was impossible to invoke LVM commands as one would on a complete installation, e.g. `vgchange` was `lvm vgchange` because some portion of the installation hadn't been propagated to the initramfs. I would invite you to poke around and see if the functionality you're after is hidden. I see no reason why the cache pool functionality should be inaccessible.

You might check to make sure the initramfs is properly installed, modules are loaded or built into the kernel, etc.

Code:
genkernel --install --no-ramdisk-modules initramfs --lvm --mdadm
Back to top
View user's profile Send private message
khayyam
Watchman
Watchman


Joined: 07 Jun 2012
Posts: 6228
Location: Room 101

PostPosted: Wed Jan 04, 2017 5:31 am    Post subject: Reply with quote

Hu wrote:
[...] that line is not immediately fatal. [...] This is clearly not what the user intended and may cause problems later when he has no configured hotplug - but the shell will not die on this line.

Hu ... I perhaps should have been clearer, however, everything subsequent will fail because there will be no device nodes due to that error.

Hu wrote:
I suggest instead telling the initramfs machinery to store those in the initramfs so that they do not need to be created explicitly.

I provided those lines because no information was given on what exists within the initramfs, and so making sure the basic directory structure, and device nodes, exist prior to running mount, or doing anything, would at least cover those as possible issues. No harm will come from doing this, even if those directories, and device nodes, exist, so it's was included to cover all bases.

Colt45 wrote:
Its dying right after persumably during lvm since that is the next step.

Colt45 ... that could be as something fails (ie, mdadm) prior, you might try the following:

Code:
echo "Scaning for software raid arrays."
if [ ! -f '/etc/mdadm.conf' ]; then
  mdadm --examine --scan > /etc/mdadm.conf
fi
mdadm --assemble --scan

echo "Scaning all disks for volume groups."
if [ -d '/etc/lvm/cache' ]; then
  rm -rf '/etc/lvm/cache';
fi
lvm vgchange -a y

Colt45 wrote:
However, none of my 'echo' commands are making it to the screen. They might as well not be there.

I'm not sure why this would happen, the only possible reason I can think of is lack of a tty.

HTH & best ... khay
Back to top
View user's profile Send private message
Colt45
Tux's lil' helper
Tux's lil' helper


Joined: 05 Sep 2007
Posts: 102
Location: Central Washington

PostPosted: Wed Jan 04, 2017 7:17 am    Post subject: Reply with quote

So, progress is happening!

1.) I was snooping and found that the latest ~ thin-provisioning-tools ebuild in portage has a `static` USEFLAG, so I rebuilt it static and dropped it into the init, eliminating the shared libs I had to load and shrinking my initramfs by over 2MB. This is the cache_check utility brought up by MageSlayer.

2.) Since I changed to khay's rescue_shell, Im actually getting dropped to a shell when it dies instead of a kernel panic
2b.) btw the flipped > in the mdev line earlier was my typo. I 'copied' it by retyping it and didn't notice it.

3.) Bad news is Im still not getting any echo's so I have no idea exaclty when its getting dropped. I have a pretty good idea its at lvm becuase of the next item, but at some point I should probably figure out why I have nothing coming from my echos

4.) While I had the shell I tried to get lvm to activate root and was unsuccessful. Its saying "/dev/mapper/vgdash-root_cache_cmeta: No such file or directory"
Im pretty sure its still not recognizing what a cache is. On a hunch I double checked that dm_cache was being built into the kernel, and it is. So Im not sure why its not recognizing it. Ive looked around and this has been done before (booting cache'd /)
and from what Ive read you need
lvm (of course)
dm_cache in the kernel (or presumably you could load it in the ramfs....mines built in)
and the cache_check utility. (which I have added)
_________________
"Hopps" - AMD Athlon X2 4450B, 4GB, 128GB Samsung SSD, (Gentoo Hardened amd64; gcc-4.9.3; 4.4.8-hardened-r1)
"Dash" - 2x Intel Xeon 5148, 12GB FB-DIMM, 3x Samsung 1TB (mdadm RAID 5), AMD HD6950, (Gentoo amd64-multilib; gcc-5.3.0; 4.4.0-pf4)
Back to top
View user's profile Send private message
MaratIK1
n00b
n00b


Joined: 21 Sep 2006
Posts: 4

PostPosted: Tue Oct 09, 2018 7:19 pm    Post subject: Reply with quote

Seems that genkernel-next supports lvm cache. Have you try it?
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Unsupported Software All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum