Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
[Solved] Kernel panics at boot on new system
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware
View previous topic :: View next topic  
Author Message
redcap
n00b
n00b


Joined: 10 Jul 2010
Posts: 35

PostPosted: Sat Dec 23, 2017 11:40 am    Post subject: [Solved] Kernel panics at boot on new system Reply with quote

Hi there,

I'm trying to setup a new Gentoo system on a Lenovo laptop. Everything went smoothly for the most part, however, the kernel I compiled
crashes early in the boot process. I bootstrapped the system using a systemrescue cd.

To configure my kernel, I took the kernel config from that cd, then

Code:

make localmodconfig


and then some minor adjustments.

The resulting kernel cfg:

[url]
https://paste.pound-python.org/show/itUweUH8hIghQbbeMO2s/
[/url]

For reference, the original kernel config:

[url]
https://paste.pound-python.org/show/ELk1XI6u7JJFFtHRN9A1/
[/url]

And the kernel version of the bootstrapping system:

Code:

root@sysresccd /mnt/gentoo % uname -a
Linux sysresccd 4.9.52-std511-amd64 #2 SMP Wed Sep 27 19:47:36 UTC 2017 x86_64 Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz GenuineIntel GNU/Linux


Here is a picture of the actual kernel output when panicing:

[url]
https://imagebin.ca/v/3lfoUZpQT53r
[/url]

When setting up the system I roughly followed the handbook but made some adjustments.
The most relevant being that I went for a lvm over luks setup with all partition apart from /boot
being encrypted. I put together a small initframfs that I packaged into the kernel at compile time.

This is my init script in the initramfs:

Code:

#!/bin/busybox sh

rescue_shell() {
    echo "Something went wrong. Dropping to a shell."
    exec sh
}

# Mount the /proc, /sys and /dev filesystems.
mount -t proc none /proc
mount -t sysfs none /sys
mount -t devtmpfs none /dev

# Do your stuff here.
echo "Waiting for /dev"
sleep 3

echo "Unlocking ..."
cryptsetup -T 5 --allow-discards luksOpen /dev/sda5 gentoocrypt

echo "Activating logical volumes"
lvm vgscan --mknodes
lvm lvchange -aly vg1/root

echo "Mounting root..."
# Mount the root filesystem.
mount -o ro /dev/mapper/vg1-root /mnt/root || rescue_shell
mount -o ro /dev/mapper/vg1-var /mnt/var || rescue_shell

# Clean up.
umount /proc
umount /sys
umount /dev

# Boot the real thing.
exec switch_root /mnt/root /sbin/init



Anyone got any ideas what might cause the kernel panic. Any help is greatly appreciated.

Thanks in advance and happy holidays :-)


Last edited by redcap on Mon Dec 25, 2017 1:10 am; edited 1 time in total
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 44212
Location: 56N 3W

PostPosted: Sat Dec 23, 2017 2:23 pm    Post subject: Reply with quote

redcap,

Your kernel loaded and everything got started, so grub is leading the kernel.
Then it wanted to kill init, which is a very bad thing.

I don't see any
Code:
echo "Unlocking ..."
in your photo,
Nor the other echos in the initrd.

Is your initrd actually being called?

Its worth adding
Code:
 || rescue_shell
to the commands where its missing, then when they fail, you can use the rescue shell to poke about.
If the kernel slill panics, your initrd is not being used.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 14395

PostPosted: Sat Dec 23, 2017 5:30 pm    Post subject: Reply with quote

You appear to have done at least some of the setup for the initramfs. From what you posted, we can't verify if you did all of it. Is /usr/src/initramfs a directory or is it a kernel initramfs manifest? If it is a directory, what is the output of find /usr/src/initramfs/ -ls? If it is a file, what is the output of cat -n /usr/src/initramfs?
Back to top
View user's profile Send private message
redcap
n00b
n00b


Joined: 10 Jul 2010
Posts: 35

PostPosted: Sat Dec 23, 2017 10:52 pm    Post subject: Reply with quote

Hi there,

I've modified the init file in the following way:

Code:

(chroot) sysresccd / # cat /usr/src/initramfs/init
#!/bin/busybox sh

echo "Starting init in initramfs..."

rescue_shell() {
    echo "Something went wrong. Dropping to a shell."
    exec sh
}

# Mount the /proc, /sys and /dev filesystems.
mount -t proc none /proc || rescue_shell
mount -t sysfs none /sys || rescue_shell
mount -t devtmpfs none /dev || rescue_shell

# Do your stuff here.
echo "Waiting for /dev"
sleep 3

echo "Unlocking ..."
cryptsetup -T 5 --allow-discards luksOpen /dev/sda5 gentoocrypt

echo "Activating logical volumes"
lvm vgscan --mknodes
lvm lvchange -aly vg1/root

echo "Mounting root..."
# Mount the root filesystem.
mount -o ro /dev/mapper/vg1-root /mnt/root || rescue_shell
mount -o ro /dev/mapper/vg1-var /mnt/var || rescue_shell

# Clean up.
umount /proc
umount /sys
umount /dev

# Boot the real thing.
exec switch_root /mnt/root /sbin/init


The first echo in the file produces no output while booting. It seems that the init file is never executed.

/usr/src/initramfs is a directory. The requested output is

Code:

(chroot) sysresccd / # find /usr/src/initramfs/ -ls
   274205      4 drwxr-xr-x  12  root     root         4096 Dec 22 17:37 /usr/src/initramfs/
   274207      4 drwxr-xr-x   2  root     root         4096 Dec 22 16:56 /usr/src/initramfs/bin
   274218   1992 -rwxr-xr-x   1  root     root      2037672 Dec 22 16:55 /usr/src/initramfs/bin/busybox
   274215      4 drwxr-xr-x   2  root     root         4096 Dec 22 16:39 /usr/src/initramfs/root
   274214      4 drwxr-xr-x   2  root     root         4096 Dec 22 16:39 /usr/src/initramfs/proc
   274210      4 drwxr-xr-x   2  root     root         4096 Dec 22 16:39 /usr/src/initramfs/lib
   274216      4 drwxr-xr-x   2  root     root         4096 Dec 22 17:32 /usr/src/initramfs/sbin
   276685   2716 -r-xr-xr-x   1  root     root      2777648 Dec 22 17:10 /usr/src/initramfs/sbin/lvm
   276687   2528 -rwxr-xr-x   1  root     root      2585224 Dec 22 17:32 /usr/src/initramfs/sbin/cryptsetup
   274212      4 drwxr-xr-x   3  root     root         4096 Dec 22 16:39 /usr/src/initramfs/mnt
   274213      4 drwxr-xr-x   3  root     root         4096 Dec 22 18:08 /usr/src/initramfs/mnt/root
   404137      4 drwxr-xr-x   2  root     root         4096 Dec 22 18:08 /usr/src/initramfs/mnt/root/var
   274209      4 drwxr-xr-x   2  root     root         4096 Dec 22 16:39 /usr/src/initramfs/etc
   274208      4 drwxr-xr-x   2  root     root         4096 Dec 22 18:09 /usr/src/initramfs/dev
   276690      0 crw-rw-rw-   1  root     root       1,   8 Dec 21 19:11 /usr/src/initramfs/dev/random
   276689      0 crw-rw-rw-   1  root     root       1,   9 Dec 21 19:11 /usr/src/initramfs/dev/urandom
   274211      4 drwxr-xr-x   2  root     root         4096 Dec 22 16:39 /usr/src/initramfs/lib64
   274217      4 drwxr-xr-x   2  root     root         4096 Dec 22 16:39 /usr/src/initramfs/sys
   276688      4 -rwxr-xr-x   1  root     root          821 Dec 23 21:49 /usr/src/initramfs/init


I also took a look at the entry created in /boot/grub/grub.cfg corresponding to the kernel. It reads as follows:

Code:

### BEGIN /etc/grub.d/10_linux ###
menuentry 'Gentoo GNU/Linux' --class gentoo --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-a17a5e94-fcb4-492e-b3a2-a79d0696ff81' {
        load_video
        if [ "x$grub_platform" = xefi ]; then
                set gfxpayload=keep
        fi
        insmod gzio
        insmod part_gpt
        insmod ext2
        set root='hd0,gpt6'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root --hint-bios=hd0,gpt6 --hint-efi=hd0,gpt6 --hint-baremetal=ahci0,gpt6  19a90a46-5fc7-439d-bf0d-3fe5e8825146
        else
          search --no-floppy --fs-uuid --set=root 19a90a46-5fc7-439d-bf0d-3fe5e8825146
        fi
        echo    'Loading Linux 4.14.8-gentoo-r1 ...'
        linux   /vmlinuz-4.14.8-gentoo-r1 root=/dev/mapper/vg1-root ro
}


I'm not quite sure if the kernel parameter root=/dev/mapper/vg1-root ro is correct, since that device is encrypted. However, when I removed that parameter in GRUB edit mode, the issue was still present.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 44212
Location: 56N 3W

PostPosted: Sat Dec 23, 2017 11:57 pm    Post subject: Reply with quote

redcap,

grub.cfg:
        echo    'Loading Linux 4.14.8-gentoo-r1 ...'
        linux   /vmlinuz-4.14.8-gentoo-r1 root=/dev/mapper/vg1-root ro


There would normally be another two lines there for the initrd.

How do you make and load the initrd?
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
redcap
n00b
n00b


Joined: 10 Jul 2010
Posts: 35

PostPosted: Sun Dec 24, 2017 12:09 am    Post subject: Reply with quote

I followed the guide at

https://wiki.gentoo.org/wiki/Custom_Initramfs#Integrated_initramfs_does_not_always_update

to create the directory structure, init script and binaries for my initramfs.

I choose to compile the initramfs into the kernel as described in

https://wiki.gentoo.org/wiki/Custom_Initramfs#Embedding_into_the_Kernel

It was my understanding that the kernel will unpack the initramfs itself and does not rely on the bootloader providing the location of the initramfs.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 44212
Location: 56N 3W

PostPosted: Sun Dec 24, 2017 12:37 am    Post subject: Reply with quote

redcap,

That's correct. The embedded initrd works. Its the only way with a EFI stub kernel if you need an initrd.
While you have a boot loader, its easier to debug and fix if the intrd is separate.

I'm surprised that you need
Code:
   276690      0 crw-rw-rw-   1  root     root       1,   8 Dec 21 19:11 /usr/src/initramfs/dev/random
   276689      0 crw-rw-rw-   1  root     root       1,   9 Dec 21 19:11 /usr/src/initramfs/dev/urandom

since DEVTMPFS should provide them.

You will also need /dev/{null,console,tty} and your HDD nodes if you need to provide a working static /dev to your initrd.

There may bo other things too like setting the $PATH, so commands look in /sbin or /bin or using the full path in the initrd init script.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
redcap
n00b
n00b


Joined: 10 Jul 2010
Posts: 35

PostPosted: Sun Dec 24, 2017 1:15 am    Post subject: Reply with quote

I must admit that is just copied

/dev/random and /dev/random

to the initrd because it was suggested in the custom initramfs guide

https://wiki.gentoo.org/wiki/Custom_Initramfs#DM-Crypt

I did not realize if or why it was necessary. So if I understood you correctly,

/usr/src/initramfs/dev

can be left empty as long as I have CONFIG_DEVTMPFS=y in the kernel and a long
enough sleep period in the init script to allow the kernel to populate /dev?

I will try to separate the initrd from the kernel to facilitate debugging.

Just to make sure that I got everything right: Your advice concerning $PATH pertains to the lvm and cryptsetup binaries but
the echo statements should be executed successfully either way as soon as the execution of the script started succesfully
since echo is a shell built-in?

Anyways, thanks for your help and I apologize for being a little dense. Getting the first running kernel on a new system has always been
the hardest part of a Gentoo install for me.
Back to top
View user's profile Send private message
redcap
n00b
n00b


Joined: 10 Jul 2010
Posts: 35

PostPosted: Mon Dec 25, 2017 1:10 am    Post subject: Reply with quote

NeddySeagoon,

thanks very much for your input. I managed to boot my system successfully.
The reason for the kernel panic where indeed missing /dev/{null,console,tty}
nodes. I simply overlooked those in the initramfs guide. So it was entirely my
own stupidity that lead to me wasting a few hours of my life. Well, at least that
error won't happen to me again.

Anyways, thanks again and enjoy the holidays :-)
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum