Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
grub: kernel not found after system, kernel update [Solved]
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Installing Gentoo
View previous topic :: View next topic  
Author Message
yarug
Tux's lil' helper
Tux's lil' helper


Joined: 01 Dec 2004
Posts: 117
Location: NL

PostPosted: Mon Dec 23, 2019 5:10 pm    Post subject: grub: kernel not found after system, kernel update [Solved] Reply with quote

I have a home router configured and had to replace a hard disk (RAID 1) which prompted me to make sure the system was up to date.

First off after replacing the hard disk (this was before updating) Grub couldn't find the root file system so I had to manually enter /dev/sda4 at the Grub prompt that was shown with the error after which booting continued.

I then linked the system to the latest profile (17.1, also following the profile update instruction), did a world update, and created a new genkernel.

It was after upgrading genkernel (to 4.0.1) and creating a new (gen)kernel/initramfs and subsequently updating the Grub configuration file with grub-mkconfig that I cannot get the system to boot anymore.

Grub loads showing the new kernel entry, but it immediately fails with:

Loading Linux 4.19.86-gentoo-x86_64
error: file '/vmlinuz-4.19.86-gentoo-x86_64' not found.
Loading initial ramdisk ...
error: you need to load the kernel first.

I have tried to pinpoint the problem, but cannot figure this out.

- grub-mkconfig worked and installed the kernel and initramfs in /boot
- my partition layout is very simple and didn't change, /dev/sda4 for root and /dev/sda2 for boot
- I tried regenerating grub.cfg with GRUB_DISABLE_LINUX_UUID=true (as I suspect UUIDs have something to do with it)
- changing root= kernel parameter to use /dev/sda4 instead of UUID does not help (this did work as a workaround after the hard disk swap and before the upgrade)

Does anyone know how I can fix this?

Thanks in advance.


Last edited by yarug on Sun Dec 29, 2019 9:17 am; edited 3 times in total
Back to top
View user's profile Send private message
yarug
Tux's lil' helper
Tux's lil' helper


Joined: 01 Dec 2004
Posts: 117
Location: NL

PostPosted: Mon Dec 23, 2019 8:30 pm    Post subject: Reply with quote

This is the Grub entry:

Code:

    load_video
   if [ "x$grub_platform" = xefi ]; then
      set gfxpayload=keep
   fi
   insmod gzio
   insmod part_gpt
   insmod ext2
   set root='hd0,gpt2'
   if [ x$feature_platform_search_hint = xy ]; then
     search --no-floppy --fs-uuid --set=root --hint-bios=hd0,gpt2 --hint-efi=hd0,gpt2 --hint-baremetal=ahci0,gpt2  c1f44c7c-ad69-49dc-9959-cfc16eb0e295
   else
     search --no-floppy --fs-uuid --set=root c1f44c7c-ad69-49dc-9959-cfc16eb0e295
   fi
   echo   'Loading Linux 4.19.86-gentoo-x86_64 ...'
   linux    /vmlinuz-4.19.86-gentoo-x86_64 root=UUID=363a15fb-07b4-4f5a-b8b9-20e422e4e0fc ro
    echo   'Loading initial ramdisk ...'
   initrd     /initramfs-4.19.86-gentoo-x86_64.img
Back to top
View user's profile Send private message
GDH-gentoo
Apprentice
Apprentice


Joined: 20 Jul 2019
Posts: 264
Location: South America

PostPosted: Mon Dec 23, 2019 9:58 pm    Post subject: Reply with quote

yarug wrote:
This is the Grub entry:

Code:
[...]
   set root='hd0,gpt2'
   if [ x$feature_platform_search_hint = xy ]; then
     search --no-floppy --fs-uuid --set=root --hint-bios=hd0,gpt2 --hint-efi=hd0,gpt2 --hint-baremetal=ahci0,gpt2  c1f44c7c-ad69-49dc-9959-cfc16eb0e295
   else
     search --no-floppy --fs-uuid --set=root c1f44c7c-ad69-49dc-9959-cfc16eb0e295
[...]
Is c1f44c7c-ad69-49dc-9959-cfc16eb0e295 the UUID of /dev/sda2? Does ls /boot show that file vmlinuz-4.19.86-gentoo-x86_64 is present?
Back to top
View user's profile Send private message
yarug
Tux's lil' helper
Tux's lil' helper


Joined: 01 Dec 2004
Posts: 117
Location: NL

PostPosted: Mon Dec 23, 2019 10:18 pm    Post subject: Reply with quote

GDH-gentoo wrote:
Is c1f44c7c-ad69-49dc-9959-cfc16eb0e295 the UUID of /dev/sda2? Does ls /boot show that file vmlinuz-4.19.86-gentoo-x86_64 is present?


Thanks for the reply GDH-gentoo.

Yes, c1f44c7c-ad69-49dc-9959-cfc16eb0e295 corresponds to /dev/sda2 (my boot partition) and /boot contains the files vmlinuz-4.19.86-gentoo-x86_64 and initramfs-4.19.86-gentoo-x86_64.img.
Back to top
View user's profile Send private message
GDH-gentoo
Apprentice
Apprentice


Joined: 20 Jul 2019
Posts: 264
Location: South America

PostPosted: Tue Dec 24, 2019 1:03 am    Post subject: Reply with quote

yarug wrote:
Yes, c1f44c7c-ad69-49dc-9959-cfc16eb0e295 corresponds to /dev/sda2 (my boot partition) and /boot contains the files vmlinuz-4.19.86-gentoo-x86_64 and initramfs-4.19.86-gentoo-x86_64.img.
Hm, it should just work then. Reboot the computer, and when GRUB's menu is showing, instead of selecting an option, press 'c' to get to the grub> prompt. Then use the ls -lh (hd0,gpt2)/ command to ask GRUB to display the contents of what should be /dev/sda2. Do you see those files in the output?

You can go back to the menu by pressing ESC after the test.
Back to top
View user's profile Send private message
yarug
Tux's lil' helper
Tux's lil' helper


Joined: 01 Dec 2004
Posts: 117
Location: NL

PostPosted: Tue Dec 24, 2019 9:06 am    Post subject: Reply with quote

Oh, this is weird.

ls -lh (hd0,gpt2) from the Grub command prompt lists my old kernels. These are from before the system/kernel update.

Is my new hard drive in the RAID 1 array somehow not syncing??

So Grub sees one hard drive of the array, but when I boot with a live/rescue USB it sees the other?

Is this why "# blkid" when booted with the live/rescue USB outputs both /dev/sda1-4 as /dev/sdb1-4 (but for example /dev/sda4 and /dev/sdb4 have the same UUID and the same PARTUUID)?

ps. apologies for the late response, I am on a west Europe time zone.
Back to top
View user's profile Send private message
yarug
Tux's lil' helper
Tux's lil' helper


Joined: 01 Dec 2004
Posts: 117
Location: NL

PostPosted: Tue Dec 24, 2019 11:28 am    Post subject: Reply with quote

To clarify:

I deleted the files listed in (hd0,gpt2)/ earlier and you don't see those in a chrooted live/rescue session (/boot mounted on /dev/sda2).

The new kernel (4.19.86) is not there, but it is there in a chrooted live/rescue session (while the old kernels are not).

In both cases (Grub's screen and chrooted live/rescue session) it seems to pick up the latest /boot/grub/grub.cfg (with 4.19.86 kernel).
Back to top
View user's profile Send private message
DONAHUE
Watchman
Watchman


Joined: 09 Dec 2006
Posts: 7602
Location: Goose Creek SC

PostPosted: Tue Dec 24, 2019 2:03 pm    Post subject: Reply with quote

possible that a separate boot or EFI system partition was or was not mounted at the time the new kerenl was compiled and installed or when grub -mkconfig was run?? usually ls -al run with separate /boot not mounted should show empty except . and .. and show kernels and asystem maps and configs and grub directory when boot partition is mounted. umount the boot partition then mount it somewhere else like /mnt or /mnt/boot and then mv all files in the boot directory to the /mnt location where the boot partition is mounted then umount from /mnt abd mount boot partition to the empty boot directory/mountpoint ls /boot check all is well, rerun grub-mkinstall and grub-mkconfig -o /boot/grub
_________________
Defund the FCC.
Back to top
View user's profile Send private message
yarug
Tux's lil' helper
Tux's lil' helper


Joined: 01 Dec 2004
Posts: 117
Location: NL

PostPosted: Tue Dec 24, 2019 10:49 pm    Post subject: Reply with quote

Hi DONAHUE. Thanks for your support. See my answers below.

Quote:
possible that a separate boot or EFI system partition was or was not mounted at the time the new kerenl was compiled and installed or when grub -mkconfig was run??

Not that I can think of.

Quote:
usually ls -al run with separate /boot not mounted should show empty except . and .. and show kernels and asystem maps and configs and grub directory when boot partition is mounted

Correct. When I enter the chrooted environment before mounting /dev/sda2 to /boot, /boot is empty. After mounting /boot it lists the kernel files:

Code:

(chroot)
00:39 0 livecd / # ls -al /boot
total 8
drwxr-xr-x  2 root root 4096 Oct 16  2016 .
drwxr-xr-x 21 root root 4096 Dec 23 16:36 ..
-rw-r--r--  1 root root    0 Oct 16  2016 .keep
(chroot)
00:39 0 livecd / # mount /dev/sda2 /boot
(chroot)
00:40 0 livecd / # ls -al /boot
total 17594
drwxr-xr-x  4 root root    1024 Dec 23 20:20 .
drwxr-xr-x 21 root root    4096 Dec 23 16:36 ..
-rw-r--r--  1 root root       0 Oct 16  2016 .keep
-rw-r--r--  1 root root 3309939 Dec 23 20:00 System.map-4.19.86-gentoo-x86_64
drwxr-xr-x  6 root root    1024 Dec 23 20:21 grub
-rw-r--r--  1 root root 8070776 Dec 23 20:17 initramfs-4.19.86-gentoo-x86_64.img
drwx------  2 root root   12288 Sep 12  2013 lost+found
-rw-r--r--  1 root root 6541984 Dec 23 20:00 vmlinuz-4.19.86-gentoo-x86_64


Quote:
umount the boot partition then mount it somewhere else like /mnt or /mnt/boot and then mv all files in the boot directory to the /mnt location where the boot partition is mounted then umount from /mnt abd mount boot partition to the empty boot directory/mountpoint ls /boot check all is well

I'm not sure I follow. There are no files in the /boot directory if /boot is not mounted (/dev/sda2).
Back to top
View user's profile Send private message
yarug
Tux's lil' helper
Tux's lil' helper


Joined: 01 Dec 2004
Posts: 117
Location: NL

PostPosted: Wed Dec 25, 2019 10:28 am    Post subject: Reply with quote

Some further investigation. I started up again with a gentoo live USB.

If I mount /dev/sda2 to /mnt/gentoo I see my new kernel (4.19.86)

Code:

livecd ~ # mount /dev/sda2 /mnt/gentoo
livecd ~ # ls -al /mnt/gentoo/
total 12946
drwxr-xr-x 4 root root    1024 Dec 25 00:52 .
drwxr-xr-x 6 root root     120 Dec 24 17:22 ..
-rw-r--r-- 1 root root       0 Oct 16  2016 .keep
-rw-r--r-- 1 root root 2961468 Dec 25 00:38 System.map-4.19.86-gentoo-x86_64
drwxr-xr-x 6 root root    1024 Dec 25 00:52 grub
-rw-r--r-- 1 root root 4954620 Dec 25 00:46 initramfs-4.19.86-gentoo-x86_64.img
drwx------ 2 root root   12288 Sep 12  2013 lost+found
-rw-r--r-- 1 root root 5268016 Dec 25 00:38 vmlinuz-4.19.86-gentoo-x86_64


However, if I mount /dev/sdb2 to /mnt/gentoo I see the list of old kernels (before system/kernel update) and this is probably what Grub sees when I boot up.

Code:

livecd ~ # umount /dev/sda2
livecd ~ # mount /dev/sdb2 /mnt/gentoo
livecd ~ # ls -al /mnt/gentoo/
total 59726
drwxr-xr-x 4 root root    1024 Oct 16  2016 .
drwxr-xr-x 6 root root     120 Dec 24 17:22 ..
-rw-r--r-- 1 root root       0 Oct 16  2016 .keep
-rw-r--r-- 1 root root 2101526 Apr 16  2014 System.map-genkernel-x86_64-3.12.13-gentoo
-rw-r--r-- 1 root root 2193027 Jan 24  2015 System.map-genkernel-x86_64-3.17.7-gentoo
-rw-r--r-- 1 root root 2192880 Jan 31  2015 System.map-genkernel-x86_64-3.17.8-gentoo-r1
-rw-r--r-- 1 root root 2264577 Jan  3  2016 System.map-genkernel-x86_64-4.1.12-gentoo
drwxr-xr-x 4 root root    1024 Jan  3  2016 grub
-rw-r--r-- 1 root root 9234244 Apr 17  2014 initramfs-genkernel-x86_64-3.12.13-gentoo
-rw-r--r-- 1 root root 9312112 Jan 24  2015 initramfs-genkernel-x86_64-3.17.7-gentoo
-rw-r--r-- 1 root root 9308860 Jan 31  2015 initramfs-genkernel-x86_64-3.17.8-gentoo-r1
-rw-r--r-- 1 root root 9426612 Jan  3  2016 initramfs-genkernel-x86_64-4.1.12-gentoo
-rw-r--r-- 1 root root 3600880 Apr 16  2014 kernel-genkernel-x86_64-3.12.13-gentoo
-rw-r--r-- 1 root root 3704384 Jan 24  2015 kernel-genkernel-x86_64-3.17.7-gentoo
-rw-r--r-- 1 root root 3704224 Jan 31  2015 kernel-genkernel-x86_64-3.17.8-gentoo-r1
-rw-r--r-- 1 root root 3840832 Jan  3  2016 kernel-genkernel-x86_64-4.1.12-gentoo
drwx------ 2 root root   12288 Sep 12  2013 lost+found


The weird thing is though that /dev/sda2(/grub.grub.cfg) contains also the corresponding (new) Grub config and that /dev/sdb2(/grub.grub.cfg) contains the old Grub config. But when booting, Grub displays the new config (from /dev/sda2) and the old kernel list (/dev/sdb2) :?:

Also I can mount /dev/sda4, but not /dev/sdb4:

Code:

livecd ~ # mount /dev/sda4 /mnt/gentoo
livecd ~ # umount /dev/sda4
livecd ~ # mount /dev/sdb4 /mnt/gentoo
mount: /mnt/gentoo: wrong fs type, bad option, bad superblock on /dev/sdb4, missing codepage or helper program, or other error.


This is the output from parted for bot /dev/sda and /dev/sdb:

Code:

livecd ~ # parted -a optimal /dev/sda
GNU Parted 3.2
Using /dev/sda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print                                                           
Model: ATA MB1000GCWCV (scsi)
Disk /dev/sda: 1000GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system     Name    Flags
 1      1049kB  34.6MB  33.6MB                  grub    bios_grub
 2      34.6MB  303MB   268MB   ext2            boot    msftdata
 3      303MB   4598MB  4295MB  linux-swap(v1)  swap    msftdata
 4      4598MB  1000GB  996GB   ext4            rootfs  msftdata

(parted) q                                                               
livecd ~ # parted -a optimal /dev/sdb
GNU Parted 3.2
Using /dev/sdb
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print                                                           
Error: The backup GPT table is corrupt, but the primary appears OK, so that will be used.
OK/Cancel? OK                                                             
Model: ATA MB1000GCWCV (scsi)
Disk /dev/sdb: 1000GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system     Name    Flags
 1      1049kB  34.6MB  33.6MB                  grub    bios_grub
 2      34.6MB  303MB   268MB   ext2            boot    msftdata
 3      303MB   4598MB  4295MB  linux-swap(v1)  swap    msftdata
 4      4598MB  1000GB  996GB   ext4            rootfs  msftdata

(parted) q                                                               


And the output from blkid:

Code:

livecd ~ # blkid
/dev/loop0: TYPE="squashfs"
/dev/sda1: PARTLABEL="grub" PARTUUID="f07b1822-c457-44f1-bcfb-9a75a15a4648"
/dev/sda2: UUID="c1f44c7c-ad69-49dc-9959-cfc16eb0e295" TYPE="ext2" PARTLABEL="boot" PARTUUID="fb2eb17c-f0a6-4799-b1e0-c856cc4caec9"
/dev/sda3: UUID="b68d0122-6c9e-44de-8ae3-3e0f38a9fb10" TYPE="swap" PARTLABEL="swap" PARTUUID="44fd6afc-6c84-4ee5-95c8-692f3c6b8423"
/dev/sda4: UUID="363a15fb-07b4-4f5a-b8b9-20e422e4e0fc" TYPE="ext4" PARTLABEL="rootfs" PARTUUID="02b72bbd-cec7-4f99-b8c0-245a2bdabe8a"
/dev/sdb1: PARTLABEL="grub" PARTUUID="f07b1822-c457-44f1-bcfb-9a75a15a4648"
/dev/sdb2: UUID="c1f44c7c-ad69-49dc-9959-cfc16eb0e295" TYPE="ext2" PARTLABEL="boot" PARTUUID="fb2eb17c-f0a6-4799-b1e0-c856cc4caec9"
/dev/sdb3: UUID="b68d0122-6c9e-44de-8ae3-3e0f38a9fb10" TYPE="swap" PARTLABEL="swap" PARTUUID="44fd6afc-6c84-4ee5-95c8-692f3c6b8423"
/dev/sdb4: UUID="363a15fb-07b4-4f5a-b8b9-20e422e4e0fc" TYPE="ext4" PARTLABEL="rootfs" PARTUUID="02b72bbd-cec7-4f99-b8c0-245a2bdabe8a"
/dev/sdc1: UUID="2019-12-23-03-48-36-03" LABEL="Gentoo amd64 20191222T214502Z" TYPE="iso9660" PTUUID="5cd52f0e" PTTYPE="dos" PARTUUID="5cd52f0e-01"
/dev/sdc2: SEC_TYPE="msdos" LABEL_FATBOOT="GENTOOLIVE" LABEL="GENTOOLIVE" UUID="913D-5710" TYPE="vfat" PARTUUID="5cd52f0e-02"



So, the questions are

a) why do I see both /dev/sda and /dev/sdb? There only should be /dev/sda as per the logical volume definition in my hardware raid setup.
b) why does grub see the new configuration file but the old kernel list (some sort of mix from /dev/sda and /dev/sdb)?

I could just copy everything from /dev/sda2 to /dev/sdb2 and try to reboot but I'm worried that I'm in some corrupted drive state and that my RAID (1) is not functioning properly.
Back to top
View user's profile Send private message
DONAHUE
Watchman
Watchman


Joined: 09 Dec 2006
Posts: 7602
Location: Goose Creek SC

PostPosted: Wed Dec 25, 2019 2:31 pm    Post subject: Reply with quote

blklid says there is no raid
you wrote:
had to replace a hard disk (RAID 1)

apparently array was not rebuilt after the disk replacement
addressing a raid by a name that the kernel assigns to a disc device seems likely to just address the device and not the raid
a decade or two ago ran software raid1 on most of two disks but not on sda1,sda2, legacy grub did not do raid at the time so sda1 sdb1 were for boot. installed to one and copied to the other to maintain redundancy.
_________________
Defund the FCC.
Back to top
View user's profile Send private message
Tony0945
Advocate
Advocate


Joined: 25 Jul 2006
Posts: 3653
Location: Illinois, USA

PostPosted: Wed Dec 25, 2019 3:18 pm    Post subject: Reply with quote

Sorry I thought this was different thread. Erased my answer.
Back to top
View user's profile Send private message
yarug
Tux's lil' helper
Tux's lil' helper


Joined: 01 Dec 2004
Posts: 117
Location: NL

PostPosted: Wed Dec 25, 2019 3:49 pm    Post subject: Reply with quote

Thanks DONAHUE. I appreciate you answering on Christmas Day. I'll investigate the B120i controller some more. From what I saw so far was that everything was reported as normal and there was a logical drive (disk name /dev/sda) created consisting of 2 hard drives, status was reported as OK, RAID Level as RAID 1. I might just take out the new drive and boot up with just the 1 original drive, see if everything is in place that way (fingers crossed) and recreate the array.

DONAHUE wrote:

blklid says there is no raid
you wrote:
had to replace a hard disk (RAID 1)

apparently array was not rebuilt after the disk replacement


Is it normal blkid shows the same UUID/PARTUUID for the 2 drives (in my case)? I would assume if it where 2 seperate drives the IDs would be different?
Back to top
View user's profile Send private message
DONAHUE
Watchman
Watchman


Joined: 09 Dec 2006
Posts: 7602
Location: Goose Creek SC

PostPosted: Wed Dec 25, 2019 5:55 pm    Post subject: Reply with quote

you wrote:
in my hardware raid setup.
and
Is it normal blkid shows the same UUID/PARTUUID for the 2 drives (in my case)? I would assume if it where 2 seperate drives the IDs would be different?

Cloning UUID/PARTUUID is possible but seems to violate the spirit and intent of Universally Unique IDentifier. HP does do strange things ...
Every linux raid I have dealt with (2 or 3 max) has been a different device with different device names from its component disks or partitions.
B120i controller linux stuff here https://access.redhat.com/articles/118133 & https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-c03871499
B120i controller gentoo google search finds results in forum
_________________
Defund the FCC.
Back to top
View user's profile Send private message
yarug
Tux's lil' helper
Tux's lil' helper


Joined: 01 Dec 2004
Posts: 117
Location: NL

PostPosted: Thu Dec 26, 2019 10:33 am    Post subject: Reply with quote

From some further reading into the B120i controller (including this forum) I understand now that:

a) the hardware is not fully supported outside Red Hat / SUSE (proprietary drivers)
b) seems to be not full/real RAID in any case (see also this post https://forums.gentoo.org/viewtopic-p-7965638.html#7965638)

I have been running this server for years in the understanding that it was (and probably lucky that it was the drive in Bay 2 that failed and not the drive in Bay 1, but I'll find that out soon).

What I will do is remove the new hard drive, boot up with only the older drive, reconfigure the HP controller to disable RAID and configure the system with software raid (mdadm).
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Installing Gentoo All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum