Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Segmentation fault with new kernel after Init process
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware
View previous topic :: View next topic  
Author Message
eric-gentoo
n00b
n00b


Joined: 11 May 2020
Posts: 10

PostPosted: Mon May 11, 2020 5:49 pm    Post subject: Segmentation fault with new kernel after Init process Reply with quote

Hello, I am having trouble booting with kernel 5.4.38. It does a segmentation fault after the Init process and in the end it says "block device UUID=0081cebf-52b7-4bc4-a396-5d9c8d74014e is not a valid root device. I don't think the root cause of the issue is the booting process not finding a valid root device. Something else is causing the kernel to segmentation fault and not finding the root device. This system is working with kernel 4.19.86 build with genkernel. Tried also building the kernel manually and it does the same thing at the same place. I did a genkernel build with the default config to see if it was something with the .config of the old kernel that was causing the issue and I still have the problem. I also build an initramfs with genkernel.

The UUID of the root partition is good:
Code:
Gentoo ~ # lsblk -f /dev/sda3
NAME FSTYPE LABEL UUID                                 FSAVAIL FSUSE% MOUNTPOINT
sda3 ext3   ROOT  0081cebf-52b7-4bc4-a396-5d9c8d74014e    8.3G    11% /
Gentoo ~ #



When it does the segmentation fault right after the Init process, I see errors like this in the console of the server.

Code:
traps: uname[198] general protection fault ip:80550bc sp:bfde498c error:0 in busybox[8049000+16c000]
traps: cut[201] general protection fault ip:80550bc sp:bfde498c error:0 in busybox[8049000+16c000]
traps: mknod[205] general protection fault ip:80550bc sp:bfde498c error:0 in busybox[8049000+16c000]
traps: mknod[206] general protection fault ip:80550bc sp:bfde498c error:0 in busybox[8049000+16c000]


I re-emerged busybox, coreutils, openrc.

One thing I am wondering about is I have not configured the partitions in the fstab file with UUID. The partitions are listed with /dev/sdaX. Is this causing an issue with the booting of newer kernels that I was not facing before?

I am having the same issue with a physical server and a VM in ESXi. Any help will be greatly appreciated. Thank you.

Code:
Gentoo ~ # emerge --info
Portage 2.3.99 (python 3.6.10-final-0, default/linux/x86/17.0, gcc-9.3.0, glibc-2.30-r8, 4.19.86-gentoo i686)
=================================================================
System uname: Linux-4.19.86-gentoo-i686-Intel-R-_Xeon-R-_CPU_E5504_@_2.00GHz-with-gentoo-2.6
KiB Mem:     3109904 total,   2164172 free
KiB Swap:     530428 total,    530428 free
Head commit of repository mv: ed218d791ccbbf68dafd30c24c737510c4f989fa

Head commit of repository spikyatlinux: 0de940c5f2c4b56b9d8240194f643db896adf606

Head commit of repository vmacs: 47c9f6a71b7ed85c63cb481c624ee7fc4892f565

Timestamp of repository gentoo: Sat, 09 May 2020 19:45:01 +0000
Head commit of repository gentoo: ff847fd1b212359109b301db4e349a433ad604d3
sh bash 4.4_p23-r1
ld GNU ld (Gentoo 2.32 p2) 2.32.0
app-shells/bash:          4.4_p23-r1::gentoo
dev-lang/perl:            5.30.1::gentoo
dev-lang/python:          2.7.18::gentoo, 3.5.7::gentoo, 3.6.10-r2::gentoo, 3.7.7-r2::gentoo, 3.8.2-r2::gentoo
dev-util/cmake:           3.14.6::gentoo
dev-util/pkgconfig:       0.29.2::gentoo
sys-apps/baselayout:      2.6-r1::gentoo
sys-apps/openrc:          0.42.1::gentoo
sys-apps/sandbox:         2.13::gentoo
sys-devel/autoconf:       2.69-r4::gentoo
sys-devel/automake:       1.16.1-r1::gentoo
sys-devel/binutils:       2.32-r1::gentoo, 2.33.1-r1::gentoo
sys-devel/gcc:            9.2.0-r2::gentoo, 9.3.0::gentoo
sys-devel/gcc-config:     2.2.1::gentoo
sys-devel/libtool:        2.4.6-r6::gentoo
sys-devel/make:           4.2.1-r4::gentoo
sys-kernel/linux-headers: 5.4::gentoo (virtual/os-headers)
sys-libs/glibc:           2.30-r8::gentoo
Repositories:

mv
    location: /var/lib/layman/mv
    sync-type: git
    sync-uri: git://anongit.gentoo.org/user/mv.git
    masters: gentoo
    priority: 50

spikyatlinux
    location: /var/lib/layman/spikyatlinux
    sync-type: git
    sync-uri: https://github.com/spikyatlinux/ebuilds_for_gentoo.git
    masters: gentoo
    priority: 50

vmacs
    location: /var/lib/layman/vmacs
    sync-type: git
    sync-uri: git://anongit.gentoo.org/user/vmacs.git
    masters: gentoo
    priority: 50

gentoo
    location: /usr/portage
    sync-type: rsync
    sync-uri: rsync://rsync2.ca.gentoo.org/gentoo-portage
    priority: 1000
    sync-rsync-verify-jobs: 1
    sync-rsync-extra-opts:
    sync-rsync-verify-metamanifest: yes
    sync-rsync-verify-max-age: 24

ACCEPT_KEYWORDS="x86"
ACCEPT_LICENSE="@FREE"
CBUILD="i686-pc-linux-gnu"
CFLAGS="-O2 -march=native -pipe"
CHOST="i686-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt /var/bind /var/spool/munin-async/.ssh"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php7.2/ext-active/ /etc/php/apache2-php7.3/ext-active/ /etc/php/apache2-php7.4/ext-active/ /etc/php/cgi-php7.2/ext-active/ /etc/php/cgi-php7.3/ext-active/ /etc/php/cgi-php7.4/ext-active/ /etc/php/cli-php7.2/ext-active/ /etc/php/cli-php7.3/ext-active/ /etc/php/cli-php7.4/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-O2 -march=native -pipe"
DISTDIR="/usr/portage/distfiles"
EMERGE_DEFAULT_OPTS="--with-bdeps=y"
ENV_UNSET="DBUS_SESSION_BUS_ADDRESS DISPLAY GOBIN PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR"
FCFLAGS="-O2 -march=i686 -pipe"
FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync multilib-strict network-sandbox news parallel-fetch pid-sandbox preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -march=i686 -pipe"
GENTOO_MIRRORS="rsync://gentoo.gossamerhost.com/gentoo-distfiles/ ftp://mirrors.tera-byte.com/pub/gentoo http://gentoo.mirrors.tera-byte.com/ rsync://mirrors.tera-byte.com/gentoo ftp://mirror.csclub.uwaterloo.ca/gentoo-distfiles/ https://mirror.csclub.uwaterloo.ca/gentoo-distfiles/ http://mirror.csclub.uwaterloo.ca/gentoo-distfiles/"
LANG="en_US.utf8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
MAKEOPTS="-j3"
PKGDIR="/var/cache/binpkgs"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
USE="acl berkdb bzip2 cli crypt dri fortran gdbm iconv libtirpc logrotate ncurses nls nptl openmp pam pcre readline seccomp split-usr ssl tcpd threads unicode x86 xattr zlib" ABI_X86="32" ADA_TARGET="gnat_2018" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1 emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="cgi cgid info rewrite status authz_host authn_core authz_core unixd dir mime alias include negotiation log_config log_forensic logio socache_shmcb setenvif headers" APACHE2_MPMS="event" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="iptables apache bind apcupsd hddtemp df disk md memory mysql network ntpd rrdtool sensors smart cpu interface ethstat dns" CPU_FLAGS_X86="mmx mmxext popcnt sse sse2 sse3 sse4_1 sse4_2 ssse3" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" GRUB_PLATFORMS="pc" INPUT_DEVICES="libinput" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" OFFICE_IMPLEMENTATION="libreoffice" POSTGRES_TARGETS="postgres10 postgres11" PYTHON_SINGLE_TARGET="python3_6" PYTHON_TARGETS="python3_6" RUBY_TARGETS="ruby24 ruby25" USERLAND="GNU" VIDEO_CARDS="amdgpu fbdev intel nouveau radeon radeonsi vesa dummy v4l" XTABLES_ADDONS="account geoip iface ipmark"
Unset:  CC, CPPFLAGS, CTARGET, CXX, INSTALL_MASK, LC_ALL, LINGUAS, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS

Gentoo ~ #
Back to top
View user's profile Send private message
mike155
Advocate
Advocate


Joined: 17 Sep 2010
Posts: 2126
Location: Frankfurt, Germany

PostPosted: Tue May 12, 2020 1:09 am    Post subject: Reply with quote

Hi eric-gentoo,

welcome to the Gentoo forums!

Please post:
  1. The output of 'lspci'
  2. The kernel config of your Linux 4.19.86-gentoo kernel, which works
  3. The kernel config of your Linux 5.4.38 kernel, which doesn't work
Please use wgetpaste to post the kernel configs.

Mike
Back to top
View user's profile Send private message
eric-gentoo
n00b
n00b


Joined: 11 May 2020
Posts: 10

PostPosted: Tue May 12, 2020 1:39 am    Post subject: Reply with quote

Hello mike155,

Thank you!

Here is the output of lspci:
http://dpaste.com/2HJADCJ


Here is the output of kernel-config-4.19.86-gentoo-x86:
http://dpaste.com/000N4V7

Here is the output of kernel-config-5.4.38-gentoo-x86:
http://dpaste.com/1B1FYQN

I will take a look myself. I did try to debug my physical server more than the VM, so I did not take a look as deep on the VM as the physical server. And the VM is less service impacting if I have to reboot it. So I will troubleshoot this VM and hope to apply the fix on the physical server. I have some screenshots of the boot process of the VM. Is there a way to share those on the forum if you need them?
Back to top
View user's profile Send private message
mike155
Advocate
Advocate


Joined: 17 Sep 2010
Posts: 2126
Location: Frankfurt, Germany

PostPosted: Tue May 12, 2020 2:29 am    Post subject: Reply with quote

I'm not surprised that your 5.4.38 doesn't work. There are many differences between the two kernel configs!

For example:
  • one is optimized for M686, the other for CORE2
  • one has CPU_FREQ enabled, the other not
  • one has PCI Express enabled, the other not
  • ...
Try the steps below:
  • Unpack the source code of the latest 5.4 kernel to /usr/src/linux-5.4
  • Make sure that /usr/src/linux is a link that points to that directory
  • chdir to /usr/src/linux-5.4
  • Copy the .config from the 4.19 kernel that works
  • Run 'make olddefconfig' to migrate the the 4.19 config to version 5.4
  • compile and install the kernel

See: https://wiki.gentoo.org/wiki/Kernel/Upgrade/en
Back to top
View user's profile Send private message
eric-gentoo
n00b
n00b


Joined: 11 May 2020
Posts: 10

PostPosted: Tue May 12, 2020 2:45 am    Post subject: Reply with quote

All those differences are do to the fact that the 5.4.38 kernel was build with the default config of genkernel so there is alot of stuff not present in the customized kernel I am running for the 4.19.86 version.

I wanted to see if the default genkernel config would make my server boot.

I will rebuild the kernel manually like you said and see how it goes tomorrow.

Thank you.
Back to top
View user's profile Send private message
eric-gentoo
n00b
n00b


Joined: 11 May 2020
Posts: 10

PostPosted: Tue May 12, 2020 2:55 pm    Post subject: genkernel 4.0+ problem? Reply with quote

I forgot that I had 2 kernel 4.19.86 in my server. One built with genkernel 3.5.x and one built with genkernel 4.0.1. The kernel built with genkernel 4.0 is showing the same problem as the 5.4 kernel. So it is not a kernel issue. I am thinking it is a Initramfs issue. I have downgraded to genkernel 3.5.3.3 and compiling a new 5.4 kernel manually and generating the initramfs from the 3.5.3.3 genkernel version.

I will post an update as soon as I can.

Thank you
Back to top
View user's profile Send private message
eric-gentoo
n00b
n00b


Joined: 11 May 2020
Posts: 10

PostPosted: Tue May 12, 2020 4:28 pm    Post subject: Reply with quote

Manually compiling a kernel and generating a initramfs with genkernel 3.5.3.3 is working fine. I will compile the kernel and generate initramfs with genkernel 3.5.3.3 now and see if it still works.

All this done with kernel version 5.4.38
Back to top
View user's profile Send private message
eric-gentoo
n00b
n00b


Joined: 11 May 2020
Posts: 10

PostPosted: Tue May 12, 2020 7:11 pm    Post subject: Reply with quote

Fully built kernel and initramfs with genkernel 3.5.3.3 is working fine.

I have rebuilt everything with genkernel 4.0.7-r1 and the problem is back. So I am thinking there is something wrong in the creation of the initramfs. Is there a way to build an initramfs without using genkernel?


Last edited by eric-gentoo on Tue May 12, 2020 7:44 pm; edited 1 time in total
Back to top
View user's profile Send private message
eric-gentoo
n00b
n00b


Joined: 11 May 2020
Posts: 10

PostPosted: Tue May 12, 2020 7:13 pm    Post subject: Reply with quote

Is there a way to compare 2 initramfs file?

I copied over the working initramfs (initramfs-genkernel-x86-5.4.38-gentoo) from genkernel 3.5.3.3 to the file name of the genkernel 4.0.7-r1 initramfs (initramfs-5.4.38-gentoo-x86.img) and the system booted correctly. So the problem is definitely with the initramfs.
Back to top
View user's profile Send private message
eric-gentoo
n00b
n00b


Joined: 11 May 2020
Posts: 10

PostPosted: Wed May 13, 2020 12:12 am    Post subject: initramfs compare Reply with quote

I found the lsinitrd command to show the content of the initramfs file. There are big differences between the initramfs from the 3.5.3.3 genkernel build and the 4.0.7-r1 genkernel initramfs, both with the same version of kernel.

genkernel 3.5.3.3 initramfs:
http://dpaste.com/1X4HK1H

genkernel 4.0.7-r1 initramfs:
http://dpaste.com/2P02755

Maybe lsinitrd is not showing everything in the genkernel 3.5.3.3 initramfs, because it has a size of 9M and it is only showing 7M of files, and no arguments and version information like the 4.0.7 initramfs.
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 15324

PostPosted: Wed May 13, 2020 1:10 am    Post subject: Reply with quote

initramfs are a particular type of archive. If you want to be thorough, you could unpack each archive into parallel trees and compare the results to see which files differ and where.
Back to top
View user's profile Send private message
eric-gentoo
n00b
n00b


Joined: 11 May 2020
Posts: 10

PostPosted: Wed May 13, 2020 2:05 am    Post subject: Reply with quote

I used lsinitrd --unpack and lsinitrd --unpackearly, to extract the files from the initramfs. Is that enough? That did not give me more files than what I see with the lsinitrd /boot/initramfs-genkernel-x86-5.4.38-gentoo for example.
Back to top
View user's profile Send private message
eric-gentoo
n00b
n00b


Joined: 11 May 2020
Posts: 10

PostPosted: Fri May 22, 2020 7:31 pm    Post subject: Continued troubleshooting Reply with quote

I am back with a bit more information on the troubleshooting I did.

I extracted the initramfs file and ran the "uname" symlink in the archive that points to busybox and it segfaults.

Code:

Gentoo ~/genkernel4_initramfs/bin # ls -l
total 1904
lrwxrwxrwx 1 root root       7 May 12 20:30 '[' -> busybox
lrwxrwxrwx 1 root root       7 May 12 20:30  ash -> busybox
-rwxr-xr-x 1 root root 1944444 May 12 20:30  busybox
lrwxrwxrwx 1 root root       7 May 12 20:30  cat -> busybox
lrwxrwxrwx 1 root root       7 May 12 20:30  cut -> busybox
lrwxrwxrwx 1 root root       7 May 12 20:30  echo -> busybox
lrwxrwxrwx 1 root root       7 May 12 20:30  mknod -> busybox
lrwxrwxrwx 1 root root       7 May 12 20:30  mount -> busybox
lrwxrwxrwx 1 root root       7 May 12 20:30  sh -> busybox
lrwxrwxrwx 1 root root       7 May 12 20:30  uname -> busybox
Gentoo ~/genkernel4_initramfs/bin # ./uname
Linux
Segmentation fault
Gentoo ~/genkernel4_initramfs/bin # ./uname -a
Linux Gentoo 5.4.38-gentoo #1 SMP Tue May 12 12:24:17 EDT 2020 i686 GNU/Linux
Segmentation fault
Gentoo ~/genkernel4_initramfs/bin #


So I guess that is why I get the following error when booting
Code:

traps: uname[198] general protection fault ip:80550bc sp:bfde498c error:0 in busybox[8049000+16c000]


And I noticed that the file size of busybox is not the same in the initramfs archive than /bin on the server:
Code:

Gentoo ~/genkernel4_initramfs/bin # ls -l /bin/busybox
-rwxr-xr-x 1 root root 2202492 May 22 11:09 /bin/busybox


Gentoo ~/genkernel4_initramfs/bin # ls -l busybox
-rwxr-xr-x 1 root root 1944444 May 12 20:30 busybox
Gentoo ~/genkernel4_initramfs/bin #


If I run the busybox command that is from the /bin directory on the server it works:
Code:

Gentoo ~/genkernel4_initramfs/bin # /bin/busybox uname
Linux
Gentoo ~/genkernel4_initramfs/bin # /bin/busybox uname -a
Linux Gentoo 5.4.38-gentoo #1 SMP Tue May 12 12:24:17 EDT 2020 i686 i686 i386 GNU/Linux
Gentoo ~/genkernel4_initramfs/bin #


I re-emerge busybox, recreated the initramfs and I still have the issue.

Anyone have an idea?

Thank you.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum