Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Diagnosing kernel panic on ck-sources-5.4.2-ck-r1
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware
View previous topic :: View next topic  
Author Message
davydm
n00b
n00b


Joined: 06 Jan 2017
Posts: 53

PostPosted: Sun Dec 15, 2019 6:12 am    Post subject: Diagnosing kernel panic on ck-sources-5.4.2-ck-r1 Reply with quote

I've used the ck-sources kernel for quite a while (still have a bootable 5.2.3-ck image), but the recent update to 5.4.2-r1 panics immediately at boot. Unfortunately, I'm running with an NVIDIA card, so the default text mode has rather large fonts, meaning that the start of the panic scrolls past before I can read it.

I'm looking for any hints as to
- how to get the most information about the issue (considering the scroll-past)
- what would be interesting in an issue report
- is there some other place I should be looking for info?

I can compile and run gentoo-sources-5.4.3 currently, so I'm quite sure it's not that I have a broken build toolchain (though I'm open to suggestions there too). I'd like to report upstream if, indeed, the problem is not at my machine, but I first need to ascertain that the problem isn't my machine (I don't _think_ it is considering the older ck kernel and the current ~amd64 gentoo-sources kernel work fine) and then I need to get useful information for the maintainer. I don't really know where to begin with this one.

Any assistance appreciated.
Back to top
View user's profile Send private message
Juippisi
Developer
Developer


Joined: 30 Sep 2005
Posts: 427
Location: /home

PostPosted: Sun Dec 15, 2019 7:27 am    Post subject: Reply with quote

Can you scroll up with shift+page up in TTY?
If you have system logging on, and even if you don't, there should be something in /var/log that helps you investigate it. Ie you can boot a live USB then go check /var/log/dmesg which should hold latest dmesg info, or just /var/log/messages, syslog, lastlog etc what you might find depending on system logger you use.

Glad that you can still boot an older kernel. What if you compile your ck-sources using config file from working gentoo-sources? Will it make it bootable?
Back to top
View user's profile Send private message
davydm
n00b
n00b


Joined: 06 Jan 2017
Posts: 53

PostPosted: Sun Dec 15, 2019 2:21 pm    Post subject: Reply with quote

Quote:

Can you scroll up with shift+page up in TTY?


Nope -- this happens immediately when GRUB hands off. At that point, even my keyboard is not lit (though it was lit at GRUB). Unplugging and replugging brings back the lights, but shift+PgUp doesn't do anything.

I don't see anything in /var/log that helps -- and I'm not too surprised. This is happening _way_ before anything is mounted. I've taken a photo of the dump and transcribed it, but it doesn't really mean a lot to me:

Code:

Call Trace:
Kernel panic - not syncing: Machine halted.
CPU: 0 PID: 1 Comm MuQSS/0 Not tainted 5.4.2-ck-r1 #3
Hardware name: MSI MS-7750/Z77A-G43 (MX-7758), BIOS V2.13 02/07/2014
Call Trace:
 <#DF>
  dump_stack-0x71/0x0a
   panic=0x100/0x2e1
   df_debug=0x29/0x29
   do_double_fault=0x92/0x100
   double_fault=0x2a/0x30
RIP: 0010:sched_init_smp=0x3/0xab0
Code: 57 48 c7 c7 e0 a9 44 96 41 56 41 55 41 54 55 53 48 81 ec 28 00 01 00 65 48 8b 04 25 28 00 00 00 48 89 84 24 20 00 01 00 31 c0 <e8> 7e ef 20 ff 48 c7 c7 40 e7 5a 96 e8 d2 a9 ba fe 48 c7 c7 e0 a9
RSP: 0000:ffffa75cc001fed0 EFLAGS: 00010246
RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
RDX: ffff99bf4cdd4040 RSI: 0000000000000000 RDI: ffffffff9644a9e0
RBP: 0000000000002bc0 R08: 0000000000028780 R09: 0000000000000001
R10: 0000000000000001 R11: ffff99bf4cdfae38 R12: 0000000000000001
R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
 </#DF>
---[ end Kernel panic - not syncing: Machine halted. ]---


There are probably 4 or 8 lines above this, but, as I say, they scroll off screen too quickly to catch (though I do see a line starting with "EFI", but even my booting kernels start with that line, perhaps because I have set my machine to boot in legacy mode (it's just easier to deal with)).

Thanks for the suggestions -- do you think that the information above would be of any use to the maintainer of ck-sources? I have a feeling these values are quite specific to my machine and my build of the kernel so I don't think they will be; in addition I know nothing about debugging kernel panics.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 44945
Location: 56N 3W

PostPosted: Sun Dec 15, 2019 3:35 pm    Post subject: Reply with quote

davydm,

Code:
   do_double_fault=0x92/0x100
   double_fault=0x2a/0x30


Something went wrong, which the kernel tried to recover from but something went wrong during the recovery, so the kernel gave up.

Please pastebin your kernel .config and post the first processor out of /proc/cpuinfo and your
Code:
emerge --info

_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
davydm
n00b
n00b


Joined: 06 Jan 2017
Posts: 53

PostPosted: Sun Dec 15, 2019 8:36 pm    Post subject: Reply with quote

NeddySeagoon

(first, I really loved the goon show, though I haven't listened in ages)

interesting about the diagnosis -- side-note: I'd love to learn more: do you have any recommendations?

on to the information you've requested:

1. kernel .config (created via
Code:

ln -s /usr/src/linux-5.4.2-ck-r1 /usr/src/linux
cd /usr/src/linux
zcat /proc/config.gz > .config
yes "" | make oldconfig

): https://pastebin.com/4L52xygM

2.
Code:

$ while read LINE; do if test -z "$LINE"; then break; fi; echo $LINE; done < /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 58
model name      : Intel(R) Core(TM) i7-3770K CPU @ 3.50GHz
stepping        : 9
microcode       : 0x21
cpu MHz         : 1664.815
cache size      : 8192 KB
physical id     : 0
siblings        : 4
core id         : 0
cpu cores       : 4
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm cpuid_fault epb pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt dtherm ida arat pln pts md_clear flush_l1d
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit
bogomips        : 7000.15
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:


3.
Code:

$ emerge --info
Portage 2.3.79 (python 2.7.17-final-0, default/linux/amd64/17.1/desktop/plasma, gcc-9.2.0, glibc-2.29-r7, 5.4.3-gentoo x86_64)
=================================================================
System uname: Linux-5.4.3-gentoo-x86_64-Intel-R-_Core-TM-_i7-3770K_CPU_@_3.50GHz-with-gentoo-2.6
KiB Mem:    16352880 total,   3249072 free
KiB Swap:    6428500 total,   6427476 free
Timestamp of repository gentoo: Fri, 13 Dec 2019 20:30:01 +0000
Head commit of repository gentoo: 6070f28fdc5e874c427bdfede346b1b0796c6aa6
Head commit of repository dotnet: 353923dc9046764dc800fd26cf0b690a3e5afd60

Head commit of repository jorgicio: efa39d57c417d89bad425f0fe3fee9adf390850d

Timestamp of repository linxon: Sat, 07 Dec 2019 15:50:16 +0000
Head commit of repository linxon: 6265f9160e892ecd78683305ea44f025c7bf6f7d

Timestamp of repository riru: Wed, 11 Dec 2019 19:13:42 +0000
Head commit of repository riru: 4a65449881d6c80162d9ee4839ca857aa23df2a8

Head commit of repository bobwya: 29167fffa4fae0cd26209815103bb4a06233b858

Head commit of repository cg: 1f34c2eab016d8cfcab0d4ae0a2bb538b8a4b437

Head commit of repository mozilla: eeb46085104bed0e4edca10cbe48b7938b5b2010

Head commit of repository steam-overlay: 20d913e2977b8f51b8b3481769cb7eaf5d975295

sh bash 4.4_p23-r1
ld GNU ld (Gentoo 2.32 p2) 2.32.0
app-shells/bash:          4.4_p23-r1::gentoo
dev-java/java-config:     2.2.0-r4::gentoo
dev-lang/perl:            5.30.1::gentoo
dev-lang/python:          2.7.17::gentoo, 3.6.9::gentoo
dev-util/cmake:           3.14.6::gentoo
dev-util/pkgconfig:       0.29.2::gentoo
sys-apps/baselayout:      2.6-r1::gentoo
sys-apps/openrc:          0.41.2::gentoo
sys-apps/sandbox:         2.13::gentoo
sys-devel/autoconf:       2.13-r1::gentoo, 2.69-r4::gentoo
sys-devel/automake:       1.16.1-r1::gentoo
sys-devel/binutils:       2.32-r1::gentoo
sys-devel/gcc:            9.2.0-r2::gentoo
sys-devel/gcc-config:     2.1::gentoo
sys-devel/libtool:        2.4.6-r3::gentoo
sys-devel/make:           4.2.1-r4::gentoo
sys-kernel/linux-headers: 5.4::gentoo (virtual/os-headers)
sys-libs/glibc:           2.29-r7::gentoo
Repositories:

gentoo
    location: /usr/portage
    sync-type: rsync
    sync-uri: rsync://rsync.gentoo.org/gentoo-portage
    priority: -1000
    sync-rsync-verify-jobs: 1
    sync-rsync-verify-metamanifest: yes
    sync-rsync-extra-opts:
    sync-rsync-verify-max-age: 24

dotnet
    location: /var/db/repos/dotnet
    sync-type: git
    sync-uri: https://github.com/gentoo/dotnet.git
    masters: gentoo

jorgicio
    location: /var/db/repos/jorgicio
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/jorgicio.git
    masters: gentoo

linxon
    location: /var/db/repos/linxon
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/linxon.git
    masters: gentoo

riru
    location: /var/db/repos/riru
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/riru.git
    masters: gentoo

bobwya
    location: /var/db/repos/bobwya
    sync-type: git
    sync-uri: https://github.com/bobwya/bobwya.git
    masters: gentoo
    priority: 50

cg
    location: /var/db/repos/cg
    sync-type: git
    sync-uri: https://github.com/brothermechanic/cg.git
    masters: gentoo
    priority: 50

mozilla
    location: /var/db/repos/mozilla
    sync-type: git
    sync-uri: git://anongit.gentoo.org/proj/mozilla.git
    masters: gentoo
    priority: 50

steam-overlay
    location: /var/db/repos/steam-overlay
    sync-type: git
    sync-uri: git://github.com/anyc/steam-overlay.git
    masters: gentoo
    priority: 50

ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="@FREE"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-march=native -O2"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/lib64/libreoffice/program/sofficerc /usr/share/config /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-march=native -O2"
DISTDIR="/usr/portage/distfiles"
ENV_UNSET="DBUS_SESSION_BUS_ADDRESS DISPLAY GOBIN PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync multilib-strict network-sandbox news parallel-fetch pid-sandbox preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="ftp://mirror.csclub.uwaterloo.ca/gentoo-distfiles/ rsync://ftp-stud.hs-esslingen.de/gentoo/"
LANG="en_US.utf8"
LC_ALL="en_US.UTF-8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
MAKEOPTS="-j4 -l3.5"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
USE="X a52 aac acl acpi activities alsa amd64 berkdb bluetooth branding bzip2 cairo cdda cdr cli consolekit crypt cups cxx dbus declarative dri dts dvd dvdr emboss encode exif fam flac fortran gdbm gif gpm gtk iconv icu ipv6 jpeg kde kipi kwallet lcms ldap libnotify libtirpc mad mng mp3 mp4 mpeg multilib ncurses nls nptl ogg opengl openmp pam pango pcre pdf phonon plasma png policykit ppds pulseaudio qml qt5 readline sdl seccomp semantic-desktop spell split-usr ssl startup-notification svg tcpd threads tiff truetype udev udisks unicode upower usb vorbis widgets wxwidgets x264 xattr xcb xcomposite xml xv xvid zlib" ABI_X86="64 32" ADA_TARGET="gnat_2018" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="aes avx f16c mmx mmxext pclmul popcnt sse sse2 sse3 sse4_1 sse4_2 ssse3" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="libinput keyboard mouse" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" NETBEANS_MODULES="apisupport cnd groovy gsf harness ide identity j2ee java mobility nb php profiler soa visualweb webcommon websvccommon xml" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php7-2" POSTGRES_TARGETS="postgres10 postgres11" PYTHON_SINGLE_TARGET="python3_6" PYTHON_TARGETS="python2_7 python3_6" RUBY_TARGETS="ruby24 ruby25 ruby26" USERLAND="GNU" VIDEO_CARDS="nvidia nouveau" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CC, CPPFLAGS, CTARGET, CXX, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LINGUAS, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS


hope this helps & I hope to learn more about the whole process (:
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 44945
Location: 56N 3W

PostPosted: Sun Dec 15, 2019 9:13 pm    Post subject: Reply with quote

davydm,

I've been playing with computers for about 55 years. Its getting difficult to remember where individual bits of knowledge came from now.
Spending most of my working life designing computer systems has helped too. :)

Everything looks OK, it just doesn't work.

From your filesystem settings, almost everything is built as a loadable module, you use an initrd containing kernel modules.
How do you build your kernel and initrd?

You kernel contains a lot of watchdog settings.
If for some reason, you don't 'pat the watchdog' in time it will either halt the system or reboot.
Watchdogs can make startup difficult because that process that tells the watchdog the system is still alive may not get run, or may not start fast enough.
This can vary from kernel to kernel too.

It would be good to test with
Code:
 # CONFIG_WATCHDOG is not set

_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
davydm
n00b
n00b


Joined: 06 Jan 2017
Posts: 53

PostPosted: Mon Dec 16, 2019 6:07 am    Post subject: Reply with quote

NeddySeagoon

I've only been playing with computers for money for about 20 years. I have much to learn!

I build the kernel and initrd with genkernel. Like most things, when there's a lot of moving parts, I tend to wrap them up in a script so I don't have to remember all of them, so here's my `rebuild-kernel` script:

Code:

#!/bin/bash
SAKAKI_BUILDKERNEL=/usr/sbin/buildkernel
if test -e $SAKAKI_BUILDKERNEL; then
    echo "Using Sakaki's buildkernel..."
    $SAKAKI_BUILDKERNEL
    exit $?
fi
RUNNING_CONFIG=/proc/config.gz
KERNEL_CONFIG=/usr/src/linux/.config

RED="\033[1;31m"
GREEN="\033[1;32m"
YELLOW="\033[1;33m"
RESET="\033[0m"

function red()
{
  echo -e "${RED}$*${RESET}"
}

function green()
{
  echo -e "${GREEN}$*${RESET}"
}

function yellow()
{
  echo -e "${YELLOW}$*${RESET}"
}

function find_latest_kernel() {
  eselect kernel list | tail -n 1 | sed -e 's/^.*linux-//' | sed -e 's/\s\\*.*//'
}

function find_selected_kernel() {
  eselect kernel list | grep "\*$" | sed -e 's/^.*linux-//' | sed -e 's/\s\\*.*//'
}

function find_running_kernel() {
  uname -r
}

function create_config_if_missing() {
  if test -e $KERNEL_CONFIG; then
    echo "$(yellow using config at $KERNEL_CONFIG)"
  else
    ZCAT="$(which zcat)"
    if test -z "$ZCAT"; then
      echo "No zcat in path"
      exit 3
    fi
    if test -e $RUNNING_CONFIG; then
      echo "$(green "creating $KERNEL_CONFIG from current running kernel")"
      $ZCAT $RUNNING_CONFIG > $KERNEL_CONFIG
    else
      echo "$(red "unable to generate new $KERNEL_CONFIG: current kernel does not support $RUNNING_CONFIG")"
      exit 2
    fi
  fi
}

function validate_selected_kernel() {
  LATEST=$(find_latest_kernel)
  ESELECTED=$(find_selected_kernel)
  RUNNING=$(uname -r)
  if test "$ESELECTED" != "$LATEST"; then
    echo "The current selected kernel is $(red ${ESELECTED}), however $(green ${LATEST}) is available."
    echo -n " Select newer kernel? (Y/n) "
    read ANSWER
    ANSWER=$(echo $ANSWER | tr [:upper:] [:lower:])
    if test -z "$ANSWER" ||
       test "$ANSWER" = "y" ||
       test "$ANSWER" = "yes"; then
      eselect kernel set $LATEST
      ESELECTED=$(find_selected_kernel)

      if test "$ESELECTED" != "$LATEST"; then
        echo "kernel selection fails; please manually run eselect"
        exit 1
      fi
    else
      echo "Continuing rebuild with $ESELECTED"
    fi
  fi
}

function build_with_genkernel() {
  if ! nice -n 19 ionice -c idle genkernel all --microcode --kernel-config=/usr/src/linux/.config --oldconfig --makeopts=-j$(nproc) --loglevel=2; then
    STATUS=$?
    echo "genkernel exits with status $STATUS; modules not rebuilt!"
    exit $STATUS
  fi

  if test "$SKIP_MODULE_REBUILD" = "n"; then
    if ! emerge @module-rebuild; then
      STATUS=$?
      echo "module rebuild exits with status $STATUS"
      exit $STATUS
    fi
  else
    yellow "Skipping @module-rebuild"
  fi
}

function update_grub() {
  grub-mkconfig -o /boot/grub/grub.cfg
}

function grok_args() {
  SKIP_MODULE_REBUILD="n"
  while test ! -z "$1"; do
    if test "$1" = "--no-module-rebuild" || test "$1" = "-n"; then
      SKIP_MODULE_REBUILD="y"
    fi
    shift
  done
}

function remind_reboot() {
  if test "$ESELECTED" != "$RUNNING"; then
    echo "Remember to reboot to switch from $(red ${RUNNING}) to $(green ${ESELECTED})"
  fi
}

grok_args
validate_selected_kernel
create_config_if_missing
build_with_genkernel
update_grub
remind_reboot


most of it is, as I'm sure you can see, boilerplate-type stuff to help me not to make a mistake like building the wrong kernel or forgetting to reboot or forgetting to rebuild 3rd-party kernel modules or forgetting to update the grub config. It's worked pretty well for the last two years or so. Currently, although I've been runing a -ck kernel for most of the time I've been using Gentoo, I'm using gentoo-sources (5.4.3), built with this script. And I was so embarrassed with the original code that I've been forced to refactor so I can post it here (:

I see that my older -ck kernel and newer gentoo kernel both also have the watchdog settings, but I'm interested to figure out the problem, so I'm rebuilding 5.4.2-ck with
Code:

# CONFIG_WATCHDOG is not set


as per your suggestion, but no change in behavior ):
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum