Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Dedicated server unbootable after update
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Installing Gentoo
View previous topic :: View next topic  
Author Message
Cygon
Tux's lil' helper
Tux's lil' helper


Joined: 05 Feb 2006
Posts: 89
Location: Germany

PostPosted: Wed Feb 26, 2020 2:09 pm    Post subject: Dedicated server unbootable after update Reply with quote

Hi!

I'm running Gentoo on a headless dedicated server. All was fine for about 3 years. Yesterday, I did the usual portage updates and today rebooted the server (as I've did many times in the past). It didn't come up again.

I can remotely boot the server into a "rescue system" (some old network-booted Debian), mount my Gentoo partition and look around.

  • /var/log/dmesg is current and shows the kernel still starts up
  • /var/log/messages wasn't touched at all
  • wtmp is current and lastlog shows that the 'reboot' command as 'still running'

So the kernel loads, has access to the root partition and then...? maybe openrc doesn't start?

How can I figure out what the issue is? I'mthankful even for ideas on what else I could check.
Back to top
View user's profile Send private message
Jaglover
Watchman
Watchman


Joined: 29 May 2005
Posts: 7653
Location: Saint Amant, Acadiana

PostPosted: Wed Feb 26, 2020 2:35 pm    Post subject: Reply with quote

Check this out: https://forums.gentoo.org/viewtopic-t-1081710-start-0.html
_________________
Please learn how to denote units correctly!
Back to top
View user's profile Send private message
Cygon
Tux's lil' helper
Tux's lil' helper


Joined: 05 Feb 2006
Posts: 89
Location: Germany

PostPosted: Wed Feb 26, 2020 4:17 pm    Post subject: Reply with quote

Thanks, but it doesn't look like my kernel is getting hung up waiting for entropy for its RNG.

And I had given it ~5 hours to boot already this morning, in case it's file system check time or such.

Code:
[...]
[    2.143738] random: fast init done
[    2.184706] EXT4-fs (md3): recovery complete
[    2.186096] EXT4-fs (md3): mounted filesystem with ordered data mode. Opts: (null)
[    2.186174] VFS: Mounted root (ext4 filesystem) readonly on device 9:3.
[    2.186688] devtmpfs: mounted
[    2.187836] Freeing unused kernel image memory: 1080K
[    2.213402] Write protecting the kernel read-only data: 14336k
[    2.214256] Freeing unused kernel image memory: 2012K
[    2.214360] Freeing unused kernel image memory: 12K
[    2.214417] rodata_test: all tests were successful
[    2.214466] Run /sbin/init as init process
[    2.618251] random: lvm: uninitialized urandom read (4 bytes read)
[    2.627233] random: lvm: uninitialized urandom read (4 bytes read)
[    2.669361] tsc: Refined TSC clocksource calibration: 2300.112 MHz
[    2.669423] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x21279eddaa4, max_idle_ns: 440795271862 ns
[    2.669585] clocksource: Switched to clocksource tsc
[    3.395204] udevd[1566]: starting version 3.2.9
[    3.396977] random: udevd: uninitialized urandom read (16 bytes read)
[    3.406499] udevd[1566]: starting eudev-3.2.9
[    3.473841] r8169 0000:02:00.0 wan0: renamed from eth0
[    3.494200]  md1:
[   13.792571]  md1:
[   13.841501] EXT4-fs (md3): re-mounted. Opts: (null)
[   13.944204] Adding 33554428k swap on /dev/sda2.  Priority:0 extents:1 across:33554428k SS
[   13.948708] Adding 33554428k swap on /dev/sdb2.  Priority:0 extents:1 across:33554428k SS


Just to be safe, I chrooted into my system, installed rng-tools and rc-update add'ed it to my boot runlevel, but no change when attempting to normally boot the system.

The RNG related stuff in the boot messages checks out, imho:

Code:
# cat /var/log/dmesg | grep -i random
[    0.000000] random: get_random_u32 called from bsp_init_amd+0x163/0x290 with crng_init=0
[    2.143738] random: fast init done
[    2.618251] random: lvm: uninitialized urandom read (4 bytes read)
[    2.627233] random: lvm: uninitialized urandom read (4 bytes read)
[    3.396977] random: udevd: uninitialized urandom read (16 bytes read)


  • I do see "[ 2.214466] Run /sbin/init as init process" in there - so it's starting (or attempting to start) OpenRC.
  • and "[ 3.473841] r8169 0000:02:00.0 wan0: renamed from eth0" means it's reading '/etc/udev/rules.d/70-persistent-net.rules'
  • but no indication that it's bringing up the wan0 interface, so its init script isn't being run.

Something must be blocking/crashing very early in the OpenRC init because /var/log/messages isn't being written and I can't find any other files with recent modifications in /var/log.
Back to top
View user's profile Send private message
Jaglover
Watchman
Watchman


Joined: 29 May 2005
Posts: 7653
Location: Saint Amant, Acadiana

PostPosted: Wed Feb 26, 2020 8:20 pm    Post subject: Reply with quote

You could make a custom minimalist runlevel and if you manage to boot into it, start troubleshooting.
_________________
Please learn how to denote units correctly!
Back to top
View user's profile Send private message
Cygon
Tux's lil' helper
Tux's lil' helper


Joined: 05 Feb 2006
Posts: 89
Location: Germany

PostPosted: Wed Feb 26, 2020 9:48 pm    Post subject: Reply with quote

I picked a somewhat more drastic solution in the meantime.

'moved the entire root filesystem into a subdirectory, extracted a fresh stage3 tarball into root, then followed the Gentoo Installation Handbook once again - voila, bootable server :)

Setting up apache, mysql, courier, openvpn, subversion and everything else will be a bit of work, but at least this will definitely get me to my goal within finite time.


Thanks for the tips, if I can I'll run a diff over the relevant configuration, maybe I can post-mortem see what might have caused the issue. But most likely I'll never know...
Back to top
View user's profile Send private message
DONAHUE
Watchman
Watchman


Joined: 09 Dec 2006
Posts: 7602
Location: Goose Creek SC

PostPosted: Wed Feb 26, 2020 11:27 pm    Post subject: Reply with quote

/var/log/rc.log is the default location for openrc to write a log.
You may have made changes to /etc/rc.conf to change the path or shut down rc_logger ...
Might be wise to have rc_logger running as you restore system in case problem recurs. Log may have retained information if it was running.

rc.conf wrote:
# rc_logger launches a logging daemon to log the entire rc process to
# /var/log/rc.log
# NOTE: Linux systems require the devfs service to be started before
# logging can take place and as such cannot log the sysinit runlevel.
#rc_logger="NO"

# Through rc_log_path you can specify a custom log file.
# The default value is: /var/log/rc.log
#rc_log_path="/var/log/rc.log"

_________________
Defund the FCC.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Installing Gentoo All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum