Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
[WORKAROUND] Weird behavior when booting with root device on
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware
View previous topic :: View next topic  
Author Message
pa4wdh
Guru
Guru


Joined: 16 Dec 2005
Posts: 323

PostPosted: Tue May 22, 2018 9:39 pm    Post subject: [WORKAROUND] Weird behavior when booting with root device on Reply with quote

Hi All,

I'm using a Soekris net5501 as a router, and of course it runs gentoo :-)
For storage there is an internal CF card, detected as sda by the kernel, and an external USB HDD containing the full OS usually detected as sdb.
At boot grub is loaded from the CF, loads the kernel from the CF, and the kernel should mount the USB HDD as root. There usually is a delay between the kernel boot and detection of the USB disk, so i'm using the "rootdelay" parameter to instruct the kernel to wait a few seconds before actually trying to mount the root device.
This setup has worked since i got the device (somewhere 2008 i think), until at least kernel 4.1.15.

Now i'm trying to update the kernel to 4.9.95, and after sorting out some USB driver issues I now have a strange situation at boot: Everything works, up to the point where the usb-storage driver logs the detection of a new device (which is my external HDD). If i don't do anything, the kernel will panic because it can't find it's root device. However, if i press enter on the serial console within a few seconds (probably within the rootdelay time), the disk is found by the SD driver, partitions become available and booting continues. It still seems slower than usual.

The timing in dmesg shows what i mean:
Code:

(.... regular boot messages ....)
[    1.229684] rtc_cmos rtc_cmos: setting system clock to 2018-05-22 21:11:22 UTC (1527023482)
[    1.232639] Waiting 5 sec before mounting root device...
[    1.249375] usb 1-1: New USB device found, idVendor=13fd, idProduct=1840
[    1.250329] usb 1-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[    1.257224] usb 1-1: Product: External       
[    1.259988] usb 1-1: Manufacturer: Generic
[    1.265225] usb 1-1: SerialNumber: 53304B584A31355031303635
[    1.267719] usb-storage 1-1:1.0: USB Mass Storage device detected
[    1.272777] scsi host2: usb-storage 1-1:1.0
[    1.313021] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0xe696d18e8b, max_idle_ns: 881590506359 ns
[    1.316512] clocksource: Switched to clocksource tsc

*** This is where i have to press enter on serial console ****

[    4.241560] scsi 2:0:0:0: Direct-Access     Generic  External         1.02 PQ: 0 ANSI: 4
[    4.291545] sd 2:0:0:0: Attached scsi generic sg1 type 0
[    4.324184] sd 2:0:0:0: [sdb] 312581808 512-byte logical blocks: (160 GB/149 GiB)
[    4.370285] sd 2:0:0:0: [sdb] Write Protect is off
[    4.399070] sd 2:0:0:0: [sdb] Mode Sense: 21 00 00 00
[    4.400154] sd 2:0:0:0: [sdb] No Caching mode page found
[    4.432085] sd 2:0:0:0: [sdb] Assuming drive cache: write through
[    5.075711]  sdb: sdb1 sdb2 sdb3 sdb4 < sdb5 sdb6 >
[    5.110373] sd 2:0:0:0: [sdb] Attached SCSI disk

*** Notice the 10 second delay ***

[   16.310982] VFS: Mounted root (ext2 filesystem) readonly on device 8:17.
[   16.365992] devtmpfs: mounted
[   16.384728] Freeing unused kernel memory: 316K
[   16.411941] Write protecting the kernel text: 3228k
[   16.441352] Write protecting the kernel read-only data: 888k

*** At this point init is started, and somehow waits a long time ***

[   33.246749] random: crng init done

*** Regular boot (openrc) starts, notice the delay ***

[  139.726177] cgroup: cgroup2: unknown option "nsdelegate"
[  153.679820] Adding 524284k swap on /dev/mapper/vghdd-swap.  Priority:-1 extents:1 across:524284k


Any ideas on how to troubleshoot this?

Thanks in advance.
_________________
The gentoo way of bringing peace to the world:
USE="-war" emerge --newuse @world

Free as in Freedom is not limited to software only:
Music: http://www.jamendo.com
Recipes: http://www.opensourcefood.com


Last edited by pa4wdh on Mon Aug 27, 2018 7:53 am; edited 1 time in total
Back to top
View user's profile Send private message
pa4wdh
Guru
Guru


Joined: 16 Dec 2005
Posts: 323

PostPosted: Tue May 22, 2018 10:03 pm    Post subject: Reply with quote

After some more tests I got some extra info.

Instead of booting with root on the USB device, i booted with the same kernel and a ramdisk image containing a minimal system.
Now when the kernel has booted and is loading the ramdisk image, the USB disk gets detected. After the ramdisk is mounted as root and init is started, again i get some weird behavior:
First, the only output on the serial console is:
INIT:
Then i have to press enter 3 times, before init shows the rest of the line (containing the version number) and starts openrc. At the point where init switches to runlevel 3, i again have to press enter a few times before it continues.
The same image but with kernel 4.1.15 boots without problems.

My impression is that this is some kind of interrupt problem, but i can't pinpoint it yet.
_________________
The gentoo way of bringing peace to the world:
USE="-war" emerge --newuse @world

Free as in Freedom is not limited to software only:
Music: http://www.jamendo.com
Recipes: http://www.opensourcefood.com
Back to top
View user's profile Send private message
pa4wdh
Guru
Guru


Joined: 16 Dec 2005
Posts: 323

PostPosted: Wed May 23, 2018 8:38 am    Post subject: Reply with quote

Some more troubleshooting learned me that this is not an interrupt problem. Both serial ports have a dedicated interrupt line, not shared with anything else, also the USB controllers have their own interrupt lines.

It now seems to be a serial console problem. I booted with console=/dev/null instead of console=ttyS0,19200n8 and this booted without any problem (with the obvious difference that i couldn't see the boot process :-) ). I tried using hardware flowcontrol (with console=ttyS0,19200n8r but this didn't make a difference.
_________________
The gentoo way of bringing peace to the world:
USE="-war" emerge --newuse @world

Free as in Freedom is not limited to software only:
Music: http://www.jamendo.com
Recipes: http://www.opensourcefood.com
Back to top
View user's profile Send private message
pa4wdh
Guru
Guru


Joined: 16 Dec 2005
Posts: 323

PostPosted: Mon Aug 27, 2018 7:51 am    Post subject: Reply with quote

After a lot of troubleshooting it turned out to be a problem with the clocksource of the kernel. I haven't really solved it, but i have a workaround that works for me.

For some reason the tsc clocksource gives problems during boot with newer kernels. The only noticable difference between a working and non working kernel is the order in which things happen. With older kernels the tsc clocksource is initialized early during boot and the clocksource is switched to tsc. With recent kernels initializing tsc is one of the last things the kernel does before it frees unused memory and starts init.

As a workaround i've added clocksource=pit to the kernel commandline, this makes sure the boot process works the way it should. However, pit's timer resolution is lower than tsc's timer resolution, so switching to tsc is still preferred. I've done this by using /etc/local.d to perform:
echo tsc > /sys/devices/system/clocksource/clocksource0/current_clocksource

Again, it's not a solution, just a workaround, any hints to get tsc going right from kernel boot are more than welcome
_________________
The gentoo way of bringing peace to the world:
USE="-war" emerge --newuse @world

Free as in Freedom is not limited to software only:
Music: http://www.jamendo.com
Recipes: http://www.opensourcefood.com
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum