Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
symlinks in custom initramfs not resolved at boot [solved]
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware
View previous topic :: View next topic  
Author Message
triquetra
Tux's lil' helper
Tux's lil' helper


Joined: 09 Apr 2012
Posts: 93
Location: Kansas, USA

PostPosted: Mon Feb 09, 2015 10:27 pm    Post subject: symlinks in custom initramfs not resolved at boot [solved] Reply with quote

I have been trying to create an initramfs with genkernel-next, but without success, as documented here.

In an attempt to troubleshoot the issue, I tried to create an initramfs with dracut, and even tried multiple permutations of a custom initramfs. None of them worked.

I have a booting system right now only because I am booting with an initramfs created with the old genkernel, but it seems I am unable to create a new functioning initramfs by any means.

I have banged my head against this problem for many hours over several days, and I am at a loss to figure out what I may be doing wrong. Any ideas?


Last edited by triquetra on Mon Feb 23, 2015 7:26 pm; edited 2 times in total
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 43209
Location: 56N 3W

PostPosted: Mon Feb 09, 2015 10:52 pm    Post subject: Reply with quote

triquetra,

Luckily systemd has nothing to do with an initramfs, or I would not be posting here.

When the black magic of automatic initramfs lets you down, you ned to fall back on the manual controls.
About the only way te debug what is happening, is to sprinkle the init script in the initfamfs with commands to show mhats happening and pauses to let you read the console.

Please post the init script from your initrd and I'll suggest some debug additions.
If you made your own directory structure, that would be useful too.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
triquetra
Tux's lil' helper
Tux's lil' helper


Joined: 09 Apr 2012
Posts: 93
Location: Kansas, USA

PostPosted: Mon Feb 09, 2015 11:14 pm    Post subject: Reply with quote

Following is what my init script currently looks like, though this is one of a couple dozen different permutations I have tried.

init:
#!/bin/busybox sh
 
rescue_shell() {
    echo $1
    busybox --install -s
    exec /bin/sh
}

# Mount the /proc and /sys filesystems.
mount -t devtmpfs none /dev 
mount -t proc none /proc
mount -t sysfs none /sys

# Assemble RAID:
( sleep 2
  mdadm --assemble --scan
  sleep 2
) &
wait

# Mount the root filesystem.
mount -o ro `findfs LABEL=gentoo` || rescue_shell "Problem mounting root.  Dropping to shell"
   
# Clean up.
umount /proc
umount /sys
umount /dev
     
# Boot the real thing.
exec switch_root /mnt/root /sbin/init


I have made my own directory structure as follows:

su -p -c "find . -print0 | cpio --null -ov --format=newc | gzip -9 > /boot/initramfs-custom-3.18.2-gentoo":

.
./bin
./bin/busybox
./lib64
./dev
./dev/null
./dev/tty
./dev/console
./dev/md126p5
./root
./proc
./mnt
./mnt/root
./sbin
./sbin/mdadm
./lib
./init
./etc
./etc/mdadm.conf
./sys
7 blocks


Thank you in advance for your help.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 43209
Location: 56N 3W

PostPosted: Tue Feb 10, 2015 6:58 pm    Post subject: Reply with quote

triquetra,

That looks "mostly harmless" and in line with the accepted wisdom.

I had similar pain a few months ago trying to get a remote box up. In the end I spent a few days debugging locally.
What I found was the busybox no longer runs its own applets without a full path and or symlinks.
This means your
Code:
# Mount the /proc and /sys filesystems.
mount -t devtmpfs none /dev
mount -t proc none /proc
mount -t sysfs none /sys
all fail, then its steadily downhill from there.

Its not until you get to
Code:
|| rescue_shell "Problem mounting root.  Dropping to shell"
that the symlinks have a chance to get written. That may fail too as busybox is at /bin/busybox and no PATH is set.

Heres a few snippits of my init. Notice the full paths.

Code:
#!/bin/busybox sh
rescue_shell() {
    echo "$@"
    echo "Something went wrong. Dropping you to a shell."
# have time to read the message
    /bin/sleep 20
    /bin/busybox --install -s
    exec /bin/sh
}

and the commented out debug
Code:
#echo "/bin"
#/bin/ls -l /bin
#echo "/sbin"
#/bin/ls -l /sbin

#/bin/sleep 20

#echo "Mount proc"
/bin/mount -t proc proc /proc
#/bin/busybox sleep 20


/bin/ls is a symlink to /bin/busybox that is created elsewhere, likewise /bin/sleep.
You will need symlinks for all of the busybox applets you want to use.
You can either set the PATH or use full path names. Setting the PATH was something I got wrong once, so I just left it in a mess.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
triquetra
Tux's lil' helper
Tux's lil' helper


Joined: 09 Apr 2012
Posts: 93
Location: Kansas, USA

PostPosted: Wed Feb 11, 2015 3:32 am    Post subject: Reply with quote

Hmmm ... no dice.

Here's my current init script:

init:
#!/bin/busybox sh
 
# Create applets
/bin/busybox --install -s

pause() {
    read -p "$*"
}

rescue_shell() {
    /bin/echo $1
    /bin/busybox --install -s
    exec /bin/sh
}

# Mount the /proc and /sys filesystems.
/bin/mount -t devtmpfs none /dev 
/bin/mount -t proc none /proc
/bin/mount -t sysfs none /sys
#echo /sbin/mdev > /proc/sys/kernel/hotplug
#mdev -s

# Assemble RAID:
( sleep 2
  /sbin/mdadm --assemble --scan
  sleep 2
) &
wait
pause "raid array just assembled.  Press Enter to continue"
# Mount the root filesystem.
/bin/mount -o ro `findfs LABEL=gentoo` || rescue_shell "Problem mounting root.  Dropping to shell"
   
# Clean up.
/bin/umount /proc
/bin/umount /sys
/bin/umount /dev
     
# Boot the real thing.
exec /sbin/switch_root /mnt/root /sbin/init


And here's what it gives me.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 43209
Location: 56N 3W

PostPosted: Wed Feb 11, 2015 9:51 pm    Post subject: Reply with quote

triquetra,

I don't see whats happening with raid assembly but /dev/sdb unknown partition table looks odd.
I can't see what it says about /dev/sda

This suggests that you are using either fakeraid in raid0 across sda and sdb, in which case you only get a partition table on sda, or that you are using a partition table that is not supported by your kernel.

The kernel can see the HDD now, so thats progress.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
triquetra
Tux's lil' helper
Tux's lil' helper


Joined: 09 Apr 2012
Posts: 93
Location: Kansas, USA

PostPosted: Wed Feb 11, 2015 11:13 pm    Post subject: Reply with quote

Yeah, some of the text is missing, and it scrolls by too fast for me to get it. Because of the stage that bootup is failing, I presume that there is no text file for me to obtain the missing text, right?

I am using mdadm with a fakeraid raid0 setup across sda and sdb. When the computer boots (with my old initramfs created with the old genkernel), it mounts at /dev/imsm0 (or something similar). Why would I get a partition table on sdb with genkernel initramfs, but not with the custom initramfs?

I'll try to post the genkernel-next boot failure screen for comparison a bit later.
Back to top
View user's profile Send private message
triquetra
Tux's lil' helper
Tux's lil' helper


Joined: 09 Apr 2012
Posts: 93
Location: Kansas, USA

PostPosted: Thu Feb 12, 2015 1:32 am    Post subject: Reply with quote

The change in my custom init script did not change the result. The kernel panic screen looks nearly identical, if not exactly so, in comparison to my previous custom initramfs.

Here's a pic of the genkernel-next failure for comparison. That seems to get farther along than the custom initramfs, but still no boot.
Back to top
View user's profile Send private message
triquetra
Tux's lil' helper
Tux's lil' helper


Joined: 09 Apr 2012
Posts: 93
Location: Kansas, USA

PostPosted: Thu Feb 19, 2015 12:22 am    Post subject: Reply with quote

Still no progress here. Any more ideas?
Back to top
View user's profile Send private message
s0be
Apprentice
Apprentice


Joined: 23 Nov 2002
Posts: 239

PostPosted: Fri Feb 20, 2015 7:41 am    Post subject: Reply with quote

Is there any chance you're missing the devicemapper or raid modules in your kernel? If you compiled them as modules and put them in your init.d, you would probably need to manually insmod them.
Back to top
View user's profile Send private message
triquetra
Tux's lil' helper
Tux's lil' helper


Joined: 09 Apr 2012
Posts: 93
Location: Kansas, USA

PostPosted: Fri Feb 20, 2015 4:20 pm    Post subject: Reply with quote

Devicemapper and raid modules are built into the kernel.
Back to top
View user's profile Send private message
juggling_ben
n00b
n00b


Joined: 25 Nov 2007
Posts: 15

PostPosted: Sun Feb 22, 2015 2:54 am    Post subject: Reply with quote

Few things:

Can you post a ls -la from inside your init directory structure? Your image shows an error code of -40 when running init, which according to my reading of the kernel code, means it can't determine the binary format of /init

Also, in your listing of your initramfs, you seem to be missing any .so files. Are you compiling the tools with the 'static' use flag? Particularly busybox.

I'd be happy to share by init script and my little build script if you'd like. It's only slightly more complicated than yours (after assembling the raid, I have to decrypt, do lvm stuff, etc) but hasn't failed me in quite some time.
Back to top
View user's profile Send private message
s0be
Apprentice
Apprentice


Joined: 23 Nov 2002
Posts: 239

PostPosted: Sun Feb 22, 2015 7:36 pm    Post subject: Reply with quote

Comparing your initrd to mine (ignoring the structural differences), you don't specify a mount point when you mount the gentoo named partition:

Code:

/bin/mount -o ro `findfs LABEL=gentoo` || rescue_shell "Problem mounting root.  Dropping to shell"


vs

Code:

mount -o ro $(findfs "$dev") /mnt/root >>/tmp/initrdlog 2>&1


in mine

You can see my initrd stuff at:

https://github.com/s0be/modular_initrd/blob/master/init
Back to top
View user's profile Send private message
triquetra
Tux's lil' helper
Tux's lil' helper


Joined: 09 Apr 2012
Posts: 93
Location: Kansas, USA

PostPosted: Mon Feb 23, 2015 12:17 am    Post subject: Reply with quote

juggling_ben wrote:
Few things:

Can you post a ls -la from inside your init directory structure? Your image shows an error code of -40 when running init, which according to my reading of the kernel code, means it can't determine the binary format of /init

Also, in your listing of your initramfs, you seem to be missing any .so files. Are you compiling the tools with the 'static' use flag? Particularly busybox.

I'd be happy to share by init script and my little build script if you'd like. It's only slightly more complicated than yours (after assembling the raid, I have to decrypt, do lvm stuff, etc) but hasn't failed me in quite some time.


ls -la:
total 56K
drwxr-xr-x 13 root root 4.0K Feb 10 16:01 .
drwxr-xr-x 11 root root 4.0K Feb 20 10:14 ..
drwxr-xr-x  2 root root 4.0K Feb  8 16:32 bin
drwxr-xr-x  2 root root 4.0K Feb  9 14:16 dev
drwxr-xr-x  2 root root 4.0K Feb  9 13:56 etc
drwxr-xr-x  2 root root 4.0K Feb  8 16:27 lib
drwxr-xr-x  2 root root 4.0K Feb  8 16:27 lib64
drwxr-xr-x  3 root root 4.0K Feb  8 16:27 mnt
drwxr-xr-x  2 root root 4.0K Feb  8 16:27 proc
drwxr-xr-x  2 root root 4.0K Feb  8 16:27 root
drwxr-xr-x  2 root root 4.0K Feb  9 14:15 sbin
drwxr-xr-x  2 root root 4.0K Feb  8 16:27 sys
drwxr-xr-x  4 root root 4.0K Feb 10 16:01 usr
-rwxr-xr-x  1 root root  786 Feb 22 18:03 init


equery u busybox:
 * Found these USE flags for sys-apps/busybox-1.23.1-r1:
 U I
 - - debug         : Enable extra debug codepaths, like asserts and extra output. If you want to get meaningful backtraces see http://www.gentoo.org/proj/en/qa/backtraces.xml
 + + ipv6          : Add support for IP version 6
 - - livecd        : !!internal use only!! DO NOT SET THIS FLAG YOURSELF!, used during livecd building
 - - make-symlinks : Create all the appropriate symlinks in /bin and /sbin.
 - - math          : Enable math support in gawk (requires libm)
 - - mdev          : Create the appropriate symlink in /sbin and install mdev.conf and support files
 + + pam           : Add support for PAM (Pluggable Authentication Modules) - DANGEROUS to arbitrarily flip
 - - savedconfig   : Use this to restore your config from /etc/portage/savedconfig ${CATEGORY}/${PN}. Make sure your USE flags allow for appropriate dependencies
 - - sep-usr       : Support a separate /usr without needing an initramfs by booting with init=/ginit
 + + static        : Make the system rescue shell (/bin/bb) static so you can recover even when glibc is broken
 - - syslog        : Enable support for syslog
 + + systemd       : Support systemd


equery u mdadm:
 * Found these USE flags for sys-fs/mdadm-3.3.2:
 U I
 + + static : !!do not set this during bootstrap!! Causes binaries to be statically linked instead of dynamically


Please share your init and build script.
Back to top
View user's profile Send private message
triquetra
Tux's lil' helper
Tux's lil' helper


Joined: 09 Apr 2012
Posts: 93
Location: Kansas, USA

PostPosted: Mon Feb 23, 2015 12:31 am    Post subject: Reply with quote

@s0be

Thanks for the reply, and sharing your initramfs. I do usually find that problems I have with gentoo are because I've overlooked something simple, but specifying the mount point on the mount line was not, unfortunately, the problem (or at least not the primary one). I changed my init to specify the mount point of /mnt/root, recompiled the initramfs, and rebooted, but got the same kernel panic.
Back to top
View user's profile Send private message
s0be
Apprentice
Apprentice


Joined: 23 Nov 2002
Posts: 239

PostPosted: Mon Feb 23, 2015 3:01 am    Post subject: Reply with quote

your error code (-40) looks like it could be:

#define ELOOP 40 /* Too many symbolic links encountered */

So I bet you have some broken symbolic links once you consider that the top level of your init tree becomes / in initramfs.

The output of:
Code:

for f in $(find -type l); do ls -l $f; done;


Will show what your symlinks are. They should all be 'relative' so that they work from both initramfs and when you're editing it.

Code:

s0be@anvil /usr/src/initramfs $ for f in $(find -type l); do ls -l $f; done;
lrwxrwxrwx 1 s0be s0be 3 Oct 11  2012 ./sbin -> bin
lrwxrwxrwx 1 s0be s0be 8 Oct 11  2012 ./usr/sbin -> ../sbin/
lrwxrwxrwx 1 s0be s0be 6 Oct 11  2012 ./usr/bin -> ../bin
Back to top
View user's profile Send private message
juggling_ben
n00b
n00b


Joined: 25 Nov 2007
Posts: 15

PostPosted: Mon Feb 23, 2015 3:40 am    Post subject: Reply with quote

So thinking about this a bit, the problem isn't related to your init script. Your boot process isn't getting that far.

triquetra wrote:
Following is what my init script currently looks like, though this is one of a couple dozen different permutations I have tried.
su -p -c "find . -print0 | cpio --null -ov --format=newc | gzip -9 > /boot/initramfs-custom-3.18.2-gentoo":

.
./bin
./bin/busybox
./lib64
./dev
./dev/null
./dev/tty
./dev/console
./dev/md126p5
./root
./proc
./mnt
./mnt/root
./sbin
./sbin/mdadm
./lib
./init
./etc
./etc/mdadm.conf
./sys
7 blocks



7 blocks? That does not sound right to me. If I only include statically-linked busybox in my own, it is > 5000 blocks. What is bin/busybox in your initramfs? (ie what does "file bin/busybox" say?) If you copied a symlink or something instead, it would fit with my earlier confusion about the 'error -40'
Back to top
View user's profile Send private message
s0be
Apprentice
Apprentice


Joined: 23 Nov 2002
Posts: 239

PostPosted: Mon Feb 23, 2015 3:52 am    Post subject: Reply with quote

juggling_ben wrote:
So thinking about this a bit, the problem isn't related to your init script. Your boot process isn't getting that far.

triquetra wrote:
Following is what my init script currently looks like, though this is one of a couple dozen different permutations I have tried.
su -p -c "find . -print0 | cpio --null -ov --format=newc | gzip -9 > /boot/initramfs-custom-3.18.2-gentoo":

.
./bin
./bin/busybox
./lib64
./dev
./dev/null
./dev/tty
./dev/console
./dev/md126p5
./root
./proc
./mnt
./mnt/root
./sbin
./sbin/mdadm
./lib
./init
./etc
./etc/mdadm.conf
./sys
7 blocks



7 blocks? That does not sound right to me. If I only include statically-linked busybox in my own, it is > 5000 blocks. What is bin/busybox in your initramfs? (ie what does "file bin/busybox" say?) If you copied a symlink or something instead, it would fit with my earlier confusion about the 'error -40'


That makes a lot of sense, I bet $INITRDPATH/bin/busybox is a symlink to /bin/busybox. That'd lead the the error -40 (ELOOP) as it'd end up being a symlink to itself, and it would give him the tiny initrd rather than one of appreciable size.
Back to top
View user's profile Send private message
triquetra
Tux's lil' helper
Tux's lil' helper


Joined: 09 Apr 2012
Posts: 93
Location: Kansas, USA

PostPosted: Mon Feb 23, 2015 7:24 pm    Post subject: Reply with quote

Yes, that was it. Thanks for the help debugging this issue. I thought I would be clever and symlink the files needed for the initramfs so that I was sure to always have the current version. Obviously, that did not work. Would hard links accomplish that goal, or is it better to just use a mkinitramfs script to build the initramfs files and directory structure each time I rebuild the initramfs?
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 43209
Location: 56N 3W

PostPosted: Mon Feb 23, 2015 7:31 pm    Post subject: Reply with quote

triquetra,

Provided that there are no kernel modules in the initrd, its just like fimware.
Why would you ever rebuild it?

Its a closed system that does some stuff and gets abandoned once it done.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
triquetra
Tux's lil' helper
Tux's lil' helper


Joined: 09 Apr 2012
Posts: 93
Location: Kansas, USA

PostPosted: Mon Feb 23, 2015 7:53 pm    Post subject: Reply with quote

@NeddySeagoon

Good question. I had just gotten into the habit (when using genkernel) of building a new initramfs every time I upgraded my kernel.
Back to top
View user's profile Send private message
s0be
Apprentice
Apprentice


Joined: 23 Nov 2002
Posts: 239

PostPosted: Mon Feb 23, 2015 9:01 pm    Post subject: Reply with quote

triquetra wrote:
Yes, that was it. Thanks for the help debugging this issue. I thought I would be clever and symlink the files needed for the initramfs so that I was sure to always have the current version. Obviously, that did not work. Would hard links accomplish that goal, or is it better to just use a mkinitramfs script to build the initramfs files and directory structure each time I rebuild the initramfs?


When it comes to initramfs, etc, making changes as rarely as possible is better, IMHO. You don't always want the 'latest and greatest' versions in it, because they change behaviors and whatnot, and pre-init is a very delicate time. I haven't upgraded the busybox in my initramfs since May 14, 2012. If it works, don't replace it ;-).
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 13855

PostPosted: Mon Feb 23, 2015 11:24 pm    Post subject: Reply with quote

Other posters make a good argument for why not to rebuild your initramfs. If you decide you want to rebuild it regularly, I recommend letting the kernel build system create it and embed it into the kernel. This has the disadvantage of keeping one copy of the initramfs for each kernel you build, but the advantage that each kernel is wholly self-contained and has an initramfs loaded even if you use a bootloader that does not support loading an initramfs.
Back to top
View user's profile Send private message
s0be
Apprentice
Apprentice


Joined: 23 Nov 2002
Posts: 239

PostPosted: Mon Feb 23, 2015 11:44 pm    Post subject: Reply with quote

I didn't want to suggest embedding it directly..... but....

It's exactly what I do. There's a problem with rebuilding kernels with embedded initramfs, here's the patch I've been using for years:

Code:

diff --git a/usr/Makefile b/usr/Makefile
index e767f01..041333b 100644
--- a/usr/Makefile
+++ b/usr/Makefile
@@ -60,15 +60,7 @@ targets := initramfs_data.cpio.gz initramfs_data.cpio.bz2 \
        initramfs_data.cpio.lzma initramfs_data.cpio.xz \
        initramfs_data.cpio.lzo initramfs_data.cpio.lz4 \
        initramfs_data.cpio
-# do not try to update files included in initramfs
-$(deps_initramfs): ;
-
-$(deps_initramfs): klibcdirs
-# We rebuild initramfs_data.cpio if:
-# 1) Any included file is newer then initramfs_data.cpio
-# 2) There are changes in which files are included (added or deleted)
-# 3) If gen_init_cpio are newer than initramfs_data.cpio
-# 4) arguments to gen_initramfs.sh changes
-$(obj)/initramfs_data.cpio$(suffix_y): $(obj)/gen_init_cpio $(deps_initramfs) klibcdirs
+
+$(obj)/initramfs_data.cpio$(suffix_y): $(obj)/gen_init_cpio
        $(Q)$(initramfs) -l $(ramfs-input) > $(obj)/.initramfs_data.cpio.d
        $(call if_changed,initfs)


to work around the issue.

Here's the error you'll see:
Code:

usr/Makefile:64: *** target pattern contains no '%'.  Stop.
Makefile:964: recipe for target 'usr' failed
make: *** [usr] Error 2


This patch might already be included in gentoo-sources, I built linux-next kernels semi-regularly as I do a lot of dev against it.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum