Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Custom initramfs is not working
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Installing Gentoo
View previous topic :: View next topic  
Author Message
IELeibowitz
n00b
n00b


Joined: 14 Mar 2020
Posts: 19

PostPosted: Fri Apr 10, 2020 1:34 pm    Post subject: Custom initramfs is not working Reply with quote

I have been trying to make a custom initramfs because I cannot find any other solution that accomplishes the boot process I want (described here for Arch's mkinitcpio).

I have copied all of my /lib/modules and /lib/firmware, as well as all of the binaries I need and the libs that they depend on, including busybox, and I've written my init script according to the examples provided in the wiki. My init mounts the /proc, /sys, and /dev filesystems and all of the modules that I was using in the initramfs that genkernel made, but it still isn't doing what I need it to do.

It isn't recognizing any of my hard drive partitions (lsblk and blkid return nothing), lsmod returns a list with incorrect formatting that's very difficult to read, and my Yubikey isn't recognized at all (despite being recognized in my OS and my other initramfs).

I am completely lost here. For some reason I can't find any help on any of this outside of the one page on Gentoo wiki. This would be so much simpler if genkernel would just allow me to customize the init script instead of being entirely automated (and I have tried customizing genkernel's image, but when I recompress it and try to boot it just causes a panic, even if I haven't edited any scripts).

Edit:
Here is my init script
Code:

#! /bin/busybox sh

rescue_shell() {
   echo "Something went wrong.  Dropping you into a shell."
   busybox --install -s
   exec /bin/sh
}

# Mount the /dev /proc and /sys filesystems
mount -t devtmpfs none /dev
mount -t proc proc /proc
mount -t sysfs none /sys
echo 0 > /proc/sys/kernel/printk

# Load modules (overkill for now)
modprobe fat
modprobe lockd
modprobe grace
modprobe sunrpc
modprobe fscache
modprobe ext4
modprobe mbcache
modprobe jbd2
modprobe reed_solomon
modprobe raid456
modprobe md_mod
modprobe async_memcpy
modprobe async_pq
modprobe raid6_pq
modprobe dm_region-hash
modprobe dm_log
modprobe async_xor
modprobe async_tx
modprobe async_xor
modprobe dm_crypt
modprobe dm_cache
modprobe dm_persistent_data
modprobe libcrc32c
modprobe dm_bufio
modprobe dm_bio_prison
modprobe dm_mod
modprobe crc_itu_t
modprobe ff_memless
modprobe usb_storage
modprobe xhci_hcd
modprobe pcmcia
modprobe pcmcia_core
modprobe nvmet_fc
modprobe megaraid_mm
modprobe scsi_transport_fc
modprobe raid_class
modprobe scsi_transport_sas
modprobe mptscsih
modprobe mptbase
modprobe scsi_transport_spi
modprobe pata_sis
modprobe libahci
modprobe nvme_fc
modprobe nvmet
modprobe rdma_cm
modprobe iw_cm
modprobe ib_cm
modprobe ib_core
modprobe configfs
modprobe crc_ccitt
modprobe nvme_fabrics
modprobe nvme_core

# Create keyfile with Yubikey
echo "Plug in your key now if you want to use it"
echo -n "enter key password followed by enter: "
read -s PW
ykchalresp -2 $PW | tr -d '\n' > crypto_keyfile
echo
echo "you may unplug your key now"

# Unlock the drive with keyfile
cryptsetup luksOpen $(findfs UUID="d65d2d6d-a34f-44f1-ad01-0f555270eac8") --key-file crypto_keyfile lvm
rm crypto_keyfile

# Set up volumes
lvm vgchange -a y /dev/mapper/vg0

# Mount root
mount -o ro /dev/vg0/root /mnt/root || rescue_shell

# Clean up
umount /proc /sys /dev

# Boot
exec switch_root /mnt/root /sbin/init
Back to top
View user's profile Send private message
Zucca
Veteran
Veteran


Joined: 14 Jun 2007
Posts: 1769
Location: KUUSANKOSKI, Finland

PostPosted: Fri Apr 10, 2020 9:57 pm    Post subject: Re: Custom initramfs is not working Reply with quote

IELeibowitz wrote:
It isn't recognizing any of my hard drive partitions (lsblk and blkid return nothing), lsmod returns a list with incorrect formatting that's very difficult to read, and my Yubikey isn't recognized at all (despite being recognized in my OS and my other initramfs).
Since no one hansn't yet answered... Be sure to mount /sys and /proc into initramfs environment at the very beginning. (Looks like you have.) I think lsblk and blkid need those to be mounted.
Also do you have the libs they need?
Code:
# qlist util-linux | grep -E 'lib(64)?'
/usr/lib64/python3.6/site-packages/libmount/__init__.py
/usr/lib64/python3.6/site-packages/libmount/__pycache__/__init__.cpython-36.pyc
/usr/lib64/python3.6/site-packages/libmount/__pycache__/__init__.cpython-36.opt-1.pyc
/usr/lib64/python3.6/site-packages/libmount/__pycache__/__init__.cpython-36.opt-2.pyc
/usr/lib64/python3.6/site-packages/libmount/pylibmount.so
/usr/lib64/pkgconfig/uuid.pc
/usr/lib64/pkgconfig/blkid.pc
/usr/lib64/pkgconfig/mount.pc
/usr/lib64/pkgconfig/smartcols.pc
/usr/lib64/pkgconfig/fdisk.pc
/usr/lib64/libblkid.so
/usr/lib64/libfdisk.so
/usr/lib64/libmount.so
/usr/lib64/libsmartcols.so
/usr/lib64/libuuid.so
/usr/lib64/libuuid.a
/usr/lib64/libmount.a
/usr/lib64/libsmartcols.a
/usr/lib64/libfdisk.a
/usr/lib64/libblkid.a
/usr/share/man/man3/libblkid.3.gz
/usr/include/libmount/libmount.h
/usr/include/libfdisk/libfdisk.h
/usr/include/libsmartcols/libsmartcols.h
/lib64/libblkid.so.1
/lib64/libfdisk.so.1
/lib64/libmount.so.1
/lib64/libsmartcols.so.1
/lib64/libuuid.so.1
/lib64/libuuid.so.1.3.0
/lib64/libsmartcols.so.1.1.0
/lib64/libmount.so.1.1.0
/lib64/libblkid.so.1.1.0
/lib64/libfdisk.so.1.1.0
And/or make sure sys-apps/util-linux has USE=static-libs.

Also you might want to have busybox mdev to populate your /dev?

If none of the above helped, please see this topic. It might have the answers?
_________________
..: Zucca :..

Code:
ERROR: '--failure' is not an option. Aborting...
Back to top
View user's profile Send private message
etnull
Guru
Guru


Joined: 26 Mar 2019
Posts: 327

PostPosted: Fri Apr 10, 2020 10:25 pm    Post subject: Reply with quote

You should also show how you make your initramfs image (what libs you are adding).
Here is what I put in mine (this is for encrypted install without lvm, but also no busybox, it may look different for you, just sharing what I have)
Code:
cd /lib64
cp -a libtinfow.so.* libtinfo.so.* libreadline.* libc.* libncurses* ld-linux-x86-64.* ld-2.* libc-* libblkid.* libuuid.* librt.* librt-2.* libpthread-* libpthread.so* libmount.* libdevmapper.* libudev.* libm.* libm-2.* libz.so.* libdl.so.* libdl-* libext2fs.so.* libcom_err.so.* libe2p.so.* /usr/src/initramfs/lib64/
cd /usr/lib64
cp -a libtinfot.so libtinfot.so.* libtinfotw.so libtinfotw.so.* libtinfo.so libcryptsetup.so.* libpopt.so.* libgcrypt.so.* libgpg-error.so.* libcrypto.so.* libargon2.so.* libjson-c.so.* libncurses* /usr/src/initramfs/usr/lib64/

#other things (I don't use busybox, apart of its switch_root which I wget separately)
cp -a /bin/sh /usr/src/initramfs/bin/
cp -a /bin/bash /usr/src/initramfs/bin/
cp -a /sbin/findfs /usr/src/initramfs/sbin/
cp -a /sbin/{fsck,fsck.ext2,fsck.ext4} /usr/src/initramfs/sbin/
cp -a /sbin/e2fsck /usr/src/initramfs/sbin/
cp -a /bin/dmesg /usr/src/initramfs/bin/
cp -a /bin/mount /usr/src/initramfs/bin/
cp -a /bin/umount /usr/src/initramfs/bin/
cp -a /bin/echo /usr/src/initramfs/bin/
cp -a /bin/sleep /usr/src/initramfs/bin/
cp -a /usr/bin/clear /usr/src/initramfs/usr/bin/
cp -a /sbin/cryptsetup /usr/src/initramfs/sbin/
cp -a /etc/terminfo/* /usr/src/initramfs/etc/terminfo/
cp -a /usr/share/terminfo/* /usr/src/initramfs/usr/share/terminfo/
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 45383
Location: 56N 3W

PostPosted: Fri Apr 10, 2020 10:44 pm    Post subject: Reply with quote

IELeibowitz,

When you get dropped to the rescue shell, what do you have in /dev ?
Not everything, /dev/null and /dev/console are essential for booting.
You block devices should be there too.

dmesg will tell if there were any problems with the modprobes.
Some of them are key to populating /dev with your block devices.

Scatter some echo statements around for debug.

You will hate this initrd every kernel upgrade as it must be rebuilt for the kernel modules.

Build all the kernel bits into the kernel binary that you need to mount root. Then the initrd becomes just the user space tools and init script.
That way the initrd is kernel agnostic. My initrd from 2009 still works with a 5.6.0 kernel.

Code:
modprobe pcmcia
modprobe pcmcia_core

PCMCIA is long gone. If you nave PCMCIA, you won't need
Code:
modprobe nvme_fc
modprobe nvmet
modprobe nvme_fabrics
modprobe nvme_core
As you won't have a PCIe bus to attach it to.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
IELeibowitz
n00b
n00b


Joined: 14 Mar 2020
Posts: 19

PostPosted: Sat Apr 11, 2020 1:05 am    Post subject: Reply with quote

Zucca:
I copied all of the libs that ldd showed me when I was copying the binaries that I needed. For some reason I get a different output to that command than what you show. Mine looks like this:
Code:
# qlist util-linux | grep -E 'lib(64)?'
/lib64/libuuid.so.1.3.0
/lib64/libblkid.so.1.1.0
/lib64/libmount.so.1.1.0
/lib64/libfdisk.so.1.1.0
/lib64/libsmartcols.so.1.1.0
/lib64/libuuid.so.1
/lib64/libsmartcols.so.1
/lib64/libmount.so.1
/lib64/libfdisk.so.1
/lib64/libblkid.so.1
/usr/include/libsmartcols/libsmartcols.h
/usr/include/libmount/libmount.h
/usr/include/libfdisk/libfdisk.h
/usr/lib64/libuuid.so
/usr/lib64/libsmartcols.so
/usr/lib64/libmount.so
/usr/lib64/libfdisk.so
/usr/lib64/libblkid.so
/usr/lib64/pkgconfig/fdisk.pc
/usr/lib64/pkgconfig/smartcols.pc
/usr/lib64/pkgconfig/mount.pc
/usr/lib64/pkgconfig/blkid.pc
/usr/lib64/pkgconfig/uuid.pc
/usr/share/man/man3/libblkid.3.bz2


I did copy all of these libs to the initramfs. I haven't used mdev, but the wiki guide says that mdev is an alternative to devtmpfs so I didn't think it was necessary. I also recompiled util-linux with static-libs just to be sure, but it didn't seem like that changed anything.

etnull:
As stated above, I simply copied the libs that ldd showed me for the binaries I was including. My /lib has the following
Code:
ld-linux-x86-64.so.2
libaio.so.1
libblkid.so.1
libblkid.so.1.1.0
libc.so.6
libdevmapper-event.so.1.02
libdevmapper.so.1.02
libdl.so.2
libeinfo.so.1
libfdisk.so.1
libfdisk.so.1.1.0
libmount.so.1
libmount.so.1.1.0
libm.so.6
libpcre.so.1
libpthread.so.0
librc.so.1
libreadline.so.7
librt.so.1
libselinux.so.1
libsmartcols.so.1
libsmartcols.so.1.1.0
libtinfo.so.6
libtinfow.so.6
libudev.so.1
libusb-1.0.so.0
libuuid.so.1
libuuid.so.1.3.0
libz.so.1


And my /usr/lib64 has these
Code:
libargon2.so.1
libblkid.a
libblkid.so
libcrypto.so.1.1
libcryptsetup.so.12
libfdisk.a
libfdisk.so
libjson-c.so.4
libmount.a
libmount.so
libpopt.so.0
libsmartcols.a
libsmartcols.so
libuuid.a
libuuid.so
libykpers-1.so.1
libyubikey.so.0
pkgconfig


NeddySeagoon:
I do have /dev/null and /dev/console, among some other things
Code:
bsg
btrfs-control
bus
console
cpu
cpu_dma_latency
fb0
full
hpet
hwrng
input
kmsg
mapper
mcelog
megadev0
mem
memory_bandwidth
network_latency
network_throughput
null
nvme-fabrics
port
ptmx
random
raw
rfkill
rtc0
sda
sda1
snapshot
tty
tty0
tty1
(...)
ttyS9
uhid
urandom
usbmon0
usbmon1
usbmon2
vcs
vcs1
vcsa
vcsa1
vcsu
vcsu1
vga_arbiter
zero


As you can see, after making a few more changes, it actually recognizes one partition (sda1), but lsblk said that it was a 28GB partition, which doesn't match any of the partitions on my disk or on the USB flash drive that I had plugged in. I have no idea what that node was referring to.

Anyway, I recompiled my kernel and some modules disappeared, so things got even worse after that. I am too tired and frustrated to continue working at this for today, so I will do some more exploration tomorrow and come back with an update. Thank you fellows for the help.
Back to top
View user's profile Send private message
Zucca
Veteran
Veteran


Joined: 14 Jun 2007
Posts: 1769
Location: KUUSANKOSKI, Finland

PostPosted: Sat Apr 11, 2020 9:02 am    Post subject: Reply with quote

IELeibowitz wrote:
I did copy all of these libs to the initramfs.
Using ldd is always the better way. I just showed some of the libs that are provided by the util-linux package. You don't need to copy them all. Not all of the listed are even libs.
IELeibowitz wrote:
I haven't used mdev, but the wiki guide says that mdev is an alternative to devtmpfs so I didn't think it was necessary.
Yup. It's not. You can always run manual commands to populate /dev from your script.
_________________
..: Zucca :..

Code:
ERROR: '--failure' is not an option. Aborting...
Back to top
View user's profile Send private message
charles17
Advocate
Advocate


Joined: 02 Mar 2008
Posts: 3238

PostPosted: Sat Apr 11, 2020 10:08 am    Post subject: Reply with quote

Zucca wrote:
IELeibowitz wrote:
I did copy all of these libs to the initramfs.
Using ldd is always the better way. ...
lddtree --copy-to-tree (wasn't this what needs python USE flag enabled for pax-utils?)
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 45383
Location: 56N 3W

PostPosted: Sat Apr 11, 2020 11:01 am    Post subject: Reply with quote

IELeibowitz,

Lets go back to the very start of booting.
Post your
Code:
lspci -nnk
so we can see what hardware you have.
I suspect you will have an ahci rotating rust rust HDD controller and maybe an nvme block device too.
That will let us work out what you need hardware wise, in your kernel.
Tell us where your root filesystem is.

Tell us if you block devices have msdos or GPT partition labels. That matters to your kernel options and sometimes, to your boot loader.
The output of
Code:
fdisk -l
on a postebin site would be really good.

Is UEFI firmware in use or legacy mode?
That matters because not all combinations of firmware and disklable work.

So that we can put all this together, put your kernel .config file onto a pastebin site.
It looks like your /dev is missing the node that holds your root filesystem. That means until your kernel works well enough to make that appear,nothing else matters.

Which version of genkernel are you using?

If I were doing this, I would build both the kernel and initrd by hand. Thats harder as it forces you to understand whats happening but the complexity is reduced as you don't have to work through a tool.

By way of debug, change your rescue shell function to
Code:
rescue_shell() {
    echo "$@"
    echo "Something went wrong. Dropping you to a shell."
# have time to read the message
    /bin/sleep 20
    /bin/busybox --install -s
    exec /bin/sh
}

Now you can pass it a message to display when something fails, like
Code:
...
modprobe fat || rescue_shell "modprobe fat failet"
...
and so on.
Do that for every command, make the messages unique and it stops at the first failed command.

Anyway lets do the analysis first.

-- edit --
That rescue_shell function is from a long time ago when busybox needed full path names.
You still need to use full path names or set PATH= to call non busybox programs.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
IELeibowitz
n00b
n00b


Joined: 14 Mar 2020
Posts: 19

PostPosted: Sat Apr 11, 2020 4:34 pm    Post subject: Reply with quote

lspci: http://dpaste.com/1FK6R3B

fdisk: http://dpaste.com/0CP46GD

kernel config: http://dpaste.com/018FDBX[/code]

My drives have gpt labels, my system is installed on an SSD, and my computer has libreboot.

I am using genkernel 4.0.4. I don't actually use it to make the kernel (I just use the make commands), but I used it to make my initramfs. I noticed that the genkernel init has a section where is says it's scanning for volume groups, immediately after all modules are loaded. What is it actually doing here, and is it something important that I should have in my script ? Unfortunately the generated init script is so obfuscated that I can't make any sense of it.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 45383
Location: 56N 3W

PostPosted: Sat Apr 11, 2020 5:28 pm    Post subject: Reply with quote

IELeibowitz,

You don't nave an nvme interface. Its SATA only. That helps.
Code:
00:1f.2 SATA controller [0106]: Intel Corporation 82801IBM/IEM (ICH9M/ICH9M-E) 4 port SATA Controller [AHCI mode] [8086:2929] (rev 03)
   Subsystem: Lenovo ThinkPad T400 [17aa:20f8]
   Kernel driver in use: ahci
   Kernel modules: ahci


As does
Code:
Device      Start        End   Sectors   Size Type
/dev/sda1    2048       6143      4096     2M BIOS boot
/dev/sda2    6144     530431    524288   256M EFI System
/dev/sda3  530432 1000213167 999682736 476.7G Linux LVM


The boot sequences goes, the boot loader loads the kernel and initrd,
The kernel and initrd together gain access to /dev/sda, find /dev/sda3 and start LVM, then decrypt the content of that that volume group.
Or is the the other way round for you.?
Decrypt /dev/sda3, then start LVM. Both ways can be made to work.

Going through your kernel
Code:
# CONFIG_EFI_PARTITION is not set
Thats the kernel name for gpt. Your kernel cannot read your GPT partition tables.
Code:
CONFIG_MSDOS_PARTITION=y
is there, so it sees the protective msdos partition table on your GPT drives.

Code:
CONFIG_HAVE_IDE=y
# CONFIG_IDE is not set
Good. Enabling that makes a mess of SATA.

Code:
CONFIG_SCSI=y
CONFIG_BLK_DEV_SD=y


Turn off CONFIG_SCSI_LOWLEVEL=y, thats for real SCSI and you don't have any. It will save kernel bloat and build time.

Code:
CONFIG_ATA=y
good.

Code:
CONFIG_SATA_AHCI=m
make that built in. You can't access your HDD without it.
Code:
CONFIG_BLK_DEV_DM=y
Good.

Code:
CONFIG_DM_DEBUG=y
be very careful of options with DEBUG is their names.
Many of your CONFIG_DM_* optins are for testing/debug.

Turn off CONFIG_FUSION=y you don't have the hardware.

Turn off CONFIG_NTFS_RW=y as it does not do what you think it does. If you need write access to NTFS volumes, use ntfs-3g and FUSE.
They can and do prevent normal operation oy whatever the relate too. Turn them all off.

CONFIG_BLK_DEV_INITRD=y and friends, good.

I don't see any major issue there apart from the partition table.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
IELeibowitz
n00b
n00b


Joined: 14 Mar 2020
Posts: 19

PostPosted: Sat Apr 11, 2020 9:20 pm    Post subject: Reply with quote

Thank you for the guidance. I have no idea how all of these things got turned off. The kernel I'm currently running functions perfectly, but I recompiled with a slightly more recent version of Linux 4.19 and it seems that things somehow got altered (despite using what I thought was the same .config). I will try this new kernel and see how it works, but I managed to get the Yubikey working with my init, so things look pretty good.
Back to top
View user's profile Send private message
IELeibowitz
n00b
n00b


Joined: 14 Mar 2020
Posts: 19

PostPosted: Sat Apr 11, 2020 10:30 pm    Post subject: Reply with quote

After making the recommended changes to my .config, I am now able to use my Yubikey to unlock my LUKS partition. However, I get dropped into a rescue shell when my init tries to mount the root volume (it says that /mnt/root is not found for some reason, even though the folder is present). I am able to mount the root inside the shell, but then the shell freezes when I get to exec switch_root (following the exact commands that I have in the init script).

I also noticed, after booting into the new kernel using genkernel's initramfs, that I'm unable to mount my /boot partition. It gives me the following error message whenever I try:
Code:
# mount /dev/sda2 /boot
mount: /boot: mount(2) system call failed: Operation not supported.


Update:
The mounting problems were because I forgot to add ext4 security labels to my kernel, but after adding them, I'm still getting the following init runtime errors:
Code:
modprobe: '4.19.114-gentoo-x86_64/kernel/drivers/ata/libahci.ko': invalid module format
modprobe: '4.19.114-gentoo-x86_64/kernel/drivers/ata/ahci.ko': invalid parameter


And when init attempts to mount:
Code:
mount: can't find /mnt/root in /etc/fstab


The strange thing is that I have no problem mounting root when I'm in the shell. It's only in init that I have problems. Also my volumes appear as /dev/dm-1, /dev/dm-2, and so on, instead of being mapped into volume group folders.
Back to top
View user's profile Send private message
IELeibowitz
n00b
n00b


Joined: 14 Mar 2020
Posts: 19

PostPosted: Sun Apr 12, 2020 1:49 am    Post subject: Reply with quote

I got the initramfs to boot properly. The problem with the mounting of my root volume is that I accidentally deleted the line that activated the volume group. Unfortunately, I couldn't get it to boot when umounting /dev /sys and /proc. For some reason it would just freeze at the switch_root. I managed to boot it by moving the mountpoints to the newroot, which is unsatisfying.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Installing Gentoo All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum