Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Compiling kernel on tmpfs
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware
View previous topic :: View next topic  
Author Message
niick
Tux's lil' helper
Tux's lil' helper


Joined: 09 Mar 2006
Posts: 91

PostPosted: Fri Nov 08, 2019 11:12 pm    Post subject: Compiling kernel on tmpfs Reply with quote

Hey guys, I want to compile the kernel on tmpfs (in memory) to avoid disk writes. Motivation is that I just had an ssd die, presumably from too many writes and I just upgraded to 32GB of RAM :-)
I have portage on tmpfs but obviously the kernel is built in;
Code:
/usr/src/linux
.
I don't want to lose the kernel on a reboot. Is anyone doing this or can suggest a method that would work?
_________________
"Give me control of a nation's money and I care not who makes the laws."
Mayer Amschel Rothschild
Back to top
View user's profile Send private message
eccerr0r
Watchman
Watchman


Joined: 01 Jul 2004
Posts: 7414
Location: almost Mile High in the USA

PostPosted: Fri Nov 08, 2019 11:17 pm    Post subject: Reply with quote

Likely your SSD did not die from too many writes, you got a dud drive unless you were *really* abusing it, or bought an abused drive.

If you truly want to save the data you'll need to rsync it back and forth. However you might well just run it on SSD, don't worry about it, all my kernels are compiled on SSD directly.
_________________
Intel Core i7 2700K@ 4.1GHz/HD3000 graphics/8GB DDR3/180GB SSD
What am I supposed watching?
Back to top
View user's profile Send private message
niick
Tux's lil' helper
Tux's lil' helper


Joined: 09 Mar 2006
Posts: 91

PostPosted: Fri Nov 08, 2019 11:22 pm    Post subject: Reply with quote

Yeah, it's probably paranoia but I'd had it a long time and just put it down to writes. Ignoring that I was looking for a method of building the kernel on tmpfs for the same rational as putting portage on tmpfs.
_________________
"Give me control of a nation's money and I care not who makes the laws."
Mayer Amschel Rothschild
Back to top
View user's profile Send private message
niick
Tux's lil' helper
Tux's lil' helper


Joined: 09 Mar 2006
Posts: 91

PostPosted: Fri Nov 08, 2019 11:31 pm    Post subject: Reply with quote

Having done some googling I found this; https://stackoverflow.com/questions/15969694/make-o-files-in-separate-folder-in-linux-kernel-compilation.

I guess this would work for me, I'd just end up compiling the entire kernel again even if I just made small changes to included modules and such.

Anyone got any experience of this??
_________________
"Give me control of a nation's money and I care not who makes the laws."
Mayer Amschel Rothschild
Back to top
View user's profile Send private message
Anon-E-moose
Advocate
Advocate


Joined: 23 May 2008
Posts: 4399
Location: Dallas area

PostPosted: Fri Nov 08, 2019 11:47 pm    Post subject: Reply with quote

I've built the kernel on tmpfs, but I run a script after installing modules

For example, I copy /usr/src/linux-5.0 to /tmp/linux-5.0, I go to /tmp/linux-5.0 build everything, install everything then run this script
Code:
kdir=`basename "$PWD"`
lmdir=`cat include/config/kernel.release`

cd /lib/modules/$lmdir

rm build source
ln -s /usr/src/$kdir build
ln -s /usr/src/$kdir source


Edit to add: it's not that I worry about too many writes (/usr/src is on an ssd) but it's faster to compile from tmpfs.

ETA2: before I do the copy to tmpfs I do all my configuring on the disk based kernel sources, I just use the tmpfs for compiling.
_________________
Asus m5a99fx, FX 8320 - nouveau, oss4, rx550 for qemu passthrough
Acer laptop E5-575, i3-7100u - i965, alsa
---both---
5.0.13 zen kernel, profile 17.1 (no-pie & modified) amd64-no-multilib
gcc 8.2.0, eudev, openrc, openbox, palemoon
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 14957

PostPosted: Sat Nov 09, 2019 12:56 am    Post subject: Reply with quote

I do out of tree builds so that I don't need to build as root. It works fine.
Back to top
View user's profile Send private message
Anon-E-moose
Advocate
Advocate


Joined: 23 May 2008
Posts: 4399
Location: Dallas area

PostPosted: Sat Nov 09, 2019 12:59 am    Post subject: Reply with quote

Hu wrote:
I do out of tree builds so that I don't need to build as root. It works fine.


I don't use portage for kernels, I prefer to do it myself.
I do the building/compiling kernel and modules as my user, but become root/sudo for installation of kernel and modules.
_________________
Asus m5a99fx, FX 8320 - nouveau, oss4, rx550 for qemu passthrough
Acer laptop E5-575, i3-7100u - i965, alsa
---both---
5.0.13 zen kernel, profile 17.1 (no-pie & modified) amd64-no-multilib
gcc 8.2.0, eudev, openrc, openbox, palemoon
Back to top
View user's profile Send private message
DaggyStyle
Watchman
Watchman


Joined: 22 Mar 2006
Posts: 5437

PostPosted: Sat Nov 09, 2019 9:19 am    Post subject: Reply with quote

here is my script that does this exactly, feel free to try
Code:

#!/bin/bash -x

function exe() {
        local cmd="$1"
        eval "${cmd}"
        if [ $? -ne 0 ]; then
                echo "run failed on cmd: ${cmd}"
                exit 1
        fi
}

KERNEL_SRC_PATH="/usr/src/linux"
KERNEL_TARGET=$(eselect kernel list | tail -1 | awk '{print $1}' | sed 's/[][]//g')

exe "eselect kernel set ${KERNEL_TARGET}"

FULL_KERNEL_LABEL="$(basename $(eselect kernel show | tail -1))"
KERNEL_VERSION="$(echo ${FULL_KERNEL_LABEL} | sed "s/-gentoo$//g;s/^linux-//g")"
MNT_POINT="/tmp/kernel"

exe "mkdir -p ${MNT_POINT}"
exe "mount -t tmpfs -o size=1G tmpfs ${MNT_POINT}"
exe "zcat /proc/config.gz > ${MNT_POINT}/.config"
exe "make -C ${KERNEL_SRC_PATH} O=${MNT_POINT} oldconfig"
exe "make -C ${KERNEL_SRC_PATH} ${MAKEOPTS} O=${MNT_POINT}"
exe "make -C ${KERNEL_SRC_PATH} ${MAKEOPTS} O=${MNT_POINT} modules_install"
exe "mount /boot/"
exe "cp ${MNT_POINT}/arch/x86/boot/bzImage /boot/kernel64-${KERNEL_VERSION}.img"
exe "GRUB_USE_LINUX_LABEL=true grub-mkconfig -o /boot/grub/grub.cfg"
exe "umount /boot"
exe "emerge @module-rebuild"
exe "cp -v ${MNT_POINT}/.config /usr/src//linux/config"
exe "umount -l ${MNT_POINT}"
exe "rm -rf ${MNT_POINT}"
exit 0

_________________
Only two things are infinite, the universe and human stupidity and I'm not sure about the former - Albert Einstein
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 14957

PostPosted: Sat Nov 09, 2019 5:35 pm    Post subject: Reply with quote

DaggyStyle wrote:
Code:
#!/bin/bash -x

function exe() {
        local cmd="$1"
        eval "${cmd}"
        if [ $? -ne 0 ]; then
                echo "run failed on cmd: ${cmd}"
                exit 1
        fi
}
Why not just use set -e and let it handle aborting? This would also get you out of using eval, which is wrong here since it causes double expansion of variables. If you want to keep the wrapper, you could remove the double expansion by writing it as:
Code:
function exe() {
   "$@" || { local rc=$?; echo "run failed on cmd: $@" >&2; exit $rc; }
}
If you make such a change, remove the double quotes in the caller and let the word splitting happen there. As an alternative, which is needed to support the lines where redirection is used, switch to single quotes in the caller and retain your definition of exe. Personally, I like letting set -e handle the early exit.
DaggyStyle wrote:
Code:
KERNEL_TARGET=$(eselect kernel list | tail -1 | awk '{print $1}' | sed 's/[][]//g')
This will return the string (none if there are no valid kernels available. You can simplify the existing expression to use just sed:
Code:
eselect kernel list | sed -n -e '${s/^\s*//;s/\s.*$//;s/[][]//g;p}'

DaggyStyle wrote:
Code:
KERNEL_VERSION="$(echo ${FULL_KERNEL_LABEL} | sed "s/-gentoo$//g;s/^linux-//g")"
This could be done using bash variable expansions.
Code:
KERNEL_VERSION="${FULL_KERNEL_LABEL%-gentoo}"
KERNEL_VERSION="${KERNEL_VERSION#linux-}"

DaggyStyle wrote:
Code:
exe "mkdir -p ${MNT_POINT}"
mkdir -p does not fail when the target exists. This may produce surprising results if the directory /tmp/kernel already exists and has content you would like to keep.
DaggyStyle wrote:
Code:
exe "mount -t tmpfs -o size=1G tmpfs ${MNT_POINT}"
By default, tmpfs is world writable and sticky. This is probably not what you want here.
DaggyStyle wrote:
Code:
exe "make -C ${KERNEL_SRC_PATH} O=${MNT_POINT} oldconfig"
exe "make -C ${KERNEL_SRC_PATH} ${MAKEOPTS} O=${MNT_POINT}"
exe "make -C ${KERNEL_SRC_PATH} ${MAKEOPTS} O=${MNT_POINT} modules_install"
These steps will run as root, but that permission is not necessary.
DaggyStyle wrote:
Code:
exe "cp -v ${MNT_POINT}/.config /usr/src//linux/config"
Why not use $KERNEL_SRC_PATH here?

Note that if the script exits early, the user must manually apply any remaining cleanups that would have happened on a successful exit.
Back to top
View user's profile Send private message
pjp
Administrator
Administrator


Joined: 16 Apr 2002
Posts: 18349

PostPosted: Sat Nov 09, 2019 9:23 pm    Post subject: Reply with quote

niick wrote:
Having done some googling I found this; https://stackoverflow.com/questions/15969694/make-o-files-in-separate-folder-in-linux-kernel-compilation.

I guess this would work for me, I'd just end up compiling the entire kernel again even if I just made small changes to included modules and such.

Anyone got any experience of this??
I combine tmpfs (writing) with squashfs (saving). The squashfs also helps for programs that expect kernel files to be available. Although I sometimes just let the buld process make assumptions about the running kernel. I need to get back to making my script more complete.
_________________
The media sells it and you live the role.
Back to top
View user's profile Send private message
DaggyStyle
Watchman
Watchman


Joined: 22 Mar 2006
Posts: 5437

PostPosted: Sun Nov 10, 2019 6:10 pm    Post subject: Reply with quote

Hu wrote:
DaggyStyle wrote:
Code:
#!/bin/bash -x

function exe() {
        local cmd="$1"
        eval "${cmd}"
        if [ $? -ne 0 ]; then
                echo "run failed on cmd: ${cmd}"
                exit 1
        fi
}
Why not just use set -e and let it handle aborting? This would also get you out of using eval, which is wrong here since it causes double expansion of variables. If you want to keep the wrapper, you could remove the double expansion by writing it as:
Code:
function exe() {
   "$@" || { local rc=$?; echo "run failed on cmd: $@" >&2; exit $rc; }
}
If you make such a change, remove the double quotes in the caller and let the word splitting happen there. As an alternative, which is needed to support the lines where redirection is used, switch to single quotes in the caller and retain your definition of exe. Personally, I like letting set -e handle the early exit.
DaggyStyle wrote:
Code:
KERNEL_TARGET=$(eselect kernel list | tail -1 | awk '{print $1}' | sed 's/[][]//g')
This will return the string (none if there are no valid kernels available. You can simplify the existing expression to use just sed:
Code:
eselect kernel list | sed -n -e '${s/^\s*//;s/\s.*$//;s/[][]//g;p}'

DaggyStyle wrote:
Code:
KERNEL_VERSION="$(echo ${FULL_KERNEL_LABEL} | sed "s/-gentoo$//g;s/^linux-//g")"
This could be done using bash variable expansions.
Code:
KERNEL_VERSION="${FULL_KERNEL_LABEL%-gentoo}"
KERNEL_VERSION="${KERNEL_VERSION#linux-}"

DaggyStyle wrote:
Code:
exe "mkdir -p ${MNT_POINT}"
mkdir -p does not fail when the target exists. This may produce surprising results if the directory /tmp/kernel already exists and has content you would like to keep.
DaggyStyle wrote:
Code:
exe "mount -t tmpfs -o size=1G tmpfs ${MNT_POINT}"
By default, tmpfs is world writable and sticky. This is probably not what you want here.
DaggyStyle wrote:
Code:
exe "make -C ${KERNEL_SRC_PATH} O=${MNT_POINT} oldconfig"
exe "make -C ${KERNEL_SRC_PATH} ${MAKEOPTS} O=${MNT_POINT}"
exe "make -C ${KERNEL_SRC_PATH} ${MAKEOPTS} O=${MNT_POINT} modules_install"
These steps will run as root, but that permission is not necessary.
DaggyStyle wrote:
Code:
exe "cp -v ${MNT_POINT}/.config /usr/src//linux/config"
Why not use $KERNEL_SRC_PATH here?

Note that if the script exits early, the user must manually apply any remaining cleanups that would have happened on a successful exit.

well, it was a script I wrote in 5 minutes which is working well, writing a proper one is on my todo list
_________________
Only two things are infinite, the universe and human stupidity and I'm not sure about the former - Albert Einstein
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum