Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Building kernel using overlayfs + tmpfs
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware
View previous topic :: View next topic  
Author Message
Havin_it
Veteran
Veteran


Joined: 17 Jul 2005
Posts: 1187
Location: Edinburgh, UK

PostPosted: Thu May 07, 2020 10:55 am    Post subject: Building kernel using overlayfs + tmpfs Reply with quote

I had this idea, and I don't see any examples of it with a bit of googling, so this makes me think it's a bad idea or otherwise someone would have done it and written about it ;) so I'd like opinions/insight.

Idea: Mount an overlayfs with /usr/src/linux as lower layer and a tmpfs location (/tmp/whatever) as upper, and do the build in the overlay. The object is to save disk I/O during the build and maybe improve performance. (I've seen that some people copy the whole source into tmpfs but this way seems a bit more efficient, plus it's a lot of RAM to tie up.)

At first I thought to then merge all the created files from the overlay into /usr/src/linux after completion, however maybe there's not even much point doing this. I seldom rebuild a kernel, and when I do, although long ago this used to be much quicker than the initial build, nowadays a rebuild always seems to take almost just as long anyway (is there a reason for this? It's a change I noticed quite a few years ago).

Is this a bad idea for some reason, or would it just not gain much? I look forward to any knowledge/views.
Back to top
View user's profile Send private message
Ionen
l33t
l33t


Joined: 06 Dec 2018
Posts: 697

PostPosted: Thu May 07, 2020 11:20 am    Post subject: Reply with quote

Different but just to mention... "If" wanted the whole source in tmpfs (or just want the process more automated), sys-kernel/gentoo-kernel may interest you. emerge will build the kernel like any other ebuild using its usual temporary directory (which could be tmpfs), then it installs a trimmed down source at /usr/src (about ~80MB big) only enough to build third party modules and/or run things like make menuconfig (when want to update your USE=savedconfig). Bootloader and further actions can be setup as usual using a /etc/kernel/postinst.d/ script (arg1 = version, arg2 = kernel location).

Edit: That aside, if ram is a concern, I'd simply not bother with any odd setup. It's not worth the time setting up, your SSDs will be perfectly fine, and speed difference is likely to be insignificant unless you're trying to set some speed record with -j128 or something :) Writes will be buffered and not really slow down anything too. I do use tmpfs to build stuff myself but it's only because I'm not worried about running out, and I'd rather i/o usage be left alone for other things on the system without relying on ionice and the like.

Edit2: Oh and you can build the kernel in a different build directory, so overlayfs isn't necessary (make O=/path/to/build/dir), gentoo-kernel does the same thing to avoid polluting the source it's going to partially install
Back to top
View user's profile Send private message
Havin_it
Veteran
Veteran


Joined: 17 Jul 2005
Posts: 1187
Location: Edinburgh, UK

PostPosted: Thu May 07, 2020 11:47 am    Post subject: Reply with quote

Hi Ionen, thanks for the tip as it's not something I was aware of. The minimal disk footprint is appealing and the functionality does sound like it would obsolete a lot of the shocking handwritten scripts I've built up over years ... however ...

1. Putting the whole build tree into RAM is something I was looking to avoid with the above idea - machine is quite old and I want to balance RAM and disk use
2. It appears (not certain this is correct) that it uses make olddefconfig so I'd no longer be able to review new config items when building a fresh kernel
3. I'd have my kernel choice limited by portage tree (current tree doesn't have an ebuild for my current backup kernel f.ex. so how could I rebuild it if I needed to?)
4. At the moment I'm more interested to know whether avoiding the many disk writes *during* the build will make significant difference.

Edit after your edits:

I guess you answered (4) above. Machine is 2012-era so DDR3 1600MHz (4GB), CPU is an old AMD APU E2-1800, which I know is the real bottleneck. Aftermarket Samsung SSD 850 about 4yrs old, don't really have any benchmark for this so thought changing the balance to do more in RAM might be helpful.

Never heard about make O= before either, that sounds like it might be the easiest route.
Back to top
View user's profile Send private message
Anon-E-moose
Advocate
Advocate


Joined: 23 May 2008
Posts: 4537
Location: Dallas area

PostPosted: Thu May 07, 2020 12:04 pm    Post subject: Reply with quote

I build my kernel on a tmpfs all the time.

From an old cheat sheet
Code:
#if building on /tmp (ram)

cp -ap /usr/src/linux-# /tmp
cd /tmp/linux-#

---

make -j14 bzImage modules
...
sudo /home/don/bin/fix-libmod


the fix-libmod script fixes the bad link generated by modules_install being run from the tmpfs
Code:
$ cat bin/fix-libmod
kdir=`basename "$PWD"`
lmdir=`cat include/config/kernel.release`

cd /lib/modules/$lmdir

rm build source
ln -s /usr/src/$kdir build
ln -s /usr/src/$kdir source


I suppose I could automate it, but it's not that often that I change kernels, and it's not that hard to manually copy stuff, etc.
_________________
PRIME x570-pro, 3700x, RX 550 & 560
Acer E5-575 (laptop), i3-7100u - i965
---both---
5.5.18 zen kernel, gcc 9.3.0, profile 17.1 (no-pie & modified) amd64-no-multilib, eudev, openrc, openbox, palemoon
Back to top
View user's profile Send private message
Havin_it
Veteran
Veteran


Joined: 17 Jul 2005
Posts: 1187
Location: Edinburgh, UK

PostPosted: Thu May 07, 2020 12:18 pm    Post subject: Reply with quote

@Anon-E-moose thanks but as I said I'm already familiar with doing the whole thing in RAM, the intent of the overlayfs idea was to find a halfway house and just avoid doing lots of disk writes during the build. I don't think the reads make as much difference, especially vs reading the whole tree into RAM before starting when a lot of it won't even be read during build.
Back to top
View user's profile Send private message
Ant P.
Watchman
Watchman


Joined: 18 Apr 2009
Posts: 6491

PostPosted: Thu May 07, 2020 11:51 pm    Post subject: Reply with quote

You're all overcomplicating a thing the kernel has provided out of the box for years.
Code:
make -C /usr/src/linux O=/tmp/kbuild oldconfig
cd /tmp/kbuild
make -j$(nproc)

(Replace /tmp/ with something mounted -o exec)
Back to top
View user's profile Send private message
Havin_it
Veteran
Veteran


Joined: 17 Jul 2005
Posts: 1187
Location: Edinburgh, UK

PostPosted: Fri May 08, 2020 12:10 am    Post subject: Reply with quote

Aha, now I realise why O= was failing when I tried it: /tmp mounted noexec. Derp.

I also went ahead and tried the overlayfs approach:
Code:

$ mkdir /tmp/kbuild{,-w,-m}
$ mount -t overlay overlay -o lowerdir=/user/src/linux-5.6.11-gentoo,upperdir=/tmp/kbuild,workdir=/tmp/kbuild-w /tmp/kbuild-m
$ cd /tmp/kbuild-m
$ make && make modules_install


I put the real dir as lowerdir rather than linux symlink as not sure whether a symlink would work for this.

[I note that despite all except the lowerdir being on the noexec /tmp, the mount was exec]

I timed the build at 66.6 min (wooooo) which is among the average of a few recent builds I've profiled, so I guess it didn't really make any appreciable gain. Ah well.
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 15324

PostPosted: Fri May 08, 2020 2:00 am    Post subject: Reply with quote

66 minutes seems very long, even for a CPU from 8 years ago. That length of time suggests to me you are building a large portion of the kernel's optional features. Do you really need all the things you enabled?
Back to top
View user's profile Send private message
pjp
Administrator
Administrator


Joined: 16 Apr 2002
Posts: 18438

PostPosted: Fri May 08, 2020 3:56 am    Post subject: Reply with quote

Ant P. wrote:
You're all overcomplicating a thing the kernel has provided out of the box for years.
Code:
make -C /usr/src/linux O=/tmp/kbuild oldconfig
cd /tmp/kbuild
make -j$(nproc)

(Replace /tmp/ with something mounted -o exec)
Do you do something similar for modules commands? I've had to put the module related variables before the make command itself, such as:
Code:
INSTALL_MOD_PATH=/install/mod/path /usr/bin/make O=/output/path modules_install
Code:
INSTALL_PATH=/install/path /usr/bin/make O=/output/path install
I thought there was an option (similar to -O), but I couldn't get it to work, so I settled for the above. Thanks for the -C reference. I think that will help clean up my script.
_________________
The media sells it and you live the role.
Back to top
View user's profile Send private message
Havin_it
Veteran
Veteran


Joined: 17 Jul 2005
Posts: 1187
Location: Edinburgh, UK

PostPosted: Fri May 08, 2020 10:20 am    Post subject: Reply with quote

Hu wrote:
66 minutes seems very long, even for a CPU from 8 years ago. That length of time suggests to me you are building a large portion of the kernel's optional features. Do you really need all the things you enabled?


Honestly can't be all that sure, it's quite possible I have dead-weight in there although not by design. The laptop has sysop duties so there are some things like raid support and lots of filesystems that are rarely/never used but considered requisite. Other stuff I'm not sure about but erring with caution like most of the crypto library stuff, xtables stuff that's proabably never used. Compared to how much stuff I do leave out it never seemed excessive.

In another thread I've been discussing ways to audit what modules are actually needed (like running a liveusb and examining what gets loaded there), so I may give that a go. Of course it also might be that my using distcc, ccache and -j / -l options are being counterproductive.

It really is a weedy CPU though even for its day. I buy what I can afford :roll:
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 15324

PostPosted: Sat May 09, 2020 12:41 am    Post subject: Reply with quote

pjp wrote:
Do you do something similar for modules commands? I've had to put the module related variables before the make command itself, such as:
Code:
INSTALL_MOD_PATH=/install/mod/path /usr/bin/make O=/output/path modules_install
Code:
INSTALL_PATH=/install/path /usr/bin/make O=/output/path install
I thought there was an option (similar to -O), but I couldn't get it to work, so I settled for the above. Thanks for the -C reference. I think that will help clean up my script.
I export the three variables ($KBUILD_OUTPUT, $INSTALL_PATH, $INSTALL_MOD_PATH) in the shell, so that all subsequent make calls will use them without requiring me to restate them.
Havin_it wrote:
It really is a weedy CPU though even for its day. I buy what I can afford :roll:
I save up and buy a bit higher end, but compensate by keeping parts for as long as they last, so my results will be a bit different from yours. Even so, I can't recall needing more than 20 minutes to build even my larger kernels.
Back to top
View user's profile Send private message
pjp
Administrator
Administrator


Joined: 16 Apr 2002
Posts: 18438

PostPosted: Sat May 09, 2020 3:54 am    Post subject: Reply with quote

Hu wrote:
I export the three variables ($KBUILD_OUTPUT, $INSTALL_PATH, $INSTALL_MOD_PATH) in the shell, so that all subsequent make calls will use them without requiring me to restate them.
For some reason I don't recall, I have avoided relying on environment variables. This seems like an instance where it may make life easier. I'll give it a try, thanks.
_________________
The media sells it and you live the role.
Back to top
View user's profile Send private message
Ant P.
Watchman
Watchman


Joined: 18 Apr 2009
Posts: 6491

PostPosted: Sat May 09, 2020 6:38 am    Post subject: Reply with quote

pjp wrote:
Do you do something similar for modules commands? I've had to put the module related variables before the make command itself, such as:
Code:
INSTALL_MOD_PATH=/install/mod/path /usr/bin/make O=/output/path modules_install
Code:
INSTALL_PATH=/install/path /usr/bin/make O=/output/path install
I thought there was an option (similar to -O), but I couldn't get it to work, so I settled for the above. Thanks for the -C reference. I think that will help clean up my script.

For the actual install part I just use something along the lines of su -c 'make modules_install install'. I'd end up invoking root either way to put the kernel in /boot/ so I just get it all done at once.
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 15324

PostPosted: Sat May 09, 2020 4:19 pm    Post subject: Reply with quote

I see two major arguments against environment variables. First, if you leave them set, you may be surprised later when they are still active and you have forgotten them. Second, if you want them to be version-specific (particularly if you build kernels for multiple systems), maintaining them can be a nuisance, particularly if you have many shells open and are not consistent about which one you use. I solve both problems by having a wrapper script that computes the right value based on context, exports the variables, then runs an interactive subshell. When I exit the subshell, the controlling script exits, and the variables are not propagated back to my regular shell. This ensures that I do not leave them set long term, and minimizes the tedium of setting them to correct values.
Back to top
View user's profile Send private message
pjp
Administrator
Administrator


Joined: 16 Apr 2002
Posts: 18438

PostPosted: Sat May 09, 2020 10:44 pm    Post subject: Reply with quote

Thanks Ant P., Hu.

My lack of using environment variables necessitates my testing it, so that helps clarify what I was anticipating. My script is up to 201 lines without comments, although it does do more than just compile the kernel. I think it is definitely time for review.
_________________
The media sells it and you live the role.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum