Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
KVM/QEMU: Binary host, VM build box, multiple binary VMs?
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Installing Gentoo
View previous topic :: View next topic  
Author Message
1clue
Advocate
Advocate


Joined: 05 Feb 2006
Posts: 2562

PostPosted: Tue Mar 05, 2019 3:07 am    Post subject: KVM/QEMU: Binary host, VM build box, multiple binary VMs? Reply with quote

Hi,

I'd like to setup a single device (8-core atom) as a KVM/QEMU host. This hardware has QuickAssist, which I want to pass on to all the VMs. It has 7x Intel NICs built-in, but unfortunately does not support VT-d. It currently has 16GB RAM, a sata3 SSD and a sata2 spinner.

I want several instances of a minimalist secure binary-only Gentoo image with these properties:

  1. No compiler.
  2. No package manager.
  3. Nothing the specific build does not need is on the disk to perform its tasks.
  4. Possibly busybox.
  5. Read-only root partition.
  6. Boot to RAM.
  7. Networked syslog.
  8. Custom (unique) kernel configuration
  9. Custom world file, but not on this image.
  10. If possible, no remote login facility, or login only from an air-gapped security network.
  11. All these images built by a single Gentoo guest VM on this host.


The purpose of this type of image is security, like you see in many firewall distros. If the system is compromised I don't want handy tools like compilers laying around ready to help build tools to attack my network or someone else's. I don't want kernel features enabled when they're not needed for that specific image. I don't want USE flags unless they're needed for that specific image.

One of these minimal binary images I want to be the host operating system. I have no idea how to do that, with the chicken-and-egg problem.

One VM image will be a build box:

  1. Full command-line Gentoo install.
  2. Networked through a tap on a bridge from the safest network on the device.
  3. Portage keeps track of this host as normal.
  4. Portage also keeps track of the host and all the minimal guest VMs.
  5. Separate kernel configs
  6. Separate USE flags
  7. Separate world files, stored on the build box
  8. The build box has sole write access to the minimal root drives.


So here are the questions:

  1. How do I (can i) set up portage on a single system to build for several unique all-binary systems?
  2. How do I make a binary-only host?
  3. How do I make the VM host be binary-only when it's built by one of its guests?
  4. Is the boot-from-ram and buildbox-writes-the-minimal-roots idea feasible? Is there a better way?


I think it may be nice to keep kernel configs and world files and other similar information on a git project on the build box. In this way the entire setup could be saved as a cohesive whole. I could see firewall rules and whatever other config being saved here too, if possible. It would enable easy comparison of kernel configs and world files, for example.
Back to top
View user's profile Send private message
nativemad
Developer
Developer


Joined: 30 Aug 2004
Posts: 911
Location: Switzerland

PostPosted: Tue Mar 05, 2019 6:32 am    Post subject: Reply with quote

Hi

I recently started to do a quite similar thing.
I have a buildscript that builds kvm-images out of some template files as well as an overlay that gets copied over the compiled rootimage (which is quite throwaway, as one can rebuild the whole thing quite automatically).
I do not use a ramdrive in the targetvm for root. Only /var is writeable on its own diskimage. The systemdrive holds a loop image that gets mounted as root. The initrd checks for new a imagefile (and kernel) and applies it upon boot. The initrd mounts the image read/write if the kernel parameter "writeable" is passed to it. That way you can try out changes on the fly to later put them in the overlay.
I use openrc-init and quite much busybox.
I don not have a process to deploy and verify newly created images so far, they need to get copied to the target manually for now.
The targetroot gets compiled via ROOT=. The downside of this approach is that a few things leak to the buildhost! Namely portage added users and groups and eselect-php. I needed to recreate these things manually in the overlay.

I'd start with a normal gentoo install in a vm and start building there.... The approach to deploy such images to bare metal should be quite the same.
I use a ramdrive on /mnt/ram to speed up buildtimes.

Here is my buildscript (forgive me, it is far from perfect):
Code:

#!/bin/bash
if [ -z $1 ]; then
        echo "we need arguments sir!"
        exit;
else
        if [ ! -e /build/$1 ]; then
                if [ ! $2 == "create" ]; then
                        echo "no build dir found and second argument is not create sir!"
                        exit;
                else
                        cp -a /build/skel /build/$1
                        echo "builddir created, make your adjustments in /build/$1 now."
                        exit;
                fi
        fi
        if [ ! -e /build/$1/root/etc/portage/make.conf ]; then
                mkdir -p /build/$1/root/etc/portage
                cp -a /build/$1/overlay/etc/portage /build/$1/root/etc/
                cp -a /build/$1/overlay/etc/locale.gen /build/$1/root/etc/
        fi
        if [ $2 == "disks" ] || [ $2 == "image" ] || [ $2 == "build" ]; then
                source /build/$1/build.conf
                eval ROOT="/build/$1/root/" PORTAGE_CONFIGROOT="/build/$1/root/" $adduses emerge --jobs 6 --tree -uDavN $packages glibc procps kbd nano bash e2fsprogs fail2ban kmod busybox acpid util-linux baselayout grub syslog-ng cronie logrotate net-snmp openssh && ROOT="/build/$1/root/" PORTAGE_CONFIGROOT="/build/$1/root/" emerge -av --depclean
                eval ROOT="/build/$1/root/" PORTAGE_CONFIGROOT="/build/$1/root/" $adduses emerge --jobs 3 -uDavN $packagesnodeps openrc netifrc gentoo-sources --nodeps
                if [ -e /build/$1/.config ]; then
                        cp /build/$1/.config /build/$1/root/usr/src/linux/
                else
                        cp /build/.config /build/$1/root/usr/src/linux/
                fi
                cd /build/$1/root/usr/src/linux
                make menuconfig
                make -j 9
                INSTALL_MOD_PATH=/build/$1/root make modules_install
                mkdir /build/$1/root/{boot,mnt,home,opt}
                cp /build/$1/root/usr/src/linux/arch/x86/boot/bzImage /build/$1/root/boot/vmlinuz
                cp /build/$1/root/usr/src/linux/arch/x86/boot/bzImage /build/$1/vmlinuz
                if [ $2 == "disks" ]; then
                        qemu-img create -f qcow2 /mnt/ram/$1.img 5G
                        nbdnr=0
                        nbdfound="no"
                        while [ $nbdfound == "no" ]; do
                                cat /proc/partitions | grep nbd$nbdnr && nbdnr=$((nbdnr + 1)) || nbdfound="yes"
                        done
                        qemu-nbd --connect=/dev/nbd$nbdnr /mnt/ram/$1.img
                        sfdisk /dev/nbd$nbdnr < /build/$1/parttable
                        mkfs.ext4 /dev/nbd${nbdnr}p1
                        mount /dev/nbd${nbdnr}p1 /build/$1/target
                elif [ $2 == "image" ]; then
                        mkdir /mnt/ram/$1target
                        mount -o bind /mnt/ram/$1target /build/$1/target
                fi
                if [ $2 == "disks" ] || [ $2 == "image" ]; then
rsync -urlopgtD --exclude=/usr/lib64/perl* --exclude=/usr/lib64/python3.6/turtledemo --exclude=/usr/lib64/python3.6/unittest \
--exclude=/usr/x86_64-pc-linux-gnu --exclude=/usr/lib64/python3.6/test --exclude=/usr/share/zoneinfo --exclude=/usr/share/doc --exclude=/usr/share/man \
--exclude=/usr/lib/gcc/*/*/include* --exclude=/usr/lib/gcc/*/*/*/include --exclude=/usr/share/gtk* --exclude=*systemd* --exclude=/share/man \
--exclude=/usr/share/locale --exclude=/usr/share/i18n --exclude=/usr/share/gcc-data --exclude=/var/db/pkg --exclude=/usr/src/* \
--exclude=usr/libexec/gcc --exclude=/usr/lib64/*.h --exclude=/usr/lib64/pkgconfig --exclude=/usr/lib/portage --exclude=/etc/portage \
--exclude=/usr/share/eselect --exclude=/usr/share/info --exclude=/var/db/pkg --exclude=/usr/lib64/php*/include --exclude=/usr/lib64/php*/lib/build \
--exclude=/var/lib/gentoo --exclude=/var/lib/portage --exclude=/var/cache/edb \
--exclude=/usr/include --exclude=/usr/share/bison --exclude=/usr/share/info /build/$1/root/* /build/$1/target/
                        sync
                        mkdir /build/$1/target/{dev,proc,sys,root}
                        echo "alias grep=grep" >/build/$1/target/root/.bash_profile
                        echo "alias poweroff=\"openrc-shutdown -p\"" >>/build/$1/target/root/.bash_profile
                        echo "alias reboot=\"openrc-shutdown -r\"" >>/build/$1/target/root/.bash_profile
                        ln -s busybox /build/$1/target/bin/grep
                        ln -s busybox /build/$1/target/bin/hostname
                        ln -s busybox /build/$1/target/bin/ifconfig
                        ln -s busybox /build/$1/target/bin/route
                        ln -s busybox /build/$1/target/bin/ping
                        ln -s busybox /build/$1/target/bin/netstat
                        ln -s busybox /build/$1/target/bin/awk
                        ln -s busybox /build/$1/target/bin/top
                        ln -s net.lo /build/$1/target/etc/init.d/net.eth0
                        ln -s net.lo /build/$1/target/etc/init.d/net.eth1
                        ln -s /etc/init.d/agetty /build/$1/target/etc/init.d/agetty.tty1
                        ln -s /proc/self/mounts /build/$1/target/etc/mtab
                        rm /build/$1/target/etc/runlevels/boot/mtab
                        ln -s /etc/init.d/{syslog-ng,cronie,acpid,net.eth0,fail2ban,sshd,snmpd,agetty.tty1} /build/$1/target/etc/runlevels/default/
                        if [ ! $startservices == "" ]; then
                                eval ln -s /etc/init.d/{$startservices} /build/$1/target/etc/runlevels/default/
                        fi
                        cp -a /build/$1/overlay/* /build/$1/target/
                        echo nameserver 172.18.0.10 >/build/$1/target/etc/resolv.conf
                        grub-install /dev/nbd$nbdnr --root-directory=/build/$1/target --modules="biosdisk part_msdos"
                        echo "set default=0" >>/build/$1/target/boot/grub/grub.cfg
                        echo "set timeout_style=menu" >>/build/$1/target/boot/grub/grub.cfg
                        echo "set timeout=10" >>/build/$1/target/boot/grub/grub.cfg
                        echo "menuentry \"tinyGentoo\" {" >>/build/$1/target/boot/grub/grub.cfg
                        echo "set root=(hd0,1)" >>/build/$1/target/boot/grub/grub.cfg
                        echo "linux /boot/vmlinuz openrc-init" >>/build/$1/target/boot/grub/grub.cfg
                        echo "initrd /boot/initramfs.gz" >>/build/$1/target/boot/grub/grub.cfg
                        echo "boot" >>/build/$1/target/boot/grub/grub.cfg
                        echo "}" >>/build/$1/target/boot/grub/grub.cfg
                        echo "menuentry \"old tinyGentoo\" {" >>/build/$1/target/boot/grub/grub.cfg
                        echo "set root=(hd0,1)" >>/build/$1/target/boot/grub/grub.cfg
                        echo "linux /boot/oldvmlinuz oldimage" >>/build/$1/target/boot/grub/grub.cfg
                        echo "initrd /boot/oldinitramfs.gz" >>/build/$1/target/boot/grub/grub.cfg
                        echo "boot" >>/build/$1/target/boot/grub/grub.cfg
                        echo "}" >>/build/$1/target/boot/grub/grub.cfg
                        chroot /build/$1/target /bin/passwd
                        sync
                        dd of=/mnt/ram/sysimage.img bs=1k count=0 seek=424000
                        mkfs.ext4 /mnt/ram/sysimage.img
                        mount /mnt/ram/sysimage.img /build/$1/targetimg
                        mv /build/$1/target/{root,bin,sbin,lib*,var,proc,sys,dev,tmp,etc,home,mnt,opt,run,usr} /build/$1/targetimg/
                        umount /build/$1/targetimg
                        cp /mnt/ram/sysimage.img /build/$1/
                        mv /mnt/ram/sysimage.img /build/$1/target/
                        cp /build/$1/initramfs.gz /build/$1/target/boot/
                        umount /build/$1/target
                        if [ $2 == "disks" ]; then
                                qemu-nbd -d /dev/nbd$nbdnr
                                mv /mnt/ram/$1.img /build/$1/
                        elif [ $2 == "image" ]; then
                                rm -rf /mnt/ram/$1target
                        fi
                fi
        fi
fi


In /build/skel I hold the template with a preconfigured fail2ban and files like conf.d/keymaps,hostname the fstab and so on. There is also build.conf that gets sourced from the buildscript and it looks like this:
Code:
packages="prosody certbot nginx sqlite"
packagesnodeps=""
startservices="net.eth1,nginx,prosody"
adduses="USE=\"sqlite\""


Some packages need libstdc++. In that case add gcc to "packagesnodeps". The rsync command in the buildscript excludes the binaries... The same thing with glibc.

Here is the make.conf that is in my template /build/skel/overlay/etc/portage/make.conf (make.profile in that folder is a symlink to /usr/portage/profiles/default/linux/amd64/17.1/no-multilib/hardened.
Code:
CFLAGS="-O2 -pipe -mtune=generic -march=x86-64"
PORTDIR="/usr/portage"
DISTDIR="/usr/portage/distfiles"
PKGDIR="/usr/portage/packages"
LC_MESSAGES=C
USE="-bindist -gd -mime -X -ipv6 -fonts -ldap -ncurses -nls -smartcard -unicode -cramfs -themes minimal symlink"
MAKEOPTS="-j9"
CPU_FLAGS_X86="mmx mmxext sse sse2 sse3"
GRUB_PLATFORMS="pc"
PYTHON_TARGETS="python3_6"
PYTHON_SINGLE_TARGET="python3_6"


Here is the initramfs init script:
Code:

#!/bin/busybox sh
touch /etc/mtab
mount -t devtmpfs none /dev
mount -t proc none /proc
rmode=ro
init=/sbin/init
if `grep writeable /proc/cmdline >/dev/null`; then rmode=rw; fi
if `grep openrc-init /proc/cmdline >/dev/null`; then init=/sbin/openrc-init; fi
mount /dev/vda1 /mnt/disk -o $rmode,noatime
if `grep oldimage /proc/cmdline >/dev/null`; then
echo "booting old image!"
mount /mnt/disk/oldsysimage.img /mnt/image -o $rmode,noatime
else
if [ -e /mnt/disk/newsysimage.img ]; then
mv /mnt/disk/sysimage.img /mnt/disk/oldsysimage.img
mv /mnt/disk/newsysimage.img /mnt/disk/sysimage.img
fi
if [ -e /mnt/disk/boot/newvmlinuz ]; then
mv /mnt/disk/boot/vmlinuz /mnt/disk/boot/oldvmlinuz
mv /mnt/disk/boot/newvmlinuz /mnt/disk/boot/vmlinuz
echo "new image applied, need to reboot!"
sync
umount /mnt/disk
reboot -f
fi
mount /mnt/disk/sysimage.img /mnt/image -o $rmode,noatime
fi
if [ -e /mnt/image/$init ]; then
echo "init found, starting it!"
mount --move /dev /mnt/image/dev
umount /proc
exec switch_root /mnt/image $init
else
echo "init not found!"
sh
fi

A generic .config for the kernel is hold in /build. If a target needs a special config, place it under its /build/targetname.
I could show you my kernelconfig if you like, but that would probably be a bit much in one post! :lol:

I guess I forgot to mention a bunch of things now, but you should get the idea.
Feel free to ask if something is not clear.
And please tell me if you have ideas to improve. :P

Cheers
_________________
Power to the people!
Back to top
View user's profile Send private message
1clue
Advocate
Advocate


Joined: 05 Feb 2006
Posts: 2562

PostPosted: Tue Mar 05, 2019 5:20 pm    Post subject: Reply with quote

Thanks for the post. I'm not ignoring you, I have a job and I also need to understand what you're talking about.

This is not what I was thinking, but it might be what I need. I need to read and digest. You being a dev, likely your approach is better than mine would have been.

Thanks.
Back to top
View user's profile Send private message
nativemad
Developer
Developer


Joined: 30 Aug 2004
Posts: 911
Location: Switzerland

PostPosted: Tue Mar 05, 2019 8:32 pm    Post subject: Reply with quote

Don't get fooled by my developer tag. I'm just a human too and it took me over a week to write (or more try and error) these scripts. :lol:

Maybe I should just answer your questions...
Quote:
1. How do I (can i) set up portage on a single system to build for several unique all-binary systems?

That is quite easy. portage knows ROOT= to install the result in a directory instead of /. There is also PORTAGE_CONFIGROOT, which tells portage where to look for instead of /etc/portage (which includes make.conf, the profile symlink, package.keywords and so on.
I organized it in /build/, with a directory for each target VM with a root and overlay directory therein. the /etc/portage part of the overlay gets copied to the root directory and portage compiles the minimum in there and the additions configured in build.conf.

Quote:
2. How do I make a binary-only host?

Just install the resulting root directory from step one. Not much different from a stage3 install actually.
My script creates a qcow2 image, creates one partition in it, copies the built root to it and installs grub. This Image would be complete, but i create another loop image with dd, format it and move everything (expect /boot) from the qcow2 image to the dd image. The dd image gets then copied in the qcow2 image. The initrd takes care of mounting the dd image.

Quote:
How do I make the VM host be binary-only when it's built by one of its guests?

Chicken Egg problem... Build a build VM first somewhere else (or on a working installation on that host that gets replaced after you've got the image ready for it).

Quote:
Is the boot-from-ram and buildbox-writes-the-minimal-roots idea feasible? Is there a better way?

Yes it is feasible. I just left the part out with boot-to-ram. There are several reasons why I don't do that.
-It needs aufs in kernel and in userspace (also in the initramfs). Without aufs (like the gentoo minimal cd does (or did?) it with tmpfs), it uses more ram for everything stored read/write.
-It is not really needed on a production server and might cause issues like with the oom killer or malicious changes and so on...

Just a bit of motivation:
I've got a jabber server with nginx that is using 74mb ram and the whole root weights 294mb (uncompressed). I guess it could be even smaller with a bit more effort. :wink:
_________________
Power to the people!
Back to top
View user's profile Send private message
1clue
Advocate
Advocate


Joined: 05 Feb 2006
Posts: 2562

PostPosted: Sun Mar 10, 2019 1:20 am    Post subject: Reply with quote

nativemad wrote:
Hi

I recently started to do a quite similar thing.
I have a buildscript that builds kvm-images out of some template files as well as an overlay that gets copied over the compiled rootimage (which is quite throwaway, as one can rebuild the whole thing quite automatically).


While I know what an overlay is, I have not yet messed with them. I don't really understand how they work in any detail. A bit nervous about doing it myself.

I like the idea that you can have a small bit of information to recreate a system. I would like more to have that small bit of information in a private git repository. I'm sure it can be done, but again I have little exposure to them.

Quote:

I do not use a ramdrive in the targetvm for root. Only /var is writeable on its own diskimage. The systemdrive holds a loop image that gets mounted as root. The initrd checks for new a imagefile (and kernel) and applies it upon boot. The initrd mounts the image read/write if the kernel parameter "writeable" is passed to it. That way you can try out changes on the fly to later put them in the overlay.


So this works sort of like a dhcp boot, only local? I might like that idea.

Another thing, even after having been a mostly-linux user since 1996 or so, I have never actually made an initrd. I tried once, never got it to work, and avoided it after that. I can't really see how to avoid it on this project though...

I like what you're doing here.

Quote:

I use openrc-init and quite much busybox.
I don not have a process to deploy and verify newly created images so far, they need to get copied to the target manually for now.
The targetroot gets compiled via ROOT=. The downside of this approach is that a few things leak to the buildhost! Namely portage added users and groups and eselect-php. I needed to recreate these things manually in the overlay.


I don't really see what systemd has to benefit this sort of system and all my Gentoo builds have been openrc, but I manage so many systemd based systems now (I use Ubuntu Server a lot, I manage a bunch of VMs) that I might try the systemd route on Gentoo just because it's getting more familiar now.

Please, this is a support thread and I do NOT want to introduce a systemd-openrc war here. I'm simply stating my position so that incoming help can understand my intent.

Re: Leaking. Have you tried not adding the users and groups from the build system but rather do it from the binary host?

Is there any advantage to an overlayfs? It seems like it would be slow to me.

Quote:

I'd start with a normal gentoo install in a vm and start building there.... The approach to deploy such images to bare metal should be quite the same.
I use a ramdrive on /mnt/ram to speed up buildtimes.


I could make a build box on the system right now, and then make a host image from that. I would almost be tempted to use the existing host install, but it's been going for awhile and I would want to clean up a bunch of stuff. I think I'll do a fresh VM.

One thing that would be really nice is if I can test the host install as a VM without doubling the install footprint. Concerns:

  1. I don't know if I can have a VM which is also workable as a bare-metal install.
  2. I don't know about a vm guest acting like a vm host. Recursive virtual machines? I know it's theoretically viable but I would be worried about re-starting a guest which is already running on the real host. I would want to be able to test that guests run, but don't want to spiral out of control.


While I'm at it, I would like to install the host root as an lvm2 volume. I've been using a real partition all this time, and it's one of the most annoying things about Linux. It's time to get over my fear of an initrd and make an lvm2-only system. I don't want to get into the details of that here, I can use Google. Unless you have some sort of opinion about doing that.

Quote:

Here is my buildscript (forgive me, it is far from perfect):


Left it out, and haven't looked yet.

Quote:

Here is the initramfs init script:


YAY! I'll definitely look closer at that.

Quote:

A generic .config for the kernel is hold in /build. If a target needs a special config, place it under its /build/targetname.
I could show you my kernelconfig if you like, but that would probably be a bit much in one post! :lol:


I don't need your kernel config. I have my own and am pretty familiar with that sort of thing. Thanks though.

Quote:

I guess I forgot to mention a bunch of things now, but you should get the idea.
Feel free to ask if something is not clear.
And please tell me if you have ideas to improve. :P

Cheers


Last edited by 1clue on Sun Mar 10, 2019 1:45 am; edited 2 times in total
Back to top
View user's profile Send private message
1clue
Advocate
Advocate


Joined: 05 Feb 2006
Posts: 2562

PostPosted: Sun Mar 10, 2019 1:32 am    Post subject: Reply with quote

nativemad wrote:
Don't get fooled by my developer tag. I'm just a human too and it took me over a week to write (or more try and error) these scripts. :lol:


I'm a developer too, just not for a Linux distro. I know how it is. Sometimes you get the bear, and sometimes the bear gets you.

Quote:

Maybe I should just answer your questions...
Quote:
1. How do I (can i) set up portage on a single system to build for several unique all-binary systems?

That is quite easy. portage knows ROOT= to install the result in a directory instead of /. There is also PORTAGE_CONFIGROOT, which tells portage where to look for instead of /etc/portage (which includes make.conf, the profile symlink, package.keywords and so on.
I organized it in /build/, with a directory for each target VM with a root and overlay directory therein. the /etc/portage part of the overlay gets copied to the root directory and portage compiles the minimum in there and the additions configured in build.conf.


Very nice! I'll be trying that soon.

Quote:

Quote:
2. How do I make a binary-only host?

Just install the resulting root directory from step one. Not much different from a stage3 install actually.
My script creates a qcow2 image, creates one partition in it, copies the built root to it and installs grub. This Image would be complete, but i create another loop image with dd, format it and move everything (expect /boot) from the qcow2 image to the dd image. The dd image gets then copied in the qcow2 image. The initrd takes care of mounting the dd image.


I was thinking to use lvm2 volumes as guest hard disks, and then install raw to that. This is on an 8-core atom, which is faster than you might think but it's still an atom.

That said it would be nice to be able to duplicate this entire setup (build box VM, all the guests and the host) on another architecture and have it figure out the differences automatically. So I could do it for example on an i7 or e3 or something.

Quote:

Quote:
How do I make the VM host be binary-only when it's built by one of its guests?

Chicken Egg problem... Build a build VM first somewhere else (or on a working installation on that host that gets replaced after you've got the image ready for it).


Actually once I thought about it, it's not a huge problem at all. I already have Gentoo on the box. I can make a VM which will be my build box, write an LVM2 disk which will be my new host, and figure out how to boot from that on the raw hardware. I know it can be done because I have Google.

Quote:

Quote:
Is the boot-from-ram and buildbox-writes-the-minimal-roots idea feasible? Is there a better way?

Yes it is feasible. I just left the part out with boot-to-ram. There are several reasons why I don't do that.
-It needs aufs in kernel and in userspace (also in the initramfs). Without aufs (like the gentoo minimal cd does (or did?) it with tmpfs), it uses more ram for everything stored read/write.
-It is not really needed on a production server and might cause issues like with the oom killer or malicious changes and so on...


How do you get around the problem of two operating systems writing to the same partition simultaneously? And for that matter how do you write your images from the build box? Do you mount the other filesystems as 9p? Or do you require that the guest be powered down and then write? That might not be feasible.

I guess I could, for the guests anyway, create and write a new lvm2 volume for the guest and then have some sort of magic for the host to figure out? Maybe your bit that looks for a new image in your initrd?

Quote:

Just a bit of motivation:
I've got a jabber server with nginx that is using 74mb ram and the whole root weights 294mb (uncompressed). I guess it could be even smaller with a bit more effort. :wink:


That's extremely nice. I think a whole lot of VMs could be incredibly tiny. For something like a firewall with remote logging it seems perfectly feasible. I would need RAM, but not necessarily much disk space.
Back to top
View user's profile Send private message
1clue
Advocate
Advocate


Joined: 05 Feb 2006
Posts: 2562

PostPosted: Sun Mar 10, 2019 1:40 am    Post subject: Reply with quote

What would be really nice is if I could quantify everything which makes each host special in a git repository, including kernel config, initrd scripts, services installed, their configurations and data, unless it's a large amount of data. Big data would have to be stored and backed up, or something.

But surely for a firewall I could have network config, iptables rules, and logging configuration in a private git repo on my network.
Back to top
View user's profile Send private message
nativemad
Developer
Developer


Joined: 30 Aug 2004
Posts: 911
Location: Switzerland

PostPosted: Sun Mar 10, 2019 4:12 pm    Post subject: Reply with quote

Git should be really easy... Just create a git repo out of /build and add */root */target */targetimg in .gitignore. That way it only tracks your manual changes with the final images and not the root that got compiled.

Lvm shouldn't be a problem either to use instead of qcow2 images... Just use the parameter "image" in the buildscript. Create the lvm volumes manually and just copy the built sysimages in there. For an update I manually mount the physical volume read/write (within the target-vm) and scp the new image/kernel/initrd there before a reboot applies them. I'm thinking about adding network/git support to the initrd, so that an update could get fetched during boot... But that would complicate the initrd quite a bit....
I only have one VM per image and no concurrent access from host and VM.

If you look at my posted make.conf, you'll see that the cflags are very generic, as I have AMD and Intel hosts where I move the VMs around. If everything runs on an ATOM (or even Intel), you probably want to be more specific... Check out qemus "-cpu" parameter, you might get around with some flag filtering and "-march=native"! :wink:

You could probably build a NAT or Host-only network on the build-VM and test the new VM images there without affecting the actual Network. VMs in a VM needs the vmx cpu flag (on the host-VM) and is called nested virtualization! :wink:

I only use the term "overlay" for a directory to store the changes that are needed to an ootb gentoo installation. That way you could also have several overlays for one root-build, if only things like hostname, ips and such are different (for a webfarm or such). That "overlay" has nothing to do with a portage overlay! :wink:

Quote:
Re: Leaking. Have you tried not adding the users and groups from the build system but rather do it from the binary host?
With a readonly-root, the passwd files need to be setup before a boot happens! So you need to copy the entries from the passwd and group files (maybe even shadow, if you have users with a password) from the build-host to the target-image during the image-buildtime (like other things in the overlay). Those users are actually needed on the buildhost so that chown works in the target-root-directory.

An overlayfs is only useful if you want some parts writeable out of a readonly media. You basically create a ramdrive and "lay it over" the desired read-only parts. Writes there get written in the ramdrive. To make livecds with persistent modifications possible, a common practice is to use a directory on the usb-drive instead of a ramdrive. As I want to be able to rebuild everything from scratch for different arches and such, it's a bad idea to store changes to configs on the VM in an overlayfs....I also doubt that such an overlay would survive a rebuild/update of the sysimage!? On the other hand I keep the actual data in /var on a second disk or even on a third disk for /data or such.

...If you know how to configure a kernel and use proper cflags, there is basically only the root= kernel parameter that needs to be different between a VM and a Host setup! Okok, maybe the "persistent" network-naming should get disabled, so that the interfaces are having the same name... :lol:

Openrc used to depend on sysvinit. Nowadays, there is openrc-init, which can replace sysvinit completely. That is what I use here (a package less). I guess any other init systems (or whole stacks) could also be used quite easily. I guess you know which name I tried to avoid here! :lol:

I started to build and use that build-environment and method quite recently. In an ideal world, a git-commit-hook as well as a cronjob (that syncs portage and checks for updates in the targets) would trigger a rebuild of the affected targetroots, create a sysimage and kernel and offer that result over git. The targetvms would check for updated files via git, download them and reboot the VMs on defined times via cron.
I really just don't know yet how to implement some kind of signing/verification during the boot process. One thing I've done was enabling the module signing from the kernel and using the virtio-net driver as module. That way the kernel needs to be booted with a valid root, or there will be no network. That is a tiny start, but some additional hash-checks like simple md5sums over the systemimage would be cool. Or some openssl signing with public keys over git!? :roll:
We get in another Chicken/Egg problem with git itself. I use gitea in a VM. And storing your keys and such on someone elses computer might be a bad idea too... :roll:
_________________
Power to the people!
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Installing Gentoo All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum