Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Building A Xen Server At The Outset
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Installing Gentoo
View previous topic :: View next topic  
Author Message
jlpoole
Guru
Guru


Joined: 01 Nov 2005
Posts: 344
Location: Salem, OR

PostPosted: Tue Nov 08, 2016 1:25 pm    Post subject: Building A Xen Server At The Outset Reply with quote

I'm going to build a Xen server on a new machine, a low wattage Intel Atom-based unit: 8 cores, 64 GB RAM. The unit arrived yesterday and I thought I might try an ambitious approach of building a XEN server from the outset using all Gentoo. In other words, the Dom0 would be Gentoo based. Previously, I built a Xen server using the Xen Project Beginner's Guide and I had something up and running in about 2 hours. I then had Gentoo clients thereon. The Guide specifies using Debian. I am very fond of Gentoo, so I thought I would try to build the server with all Gentoo in the same manner as the Guide, but supplanting Gentoo for Debian.

The first question I found myself wondering about: does the Gentoo Xen Wiki presuppose an already-built Gentoo system? The answer appears to be "yes." Despite that, I have tried to blend together the Wiki with the Gentoo Handbook so the installation is one step, but I have run into various snags, much of which surround my choice for configurations at various steps. Example, I chose ext2 for /boot, but when trying to install Grub2, Grub2 aborts with " will not proceed with blocklists."

As I'm now several hours into this attempt, the temptation of the Xen-Debian install becomes more enticing -- I can have a ready-working hypervisor and spend my quality time working on the guest/clients. So, I thought I would post here to register my interest in having an install guide with the goal of creating a Xen hypervisor running on Gentoo.

I think it would be a credit to Gentoo if there were an installation guide that builds the hypervisor in one step. I'd volunteer using my machine as a test subject, but I don't have the luxury of time to keep it out of production. I guess this is all to say I'm someone who would like to build a Gentoo based XEN server without having to build a regular system and then convert it to a Xen server. Perhaps this is unrealistic?
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 43194
Location: 56N 3W

PostPosted: Tue Nov 08, 2016 4:18 pm    Post subject: Reply with quote

jlpoole,

Code:
Grub2 aborts with " will not proceed with blocklists."
is not due to your choice of ext2 for boot. You have used GPT and not told grub about its 2Mb partition.

MSDOS disk labels, left some free space between the partition table and the start of the the first partition. Over the years, boot loaders and other things have helped themselves to this space.
legacy grub, or grub2 with an MSDOS disk label, uses this area.

With a GPT disk label, the first copf of the partition table starts in LBA 1. There is no free space. Grub2 still needs to install this code though, which is what the 2Mb BIOS boot partition is for.

Block lists are fragile but can be made to work. That's the way lilo loads the kernel and initrd.
grub, (both of them) read the /boot filesystem, they do not load the contents of a list of blocks and hope the filesystem has not changed since the block list was created.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
jlpoole
Guru
Guru


Joined: 01 Nov 2005
Posts: 344
Location: Salem, OR

PostPosted: Tue Nov 08, 2016 5:40 pm    Post subject: Reply with quote

Thanks, Neddy. I'm installing Debian now to move forward.

Has anyone built a Xen server using Gentoo on an initial build?
Back to top
View user's profile Send private message
1clue
Advocate
Advocate


Joined: 05 Feb 2006
Posts: 2549

PostPosted: Tue Nov 08, 2016 6:09 pm    Post subject: Reply with quote

I'm interested in your project.

I built a box awhile back off of this: http://www.supermicro.com/products/motherboard/Atom/X10/A1SRM-LN7F-2758.cfm with 16g RAM.

It's a kvm host running Gentoo. My intent is to host pfSense and Gentoo guests for layered firewall, and possibly more as performance allows. The guests are on a slow back burner, it's been almost 2 years now.

The system works, but as yet if I force the host to ignore the extra interfaces then the guests seem to ignore them too, due to no vt-d support.

I think we're pretty close architecturally. I'd really like to see how this goes. I'm also interested in the differences between kvm and xen, I never really figured xen out.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 43194
Location: 56N 3W

PostPosted: Tue Nov 08, 2016 6:25 pm    Post subject: Reply with quote

Team,

I've done it with KVMs on an HP-Microserver.
Its on the wayback machine as I've never ported it to the wiki.

My network interfaces are all bridged an the four port NIC I have has a hardware bug that prevents me using passthrough.
Still, I don't need to assign IPs to those interfaces that are not required on the host.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
Atom2
Apprentice
Apprentice


Joined: 01 Aug 2011
Posts: 185

PostPosted: Tue Nov 08, 2016 8:25 pm    Post subject: Reply with quote

jlpoole,
jlpoole wrote:
Has anyone built a Xen server using Gentoo on an initial build?
I have done this and its running nicely in a production environment. The hardware is based on an Intel S2600CP4 motherboard with 2 Xeon E5-2620 CPUs (2x6 cores, total 24 w/ hyperthreading) and 112GB ECC RAM. The NUMA node 0 has access to 64GB local RAM and the NUMA node 1 has access to 48GB local RAM.

The system boots off a software RAID1 LVM partition using ext2 as the filesystem. The root filesystem for dom0 is btrfs (btrfs RAID1 setup). The main "disks" for dom0 are based on SSDs (one Samsung 850 Pro and one Samsung 840 Pro, each 512GB in size). Furthermore dom0 has access to two 500GB spinning disks (WD RE in a software RAID1 setup) used for mostly write data (i.e. portage tree, log files, packages, etc) to save write cycles on the SSDs. All disks are using LVM; SSDs als use thin-provisioning. Shared directories (like the portage tree for gentoo domUs/dom0) are shared via NFS4. All systems (except Win7) log to dom0 via syslog-ng and all domUs (except Win7) display their console output into a GNU-screen session on dom0.

The server setup is roughly as follows:
0.) dom0 is gentoo hardened sources (latest stable kernel) using the 4 onboard NICs as one bonded device; all domUs are bridged to the bonded device
1.) HVM domU pfsense based on FreeBSD (with 2 Intel NICs as PCI-passthrough) acting as the internet gateway/firewall through ADSL (1st NIC) and providing DNS and DHCP to all client PCs (all domU servers use fixed IP addresses); connected is also a WiFi guest AP through the 2nd NIC which allows Internet access only (the internal WiFi AP is directly connected to the switch)
2.) HVM domU sambadc1 (FreeBSD based) samba AD domain controller 1; also provides DNS services for the samba sub-domain
3.) HVM domU sambadc2 (FreeBSD based) samba AD domain controller 2; also provides DNS services for the samba sub-domain
4.) HVM domU sambafs1 (FreeBSD based) samba member file server (with 48GB RAM) using ZFS as the filesystem (RAIDZ2 storage pool with the onboard SCU SATA controller passed through) - currently 4 disks, effectively RAID6
5.) HVM domU poudriere (FreeBSD based) used to nightly sync and build (in tmpfs) FreeBSD packages from the sources and creating a package repository
6.) PV domU master (gentoo based) which contains all (unconfigured - i.e. only emerged) packages needed by any of the gentoo domUs; this image is snapshoted (LVM) and made available to all gentoo domUs as the r/o lowerdir of an overlayfs filesystem with a small r/w 1GB overlay for storing domU specific configuration data
7.) PV domU portage (gentoo based) which is used as a build host for all gentoo VMs and syncs the tree; all required packages are built nightly in a tmpfs based chroot environment (including catering for different use flags between dom0 and domUs); this is the only gentoo domU that's not based off the common master image, but rather uses a RAID1 btrfs root filesystem and a RAID1 ext2 boot partition.
8.) PV domU smtp (gentoo based) running exim (virtual mail server setup - users don't require linux logons) as the SMTP server (in- and outbound); mtaspool space is provided through NFS shared space from the ZFS pool on sambafs1
9.) PV domU imap (gentoo based) running dovecot as the IMAP server; storage for the IMAP server is provided through NFS shared space from the ZFS pool on sambafs1
10.) PV domU pki (gentoo based; mostly offline) used to create and maintain a self-signed PKI infrastructure using a self-developped script
11.) PV domU icinga (gentoo based) running icinga used to monitor all VMs and other system (e.g. printers, WIN7 workstations); total monitored hosts: 32; total monitored services: 358
12.) HVM domU win7 (Windows 7 based) planned for remote access via VPN to the site (RAS is not implemeted yet) and also integrated into the samba AD domain
13.) PV domU mysql (gentoo based) running the a mysql server providing databases used for icinga, exim, imap and spamassassin/clamav
14.) PV domU www (gentoo based) running apache for icinga and a few other applications
15.) PV domU milter (gentoo based) running spamassassin and clamav for mail filtering (in- and outbound)

dom0, pfsense, and sambafs1 run on NUMA node 0, the rest is running on NUMA node 1.

All running VMs (except Win7) are subject to nightly backups to the ZFS storage pool using rsync and keeping 14 daily snapshots, 5 weekly snapshots, 12 monthly snapshots, and all yearly snapshots. Furthermore there's a monthly copy of the local data from the overlay to the second SSD (840 Pro) that uses the same layout as the first.

To startup and stop all VMs in the correct sequence I have heavily modified XEN's OpenRC startup scripts and I use OpenRC's native dependency system as the basis. Dependencies are described in a configuration file using "need" "provide" "after" and "before" descriptive statements. Communication between the domUs and the dom0 to signal "VM ready" (which us required for startup and dependencies) is based upon socat on a pre-defined port using PKI certificates.

VMs (plus WIN7 client workstations) also signal up-status and down-status to the icinga monitoring system to start/stop monitoring these hosts.

So in a nutshell, this is doable and XEN together with gentoo is very stable indeed. As always, there's clearly room for improvement and I am planning on using a simular setup at home to mimic the configuration. I have a few ideas that I would like to implement, but that probably takes a bit of time before it is up and running.

I hope I haven't left out anything important from my setup. Atom2

_________________________________________
Edit: corrected a few typos and added formatting


Last edited by Atom2 on Tue Nov 08, 2016 10:23 pm; edited 3 times in total
Back to top
View user's profile Send private message
jlpoole
Guru
Guru


Joined: 01 Nov 2005
Posts: 344
Location: Salem, OR

PostPosted: Tue Nov 08, 2016 8:42 pm    Post subject: Reply with quote

Atom2, I am humbled by what you have accomplished. I'm not clear, though, on whether you configured your server first by installing Gentoo in the normal way and then modifying it to be a Xen server, or whether you started with a bare metal platform and performed your own Gentoo-Handbook-like Xen install so that your first boot-up was a Dom0.
Back to top
View user's profile Send private message
Atom2
Apprentice
Apprentice


Joined: 01 Aug 2011
Posts: 185

PostPosted: Tue Nov 08, 2016 8:55 pm    Post subject: Reply with quote

jlpoole,
jlpoole wrote:
I'm not clear, though, on whether you configured your server first by installing Gentoo in the normal way and then modifying it to be a Xen server, or whether you started with a bare metal platform and performed your own Gentoo-Handbook-like Xen install so that your first boot-up was a Dom0.
I started installing gentoo by following the handbook and configured it (e.g. LVM, configuration files, NICs, bonding, etc.) so that it worked as a real gentoo system; BTW this gentoo system is still available as a boot entry under grub2 using the exact same kernel as the dom0 under xen, but not starting the hypervisor before booting the kernel. If started as a standard gentoo system, it is just not making use of the embedded xen code in the kernel.

After that it was just changing the kernel to include the required configuration options for XEN dom0 (my dom0 does not contain any frontend code, only the backend code required for dom0; the frontend code is only required for the domUs), recompiling and installing the kernel and then emerging xen and xen-tools. I am not really aware of any other option - to install (read: emerge) the xen packages you need a running linux (read: gentoo) system. In my view there's no other way, so I don't really get what you have in mind, but I am eager to learn ...

Finally it was a matter of adding XEN entries to the grub2 bootloader and startup XEN - which basically resulted in a working dom0 running under the XEN hypervisor. From there on it was all about creating the domUs and configuring/orchestrating the system in such a way that it all plays together nicely.
Back to top
View user's profile Send private message
jlpoole
Guru
Guru


Joined: 01 Nov 2005
Posts: 344
Location: Salem, OR

PostPosted: Tue Nov 08, 2016 10:13 pm    Post subject: Reply with quote

Atom2, thank you for the clarification.

Atom2 wrote:
jlpoole,
...
In my view there's no other way, so I don't really get what you have in mind, but I am eager to learn ...


What I tried was using the DVD to start up an environment, partition the disk, and then unzip an image into a mount root -- the standard approach. But, once in the chrooted environment, then using emerge to add the xen packages and compile the kernel as if I had already successfully installed a Gentoo system on this machine so at the first reboot I'm into a Dom0. As I'm doing the recommended Debian install, I see that it, too, is a two stage process: 1) get a Debian system up and running, then 2) create the Dom0. I guess I recalled when I built a Xen server three years ago it was all done in one install -- it was so much faster than a carefully-considered Gentoo install.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Installing Gentoo All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum