Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
QEMU/KVM unable to boot amd64 guests on amd64 host
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware
View previous topic :: View next topic  
Author Message
sylence
n00b
n00b


Joined: 08 May 2020
Posts: 10

PostPosted: Fri May 08, 2020 2:20 pm    Post subject: QEMU/KVM unable to boot amd64 guests on amd64 host Reply with quote

Hey,

I'm struggling with my QEMU/KVM setup. The Gentoo system in question is more than 10 years old and was continuously updated, this could very well be an issue of me having messed up something at some point. Hardware-wise I'm running on an Intel Core2Quad CPU.

My main issue is that QEMU with enabled KVM works fine with 32 bit guests, but is unable to boot anything that is built for 64 bits. I simplified the problem to simply invoking
Code:
qemu-system-x86_64 --enable-kvm -m 384 -boot d -cdrom debian-10.3.0-amd64-netinst.iso

This will start the VM, the boot menu will appear, but booting any 64bit linux kernel like that will simply reboot the virtual machine without any error message. I verified that by manually invoking the kernel from the GRUB2 shell:
Code:
linux /boot/.../vmlinuz
boot

will just lead to an immediate restart of the VM.
Playing around with various QEMU settings such as "-cpu host" didn't change that, however as soon as I use that switch to simulate a 32 bit CPU, e.g. "-cpu pentium3", any attempt to boot a linux kernel will hang with a message that I need a 64 bit CPU. Which leads me to believe that my regular VM is a 64bit one and something else is wrong here. Creating the VM without KVM support (without --enable-kvm) "just works" with all my 64 bit ISOs. Moreover, with enabled KVM, 32 bit guests work correctly as well. I tried various ISOs for different operating systems, but none of those 64 bit images work: Windows bluescreens with "IRQL_NOT_LESS_OR_EQUAL", Solaris just reboots without any errors.

I'm currently running QEMU 4.2.0 and kernel 4.19.82 with KVM support:
Code:
CONFIG_HAVE_KVM=y
CONFIG_HAVE_KVM_IRQCHIP=y
CONFIG_HAVE_KVM_IRQFD=y
CONFIG_HAVE_KVM_IRQ_ROUTING=y
CONFIG_HAVE_KVM_EVENTFD=y
CONFIG_KVM_MMIO=y
CONFIG_KVM_ASYNC_PF=y
CONFIG_HAVE_KVM_MSI=y
CONFIG_HAVE_KVM_CPU_RELAX_INTERCEPT=y
CONFIG_KVM_VFIO=y
CONFIG_KVM_GENERIC_DIRTYLOG_READ_PROTECT=y
CONFIG_KVM_COMPAT=y
CONFIG_HAVE_KVM_IRQ_BYPASS=y
CONFIG_KVM=m
CONFIG_KVM_INTEL=m


PS: 64bit guests in Virtualbox run fine with VT-x enabled.

Any help is greatly appreciated!
Back to top
View user's profile Send private message
mike155
Advocate
Advocate


Joined: 17 Sep 2010
Posts: 2126
Location: Frankfurt, Germany

PostPosted: Fri May 08, 2020 3:51 pm    Post subject: Reply with quote

Hi sylence,

welcome to the Gentoo forums!

Is virtualization enabled in the BIOS? It is/was disabled by default, at least on older machines.

Mike
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 15324

PostPosted: Sat May 09, 2020 12:22 am    Post subject: Reply with quote

Are you sure that the 64-bit guest you are trying to run is actually compatible with your real CPU? When using software emulation without KVM, qemu can emulate instructions that you do not have. When using KVM acceleration, I am not sure if qemu will fall back to emulating instructions that are not natively supported by the CPU.
Back to top
View user's profile Send private message
sylence
n00b
n00b


Joined: 08 May 2020
Posts: 10

PostPosted: Sun May 10, 2020 1:03 pm    Post subject: Reply with quote

Hey,

mike155 wrote:
Is virtualization enabled in the BIOS? It is/was disabled by default, at least on older machines.

Yes, I just verified that - it's enabled. If I understood the documentation correctly, KVM would complain in case it's disabled.

Hu wrote:
Are you sure that the 64-bit guest you are trying to run is actually compatible with your real CPU? When using software emulation without KVM, qemu can emulate instructions that you do not have. When using KVM acceleration, I am not sure if qemu will fall back to emulating instructions that are not natively supported by the CPU.

Well, the ISOs I'm trying to boot are mostly just regular x86_64 Linux distributions such as the Debian installer or the grml live CD. The 64bit version of Windows 7 doesn't work as well (bluescreen - see my initial post). All of those should run fine on my Core2Quad CPU. The crashes and reboots happen with '-cpu host' (passthrough) as well as '-cpu qemu64' or any other 64 bit option in the built-in CPU list. However, a 32bit ISO boots correctly with any 32 bit CPU (such as '-cpu qemu32'), even with KVM enabled.

I'm really out of ideas here. :(
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 45435
Location: 56N 3W

PostPosted: Sun May 10, 2020 1:41 pm    Post subject: Reply with quote

sylence,

Do you have a 64 bit install on the host?

Most virtualisers will allow you to run 64 bit guests on 32 bit hosts, but not all of them.
What does the hosts
Code:
uname -a
show.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
sylence
n00b
n00b


Joined: 08 May 2020
Posts: 10

PostPosted: Sun May 10, 2020 1:52 pm    Post subject: Reply with quote

Yeah, the host's Gentoo installation is 64 bit:
Code:
Linux shade 4.19.82-gentoo.shade.v2 #2 SMP Thu Apr 9 12:49:07 CEST 2020 x86_64 Intel(R) Core(TM)2 Quad CPU Q9550 @ 2.83GHz GenuineIntel GNU/Linux
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 45435
Location: 56N 3W

PostPosted: Sun May 10, 2020 1:55 pm    Post subject: Reply with quote

sylence,

Do you have
Code:
$ ls -l /dev/kvm
crw-rw---- 1 root kvm 10, 232 Dec 30  2017 /dev/kvm


and is the user running qemu-system-x86_64 in the kvm group?
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
sylence
n00b
n00b


Joined: 08 May 2020
Posts: 10

PostPosted: Sun May 10, 2020 2:17 pm    Post subject: Reply with quote

Looks fine (the user running qemu is 'sylence'):
Code:
# ls -l /dev/kvm
crw-rw---- 1 root kvm 10, 232 Mai 10 15:19 /dev/kvm
# grep kvm /etc/group
kvm:x:78:qemu,sylence
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 45435
Location: 56N 3W

PostPosted: Sun May 10, 2020 2:21 pm    Post subject: Reply with quote

sylence,

That's fine if sylence has logged out and back in again since being added to the kvm group.

If kvm is not listed in groups, when run as sylence, thats a problem.
Code:
$ groups
tty wheel uucp audio cdrom video games kvm cdrw usb users vboxusers scanner wireshark plugdev roy

_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
sylence
n00b
n00b


Joined: 08 May 2020
Posts: 10

PostPosted: Sun May 10, 2020 2:54 pm    Post subject: Reply with quote

Looks correct as well.
Code:
# groups
wheel cdrom video games kvm usb users libvirt docker sylence plugdev qemu vboxusers wireshark


In the meantime, I removed the VirtualBox kernel modules I had installed as well, just to be sure: So far, same behaviour.
Back to top
View user's profile Send private message
mike155
Advocate
Advocate


Joined: 17 Sep 2010
Posts: 2126
Location: Frankfurt, Germany

PostPosted: Sun May 10, 2020 3:17 pm    Post subject: Reply with quote

Looks good so far...
  1. Please post the output of
    Code:
    emerge --info

  2. Please post your kernel config using wgetpaste.
Back to top
View user's profile Send private message
sylence
n00b
n00b


Joined: 08 May 2020
Posts: 10

PostPosted: Sun May 10, 2020 3:39 pm    Post subject: Reply with quote

Kernel config
emerge --info
Back to top
View user's profile Send private message
mike155
Advocate
Advocate


Joined: 17 Sep 2010
Posts: 2126
Location: Frankfurt, Germany

PostPosted: Sun May 10, 2020 4:14 pm    Post subject: Reply with quote

Thanks for the data. I couldn't find anything wrong in the output of 'emerge --info' and your kernel config.

You seem to build KVM as modules:
Code:
CONFIG_KVM=m
CONFIG_KVM_INTEL=m

Are all KVM modules loaded when you start QEMU/KVM? Have you tried to compile KVM directly into the kernel?
Back to top
View user's profile Send private message
sylence
n00b
n00b


Joined: 08 May 2020
Posts: 10

PostPosted: Sun May 10, 2020 4:17 pm    Post subject: Reply with quote

If by all you mean all two of them (kvm and kvm_intel), then yes, those are loaded when I'm starting QEMU.
I haven't tried yet to build them into the kernel and can try to do that, but would be surprised if that changes anything.
Thanks for having a look at my configuration, though. :)
Back to top
View user's profile Send private message
alamahant
Apprentice
Apprentice


Joined: 23 Mar 2019
Posts: 282

PostPosted: Sun May 10, 2020 4:54 pm    Post subject: Reply with quote

Do you have libvirt also installed?
Maybe if you had a full qemu/libvirt/virt-manager setup and created a guest graphically?
Sometimes trying to be smartly minimal in many things ends up inviting more problems than not..........
I have seen this in USE flags for example, the kernel .config flags and in many other places...
We strive for smart seek minimal setups(which is a fantastic thing),but some times we break things up like this.................
:)
Back to top
View user's profile Send private message
sylence
n00b
n00b


Joined: 08 May 2020
Posts: 10

PostPosted: Sun May 10, 2020 5:15 pm    Post subject: Reply with quote

I do have libvirt and virt-manager installed. That's where I noticed that something is wrong - my original goal was to set up an Android VM, which didn't work. Only then I noticed that I can't boot anything that's x86_64. The minimal QEMU command above is just my attempt to reduce the problem to something tangible. Sadly, the 'normal' way of managing KVM VMs doesn't work either. ;)
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 45435
Location: 56N 3W

PostPosted: Sun May 10, 2020 5:24 pm    Post subject: Reply with quote

sylence,

What error do you get from virtmanager?
I'll guess, none at all, the guests just reboot.

That points the finger at the host. I have an AMD and Intel system that are KVM hosts. I can share the kernel .configs if you want.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
sylence
n00b
n00b


Joined: 08 May 2020
Posts: 10

PostPosted: Sun May 10, 2020 5:33 pm    Post subject: Reply with quote

NeddySeagoon wrote:
What error do you get from virtmanager?
I'll guess, none at all, the guests just reboot.

Correct, no error anywhere. The guest just reboots (or bluescreens in the case of a Windows guest). I was playing around with QEMUs logging options ("-d") for invalid opcodes (amongst others), but didn't get anything from there either. I'm not sure if another kernel config would help. Guess I'll try to two things now: Building a 5.x kernel (maybe it's really just a very specific kernel bug) and booting some other distribution to see whether KVM works from there. Could as well be a hardware issue with the VT-X extension.
Back to top
View user's profile Send private message
mike155
Advocate
Advocate


Joined: 17 Sep 2010
Posts: 2126
Location: Frankfurt, Germany

PostPosted: Sun May 10, 2020 6:26 pm    Post subject: Reply with quote

You could also try a BIOS update. I found some reports from users with a Core 2 CPU who reported that virtualization worked for them only after a BIOS update.

12 years ago, virtualization was quite new and mainboard and BIOS manufactures had to puzzle out how to use it...
Back to top
View user's profile Send private message
ununu
n00b
n00b


Joined: 19 Apr 2020
Posts: 31

PostPosted: Sun May 10, 2020 7:59 pm    Post subject: too little ram Reply with quote

greets,

this could be due to little RAM on your qemu options

Quote:
-m 384


I think 64 bit distros needs a little bit more due to KASLR or something like that.

Also make sure that qemu is built with:

Code:
QEMU_SOFTMMU_TARGETS="i386 x86_64"
QEMU_USER_TARGETS="i386 x86_64"


I forgot to add this:

Code:
-cpu core2duo

or
Code:
-cpu Nehalem


sylence wrote:
Hey,
...
system is more than 10 years old ... running on an Intel Core2Quad CPU.


:cough:AMD:cough: <---
Back to top
View user's profile Send private message
sylence
n00b
n00b


Joined: 08 May 2020
Posts: 10

PostPosted: Sun May 10, 2020 9:46 pm    Post subject: Reply with quote

Problem solved, even though I'm still not sure what exactly the problem was.

I tried various things, all of them without success:
* BIOS is up to date (I used VT-x a lot previously with VirtualBox, under Windows etc., so that should work)
* Not enough RAM: The problem persisted also with 2GB or 4GB of RAM (-m 2048, -m 4096)
* My QEMU_*_TARGETS are both set to just "x86_64", which should give me at the very minimum support for 64 bit
* Different "-cpu" settings didn't help

I verified that it's not a hardware problem by booting another USB live linux (also with kernel 4.19) and running QEMU/KVM from there (same command, same ISO) - this worked fine. As a last resort I finally upgraded my kernel to 5.4.38. I built that one with the exact same config as I used before for 4.19.x (make olddefconfig). Now it works fine, no idea what was wrong with the old one. I also tried to integrate KVM into the kernel or build it as a set of modules - no difference.

Well, thanks for all the suggestions! :)
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum