Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Crocodillian's Yet Another Useless VFIO Guide
View unanswered posts
View posts from last 24 hours

 
This topic is locked: you cannot edit posts or make replies.    Gentoo Forums Forum Index Kernel & Hardware
View previous topic :: View next topic  
Author Message
Crocodillian
n00b
n00b


Joined: 16 Oct 2018
Posts: 3
Location: San Francisco, CA, United States

PostPosted: Sun Oct 28, 2018 8:24 am    Post subject: Crocodillian's Yet Another Useless VFIO Guide Reply with quote

In this guide I will explain how to set up a windows VM with GPU passthrough
using virt-manager.

These are not the nicest way to do things, but it will get you up and running.

Big thanks to the VFIO discord guys, without whom I would have never gotten
anything to work:

discord.gg/utg3DG

I'm using lots of information from the arch wiki page:

https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF

First, let's check if your hardware supports IOMMU.

Make sure IOMMU or VT-d is enabled in your EFI settings.

Let's add some kernel parameters.

This is generally GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub.

I have the following:

amd_iommu=on iommu=pt vfio_iommu_type1.allow_unsafe_interrupts=1

the last two are optional.

For intel use intel_iommu=on.

Next we are going to enable some things in the kernel.

You can just edit /usr/src/linux/.config and then run genkernel --oldconfig.

You can also use make menuconfig and search for options with /, you get back a
result list and you can press the associated number to jump to that section in
menuconfig.

Check that these options are enabled in your kernel:

# iommu options
CONFIG_IOMMU_API=y
CONFIG_IOMMU_SUPPORT=y
CONFIG_IOMMU_IOVA=y
CONFIG_AMD_IOMMU=y # for AMD
CONFIG_AMD_IOMMU_V2=y # for AMD
CONFIG_INTEL_IOMMU=y # for intel

# kvm options
CONFIG_HAVE_KVM=y
CONFIG_HAVE_KVM_IRQCHIP=y
CONFIG_HAVE_KVM_IRQFD=y
CONFIG_HAVE_KVM_IRQ_ROUTING=y
CONFIG_HAVE_KVM_EVENTFD=y
CONFIG_KVM_MMIO=y
CONFIG_KVM_ASYNC_PF=y
CONFIG_HAVE_KVM_MSI=y
CONFIG_HAVE_KVM_CPU_RELAX_INTERCEPT=y
CONFIG_KVM_VFIO=y
CONFIG_KVM_GENERIC_DIRTYLOG_READ_PROTECT=y
CONFIG_KVM_COMPAT=y
CONFIG_HAVE_KVM_IRQ_BYPASS=y
CONFIG_VIRTUALIZATION=y
CONFIG_KVM=m
CONFIG_KVM_INTEL=m
CONFIG_KVM_AMD=m
CONFIG_KVM_MMU_AUDIT=y
CONFIG_VHOST_NET=m
CONFIG_VHOST=m

# vfio
CONFIG_VFIO_IOMMU_TYPE1=m
CONFIG_VFIO_VIRQFD=m
CONFIG_VFIO=m
CONFIG_VFIO_PCI=m
CONFIG_VFIO_PCI_VGA=y
CONFIG_VFIO_PCI_MMAP=y
CONFIG_VFIO_PCI_INTX=y
CONFIG_VFIO_PCI_IGD=y
CONFIG_VFIO_MDEV=m
CONFIG_VFIO_MDEV_DEVICE=m
CONFIG_IRQ_BYPASS_MANAGER=m

# cgroups
CONFIG_CGROUPS=y
CONFIG_BLK_CGROUP=y
CONFIG_CGROUP_WRITEBACK=y
CONFIG_CGROUP_SCHED=y
CONFIG_CGROUP_PIDS=y
CONFIG_CGROUP_RDMA=y
CONFIG_CGROUP_FREEZER=y
CONFIG_CGROUP_HUGETLB=y
CONFIG_CGROUP_DEVICE=y
CONFIG_CGROUP_CPUACCT=y
CONFIG_CGROUP_PERF=y
CONFIG_CGROUP_BPF=y
CONFIG_CGROUP_DEBUG=y
CONFIG_SOCK_CGROUP_DATA=y
CONFIG_CGROUP_NET_PRIO=y
CONFIG_CGROUP_NET_CLASSID=y
CONFIG_BLK_CGROUP_IOLATENCY=y

# misc
PCI_QUIRKS=y
CONFIG_MACVLAN=m
CONFIG_HIGH_RES_TIMERS=y
CONFIG_HPET_TIMER=y
CONFIG_HPET_EMULATE_RTC=y
CONFIG_BALLOON_COMPACTION=y
CONFIG_COMPACTION=y
CONFIG_MIGRATION=y
CONFIG_KSM=y
CONFIG_TRANSPARENT_HUGEPAGE=y
CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS=y

Also enable I2C for your system if you want to switch monitor inputs with DDC.

make sure your gpu drivers are modules, e.g. amdgpu.

If you are going to pass through a Vega, you may want to apply the vega reset
bug mitigation patch. The patch does not work perfectly, but it significantly
increases the chances of restarting your guest VM working at all.

The patch is here:

https://gist.github.com/numinit/1bbabff521e0451e5470d740e0eb82fd

You need to check that the IDs in the patch match your cards, to do so do a:

lspci -vn | grep "VGA controller"

if that shows e.g.:

33:00.0 0300: 1002:6863 (prog-if 00 [VGA controller])

Then the ID I want is 0x6863.

Do the same for the HDMI audio ID.

See below for finding the GPU in lspci.

If you use the patch, just in case, set managed='no' for the passed through
pcie devices later in the XML when you get to that part.

On AMD systems, add:

/etc/modprobe.d/kvm.conf

with:

options kvm ignore_msrs=1

For DDC add:

/etc/modules-load.d/ddc.conf

with:

i2c-dev

Update kernel and grub and reboot:

genkernel --oldconfig all
grub-mkconfig -o /boot/grub/grub.cfg

then check:

journalctl -a --since=yesterday | grep -e DMAR -e IOMMU

you should have some "Found IOMMU" messages.

Now we are going to install the necessary software.

I recommend adding the following use flags to e.g.

/etc/portage/package.use/vfio

app-emulation/qemu caps opengl alsa pulseaudio python spice ssh systemtap usb usbredir vde virgl virtfs glusterfs gnutls sdl2
app-emulation/libvirt pcap virt-network
>=net-dns/dnsmasq-2.79 script
sys-firmware/edk2-ovmf binary

If you want latest everything, which generally works nicely, add the following
to /etc/portage/package.accept_keywords

~app-emulation/libvirt-9999 **
~dev-python/libvirt-python-9999 **
~app-emulation/virt-manager-9999 **
~app-emulation/qemu-3.0.0 **
~sys-firmware/edk2-ovmf-9999 **

You can use app-portage/smart-live-rebuild to update the 9999 git packages.

We'll add a patch to qemu to make pulseaudio work well:

Code:

curl -LO 'https://gist.githubusercontent.com/Vaporeon/c879636f9147bd696fb888321ffd5655/raw/57fe4b41a84d46b908fcb9d8e9756e27f3b75940/pa-fixes.patch'
mkdir -p /etc/portage/patches/app-emulation/qemu-3.0.0/
mv pa-fixes.patch /etc/portage/patches/app-emulation/qemu-3.0.0/


Then install all the software:

emerge -v app-emulation/virt-manager

Make sure you are in the following groups (you can just add your user to them
with sudo vigr):

qemu:x:77:<YOU>
libvirt:x:994:<YOU>
audio:x:18:<YOU>
input:x:97:<YOU>
kvm:x:78:qemu,<YOU>
disk:x:6:root,adm,<YOU>

The disk group is probably necessary if you are going to use raw disk access
to boot your windows dual boot in the VM.

Next we're going to set up the windows VM in virt-manager, first without the
passthrough GPU.

Edit /etc/libvirt/qemu.conf and set:

user = "<your regular user>"
group = "root"
clear_emulator_capabilities = 0

You may not need to set group to root, you can try setting it to "libvirt" and
only change it if you have permissions problems, or your evdev peripherals are
not getting passed through.

Then do a:

systemctl restart libvirtd

If you are going to use your dual-boot windows partition, create the VM with a
small disk which you will later remove, then add your disk in:

sudo virsh edit <vm-name>

(You may have to use sudo for virsh edit despite being in the appropriate
groups.)

like so:

<disk type='block' device='disk'>
<driver name='qemu' type='raw' cache='directsync' io='threads' discard='unmap' detect_zeroes='unmap'/>
<source dev='/dev/disk/by-id/nvme-Samsung_SSD_960_PRO_2TB_S3EXNX0JC00132X'/>
<target dev='sda' bus='virtio'/>
</disk>

Change the top line of the XML to:

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>

And at the bottom add this element:

<qemu:commandline>
<qemu:env name='QEMU_AUDIO_DRV' value='pa'/>
<qemu:env name='QEMU_PA_SERVER' value='/run/user/1000/pulse/native'/>
</qemu:commandline>

If virsh complains about invalid XML, save the file from your editor, and then
run:

virsh define win10.xml

Set your sound chipset to ICH9 in the virt-manager GUI.

Make sure the bios type is set to EFI.

Under the <os> tag add the following:

<loader readonly='yes' type='pflash'>/usr/share/edk2-ovmf/OVMF_CODE.fd</loader>
<nvram>/var/lib/libvirt/qemu/nvram/win10_VARS.fd</nvram>

Then copy the vars, e.g.:

cp /usr/share/edk2-ovmf/OVMF_VARS.fd /var/lib/libvirt/qemu/nvram/win10_VARS.fd

If your hardware clock is UTC, as is recommended for systemd, and you already
made the necessary adjustment in windows, make sure the <clock> tag is:

<clock offset='utc'>

For your VM cpu, it's best to use host-passthrough. For AMD use something
like:

<cpu mode='host-passthrough' check='none'>
<topology sockets='1' cores='8' threads='2'/>
<cache mode='passthrough'/>
<feature policy='require' name='topoext'/>
</cpu>

Set your number of sockets, number of cores per socket, and number of threads
per core. For Intel you don't need the topext feature.

To install guest drivers for windows, use this:

https://www.spice-space.org/download/windows/spice-guest-tools/spice-guest-tools-latest.exe

Make sure your new VM boots and works correctly.

If windows refuses to boot with a black or blue screen, try changing the cpu
model to core2duo.

For activation mitigation use the well known pirate utilities. There is no
other good way.

At this point you should have a working windows VM with sound, now we are
going to do the GPU and peripherals passthrough part.

The following instructions are for genkernel-next, if you are using something
else for your initramfs, do something similar.

Edit /etc/genkernel.conf

Comment out any SPLASH and PLYMOUTH options, because this copies your GPU
drivers to the initramfs and we don't want them there to interfere with
binding vfio-pci.

Add these lines:

AMODULES_VIRTIO="vfio-pci vfio_virqfd vfio vfio_iommu_type1 irqbypass"

INITRAMFS_OVERLAY="/etc/initramfs-overlay"

Now find the PCI IDs of your GPU and associated HDMI audio device.

E.g.:

lspci -vv | grep "VGA controller"
12:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 41) (prog-if 00 [VGA controller])
23:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Vega 10 XTX [Radeon Vega Frontier Edition] (prog-if 00 [VGA controller])
33:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Vega 10 XTX [Radeon Vega Frontier Edition] (prog-if 00 [VGA controller])

I pass through my second Vega which is 33:00.0 and its HDMI audio device which
is 33:00.1.

Now we'll make a script in the initramfs to bind them:

mkdir -p /etc/initramfs-overlay/usr/bin/

edit /etc/initramfs-overlay/usr/bin/vfio-bind:

Code:

#!/bin/sh

for dev in "33:00.0" "33:00.1"; do
    echo "vfio-pci" > "/sys/bus/pci/devices/0000:${dev}/driver_override"
    echo "0000:${dev}" > /sys/bus/pci/drivers/vfio-pci/bind
done


set the device ids in the loop to the ones you want to pass through.

chmod +x /etc/initramfs-overlay/usr/bin/vfio-bind:

Then we'll make a custom linuxrc to invoke the script:

cp /usr/share/genkernel/defaults/linuxrc /etc/linuxrc

Edit /etc/linuxrc and below modules_init put something like this:

Code:

modules_init

case "$(cat /proc/cmdline)" in
    *dovfio*)
        sh /usr/bin/vfio-bind
        ;;
esac


Now we need to add a grub option for enabling vfio.

Edit /etc/grub.d/10_linux:

Find the code that looks like this:

Code:

  linux_entry "${OS}" "${version}" advanced \
              "${GRUB_CMDLINE_LINUX} ${GRUB_CMDLINE_LINUX_DEFAULT}"


Add another entry below it that looks like this:

Code:

  linux_entry "${OS} - vfio" "${version}" simple \
              "${GRUB_CMDLINE_LINUX} ${GRUB_CMDLINE_LINUX_DEFAULT} dovirtio dovfio"


For this option we are adding the dovirtio and dovfio kernel parameters.

Now regenerate the initramfs and the grub config:

genkernel --oldconfig --virtio --linuxrc=/etc/linuxrc initramfs

grub-mkconfig -o /boot/grub/grub.cfg

Now reboot to the new vfio entry and check that vfio-pci is bound to your GPU,
with e.g.:

lspci -vvk -s 33:00.0

it should say:

Kernel driver in use: vfio-pci

You can now test booting your VM with the passed through GPU.

In the virt-manager GUI, pass through the PCI ids for your GPU and HDMI audio
device.

You may want to find them in virsh edit and set managed='no'.

Now try booting the VM with a monitor connected to the passthrough GPU.

If you will be using e.g. looking-glass, you can get a dummy plug for your
passthrough GPU.

If windows does not boot, try setting cpu model to core2duo.

If you see windows on your passed through gpu, great.

There is one more step we need to do, to pass through a mouse and keyboard via
evdev.

This is explained here:

https://passthroughpo.st/using-evdev-passthrough-seamless-vm-input/

In virsh edit, make sure you have both ps2 and virtio mouse and keyboard
devices:

<input type='mouse' bus='virtio'>
</input>
<input type='keyboard' bus='virtio'>
</input>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>

Now we need to find the main keyboard and mouse input devices.

Go into /dev/input/by-id and /dev/input/by-path and try moving your mouse and
typing things on the keyboard while you do a:

cat *

you should see it outputting control characters.

Find your specific keyboard and mouse devices.

Passthrough the devices in virsh edit, e.g. I have:

Code:

<qemu:commandline>
  <qemu:arg value='-object'/>
  <qemu:arg value='input-linux,id=mouse,evdev=/dev/input/by-path/pci-0000:03:00.3-usb-0:1.3:1.1-event-mouse'/>
  <qemu:arg value='-object'/>
  <qemu:arg value='input-linux,id=keyboard,evdev=/dev/input/by-id/ckb-CORSAIR_K70_RGB_MK.2_Mechanical_Gaming_Keyboard_vKB_-event,grab_all=on,repeat=on'/>


The options grab_all=on,repeat=on are for keyboard devices.

Now add the devices to the qemu acl.

Edit /etc/libvirt/qemu.conf and change cgroup_device_acl to something like
this:

Code:

cgroup_device_acl = [
   "/dev/null", "/dev/full", "/dev/zero",
   "/dev/random", "/dev/urandom",
   "/dev/ptmx", "/dev/kvm", "/dev/kqemu",
   "/dev/rtc","/dev/hpet",
   "/dev/input/by-path/pci-0000:03:00.3-usb-0:1.3:1.1-event-mouse",
   "/dev/input/by-id/ckb-CORSAIR_K70_RGB_MK.2_Mechanical_Gaming_Keyboard_vKB_-event"
]


with your mouse and keyboard devices as the last two devices.

Do a:

systemctl restart libvirtd

Now you can switch your mouse and keyboard between the host and guest by
pressing both of the CTRL buttons.

This should be enough to get you up and running. There are endless tinkering
possibilities with this.
Back to top
View user's profile Send private message
Display posts from previous:   
This topic is locked: you cannot edit posts or make replies.    Gentoo Forums Forum Index Kernel & Hardware All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum