Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
[SOLVED] Setting up IOMMU and QEMU for VGA Passthrough
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware
View previous topic :: View next topic  
Author Message
NilPointer
n00b
n00b


Joined: 02 Feb 2015
Posts: 37

PostPosted: Sun Aug 28, 2016 7:31 am    Post subject: [SOLVED] Setting up IOMMU and QEMU for VGA Passthrough Reply with quote

I'm trying to achieve the VGA Passthrough. I've got following hardware:

Gigabyte GA-990X-GAMING SLI with IOMMU controller.
AMD FX-8350 CPU with IOMMU support.
Two AMD Radeon RX480 GPUs.

I want to give the spare GPU to the VM with PCI passthrough. I've enabled IOMMU in BIOS, compiled the kernel with relevant modules and cmdline options.

This is my customized cmdline for kernel:

Code:
GRUB_CMDLINE_LINUX="max_loop=64 iommu=pt iommu=1 amd_iommu=fullflush"


And these lines appear in dmesg:

Code:
[    1.176991] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40
[    1.177107] AMD-Vi: Interrupt remapping enabled
[    1.177302] AMD-Vi: IO/TLB flush on unmap enabled
...
[    1.089694] iommu: Adding device 0000:00:00.0 to group 0
[    1.089818] iommu: Using direct mapping for device 0000:00:00.0
[    1.090242] iommu: Adding device 0000:00:02.0 to group 1
[    1.090398] iommu: Using direct mapping for device 0000:00:02.0
[    1.090820] iommu: Adding device 0000:00:03.0 to group 2
[    1.090941] iommu: Using direct mapping for device 0000:00:03.0
[    1.091371] iommu: Adding device 0000:00:09.0 to group 3
[    1.091519] iommu: Using direct mapping for device 0000:00:09.0
[    1.091941] iommu: Adding device 0000:00:11.0 to group 4
[    1.092063] iommu: Using direct mapping for device 0000:00:11.0
[    1.092507] iommu: Adding device 0000:00:12.0 to group 5
[    1.092630] iommu: Using direct mapping for device 0000:00:12.0
[    1.092784] iommu: Adding device 0000:00:12.2 to group 5
[    1.093215] iommu: Adding device 0000:00:13.0 to group 6
[    1.093343] iommu: Using direct mapping for device 0000:00:13.0
[    1.093471] iommu: Adding device 0000:00:13.2 to group 6
[    1.093895] iommu: Adding device 0000:00:14.0 to group 7
[    1.094043] iommu: Using direct mapping for device 0000:00:14.0
[    1.094471] iommu: Adding device 0000:00:14.2 to group 8
[    1.094592] iommu: Using direct mapping for device 0000:00:14.2
[    1.095014] iommu: Adding device 0000:00:14.3 to group 9
[    1.095163] iommu: Using direct mapping for device 0000:00:14.3
[    1.098073] iommu: Adding device 0000:00:14.4 to group 10
[    1.098194] iommu: Using direct mapping for device 0000:00:14.4
[    1.098629] iommu: Adding device 0000:00:14.5 to group 11
[    1.098750] iommu: Using direct mapping for device 0000:00:14.5
[    1.099207] iommu: Adding device 0000:00:15.0 to group 12
[    1.099333] iommu: Using direct mapping for device 0000:00:15.0
[    1.099463] iommu: Adding device 0000:00:15.3 to group 12
[    1.099894] iommu: Adding device 0000:00:16.0 to group 13
[    1.100039] iommu: Using direct mapping for device 0000:00:16.0
[    1.100167] iommu: Adding device 0000:00:16.2 to group 13
[    1.100627] iommu: Adding device 0000:01:00.0 to group 14
[    1.100750] iommu: Using direct mapping for device 0000:01:00.0
[    1.100891] iommu: Adding device 0000:01:00.1 to group 14
[    1.101359] iommu: Adding device 0000:02:00.0 to group 15
[    1.101485] iommu: Using direct mapping for device 0000:02:00.0
[    1.101631] iommu: Adding device 0000:02:00.1 to group 15
[    1.102056] iommu: Adding device 0000:03:00.0 to group 16
[    1.102176] iommu: Using direct mapping for device 0000:03:00.0
[    1.102297] iommu: Adding device 0000:04:06.0 to group 10
[    1.102452] iommu: Adding device 0000:05:00.0 to group 12
[    1.102578] iommu: Adding device 0000:06:00.0 to group 12


The GPUs are there:

Code:
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Device 67df (rev c7)
01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Device aaf0
02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Device 67df (rev c7)
02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Device aaf0


They're both being bound to amdgpu driver on boot and that's unwanted. How can I bind 02.00.0 to vfio-pci driver? Apparently, it's easier to bind devices with different vendor/device IDs. But in my case, there's two GPUs of same kind.

I've tried unbinding device with following script and rebinding it to vfio-pci driver manually:

Code:
#!/bin/sh

modprobe -i vfio-pci
echo 0000:02:00.0 > /sys/bus/pci/drivers/amdgpu/unbind
echo 0000:02:00.0 > /sys/bus/pci/drivers/vfio-pci/bind


Running these commands cause nothing unusual in the first 5 seconds, then everything starts to stutter (mouse, probably, moves 1 frame per second) and few seconds later, machine completely deadlocks, not responding to mouse, keyboard or SSH. How could I force vfio-pci driver for second GPU? Am I doing something wrong?


Last edited by NilPointer on Mon Aug 29, 2016 8:28 pm; edited 1 time in total
Back to top
View user's profile Send private message
szatox
Veteran
Veteran


Joined: 27 Aug 2013
Posts: 1729

PostPosted: Sun Aug 28, 2016 9:48 am    Post subject: Reply with quote

I'd try blacklisting radeon so it doesn't load at boot.
Bind one GPU to virtio (pci_stub module? I'm not quite sure which one should bind it) and then load radeon so it will claim the other, unused device.
Back to top
View user's profile Send private message
NilPointer
n00b
n00b


Joined: 02 Feb 2015
Posts: 37

PostPosted: Mon Aug 29, 2016 8:26 pm    Post subject: Reply with quote

Looks like I've made it. Thanks for advice. I've blacklisted amdgpu driver and removed amdgpu from /etc/conf.d/modules. Then, I've used the following script, which I've found there
http://vfio.blogspot.ru/2015/05/vfio-gpu-how-to-series-part-3-host.html:

Code:
#!/bin/sh

for i in $(find /sys/devices/pci* -name boot_vga); do
        if [ $(cat $i) -eq 0 ]; then
                GPU=$(dirname $i)
                AUDIO=$(echo $GPU | sed -e "s/0$/1/")
                echo "vfio-pci" > $GPU/driver_override
                if [ -d $AUDIO ]; then
                        echo "vfio-pci" > $AUDIO/driver_override
                fi
        fi
done

modprobe -i vfio-pci


That adds driver override for non-boot VGA adapter (which is 0000:02:00.0 in my case, and to it's companion HDMI audio) and loads vfio-pci, which binds to those. Then I can modprobe amdgpu which will bind to boot adapter and load XOrg. I think I'll add that script to my boot sequence (with "modprobe amdgpu" appended).

Finally, qemu starts (although only as root) and second GPU actually outputs signal! Woo-hoo! Guess that means GPU passthrough works!
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum