Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
permission denied on /dev/vfio/1 gpu passthrough kvm
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware
View previous topic :: View next topic  
Author Message
tse
n00b
n00b


Joined: 03 Jan 2018
Posts: 1

PostPosted: Wed Jan 03, 2018 4:47 pm    Post subject: permission denied on /dev/vfio/1 gpu passthrough kvm Reply with quote

Hello, I'm trying to achieve PCI-express GPU passthrough to a Window 10 VM, however, when trying to start the virtual machine (as root) with the hardware attached, this is what I get:
Code:

virsh # start win10-clone
error: Failed to start domain win10-clone
error: internal error: process exited while connecting to monitor: 2018-01-03T16:58:47.761436Z qemu-system-x86_64: -chardev pty,id=charserial0: char device redirected to /dev/pts/4 (label charserial0)
2018-01-03T16:58:47.786124Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.0,addr=0x2: vfio error: 0000:01:00.0: failed to open /dev/vfio/1: Permission denied


The file however, exists:
Code:

# file /dev/vfio/1
/dev/vfio/1: character special (250/0)
# ls -l /dev/vfio/1
crw------- 1 root root 250, 0 Jan  3 16:51 /dev/vfio/1


I followed the steps detailed in https://wiki.installgentoo.com/index.php/PCI_passthrough for host preparation, and in https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF#Setting_up_the_guest_OS for setting up the guest.

Before running virsh, I:
Code:

/etc/init.d/libvirtd stop
/etc/init.d/libvirt-guests stop
./vfio-bind 0000:01:00.0 0000:01:00.1
/etc/init.d/libvirtd start
/etc/init.d/libvirt-guests start


I've tried playing around with /etc/libvirt/qemu.conf, namely with the "group", "user", and "dynamic_ownership" settings - even though I'm running everything as root - and with "cgroup_device_acl" (by adding "/dev/vfio/1" to the list).
I cannot figure out what the problem in my setup is or what to try next. Is there anyone that can shine some light on this?

Thank you!

Following is the system information that I think is relevant for this problem:

lspci -vv | grep -i acscap
Code:

# lspci -vv | grep -i acscap
#


dmesg | grep -e DMAR -e IOMMU
Code:

# dmesg |  grep -e DMAR -e IOMMU
[    0.000000] ACPI: DMAR 0x000000008A430B40 0000A8 (v01 INTEL  SKL      00000001 INTL 00000001)
[    0.000000] DMAR: IOMMU enabled
[    0.032444] DMAR: Host address width 39
[    0.032446] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
[    0.032452] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap 1c0000c40660462 ecap 7e3ff0505e
[    0.032455] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[    0.032459] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap d2008c40660462 ecap f050da
[    0.032462] DMAR: RMRR base: 0x00000089f07000 end: 0x00000089f26fff
[    0.032476] DMAR: RMRR base: 0x0000008b800000 end: 0x0000008fffffff
[    0.032478] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.032480] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[    0.033866] DMAR-IR: Enabled IRQ remapping in xapic mode
[    7.518966] DMAR: No ATSR found
[    7.519126] DMAR: dmar0: Using Queued invalidation
[    7.519131] DMAR: dmar1: Using Queued invalidation
[    7.519328] DMAR: Setting RMRR:
[    7.519348] DMAR: Setting identity map for device 0000:00:02.0 [0x8b800000 - 0x8fffffff]
[    7.519369] DMAR: Setting identity map for device 0000:00:14.0 [0x89f07000 - 0x89f26fff]
[    7.519375] DMAR: Prepare 0-16MiB unity mapping for LPC
[    7.519391] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
[    7.519412] DMAR: Intel(R) Virtualization Technology for Directed I/O


dmesg |grep vfio:
Code:

# dmesg |grep vfio
[    0.000000] Command line: BOOT_IMAGE=/kernel-genkernel-x86_64-4.9.72-gentoo root=/dev/mapper/vg0-root ro dolvm crypt_root=UUID=5016f5cf-65f7-4d8f-8c14-113c48778ad6 root=/dev/mapper/vg0-root iommu=on intel_iommu=on vfio_pci.ids=10de:1b81,10de:10f0 pcie_acs_override=downstream
[    0.000000] Kernel command line: BOOT_IMAGE=/kernel-genkernel-x86_64-4.9.72-gentoo root=/dev/mapper/vg0-root ro dolvm crypt_root=UUID=5016f5cf-65f7-4d8f-8c14-113c48778ad6 root=/dev/mapper/vg0-root iommu=on intel_iommu=on vfio_pci.ids=10de:1b81,10de:10f0 pcie_acs_override=downstream
[    7.627739] vfio_pci: add [10de:1b81[ffff:ffff]] class 0x000000/00000000
[    7.639685] vfio_pci: add [10de:10f0[ffff:ffff]] class 0x000000/00000000


lspci -nnk | grep -A3 -i nvidia
Code:

# lspci -nnk | grep -A3 -i nvidia
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP104 [GeForce GTX 1070] [10de:1b81] (rev a1)
   Subsystem: Micro-Star International Co., Ltd. [MSI] GP104 [GeForce GTX 1070] [1462:3302]
   Kernel driver in use: vfio-pci
   Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia
01:00.1 Audio device [0403]: NVIDIA Corporation GP104 High Definition Audio Controller [10de:10f0] (rev a1)
   Subsystem: Micro-Star International Co., Ltd. [MSI] GP104 High Definition Audio Controller [1462:3302]
   Kernel driver in use: vfio-pci
   Kernel modules: snd_hda_intel


./iommu_groups.sh
Code:

# ./iommu_groups.sh
IOMMU Group 0 00:00.0 Host bridge [0600]: Intel Corporation Skylake Host Bridge/DRAM Registers [8086:191f] (rev 07)
IOMMU Group 10 00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (2) I219-V [8086:15b8] (rev 31)
IOMMU Group 1 00:01.0 PCI bridge [0604]: Intel Corporation Skylake PCIe Controller (x16) [8086:1901] (rev 07)
IOMMU Group 1 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP104 [GeForce GTX 1070] [10de:1b81] (rev a1)
IOMMU Group 1 01:00.1 Audio device [0403]: NVIDIA Corporation GP104 High Definition Audio Controller [10de:10f0] (rev a1)
IOMMU Group 2 00:02.0 VGA compatible controller [0300]: Intel Corporation HD Graphics 530 [8086:1912] (rev 06)
IOMMU Group 3 00:14.0 USB controller [0c03]: Intel Corporation Sunrise Point-H USB 3.0 xHCI Controller [8086:a12f] (rev 31)
IOMMU Group 4 00:16.0 Communication controller [0780]: Intel Corporation Sunrise Point-H CSME HECI #1 [8086:a13a] (rev 31)
IOMMU Group 5 00:17.0 SATA controller [0106]: Intel Corporation Sunrise Point-H SATA controller [AHCI mode] [8086:a102] (rev 31)
IOMMU Group 6 00:1b.0 PCI bridge [0604]: Intel Corporation Sunrise Point-H PCI Root Port #17 [8086:a167] (rev f1)
IOMMU Group 6 00:1b.2 PCI bridge [0604]: Intel Corporation Sunrise Point-H PCI Root Port #19 [8086:a169] (rev f1)
IOMMU Group 7 00:1c.0 PCI bridge [0604]: Intel Corporation Sunrise Point-H PCI Express Root Port #1 [8086:a110] (rev f1)
IOMMU Group 7 00:1c.1 PCI bridge [0604]: Intel Corporation Sunrise Point-H PCI Express Root Port #2 [8086:a111] (rev f1)
IOMMU Group 7 00:1c.2 PCI bridge [0604]: Intel Corporation Sunrise Point-H PCI Express Root Port #3 [8086:a112] (rev f1)
IOMMU Group 7 00:1c.4 PCI bridge [0604]: Intel Corporation Sunrise Point-H PCI Express Root Port #5 [8086:a114] (rev f1)
IOMMU Group 7 07:00.0 PCI bridge [0604]: Intel Corporation DSL6540 Thunderbolt 3 Bridge [Alpine Ridge 4C 2015] [8086:1578]
IOMMU Group 7 08:00.0 PCI bridge [0604]: Intel Corporation DSL6540 Thunderbolt 3 Bridge [Alpine Ridge 4C 2015] [8086:1578]
IOMMU Group 7 08:01.0 PCI bridge [0604]: Intel Corporation DSL6540 Thunderbolt 3 Bridge [Alpine Ridge 4C 2015] [8086:1578]
IOMMU Group 7 08:02.0 PCI bridge [0604]: Intel Corporation DSL6540 Thunderbolt 3 Bridge [Alpine Ridge 4C 2015] [8086:1578]
IOMMU Group 7 08:04.0 PCI bridge [0604]: Intel Corporation DSL6540 Thunderbolt 3 Bridge [Alpine Ridge 4C 2015] [8086:1578]
IOMMU Group 7 0b:00.0 USB controller [0c03]: Intel Corporation DSL6540 USB 3.1 Controller [Alpine Ridge] [8086:15b6]
IOMMU Group 8 00:1d.0 PCI bridge [0604]: Intel Corporation Sunrise Point-H PCI Express Root Port #9 [8086:a118] (rev f1)
IOMMU Group 8 00:1d.4 PCI bridge [0604]: Intel Corporation Sunrise Point-H PCI Express Root Port #13 [8086:a11c] (rev f1)
IOMMU Group 9 00:1f.0 ISA bridge [0601]: Intel Corporation Sunrise Point-H LPC Controller [8086:a145] (rev 31)
IOMMU Group 9 00:1f.2 Memory controller [0580]: Intel Corporation Sunrise Point-H PMC [8086:a121] (rev 31)
IOMMU Group 9 00:1f.3 Audio device [0403]: Intel Corporation Sunrise Point-H HD Audio [8086:a170] (rev 31)
IOMMU Group 9 00:1f.4 SMBus [0c05]: Intel Corporation Sunrise Point-H SMBus [8086:a123] (rev 31)

cat win10-clone.xml
Code:

# cat win10-clone.xml
<!--
WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:
  virsh edit win10-clone
or other application using the libvirt API.
-->

<domain type='kvm'>
  <name>win10-clone</name>
  <uuid>a673c9c8-a1f5-4c3d-97a7-16096bcf1b01</uuid>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <vcpu placement='static'>2</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.10'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/edk2-ovmf/OVMF_CODE.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/win10_VARS.fd</nvram>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
    </hyperv>
    <vmport state='off'/>
  </features>
  <cpu mode='host-model' check='partial'>
    <model fallback='allow'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
    <timer name='hypervclock' present='yes'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/home/tse/kvm_vm/test1-clone.qcow2'/>
      <target dev='sda' bus='scsi'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='1' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>
    </controller>
    <controller type='scsi' index='0' model='virtio-scsi'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </controller>
    <interface type='direct'>
      <mac address='52:54:00:45:6f:82'/>
      <source dev='enp0s31f6' mode='bridge'/>
      <model type='rtl8139'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <sound model='ac97'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </sound>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </hostdev>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='2'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='3'/>
    </redirdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </memballoon>
  </devices>
</domain>


cat iommu_groups.sh
Code:

# cat iommu_groups.sh
#!/bin/bash
shopt -s nullglob
for d in /sys/kernel/iommu_groups/*/devices/*; do
    n=${d#*/iommu_groups/*}; n=${n%%/*}
    printf 'IOMMU Group %s ' "$n"
    lspci -nns "${d##*/}"
done;


cat vfio-bind
Code:

# cat vfio-bind
#!/bin/bash
modprobe vfio-pci
for dev in "$@"; do
        vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
        device=$(cat /sys/bus/pci/devices/$dev/device)
        if [ -e /sys/bus/pci/devices/$dev/driver ]; then
                echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
        fi
        echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id
   echo "done"
done


emerge --pretend --verbose qemu
Code:

[ebuild   R    ] app-emulation/qemu-2.10.1-r1::gentoo  USE="aio alsa bzip2 caps curl fdt filecaps jpeg ncurses nls opengl pin-upstream-blobs png python seccomp spice usb usbredir vhost-net vnc xattr -accessibility -bluetooth -debug (-glusterfs) -gnutls -gtk -gtk2 -infiniband -iscsi -lzo -nfs -numa -pulseaudio -rbd -sasl -sdl -sdl2 (-selinux) -smartcard -snappy -ssh -static -static-user -systemtap -tci {-test} -vde -virgl -virtfs -vte -xen -xfs" LINGUAS="-bg -de_DE -fr_FR -hu -it -tr -zh_CN" PYTHON_TARGETS="python2_7" QEMU_SOFTMMU_TARGETS="x86_64 -aarch64 -alpha -arm -cris -i386 -lm32 -m68k -microblaze -microblazeel -mips -mips64 -mips64el -mipsel -moxie -nios2 -or1k -ppc -ppc64 -ppcemb -s390x -sh4 -sh4eb -sparc -sparc64 -tricore -unicore32 -xtensa -xtensaeb" QEMU_USER_TARGETS="-aarch64 -alpha -arm -armeb -cris -hppa -i386 -m68k -microblaze -microblazeel -mips -mips64 -mips64el -mipsel -mipsn32 -mipsn32el -nios2 -or1k -ppc -ppc64 -ppc64abi32 -ppc64le -s390x -sh4 -sh4eb -sparc -sparc32plus -sparc64 -tilegx -x86_64"
Back to top
View user's profile Send private message
blopsalot
Apprentice
Apprentice


Joined: 28 Jan 2017
Posts: 231

PostPosted: Mon Jan 08, 2018 2:54 pm    Post subject: Reply with quote

i would try starting with a simpler qemu command, try to turn off kvm. make sure vfio grabbing device first. im not familar with libvirt tho.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum