So I've been digging and reading for the past 4 days and installed ubuntu a good 4 or 5 times trying to get this working, let me explain my journey.
Xen has been the recomended way to go about it from when I first started researching PCI pass through, unfortunately I cannot get the Xen kernel to boot with Xorg at all. I had first set it up with cryptsetup and I could type my password and unlock the partitions but once that was done my already black screen lost signal, set it up without cryptsetup (thinking that was the problem), now I didnt have to type my crypt password to turn my monitor off. Had it working with arch a while back with a different GPU/monitors (still most recent nvidia drivers), unfortunately couldnt get nvidia drivers to work with my monitors on arch again once I returned to linux.
So Xen didnt work, I tried straight up virt-manager, no luck. With default settings it gets to starting windows, but no further. With PCI passthrough it just doesn't work, getting stuck at creating domain and creating a lock on KVM causing a kernel panic 90% of the time I reboot.
Now onto my final solution, a mixture of the puget guide, and the debian guide. This gets me either:
after binding and running:
echo '0000:01:00.1' | sudo tee /sys/bus/pci/devices/0000:01:00.1/driver/unbind
or:
when not unbinding the HDMI's driver with tee /sys/bus/pci/devices/0000:01:00.1/driver/unbind.
Here is my lspci -vnn:
dmesg | grep -i iommu
In there you can already see the line used to boot in grub, I added the PCI stub amd_iommu=on and iommu=pt.
lastly, my /usr/vm-win:
I believe I tried with the ROM entry as the debian guide recomends, I dont have error logs from that and would be willing to try it again if someone see fit?
Usually how I go about run it, ofc with sudo:
modprobe vfio
modprobe vfio_pci
echo 10de 13c2 | sudo tee /sys/bus/pci/drivers/vfio-pci/new_id
echo 10de 0fbb | sudo tee /sys/bus/pci/drivers/vfio-pci/new_id
echo '0000:01:00.1' | sudo tee /sys/bus/pci/devices/0000:01:00.1/driver/unbind
/usr/vm-win
I also get a long list of: "host doesn't support requested feature: CPUID.8000000H.EDX [bit #]"
they repeat four times and go through bit 0 - bit 24 skipping a few numbers.
Recomendations? Help? Need to see more logs?
Anything would be appreciated, I removed windows thinking I could use xen and move on with my life. ](*,)
I really dont want to reinstall again or move distros.
Xen has been the recomended way to go about it from when I first started researching PCI pass through, unfortunately I cannot get the Xen kernel to boot with Xorg at all. I had first set it up with cryptsetup and I could type my password and unlock the partitions but once that was done my already black screen lost signal, set it up without cryptsetup (thinking that was the problem), now I didnt have to type my crypt password to turn my monitor off. Had it working with arch a while back with a different GPU/monitors (still most recent nvidia drivers), unfortunately couldnt get nvidia drivers to work with my monitors on arch again once I returned to linux.
So Xen didnt work, I tried straight up virt-manager, no luck. With default settings it gets to starting windows, but no further. With PCI passthrough it just doesn't work, getting stuck at creating domain and creating a lock on KVM causing a kernel panic 90% of the time I reboot.
Now onto my final solution, a mixture of the puget guide, and the debian guide. This gets me either:
Code:
qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error, group 14 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.
qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 14
qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failedecho '0000:01:00.1' | sudo tee /sys/bus/pci/devices/0000:01:00.1/driver/unbind
or:
Code:
qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error opening /dev/vfio/14: No such file or directory
qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 14
qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failedHere is my lspci -vnn:
Code:
00:00.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (external gfx0 port B) [1002:5a14] (rev 02)
Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (external gfx0 port B) [1002:5a14]
Flags: fast devsel
Capabilities: <access denied>
00:00.2 IOMMU [0806]: Advanced Micro Devices, Inc. [AMD/ATI] RD990 I/O Memory Management Unit (IOMMU) [1002:5a23]
Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] RD990 I/O Memory Management Unit (IOMMU) [1002:5a23]
Flags: fast devsel, IRQ 26
Capabilities: <access denied>
00:02.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (PCI express gpp port B) [1002:5a16] (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 27
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 0000e000-0000efff
Memory behind bridge: fd000000-fe0fffff
Prefetchable memory behind bridge: 00000000c0000000-00000000d1ffffff
Capabilities: <access denied>
Kernel driver in use: pcieport
Kernel modules: shpchp
00:09.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (PCI express gpp port H) [1002:5a1c] (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 24
Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
Memory behind bridge: fe300000-fe3fffff
Capabilities: <access denied>
Kernel driver in use: pcieport
Kernel modules: shpchp
00:0a.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (external gfx1 port A) [1002:5a1d] (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 28
Bus: primary=00, secondary=03, subordinate=03, sec-latency=0
I/O behind bridge: 0000d000-0000dfff
Memory behind bridge: fe200000-fe2fffff
Capabilities: <access denied>
Kernel driver in use: pcieport
Kernel modules: shpchp
00:11.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 SATA Controller [AHCI mode] [1002:4391] (rev 40) (prog-if 01 [AHCI 1.0])
Subsystem: Gigabyte Technology Co., Ltd SB7x0/SB8x0/SB9x0 SATA Controller [AHCI mode] [1458:b002]
Flags: bus master, 66MHz, medium devsel, latency 32, IRQ 19
I/O ports at f090 [size=8]
I/O ports at f080 [size=4]
I/O ports at f070 [size=8]
I/O ports at f060 [size=4]
I/O ports at f050 [size=16]
Memory at fe40b000 (32-bit, non-prefetchable) [size=1K]
Capabilities: <access denied>
Kernel driver in use: ahci
Kernel modules: ahci
00:12.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller [1002:4397] (prog-if 10 [OHCI])
Subsystem: Gigabyte Technology Co., Ltd SB7x0/SB8x0/SB9x0 USB OHCI0 Controller [1458:5004]
Flags: bus master, 66MHz, medium devsel, latency 32, IRQ 18
Memory at fe40a000 (32-bit, non-prefetchable) [size=4K]
Kernel driver in use: ohci-pci
00:12.2 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller [1002:4396] (prog-if 20 [EHCI])
Subsystem: Gigabyte Technology Co., Ltd SB7x0/SB8x0/SB9x0 USB EHCI Controller [1458:5004]
Flags: bus master, 66MHz, medium devsel, latency 32, IRQ 17
Memory at fe409000 (32-bit, non-prefetchable) [size=256]
Capabilities: <access denied>
Kernel driver in use: ehci-pci
00:13.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller [1002:4397] (prog-if 10 [OHCI])
Subsystem: Gigabyte Technology Co., Ltd SB7x0/SB8x0/SB9x0 USB OHCI0 Controller [1458:5004]
Flags: bus master, 66MHz, medium devsel, latency 32, IRQ 18
Memory at fe408000 (32-bit, non-prefetchable) [size=4K]
Kernel driver in use: ohci-pci
00:13.2 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller [1002:4396] (prog-if 20 [EHCI])
Subsystem: Gigabyte Technology Co., Ltd SB7x0/SB8x0/SB9x0 USB EHCI Controller [1458:5004]
Flags: bus master, 66MHz, medium devsel, latency 32, IRQ 17
Memory at fe407000 (32-bit, non-prefetchable) [size=256]
Capabilities: <access denied>
Kernel driver in use: ehci-pci
00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 SMBus Controller [1002:4385] (rev 42)
Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 SMBus Controller [1002:4385]
Flags: 66MHz, medium devsel
Kernel driver in use: piix4_smbus
Kernel modules: i2c_piix4, sp5100_tco
00:14.1 IDE interface [0101]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 IDE Controller [1002:439c] (rev 40) (prog-if 8a [Master SecP PriP])
Subsystem: Gigabyte Technology Co., Ltd SB7x0/SB8x0/SB9x0 IDE Controller [1458:5002]
Flags: bus master, 66MHz, medium devsel, latency 32, IRQ 17
I/O ports at 01f0 [size=8]
I/O ports at 03f4
I/O ports at 0170 [size=8]
I/O ports at 0374
I/O ports at f000 [size=16]
Kernel driver in use: pata_atiixp
Kernel modules: pata_atiixp, pata_acpi
00:14.2 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 Azalia (Intel HDA) [1002:4383] (rev 40)
Subsystem: Gigabyte Technology Co., Ltd SBx00 Azalia (Intel HDA) [1458:a132]
Flags: bus master, slow devsel, latency 32, IRQ 16
Memory at fe400000 (64-bit, non-prefetchable) [size=16K]
Capabilities: <access denied>
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 LPC host controller [1002:439d] (rev 40)
Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 LPC host controller [1002:439d]
Flags: bus master, 66MHz, medium devsel, latency 0
00:14.4 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 PCI to PCI Bridge [1002:4384] (rev 40) (prog-if 01 [Subtractive decode])
Flags: bus master, VGA palette snoop, 66MHz, medium devsel, latency 64
Bus: primary=00, secondary=04, subordinate=04, sec-latency=64
I/O behind bridge: 0000c000-0000cfff
Memory behind bridge: fe100000-fe1fffff
00:14.5 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI2 Controller [1002:4399] (prog-if 10 [OHCI])
Subsystem: Gigabyte Technology Co., Ltd SB7x0/SB8x0/SB9x0 USB OHCI2 Controller [1458:5004]
Flags: bus master, 66MHz, medium devsel, latency 32, IRQ 18
Memory at fe406000 (32-bit, non-prefetchable) [size=4K]
Kernel driver in use: ohci-pci
00:16.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller [1002:4397] (prog-if 10 [OHCI])
Subsystem: Gigabyte Technology Co., Ltd SB7x0/SB8x0/SB9x0 USB OHCI0 Controller [1458:5004]
Flags: bus master, 66MHz, medium devsel, latency 32, IRQ 18
Memory at fe405000 (32-bit, non-prefetchable) [size=4K]
Kernel driver in use: ohci-pci
00:16.2 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller [1002:4396] (prog-if 20 [EHCI])
Subsystem: Gigabyte Technology Co., Ltd SB7x0/SB8x0/SB9x0 USB EHCI Controller [1458:5004]
Flags: bus master, 66MHz, medium devsel, latency 32, IRQ 17
Memory at fe404000 (32-bit, non-prefetchable) [size=256]
Capabilities: <access denied>
Kernel driver in use: ehci-pci
00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 0 [1022:1600]
Flags: fast devsel
Capabilities: <access denied>
00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 1 [1022:1601]
Flags: fast devsel
00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 2 [1022:1602]
Flags: fast devsel
Kernel modules: amd64_edac_mod
00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 3 [1022:1603]
Flags: fast devsel
Capabilities: <access denied>
Kernel driver in use: k10temp
Kernel modules: k10temp
00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 4 [1022:1604]
Flags: fast devsel
Kernel driver in use: fam15h_power
Kernel modules: fam15h_power
00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 5 [1022:1605]
Flags: fast devsel
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1) (prog-if 00 [VGA controller])
Subsystem: ASUSTeK Computer Inc. GM204 [GeForce GTX 970] [1043:853f]
Flags: bus master, fast devsel, latency 0, IRQ 35
Memory at fd000000 (32-bit, non-prefetchable) [size=16M]
Memory at c0000000 (64-bit, prefetchable) [size=256M]
Memory at d0000000 (64-bit, prefetchable) [size=32M]
I/O ports at e000 [size=128]
[virtual] Expansion ROM at fe000000 [disabled] [size=512K]
Capabilities: <access denied>
Kernel driver in use: nvidia
Kernel modules: nvidiafb, nouveau, nvidia_361
01:00.1 Audio device [0403]: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)
Subsystem: ASUSTeK Computer Inc. GM204 High Definition Audio Controller [1043:853f]
Flags: fast devsel, IRQ 33
Memory at fe080000 (32-bit, non-prefetchable) [size=16K]
Capabilities: <access denied>
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel
02:00.0 USB controller [0c03]: VIA Technologies, Inc. VL805 USB 3.0 Host Controller [1106:3483] (rev 01) (prog-if 30 [XHCI])
Subsystem: Gigabyte Technology Co., Ltd VL805 USB 3.0 Host Controller [1458:5007]
Flags: bus master, fast devsel, latency 0, IRQ 30
Memory at fe300000 (64-bit, non-prefetchable) [size=4K]
Capabilities: <access denied>
Kernel driver in use: xhci_hcd
03:00.0 SATA controller [0106]: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller [1b4b:9172] (rev 11) (prog-if 01 [AHCI 1.0])
Subsystem: Gigabyte Technology Co., Ltd 88SE9172 SATA 6Gb/s Controller [1458:b000]
Flags: bus master, fast devsel, latency 0, IRQ 32
I/O ports at d040 [size=8]
I/O ports at d030 [size=4]
I/O ports at d020 [size=8]
I/O ports at d010 [size=4]
I/O ports at d000 [size=16]
Memory at fe210000 (32-bit, non-prefetchable) [size=512]
Expansion ROM at fe200000 [disabled] [size=64K]
Capabilities: <access denied>
Kernel driver in use: ahci
Kernel modules: ahci
04:06.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8169 PCI Gigabit Ethernet Controller [10ec:8169] (rev 10)
Subsystem: Realtek Semiconductor Co., Ltd. RTL8169/8110 Family PCI Gigabit Ethernet NIC [10ec:8169]
Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 20
I/O ports at c000 [size=256]
Memory at fe121000 (32-bit, non-prefetchable) [size=256]
Expansion ROM at fe100000 [disabled] [size=128K]
Capabilities: <access denied>
Kernel driver in use: r8169
Kernel modules: r8169
04:0e.0 FireWire (IEEE 1394) [0c00]: VIA Technologies, Inc. VT6306/7/8 [Fire II(M)] IEEE 1394 OHCI Controller [1106:3044] (rev c0) (prog-if 10 [OHCI])
Subsystem: Gigabyte Technology Co., Ltd GA-7VT600-1394 Motherboard [1458:1000]
Flags: bus master, medium devsel, latency 32, IRQ 22
Memory at fe120000 (32-bit, non-prefetchable) [size=2K]
I/O ports at c100 [size=128]
Capabilities: <access denied>
Kernel driver in use: firewire_ohci
Kernel modules: firewire_ohciCode:
[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.4.0-31-generic.efi.signed root=UUID=54db22e2-5246-420f-8be7-9e879b60f331 ro quiet splash iommu=pt amd_iommu=on pci-stub.ids=10de:13c2,10de:0fbb vt.handoff=7
[ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.4.0-31-generic.efi.signed root=UUID=54db22e2-5246-420f-8be7-9e879b60f331 ro quiet splash iommu=pt amd_iommu=on pci-stub.ids=10de:13c2,10de:0fbb vt.handoff=7
[ 1.180773] iommu: Adding device 0000:00:00.0 to group 0
[ 1.180782] iommu: Using direct mapping for device 0000:00:00.0
[ 1.181078] iommu: Adding device 0000:00:02.0 to group 1
[ 1.181085] iommu: Using direct mapping for device 0000:00:02.0
[ 1.181379] iommu: Adding device 0000:00:09.0 to group 2
[ 1.181386] iommu: Using direct mapping for device 0000:00:09.0
[ 1.181681] iommu: Adding device 0000:00:0a.0 to group 3
[ 1.181688] iommu: Using direct mapping for device 0000:00:0a.0
[ 1.181982] iommu: Adding device 0000:00:11.0 to group 4
[ 1.181990] iommu: Using direct mapping for device 0000:00:11.0
[ 1.182327] iommu: Adding device 0000:00:12.0 to group 5
[ 1.182334] iommu: Using direct mapping for device 0000:00:12.0
[ 1.182347] iommu: Adding device 0000:00:12.2 to group 5
[ 1.182648] iommu: Adding device 0000:00:13.0 to group 6
[ 1.182655] iommu: Using direct mapping for device 0000:00:13.0
[ 1.182668] iommu: Adding device 0000:00:13.2 to group 6
[ 1.182963] iommu: Adding device 0000:00:14.0 to group 7
[ 1.182970] iommu: Using direct mapping for device 0000:00:14.0
[ 1.183264] iommu: Adding device 0000:00:14.1 to group 8
[ 1.183271] iommu: Using direct mapping for device 0000:00:14.1
[ 1.183565] iommu: Adding device 0000:00:14.2 to group 9
[ 1.183573] iommu: Using direct mapping for device 0000:00:14.2
[ 1.183866] iommu: Adding device 0000:00:14.3 to group 10
[ 1.183873] iommu: Using direct mapping for device 0000:00:14.3
[ 1.184168] iommu: Adding device 0000:00:14.4 to group 11
[ 1.184175] iommu: Using direct mapping for device 0000:00:14.4
[ 1.184468] iommu: Adding device 0000:00:14.5 to group 12
[ 1.184475] iommu: Using direct mapping for device 0000:00:14.5
[ 1.184777] iommu: Adding device 0000:00:16.0 to group 13
[ 1.184784] iommu: Using direct mapping for device 0000:00:16.0
[ 1.184797] iommu: Adding device 0000:00:16.2 to group 13
[ 1.185111] iommu: Adding device 0000:01:00.0 to group 14
[ 1.185118] iommu: Using direct mapping for device 0000:01:00.0
[ 1.185139] iommu: Adding device 0000:01:00.1 to group 14
[ 1.185434] iommu: Adding device 0000:02:00.0 to group 15
[ 1.185442] iommu: Using direct mapping for device 0000:02:00.0
[ 1.185737] iommu: Adding device 0000:03:00.0 to group 16
[ 1.185745] iommu: Using direct mapping for device 0000:03:00.0
[ 1.185753] iommu: Adding device 0000:04:06.0 to group 11
[ 1.185760] iommu: Adding device 0000:04:0e.0 to group 11
[ 1.185762] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40lastly, my /usr/vm-win:
Code:
#!/bin/bash
sudo qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host \
-smp 4,sockets=1,cores=4,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \
-drive file=/media/vuvffufg/Media/VMs/Win.img,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk \
-drive file=/media/vuvffufg/Media/VMs/Windows.iso,id=isocd -device ide-cd,bus=ide.1,drive=isocd \
-boot menu=on
exit 0Usually how I go about run it, ofc with sudo:
modprobe vfio
modprobe vfio_pci
echo 10de 13c2 | sudo tee /sys/bus/pci/drivers/vfio-pci/new_id
echo 10de 0fbb | sudo tee /sys/bus/pci/drivers/vfio-pci/new_id
echo '0000:01:00.1' | sudo tee /sys/bus/pci/devices/0000:01:00.1/driver/unbind
/usr/vm-win
I also get a long list of: "host doesn't support requested feature: CPUID.8000000H.EDX [bit #]"
they repeat four times and go through bit 0 - bit 24 skipping a few numbers.
Recomendations? Help? Need to see more logs?
Anything would be appreciated, I removed windows thinking I could use xen and move on with my life. ](*,)
I really dont want to reinstall again or move distros.