So as much as I feel I have tried everything I obviously have not but I am at a complete brick wall now. Especially considering one works but windows does not.
Please Ubuntu gods give me some pointers.
Ubuntu running in a VM works, Nvidia-smi and external screen turns on with display manager - PERFECT :)
Windows 10 (or 7 also tried) continue to get error 43 whatever I do - NOT GOOD :(
I am a little confused as to why my Ubuntu running in a VM can operate perfectly with official NVidia windows drivers but I fail to get anywhere with windows.
I will go into in detail below (in the hope it can help others in the future as well as thought checking myself).
H/W Configuration:
S/W Configuration:
GRUB default command line ( probably overkill )
/etc/modprobe.d/local.conf
/etc/initramfs-tools/modules.conf
/etc/modprobe.d/blacklist.conf
If I check the passthrough, lspci -nnk I see they are indeed using the right driver.
Checking the kernel log, dmesg|grep -i vfio
I would say that looks ideal.
VM Configuration
Hiding the VM from nvidia, all configurations have the following:
The PCIE passthrough part:
For Ubuntu VM I am running UEFI and the GPU is passed through, confirmed with NVidia-smi.
If I want to do it via Qemu command line with novga then I run the following:
Onto Windows, I cannot for the life of me get windows to not give an error 43.
I have tried
Here is my full windows config:
I really appreciate any help with this. thanks :)
Please Ubuntu gods give me some pointers.
Ubuntu running in a VM works, Nvidia-smi and external screen turns on with display manager - PERFECT :)
Windows 10 (or 7 also tried) continue to get error 43 whatever I do - NOT GOOD :(
I am a little confused as to why my Ubuntu running in a VM can operate perfectly with official NVidia windows drivers but I fail to get anywhere with windows.
I will go into in detail below (in the hope it can help others in the future as well as thought checking myself).
H/W Configuration:
- Ubuntu server 17.04
- i7700k
- Geforce 1080
- 32GB RAM
- Intel IGP
S/W Configuration:
GRUB default command line ( probably overkill )
Code:
pcie_acs_override=downstream nofb nomodeset vga=normal video=vesafb:off i915.enable_hd_vgaarb=1 i915.modeset=0 intel_iommu=on iommu=pt rd.driver.pre=vfio-pciCode:
options kvm allow_unsafe_assigned_interrupts=1
options vfio-pci ids=10de:1b80,10de:10f0
options vfio-pci disable_vga=1Code:
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd/etc/modprobe.d/blacklist.conf
Code:
blacklist nouveau
blacklist lbm-nouveauIf I check the passthrough, lspci -nnk I see they are indeed using the right driver.
Code:
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP104 [GeForce GTX 1080] [10de:1b80] (rev a1)
Subsystem: Dell GP104 [GeForce GTX 1080] [1028:3366]
Kernel driver in use: vfio-pci
Kernel modules: nvidiafb, nouveau
01:00.1 Audio device [0403]: NVIDIA Corporation GP104 High Definition Audio Controller [10de:10f0] (rev a1)
Subsystem: Dell GP104 High Definition Audio Controller [1028:3366]
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intelChecking the kernel log, dmesg|grep -i vfio
Code:
[ 3.136766] VFIO - User Level meta-driver version: 0.3
[ 3.140229] vfio-pci 0000:01:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=io+mem
[ 3.157852] vfio_pci: add [10de:1b80[ffff:ffff]] class 0x000000/00000000
[ 3.177847] vfio_pci: add [10de:10f0[ffff:ffff]] class 0x000000/00000000VM Configuration
Hiding the VM from nvidia, all configurations have the following:
Code:
<features>
<acpi/>
<apic/>
<hyperv>
<relaxed state='on'/>
<vapic state='on'/>
<spinlocks state='on' retries='8191'/>
<vendor_id state='on' value='yesplease'/>
</hyperv>
<kvm>
<hidden state='on'/>
</kvm>
</features>
<cpu mode='host-passthrough'/>Code:
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</source>
<address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
</source>
<address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
</hostdev>If I want to do it via Qemu command line with novga then I run the following:
Code:
qemu-system-x86_64 -machine pc-i440fx-zesty,accel=kvm -cpu Skylake-Client,kvm=off -smp 1,sockets=1,cores=1,threads=1 -drive file=/usr/share/OVMF/OVMF_CODE.fd,if=pflash,format=raw,unit=0,readonly=on -drive file=/var/lib/libvirt/qemu/nvram/ub_VARS.fd,if=pflash,format=raw,unit=1 -m 1024 -drive file=/var/lib/libvirt/images/ub.qcow2,format=qcow2,if=none,id=drive-ide0-0-0 -device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.0,addr=0x8 -device vfio-pci,host=01:00.1,id=hostdev1,bus=pci.0,addr=0x9 -vga noneOnto Windows, I cannot for the life of me get windows to not give an error 43.
I have tried
- various Bios and UEFI configurations of both Q35 and I440FX
- the hiding configuration above
- various NVidia drivers (if anyone has a tip for one that works please let me know)
- Enabling MSI for the devices in windows via the registry (only the HD audio device gets a negative IRQ the card stays yellow)
Here is my full windows config:
Code:
<domain type='kvm'>
<name>win10q35</name>
<uuid>0a706dcb-9b07-4d72-8c05-6ca16d25835b</uuid>
<memory unit='KiB'>12582912</memory>
<currentMemory unit='KiB'>12582912</currentMemory>
<vcpu placement='static'>4</vcpu>
<os>
<type arch='x86_64' machine='pc-q35-2.8'>hvm</type>
<loader readonly='yes' type='pflash'>/usr/share/OVMF/OVMF_CODE.fd</loader>
<nvram>/var/lib/libvirt/qemu/nvram/win10q35_VARS.fd</nvram>
<boot dev='hd'/>
</os>
<features>
<acpi/>
<apic/>
<hyperv>
<relaxed state='on'/>
<vapic state='on'/>
<spinlocks state='on' retries='8191'/>
<vendor_id state='on' value='yesplease'/>
</hyperv>
<kvm>
<hidden state='on'/>
</kvm>
</features>
<cpu mode='host-passthrough'/>
<clock offset='localtime'>
<timer name='hpet' present='no'/>
<timer name='hypervclock' present='yes'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>destroy</on_reboot>
<on_crash>destroy</on_crash>
<pm>
<suspend-to-mem enabled='no'/>
<suspend-to-disk enabled='no'/>
</pm>
<devices>
<emulator>/usr/bin/kvm-spice</emulator>
<disk type='file' device='cdrom'>
<driver name='qemu' type='raw'/>
<source file='/var/lib/libvirt/images/en_windows_10_enterprise_n_version_1703_updated_march_2017_x64_dvd_10189280.iso'/>
<target dev='sdb' bus='sata'/>
<readonly/>
<address type='drive' controller='0' bus='0' target='0' unit='1'/>
</disk>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2'/>
<source file='/var/lib/libvirt/images/win10q35.qcow2'/>
<target dev='sdc' bus='sata'/>
<address type='drive' controller='0' bus='0' target='0' unit='2'/>
</disk>
<controller type='usb' index='0' model='ich9-ehci1'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x7'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci1'>
<master startport='0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x0' multifunction='on'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci2'>
<master startport='2'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x1'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci3'>
<master startport='4'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x2'/>
</controller>
<controller type='sata' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
</controller>
<controller type='pci' index='0' model='pcie-root'/>
<controller type='pci' index='1' model='dmi-to-pci-bridge'>
<model name='i82801b11-bridge'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
</controller>
<controller type='pci' index='2' model='pci-bridge'>
<model name='pci-bridge'/>
<target chassisNr='2'/>
<address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</controller>
<controller type='pci' index='3' model='pcie-root-port'>
<model name='ioh3420'/>
<target chassis='3' port='0x10'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
</controller>
<controller type='pci' index='4' model='pcie-root-port'>
<model name='ioh3420'/>
<target chassis='4' port='0x18'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</controller>
<controller type='pci' index='5' model='pcie-root-port'>
<model name='ioh3420'/>
<target chassis='5' port='0x20'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</controller>
<controller type='pci' index='6' model='pcie-root-port'>
<model name='ioh3420'/>
<target chassis='6' port='0x28'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
</controller>
<controller type='virtio-serial' index='0'>
<address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
</controller>
<interface type='network'>
<mac address='52:54:00:1a:37:56'/>
<source network='default'/>
<model type='rtl8139'/>
<address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/>
</interface>
<input type='tablet' bus='usb'>
<address type='usb' bus='0' port='1'/>
</input>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<graphics type='spice' autoport='yes'>
<listen type='address'/>
</graphics>
<video>
<model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
</video>
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</source>
<address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
</source>
<address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
</hostdev>
<memballoon model='virtio'>
<address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
</memballoon>
</devices>
</domain>I really appreciate any help with this. thanks :)