Hi All,
Im building a new system based on Ubuntu 17.04. I need to retain a Windows environment so decided to VM it via QEMU/KVM. The Windows VM needs to have direct access to the GPU and this is where I ran into problems.
In a nutshell:
1. I blacklisted the amdgpu driver that my RX 570 uses. Ubuntu is left with a Radeon HD5400 running on the radeon driver.
2. I succeeded in isolating the GPU from Ubuntu via pci-stub.
3. I can pass the GPU and what I think is its HDMI sound device to the VM via virt-manager. Adding both devices as a pci host device.
4. Windows 7 VM picks up a single new device.
5. Installing the most recent drivers or manufacturer drivers results in a BSOD upon boot most of the time.
I managed to boot into Windows twice with the RX 570 drivers installed. When I did, the system felt slow, laggy and eventually froze. Im using hugepages, passing CPU cores directly and running OS disk on a SSD. Its blazing fast running on kemus QXL/VGA adapter.
There are two variations in my config from most of the guides I found online:
A. I use pci-stub rather than vfio.
B. I use SeaBIOS rather than UEFI.
The first difference (A) is a result of following a certain guide (see link below), the second difference (B) is my omission during VM setup. Can someone help me troubleshoot the issue / help me understand where Ive gone wrong? Specific questions I have:
Q1: Is the odd Windows behaviour upon passthrough related to the use of SeaBIOS rather than UEFI? I really hope it isnt. My Windows 7 VM is fully configured and migrating to UEFI would be tough; starting afresh would require rework plus I struggle to boot from the Windows CD with UEFI mode in VM.
Q2: Is the odd Windows behaviour upon passthrough related to the use of pci-stub rather than vfio? If the role of either is to isolate a device and pass to a VM, then it shouldnt matter too much which I use?
Q3: Have I missed anything on the Windows configuration part? Its got all the updates required and recommended, drivers, etc. One thing I noticed on Ubuntu is that the GPU is MSI capable but not using it. Quite likely the case in the VM too but cant tweak the driver settings, hence I cant tell whether this is MSI related.
Q4: Do I need separate / dedicated interaction devices? When using the host VGA, I used the same mouse and keyboard and it worked great. In theory, nothing should change when I continue using the same devices, but screen output is delivered via another HDMI port.
Main guide I followed: https://davidyat.es/2016/09/08/gpu-passthrough/
Command outputs:
uname -a
dmesg | grep AMD-Vi
lspci | grep VGA
lsmod | grep vfio
dmesg | grep pci-stub
dmesg | grep VFIO
Windows VM config
Thanks!
Janusz
Im building a new system based on Ubuntu 17.04. I need to retain a Windows environment so decided to VM it via QEMU/KVM. The Windows VM needs to have direct access to the GPU and this is where I ran into problems.
In a nutshell:
1. I blacklisted the amdgpu driver that my RX 570 uses. Ubuntu is left with a Radeon HD5400 running on the radeon driver.
2. I succeeded in isolating the GPU from Ubuntu via pci-stub.
3. I can pass the GPU and what I think is its HDMI sound device to the VM via virt-manager. Adding both devices as a pci host device.
4. Windows 7 VM picks up a single new device.
5. Installing the most recent drivers or manufacturer drivers results in a BSOD upon boot most of the time.
I managed to boot into Windows twice with the RX 570 drivers installed. When I did, the system felt slow, laggy and eventually froze. Im using hugepages, passing CPU cores directly and running OS disk on a SSD. Its blazing fast running on kemus QXL/VGA adapter.
There are two variations in my config from most of the guides I found online:
A. I use pci-stub rather than vfio.
B. I use SeaBIOS rather than UEFI.
The first difference (A) is a result of following a certain guide (see link below), the second difference (B) is my omission during VM setup. Can someone help me troubleshoot the issue / help me understand where Ive gone wrong? Specific questions I have:
Q1: Is the odd Windows behaviour upon passthrough related to the use of SeaBIOS rather than UEFI? I really hope it isnt. My Windows 7 VM is fully configured and migrating to UEFI would be tough; starting afresh would require rework plus I struggle to boot from the Windows CD with UEFI mode in VM.
Q2: Is the odd Windows behaviour upon passthrough related to the use of pci-stub rather than vfio? If the role of either is to isolate a device and pass to a VM, then it shouldnt matter too much which I use?
Q3: Have I missed anything on the Windows configuration part? Its got all the updates required and recommended, drivers, etc. One thing I noticed on Ubuntu is that the GPU is MSI capable but not using it. Quite likely the case in the VM too but cant tweak the driver settings, hence I cant tell whether this is MSI related.
Q4: Do I need separate / dedicated interaction devices? When using the host VGA, I used the same mouse and keyboard and it worked great. In theory, nothing should change when I continue using the same devices, but screen output is delivered via another HDMI port.
Main guide I followed: https://davidyat.es/2016/09/08/gpu-passthrough/
Command outputs:
uname -a
Code:
Linux X 4.10.0-40-generic #44-Ubuntu SMP Thu Nov 9 14:49:09 UTC 2017 x86_64 x86_64 x86_64 GNU/LinuxCode:
[ 0.984726] AMD-Vi: IOMMU performance counters supported
[ 0.986773] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40
[ 0.986774] AMD-Vi: Extended features (0xf77ef22294ada):
[ 0.986776] AMD-Vi: Interrupt remapping enabled
[ 0.986776] AMD-Vi: virtual APIC enabled
[ 0.986884] AMD-Vi: Lazy IO/TLB flushing enabledCode:
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/580] (rev ef)
0b:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cedar [Radeon HD 5000/6000/7350/8350 Series]Code:
vfio_pci 45056 0
vfio_virqfd 16384 1 vfio_pci
irqbypass 16384 2 kvm,vfio_pci
vfio_iommu_type1 24576 0
vfio 32768 2 vfio_iommu_type1,vfio_pciCode:
[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.10.0-40-generic root=UUID=ba28b386-e9e6-4b50-a76e-ccb6b82d846a ro amdgpu.blacklist=1 quiet splash amd_iommu=on pci-stub.ids=1002:67df,1002:aaf0 vt.handoff=7
[ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.10.0-40-generic root=UUID=ba28b386-e9e6-4b50-a76e-ccb6b82d846a ro amdgpu.blacklist=1 quiet splash amd_iommu=on pci-stub.ids=1002:67df,1002:aaf0 vt.handoff=7
[ 3.536165] pci-stub: add 1002:67DF sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[ 3.536176] pci-stub 0000:01:00.0: claimed by stub
[ 3.536184] pci-stub: add 1002:AAF0 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[ 3.536187] pci-stub 0000:01:00.1: claimed by stubCode:
[ 4.131697] VFIO - User Level meta-driver version: 0.3Code:
<domain type='kvm'>
<name>Win7VM</name>
<uuid>4553eac9-6699-49d6-b9e7-d520945250dc</uuid>
<memory unit='KiB'>8388608</memory>
<currentMemory unit='KiB'>8388608</currentMemory>
<memoryBacking>
<hugepages/>
</memoryBacking>
<vcpu placement='static' current='6'>12</vcpu>
<os>
<type arch='x86_64' machine='pc-i440fx-zesty'>hvm</type>
</os>
<features>
<acpi/>
<apic/>
<hyperv>
<relaxed state='on'/>
<vapic state='on'/>
<spinlocks state='on' retries='8191'/>
</hyperv>
</features>
<cpu mode='host-passthrough'>
<topology sockets='1' cores='6' threads='2'/>
</cpu>
<clock offset='localtime'>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='pit' tickpolicy='delay'/>
<timer name='hpet' present='no'/>
<timer name='hypervclock' present='yes'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>restart</on_crash>
<pm>
<suspend-to-mem enabled='no'/>
<suspend-to-disk enabled='no'/>
</pm>
<devices>
<emulator>/usr/bin/kvm-spice</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='raw' cache='none' io='native'/>
<source file='/vm/WindowsVM/Win7VM_D2.img'/>
<target dev='vda' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
</disk>
<disk type='file' device='disk'>
<driver name='qemu' type='raw' cache='none' io='native'/>
<source file='/var/lib/libvirt/images/Win7VM_D1.img'/>
<target dev='vdb' bus='virtio'/>
<boot order='1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</disk>
<disk type='block' device='cdrom'>
<driver name='qemu' type='raw'/>
<source dev='/dev/sr0'/>
<target dev='hdb' bus='ide'/>
<readonly/>
<address type='drive' controller='0' bus='0' target='0' unit='1'/>
</disk>
<controller type='usb' index='0' model='ich9-ehci1'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci1'>
<master startport='0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci2'>
<master startport='2'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci3'>
<master startport='4'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>
</controller>
<controller type='pci' index='0' model='pci-root'/>
<controller type='ide' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
</controller>
<controller type='virtio-serial' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
</controller>
<interface type='network'>
<mac address='52:54:00:35:b3:4e'/>
<source network='default'/>
<model type='rtl8139'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<serial type='pty'>
<target port='0'/>
</serial>
<console type='pty'>
<target type='serial' port='0'/>
</console>
<channel type='spicevmc'>
<target type='virtio' name='com.redhat.spice.0'/>
<address type='virtio-serial' controller='0' bus='0' port='1'/>
</channel>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<graphics type='spice' autoport='yes'>
<listen type='address'/>
</graphics>
<sound model='ich6'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</sound>
<video>
<model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
</video>
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</source>
<address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
</source>
<address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/>
</hostdev>
<redirdev bus='usb' type='spicevmc'>
<address type='usb' bus='0' port='2'/>
</redirdev>
<redirdev bus='usb' type='spicevmc'>
<address type='usb' bus='0' port='3'/>
</redirdev>
<memballoon model='virtio'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
</memballoon>
</devices>
</domain>Janusz