Just a quick note to help others trying the Asrock FM2A88M Extreme 4+ with the AMD a8-5600k processor.
After many different tries with xen 4.1-3 and HVM i finally found a solution that works and restarts without windows bsod errors or crashes.
Hardware :
Asrock FM2A88M Extreme 4+
AMD A8-5600K
16 Gb ddr3 ram.
Nvidia Quadro fx570 -pcie - 1st Slot and selected as primary.
AMD HD6950 - 2nd Pcie slot.
Kernel : 3.13.0-19
Config items ;
Onboard Video (APU) is disabled at present,it has been enabled and does work but radeon blacklisting confuses the issue.
Bios - Version 2.00 January 2014,although i did let Asrock know of the ioapic issue in the bios tables,they seem content with
using redirects for this.
Asrock ; --
Communications with Asrock have been fruitful ,thankyou for your support :),most of their boards work with Vt-d and IOMMU ,and they are very keen to help.
(A)-
Mainboard bios settings have the onboard coms ports and onboard video disabled,the coms ports are disabled as the local apic 0x05 is not defined and the only work around as you can see by the iovrs redirect in the grub cli below,0x05 being the com port1 on the mainboard with its parent id being the pci isa bridge port.
1 -
Basic Xen install of Xen Hypervisor AMD 64 via sudo apt-get,along with xen tools and qemu.
2 -
I use the pciback script to attached the pci devices in /etc/xen/pciback.conf to the hvm.
pciback.conf
0000:00:01.0 # apu
0000:00:01.1 # apu sound
0000:02:00.0
0000:02:00.1
#
0000:00:13.0 # USB and sound
0000:00:10.1
0000:00:10.0
0000:00:14.2
3-
I use the /etc/initramfs/modules method of hiding the devices via the xen-pciback module,config is shown below and not hide or enable the module
in the grub config for xen.
/etc/initramfs/modules ; Hiding both the hd6950 and the onboard apu adapter.
xen-pciback passthrough=1 hide=(02:00.0)(02:00.1)(00:01.1)(00:01.0)
update-initramfs -k all -c after modify this file ,update-grub and reboot.
3-
I dont use an LVM or PV as this is HVM combined with PV thanks to the dev team ,and i created a boot image using dd and specified that in the hvm.cfg file shwon below.
Show here is the important parts of /etc/xen/hvm.cfg using the xl toolstack ,as xm and 4.3 worked well with the network manager on the host.
vcpus = "5"
pae = 1
acpi = 1
apic = 1
disk = ['file:/yourdrive/vm1.img,hda,w' , 'phy:/dev/cdrom,hdc:cdrom,r' ] # image created using dd and mkdosfs tools.
vif = [ 'bridge=xenbr0,mac=00:16:3E:69:7f:02,type=ioemu,mo del=e1000' ]
pci=[ '02:00.0', '02:00.1', '00:10.1', '00:10.0', '00:14.2', '00:13.0' ]
# xen_platform_pci = '1'
# pci_power_mgmt = '1'
4-
Networking in Xen 4.4 required the manual setup w/o network manager as since 4.3 and xm tools it is broken ,as far as the xen bridge dhcp is concerend,this could be issues with nm security ? ,but this was amjor headache trying to play with nm and bridges so out the window went the nm method and the old manual methods that work ok.
Setup in /etc/network/interfaces ;
# interfaces(5) file used by ifup(8) and ifdown(8)
auto lo xenbr0
iface lo inet loopback
allow-hotplug eth0
allow-hotplug eth1
iface eth0 inet static
iface eth1 inet static
iface xenbr0 inet static
bridge_ports eth0
bridge_ports eth1
address 192.168.0.1 # IP of Linux Mint dom0
broadcast 192.168.0.255
netmask 255.255.255.0
gateway 192.168.0.2 # IP of router or modem
dns-nameservers 192.168.0.2 # IP(s) of DNS name server(s)
bridge_stp off # disable Spanning Tree Protocol - optional
Setup in /etc/rc.local ;
/sbin/ifconfig xenbr0 192.168.0.1
/sbin/route add default gw 192.168.0.2
/bin/cp /myresolv.conf /etc/
4 - Grub Config - /etc/default/grub
GRUB_DEFAULT="Xen 4.1-amd64"
GRUB_TIMEOUT="10"
GRUB_DISTRIBUTOR="`lsb_release -i -s 2> /dev/null || echo Debian`"
#
GRUB_CMDLINE_LINUX_DEFAULT="radeon.blacklist=1"
#
# GRUB_CMDLINE_LINUX=""
#
GRUB_CMDLINE_XEN="iommu=soft swiotlb=force radeon.blacklist=1 e820-mtrr-clip=false extra_guest_irqs=,18 lapic x2apic=false irqpoll amd_iommu_dump debug,verbose loglvl=all guest_loglvl=all apic_verbosity=debug e820-verbose=true ivrs_ioapic[5]=00:14.0"
The key points in the stability of the guest was the grub config using soft options ,as the iommu=1 was causing instability in the host and guest acpi with shown in the /var/log/xen logs for the guest.
The instability is probably related to the guest ATI drivers and hardware /acpi calls -BSOD fun # ,as the io ranges that are in the bsod are the memory ranges of both the pcie bridge and the shared by the pcie adapter in the host and guest - ffff8803e8d2209c ffff880405e03e08 ffffffff81715a64 ffff8803e8d22000,irqpolling does nothing .
It could be noted also that the win guest doesnt /wouldnt align irq18 correctly ,nor does xen as shown by the cpuc call issues on shutdown - This was using HVM access not the iommu=soft .
Logs ;
* using iommu=1 ,NOT recommended on this hardware .
kernel: [ 8247.094982] irq 18: nobody cared (try booting with the "irqpoll" option)
kernel: [ 8247.094995] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.13.0-24-generic #46-Ubuntu
kernel: [ 8247.095000] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./FM2A88M Extreme4+, BIOS P2.00 01/17/2014
kernel: [ 8247.095004] ffff8803e8d2209c ffff880405e03e08 ffffffff81715a64 ffff8803e8d22000
kernel: [ 8247.095013] ffff880405e03e30 ffffffff810c19d2 ffff8803e8d22000 0000000000000012
kernel: [ 8247.095020] 0000000000000000 ffff880405e03e70 ffffffff810c1e6c 0000000000000012
kernel: [ 8247.095028] Call Trace:
kernel: [ 8247.095032] <IRQ> [<ffffffff81715a64>] dump_stack+0x45/0x56
kernel: [ 8247.095050] [<ffffffff810c19d2>] __report_bad_irq+0x32/0xd0
kernel: [ 8247.095058] [<ffffffff810c1e6c>] note_interrupt+0x1ac/0x200
kernel: [ 8247.095065] [<ffffffff810bf749>] handle_irq_event_percpu+0xd9/0x1d0
kernel: [ 8247.095072] [<ffffffff810bf87d>] handle_irq_event+0x3d/0x60
kernel: [ 8247.095078] [<ffffffff810c29ea>] handle_fasteoi_irq+0x5a/0x100
kernel: [ 8247.095086] [<ffffffff8142917f>] __xen_evtchn_do_upcall+0x20f/0x320
kernel: [ 8247.095094] [<ffffffff8142b36b>] xen_evtchn_do_upcall+0x2b/0x50
kernel: [ 8247.095102] [<ffffffff81727ebe>] xen_do_hypervisor_callback+0x1e/0x30
kernel: [ 8247.095105] <EOI> [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
kernel: [ 8247.095118] [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
kernel: [ 8247.095126] [<ffffffff81009e60>] ? xen_safe_halt+0x10/0x20
kernel: [ 8247.095133] [<ffffffff8101c5ff>] ? default_idle+0x1f/0xc0
kernel: [ 8247.095140] [<ffffffff8101cec6>] ? arch_cpu_idle+0x26/0x30
kernel: [ 8247.095145] [<ffffffff810beb85>] ? cpu_startup_entry+0xc5/0x290
kernel: [ 8247.095153] [<ffffffff81703f37>] ? rest_init+0x77/0x80
kernel: [ 8247.095162] [<ffffffff81d34f70>] ? start_kernel+0x438/0x443
kernel: [ 8247.095169] [<ffffffff81d34941>] ? repair_env_string+0x5c/0x5c
kernel: [ 8247.095177] [<ffffffff81d345ee>] ? x86_64_start_reservations+0x2a/0x2c
kernel: [ 8247.095183] [<ffffffff81d384a7>] ? xen_start_kernel+0x55e/0x56a
kernel: [ 8247.095188] handlers:
kernel: [ 8247.095194] [<ffffffff81539520>] usb_hcd_irq
kernel: [ 8247.095198] [<ffffffff81539520>] usb_hcd_irq
kernel: [ 8247.095202] Disabling IRQ #18
* The above issue does not occur with the grub options for iommu and swiotlb=force
3-
The normal update-grub is required.
4-
Grub setup was interesting as unlike the Intel Vt-d setup i have on another asrock mainboard the AMD board has an ioapic issues with com 1 ( ioapic 0x05 ) ,you will see this in the xl dmesg with a standard xen grub w/o the redirection of the 0x05 apic back to the pci-isa bridge.
5-
This setup runs pci passthrough successfully with the latest AMD drivers on a w7 guest,drivers only not ccc,which requries a manual extract of the package utilising 7zip or archiver.
6-
Updates i think would benefit users & admins in Xen future releases ;
Ability to define and reserve the irq for pci passthough and memory addresses in the pv /hvm config file .
Ability to configure the qemu /pv bios for various settings realted to pcie lane control /gen.
Win8 compatibility ,this isnt workign as yet.
AMD chipset option in the HVM as there seems to be some issues with the AMD machines.
7- Updated - ATI BSOD FIX
Sorry forgot to include my workaround for the bsod caused by the amdkmdag.sys driver loading early with two adapters and the ati cards
ability to search for crossfire pairs on boot prior to loading the first screen,which is what passthrough card seems to be doing via debuggin ,
although i didnt have these on a predictable basis on boot only.
Win reliability - BSOD atikdag.sys & atikmpag.sys issues.
a- The workaround for this is to enable the hidden device ( amdkmdag.sys) ( non -pnp driver) ,see this in the device manager afte showing hidden devices.
This must be modified to start after the ati device has detected and rendered,to do this the following steps must be carried out.
b- Backup your registry and create a restore point.
c- The registry keys /HKEY_LOCAL_MACHINE\SYSTEM\CONTROLSET001\SERVICES\A MDKMDAG Start option to be changed from
the existing demand startup parameter to the automatic startup requirement,this is done by changed the Start hex value from 3 to the value of 2.
d- Also change the ULPS function EnableULPS from 1 to 0 .
e- The hidden device in device manager should now show it start point as automatic and not demand.
f - Shutdown or Reboot the xen Domu.
Not too much !.
;).
Thankyou Xen team ! .
All the best ! .
After many different tries with xen 4.1-3 and HVM i finally found a solution that works and restarts without windows bsod errors or crashes.
Hardware :
Asrock FM2A88M Extreme 4+
AMD A8-5600K
16 Gb ddr3 ram.
Nvidia Quadro fx570 -pcie - 1st Slot and selected as primary.
AMD HD6950 - 2nd Pcie slot.
Kernel : 3.13.0-19
Config items ;
Onboard Video (APU) is disabled at present,it has been enabled and does work but radeon blacklisting confuses the issue.
Bios - Version 2.00 January 2014,although i did let Asrock know of the ioapic issue in the bios tables,they seem content with
using redirects for this.
Asrock ; --
Communications with Asrock have been fruitful ,thankyou for your support :),most of their boards work with Vt-d and IOMMU ,and they are very keen to help.
(A)-
Mainboard bios settings have the onboard coms ports and onboard video disabled,the coms ports are disabled as the local apic 0x05 is not defined and the only work around as you can see by the iovrs redirect in the grub cli below,0x05 being the com port1 on the mainboard with its parent id being the pci isa bridge port.
1 -
Basic Xen install of Xen Hypervisor AMD 64 via sudo apt-get,along with xen tools and qemu.
2 -
I use the pciback script to attached the pci devices in /etc/xen/pciback.conf to the hvm.
pciback.conf
0000:00:01.0 # apu
0000:00:01.1 # apu sound
0000:02:00.0
0000:02:00.1
#
0000:00:13.0 # USB and sound
0000:00:10.1
0000:00:10.0
0000:00:14.2
3-
I use the /etc/initramfs/modules method of hiding the devices via the xen-pciback module,config is shown below and not hide or enable the module
in the grub config for xen.
/etc/initramfs/modules ; Hiding both the hd6950 and the onboard apu adapter.
xen-pciback passthrough=1 hide=(02:00.0)(02:00.1)(00:01.1)(00:01.0)
update-initramfs -k all -c after modify this file ,update-grub and reboot.
3-
I dont use an LVM or PV as this is HVM combined with PV thanks to the dev team ,and i created a boot image using dd and specified that in the hvm.cfg file shwon below.
Show here is the important parts of /etc/xen/hvm.cfg using the xl toolstack ,as xm and 4.3 worked well with the network manager on the host.
vcpus = "5"
pae = 1
acpi = 1
apic = 1
disk = ['file:/yourdrive/vm1.img,hda,w' , 'phy:/dev/cdrom,hdc:cdrom,r' ] # image created using dd and mkdosfs tools.
vif = [ 'bridge=xenbr0,mac=00:16:3E:69:7f:02,type=ioemu,mo del=e1000' ]
pci=[ '02:00.0', '02:00.1', '00:10.1', '00:10.0', '00:14.2', '00:13.0' ]
# xen_platform_pci = '1'
# pci_power_mgmt = '1'
4-
Networking in Xen 4.4 required the manual setup w/o network manager as since 4.3 and xm tools it is broken ,as far as the xen bridge dhcp is concerend,this could be issues with nm security ? ,but this was amjor headache trying to play with nm and bridges so out the window went the nm method and the old manual methods that work ok.
Setup in /etc/network/interfaces ;
# interfaces(5) file used by ifup(8) and ifdown(8)
auto lo xenbr0
iface lo inet loopback
allow-hotplug eth0
allow-hotplug eth1
iface eth0 inet static
iface eth1 inet static
iface xenbr0 inet static
bridge_ports eth0
bridge_ports eth1
address 192.168.0.1 # IP of Linux Mint dom0
broadcast 192.168.0.255
netmask 255.255.255.0
gateway 192.168.0.2 # IP of router or modem
dns-nameservers 192.168.0.2 # IP(s) of DNS name server(s)
bridge_stp off # disable Spanning Tree Protocol - optional
Setup in /etc/rc.local ;
/sbin/ifconfig xenbr0 192.168.0.1
/sbin/route add default gw 192.168.0.2
/bin/cp /myresolv.conf /etc/
4 - Grub Config - /etc/default/grub
GRUB_DEFAULT="Xen 4.1-amd64"
GRUB_TIMEOUT="10"
GRUB_DISTRIBUTOR="`lsb_release -i -s 2> /dev/null || echo Debian`"
#
GRUB_CMDLINE_LINUX_DEFAULT="radeon.blacklist=1"
#
# GRUB_CMDLINE_LINUX=""
#
GRUB_CMDLINE_XEN="iommu=soft swiotlb=force radeon.blacklist=1 e820-mtrr-clip=false extra_guest_irqs=,18 lapic x2apic=false irqpoll amd_iommu_dump debug,verbose loglvl=all guest_loglvl=all apic_verbosity=debug e820-verbose=true ivrs_ioapic[5]=00:14.0"
The key points in the stability of the guest was the grub config using soft options ,as the iommu=1 was causing instability in the host and guest acpi with shown in the /var/log/xen logs for the guest.
The instability is probably related to the guest ATI drivers and hardware /acpi calls -BSOD fun # ,as the io ranges that are in the bsod are the memory ranges of both the pcie bridge and the shared by the pcie adapter in the host and guest - ffff8803e8d2209c ffff880405e03e08 ffffffff81715a64 ffff8803e8d22000,irqpolling does nothing .
It could be noted also that the win guest doesnt /wouldnt align irq18 correctly ,nor does xen as shown by the cpuc call issues on shutdown - This was using HVM access not the iommu=soft .
Logs ;
* using iommu=1 ,NOT recommended on this hardware .
kernel: [ 8247.094982] irq 18: nobody cared (try booting with the "irqpoll" option)
kernel: [ 8247.094995] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.13.0-24-generic #46-Ubuntu
kernel: [ 8247.095000] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./FM2A88M Extreme4+, BIOS P2.00 01/17/2014
kernel: [ 8247.095004] ffff8803e8d2209c ffff880405e03e08 ffffffff81715a64 ffff8803e8d22000
kernel: [ 8247.095013] ffff880405e03e30 ffffffff810c19d2 ffff8803e8d22000 0000000000000012
kernel: [ 8247.095020] 0000000000000000 ffff880405e03e70 ffffffff810c1e6c 0000000000000012
kernel: [ 8247.095028] Call Trace:
kernel: [ 8247.095032] <IRQ> [<ffffffff81715a64>] dump_stack+0x45/0x56
kernel: [ 8247.095050] [<ffffffff810c19d2>] __report_bad_irq+0x32/0xd0
kernel: [ 8247.095058] [<ffffffff810c1e6c>] note_interrupt+0x1ac/0x200
kernel: [ 8247.095065] [<ffffffff810bf749>] handle_irq_event_percpu+0xd9/0x1d0
kernel: [ 8247.095072] [<ffffffff810bf87d>] handle_irq_event+0x3d/0x60
kernel: [ 8247.095078] [<ffffffff810c29ea>] handle_fasteoi_irq+0x5a/0x100
kernel: [ 8247.095086] [<ffffffff8142917f>] __xen_evtchn_do_upcall+0x20f/0x320
kernel: [ 8247.095094] [<ffffffff8142b36b>] xen_evtchn_do_upcall+0x2b/0x50
kernel: [ 8247.095102] [<ffffffff81727ebe>] xen_do_hypervisor_callback+0x1e/0x30
kernel: [ 8247.095105] <EOI> [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
kernel: [ 8247.095118] [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
kernel: [ 8247.095126] [<ffffffff81009e60>] ? xen_safe_halt+0x10/0x20
kernel: [ 8247.095133] [<ffffffff8101c5ff>] ? default_idle+0x1f/0xc0
kernel: [ 8247.095140] [<ffffffff8101cec6>] ? arch_cpu_idle+0x26/0x30
kernel: [ 8247.095145] [<ffffffff810beb85>] ? cpu_startup_entry+0xc5/0x290
kernel: [ 8247.095153] [<ffffffff81703f37>] ? rest_init+0x77/0x80
kernel: [ 8247.095162] [<ffffffff81d34f70>] ? start_kernel+0x438/0x443
kernel: [ 8247.095169] [<ffffffff81d34941>] ? repair_env_string+0x5c/0x5c
kernel: [ 8247.095177] [<ffffffff81d345ee>] ? x86_64_start_reservations+0x2a/0x2c
kernel: [ 8247.095183] [<ffffffff81d384a7>] ? xen_start_kernel+0x55e/0x56a
kernel: [ 8247.095188] handlers:
kernel: [ 8247.095194] [<ffffffff81539520>] usb_hcd_irq
kernel: [ 8247.095198] [<ffffffff81539520>] usb_hcd_irq
kernel: [ 8247.095202] Disabling IRQ #18
* The above issue does not occur with the grub options for iommu and swiotlb=force
3-
The normal update-grub is required.
4-
Grub setup was interesting as unlike the Intel Vt-d setup i have on another asrock mainboard the AMD board has an ioapic issues with com 1 ( ioapic 0x05 ) ,you will see this in the xl dmesg with a standard xen grub w/o the redirection of the 0x05 apic back to the pci-isa bridge.
5-
This setup runs pci passthrough successfully with the latest AMD drivers on a w7 guest,drivers only not ccc,which requries a manual extract of the package utilising 7zip or archiver.
6-
Updates i think would benefit users & admins in Xen future releases ;
Ability to define and reserve the irq for pci passthough and memory addresses in the pv /hvm config file .
Ability to configure the qemu /pv bios for various settings realted to pcie lane control /gen.
Win8 compatibility ,this isnt workign as yet.
AMD chipset option in the HVM as there seems to be some issues with the AMD machines.
7- Updated - ATI BSOD FIX
Sorry forgot to include my workaround for the bsod caused by the amdkmdag.sys driver loading early with two adapters and the ati cards
ability to search for crossfire pairs on boot prior to loading the first screen,which is what passthrough card seems to be doing via debuggin ,
although i didnt have these on a predictable basis on boot only.
Win reliability - BSOD atikdag.sys & atikmpag.sys issues.
a- The workaround for this is to enable the hidden device ( amdkmdag.sys) ( non -pnp driver) ,see this in the device manager afte showing hidden devices.
This must be modified to start after the ati device has detected and rendered,to do this the following steps must be carried out.
b- Backup your registry and create a restore point.
c- The registry keys /HKEY_LOCAL_MACHINE\SYSTEM\CONTROLSET001\SERVICES\A MDKMDAG Start option to be changed from
the existing demand startup parameter to the automatic startup requirement,this is done by changed the Start hex value from 3 to the value of 2.
d- Also change the ULPS function EnableULPS from 1 to 0 .
e- The hidden device in device manager should now show it start point as automatic and not demand.
f - Shutdown or Reboot the xen Domu.
Not too much !.
;).
Thankyou Xen team ! .
All the best ! .