[server] Trouble with PCI Passthrough of NIC to Guest VM on KVM

I recently acquired an HP ProLiant DL380 G7, and I'm wanting to create a virtualized environment with it. I figure it'll be a neat learning experience, but I've hit a point where I need some help. My intended architecture will look something like this:

• The host OS will be Ubuntu Server 16.04 LTS, running KVM (or libvirt, or virsh - to be honest, I'm not sure what the appropriate terminology here is, I'm welcome to corrections).
• My first virtual machine will be a pfSense router. I intend for one of my server's four NICs to be passed through to this VM, effectively exposing it to the outside world.
• The router's 2nd nic will be attached to a vswitch, and other VMs will attach to this vswitch as well, using the pfSense router for DHCP and Internet access. The host will also receive internet access through this vswitch, via a virtual interface.
• A 2nd physical interface on the server will be routed to this vswitch, so that physical hosts can also use the virtualized pfsense router for internet access.
• One virtual machine will run Windows, and will have a dedicated GPU attached to it via pci-passthrough.

So, that's the plan. And its coming along quite well. I've verified that guests are able to attach to the vswitch and get an IP address from the dhcp on the pfsense router. But I'm having issues getting one of my NICs passed through to the pfsense router. Currently, its wan interface is NAT'ed through the host, which I feel defeats the purpose of the router. I've been trying to get pcie passthrough working on this one particular nic, and failing. I'd like to ask you all for help with getting this working and with clearing up any misconceptions I've developed along the way.

The Problem
I'm unable to pass my nic "enp3s0f0" through to a KVM virtual machine. When I attempt to start said virtual machine, I receive one of a few handfuls of errors.

• First error: "fio: error, group 14 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver"
  
  • I've tried to fix this one by finding all devices in group 14, and binding them to the vfio driver. Group 14 contained all four of my network interface cards and my iLO controller.
  • I used this script to perform the binding:
    
    Code:

    ---

    #!/bin/bash
modprobe vfio-pci
for dev in "$@"; do
        vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
        device=$(cat /sys/bus/pci/devices/$dev/device)
        if [ -e /sys/bus/pci/devices/$dev/driver ]; then
                echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
        fi
        echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id
              echo $vendor $device
done

    ---

  • This caused my host to loose network connectivity. All my ssh sessions died, and I lost iLO connectivity. (For reference, iLO is sharing my first Ethernet interface. The interface essentially has 2 mac addresses on it. I'm doing it this way instead of using the dedicated iLO port because I was out of interfaces on my switch.)
• Next error: "failed to set iommu for container: operation not permitted"
  
  • I don't even know where to start with this one.

First off, I'll establish some sanity checks.

• I've ensured that virtualization is on in my bios, including Intel VT-D. I've also added this line to /etc/default/grub:
  
  Code:

  ---

  GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on pcie_acs_override=downstream"

  ---

  I'm not entirely sure what pcie_acs_override does, but it was suggested during my googling. Can anyone explain? I do like understanding how these things work.
• I can verify that iommu is on by running "dmesg | grep -i "iommu" and seeing that iommu is adding devices to groups. I can see where my nic, with device ID 0000:03:00.0 is being added to group 14, along with my other 3 nics and the iLO hardware.
• When I run
  Code:

  ---

  lspci -nnk

  ---

  , and I look for all the devices in group 14, I see, "kernel driver in use: vfio-pci" after executing the aforementioned script.

If any more output is needed, please let me know and I'll be happy to provide. I'm sure you can understand how much I'd like to get over this hurdle, especially since I'm wanting to do GPU passthrough in the future.