Cannot get vfio-pci to grab nvidia 980ti

I am running a clean install of Kubuntu 17.10 with the following hardware.
I7-7700k
16gb ram
ga-z270xp-sli Mobo
GTX 980ti (pcie slot1)
GTX 1050 (pcie slot2)

I am desperately trying to switch back to Linux, but I need a Windows VM running for gaming and CAD work. I have been trying all week to get the nouveau drivers to let go of my 980ti as I would like to pass that card through to a KVM running Windows10. While keeping the 1050 for Linux to use.

I have followed numerous tutorials that walk through how to do this, but no matter what I do I end up with the same result. VFIO will grab the cards HDMI audio chip, but not the GPU.

I have tried physically swapping the cards pcie slots, and it made no difference.

My last ditch effort was to remove to the 1050, and install an old AMD HD5450 for Linux to use, so I could just black list the nouveau drivers completely, but sadly this just resulted in kubuntu not booting at all. 😭

I can use the HD5450 if I absolutely have to, but I would much much rather use the 1050.

Please any help is appreciated. I have done this through unraid numerous times and it so easy. But now I’d like to actually do it on my desktop.