Windows 10 virtual machine on PopOS 20.04 with GPU passthrough

July 6, 2021
Tags: GPU passthrough, Pop OS, VFIO, Virtual Machine, Windows 10

In this guide I will go through how I got Windows 10 to run on a virtual machine under PopOS 20.04 with GPU passthrough. I will passthrough a Nvidia RTX 2080 GPU on the first PCI-e slot of the motherboard to Windows 10. Here’s the whole specs of my system:

AMD Ryzen 3950X (later updated to 5950X)
Asrock X570 Creator motherboard (BIOS version: 3.30, AGESA version: AMD AGESA Combo-AM4 V2 1.1.0.0 patch D)
MSI RTX 2080 Sea Hawk EK X GPU (on the first PCI-e x16 slot, for Windows 10)
AMD Radeon VII GPU (on the second PCI-e x16 slot, for PopOS)
64GB DDR4
Inateck 2 port USB 3.0 PCI-E card (on the third PCI-e x4 slot for Windows 10)
PopOS 20.04 as host OS
Windows 10 as guest OS

USB on Windows 10

As you can see from the specs I will also passthrough an USB PCI-e card, as it can be quite difficult to passthrough any USB ports that are directly on your motherboard. For example I tried to passthrough some USB ports in my motherboard to Windows, but unfortunately it didn’t work or it resulted in hanging up the whole system. That’s why I would recommend getting a seperate USB PCI-e card. Other solution is to passthrough each USB device by device (instead of passthrough the whole USB port), but in that case when you want to passthrough a new USB device, you would need to then add that device to the list of devices that you want to be passed through. Second problem that you might encounter with this solution, is that if your USB device gets disconneted and reconnected again, the virtual machine might not find the USB device again until you restart the virtual machine. For example I have a hardware KVM which I use to switch my mouse and keyboard to different computers, in this case the seperate USB PCI-e card is much better solution.

Installing SSH server

My second recommendation would be to install SSH server on your PopOS. Especially if you are going to passthrough your GPU in the first PCI-e slot of your motherboard, as you might stumble to a problem that after you have backlisted the GPU and restarted your computer, PopOS can’t start X Server (basically meaning a blank screen with no graphical interface). We will talk about more how to solve this problem later, if you happen to encounter it. With an SSH server running, you can remote connect to your computer from another computer. To install SSH server run the following command:

sudo apt install openssh-server

Then I would just in case test that you can connect to the computer from another computer. By using the following command from the other computer (if you are using Linux or Mac on the other computer):

ssh your_username@your_ip_address

BIOS settings

You should first enable IOMMU in BIOS (IOMMU allows us to passthrough PCI-e devices). For example in my Asrock motherboard the setting can be found under: Advanced -> AMD CBS -> NBIO Common Settings -> IOMMU

You might also need to enable SR-IOV Support under Advanced -> PCI Configuration -> SR-IOV support. And you might need to enable CSM under Boot -> CSM (Compatibility Support Module) -> CSM

Installing Qemu and enabling IOMMU

First we will start by installing Qemu (handles the virtualization), Virt-manager (graphical interface for QEMU so we can set up the VM a bit more easier) and all the other needed software:

sudo apt install qemu-kvm qemu-utils virt-manager libvirt-daemon-system libvirt-clients bridge-utils ovmf

Enabling IOMMU for AMD CPUs:

sudo kernelstub -o "amd_iommu=on amd_iommu=pt"

Restart your computer.

Identifying PCI bus / device ids

Next we want to find out the PCI bus ids and device ids for the GPU and USB card that we want to passthrough:

lspci -nnv

This should print out a really long list of all the different devices in your computer. You need to look through this list and find your GPU and USB card. For example here’s my Nvidia card:

35:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU104 [GeForce RTX 2080 Rev. A] [10de:1e87] 
(rev a1) (prog-if 00 [VGA controller])
Subsystem: Micro-Star International Co., Ltd. \[MSI\] TU104 \[GeForce RTX 2080 Rev. A\] \[1462:3728\]
Flags: fast devsel, IRQ 255
Memory at dc000000 (32-bit, non-prefetchable) \[size=16M\]
Memory at 90000000 (64-bit, prefetchable) \[size=256M\]
Memory at a0000000 (64-bit, prefetchable) \[size=32M\]
I/O ports at e000 \[size=128\]
Expansion ROM at dd000000 \[disabled\] \[size=512K\]
Capabilities: 
Kernel driver in use: vfio-pci
Kernel modules: nvidiafb, nouveau, nvidia\_drm, nvidia

35:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:10f8] (rev a1)
Subsystem: Micro-Star International Co., Ltd. \[MSI\] Device \[1462:3728\]
Flags: fast devsel, IRQ 255
Memory at dd080000 (32-bit, non-prefetchable) \[disabled\] \[size=16K\]
Capabilities: 
Kernel driver in use: vfio-pci
Kernel modules: snd\_hda\_intel

35:00.2 USB controller [0c03]: NVIDIA Corporation Device [10de:1ad8] (rev a1) (prog-if 30 [XHCI])
Subsystem: Micro-Star International Co., Ltd. \[MSI\] Device \[1462:3728\]
Flags: fast devsel, IRQ 80
Memory at a2000000 (64-bit, prefetchable) \[size=256K\]
Memory at a2040000 (64-bit, prefetchable) \[size=64K\]
Capabilities: 
Kernel driver in use: xhci\_hcd

35:00.3 Serial bus controller [0c80]: NVIDIA Corporation Device [10de:1ad9] (rev a1)
Subsystem: Micro-Star International Co., Ltd. \[MSI\] Device \[1462:3728\]
Flags: fast devsel, IRQ 255
Memory at dd084000 (32-bit, non-prefetchable) \[disabled\] \[size=4K\]
Capabilities: 
Kernel driver in use: vfio-pci
Kernel modules: i2c\_nvidia\_gpu

And here’s my USB card:

2b:00.0 USB controller [0c03]: Fresco Logic FL1100 USB 3.0 Host Controller [1b73:1100] (rev 10) (prog-if 30 [XHCI])
Subsystem: Fresco Logic FL1100 USB 3.0 Host Controller [1b73:1100]
Flags: fast devsel, IRQ 27
Memory at eaa00000 (64-bit, non-prefetchable) [size=64K]
Memory at eaa11000 (64-bit, non-prefetchable) [size=4K]
Memory at eaa10000 (64-bit, non-prefetchable) [size=4K]
Capabilities: 
Kernel driver in use: vfio-pci
Kernel modules: xhci_pci

All the different devices that are related to the GPU are listed consecutively, and they will belong to the same group (in my example: 35:00). You can see from the list that my GPU card has been listed four times: VGA compatible controller (video), Audio device (as most cards can output audio through HDMI/DisplayPort, so you will probably also have this one), USB controller (this might be a bit more rare as my Nvidia card has also an USB-C connector, so you might not have this entry), Serial bus controller. You should then write down the GPU’s PCI bus ids (35:00.0, 35:00.1, 35:00.2, 35:00.3), device ids (10de:1e87, 10de:10f8, 10de:1ad8, 10de:1ad9) and also what kind of device it is (VGA, Audio, USB and Serial Bus). For the device ids it is enough to write down only the ids that are listed on the title of each entry. For example I would write down on my VGA combatible controller only 10de:1e87 (don’t need the 1462:3728).

For my USB card , I would write down PCI bus id 2b:00.0 and device id 1b73:1100.

Blacklisting your GPU and USB card

Next we will blacklist our GPU that we want to passthrough to Windows. Before we blacklist the GPU, there’s few things you should know: The BIOS menu will still show up in the first GPU, but when Pop OS starts to load up, you will see video output only on the second GPU. Second thing is that you might not get any video output at all in PopOS, if the X Server fails to start, but don’t panic, we will fix that problem later. To blacklist our GPU and USB card, run the following command, but remember to replace with your own device ids:

sudo kernelstub --add-options "vfio-pci.ids=10de:1e87, 10de:10f8, 10de:1ad8, 10de:1ad9, 1b73:1100"

And if you happen to make a mistake, for example if you put wrong device ids, you can delete the changes with the following command (for example if I would want to delete the above changes from my kernelstub, I would use the following command):

sudo kernelstub --delete-options "vfio-pci.ids=10de:1e87, 10de:10f8, 10de:1ad8, 10de:1ad9, 1b73:1100"

Next we will run the following command (some certain Windows version require it):

sudo kernelstub --add-options "kvm.ignore_msrs=1"

Next we will disable the EFI/VESA framebuffer when the PopOS is booting (as otherwise the Pop OS will reserve the first GPU card, and then the GPU passthrough might not work):

sudo kernelstub --add-options "video=efifb:off"

Last the step before rebooting, is to blacklist the Nvidia/Nouveau drivers in PopOS, so the Nvidia drivers don’t try to initiliaze the GPU (and then the GPU passthrough would fail). Go to edit the following file (in my example I’m using Emacs to edit it):

sudo emacs /etc/modprobe.d/blacklist.conf

Add the following lines to the end of the file:

blacklist nvidia
blacklist nouveau

Finally reboot your computer. After rebooting your computer you can run the “ls -nnv” command again, and now you should see on your GPU and USB card devices that have “Kernel driver in use” as “vfio-pci” (except for the USB-C port that’s on the Nvidia GPU, if you happen to have an USB-C port in your Nvidia GPU). For example for my VGA controller it would show the following:

35:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU104 [GeForce RTX 2080 Rev. A] [10de:1e87] 
(rev a1) (prog-if 00 [VGA controller])
Subsystem: Micro-Star International Co., Ltd. \[MSI\] TU104 \[GeForce RTX 2080 Rev. A\] \[1462:3728\]
Flags: fast devsel, IRQ 255
Memory at dc000000 (32-bit, non-prefetchable) \[size=16M\]
Memory at 90000000 (64-bit, prefetchable) \[size=256M\]
Memory at a0000000 (64-bit, prefetchable) \[size=32M\]
I/O ports at e000 \[size=128\]
Expansion ROM at dd000000 \[disabled\] \[size=512K\]
Capabilities: 
Kernel driver in use: vfio-pci
Kernel modules: nvidiafb, nouveau, nvidia\_drm, nvidia

Problems after rebooting

If you didn’t have any problems after rebooting then you can skip this section. But if you have any problems after rebooting, like no video signal on your second GPU after PopOS has started to boot (no PopOS boot information and no login screen). Then you might need to have to edit your xorg.conf. If you encrypted your PopOS hard drive then after booting, wait a bit, and then input your encryption password blindly (even though you can’t see the input encryption screen). And then wait a bit again. And then try to SSH from another computer to your computer. Create xorg.conf.d folder under /etc/X11/

sudo mkdir /etc/X11/xorg.conf.d

Then create the 10-gpus.conf file to under that folder:

sudo emacs /etc/X11/xorg.conf.d/10-gpus.conf

And then add the following to the file (this applies to only AMD GPU):

# For AMD
Section "Device"
  Identifier "AMD"
  Driver "amdgpu"
EndSection

Then disconnect SSH and reboot your computer. Hopefully now you can see PopOS booting and the encryption/login screen.

But if you can’t still see anything, then I would recommend to turn off the computer and then switch the GPUs’ places (so put the AMD GPU to the first slot and then Nvidia GPU to the second slot). You should then able to boot normally and you can then try to backtrack, if you missed anything.

Creating a virtual disk for the Windows VM

Now we will create a virtual disk for the Windows VM that will host our Windows VM install. There are different approaches for this (for example you could passthrough a whole NVMe SSD drive), but I have usually created just img-file. Maybe one advantage for this is that if you want to make a backup of the Windows install, then you just copy the img-file. Of course one disadvantage will be that the file size will be as big as the size of the virtual disk (even though you haven’t filled up your virtual disk). We can create a virtual disk with the following command. The 512G option means that the virtual disk will 512GB big. You can make the disk bigger or smaller as you want. The virtual disk can be made bigger later, if you find that you need more space (I will cover this one in a later post).

fallocate -l 512G win10.img

Download necessary files

Download Windows 10 installation ISO from Microsoft (https://www.microsoft.com/software-download/windows10ISO). Then download Virtio drivers for the virtual disk, as otherwise Windows installer won’t recognize the virtual disk. The Virtio drivers can be found at https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/ (Download the stable-virtio version that has a version number on it and that ends with .iso). You can optionally download your GPU drivers and put them on a USB stick, if you prefer to install the GPU drivers without Windows automatic driver installation.

(Optional) Download vBIOS

This is an optional step as in new Nvidia drivers for Windows there shouldn’t be any problems running consumer Nvidia GPUs under VM. But in older Nvidia drivers, Nvidia blocked running consumer GPUs under VMs. Basically after you installed Nvidia drivers in Windows, the drivers wouldn’t load after reboot and Windows’ Device Manager would show Error 43 under the Nvidia GPU. You wouldn’t able to change resolution or use 3D accelaration. To counter the Error 43, we will provide a fresh copy of the GPU’s vBIOS. For example in my case as I will passthrough the GPU in the first PCI-e slot, when I start up my computer the BIOS will show up on Nvidia GPU (before the video output would move to my second GPU when the PopOS will start to load). At this point when the BIOS is loaded, the Nvidia GPU will be marked as initialized. But when I go to my VM side, and I start up my Windows VM, the Nvidia CPU would know that it has been initiliazed in another OS first (so it is running under a VM). And at this point the Nvidia drivers would block using the Nvidia GPU and not load the drivers.

This step doesn’t affect, if you use already new enough drivers (it will work with those too). But you can of course try first without doing this step and then come back if you encounter to the Error 43.

To fix Error 43, we will first download a copy of the GPU’s vBIOS from https://www.techpowerup.com/vgabios/. Find the exact GPU model that you are using and download its vBIOS (you can’t use another manufacturer’s or another version of vBIOS, it needs to be the same manufacturer and exactly the same model otherwise it can be harmful to your GPU).

Then we create vgabios folder under usr/share folder (because usr/share folder is accessible for Qemu, for example if we put the vBIOS under our own home directory PopOS’ AppArmor security might block access to it):

sudo mkdir /usr/share/vgabios

Then we just go to the folder that has our donwloaded vBIOS and move it to the vgabios folder (for example my ROM-file is named MSI):

sudo cp MSI.rom /usr/share/vgabios/MSI.rom

We will add this vBIOS file later to the VM settings, when we get to that point.

Creating the VM in Virtio

After all the pre-configurations we are ready to create the VM in Virtio. Launch Virt-manager. And select File -> New Virtual Machine

From the new window select “Local install media (ISO image or CDROM”).

In the next window click “Browse”, then “Browse Local”, and then select the Windows installer ISO that you downloaded previously.

Next you can choose how much memory and CPUs (threads), you want to assign to your VM. Leave some threads and memory to Pop OS (don’t assign everything to Windows). For example I used 24 threads (so 12 cores) and 49152MiB.

Next we will assign the virtual disk where Windows will be installed that we created before hand. Select “Select or create custom storage” option, and then click “Manage”, then “Browse Local”, and then find your virtual disk.

In the next window enable the “Customize configuration before install” option and hit Finish.

Now we will add the Virtio ISO that we downloaded previously. Click “Add Hardware”.

Choose “Storage”, and then as “Device Type” “CDROM device” and “Bus Type” as”SATA”. Click the “Manage”, and then “Browse Local”, and find the location where the Virtio ISO is.

Next we will add the GPU devices and the USB card that we want to passthrough. At the beginning of this guide we wrote down the PCI bus ids of the GPU and USB card. Click the “Add Hardware” again, then select “PCI Host Device”, and select the PCI bus id that you had written down before. And repeat this until you have added all the devices that you want to passthrough.

(Optional step) Depending on the Nvidia drivers, you might encounter Error 43 inside Windows. To avoid this we will give a fresh copy of CPU’s vBios that we downloaed earlierly to the VM. Select your VGA controller (in my case it’s the 0000:32:00:0). And go to the XML-tab and add the line “<rom bar=”on” file=”/usr/share/vgabios/MSI.rom”/>”. And click Apply. You don’t need to add the vBios to other deviced related to your GPU.

Installing Windows

Finally we are ready to start the VM. Optionally if you prefer to Windows not automatically install your GPU drivers, but install the drivers yourself, then you should now disable your network connection. Now we click the Run button to start the VM. And then the Monitor button to open the monitor tab. Because we don’t yet have any drivers, you might not get any video output through your GPU at this point. But don’t panic, we can see video output inside the monitor tab (of course this won’t be GPU accelerated, but this shouldn’t matter for the intallation process).

During the Windows installation you might notice that Windows can’t find a disk to install the OS. This is because we need to load the Virtio drivers that we donwloaded earlierly. Click “Load driver” and point to the Virtio disc. The drivers should be inside the “amd64/w10” folder. Now you should able to continue with the installation.

After Windows has finished installing, we can install the Nvidia drivers. Depending if you disabled your network connection, then you need to insert the USB stick with the GPU drivers and install from there. Or if you didn’t disable your network connection, then Windows should automatically recognize your GPU and install the correct drivers. After the drivers have finished installing, you should able to extend the desktop to the monitor that is connected to your GPU. You can disable the “virtual monitor” from Windows’ monitor settings. Or you can even delete the Spice Server/Screen in your VM settings to completely disable the “virtual monitor”.

Now you should have a Windows VM up and running! I hope that this guide has been helpful. If you notice that there’s some errors in the guide, or if you have encountered any problems, please leave a comment. I posted some problems that I have encountered myself below.

Problems:

Slow upload speed with Dropbox

I noticed that some times I had really slow upload speeds in Dropbox some times for some reason. I haven’t yet found a reason why this happens when I use the shared ethernet connection. Currently I have passthrough the second Ethernet connection to the VM (as my motherboard has two Ethernet ports). And I haven’t noticed any slow down anymore.

VM not starting up

If for some reason, your VM doesn’t start, for example you get an error, but the error message isn’t that clear what’s the problem, then you can try to see Qemu’s log file (replace the “win10” with the name that you gave to your VM in Virt-manager):

sudo emacs /var/log/libvirt/qemu/win10.log

How to check what you have added to your kernelstub

If you want to check what you have added to your kernelstub, for example you have some problems with the VM and you want to double check, if everything is correct in the kernelstub. Or if you bought a new GPU, and you want to remove the old one, but you don’t remeber exactly what you have added to the kernelstub. You can quickly check with the following command:

cat /proc/cmdline

For example I get the following output (remember not to remove the initrd or root, as they relate which hard drive to use to boot your Pop OS):

initrd=\EFI\Pop_OS-********-****-****-****-************\initrd.img root=UUID=********-****-****-****-************ ro kvm.ignore_msrs=1 amd_iommu=on amd_iommu=pt video=efifb:off vfio-pci.ids=10de:1e87,10de:10f8,10de:1ad8,10de:1ad9,1b73:1100

Ignored rdmsr problem when booting PopOS

If you are booting your PopOS, but the bootup fails, and you get a black screen, and the following error message (the number values might be different for you):

[27.508658] kvm [2055]: ignored rdmsr: 0x48b data 0x0

This might be that the IOMMU and SR-IOV settings in your motherboard’s BIOS have been disabled (because of a BIOS reset). This might happen if you have updated your motherboard’s BIOS. Or for example if there has been some electric blackout during at the same time as you were using your computer and the computer then has shutdown incorrectly and caused a BIOS reset. So you should then check your BIOS settings that the IOMMU and SR-IOV are enabled (and that other settings are also correct).

Updates

-21. November 2021 added ignored rdmsr problem

Useful links and sources:

Few useful guides from Mathias Hueber that have helped me a lot:

https://mathiashueber.com/pci-passthrough-ubuntu-2004-virtual-machine/

https://mathiashueber.com/windows-virtual-machine-gpu-passthrough-ubuntu/

Level1Techs have really good forums about VMs:

https://forum.level1techs.com/c/software/vfio/132

Discussion about vBIOS and Pop OS’ AppArmor:

https://askubuntu.com/questions/985854/gpu-passthrough-problem-on-adding-dumped-rom

Resizing your virtual machine's IMG virtual disk image