Check that the integrated GPU is the primary GPU.
If necessary, enable IOMMU in the BIOS.
Edit /etc/default/grub. Append
intel_iommu=on
to the value ofGRUB_CMDLINE_LINUX_DEFAULT
:... GRUB_CMDLINE_LINUX_DEFAULT="quiet splash intel_iommu=on" ...
Update grub:
$ sudo update-grub
Reboot
Identify the PCIe bus the GPU we're passing through is on:
$ lspci -nnk ... 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM107 [GeForce GTX 750] [10de:1381] (rev a2) Subsystem: Gigabyte Technology Co., Ltd GM107 [GeForce GTX 750] [1458:362e] Kernel driver in use: nvidia Kernel modules: nvidiafb, nouveau, nvidia_384_drm, nvidia_384 01:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fbc] (rev a1) Subsystem: Gigabyte Technology Co., Ltd Device [1458:362e] Kernel driver in use: snd_hda_intel Kernel modules: snd_hda_intel ...
Check what group the graphics card for Windows is in:
$ find /sys/kernel/iommu_groups/ -type l
Check that only the graphics card for Windows is in the IOMMU group:
$ ls -lhA /sys/bus/pci/devices/0000\:01\:00.0/iommu_group/devices/ total 0 ... 0000:00:01.0 -> ../../../../devices/pci0000:00/0000:00:01.0 ... 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0 ... 0000:01:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.1
Blacklist the GPU we're passing through to the VM so that the graphics driver can't grab it. We use the pci-stub module to claim the card before nvidia or nouveau can. Add "pci-stub" to /etc/initramfs-tools/modules:
$ echo "pci_stub" | sudo tee -a /etc/initramfs-tools/modules
(Credit: Weber K.) You can add options to kernel modules in /etc/initramfs-tools/modules, but we will add them in /lib/modprobe.d/pci-stub.conf. Pass pci-stub the IDs of the graphics controller and the audio device (as found with
lspci -nnk
above). Set pci-stub as a dependency for drm, otherwise the graphics driver will be loaded before the pci-stub driver. (Check dmesg to see when this happens.)$ sudo vim /lib/modprobe.d/pci-stub.conf options pci-stub ids=10de:1381,10de:0fbc softdep drm pre: pci-stub
Update the existing initramfs image:
$ sudo update-initramfs -u
Reboot
Confirm that pci-stub claimed the devices:
$ lspci -nnk ... 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM107 [GeForce GTX 750] [10de:1381] (rev a2) Subsystem: Gigabyte Technology Co., Ltd GM107 [GeForce GTX 750] [1458:362e] Kernel driver in use: pci-stub Kernel modules: nvidiafb, nouveau, nvidia_384_drm, nvidia_384 01:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fbc] (rev a1) Subsystem: Gigabyte Technology Co., Ltd Device [1458:362e] Kernel driver in use: pci-stub Kernel modules: snd_hda_intel ...
Create a "vfio-bind" script to replace the pci-stub placeholder driver with vfio-pci. Put it in /usr/local/bin/vfio-bind:
$ sudo vim /usr/local/bin/vfio-bind #!/bin/bash modprobe vfio-pci for dev in "$@" do vendor=$(cat /sys/bus/pci/devices/$dev/vendor) device=$(cat /sys/bus/pci/devices/$dev/device) if [ -e /sys/bus/pci/devices/$dev/driver ]; then echo $dev > /sys/bus/pci/devices/$dev/driver/unbind fi echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id done $ sudo chmod +x /usr/local/bin/vfio-bind
Run the script with the PCIe bus IDs of the devices in the IOMMU group:
$ sudo vfio-bind 0000:01:00.0 0000:01:00.1
Confirm that the devices are now using the vfio-pci driver:
$ lspci -nnk ... 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM107 [GeForce GTX 750] [10de:1381] (rev a2) Subsystem: Gigabyte Technology Co., Ltd GM107 [GeForce GTX 750] [1458:362e] Kernel driver in use: vfio-pci Kernel modules: nvidiafb, nouveau, nvidia_384_drm, nvidia_384 01:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fbc] (rev a1) Subsystem: Gigabyte Technology Co., Ltd Device [1458:362e] Kernel driver in use: vfio-pci Kernel modules: snd_hda_intel ...
Install QEMU, KVM, and the OVMF UEFI BIOS:
$ sudo apt-get install qemu-kvm ovmf
You will need to run QEMU as root in order for it to use the vfio device. You will need to give root permission to use your display using a .Xauthority file. Here is a wrapper to do that. (Credit: ton)
$ vim ~/bin/xsudo #!/bin/sh touch ~/.Xauthority xauth generate :0 . trusted sudo $@ rm ~/.Xauthority
Copy the OVMF variables image to support UEFI variables:
$ cp /usr/share/OVMF/OVMF_VARS.fd ovmf_vars.fd
You will need to dedicate a keyboard and mouse to Windows. Use
lsusb
to find their vendor and product IDs:$ lsusb ... Bus 003 Device 004: ID 2516:0015 Cooler Master Co., Ltd. Keyboard Bus 003 Device 007: ID 046d:c083 Logitech, Inc. ...
Create a script for your VM.
Here is the script. We will unpack it next.
$ vim windows #!/bin/sh vfio-bind 0000:01:00.0 0000:01:00.1 qemu-system-x86_64 \ -enable-kvm \ -monitor stdio \ -name win10 \ \ -machine type=q35,accel=kvm \ -cpu host,kvm=off,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_vendor_id=NoFortyThree \ -smp 3 \ -m 8G \ -usb \ -vga none \ \ -device ich9-intel-hda \ -device hda-micro,audiodev=hda \ -audiodev pa,id=hda,server=unix:/run/user/$UID/pulse/native \ \ -drive if=pflash,format=raw,readonly,file=/usr/share/OVMF/OVMF_CODE.fd \ -drive if=pflash,format=raw,file=ovmf_vars.fd \ \ -device vfio-pci,host=01:00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1 \ \ -device usb-host,vendorid=0x2516,productid=0x0015 \ -device usb-host,vendorid=0x046d,productid=0xc083 \ \ -drive index=0,media=disk,file=$WINDOWS_IMG,format=raw \ $ chmod +x windows
We chose
-machine type=q35
to use a PCIe bus.We told the VM to hide the fact that we are using KVM with
kvm=off
. This is to avoid a "bug" in NVIDIA's driver for Windows.We enabled Hyper-V enlightenments with
hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time
.We set
hv_vendor_id=NoFortyThree
to avoid the NVIDIA driver throwing Error 43.We chose Intel HD Audio for audio devices. We will need to run the script as root. For root to use normal user 1001's pulseaudio server, we need to copy their pulseaudio config to root's:
$ sudo mkdir -p /root/.config $ sudo cp -a ~/.config/pulse /root/.config/ $ sudo chown -R root.root /root/.config/pulse
We are using drive interface
if=pflash
for the BIOS to support UEFI variables.GPU video and audio use the
vfio-pci
device.The dedicated keyboard and mouse are set with the
usb-host
device.
If you are reinstalling Windows:
Wipe all file system data:
$ sudo wipefs -a --force /dev/sdxX
Use qemu-img to write "MSEdge - Win10-disk001.vmdk" over the old drive:
$ sudo qemu-img dd -f vmdk -O raw bs=1M if="MSEdge - Win10-disk001.vmdk" of=/dev/sdxX
Skip to "Convert the VM to UEFI" below.
Download a Windows VM for VirtualBox from developer.microsoft.com.
If you can, use a 50GB native disk partition for Windows. (We will refer to it as
/dev/sdxX
.) Otherwise, use a "raw"-format file.If you are using a partition, write the Windows VM drive image to the partition:
$ unzip MSEdge.Win10.VirtualBox.zip $ tar -xvf "MSEdge - Win10.ova" $ sudo qemu-img dd -f vmdk -O raw bs=1M if="MSEdge - Win10-disk001.vmdk" of=/dev/sdxX
If you are using a file, convert the drive image to raw format:
$ unzip MSEdge.Win10.VirtualBox.zip $ tar -xvf "MSEdge - Win10.ova" $ sudo qemu-img convert -f vmdk -O raw "MSEdge - Win10-disk001.vmdk" windows.img
Update the VM script.
$ vim windows ... # /dev/sdxX WINDOWS_IMG=/dev/disk/by-partuuid/xxxxxxxx-XX ... -drive index=0,media=disk,file=$WINDOWS_IMG,format=raw \ ...
Microsoft assumes you will not be using a UEFI BIOS with the VM. But we need a UEFI BIOS for passthrough to avoid the issues with using Seabios. Since version 1703, Windows 10 includes the MBR2GPT command line tool to convert a drive from MBR to GPT and to create the EFI partition. The following is based on instructions at Windows Central
Boot Windows with SeaBIOS.
$ vim windows_seabios #!/bin/sh WINDOWS_IMG=/dev/sdxX qemu-system-x86_64 \ -enable-kvm \ -machine type=q35,accel=kvm \ -cpu host,kvm=off,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_vendor_id=NoFortyThree \ -m 4G \ -soundhw hda \ -usb \ -drive index=0,media=disk,file=$WINDOWS_IMG,format=raw \ -monitor stdio $ chmod +x windows_seabios $ xsudo ./windows_seabios
Log in with the password "Passw0rd!".
Open Settings. Search for "Advanced startup" and choose "Change advanced startup options". Click Restart now.
Choose Troubleshoot > Advanced options > Command prompt. Wait a long time.
When prompted, select "IEUser" and enter the password "Passw0rd!".
At the command prompt, type:
> mbr2gpt /validate > mbr2gpt /convert
Exit the command prompt and choose Turn off your PC.
Boot the VM using the
windows
VM script. If you are confronted with the UEFI shell, enter ...> exit
... to return to the BIOS. Choose Boot manager and select the hard drive Windows is installed on.
Use Chocolatey for software management. You can find instructions at Chocolatey Docs.
Install the NVIDIA driver. Don't time out if it takes ages to download.
> choco install -y --execution-timeout=0 nvidia-display-driver
Install Steam.
> choco install -y steam
Assign more processors to Windows.
nproc
will tell you how many processors are available. e.g.$ nproc 8 $ vim windows ... -smp 4 \ ...
Use virtio SCSI drivers. Previous versions of Windows allowed you to use the Recovery Drive to set the boot drive to use virtio drivers, but Windows 10 will throw "stop code: inaccessible boot device", and requires a longer process. Credit: harrymc and llegolas.
Download Windows VirtIO drivers from RedHat:
$ wget https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/stable-virtio/virtio-win.iso
Set a second drive as a virtio SCSI drive. Add the virtio driver CD.
$ vim windows ... # /dev/sdd3 DATA_IMG=/dev/disk/by-partuuid/c6a2a56c-2e66-4642-9785-c18cae539c0f ... -device virtio-scsi-pci,id=scsi0 \ -drive id=disk1,file=$DATA_IMG,format=raw,cache=writeback,if=none \ -device scsi-hd,drive=disk1 \ -drive index=2,media=cdrom,file=virtio-win.iso \ ...
Boot Windows. Open Device Manager. Right-click on "PCI SCSI Controller" and select "Update driver". Load the driver from D:vioscsiw10amd64.
Run Command Prompt as administrator and set Windows to boot into safe mode:
> bcdedit /set {current} safeboot minimal
Shut down Windows and change the boot device type to virtio:
$ vim windows ... -device virtio-scsi-pci,id=scsi0 \ -drive id=disk0,file=$WINDOWS_IMG,format=raw,cache=none,if=none \ -device scsi-hd,drive=disk0 \ -drive id=disk1,file=$DATA_IMG,format=raw,cache=writeback,if=none \ -device scsi-hd,drive=disk1 \ ...
Start the VM. Windows will enter in safe mode.
Note
In Safe mode all boot-start drivers will be enabled and loaded, including the virtio driver. Since there is now a drive installed to use it, the kernel will now make it part of the drivers that are to be loaded on boot and not disable it again.
Run Command Prompt as administrator again, and set Windows no longer to boot into safe mode with:
> bcdedit /deletevalue {current} safeboot
Reboot.
Further reading:
-
> The example below shows the use of the virtio-scsi-pci driver. In this case I defined an ioh3420 root port driver (PCIe) where I attached the SCSI devices:
-device pcie-root-port,bus=pcie.0,addr=1c.0,multifunction=on, port=1,chassis=1,id=root.1 \ -object iothread,id=io1 \ -device virtio-scsi-pci,id=scsi0,iothread=io1,num_queues=4,bus=pcie.0 \ -drive id=scsi0,file=/dev/sdb1,if=none,format=raw,aio=threads,cache=none \ -device scsi-hd,drive=scsi0 \
-
Use virtio network drivers. Note that if VM's usage is high, this seems to be able to cripple network speed for the host.
Load the vhost-net kernel module, and set the Ethernet controller to virtio
$ vim windows ... modprobe vhost-net ... -netdev user,id=net0 \ -device virtio-net,netdev=net0 \ ...
Boot Windows. Open Device Manager. Right-click on "Ethernet Controller" and choose "Update driver". Load the driver from D:NetKVMw10amd64.
Disable Hibernate: Run "cmd" as administrator, and
> powercfg -h off
Disable Suspend: Start > Settings > System > Power & sleep > Sleep: "Never"
Disable Cortana:
- Run "gpedit.msc"
- Go to Computer Configuration > Administrative Templates > Windows Components > Search
- Find "Allow Cortana", and set it to "Disabled". Click "OK".
- Reboot, or log out and log in.
You can tweak the audio quality with
QEMU_PA_SAMPLES
andQEMU_AUDIO_TIMER_PERIOD
. You can find the available variables and their defaults with:$ qemu-system-x86_64 -audio-help
Use NAT networking with a TAP interface.
Further reading:
If you are going to assign more than 4GB of RAM to Windows, hugepage support will improve performance. hugepages is installed in Ubuntu by default. The following is based on Ubuntu Community Help Wiki and ArchWiki.
Confirm that hugepage size is 2048 KB:
$ cat /proc/meminfo | grep Hugepagesize Hugepagesize: 2048 kB
If we want to assign 6 GB to Windows, that will be 6 × 1024 × 1024 ÷ 2048 = 6 × 1024 ÷ 2 = 3072 hugepages. Round up to 3100. If we want to assign 12 GB to Windows, that will be 12 × 1024 ÷ 2 = 6144 hugepages. Round up to 6200. Add the following to the script to reserve 3100 (for example) hugepages:
$ vim windows ... sysctl vm.nr_hugepages=3100 ... -m 6G \ -mem-path /dev/hugepages \ ...
You can check, while the VM is running, how many pages are used:
$ cat /proc/meminfo | grep HugePages
CPU pinning:
Further reading:
If you want to use the NVIDIA in Linux again, use the following steps to take the graphics card back:
Comment out "pci_stub" in /etc/initramfs-tools/modules
Move /lib/modprobe.d/pci-stub.conf to ~/doc/
$ sudo mv /lib/modprobe.d/pci-stub.conf ~/doc/
Update the initramfs image:
$ sudo update-initramfs -u
Reboot.
If necessary, check prime graphics device and monitor configuration:
$ xsudo nvidia-settings $ rm ~/.config/monitors.xml
Reboot, log in, and configure monitors.