Skip to content

Latest commit

 

History

History
405 lines (369 loc) · 19.8 KB

README.md

File metadata and controls

405 lines (369 loc) · 19.8 KB
Raw

Yet Another VFIO Guide

This guide describes how to create my personal VFIO setup on Ubuntu 19.04.

Table of Contents

Hardware

  • CPU: Ryzen 7 1700
  • Motherboard: ASUS ROG STRIX X370-F GAMING
  • RAM: 16 GB
  • Host GPU: RX 570
  • Guest GPU: GTX 980 Ti

System Configuration

Set Kernel Parameters

Edit /etc/default/grub

Before:
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"
After:
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash amd_iommu=on iommpu=pt video=efifb:off vfio-pci.ids=10de:17c8,10de:0fb0 isolcpus=8-15 nohz_full=8-15 rcu_nocbs=8-15"
  • amd_iommu=on: Enable IOMMU
  • iommu=pt: Improves performance. See this for a (very) detailed description.
  • video=efifb:off: No display output from guest GPU without this. See this post on The Passthrough POST
  • vfio-pci.ids=10de:17c8,10de:0fb0: Use vfio-pci driver for guest GPU
  • isolcpus=8-15: Isolates and pins CPU cores so that they are exclusively used by the guest OS. Caution: Disables host OS to use these CPU cores
  • nohz_full=8-15 and rcu_nocbs=8-15: Removes microstutters in the guest OS. I don't know exactly why this works. See this old Reddit post

Blacklist GPU to prevent driver loading

Create /etc/modprobe.d/vfio.conf and add these parameters:

  options vfio-pci ids=10de:17c8,10de:0fb0 
  options kvm ignore_msrs=1
  • options vfio-pci ids=10de:17c8,10de:0fb0: Further blacklisting of guest GPU
  • options kvm ignore_msrs=1: Windows will not boot without this. See this post on the Level1Techs forum

Load VFIO kernel modules

Edit /etc/initramfs-tools/modules and append these parameters:

vfio_pci
vfio
vfio_iommu_type1
vfio_virqfd

Update GRUB and Initramfs

Next, update GRUB and Initramfs to feature all the changes made above by running update grub && update-initramfs -u

Set up QEMU

Install virtualization packages

Run apt install qemu qemu-kvm libvirt0 ovmf virt-manager to install packages needed for virtualization.

Set path of OVMF firmware

Edit etc/libvirt/qemu.conf and find the commented line that starts with nvram. Edit these lines to reflect the path of your OVMF files. Always restart libvirtd after editing by executing systemctl restart libvirtd.service

Before
#nvram = [
#   "/usr/share/OVMF/OVMF_CODE.fd:/usr/share/OVMF/OVMF_VARS.fd",
#   "/usr/share/OVMF/OVMF_CODE.secboot.fd:/usr/share/OVMF/OVMF_VARS.fd",
#   "/usr/share/AAVMF/AAVMF_CODE.fd:/usr/share/AAVMF/AAVMF_VARS.fd",
#   "/usr/share/AAVMF/AAVMF32_CODE.fd:/usr/share/AAVMF/AAVMF32_VARS.fd",
#   "/usr/share/OVMF/OVMF_CODE.ms.fd:/usr/share/OVMF/OVMF_VARS.ms.fd"
#]
After
nvram = [
   "/usr/share/OVMF/OVMF_CODE.fd:/usr/share/OVMF/OVMF_VARS.fd",
]
#   "/usr/share/OVMF/OVMF_CODE.secboot.fd:/usr/share/OVMF/OVMF_VARS.fd",
#   "/usr/share/AAVMF/AAVMF_CODE.fd:/usr/share/AAVMF/AAVMF_VARS.fd",
#   "/usr/share/AAVMF/AAVMF32_CODE.fd:/usr/share/AAVMF/AAVMF32_VARS.fd",
#   "/usr/share/OVMF/OVMF_CODE.ms.fd:/usr/share/OVMF/OVMF_VARS.ms.fd"

Optional: Change user for QEMU

This is only needed if evdev passthrough will be used. As I understand it is best practice to execute QEMU as a non-login user created for this purpose. One may also change this value to the own user or root if so preferred.

First, create a new user by executing useradd -s /usr/sbin/nologin -r -M -d /dev/null vfio. Then, edit etc/libvirt/qemu.conf once again. The relevant line is around 440.

Before:
#user = root
After:
user = vfio

Create and configure the VM

This can be split into two parts. In the first part a virtual machine will be created with virt-manager. The second part configures the VM to improve performance and (most importantly) work around the well-known Error 43.

Create VM in virt-manager

Initial VM creation

Open virt-manager and create a new virtual machine. A window should open.

  • Step 1: Choose local install media
  • Step 2: Select you install media. You might have to create a storage pool first. In this case choose filesystem directory.
  • Step 3: Select 8144 and 8 for memory and CPU respectively.
  • Step 4: Depending on your setup you can choose to
    • use the default disk image
    • create a disk image in a place of your choosing
    • specify a whole disk as storage location, e.g. /dev/sdb3
  • Step 5: Important: Tick Customize configuration before install

Basic configuration

A couple of steps need to be done in the new configuration overview window.

  • Choose the correct firmware in the Overview tab. Switch from BIOS to UEFI. The UEFI entry should feature the previously specified nvram path.
  • Change the CPU configuration in the CPUs tab.
    • Deselect Copy host CPU configuration and set the model to host-passthrough.
    • Select Manually set CPU topology. In this case 1 socket, 4 cores and 2 threads (per core).
  • Change the Disk Bus to VirtIO in the SATA Disk 1 tab
  • Optional: Remove the Tablet, Display Spice and Video QXL devices. We're doing GPU-passthrough after all.
  • Really Optional: Remove the Sound ich9 device. I just can't get guest to host audio passthrough to work. Your mileage may vary.
  • Add Hardware: Select all devices that you want to use inside your VM.
    • Select both Guest GPU devices under PCI Host device
    • Add a CDROM containing the VirtIO drivers that can be downloaded from here
    • Optional: Select a USB controller or HD audio device under PCI Host device
    • Optional: Select single USB controller
    • Optional: Select additional storage and/or further devices. Just remember that you need some kind of input device to install windows if you aren't using evdev passthrough.

Hit Begin Installation once you configured the VM to your liking. Kill the VM before installing windows since your new VM is not fully configured.

Advanced configuration with virsh

The following steps are performed inside the VM's XML file. Edit by executing virsh edit VM_NAME.

XML Namespace

Declare the XML namespace in the first line

Before:

<domain type='kvm'>

After:

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>

CPU configuration

Change the CPU configuration to reflect the isolated cpu cores and number of cores of your VM.

Before:
<vcpu placement='static' current='1'>8</vcpu>
After:
<vcpu placement='static'>8</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='8'/>
    <vcpupin vcpu='1' cpuset='9'/>
    <vcpupin vcpu='2' cpuset='10'/>
    <vcpupin vcpu='3' cpuset='11'/>
    <vcpupin vcpu='4' cpuset='12'/>
    <vcpupin vcpu='5' cpuset='13'/>
    <vcpupin vcpu='6' cpuset='14'/>
    <vcpupin vcpu='7' cpuset='15'/>
    <emulatorpin cpuset='0-1'/>
  </cputune>

Fix Error 43

Add a fake vendor_id and hide KVM in the features section.

Before:
<features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
    </hyperv>
    <vmport state='off'/>
  </features>
After:
<features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='1234567890ab'/>
    </hyperv>
    <kvm>
      <hidden state='on'/>
    </kvm>
  </features>

Installing Windows

Now you are finally ready to start the VM and install Windows. If you get stuck in the BIOS/UEFI, type exit and select the Windows CD in the Boot Manager.

Windows won't recognize the VirtIO disk until install the drivers. Load drivers from the CD's viosci/win10/amd64 and vstor/win10/amd64 folder.

Optional: Evdev Passthrough

Evdev passthrough works like a virtual USB/KVM switch. Clicking both CTRLs on your keyboard switches your input devices between guest and host. The Passthrough POST has a really nice tutorial explaining the matter much better.

I did not end up using it (even though I would have really liked to). Sometimes when switching from guest to host audio in the Windows guest hangs until i disable and enable the audio device (HDMI audio from 980 Ti). Instead I use Synergy to switch mouse and keyboard between guest and host.

Finding the correct input devices

Your mouse and keyboard actions (in Linux fashion) are represented as a file under /dev/input. Try to find the event files that correspondent to the actual key events by first listing the content of /dev/input/by-id.

Example:
$ ls -l /dev/input/by-id

total 0
lrwxrwxrwx 1 root root  9 Jul 14 18:28 usb-Logitech_Gaming_Mouse_G502_0C65344F3231-event-if01 -> ../event4
lrwxrwxrwx 1 root root  9 Jul 14 18:28 usb-Logitech_Gaming_Mouse_G502_0C65344F3231-event-mouse -> ../event2
lrwxrwxrwx 1 root root  9 Jul 14 18:28 usb-Logitech_Gaming_Mouse_G502_0C65344F3231-if01-event-kbd -> ../event3
lrwxrwxrwx 1 root root  9 Jul 14 18:28 usb-Logitech_Gaming_Mouse_G502_0C65344F3231-mouse -> ../mouse0
lrwxrwxrwx 1 root root 10 Jul 14 18:28 usb-Logitech_Logitech_G710_Keyboard-event-if01 -> ../event12
lrwxrwxrwx 1 root root 10 Jul 14 18:28 usb-Logitech_Logitech_G710_Keyboard-event-kbd -> ../event10
lrwxrwxrwx 1 root root 10 Jul 14 18:28 usb-Logitech_Logitech_G710_Keyboard-if01-event-kbd -> ../event11

Then, cat devices that sound plausible. In my case all of the above... If the output is random gibberish you have found one of you input devices. Repeat until all devices have been found. Note the event file in /dev/input that the file symlinks to. You will need that path (e.g. /dev/input/event10/) for later configuration

Example:
$ cat /dev/input/by-id/usb-Logitech_Logitech_G710_Keyboard-event-kbd

+]�`d-k+]Pz-k+]Pz -k+]Pz.k+]��.k+]��.k+]��2/k+]�# /k+]�#/k+]�#3/k+]�/k+]�/k+]�/k+]�!/k+]�/k+]�4/k+]�� /k+]��/k+]��/k+]%�!/k+]%�/k+]%�7k+]I<

Add devices to your VM

Add all devices found the previous step to your VM by editing its XML near the bottom of the file via virsh edit VM_NAME. Add grab_all=on,repeat=on to your keyboard.

Before:
  </devices>
</domain>
After:
  </devices>
  <qemu:commandline>
    <qemu:arg value='-object'/>
    <qemu:arg value='input-linux,id=mouse1,evdev=/dev/input/event2'/>
    <qemu:arg value='-object'/>
    <qemu:arg value='input-linux,id=mouse2,evdev=/dev/input/event3'/>
    <qemu:arg value='-object'/>
    <qemu:arg value='input-linux,id=kbd1,evdev=/dev/input/event10,grab_all=on,repeat=on'/>
    <qemu:arg value='-object'/>
    <qemu:arg value='input-linux,id=kbd2,evdev=/dev/input/event12'/>
  </qemu:commandline>
</domain>

For better performance you can add mouse and keyboard as VirtIO devices. This requires you to install the vioinput drivers from the virtio-win driver package used during Windows installation.

Before:
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
After:
    <input type='mouse' bus='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0e' function='0x0'/>
    </input>
    <input type='keyboard' bus='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0f' function='0x0'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>

Add files to QEMU

Add the same devices/files to the cgroup_devices_acl to your QEMU config in etc/libvirt/qemu.conf/ The relevant section should be around line 480.

Before:
# cgroup_device_acl = [
#    "/dev/null", "/dev/full", "/dev/zero",
#    "/dev/random", "/dev/urandom",
#    "/dev/ptmx", "/dev/kvm", "/dev/kqemu",
#    "/dev/rtc","/dev/hpet", "/dev/sev"
#
# ]
After:
cgroup_device_acl = [
    "/dev/null", "/dev/full", "/dev/zero",
    "/dev/random", "/dev/urandom",
    "/dev/ptmx", "/dev/kvm", "/dev/kqemu",
    "/dev/rtc","/dev/hpet", "/dev/sev",
    "/dev/input/event2",
    "/dev/input/event3",
    "/dev/input/event10",
    "/dev/input/event12",
]

Restart libvirtd with systemctl restart libvirtd.service.

Add exceptions to AppArmor

QEMU won't be able to access these devices if they aren't whitelisted. Therefore you have to add them as exception in /etc/apparmor.d/abstractions/libvirt-qemu. Add the following lines and restart AppArmor with systemctl restart apparmor.service:

  /dev/input/event2 rw,
  /dev/input/event3 rw,
  /dev/input/event10 rw,
  /dev/input/event12 rw,

Finally, add the previously created user vfio to the input group: usermod -a -G input vfio

Optional: Execute Scripts on VM Startup and Shutdown

It is possible to hook scripts to the startup and shutdown of your virtual machine. For example, I want to disconnect my second screen from gnome and start synergy.

Setup

Execute the following:

$ sudo mkdir -p /etc/libvirt/hooks
$ sudo wget 'https://raw.githubusercontent.com/PassthroughPOST/VFIO-Tools/master/libvirt_hooks/qemu' -O /etc/libvirt/hooks/qemu 
$ sudo chmod +x /etc/libvirt/hooks/qemu
$ sudo systemctl restart libvirtd.service

Exemplary Hooks

For more details look at the tutorial linked above. Examples:

/etc/libvirt/hooks/qemu.d/win10/started/begin/startup.sh

#!/bin/bash
su - bastian -c "DISPLAY=:0.0 xrandr --output HDMI-A-1 --off"
systemctl start [email protected]

/etc/libvirt/hooks/qemu.d/win10/stopped/end/shutdown.sh

#!/bin/bash
su - bastian -c "DISPLAY=:0.0 xrandr --output DisplayPort-1 --auto --output HDMI-A-1 --auto --right-of DisplayPort-1"
systemctl stop [email protected]

Optional: Add Synergy as Systemd Service

To create a systemd service for Synergy server, create /lib/systemd/system/[email protected] with the following content:

[Unit]
Description=Synergy for sharing mouse and keyboard
After=network.target

[Service]
ExecStart=/usr/bin/synergys -f -n H440-Linux -c /etc/synergy.conf -a 192.168.122.1:24800 -l /var/log/synergy.log
User=%i

[Install]
WantedBy=multi-user.target

Optional: Configure X and Gnome

Avoid Login Loop

I have to reconfigure X after starting the VM for the first time. Otherwise I am stuck in a login loop after rebooting Ubuntu. I guess that X wants to talk to the blacklisted guest GPU which crashes it. To avoid this, execute this:

$ sudo X -configure
$ sudo cp xorg.conf.new /etc/X11/xorg.conf

Add HiDPI support to xrandr output switching

By default, turning a display on via xrandr sets Gnome's HiDPI scale to 100%. Since I prefer 200% scaling I had to run the following commands as fix:

$ gsettings set org.gnome.settings-daemon.plugins.xsettings overrides "[{'Gdk/WindowScalingFactor', <2>}]"
$ gsettings set org.gnome.desktop.interface scaling-factor 2

Credits/Sources