pci-e passthrough with QEMU on intel / nvidia platform · Neg blog

Neg blog

~/dev/src/github

© 2017. All rights reserved.

pci-e passthrough with QEMU on intel / nvidia platform

11 Jun 2017

I will tell you some “success story” about pci-e passthrough Geforce 980 GTX videocard with QEMU / KVM to Windows 10 guest system for gaming. As the result you’ll get system almost with no overhead(1-10%). If you are interested in this welcome under cut.

The config of my desktop:

Also we need monitor which support simultaneous working with several outputs as hdmi/dvi or whatever, for example my asus pa249q supports that. Also you may simply use a couple of monitors.

Soft which we need:

Also you can use manual from here https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF

As you can see I’ve used intel gpu for host and nvidia 980 gpu for guests.

You need UEFI compatible system with support of VT-d. Also VT-d should be enabled in UEFI-setup.

For example the list of Intel compatible CPUs is here: http://ark.intel.com/search/advanced?s=t&VTX=true&VTD=true

Then you can use following script to get IOMMU groups:

```
#!/bin/bash
shopt -s nullglob
for d in /sys/kernel/iommu_groups/*/devices/*; do 
    n=${d#*/iommu_groups/*}; n=${n%%/*}
    printf 'IOMMU Group %s ' "$n"
    lspci -nns "${d##*/}"
done;

For my system I’ve the following output:

[~/bin/qemu] >> s ~/bin/qemu/iommu/group_valid
IOMMU Group 0 00:00.0 Host bridge [0600]: Intel Corporation 4th Gen Core Processor DRAM Controller [8086:0c00] (rev 06)
IOMMU Group 10 00:1c.3 PCI bridge [0604]: Intel Corporation 9 Series Chipset Family PCI Express Root Port 4 [8086:8c96] (rev d0)
IOMMU Group 11 00:1c.4 PCI bridge [0604]: Intel Corporation 9 Series Chipset Family PCI Express Root Port 5 [8086:8c98] (rev d0)
IOMMU Group 12 00:1c.7 PCI bridge [0604]: Intel Corporation 9 Series Chipset Family PCI Express Root Port 8 [8086:8c9e] (rev d0)
IOMMU Group 13 00:1d.0 USB controller [0c03]: Intel Corporation 9 Series Chipset Family USB EHCI Controller #1 [8086:8ca6]
IOMMU Group 14 00:1f.0 ISA bridge [0601]: Intel Corporation 9 Series Chipset Family Z97 LPC Controller [8086:8cc4]
IOMMU Group 14 00:1f.2 SATA controller [0106]: Intel Corporation 9 Series Chipset Family SATA Controller [AHCI Mode] [8086:8c82]
IOMMU Group 14 00:1f.3 SMBus [0c05]: Intel Corporation 9 Series Chipset Family SMBus Controller [8086:8ca2]
IOMMU Group 15 03:00.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1184]
IOMMU Group 16 04:01.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1184]
IOMMU Group 17 04:03.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1184]
IOMMU Group 17 06:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 11)
IOMMU Group 18 04:05.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1184]
IOMMU Group 19 04:07.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1184]
IOMMU Group 19 08:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 02)
IOMMU Group 1 00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller [8086:0c01] (rev 06)
IOMMU Group 1 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 980] [10de:13c0] (rev a1)
IOMMU Group 1 01:00.1 Audio device [0403]: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)
IOMMU Group 20 09:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller [1b21:1142]
IOMMU Group 21 0a:00.0 Non-Volatile memory controller [0108]: OCZ Technology Group, Inc. Device [1b85:6018] (rev 01)
IOMMU Group 2 00:02.0 VGA compatible controller [0300]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller [8086:0412] (rev 06)
IOMMU Group 3 00:03.0 Audio device [0403]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller [8086:0c0c] (rev 06)
IOMMU Group 4 00:14.0 USB controller [0c03]: Intel Corporation 9 Series Chipset Family USB xHCI Controller [8086:8cb1]
IOMMU Group 5 00:16.0 Communication controller [0780]: Intel Corporation 9 Series Chipset Family ME Interface #1 [8086:8cba]
IOMMU Group 6 00:19.0 Ethernet controller [0200]: Intel Corporation Ethernet Connection (2) I218-V [8086:15a1]
IOMMU Group 7 00:1a.0 USB controller [0c03]: Intel Corporation 9 Series Chipset Family USB EHCI Controller #2 [8086:8cad]
IOMMU Group 8 00:1b.0 Audio device [0403]: Intel Corporation 9 Series Chipset Family HD Audio Controller [8086:8ca0]
IOMMU Group 9 00:1c.0 PCI bridge [0604]: Intel Corporation 9 Series Chipset Family PCI Express Root Port 1 [8086:8c90] (rev d0)

The minimal unit which can be passed by qemu is IOMMU group, if you have several devices on the same group you may try to use acs-patch, but use it carefully. I don’t recomment to apply this hack to non-videocard devices. And also you will probably need patched linux kernel: linux-vfio(AUR) You can read better explaination of this on vfio blog:

https://vfio.blogspot.ru/2014/08/iommu-groups-inside-and-out.html

But I’ve not try it, use it for your own risk.

In my case you need to take

IOMMU Group 1 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 980] [10de:13c0] (rev a1)
IOMMU Group 1 01:00.1 Audio device [0403]: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)

But leave pci-e controller to the host system. To get it add this to your /etc/modprobe.d/vfio.conf:

options vfio-pci ids=10de:13c0,10de:0fbb

And that’s it. Now you bypassed your GPU to the vfio driver.

Now you need to reboot. Then check it with

>> for i in 10de:13c0 10de:0fbb; do sudo lspci -nnk -d "${i}"; done
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 980] [10de:13c0] (rev a1)
        Subsystem: NVIDIA Corporation Device [10de:1116]
        Kernel driver in use: vfio-pci
        Kernel modules: nouveau, nvidia_drm, nvidia
01:00.1 Audio device [0403]: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)
        Subsystem: NVIDIA Corporation Device [10de:1116]
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel

If you get Kernel driver in use: vfio-pci then everything is ok any you can to go further.

I’ve installed Windows 10 for that.

Here is my main qemu parameters for that:

-enable-kvm

We are need to use KVM instead of emulation of course

-m 8192

I’ve bypassed half of my host memory to vm.

-smp sockets=1,cores=4,threads=2

And also copy all my CPU topology here

-cpu host,kvm=off

We are using host cpu with kvm=off to trick out nvidia which prevents to be used on the latest versions of nvidia driver.

-device virtio-scsi-pci,id=scsi

We need it for faster hdd/ssd io-performance.

and then use options like this:

    -drive file="${iso_path}/${win_iso},index=0,media=cdrom"
    -drive file="${iso_path}/virtio-win-0.1.137.iso,index=1,media=cdrom"
    -boot menu=on

Also we need ovmf binary and vars, you may alsa copy it from ovmf_code_x64.bin to your vm directory as me. To find location of it you may use pacaur -Ql ovmf-git or mlocate stuff.

-drive if=pflash,format=raw,readonly,file=/one/vm/ovmf_code_x64.bin
-drive if=pflash,format=raw,file=/one/vm/ovmf_vars_x64.bin
-drive file="/one/vm/${disk_name}.qcow2",id=disk,format=qcow2,if=none,cache=writeback -device scsi-hd,drive=disk

And here we added hdd drive itself. You can create it with s qemu-img create -f qcow2 w7.qcow2 320G

Now you can to install Windows 10.

When you’ve installed that you may bypass your GPU with

    -vga none
    -device vfio-pci,host=01:00.0,multifunction=on
    -device vfio-pci,host=01:00.1

respecting ids from virtio driver which we added later using IOMMU Groups info.

Also you will get some problem with keyboard and mouse, because of vm cannot to see it. You may to add something like this. But by carefull, because of while vm is running you will loose your keyboard/mouse on the host.

    -usb
    -usbdevice host:1e7d:2e22 # Roccat kone rtd [ Mouse ]
    -usbdevice host:046d:c32b # Logitech G910 [Keyboard]

To prevent this you may use synergy.

The one of the biggest troubles here is sound quality. To prevent this I recommend you -soundhw ac97. Also you need to enable test-singning free mode. To enable test-signing use Bcdedit.exe -set TESTSIGNING ON. Then install ac97 driver for windows 7 x64 as usual. Also I’ve get best result with pulseaudio and following options:

QEMU_AUDIO_TIMER_PERIOD=150
QEMU_PA_SAMPLES=1024
QEMU_PA_SERVER=localhost
PULSE_SERVER=localhost
QEMU_AUDIO_DRV=pa

Also here is my client.conf for Pulseaudio:

autospawn = no
enable-shm = yes

Here is daemon.conf:

daemonize = no
fail = yes
allow-module-loading = yes
allow-exit = no
use-pid-file = yes
system-instance = no
local-server-type = user
enable-shm = yes
shm-size-bytes = 0 # setting this 0 will use the system-default, usually 64 MiB
lock-memory = no
cpu-limit = no
high-priority = yes
nice-level = -11
realtime-scheduling = yes
realtime-priority = 5
exit-idle-time=-1
exit-idle-time = 20
scache-idle-time = 20
load-default-script-file = yes
default-script-file = /etc/pulse/default.pa
resample-method = auto
flat-volumes = no
rlimit-fsize = -1
rlimit-data = -1
rlimit-stack = -1
rlimit-core = -1
rlimit-as = -1
rlimit-rss = -1
rlimit-nproc = -1
rlimit-nofile = 256
rlimit-memlock = -1
rlimit-locks = -1
rlimit-sigpending = -1
rlimit-msgqueue = -1
rlimit-nice = 31
rlimit-rtprio = 9
rlimit-rttime = 200000
default-sample-format = float32ne
default-sample-rate = 192000
alternate-sample-rate = 192000
default-fragments = 2
default-fragment-size-msec = 114

And here is default.pa:

.nofail
.fail
load-module module-device-restore
load-module module-stream-restore
load-module module-card-restore
load-module module-augment-properties
load-module module-switch-on-port-available
.ifexists module-udev-detect.so
load-module module-udev-detect
.else
load-module module-detect use_ucm=0 tshed=0
.endif
.ifexists module-jackdbus-detect.so
.nofail
load-module module-jackdbus-detect channels=2
.fail
.endif
.ifexists module-bluetooth-policy.so
load-module module-bluetooth-policy
.endif
.ifexists module-bluetooth-discover.so
load-module module-bluetooth-discover
.endif
.ifexists module-esound-protocol-unix.so
load-module module-esound-protocol-unix
.endif
load-module module-native-protocol-unix
load-module module-native-protocol-tcp auth-ip-acl=127.0.0.1
.ifexists module-gconf.so
.nofail
load-module module-gconf
.fail
.endif
load-module module-default-device-restore
load-module module-rescue-streams
load-module module-always-sink
load-module module-intended-roles
load-module module-suspend-on-idle
.ifexists module-console-kit.so
load-module module-console-kit
.endif
.ifexists module-systemd-login.so
load-module module-systemd-login
.endif
load-module module-position-event-sounds
load-module module-filter-heuristics
load-module module-filter-apply

Also you need to calculate opts

default-fragments = [???]
default-fragment-size-msec = [???]

for yourself, here is the method:

[/etc/pulse] >> pactl list sinks |& grep -i buff
device.buffering.buffer_size = "1048572"
device.buffering.fragment_size = "524286"
[/etc//pulse] >> echo $[1048572./(192000*24)]
0.22755468749999999
[/etc//pulse] >> echo $[524286./(192000*24)]
0.11377734375

Where 192000@24 bits are settings for my pulseadio and soundcard.

Here we have 227.55 msesc for device.buffering.buffer_size and 113.7773 msecs for device.buffering.fragment_size.

To compute default fragments count we are using:

echo $[227.55/113.7773]
1.9999595701427264

And default-fragment-size-msec is device.buffering.fragment_size

So we got:

default-fragments = 2
default-fragment-size-msec = 114

After all this manipulations all sound should be clear. Also Winwdows 7 haven’t this problem. Maybe this is because of new high-performance sound engine inside Windows 8.1/10.

Also here is my full script to run QEMU:

#!/bin/zsh

# Script to run windows 10 with geforce 980gtx gpu ;)

local disk_name="w10"
local iso_path="/home/neg/1st_level/iso"
local win10_iso="en_windows_10_education_version_1607_updated_jul_2016_x64_dvd_9055880.iso"
local win_iso="${win10_iso}"

local -a qemu_params=(
    -enable-kvm
    -m 8192
    -smp sockets=1,cores=4,threads=2
    -cpu host,kvm=off
    -device virtio-scsi-pci,id=scsi
    -drive if=pflash,format=raw,readonly,file=/one/vm/ovmf_code_x64.bin
    -drive if=pflash,format=raw,file=/one/vm/ovmf_vars_x64.bin
    -drive file="/one/vm/${disk_name}.qcow2",id=disk,format=qcow2,if=none,cache=writeback -device scsi-hd,drive=disk
)

local -a qemu_cpu_norm_params=(
    -cpu host,kvm=off,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff
)

local -a qemu_cpu_slow_params=(
    -cpu host,kvm=off,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv-vendor-id=servo,hv-vpindex,hv-reset,hv-runtime,hv-crash
)

local -a qemu_rtc_params=(
    -rtc base=utc,driftfix=slew -no-hpet -global kvm-pit.lost_tick_policy=discard
)

local -a qemu_source_params=(
    -drive file="${iso_path}/${win_iso},index=0,media=cdrom"
    -drive file="${iso_path}/virtio-win-0.1.137.iso,index=1,media=cdrom"
    -boot menu=on
)

local -a qemu_pass_video=(
    -vga none
    -device vfio-pci,host=01:00.0,multifunction=on
    -device vfio-pci,host=01:00.1
)

local -a qemu_pass_input=(
    -usb
    -usbdevice host:1e7d:2e22 # Roccat kone rtd [ Mouse ]
    -usbdevice host:046d:c32b # Logitech G910 [Keyboard]
)

local -a qemu_usb_host=(
  -device ich9-usb-uhci3,id=uhci
  -device usb-ehci,id=ehci
  -device nec-usb-xhci,id=xhci
)

local -a qemu_pass_emu=(
    # --
)

local -a qemu_emulated_sound=(
    -soundhw ac97
)

local -a qemu_share=(
    -net nic -net user,smb=/mnt/qemu 
)

local -a qemu_separated_console=(
    -qmp unix:/home/neg/1st_level/qmp.socket,server --monitor stdio \
    /home/neg/bin/scripts/qmp/qmp-shell /home/neg/1st_level/qmp.socket
)

local -a qemu_network=(
    -net nic -net tap,ifname=tap0,script=/home/neg/bin/qemu/qemu-ifup,downscript=/home/neg/bin/qemu/qemu-ifdown
)

if [[ ! $(pidof synergyc) ]]; then
    sudo -u neg "synergy" & print "Autostart synergy"
fi

QEMU_AUDIO_TIMER_PERIOD=150 \
QEMU_PA_SAMPLES=1024 \
QEMU_PA_SERVER=localhost \
PULSE_SERVER=localhost \
QEMU_AUDIO_DRV=pa \
qemu-system-x86_64 "${qemu_params[@]}" \
    "${qemu_emulated_sound[@]}" \
    "${qemu_source_params[@]}" \
    "${qemu_pass_video[@]}" \
    "${qemu_pass_input[@]}"  \
    "${qemu_network[@]}"

Related Posts

comments powered by Disqus