nixos/qemu-vm: add opengl hardware acceleration option

[kira-bruneau]

Description of changes

VMs built with hardware.opengl.enable = true will now use opengl hardware acceleration backed by VirGL.

This makes it possible to run most Wayland compositors in a VM, and enables hardware acceleration for X11 desktop environments.

Hardware acceleration can also be enabled when running a test VM interactively with:

{
  interactive.nodes.machine = {
    virtualisation.opengl = true;
  };
}

nix run .#nixosTests.gnome.driverInteractive
>>> start_all()

Compositors like gamescope which rely on Vulkan still aren't supported out of the box. Although, it looks like we could eventually support this if we enable & use rutabaga when building qemu.

NOTE: I haven't tested this on aarch64-linux & darwin, but I assume it should work based on the similar usage instructions I found at https://github.com/knazarov/homebrew-qemu-virgl#usage---m1-macs.

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

---

Add a 👍 reaction to pull requests you find important.

[RaitoBezarius]

Are the mkMerge necessary for the merging or could you get away with a simple lib.optional and ++ ? Other than that, it looks mostly good to me. Is there a way to easily verify it's doing what it's supposed to do on a x86 machine? Just running some FPS stuff?

[kira-bruneau]

Are the mkMerge necessary for the merging or could you get away with a simple lib.optional and ++ ?

Oh I would normally would use lib.optional(s) but I was just following along with what was already defined in the module for consistency.

I was just starting to replace all uses of mkMerge with lib.optional(s), but then looked into it more and based on https://discourse.nixos.org/t/mkif-vs-if-then/28521/2, I think it would be better to stick with mkMerge.

Is there a way to easily verify it's doing what it's supposed to do on a x86 machine? Just running some FPS stuff?

I was verifying the change through my nixos config with --override-input, but here's a more minimal flake to try out:

{
  inputs.nixpkgs.url = "github:kira-bruneau/nixpkgs/qemu-vm-opengl";

  outputs = { self, nixpkgs }: {
    packages.x86_64-linux.default = (nixpkgs.lib.nixosSystem {
      pkgs = nixpkgs.legacyPackages.x86_64-linux;
      modules = [
        ({ pkgs, ... }: {
          system.stateVersion = "23.11";
          programs.sway.enable = true;
          services.getty.autologinUser = "root";
          programs.bash.loginShellInit = ''
            if [ "$(tty)" = "/dev/tty1" ]; then
              sway --config ${pkgs.writeText "glxgears" ''
                exec ${pkgs.mangohud}/bin/mangohud ${pkgs.mesa-demos}/bin/glxgears -fullscreen
              ''}
            fi
          '';
        })
      ];
    }).config.system.build.vm;
  };
}

Before:

After:

I didn't add a new test for this because hardware acceleration can't be used within the context of a test, but I updated a bunch of existing tests to enable it when running interactively.

For example:

nix run github:kira-bruneau/nixpkgs/qemu-vm-opengl#nixosTests.gnome.driverInteractive
>>> start_all()

[rnhmjoj]

Is it safe to enable VirGL by default? What if the guest has opengl.enable but the host doesn't support OpenGL?

Also, is it really that important to enable it in all these tests? I'd rather keep the changes limited to qemu-vm.nix.

[rnhmjoj]

I tested with glxgears and it's working all right, the are a few problems, though:

1. If you resize the qemu window the video output gets stretched out, without maintaing the aspect ratio, even if "zoom to fit" is disabled. This is pretty ugly.

2. If the resolution is larger than the initial window size you get either a completely black window and the output is not properly refreshed.

3. If you use SPICE¹ you get a confusing "OpenGL is not supported by the display" and qemu refuses to start.

Given that the xserver module enables hardware.opengl.enable, VirGL will be pretty much always enabled. I don't think it's ok with those issues.

Footnotes

1. Try adding

     services.spice-vdagentd.enable = true;
   
     virtualisation.qemu.options = [
       "-vga qxl"
       "-spice port=5924,disable-ticketing=on"
       "-device virtio-serial"
       "-chardev spicevmc,id=vdagent,debug=0,name=vdagent"
       "-device virtserialport,chardev=vdagent,name=com.redhat.spice.0"
     ];
   

   and connecting with nix-shell -p spice-gtk --run 'spicy -p 5924' ↩

[rnhmjoj]

ping

[kira-bruneau]

Oops sorry!! I forgot to get back to this.

I was trying to figure out a general way to enable this by default without breaking cli-only & spice setups, but I couldn't really come up with a good solution - so I'll keep the option, but leave it disabled by default.

Also, is it really that important to enable it in all these tests? I'd rather keep the changes limited to qemu-vm.nix.

It's not essential, but I think it would be nice to have, since those VMs are a lot more pleasant to debug with hardware acceleration. The changes I made only apply to the interactive driver, which wouldn't have any impact on how the tests runs in hydra.

[tomfitzhenry]

I like this idea, and it'd be great to have more performant graphics on NixOS VMs.

it looks like we could eventually support this if we enable & use rutabaga when building qemu.

If/when this happens, what option would we use?

Perhaps instead of virtualisation.opengl, it's worth introducing a virtualisation.display (or virtualisation.gpu) enum now, with value virtio-gpu-gl-pci. If/when we introduce rutabaga, we can add another value virtio-gpu-rutabaga.

[tomfitzhenry]

When I checkout this PR, and run a test in interactive mode, qemu fails to start:

$ nix-build -A nixosTests.phosh.driverInteractive
$ ./result/bin/nixos-test-driver
[...]
>>> start_all()
start all VMs
phone: starting vm
mke2fs 1.47.0 (5-Feb-2023)
qemu: GtkGLArea console lacks DMABUF support.
qemu-kvm: ../src/dispatch_common.c:885: epoxy_get_proc_address: Assertion `0 && "Couldn't find current GLX or EGL context.\n"' failed.

I'm running this on Debian Testing, on Wayland, with an amdgpu card/driver.

$ echo $XDG_SESSION_TYPE 
wayland
$ lspci | grep -i VGA
61:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Lexa XT [Radeon PRO WX 3200] (rev 10)
$ dpkg -s xserver-xorg-video-amdgpu | grep Status
Status: install ok installed

I'll test this later on my NixOS machine. Maybe the nix provided Qemu is unable to find the EGL implementation on my Debian Testing machine. My graphics stack-fu is weak.

[tomfitzhenry]

I'll test this later on my NixOS machine. Maybe the nix provided Qemu is unable to find the EGL implementation on my Debian Testing machine. My graphics stack-fu is weak.

Works fine on my NixOS machine.

$ echo $XDG_SESSION_TYPE
wayland
$ lspci | grep -i VGA
00:02.0 VGA compatible controller: Intel Corporation CometLake-S GT2 [UHD Graphics 630] (rev 03)

I'm not sure whether running nixosTests from non-NixOS is a supported usecase. It would be a shame to hamstring ourselves (see the workarounds that this PR removes) to a lowest-common denominator of QEMU support. Is there a way to get working OpenGL passthrough from QEMU, when run via nix-build on a non-NixOS distro?

[rnhmjoj]

OpenGL probably is not working on non-NixOS for the usual reasons. Have you tried NixGL?

[tomfitzhenry]

TIL the usual reasons #9415 . I'll try NixGL sometime, thanks.

With respect to this PR, I think it's fine if non-NixOS users are unable to interactively run NixOS tests for the few tests that rely on OpenGL.

[tomfitzhenry]

I'll try NixGL sometime, thanks.

I tested whether NixGL allowed this PR's tests to be run from Debian Testing. It worked!

[rnhmjoj]

With respect to this PR, I think it's fine if non-NixOS users are unable to interactively run NixOS tests for the few tests that rely on OpenGL.

It would be better if the test simply turned OpenGL off instead of hard failing, though.