1Virtio-GPU Venus 2================ 3 4Venus is a Virtio-GPU protocol for Vulkan command serialization. The protocol 5definition and codegen are hosted at `venus-protocol 6<https://gitlab.freedesktop.org/virgl/venus-protocol>`__. The renderer is 7hosted at `virglrenderer 8<https://gitlab.freedesktop.org/virgl/virglrenderer>`__. 9 10Requirements 11------------ 12 13The Venus renderer requires 14 15- Vulkan 1.1 16- :ext:`VK_EXT_external_memory_dma_buf` 17- :ext:`VK_EXT_image_drm_format_modifier` 18- :ext:`VK_EXT_queue_family_foreign` 19 20from the host driver. However, it violates the spec in some places currently 21and also relies on implementation-defined behaviors in others. It is not 22expected to work on all drivers meeting the requirements. It has only been 23tested with 24 25- ANV 21.1 or later 26- RADV 21.1 or later (the host kernel must have 27 ``CONFIG_TRANSPARENT_HUGEPAGE`` disabled because of this `KVM issue 28 <https://github.com/google/security-research/security/advisories/GHSA-7wq5-phmq-m584>`__) 29- TURNIP 22.0 or later 30- Mali r32p0 or later 31 32The Venus driver requires supports for 33 34- ``VIRTGPU_PARAM_3D_FEATURES`` 35- ``VIRTGPU_PARAM_CAPSET_QUERY_FIX`` 36- ``VIRTGPU_PARAM_RESOURCE_BLOB`` 37- ``VIRTGPU_PARAM_HOST_VISIBLE`` 38- ``VIRTGPU_PARAM_CROSS_DEVICE`` 39- ``VIRTGPU_PARAM_CONTEXT_INIT`` 40 41from the virtio-gpu kernel driver, unless vtest is used. That usually means 42the guest kernel should be at least 5.16 or have the parameters back ported, 43paired with hypervisors such as `crosvm 44<https://chromium.googlesource.com/chromiumos/platform/crosvm>`__, or `patched 45qemu 46<https://www.collabora.com/news-and-blog/blog/2021/11/26/venus-on-qemu-enabling-new-virtual-vulkan-driver/>`__. 47 48vtest 49----- 50 51The simplest way to test Venus is to use virglrenderer's vtest server. To 52build virglrenderer with Venus support and to start the vtest server, 53 54.. code-block:: sh 55 56 $ git clone https://gitlab.freedesktop.org/virgl/virglrenderer.git 57 $ cd virglrenderer 58 $ meson out -Dvenus=true 59 $ meson compile -C out 60 $ meson devenv -C out 61 $ ./vtest/virgl_test_server --venus 62 $ exit 63 64In another shell, 65 66.. code-block:: sh 67 68 $ export VK_DRIVER_FILES=<path-to-virtio_icd.x86_64.json> 69 $ export VN_DEBUG=vtest 70 $ vulkaninfo 71 $ vkcube 72 73If the host driver of the system is not new enough, it is a good idea to build 74the host driver as well when building the Venus driver. Just remember to set 75:envvar:`VK_DRIVER_FILES` when starting the vtest server so that the vtest 76server finds the locally built host driver. 77 78Virtio-GPU 79---------- 80 81The driver requires ``VIRTGPU_PARAM_CONTEXT_INIT`` from the virtio-gpu kernel 82driver, which was upstreamed in kernel 5.16. 83 84crosvm is written in Rust. To build crosvm, make sure Rust has been installed 85and 86 87.. code-block:: sh 88 89 $ git clone --recurse-submodules \ 90 https://chromium.googlesource.com/chromiumos/platform/crosvm 91 $ cd crosvm 92 $ RUSTFLAGS="-L<path-to-virglrenderer>/out/src" cargo build \ 93 --features "x wl-dmabuf virgl_renderer virgl_renderer_next default-no-sandbox" 94 95Note that crosvm must be built with ``default-no-sandbox`` or started with 96``--disable-sandbox`` in this setup. 97 98This is how one might want to start crosvm 99 100.. code-block:: sh 101 102 $ sudo LD_LIBRARY_PATH=<...> VK_DRIVER_FILES=<...> ./target/debug/crosvm run \ 103 --gpu vulkan=true \ 104 --gpu-render-server path=<path-to-virglrenderer>/out/server/virgl_render_server \ 105 --display-window-keyboard \ 106 --display-window-mouse \ 107 --net "host-ip 192.168.0.1,netmask=255.255.255.0,mac=12:34:56:78:9a:bc" \ 108 --rwdisk disk.img \ 109 -p root=/dev/vda1 \ 110 <path-to-bzImage> 111 112assuming a working system is installed to partition 1 of ``disk.img``. 113``sudo`` or ``CAP_NET_ADMIN`` is needed to set up the TAP network device. 114 115Virtio-GPU and Virtio-WL 116------------------------ 117 118In this setup, the guest userspace uses Xwayland and a special Wayland 119compositor to connect guest X11/Wayland clients to the host Wayland 120compositor, using Virtio-WL as the transport. This setup is more tedious, but 121that should hopefully change over time. 122 123For now, the guest kernel must be built from the ``chromeos-5.10`` branch of 124the `Chrome OS kernel 125<https://chromium.googlesource.com/chromiumos/third_party/kernel>`__. 126 127To build minigbm and to enable minigbm support in virglrenderer, 128 129.. code-block:: sh 130 131 $ git clone https://chromium.googlesource.com/chromiumos/platform/minigbm 132 $ cd minigbm 133 $ CFLAGS=-DDRV_<I915-or-your-driver> OUT=out DESTDIR=out/install make install 134 $ cd ../virglrenderer 135 $ meson configure out -Dminigbm_allocation=true 136 $ meson compile -C out 137 138Make sure a host Wayland compositor is running. Replace 139``--display-window-keyboard --display-window-mouse`` by 140``--wayland-sock=<path-to-wayland-socket>`` when starting crosvm. 141 142In the guest, build and start Sommelier, the special Wayland compositor, 143 144.. code-block:: sh 145 146 $ git clone https://chromium.googlesource.com/chromiumos/platform2 147 $ cd platform2/vm_tools/sommelier 148 $ meson out -Dxwayland_path=/usr/bin/Xwayland -Dxwayland_gl_driver_path=/usr/lib/dri 149 $ meson compile -C out 150 $ sudo chmod 777 /dev/wl0 151 $ ./out/sommelier -X --glamor 152 --xwayland-gl-driver-path=<path-to-locally-built-gl-driver> \ 153 sleep infinity 154 155Optional Requirements 156--------------------- 157 158When virglrenderer is built with ``-Dminigbm_allocation=true``, the Venus 159renderer might need to import GBM BOs. The imports will fail unless the host 160driver supports the formats, especially multi-planar ones, and the DRM format 161modifiers of the GBM BOs. 162 163In the future, if virglrenderer's ``virgl_renderer_export_fence`` is 164supported, the Venus renderer will require :ext:`VK_KHR_external_fence_fd` 165with ``VK_EXTERNAL_FENCE_HANDLE_TYPE_SYNC_FD_BIT`` from the host driver. 166 167VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT 168----------------------------------- 169 170The Venus renderer makes assumptions about ``VkDeviceMemory`` that has 171``VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT``. The assumptions are illegal and rely 172on the current behaviors of the host drivers. It should be possible to remove 173some of the assumptions and incrementally improve compatibilities with more 174host drivers by imposing platform-specific requirements. But the long-term 175plan is to create a new Vulkan extension for the host drivers to address this 176specific use case. 177 178The Venus renderer assumes a device memory that has 179``VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT`` can be exported as a mmapable dma-buf 180(in the future, the plan is to export the device memory as an opaque fd). It 181chains ``VkExportMemoryAllocateInfo`` to ``VkMemoryAllocateInfo`` without 182checking if the host driver can export the device memory. 183 184The dma-buf is mapped (in the future, the plan is to import the opaque fd and 185call ``vkMapMemory``) but the mapping is not accessed. Instead, the mapping 186is passed to ``KVM_SET_USER_MEMORY_REGION``. The hypervisor, host KVM, and 187the guest kernel work together to set up a write-back or write-combined guest 188mapping (see ``virtio_gpu_vram_mmap`` of the virtio-gpu kernel driver). CPU 189accesses to the device memory are via the guest mapping, and are assumed to be 190coherent when the device memory also has 191``VK_MEMORY_PROPERTY_HOST_COHERENT_BIT``. 192 193While the Venus renderer can force a ``VkDeviceMemory`` external, it does not 194force a ``VkImage`` or a ``VkBuffer`` external. As a result, it can bind an 195external device memory to a non-external resource. 196