1LLVMpipe 2======== 3 4Introduction 5------------ 6 7The Gallium LLVMpipe driver is a software rasterizer that uses LLVM to 8do runtime code generation. Shaders, point/line/triangle rasterization 9and vertex processing are implemented with LLVM IR which is translated 10to x86, x86-64, or ppc64le machine code. Also, the driver is 11multithreaded to take advantage of multiple CPU cores (up to 32 at this 12time). It's the fastest software rasterizer for Mesa. 13 14Requirements 15------------ 16 17- For x86 or amd64 processors, 64-bit mode is recommended. Support for 18 SSE2 is strongly encouraged. Support for SSE3 and SSE4.1 will yield 19 the most efficient code. The fewer features the CPU has the more 20 likely it is that you will run into underperforming, buggy, or 21 incomplete code. 22 23 For ppc64le processors, use of the Altivec feature (the Vector 24 Facility) is recommended if supported; use of the VSX feature (the 25 Vector-Scalar Facility) is recommended if supported AND Mesa is built 26 with LLVM version 4.0 or later. 27 28 See ``/proc/cpuinfo`` to know what your CPU supports. 29 30- Unless otherwise stated, LLVM version 3.9 or later is required. 31 32 For Linux, on a recent Debian based distribution do: 33 34 .. code-block:: sh 35 36 aptitude install llvm-dev 37 38 If you want development snapshot builds of LLVM for Debian and 39 derived distributions like Ubuntu, you can use the APT repository at 40 `apt.llvm.org <https://apt.llvm.org/>`__, which are maintained by 41 Debian's LLVM maintainer. 42 43 For a RPM-based distribution do: 44 45 .. code-block:: sh 46 47 yum install llvm-devel 48 49 If you want development snapshot builds of LLVM for Fedora, you can 50 use the Copr repository at `fedora-llvm-team/llvm-snapshots <https://copr.fedorainfracloud.org/coprs/g/fedora-llvm-team/llvm-snapshots/>`__, 51 which is maintained by Red Hat's LLVM team. 52 53 For Windows you will need to build LLVM from source with MSVC or 54 MINGW (either natively or through cross compilers) and CMake, and set 55 the ``LLVM`` environment variable to the directory you installed it 56 to. LLVM will be statically linked, so when building on MSVC it needs 57 to be built with a matching CRT as Mesa, and you'll need to pass 58 ``-DLLVM_USE_CRT_xxx=yyy`` as described below. 59 60 61 +-----------------+----------------------------------------------------------------+ 62 | LLVM build-type | Mesa build-type | 63 | +--------------------------------+-------------------------------+ 64 | | debug,checked | release,profile | 65 +=================+================================+===============================+ 66 | Debug | ``-DLLVM_USE_CRT_DEBUG=MTd`` | ``-DLLVM_USE_CRT_DEBUG=MT`` | 67 +-----------------+--------------------------------+-------------------------------+ 68 | Release | ``-DLLVM_USE_CRT_RELEASE=MTd`` | ``-DLLVM_USE_CRT_RELEASE=MT`` | 69 +-----------------+--------------------------------+-------------------------------+ 70 71 You can build only the x86 target by passing 72 ``-DLLVM_TARGETS_TO_BUILD=X86`` to CMake. 73 74Building 75-------- 76 77To build everything on Linux invoke meson as: 78 79.. code-block:: sh 80 81 mkdir build 82 cd build 83 meson -D glx=xlib -D gallium-drivers=swrast 84 ninja 85 86To build on Android requires the additional step of building LLVM 87for Android using the NDK. Before following the steps in 88:doc:`Android's documentation <../android>` you must build a version 89of LLVM that targets the NDK with all the required libraries for 90llvmpipe, and then create a wrap file so the meson knows where to 91find the LLVM libraries. It can be a bit tricky to get LLVM to build 92properly using the Android NDK, so the below cmake command can be 93used as a reference to configure LLVM to build with the NDK for x86. 94You need to set the ``ANDROID_NDK_ROOT`` and ``INSTALL_PREFIX`` 95environment variable appropriately. 96 97.. code-block:: sh 98 99 cmake ../llvm-project-18.1.1.src/llvm \ 100 -DCMAKE_TOOLCHAIN_FILE=${ANDROID_NDK_ROOT}/build/cmake/android.toolchain.cmake \ 101 -DANDROID_ABI=x86_64 \ 102 -DANDROID_PLATFORM=android-23 \ 103 -DANDROID_NDK=${ANDROID_NDK_ROOT} \ 104 -DCMAKE_ANDROID_ARCH_ABI=x86_64 \ 105 -DCMAKE_ANDROID_NDK=${ANDROID_NDK_ROOT} \ 106 -DCMAKE_BUILD_TYPE=MinSizeRel \ 107 -DCMAKE_SYSTEM_NAME=Android \ 108 -DCMAKE_SYSTEM_VERSION=23 \ 109 -DCMAKE_INSTALL_PREFIX=${INSTALL_PREFIX} \ 110 -DCMAKE_CXX_FLAGS='-march=x86-64 --target=x86_64-linux-android23 -fno-rtti' \ 111 -DLLVM_HOST_TRIPLE=x86_64-linux-android23 \ 112 -DLLVM_TARGETS_TO_BUILD=X86 \ 113 -DLLVM_BUILD_LLVM_DYLIB=OFF \ 114 -DLLVM_BUILD_TESTS=OFF \ 115 -DLLVM_BUILD_EXAMPLES=OFF \ 116 -DLLVM_BUILD_DOCS=OFF \ 117 -DLLVM_BUILD_TOOLS=OFF \ 118 -DLLVM_ENABLE_RTTI=OFF \ 119 -DLLVM_BUILD_INSTRUMENTED_COVERAGE=OFF \ 120 -DLLVM_NATIVE_TOOL_DIR=${ANDROID_NDK_ROOT}toolchains/llvm/prebuilt/linux-x86_64/bin \ 121 -DLLVM_ENABLE_PIC=False 122 123 make -j$(nproc) install 124 125 126You will also need to create a wrap file, so that meson is able 127to find the LLVM libraries built with the NDK. The process for this 128is described in :doc:`meson documentation <../meson>`. For example a 129file like this would work ``subprojects/llvm/meson.build`` where 130``INSTALL_PREFIX`` is replaced with the path LLVM was installed to. 131 132.. code-block:: 133 134 project('llvm', ['cpp']) 135 136 cpp = meson.get_compiler('cpp') 137 138 _deps = [] 139 _search = join_paths('$INSTALL_PREFIX', 'lib') 140 141 foreach d: ['libLLVMAggressiveInstCombine', 'libLLVMAnalysis', 'libLLVMAsmParser', 'libLLVMAsmPrinter', 'libLLVMBinaryFormat', 'libLLVMBitReader', 'libLLVMBitstreamReader', 'libLLVMBitWriter', 'libLLVMCFGuard', 'libLLVMCFIVerify', 'libLLVMCodeGen', 'libLLVMCodeGenTypes', 'libLLVMCore', 'libLLVMCoroutines', 'libLLVMCoverage', 'libLLVMDebugInfoBTF', 'libLLVMDebugInfoCodeView', 'libLLVMDebuginfod', 'libLLVMDebugInfoDWARF', 'libLLVMDebugInfoGSYM', 'libLLVMDebugInfoLogicalView', 'libLLVMDebugInfoMSF', 'libLLVMDebugInfoPDB', 'libLLVMDemangle', 'libLLVMDiff', 'libLLVMDlltoolDriver', 'libLLVMDWARFLinker', 'libLLVMDWARFLinkerClassic', 'libLLVMDWARFLinkerParallel', 'libLLVMDWP', 'libLLVMExecutionEngine', 'libLLVMExegesis', 'libLLVMExegesisX86', 'libLLVMExtensions', 'libLLVMFileCheck', 'libLLVMFrontendDriver', 'libLLVMFrontendHLSL', 'libLLVMFrontendOffloading', 'libLLVMFrontendOpenACC', 'libLLVMFrontendOpenMP', 'libLLVMFuzzerCLI', 'libLLVMFuzzMutate', 'libLLVMGlobalISel', 'libLLVMHipStdPar', 'libLLVMInstCombine', 'libLLVMInstrumentation', 'libLLVMInterfaceStub', 'libLLVMInterpreter', 'libLLVMipo', 'libLLVMIRPrinter', 'libLLVMIRReader', 'libLLVMJITLink', 'libLLVMLibDriver', 'libLLVMLineEditor', 'libLLVMLinker', 'libLLVMLTO', 'libLLVMMC', 'libLLVMMCA', 'libLLVMMCDisassembler', 'libLLVMMCJIT', 'libLLVMMCParser', 'libLLVMMIRParser', 'libLLVMObjCARCOpts', 'libLLVMObjCopy', 'libLLVMObject', 'libLLVMObjectYAML', 'libLLVMOption', 'libLLVMOrcDebugging', 'libLLVMOrcJIT', 'libLLVMOrcShared', 'libLLVMOrcTargetProcess', 'libLLVMPasses', 'libLLVMProfileData', 'libLLVMRemarks', 'libLLVMRuntimeDyld', 'libLLVMScalarOpts', 'libLLVMSelectionDAG', 'libLLVMSupport', 'libLLVMSymbolize', 'libLLVMTableGen', 'libLLVMTableGenCommon', 'libLLVMTableGenGlobalISel', 'libLLVMTarget', 'libLLVMTargetParser', 'libLLVMTextAPI', 'libLLVMTextAPIBinaryReader', 'libLLVMTransformUtils', 'libLLVMVectorize', 'libLLVMWindowsDriver', 'libLLVMWindowsManifest', 'libLLVMX86AsmParser', 'libLLVMX86CodeGen', 'libLLVMX86Desc', 'libLLVMX86Disassembler', 'libLLVMX86Info', 'libLLVMX86TargetMCA', 'libLLVMXRay'] 142 _deps += cpp.find_library(d, dirs : _search) 143 endforeach 144 145 dep_llvm = declare_dependency( 146 include_directories : include_directories('$INSTALL_PREFIX/include'), 147 dependencies : _deps, 148 version : '6.0.0', 149 ) 150 151 has_rtti = false 152 irbuilder_h = files('$INSTALL_PREFIX/include/llvm/IR/IRBuilder.h') 153 154Afterwards you can continue following the instructors to build mesa 155on :doc:`Android <../android>` and follow the steps to add the driver 156directly to an Android OS image. Please note that you'll need to set 157``-Ddri-search-path`` when building the driver, so that the loader 158will be able to find the driver libraries in the vendor partition 159``/vendor/lib(64)/egl``. 160 161Using 162----- 163 164Environment variables 165~~~~~~~~~~~~~~~~~~~~~ 166 167.. envvar:: LP_NATIVE_VECTOR_WIDTH 168 169 We can use it to override vector bits. Because sometimes it turns 170 out LLVMpipe can be fastest by using 128 bit vectors, 171 yet use AVX instructions. 172 173.. envvar:: GALLIUM_NOSSE 174 175 Deprecated in favor of ``GALLIUM_OVERRIDE_CPU_CAPS``, 176 use ``GALLIUM_OVERRIDE_CPU_CAPS=nosse`` instead. 177 178.. envvar:: LP_FORCE_SSE2 179 180 Deprecated in favor of ``GALLIUM_OVERRIDE_CPU_CAPS`` 181 use ``GALLIUM_OVERRIDE_CPU_CAPS=sse2`` instead. 182 183Linux 184~~~~~ 185 186On Linux, building will create a drop-in alternative for ``libGL.so`` 187into 188 189:: 190 191 build/foo/gallium/targets/libgl-xlib/libGL.so 192 193or 194 195:: 196 197 lib/gallium/libGL.so 198 199To use it set the ``LD_LIBRARY_PATH`` environment variable accordingly. 200 201Windows 202~~~~~~~ 203 204On Windows, building will create 205``build/windows-x86-debug/gallium/targets/libgl-gdi/opengl32.dll`` which 206is a drop-in alternative for system's ``opengl32.dll``, which will use 207the Mesa ICD, ``build/windows-x86-debug/gallium/targets/wgl/libgallium_wgl.dll``. 208To use it put both DLLs in the same directory as your application. It can also 209be used by replacing the native ICD driver, but it's quite an advanced usage, so if 210you need to ask, don't even try it. 211 212There is however an easy way to replace the OpenGL software renderer 213that comes with Microsoft Windows 7 (or later) with LLVMpipe (that is, 214on systems without any OpenGL drivers): 215 216- copy 217 ``build/windows-x86-debug/gallium/targets/wgl/libgallium_wgl.dll`` to 218 ``C:\Windows\SysWOW64\mesadrv.dll`` 219 220- load this registry settings: 221 222 :: 223 224 REGEDIT4 225 226 ; https://technet.microsoft.com/en-us/library/cc749368.aspx 227 ; https://www.msfn.org/board/topic/143241-portable-windows-7-build-from-winpe-30/page-5#entry942596 228 [HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\Windows NT\CurrentVersion\OpenGLDrivers\MSOGL] 229 "DLL"="mesadrv.dll" 230 "DriverVersion"=dword:00000001 231 "Flags"=dword:00000001 232 "Version"=dword:00000002 233 234- Ditto for 64 bits drivers if you need them. 235 236Profiling 237--------- 238 239Linux perf integration 240~~~~~~~~~~~~~~~~~~~~~~ 241 242On Linux, it is possible to have symbol resolution of JIT code with 243`Linux perf <https://perf.wiki.kernel.org/>`__: 244 245:: 246 247 perf record -g /my/application 248 perf report 249 250When run inside Linux perf, LLVMpipe will create a 251``/tmp/perf-XXXXX.map`` file with symbol address table. It also dumps 252assembly code to ``/tmp/perf-XXXXX.map.asm``, which can be used by the 253``bin/perf-annotate-jit.py`` script to produce disassembly of the 254generated code annotated with the samples. 255 256You can obtain a call graph via 257`Gprof2Dot <https://github.com/jrfonseca/gprof2dot#linux-perf>`__. 258 259FlameGraph support 260~~~~~~~~~~~~~~~~~~~~~~ 261 262Outside Linux, it is possible to generate a 263`FlameGraph <https://github.com/brendangregg/FlameGraph>`__ 264with resolved JIT symbols. 265 266Set the environment variable ``JIT_SYMBOL_MAP_DIR`` to a directory path, 267and run your LLVMpipe program. Follow the FlameGraph instructions: 268capture traces using a supported tool (for example DTrace), 269and fold the stacks using the associated script 270(``stackcollapse.pl`` for DTrace stacks). 271 272LLVMpipe will create a ``jit-symbols-XXXXX.map`` file containing the symbol 273address table inside the chosen directory. It will also dump the JIT 274disassemblies to ``jit-symbols-XXXXX.map.asm``. Run your folded traces and 275both output files through the ``bin/flamegraph_map_lp_jit.py`` script to map 276addresses to JIT symbols, and annotate the disassembly with the sample counts. 277 278Unit testing 279------------ 280 281Building will also create several unit tests in 282``build/linux-???-debug/gallium/drivers/llvmpipe``: 283 284- ``lp_test_blend``: blending 285- ``lp_test_conv``: SIMD vector conversion 286- ``lp_test_format``: pixel unpacking/packing 287 288Some of these tests can output results and benchmarks to a tab-separated 289file for later analysis, e.g.: 290 291:: 292 293 build/linux-x86_64-debug/gallium/drivers/llvmpipe/lp_test_blend -o blend.tsv 294 295Development Notes 296----------------- 297 298- When looking at this code for the first time, start in lp_state_fs.c, 299 and then skim through the ``lp_bld_*`` functions called there, and 300 the comments at the top of the ``lp_bld_*.c`` functions. 301- The driver-independent parts of the LLVM / Gallium code are found in 302 ``src/gallium/auxiliary/gallivm/``. The filenames and function 303 prefixes need to be renamed from ``lp_bld_`` to something else 304 though. 305- We use LLVM-C bindings for now. They are not documented, but follow 306 the C++ interfaces very closely, and appear to be complete enough for 307 code generation. See `this stand-alone 308 example <https://npcontemplation.blogspot.com/2008/06/secret-of-llvm-c-bindings.html>`__. 309 See the ``llvm-c/Core.h`` file for reference. 310 311.. _recommended_reading: 312 313Recommended Reading 314------------------- 315 316- Rasterization 317 318 - `Triangle Scan Conversion using 2D Homogeneous 319 Coordinates <https://userpages.cs.umbc.edu/olano/papers/2dh-tri/>`__ 320 - `Rasterization on 321 Larrabee <https://www.drdobbs.com/parallel/rasterization-on-larrabee/217200602>`__ 322 - `Rasterization using half-space 323 functions <http://web.archive.org/web/20110820052005/http://www.devmaster.net/codespotlight/show.php?id=17>`__ 324 - `Advanced 325 Rasterization <http://web.archive.org/web/20140514220546/http://devmaster.net/posts/6145/advanced-rasterization>`__ 326 - `Optimizing Software Occlusion 327 Culling <https://fgiesen.wordpress.com/2013/02/17/optimizing-sw-occlusion-culling-index/>`__ 328 329- Texture sampling 330 331 - `Perspective Texture 332 Mapping <https://chrishecker.com/Miscellaneous_Technical_Articles#Perspective_Texture_Mapping>`__ 333 - `Texturing As In 334 Unreal <https://www.flipcode.com/archives/Texturing_As_In_Unreal.shtml>`__ 335 - `Run-Time MIP-Map 336 Filtering <http://web.archive.org/web/20220709145555/http://www.gamasutra.com/view/feature/3301/runtime_mipmap_filtering.php>`__ 337 - `Will "brilinear" filtering 338 persist? <https://alt.3dcenter.org/artikel/2003/10-26_a_english.php>`__ 339 - `Trilinear 340 filtering <http://ixbtlabs.com/articles2/gffx/nv40-rx800-3.html>`__ 341 - `Texture tiling and 342 swizzling <https://fgiesen.wordpress.com/2011/01/17/texture-tiling-and-swizzling/>`__ 343 344- SIMD 345 346 - `Whole-Function 347 Vectorization <https://compilers.cs.uni-saarland.de/projects/wfv/#pubs>`__ 348 349- Optimization 350 351 - `Optimizing Pixomatic For Modern x86 352 Processors <https://www.drdobbs.com/optimizing-pixomatic-for-modern-x86-proc/184405807>`__ 353 - `Intel 64 and IA-32 Architectures Optimization Reference 354 Manual <https://www.intel.com/content/www/us/en/content-details/779559/intel-64-and-ia-32-architectures-optimization-reference-manual.html>`__ 355 - `Software optimization 356 resources <https://www.agner.org/optimize/>`__ 357 - `Intel Intrinsics 358 Guide <https://www.intel.com/content/www/us/en/docs/intrinsics-guide/index.html>`__ 359 360- LLVM 361 362 - `LLVM Language Reference 363 Manual <https://llvm.org/docs/LangRef.html>`__ 364 - `The secret of LLVM C 365 bindings <https://npcontemplation.blogspot.com/2008/06/secret-of-llvm-c-bindings.html>`__ 366 367- General 368 369 - `A trip through the Graphics 370 Pipeline <https://fgiesen.wordpress.com/2011/07/09/a-trip-through-the-graphics-pipeline-2011-index/>`__ 371 - `WARP Architecture and 372 Performance <https://learn.microsoft.com/en-us/windows/win32/direct3darticles/directx-warp#warp-architecture-and-performance>`__ 373