1# Precompiling 2 3Precompiling is compiling Python source files (`.py` files) into byte code 4(`.pyc` files) at build time instead of runtime. Doing it at build time can 5improve performance by skipping that work at runtime. 6 7Precompiling is disabled by default, so you must enable it using flags or 8attributes to use it. 9 10## Overhead of precompiling 11 12While precompiling helps runtime performance, it has two main costs: 131. Increasing the size (count and disk usage) of runfiles. It approximately 14 double the count of the runfiles because for every `.py` file, there is also 15 a `.pyc` file. Compiled files are generally around the same size as the 16 source files, so it approximately doubles the disk usage. 172. Precompiling requires running an extra action at build time. While 18 compiling itself isn't that expensive, the overhead can become noticable 19 as more files need to be compiled. 20 21## Binary-level opt-in 22 23Because of the costs of precompiling, it may not be feasible to globally enable it 24for your repo for everything. For example, some binaries may be 25particularly large, and doubling the number of runfiles isn't doable. 26 27If this is the case, there's an alternative way to more selectively and 28incrementally control precompiling on a per-binry basis. 29 30To use this approach, the two basic steps are: 311. Disable pyc files from being automatically added to runfiles: 32 {bzl:obj}`--@rules_python//python/config_settings:precompile_add_to_runfiles=decided_elsewhere`, 332. Set the `pyc_collection` attribute on the binaries/tests that should or should 34 not use precompiling. 35 36The default for the `pyc_collection` attribute is controlled by the flag 37{bzl:obj}`--@rules_python//python/config_settings:pyc_collection`, so you 38can use an opt-in or opt-out approach by setting its value: 39* targets must opt-out: `--@rules_python//python/config_settings:pyc_collection=include_pyc` 40* targets must opt-in: `--@rules_python//python/config_settings:pyc_collection=disabled` 41 42## Advanced precompiler customization 43 44The default implementation of the precompiler is a persistent, multiplexed, 45sandbox-aware, cancellation-enabled, json-protocol worker that uses the same 46interpreter as the target toolchain. This works well for local builds, but may 47not work as well for remote execution builds. To customize the precompiler, two 48mechanisms are available: 49 50* The exec tools toolchain allows customizing the precompiler binary used with 51 the `precompiler` attribute. Arbitrary binaries are supported. 52* The execution requirements can be customized using 53 `--@rules_python//tools/precompiler:execution_requirements`. This is a list 54 flag that can be repeated. Each entry is a key=value that is added to the 55 execution requirements of the `PyCompile` action. Note that this flag 56 is specific to the rules_python precompiler. If a custom binary is used, 57 this flag will have to be propagated from the custom binary using the 58 `testing.ExecutionInfo` provider; refer to the `py_interpreter_program` an 59 60The default precompiler implementation is an asynchronous/concurrent 61implementation. If you find it has bugs or hangs, please report them. In the 62meantime, the flag `--worker_extra_flag=PyCompile=--worker_impl=serial` can 63be used to switch to a synchronous/serial implementation that may not perform 64as well, but is less likely to have issues. 65 66The `execution_requirements` keys of most relevance are: 67* `supports-workers`: 1 or 0, to indicate if a regular persistent worker is 68 desired. 69* `supports-multiplex-workers`: 1 o 0, to indicate if a multiplexed persistent 70 worker is desired. 71* `requires-worker-protocol`: json or proto; the rules_python precompiler 72 currently only supports json. 73* `supports-multiplex-sandboxing`: 1 or 0, to indicate if sanboxing is of the 74 worker is supported. 75* `supports-worker-cancellation`: 1 or 1, to indicate if requests to the worker 76 can be cancelled. 77 78Note that any execution requirements values can be specified in the flag. 79 80## Known issues, caveats, and idiosyncracies 81 82* Precompiling requires Bazel 7+ with the Pystar rule implementation enabled. 83* Mixing rules_python PyInfo with Bazel builtin PyInfo will result in pyc files 84 being dropped. 85* Precompiled files may not be used in certain cases prior to Python 3.11. This 86 occurs due Python adding the directory of the binary's main `.py` file, which 87 causes the module to be found in the workspace source directory instead of 88 within the binary's runfiles directory (where the pyc files are). This can 89 usually be worked around by removing `sys.path[0]` (or otherwise ensuring the 90 runfiles directory comes before the repos source directory in `sys.path`). 91* The pyc filename does not include the optimization level (e.g. 92 `foo.cpython-39.opt-2.pyc`). This works fine (it's all byte code), but also 93 means the interpreter `-O` argument can't be used -- doing so will cause the 94 interpreter to look for the non-existent `opt-N` named files. 95