1*61c4878aSAndroid Build Coastguard Worker.. _module-pw_env_setup: 2*61c4878aSAndroid Build Coastguard Worker 3*61c4878aSAndroid Build Coastguard Worker------------ 4*61c4878aSAndroid Build Coastguard Workerpw_env_setup 5*61c4878aSAndroid Build Coastguard Worker------------ 6*61c4878aSAndroid Build Coastguard WorkerA classic problem in the embedded space is reducing the time from git clone 7*61c4878aSAndroid Build Coastguard Workerto having a binary executing on a device. The issue is that an entire suite 8*61c4878aSAndroid Build Coastguard Workerof tools is needed for non-trivial production embedded projects. For example: 9*61c4878aSAndroid Build Coastguard Worker 10*61c4878aSAndroid Build Coastguard Worker- A C++ compiler for your target device, and also for your host 11*61c4878aSAndroid Build Coastguard Worker- A build system or three; for example, GN, Ninja, CMake, Bazel 12*61c4878aSAndroid Build Coastguard Worker- A code formatting program like clang-format 13*61c4878aSAndroid Build Coastguard Worker- A debugger like OpenOCD to flash and debug your embedded device (OpenOCD 14*61c4878aSAndroid Build Coastguard Worker support removed for Windows) 15*61c4878aSAndroid Build Coastguard Worker- A known Python version with known modules installed for scripting 16*61c4878aSAndroid Build Coastguard Worker- A Go compiler for the Go-based command line tools 17*61c4878aSAndroid Build Coastguard Worker 18*61c4878aSAndroid Build Coastguard Worker...and so on 19*61c4878aSAndroid Build Coastguard Worker 20*61c4878aSAndroid Build Coastguard WorkerIn the server space, container solutions like Docker or Podman solve this; 21*61c4878aSAndroid Build Coastguard Workerhowever, in our experience container solutions are a mixed bag for embedded 22*61c4878aSAndroid Build Coastguard Workersystems development where one frequently needs access to native system 23*61c4878aSAndroid Build Coastguard Workerresources like USB devices, or must operate on Windows. 24*61c4878aSAndroid Build Coastguard Worker 25*61c4878aSAndroid Build Coastguard Worker``pw_env_setup`` is our compromise solution for this problem that works on Mac, 26*61c4878aSAndroid Build Coastguard WorkerWindows, and Linux. It leverages the Chrome packaging system `CIPD`_ to 27*61c4878aSAndroid Build Coastguard Workerbootstrap a Python installation, which in turn inflates a virtual 28*61c4878aSAndroid Build Coastguard Workerenvironment. The tooling is installed into your workspace, and makes no 29*61c4878aSAndroid Build Coastguard Workerchanges to your system. This tooling is designed to be reused by any 30*61c4878aSAndroid Build Coastguard Workerproject. 31*61c4878aSAndroid Build Coastguard Worker 32*61c4878aSAndroid Build Coastguard Worker.. _CIPD: https://github.com/luci/luci-go/tree/HEAD/cipd 33*61c4878aSAndroid Build Coastguard Worker 34*61c4878aSAndroid Build Coastguard WorkerUsers interact with ``pw_env_setup`` with two commands: ``. bootstrap.sh`` and 35*61c4878aSAndroid Build Coastguard Worker``. activate.sh``. The bootstrap command always pulls down the current versions 36*61c4878aSAndroid Build Coastguard Workerof CIPD packages and sets up the Python virtual environment. The activate 37*61c4878aSAndroid Build Coastguard Workercommand reinitializes a previously configured environment, and if none is found, 38*61c4878aSAndroid Build Coastguard Workerruns bootstrap. 39*61c4878aSAndroid Build Coastguard Worker 40*61c4878aSAndroid Build Coastguard Worker.. note:: 41*61c4878aSAndroid Build Coastguard Worker 42*61c4878aSAndroid Build Coastguard Worker On Windows the scripts used to set up the environment are ``bootstrap.bat`` 43*61c4878aSAndroid Build Coastguard Worker and ``activate.bat``. 44*61c4878aSAndroid Build Coastguard Worker 45*61c4878aSAndroid Build Coastguard Worker ``bootstrap.fish`` and ``activate.fish`` are also available for `Fish shell 46*61c4878aSAndroid Build Coastguard Worker <https://fishshell.com/>`_ users. 47*61c4878aSAndroid Build Coastguard Worker 48*61c4878aSAndroid Build Coastguard Worker For simplicity they will be referred to with the ``.sh`` endings unless the 49*61c4878aSAndroid Build Coastguard Worker distinction is relevant. 50*61c4878aSAndroid Build Coastguard Worker 51*61c4878aSAndroid Build Coastguard WorkerOn POSIX systems, the environment can be deactivated by running ``deactivate``. 52*61c4878aSAndroid Build Coastguard Worker 53*61c4878aSAndroid Build Coastguard Worker================================== 54*61c4878aSAndroid Build Coastguard WorkerUsing pw_env_setup in your project 55*61c4878aSAndroid Build Coastguard Worker================================== 56*61c4878aSAndroid Build Coastguard Worker 57*61c4878aSAndroid Build Coastguard WorkerDownstream Projects Using Pigweed's Packages 58*61c4878aSAndroid Build Coastguard Worker******************************************** 59*61c4878aSAndroid Build Coastguard Worker 60*61c4878aSAndroid Build Coastguard WorkerProjects using Pigweed can leverage ``pw_env_setup`` to install Pigweed's 61*61c4878aSAndroid Build Coastguard Workerdependencies or their own dependencies. Projects that only want to use Pigweed's 62*61c4878aSAndroid Build Coastguard Workerdependencies without modifying them can just source Pigweed's ``bootstrap.sh`` 63*61c4878aSAndroid Build Coastguard Workerand ``activate.sh`` scripts. 64*61c4878aSAndroid Build Coastguard Worker 65*61c4878aSAndroid Build Coastguard WorkerAn example of what your project's `bootstrap.sh` could look like is below. This 66*61c4878aSAndroid Build Coastguard Workerassumes `bootstrap.sh` is at the top level of your repository. 67*61c4878aSAndroid Build Coastguard Worker 68*61c4878aSAndroid Build Coastguard Worker.. code-block:: bash 69*61c4878aSAndroid Build Coastguard Worker 70*61c4878aSAndroid Build Coastguard Worker # Do not include a "#!" line, this must be sourced and not executed. 71*61c4878aSAndroid Build Coastguard Worker 72*61c4878aSAndroid Build Coastguard Worker # This assumes the user is sourcing this file from it's parent directory. See 73*61c4878aSAndroid Build Coastguard Worker # below for a more flexible way to handle this. 74*61c4878aSAndroid Build Coastguard Worker PROJ_SETUP_SCRIPT_PATH="$(pwd)/bootstrap.sh" 75*61c4878aSAndroid Build Coastguard Worker 76*61c4878aSAndroid Build Coastguard Worker export PW_PROJECT_ROOT="$(_python_abspath "$(dirname "$PROJ_SETUP_SCRIPT_PATH")")" 77*61c4878aSAndroid Build Coastguard Worker 78*61c4878aSAndroid Build Coastguard Worker # You may wish to check if the user is attempting to execute this script 79*61c4878aSAndroid Build Coastguard Worker # instead of sourcing it. See below for an example of how to handle that 80*61c4878aSAndroid Build Coastguard Worker # situation. 81*61c4878aSAndroid Build Coastguard Worker 82*61c4878aSAndroid Build Coastguard Worker # Source Pigweed's bootstrap utility script. 83*61c4878aSAndroid Build Coastguard Worker # Using '.' instead of 'source' for POSIX compatibility. Since users don't use 84*61c4878aSAndroid Build Coastguard Worker # dash directly, using 'source' in most documentation so users don't get 85*61c4878aSAndroid Build Coastguard Worker # confused and try to `./bootstrap.sh`. 86*61c4878aSAndroid Build Coastguard Worker . "$PW_PROJECT_ROOT/third_party/pigweed/pw_env_setup/util.sh" 87*61c4878aSAndroid Build Coastguard Worker 88*61c4878aSAndroid Build Coastguard Worker pw_check_root "$PW_ROOT" 89*61c4878aSAndroid Build Coastguard Worker _PW_ACTUAL_ENVIRONMENT_ROOT="$(pw_get_env_root)" 90*61c4878aSAndroid Build Coastguard Worker export _PW_ACTUAL_ENVIRONMENT_ROOT 91*61c4878aSAndroid Build Coastguard Worker SETUP_SH="$_PW_ACTUAL_ENVIRONMENT_ROOT/activate.sh" 92*61c4878aSAndroid Build Coastguard Worker pw_bootstrap --args... # See below for details about args. 93*61c4878aSAndroid Build Coastguard Worker pw_finalize bootstrap "$SETUP_SH" 94*61c4878aSAndroid Build Coastguard Worker 95*61c4878aSAndroid Build Coastguard Worker 96*61c4878aSAndroid Build Coastguard WorkerBazel Usage 97*61c4878aSAndroid Build Coastguard Worker----------- 98*61c4878aSAndroid Build Coastguard WorkerIt is possible to pull in a CIPD dependency into Bazel using WORKSPACE rules 99*61c4878aSAndroid Build Coastguard Workerrather than using `bootstrap.sh`. e.g. 100*61c4878aSAndroid Build Coastguard Worker 101*61c4878aSAndroid Build Coastguard Worker.. code-block:: python 102*61c4878aSAndroid Build Coastguard Worker 103*61c4878aSAndroid Build Coastguard Worker # WORKSPACE 104*61c4878aSAndroid Build Coastguard Worker 105*61c4878aSAndroid Build Coastguard Worker load("//pw_env_setup/bazel/cipd_setup:cipd_rules.bzl", "pigweed_deps") 106*61c4878aSAndroid Build Coastguard Worker 107*61c4878aSAndroid Build Coastguard Worker # Setup CIPD client and packages. 108*61c4878aSAndroid Build Coastguard Worker # Required by: pigweed. 109*61c4878aSAndroid Build Coastguard Worker # Used by modules: all. 110*61c4878aSAndroid Build Coastguard Worker pigweed_deps() 111*61c4878aSAndroid Build Coastguard Worker 112*61c4878aSAndroid Build Coastguard Worker load("@cipd_deps//:cipd_init.bzl", "cipd_init") 113*61c4878aSAndroid Build Coastguard Worker 114*61c4878aSAndroid Build Coastguard Worker cipd_init() 115*61c4878aSAndroid Build Coastguard Worker 116*61c4878aSAndroid Build Coastguard Worker 117*61c4878aSAndroid Build Coastguard WorkerThis will make the entire set of Pigweeds remote repositories available to your 118*61c4878aSAndroid Build Coastguard Workerproject. Though these repositories will only be donwloaded if you use them. To 119*61c4878aSAndroid Build Coastguard Workerget a full list of the remote repositories that this configures, run: 120*61c4878aSAndroid Build Coastguard Worker 121*61c4878aSAndroid Build Coastguard Worker.. code-block:: sh 122*61c4878aSAndroid Build Coastguard Worker 123*61c4878aSAndroid Build Coastguard Worker bazel query //external:all | grep cipd_ 124*61c4878aSAndroid Build Coastguard Worker 125*61c4878aSAndroid Build Coastguard WorkerAll files and executables in each CIPD remote repository is exported and visible 126*61c4878aSAndroid Build Coastguard Workereither directely (`@cipd_<dep>//:<file>`) or from 'all' filegroup 127*61c4878aSAndroid Build Coastguard Worker(`@cipd_<dep>//:all`). 128*61c4878aSAndroid Build Coastguard Worker 129*61c4878aSAndroid Build Coastguard WorkerFrom here it is possible to get access to the Bloaty binaries using the 130*61c4878aSAndroid Build Coastguard Workerfollowing command. For example; 131*61c4878aSAndroid Build Coastguard Worker 132*61c4878aSAndroid Build Coastguard Worker.. code-block:: sh 133*61c4878aSAndroid Build Coastguard Worker 134*61c4878aSAndroid Build Coastguard Worker bazel run @cipd_pigweed_third_party_bloaty_embedded_linux_amd64//:bloaty \ 135*61c4878aSAndroid Build Coastguard Worker -- --help 136*61c4878aSAndroid Build Coastguard Worker 137*61c4878aSAndroid Build Coastguard WorkerUser-Friendliness 138*61c4878aSAndroid Build Coastguard Worker----------------- 139*61c4878aSAndroid Build Coastguard Worker 140*61c4878aSAndroid Build Coastguard WorkerYou may wish to allow sourcing `bootstrap.sh` from a different directory. In 141*61c4878aSAndroid Build Coastguard Workerthat case you'll need the following at the top of `bootstrap.sh`. 142*61c4878aSAndroid Build Coastguard Worker 143*61c4878aSAndroid Build Coastguard Worker.. code-block:: bash 144*61c4878aSAndroid Build Coastguard Worker 145*61c4878aSAndroid Build Coastguard Worker _python_abspath () { 146*61c4878aSAndroid Build Coastguard Worker python -c "import os.path; print(os.path.abspath('$@'))" 147*61c4878aSAndroid Build Coastguard Worker } 148*61c4878aSAndroid Build Coastguard Worker 149*61c4878aSAndroid Build Coastguard Worker # Use this code from Pigweed's bootstrap to find the path to this script when 150*61c4878aSAndroid Build Coastguard Worker # sourced. This should work with common shells. PW_CHECKOUT_ROOT is only used in 151*61c4878aSAndroid Build Coastguard Worker # presubmit tests with strange setups, and can be omitted if you're not using 152*61c4878aSAndroid Build Coastguard Worker # Pigweed's automated testing infrastructure. 153*61c4878aSAndroid Build Coastguard Worker if test -n "$PW_CHECKOUT_ROOT"; then 154*61c4878aSAndroid Build Coastguard Worker PROJ_SETUP_SCRIPT_PATH="$(_python_abspath "$PW_CHECKOUT_ROOT/bootstrap.sh")" 155*61c4878aSAndroid Build Coastguard Worker unset PW_CHECKOUT_ROOT 156*61c4878aSAndroid Build Coastguard Worker # Shell: bash. 157*61c4878aSAndroid Build Coastguard Worker elif test -n "$BASH"; then 158*61c4878aSAndroid Build Coastguard Worker PROJ_SETUP_SCRIPT_PATH="$(_python_abspath "$BASH_SOURCE")" 159*61c4878aSAndroid Build Coastguard Worker # Shell: zsh. 160*61c4878aSAndroid Build Coastguard Worker elif test -n "$ZSH_NAME"; then 161*61c4878aSAndroid Build Coastguard Worker PROJ_SETUP_SCRIPT_PATH="$(_python_abspath "${(%):-%N}")" 162*61c4878aSAndroid Build Coastguard Worker # Shell: dash. 163*61c4878aSAndroid Build Coastguard Worker elif test ${0##*/} = dash; then 164*61c4878aSAndroid Build Coastguard Worker PROJ_SETUP_SCRIPT_PATH="$(_python_abspath \ 165*61c4878aSAndroid Build Coastguard Worker "$(lsof -p $$ -Fn0 | tail -1 | sed 's#^[^/]*##;')")" 166*61c4878aSAndroid Build Coastguard Worker # If everything else fails, try $0. It could work. 167*61c4878aSAndroid Build Coastguard Worker else 168*61c4878aSAndroid Build Coastguard Worker PROJ_SETUP_SCRIPT_PATH="$(_python_abspath "$0")" 169*61c4878aSAndroid Build Coastguard Worker fi 170*61c4878aSAndroid Build Coastguard Worker 171*61c4878aSAndroid Build Coastguard WorkerYou may also wish to check if the user is attempting to execute `bootstrap.sh` 172*61c4878aSAndroid Build Coastguard Workerinstead of sourcing it. Executing `bootstrap.sh` would download everything 173*61c4878aSAndroid Build Coastguard Workerrequired for the environment, but cannot modify the environment of the parent 174*61c4878aSAndroid Build Coastguard Workerprocess. To check for this add the following. 175*61c4878aSAndroid Build Coastguard Worker 176*61c4878aSAndroid Build Coastguard Worker.. code-block:: bash 177*61c4878aSAndroid Build Coastguard Worker 178*61c4878aSAndroid Build Coastguard Worker # Check if this file is being executed or sourced. 179*61c4878aSAndroid Build Coastguard Worker _pw_sourced=0 180*61c4878aSAndroid Build Coastguard Worker # If not running in Pigweed's automated testing infrastructure the 181*61c4878aSAndroid Build Coastguard Worker # SWARMING_BOT_ID check is unnecessary. 182*61c4878aSAndroid Build Coastguard Worker if [ -n "$SWARMING_BOT_ID" ]; then 183*61c4878aSAndroid Build Coastguard Worker # If set we're running on swarming and don't need this check. 184*61c4878aSAndroid Build Coastguard Worker _pw_sourced=1 185*61c4878aSAndroid Build Coastguard Worker elif [ -n "$ZSH_EVAL_CONTEXT" ]; then 186*61c4878aSAndroid Build Coastguard Worker case $ZSH_EVAL_CONTEXT in *:file) _pw_sourced=1;; esac 187*61c4878aSAndroid Build Coastguard Worker elif [ -n "$KSH_VERSION" ]; then 188*61c4878aSAndroid Build Coastguard Worker [ "$(cd $(dirname -- $0) && pwd -P)/$(basename -- $0)" != \ 189*61c4878aSAndroid Build Coastguard Worker "$(cd $(dirname -- ${.sh.file}) && pwd -P)/$(basename -- ${.sh.file})" ] \ 190*61c4878aSAndroid Build Coastguard Worker && _pw_sourced=1 191*61c4878aSAndroid Build Coastguard Worker elif [ -n "$BASH_VERSION" ]; then 192*61c4878aSAndroid Build Coastguard Worker (return 0 2>/dev/null) && _pw_sourced=1 193*61c4878aSAndroid Build Coastguard Worker else # All other shells: examine $0 for known shell binary filenames 194*61c4878aSAndroid Build Coastguard Worker # Detects `sh` and `dash`; add additional shell filenames as needed. 195*61c4878aSAndroid Build Coastguard Worker case ${0##*/} in sh|dash) _pw_sourced=1;; esac 196*61c4878aSAndroid Build Coastguard Worker fi 197*61c4878aSAndroid Build Coastguard Worker 198*61c4878aSAndroid Build Coastguard Worker _pw_eval_sourced "$_pw_sourced" 199*61c4878aSAndroid Build Coastguard Worker 200*61c4878aSAndroid Build Coastguard WorkerDownstream Projects Using Different Packages 201*61c4878aSAndroid Build Coastguard Worker******************************************** 202*61c4878aSAndroid Build Coastguard WorkerProjects depending on Pigweed but using additional or different packages should 203*61c4878aSAndroid Build Coastguard Workercopy the Pigweed `sample project`'s ``bootstrap.sh`` and ``pigweed.json`` and 204*61c4878aSAndroid Build Coastguard Workerupdate the call to ``pw_bootstrap``. Search for "downstream" for other places 205*61c4878aSAndroid Build Coastguard Workerthat may require changes, like setting the ``PW_ROOT`` and ``PW_PROJECT_ROOT`` 206*61c4878aSAndroid Build Coastguard Workerenvironment variables. Explanations of parts of ``pigweed.json`` are described 207*61c4878aSAndroid Build Coastguard Workerhere. 208*61c4878aSAndroid Build Coastguard Worker 209*61c4878aSAndroid Build Coastguard Worker.. _sample project: https://pigweed.googlesource.com/pigweed/sample_project/+/HEAD 210*61c4878aSAndroid Build Coastguard Worker 211*61c4878aSAndroid Build Coastguard Worker``pw.pw_env_setup.root_variable`` 212*61c4878aSAndroid Build Coastguard Worker Variable used to point to the root of the source tree. Optional, can always 213*61c4878aSAndroid Build Coastguard Worker use ``PW_PROJECT_ROOT`` instead. (That variable will be set regardless of 214*61c4878aSAndroid Build Coastguard Worker whether this is provided.) 215*61c4878aSAndroid Build Coastguard Worker 216*61c4878aSAndroid Build Coastguard Worker``pw.pw_env_setup.relative_pigweed_root`` 217*61c4878aSAndroid Build Coastguard Worker Location of the Pigweed submodule within the source tree. Optional—environment 218*61c4878aSAndroid Build Coastguard Worker setup will work correctly without this. If present, will confirm that it's 219*61c4878aSAndroid Build Coastguard Worker correct. May be used by other tooling. 220*61c4878aSAndroid Build Coastguard Worker 221*61c4878aSAndroid Build Coastguard Worker``pw.pw_env_setup.cipd_package_files`` 222*61c4878aSAndroid Build Coastguard Worker CIPD package file. JSON file consisting of a list of additional CIPD package 223*61c4878aSAndroid Build Coastguard Worker files to import and a list of dictionaries with "path", "platforms", "subdir", 224*61c4878aSAndroid Build Coastguard Worker "tags", and "version_file" keys. Both top-level lists are optional. An 225*61c4878aSAndroid Build Coastguard Worker example is below. Only "path", "platforms", and "tags" are required. If 226*61c4878aSAndroid Build Coastguard Worker "version_file" is specified then ``pw doctor`` will fail if that version file 227*61c4878aSAndroid Build Coastguard Worker is not present. If "subdir" is specified then this packages will be installed 228*61c4878aSAndroid Build Coastguard Worker in a subdirectory of the directory created for packages in this file. 229*61c4878aSAndroid Build Coastguard Worker 230*61c4878aSAndroid Build Coastguard Worker.. code-block:: json 231*61c4878aSAndroid Build Coastguard Worker 232*61c4878aSAndroid Build Coastguard Worker { 233*61c4878aSAndroid Build Coastguard Worker "included_files": [ 234*61c4878aSAndroid Build Coastguard Worker "foo.json" 235*61c4878aSAndroid Build Coastguard Worker ], 236*61c4878aSAndroid Build Coastguard Worker "packages": [ 237*61c4878aSAndroid Build Coastguard Worker { 238*61c4878aSAndroid Build Coastguard Worker "path": "infra/3pp/tools/go/${platform}", 239*61c4878aSAndroid Build Coastguard Worker "platforms": [ 240*61c4878aSAndroid Build Coastguard Worker "linux-amd64", 241*61c4878aSAndroid Build Coastguard Worker "linux-arm64", 242*61c4878aSAndroid Build Coastguard Worker "mac-amd64", 243*61c4878aSAndroid Build Coastguard Worker "windows-amd64" 244*61c4878aSAndroid Build Coastguard Worker ], 245*61c4878aSAndroid Build Coastguard Worker "subdir": "pa/th", 246*61c4878aSAndroid Build Coastguard Worker "tags": [ 247*61c4878aSAndroid Build Coastguard Worker "version:[email protected]" 248*61c4878aSAndroid Build Coastguard Worker ], 249*61c4878aSAndroid Build Coastguard Worker "version_file": ".versions/go.cipd_version" 250*61c4878aSAndroid Build Coastguard Worker } 251*61c4878aSAndroid Build Coastguard Worker ] 252*61c4878aSAndroid Build Coastguard Worker } 253*61c4878aSAndroid Build Coastguard Worker 254*61c4878aSAndroid Build Coastguard Worker``pw.pw_env_setup.project_actions`` 255*61c4878aSAndroid Build Coastguard Worker A list of plugins to load and run after CIPD setup, but prior to virtualenv 256*61c4878aSAndroid Build Coastguard Worker setup, for e.g. downloading project-specific tools or artifacts needed by 257*61c4878aSAndroid Build Coastguard Worker later steps. Particularly useful for downstream projects with limited CIPD 258*61c4878aSAndroid Build Coastguard Worker access. 259*61c4878aSAndroid Build Coastguard Worker 260*61c4878aSAndroid Build Coastguard Worker A plugin is specified as a dictionary with two keys: "import_path" and 261*61c4878aSAndroid Build Coastguard Worker "module_name". The "import_path" is relative to the root of the checkout. 262*61c4878aSAndroid Build Coastguard Worker 263*61c4878aSAndroid Build Coastguard Worker The specified module must provide a "run_actions" method which takes a single 264*61c4878aSAndroid Build Coastguard Worker argument, "env_vars", which is a pw_env_setup.Environment instance. 265*61c4878aSAndroid Build Coastguard Worker 266*61c4878aSAndroid Build Coastguard Worker Sample plugin and pigweed.json blob: 267*61c4878aSAndroid Build Coastguard Worker 268*61c4878aSAndroid Build Coastguard Worker.. code-block:: python 269*61c4878aSAndroid Build Coastguard Worker 270*61c4878aSAndroid Build Coastguard Worker """Sample pw_env_setup project action plugin. 271*61c4878aSAndroid Build Coastguard Worker 272*61c4878aSAndroid Build Coastguard Worker A sample/starter project action plugin template for pw_env_setup. 273*61c4878aSAndroid Build Coastguard Worker """ 274*61c4878aSAndroid Build Coastguard Worker def run_action(**kwargs): 275*61c4878aSAndroid Build Coastguard Worker """Sample project action.""" 276*61c4878aSAndroid Build Coastguard Worker if "env" not in kwargs: 277*61c4878aSAndroid Build Coastguard Worker raise ValueError(f"Missing required kwarg 'env', got %{kwargs}") 278*61c4878aSAndroid Build Coastguard Worker 279*61c4878aSAndroid Build Coastguard Worker kwargs["env"].prepend("PATH", "PATH_TO_NEW_TOOLS") 280*61c4878aSAndroid Build Coastguard Worker raise NotImplementedError("Sample project action running!") 281*61c4878aSAndroid Build Coastguard Worker 282*61c4878aSAndroid Build Coastguard Worker.. code-block:: json 283*61c4878aSAndroid Build Coastguard Worker 284*61c4878aSAndroid Build Coastguard Worker "project_actions" : [ 285*61c4878aSAndroid Build Coastguard Worker { 286*61c4878aSAndroid Build Coastguard Worker "import_path": "pw_env_setup", 287*61c4878aSAndroid Build Coastguard Worker "module_name": "sample_project_action" 288*61c4878aSAndroid Build Coastguard Worker } 289*61c4878aSAndroid Build Coastguard Worker ], 290*61c4878aSAndroid Build Coastguard Worker 291*61c4878aSAndroid Build Coastguard Worker``pw.pw_env_setup.virtualenv.gn_args`` 292*61c4878aSAndroid Build Coastguard Worker Any necessary GN args to be used when installing Python packages. 293*61c4878aSAndroid Build Coastguard Worker 294*61c4878aSAndroid Build Coastguard Worker``pw.pw_env_setup.virtualenv.gn_targets`` 295*61c4878aSAndroid Build Coastguard Worker Target for installing Python packages. Downstream projects will need to 296*61c4878aSAndroid Build Coastguard Worker create targets to install their packages or only use Pigweed Python packages. 297*61c4878aSAndroid Build Coastguard Worker 298*61c4878aSAndroid Build Coastguard Worker``pw.pw_env_setup.virtualenv.gn_root`` 299*61c4878aSAndroid Build Coastguard Worker The root directory of your GN build tree, relative to ``PW_PROJECT_ROOT``. 300*61c4878aSAndroid Build Coastguard Worker This is the directory your project's ``.gn`` file is located in. If you're 301*61c4878aSAndroid Build Coastguard Worker only installing Pigweed Python packages, use the location of the Pigweed 302*61c4878aSAndroid Build Coastguard Worker submodule. 303*61c4878aSAndroid Build Coastguard Worker 304*61c4878aSAndroid Build Coastguard Worker``pw.pw_env_setup.virtualenv.requirements`` 305*61c4878aSAndroid Build Coastguard Worker A list of Python Pip requirements files for installing into the Pigweed 306*61c4878aSAndroid Build Coastguard Worker virtualenv. Each file will be passed as additional ``--requirement`` argument 307*61c4878aSAndroid Build Coastguard Worker to a single ```pip install`` at the beginning of bootstrap's ``Python 308*61c4878aSAndroid Build Coastguard Worker environment`` setup stage. See the `Requirements Files documentation`_ for 309*61c4878aSAndroid Build Coastguard Worker details on what can be specified using requirements files. 310*61c4878aSAndroid Build Coastguard Worker 311*61c4878aSAndroid Build Coastguard Worker``pw.pw_env_setup.virtualenv.constraints`` 312*61c4878aSAndroid Build Coastguard Worker A list of Python Pip constraints files. These constraints will be passed to 313*61c4878aSAndroid Build Coastguard Worker every ``pip`` invocation as an additional ``--constraint`` argument during 314*61c4878aSAndroid Build Coastguard Worker bootstrap. virtualenv. See the `Constraints Files documentation`_ for details 315*61c4878aSAndroid Build Coastguard Worker on formatting. 316*61c4878aSAndroid Build Coastguard Worker 317*61c4878aSAndroid Build Coastguard Worker``pw.pw_env_setup.virtualenv.system_packages`` 318*61c4878aSAndroid Build Coastguard Worker A boolean value that can be used the give the Python virtual environment 319*61c4878aSAndroid Build Coastguard Worker access to the system site packages. Defaults to ``false``. 320*61c4878aSAndroid Build Coastguard Worker 321*61c4878aSAndroid Build Coastguard Worker``pw.pw_env_setup.virtualenv.pip_install_offline`` 322*61c4878aSAndroid Build Coastguard Worker A boolean value that adds ``--no-index`` to all ``pip install`` commands that 323*61c4878aSAndroid Build Coastguard Worker are part of bootstrap. This forces pip to not reach out to the internet 324*61c4878aSAndroid Build Coastguard Worker (usually `pypi.org <https://pypi.org/>`_) to download packages. Using this 325*61c4878aSAndroid Build Coastguard Worker option requires setting 326*61c4878aSAndroid Build Coastguard Worker ``pw.pw_env_setup.virtualenv.pip_install_find_links``. Defaults to 327*61c4878aSAndroid Build Coastguard Worker ``false``. 328*61c4878aSAndroid Build Coastguard Worker 329*61c4878aSAndroid Build Coastguard Worker .. seealso:: 330*61c4878aSAndroid Build Coastguard Worker The Python GN guide for offline pip installation: 331*61c4878aSAndroid Build Coastguard Worker :ref:`docs-python-build-installing-offline` 332*61c4878aSAndroid Build Coastguard Worker 333*61c4878aSAndroid Build Coastguard Worker``pw.pw_env_setup.virtualenv.pip_install_find_links`` 334*61c4878aSAndroid Build Coastguard Worker List of paths to folders containing Python wheels (``*.whl``) or source tar 335*61c4878aSAndroid Build Coastguard Worker files (``*.tar.gz``). Pip will check each of these directories when looking 336*61c4878aSAndroid Build Coastguard Worker for potential install candidates. Each path will be passed to all ``pip 337*61c4878aSAndroid Build Coastguard Worker install`` commands as ``--find-links PATH``. 338*61c4878aSAndroid Build Coastguard Worker 339*61c4878aSAndroid Build Coastguard Worker .. tip:: 340*61c4878aSAndroid Build Coastguard Worker Environment variables may be used in these paths. For example: 341*61c4878aSAndroid Build Coastguard Worker 342*61c4878aSAndroid Build Coastguard Worker .. code-block:: json 343*61c4878aSAndroid Build Coastguard Worker 344*61c4878aSAndroid Build Coastguard Worker "virtualenv": { 345*61c4878aSAndroid Build Coastguard Worker "pip_install_find_links": [ 346*61c4878aSAndroid Build Coastguard Worker "${PW_PROJECT_ROOT}/pip_cache" 347*61c4878aSAndroid Build Coastguard Worker ] 348*61c4878aSAndroid Build Coastguard Worker } 349*61c4878aSAndroid Build Coastguard Worker 350*61c4878aSAndroid Build Coastguard Worker``pw.pw_env_setup.virtualenv.pip_install_require_hashes`` 351*61c4878aSAndroid Build Coastguard Worker Adds ``--require-hashes`` This option enforces hash checking on Python 352*61c4878aSAndroid Build Coastguard Worker package files. Defaults to ``false``. 353*61c4878aSAndroid Build Coastguard Worker 354*61c4878aSAndroid Build Coastguard Worker``pw.pw_env_setup.virtualenv.pip_install_disable_cache`` 355*61c4878aSAndroid Build Coastguard Worker A boolean value that adds ``--no-cache-dir`` to all ``pip install`` commands 356*61c4878aSAndroid Build Coastguard Worker that are part of bootstrap. This forces pip to ignore any previously cached 357*61c4878aSAndroid Build Coastguard Worker Python packages. On most systems this is located in 358*61c4878aSAndroid Build Coastguard Worker ``~/.cache/pip/``. Defaults to ``false``. 359*61c4878aSAndroid Build Coastguard Worker 360*61c4878aSAndroid Build Coastguard Worker``pw.pw_env_setup.optional_submodules`` 361*61c4878aSAndroid Build Coastguard Worker By default environment setup will check that all submodules are present in 362*61c4878aSAndroid Build Coastguard Worker the checkout. Any submodules in this list are excluded from that check. 363*61c4878aSAndroid Build Coastguard Worker 364*61c4878aSAndroid Build Coastguard Worker``pw.pw_env_setup.required_submodules`` 365*61c4878aSAndroid Build Coastguard Worker If this is specified instead of ``optional_submodules`` bootstrap will only 366*61c4878aSAndroid Build Coastguard Worker complain if one of the required submodules is not present. Combining this 367*61c4878aSAndroid Build Coastguard Worker with ``optional_submodules`` is not supported. 368*61c4878aSAndroid Build Coastguard Worker 369*61c4878aSAndroid Build Coastguard Worker``pw.pw_env_setup.pw_packages`` 370*61c4878aSAndroid Build Coastguard Worker A list of packages to install using :ref:`pw_package <module-pw_package>` 371*61c4878aSAndroid Build Coastguard Worker after the rest of bootstrap completes. 372*61c4878aSAndroid Build Coastguard Worker 373*61c4878aSAndroid Build Coastguard Worker``pw.pw_env_setup.gni_file`` 374*61c4878aSAndroid Build Coastguard Worker Location to write a ``.gni`` file containing paths to many things within the 375*61c4878aSAndroid Build Coastguard Worker environment directory. Defaults to 376*61c4878aSAndroid Build Coastguard Worker ``build_overrides/pigweed_environment.gni``. 377*61c4878aSAndroid Build Coastguard Worker 378*61c4878aSAndroid Build Coastguard Worker``pw.pw_env_setup.json_file`` 379*61c4878aSAndroid Build Coastguard Worker Location to write a ``.json`` file containing step-by-step modifications to 380*61c4878aSAndroid Build Coastguard Worker the environment, for reading by tools that don't inherit an environment from 381*61c4878aSAndroid Build Coastguard Worker a sourced ``bootstrap.sh``. 382*61c4878aSAndroid Build Coastguard Worker 383*61c4878aSAndroid Build Coastguard Worker``pw.pw_env_setup.rosetta`` 384*61c4878aSAndroid Build Coastguard Worker Whether to use Rosetta to use amd64 packages on arm64 Macs. Accepted values 385*61c4878aSAndroid Build Coastguard Worker are ``never``, ``allow``, and ``force``. For now, ``allow`` means ``force``. 386*61c4878aSAndroid Build Coastguard Worker At some point in the future ``allow`` will be changed to mean ``never``. 387*61c4878aSAndroid Build Coastguard Worker 388*61c4878aSAndroid Build Coastguard WorkerAn example of a config file is below. 389*61c4878aSAndroid Build Coastguard Worker 390*61c4878aSAndroid Build Coastguard Worker.. code-block:: json 391*61c4878aSAndroid Build Coastguard Worker 392*61c4878aSAndroid Build Coastguard Worker { 393*61c4878aSAndroid Build Coastguard Worker "pw": { 394*61c4878aSAndroid Build Coastguard Worker "pw_env_setup": { 395*61c4878aSAndroid Build Coastguard Worker "root_variable": "EXAMPLE_ROOT", 396*61c4878aSAndroid Build Coastguard Worker "cipd_package_files": [ 397*61c4878aSAndroid Build Coastguard Worker "pigweed/pw_env_setup/py/pw_env_setup/cipd_setup/pigweed.json", 398*61c4878aSAndroid Build Coastguard Worker "pigweed/pw_env_setup/py/pw_env_setup/cipd_setup/luci.json" 399*61c4878aSAndroid Build Coastguard Worker "tools/myprojectname.json" 400*61c4878aSAndroid Build Coastguard Worker ], 401*61c4878aSAndroid Build Coastguard Worker "virtualenv": { 402*61c4878aSAndroid Build Coastguard Worker "gn_root": ".", 403*61c4878aSAndroid Build Coastguard Worker "gn_targets": [ 404*61c4878aSAndroid Build Coastguard Worker ":python.install", 405*61c4878aSAndroid Build Coastguard Worker ], 406*61c4878aSAndroid Build Coastguard Worker "system_packages": false 407*61c4878aSAndroid Build Coastguard Worker }, 408*61c4878aSAndroid Build Coastguard Worker "pw_packages": [], 409*61c4878aSAndroid Build Coastguard Worker "optional_submodules": [ 410*61c4878aSAndroid Build Coastguard Worker "optional/submodule/one", 411*61c4878aSAndroid Build Coastguard Worker "optional/submodule/two" 412*61c4878aSAndroid Build Coastguard Worker ], 413*61c4878aSAndroid Build Coastguard Worker "gni_file": "tools/environment.gni", 414*61c4878aSAndroid Build Coastguard Worker "json_file": "tools/environment.json", 415*61c4878aSAndroid Build Coastguard Worker "rosetta": "allow" 416*61c4878aSAndroid Build Coastguard Worker } 417*61c4878aSAndroid Build Coastguard Worker } 418*61c4878aSAndroid Build Coastguard Worker } 419*61c4878aSAndroid Build Coastguard Worker 420*61c4878aSAndroid Build Coastguard WorkerOnly the packages necessary for almost all projects based on Pigweed are 421*61c4878aSAndroid Build Coastguard Workerincluded in the ``cipd_setup/pigweed.json`` file. A number of other files are 422*61c4878aSAndroid Build Coastguard Workerpresent in that directory for projects that need more than the minimum. 423*61c4878aSAndroid Build Coastguard WorkerInternal-Google projects using LUCI should at least include ``luci.json``. 424*61c4878aSAndroid Build Coastguard Worker 425*61c4878aSAndroid Build Coastguard WorkerIn case the CIPD packages need to be referenced from other scripts, variables 426*61c4878aSAndroid Build Coastguard Workerlike ``PW_${BASENAME}_CIPD_INSTALL_DIR`` point to the CIPD install directories, 427*61c4878aSAndroid Build Coastguard Workerwhere ``${BASENAME}`` is ``"PIGWEED"`` for 428*61c4878aSAndroid Build Coastguard Worker``"pigweed/pw_env_setup/py/pw_env_setup/cipd_setup/pigweed.json"`` and 429*61c4878aSAndroid Build Coastguard Worker``"LUCI"`` for 430*61c4878aSAndroid Build Coastguard Worker``"pigweed/pw_env_setup/py/pw_env_setup/cipd_setup/luci.json"``. This example 431*61c4878aSAndroid Build Coastguard Workerwould set the following environment variables. 432*61c4878aSAndroid Build Coastguard Worker 433*61c4878aSAndroid Build Coastguard Worker- ``PW_LUCI_CIPD_INSTALL_DIR`` 434*61c4878aSAndroid Build Coastguard Worker- ``PW_MYPROJECTNAME_CIPD_INSTALL_DIR`` 435*61c4878aSAndroid Build Coastguard Worker- ``PW_PIGWEED_CIPD_INSTALL_DIR`` 436*61c4878aSAndroid Build Coastguard Worker 437*61c4878aSAndroid Build Coastguard WorkerThese directories are also referenced in the gni_file specified by the 438*61c4878aSAndroid Build Coastguard Workerenvironment config file as ``dir_cipd_${BASENAME}``. This allows the GN build to 439*61c4878aSAndroid Build Coastguard Workerreliably reference these directories without using GN ``getenv()`` calls or 440*61c4878aSAndroid Build Coastguard Workerhardcoding paths. 441*61c4878aSAndroid Build Coastguard Worker 442*61c4878aSAndroid Build Coastguard WorkerIn addition, ``PW_${BASENAME}_CIPD_INSTALL_DIR`` and 443*61c4878aSAndroid Build Coastguard Worker``PW_${BASENAME}_CIPD_INSTALL_DIR/bin`` are both added to ``PATH`` for each 444*61c4878aSAndroid Build Coastguard Workerpackage directory. 445*61c4878aSAndroid Build Coastguard Worker 446*61c4878aSAndroid Build Coastguard WorkerIf multiple packages install executables with the same name, the file mentioned 447*61c4878aSAndroid Build Coastguard Workerlast topologically takes priority. For example, with the file contents below, 448*61c4878aSAndroid Build Coastguard Worker``d.json``'s entries will appear in ``PATH`` before ``c.json``'s, which will 449*61c4878aSAndroid Build Coastguard Workerappear before ``b.json``'s, which will appear before ``a.json``'s. 450*61c4878aSAndroid Build Coastguard Worker 451*61c4878aSAndroid Build Coastguard Worker.. code-block:: json 452*61c4878aSAndroid Build Coastguard Worker :caption: :octicon:`file;1em` pigweed.json 453*61c4878aSAndroid Build Coastguard Worker 454*61c4878aSAndroid Build Coastguard Worker { 455*61c4878aSAndroid Build Coastguard Worker "pw": { 456*61c4878aSAndroid Build Coastguard Worker "pw_env_setup": { 457*61c4878aSAndroid Build Coastguard Worker "cipd_package_files": [ 458*61c4878aSAndroid Build Coastguard Worker "a.json", 459*61c4878aSAndroid Build Coastguard Worker "b.json", 460*61c4878aSAndroid Build Coastguard Worker "d.json" 461*61c4878aSAndroid Build Coastguard Worker ] 462*61c4878aSAndroid Build Coastguard Worker } 463*61c4878aSAndroid Build Coastguard Worker } 464*61c4878aSAndroid Build Coastguard Worker } 465*61c4878aSAndroid Build Coastguard Worker 466*61c4878aSAndroid Build Coastguard Worker.. code-block:: json 467*61c4878aSAndroid Build Coastguard Worker :caption: :octicon:`file;1em` a.json 468*61c4878aSAndroid Build Coastguard Worker 469*61c4878aSAndroid Build Coastguard Worker { 470*61c4878aSAndroid Build Coastguard Worker "package_files": [ 471*61c4878aSAndroid Build Coastguard Worker // ... 472*61c4878aSAndroid Build Coastguard Worker ] 473*61c4878aSAndroid Build Coastguard Worker } 474*61c4878aSAndroid Build Coastguard Worker 475*61c4878aSAndroid Build Coastguard Worker.. code-block:: json 476*61c4878aSAndroid Build Coastguard Worker :caption: :octicon:`file;1em` b.json 477*61c4878aSAndroid Build Coastguard Worker 478*61c4878aSAndroid Build Coastguard Worker { 479*61c4878aSAndroid Build Coastguard Worker "included_files": ["c.json"], 480*61c4878aSAndroid Build Coastguard Worker "package_files": [ 481*61c4878aSAndroid Build Coastguard Worker // ... 482*61c4878aSAndroid Build Coastguard Worker ] 483*61c4878aSAndroid Build Coastguard Worker } 484*61c4878aSAndroid Build Coastguard Worker 485*61c4878aSAndroid Build Coastguard Worker.. code-block:: json 486*61c4878aSAndroid Build Coastguard Worker :caption: :octicon:`file;1em` c.json 487*61c4878aSAndroid Build Coastguard Worker 488*61c4878aSAndroid Build Coastguard Worker { 489*61c4878aSAndroid Build Coastguard Worker "package_files": [ 490*61c4878aSAndroid Build Coastguard Worker // ... 491*61c4878aSAndroid Build Coastguard Worker ] 492*61c4878aSAndroid Build Coastguard Worker } 493*61c4878aSAndroid Build Coastguard Worker 494*61c4878aSAndroid Build Coastguard Worker.. code-block:: json 495*61c4878aSAndroid Build Coastguard Worker :caption: :octicon:`file;1em` d.json 496*61c4878aSAndroid Build Coastguard Worker 497*61c4878aSAndroid Build Coastguard Worker { 498*61c4878aSAndroid Build Coastguard Worker "package_files": [ 499*61c4878aSAndroid Build Coastguard Worker // ... 500*61c4878aSAndroid Build Coastguard Worker ] 501*61c4878aSAndroid Build Coastguard Worker } 502*61c4878aSAndroid Build Coastguard Worker 503*61c4878aSAndroid Build Coastguard Worker.. code-block:: 504*61c4878aSAndroid Build Coastguard Worker :caption: Effective File Loading Order 505*61c4878aSAndroid Build Coastguard Worker 506*61c4878aSAndroid Build Coastguard Worker pigweed.json 507*61c4878aSAndroid Build Coastguard Worker a.json 508*61c4878aSAndroid Build Coastguard Worker b.json 509*61c4878aSAndroid Build Coastguard Worker c.json 510*61c4878aSAndroid Build Coastguard Worker d.json 511*61c4878aSAndroid Build Coastguard Worker 512*61c4878aSAndroid Build Coastguard WorkerPinning Python Packages 513*61c4878aSAndroid Build Coastguard Worker*********************** 514*61c4878aSAndroid Build Coastguard WorkerPython modules usually express dependencies as ranges, which makes it easier to 515*61c4878aSAndroid Build Coastguard Workerinstall many Python packages that might otherwise have conflicting dependencies. 516*61c4878aSAndroid Build Coastguard WorkerHowever, this means version of packages can often change underneath us and 517*61c4878aSAndroid Build Coastguard Workerbuilds will not be hermetic. 518*61c4878aSAndroid Build Coastguard Worker 519*61c4878aSAndroid Build Coastguard WorkerTo ensure versions don't change without approval, Pigweed by default pins the 520*61c4878aSAndroid Build Coastguard Workerversions of packages it depends on using a `pip constraints file`_. To pin the 521*61c4878aSAndroid Build Coastguard Workerversions of additional packages your project depends on, run 522*61c4878aSAndroid Build Coastguard Worker``pw python-packages list <path/to/constraints/file>`` and then add 523*61c4878aSAndroid Build Coastguard Worker``pw_build_PIP_CONSTRAINTS = ["//path/to/constraints/file"]`` to your project's 524*61c4878aSAndroid Build Coastguard Worker``.gn`` file (see `Pigweed's .gn file`_ for an example). 525*61c4878aSAndroid Build Coastguard Worker 526*61c4878aSAndroid Build Coastguard Worker.. _pip constraints file: https://pip.pypa.io/en/stable/user_guide/#constraints-files 527*61c4878aSAndroid Build Coastguard Worker.. _default constraints: https://cs.pigweed.dev/pigweed/+/main:pw_env_setup/py/pw_env_setup/virtualenv_setup/constraint.list 528*61c4878aSAndroid Build Coastguard Worker.. _Pigweed's .gn file: https://cs.pigweed.dev/pigweed/+/main:.gn 529*61c4878aSAndroid Build Coastguard Worker 530*61c4878aSAndroid Build Coastguard WorkerTo update packages, set ``pw_build_PIP_CONSTRAINTS = []``, delete the 531*61c4878aSAndroid Build Coastguard Workerenvironment, and bootstrap again. Then run the ``list`` command from above 532*61c4878aSAndroid Build Coastguard Workeragain, and run ``pw presubmit``. 533*61c4878aSAndroid Build Coastguard Worker 534*61c4878aSAndroid Build Coastguard WorkerEnvironment Variables 535*61c4878aSAndroid Build Coastguard Worker********************* 536*61c4878aSAndroid Build Coastguard WorkerInput Variables 537*61c4878aSAndroid Build Coastguard Worker--------------- 538*61c4878aSAndroid Build Coastguard WorkerThe following environment variables affect env setup behavior. Most users will 539*61c4878aSAndroid Build Coastguard Workernever need to set these. 540*61c4878aSAndroid Build Coastguard Worker 541*61c4878aSAndroid Build Coastguard Worker``CIPD_CACHE_DIR`` 542*61c4878aSAndroid Build Coastguard Worker Location of CIPD cache dir. Read by CIPD, but if unset will be defaulted to 543*61c4878aSAndroid Build Coastguard Worker ``$HOME/.cipd-cache-dir``. 544*61c4878aSAndroid Build Coastguard Worker 545*61c4878aSAndroid Build Coastguard Worker``PW_NO_CIPD_CACHE_DIR`` 546*61c4878aSAndroid Build Coastguard Worker Disables the CIPD cache. 547*61c4878aSAndroid Build Coastguard Worker 548*61c4878aSAndroid Build Coastguard Worker``PW_ACTIVATE_SKIP_CHECKS`` 549*61c4878aSAndroid Build Coastguard Worker If set, skip running ``pw doctor`` at end of bootstrap/activate. Intended to 550*61c4878aSAndroid Build Coastguard Worker be used by automated tools but not interactively. 551*61c4878aSAndroid Build Coastguard Worker 552*61c4878aSAndroid Build Coastguard Worker``PW_BANNER_FUNC`` 553*61c4878aSAndroid Build Coastguard Worker Command to print a banner at the beginning of bootstrap. 554*61c4878aSAndroid Build Coastguard Worker 555*61c4878aSAndroid Build Coastguard Worker``PW_BOOTSTRAP_PYTHON`` 556*61c4878aSAndroid Build Coastguard Worker Python executable to be used, for example "python3". Defaults to 557*61c4878aSAndroid Build Coastguard Worker "python3" if that's in ``PATH``, then tries "python". 558*61c4878aSAndroid Build Coastguard Worker 559*61c4878aSAndroid Build Coastguard Worker``PW_CIPD_SERVICE_ACCOUNT_JSON`` 560*61c4878aSAndroid Build Coastguard Worker Value to pass as ``-service-account-json`` to CIPD invocations. This should 561*61c4878aSAndroid Build Coastguard Worker point either to a service account JSON key file, or be the magical value 562*61c4878aSAndroid Build Coastguard Worker ``:gce`` to tell the tool to fetch tokens from GCE metadata server. 563*61c4878aSAndroid Build Coastguard Worker 564*61c4878aSAndroid Build Coastguard Worker``PW_ENVIRONMENT_ROOT`` 565*61c4878aSAndroid Build Coastguard Worker Location to which packages are installed. Defaults to ``environment`` folder 566*61c4878aSAndroid Build Coastguard Worker within the checkout root. This variable is cleared after environment setup is 567*61c4878aSAndroid Build Coastguard Worker complete. 568*61c4878aSAndroid Build Coastguard Worker 569*61c4878aSAndroid Build Coastguard Worker``PW_ENVSETUP_DISABLE_SPINNER`` 570*61c4878aSAndroid Build Coastguard Worker Disable the spinner during env setup. Intended to be used when the output is 571*61c4878aSAndroid Build Coastguard Worker being redirected to a log. 572*61c4878aSAndroid Build Coastguard Worker 573*61c4878aSAndroid Build Coastguard Worker``PW_ENVSETUP_DISABLE_SPINNER`` 574*61c4878aSAndroid Build Coastguard Worker Disable the console spinner that runs when waiting for env setup steps to 575*61c4878aSAndroid Build Coastguard Worker complete. 576*61c4878aSAndroid Build Coastguard Worker 577*61c4878aSAndroid Build Coastguard Worker``PW_ENVSETUP_NO_BANNER`` 578*61c4878aSAndroid Build Coastguard Worker Skip printing the banner. 579*61c4878aSAndroid Build Coastguard Worker 580*61c4878aSAndroid Build Coastguard Worker``PW_ENVSETUP_QUIET`` 581*61c4878aSAndroid Build Coastguard Worker Disables all non-error output. 582*61c4878aSAndroid Build Coastguard Worker 583*61c4878aSAndroid Build Coastguard Worker``PW_PROJECT_ROOT`` 584*61c4878aSAndroid Build Coastguard Worker The absolute path of the project using Pigweed's env setup. For Pigweed this 585*61c4878aSAndroid Build Coastguard Worker is the same as ``PW_ROOT``. This should be set by the project's bootstrap 586*61c4878aSAndroid Build Coastguard Worker script. 587*61c4878aSAndroid Build Coastguard Worker 588*61c4878aSAndroid Build Coastguard Worker``PW_ROOT`` 589*61c4878aSAndroid Build Coastguard Worker The absolute path to the Pigweed repository within ``PW_PROJECT_ROOT``. This 590*61c4878aSAndroid Build Coastguard Worker should be set by the project's bootstrap script. 591*61c4878aSAndroid Build Coastguard Worker 592*61c4878aSAndroid Build Coastguard WorkerOutput Variables 593*61c4878aSAndroid Build Coastguard Worker---------------- 594*61c4878aSAndroid Build Coastguard WorkerThe following environment variables are set by env setup. 595*61c4878aSAndroid Build Coastguard Worker 596*61c4878aSAndroid Build Coastguard Worker``PATH`` 597*61c4878aSAndroid Build Coastguard Worker System executable search path. Many of the environment variables below are 598*61c4878aSAndroid Build Coastguard Worker also added to this variable. 599*61c4878aSAndroid Build Coastguard Worker 600*61c4878aSAndroid Build Coastguard Worker``_PW_ACTUAL_ENVIRONMENT_ROOT`` 601*61c4878aSAndroid Build Coastguard Worker Location the environment was installed into. Separate from 602*61c4878aSAndroid Build Coastguard Worker ``PW_ENVIRONMENT_ROOT`` because setting that implicitly and switching to 603*61c4878aSAndroid Build Coastguard Worker another project directory causes unexpected behavior. 604*61c4878aSAndroid Build Coastguard Worker 605*61c4878aSAndroid Build Coastguard Worker``PW_CIPD_INSTALL_DIR`` 606*61c4878aSAndroid Build Coastguard Worker Top-level CIPD install directory. This is where the ``cipd`` executable is. 607*61c4878aSAndroid Build Coastguard Worker 608*61c4878aSAndroid Build Coastguard Worker``PW_*_CIPD_INSTALL_DIR`` 609*61c4878aSAndroid Build Coastguard Worker Each CIPD package file is installed into its own directory. This allows other 610*61c4878aSAndroid Build Coastguard Worker tools to determine what those directories are. The ``*`` is replaced with an 611*61c4878aSAndroid Build Coastguard Worker all-caps version of the basename of the package file, without the extension. 612*61c4878aSAndroid Build Coastguard Worker (E.g., "path/foo.json" becomes ``PW_FOO_CIPD_INSTALL_DIR``.) 613*61c4878aSAndroid Build Coastguard Worker 614*61c4878aSAndroid Build Coastguard Worker``PW_PACKAGE_ROOT`` 615*61c4878aSAndroid Build Coastguard Worker Location that packages installed by ``pw package`` will be installed to. 616*61c4878aSAndroid Build Coastguard Worker 617*61c4878aSAndroid Build Coastguard Worker``VIRTUAL_ENV`` 618*61c4878aSAndroid Build Coastguard Worker Path to Pigweed's virtualenv. 619*61c4878aSAndroid Build Coastguard Worker 620*61c4878aSAndroid Build Coastguard WorkerNon-Shell Environments 621*61c4878aSAndroid Build Coastguard Worker********************** 622*61c4878aSAndroid Build Coastguard WorkerIf using this outside of bash—for example directly from an IDE or CI 623*61c4878aSAndroid Build Coastguard Workersystem—users can process the ``actions.json`` file that's generated in the 624*61c4878aSAndroid Build Coastguard Workerlocation specified by the environment config. It lists variables to set, clear, 625*61c4878aSAndroid Build Coastguard Workerand modify. An example ``actions.json`` is shown below. The "append" and 626*61c4878aSAndroid Build Coastguard Worker"prepend" actions are listed in the order they should be applied, so the 627*61c4878aSAndroid Build Coastguard Worker``<pigweed-root>/out/host/host_tools`` entry should be at the beginning of 628*61c4878aSAndroid Build Coastguard Worker``PATH`` and not in the middle somewhere. 629*61c4878aSAndroid Build Coastguard Worker 630*61c4878aSAndroid Build Coastguard Worker.. code-block:: json 631*61c4878aSAndroid Build Coastguard Worker 632*61c4878aSAndroid Build Coastguard Worker { 633*61c4878aSAndroid Build Coastguard Worker "modify": { 634*61c4878aSAndroid Build Coastguard Worker "PATH": { 635*61c4878aSAndroid Build Coastguard Worker "append": [], 636*61c4878aSAndroid Build Coastguard Worker "prepend": [ 637*61c4878aSAndroid Build Coastguard Worker "<pigweed-root>/environment/cipd", 638*61c4878aSAndroid Build Coastguard Worker "<pigweed-root>/environment/cipd/pigweed", 639*61c4878aSAndroid Build Coastguard Worker "<pigweed-root>/environment/cipd/pigweed/bin", 640*61c4878aSAndroid Build Coastguard Worker "<pigweed-root>/environment/cipd/luci", 641*61c4878aSAndroid Build Coastguard Worker "<pigweed-root>/environment/cipd/luci/bin", 642*61c4878aSAndroid Build Coastguard Worker "<pigweed-root>/environment/pigweed-venv/bin", 643*61c4878aSAndroid Build Coastguard Worker "<pigweed-root>/out/host/host_tools" 644*61c4878aSAndroid Build Coastguard Worker ], 645*61c4878aSAndroid Build Coastguard Worker "remove": [] 646*61c4878aSAndroid Build Coastguard Worker } 647*61c4878aSAndroid Build Coastguard Worker }, 648*61c4878aSAndroid Build Coastguard Worker "set": { 649*61c4878aSAndroid Build Coastguard Worker "PW_PROJECT_ROOT": "<pigweed-root>", 650*61c4878aSAndroid Build Coastguard Worker "PW_ROOT": "<pigweed-root>", 651*61c4878aSAndroid Build Coastguard Worker "_PW_ACTUAL_ENVIRONMENT_ROOT": "<pigweed-root>/environment", 652*61c4878aSAndroid Build Coastguard Worker "PW_CIPD_INSTALL_DIR": "<pigweed-root>/environment/cipd", 653*61c4878aSAndroid Build Coastguard Worker "CIPD_CACHE_DIR": "<home>/.cipd-cache-dir", 654*61c4878aSAndroid Build Coastguard Worker "PW_PIGWEED_CIPD_INSTALL_DIR": "<pigweed-root>/environment/cipd/pigweed", 655*61c4878aSAndroid Build Coastguard Worker "PW_LUCI_CIPD_INSTALL_DIR": "<pigweed-root>/environment/cipd/luci", 656*61c4878aSAndroid Build Coastguard Worker "VIRTUAL_ENV": "<pigweed-root>/environment/pigweed-venv", 657*61c4878aSAndroid Build Coastguard Worker "PYTHONHOME": null, 658*61c4878aSAndroid Build Coastguard Worker "__PYVENV_LAUNCHER__": null 659*61c4878aSAndroid Build Coastguard Worker } 660*61c4878aSAndroid Build Coastguard Worker } 661*61c4878aSAndroid Build Coastguard Worker 662*61c4878aSAndroid Build Coastguard WorkerMany of these variables are directly exposed to the GN build as well, through 663*61c4878aSAndroid Build Coastguard Workerthe GNI file specified in the environment config file. 664*61c4878aSAndroid Build Coastguard Worker 665*61c4878aSAndroid Build Coastguard Worker.. code-block:: 666*61c4878aSAndroid Build Coastguard Worker 667*61c4878aSAndroid Build Coastguard Worker declare_args() { 668*61c4878aSAndroid Build Coastguard Worker pw_env_setup_CIPD_LUCI = "<environment-root>/cipd/packages/luci" 669*61c4878aSAndroid Build Coastguard Worker pw_env_setup_CIPD_PIGWEED = "<environment-root>/cipd/packages/pigweed" 670*61c4878aSAndroid Build Coastguard Worker pw_env_setup_PACKAGE_ROOT = "<environment-root>/packages" 671*61c4878aSAndroid Build Coastguard Worker pw_env_setup_VIRTUAL_ENV = "<environment-root>/pigweed-venv" 672*61c4878aSAndroid Build Coastguard Worker } 673*61c4878aSAndroid Build Coastguard Worker 674*61c4878aSAndroid Build Coastguard WorkerIt's straightforward to use these variables. 675*61c4878aSAndroid Build Coastguard Worker 676*61c4878aSAndroid Build Coastguard Worker.. code-block:: cpp 677*61c4878aSAndroid Build Coastguard Worker 678*61c4878aSAndroid Build Coastguard Worker import("//build_overrides/pigweed_environment.gni") 679*61c4878aSAndroid Build Coastguard Worker 680*61c4878aSAndroid Build Coastguard Worker deps = [ "$pw_env_setup_CIPD_PIGWEED/..." ] 681*61c4878aSAndroid Build Coastguard Worker 682*61c4878aSAndroid Build Coastguard WorkerImplementation 683*61c4878aSAndroid Build Coastguard Worker************** 684*61c4878aSAndroid Build Coastguard WorkerThe environment is set up by installing CIPD and Python packages in 685*61c4878aSAndroid Build Coastguard Worker``PW_ENVIRONMENT_ROOT`` or ``<checkout>/environment``, and saving modifications 686*61c4878aSAndroid Build Coastguard Workerto environment variables in setup scripts in those directories. To support 687*61c4878aSAndroid Build Coastguard Workermultiple operating systems this is done in an operating system-agnostic manner 688*61c4878aSAndroid Build Coastguard Workerand then written into operating system-specific files to be sourced now and in 689*61c4878aSAndroid Build Coastguard Workerthe future when running ``activate.sh`` instead of ``bootstrap.sh``. In the 690*61c4878aSAndroid Build Coastguard Workerfuture these could be extended to C shell and PowerShell. A logical mapping of 691*61c4878aSAndroid Build Coastguard Workerhigh-level commands to system-specific initialization files is shown below. 692*61c4878aSAndroid Build Coastguard Worker 693*61c4878aSAndroid Build Coastguard Worker.. grid:: 1 694*61c4878aSAndroid Build Coastguard Worker :padding: 0 695*61c4878aSAndroid Build Coastguard Worker 696*61c4878aSAndroid Build Coastguard Worker .. grid-item-card:: 697*61c4878aSAndroid Build Coastguard Worker :columns: 12 698*61c4878aSAndroid Build Coastguard Worker :class-header: font-monospace 699*61c4878aSAndroid Build Coastguard Worker 700*61c4878aSAndroid Build Coastguard Worker SET $PW_ROOT /home/$USER/pigweed 701*61c4878aSAndroid Build Coastguard Worker ^^^ 702*61c4878aSAndroid Build Coastguard Worker 703*61c4878aSAndroid Build Coastguard Worker .. grid:: 2 704*61c4878aSAndroid Build Coastguard Worker :margin: 0 705*61c4878aSAndroid Build Coastguard Worker :padding: 0 706*61c4878aSAndroid Build Coastguard Worker 707*61c4878aSAndroid Build Coastguard Worker .. grid-item:: **Windows** 708*61c4878aSAndroid Build Coastguard Worker 709*61c4878aSAndroid Build Coastguard Worker .. grid-item:: **Linux & Mac (sh-compatible shells)** 710*61c4878aSAndroid Build Coastguard Worker 711*61c4878aSAndroid Build Coastguard Worker .. grid:: 2 712*61c4878aSAndroid Build Coastguard Worker :margin: 0 713*61c4878aSAndroid Build Coastguard Worker :padding: 0 714*61c4878aSAndroid Build Coastguard Worker 715*61c4878aSAndroid Build Coastguard Worker .. grid-item:: 716*61c4878aSAndroid Build Coastguard Worker 717*61c4878aSAndroid Build Coastguard Worker .. code-block:: dosbatch 718*61c4878aSAndroid Build Coastguard Worker 719*61c4878aSAndroid Build Coastguard Worker set PW_ROOT /home/%USER%/pigweed 720*61c4878aSAndroid Build Coastguard Worker 721*61c4878aSAndroid Build Coastguard Worker .. grid-item:: 722*61c4878aSAndroid Build Coastguard Worker 723*61c4878aSAndroid Build Coastguard Worker .. code-block:: shell 724*61c4878aSAndroid Build Coastguard Worker 725*61c4878aSAndroid Build Coastguard Worker PW_ROOT="/home/$USER/pigweed" 726*61c4878aSAndroid Build Coastguard Worker export PW_ROOT 727*61c4878aSAndroid Build Coastguard Worker 728*61c4878aSAndroid Build Coastguard Worker.. grid:: 1 729*61c4878aSAndroid Build Coastguard Worker :padding: 0 730*61c4878aSAndroid Build Coastguard Worker 731*61c4878aSAndroid Build Coastguard Worker .. grid-item-card:: 732*61c4878aSAndroid Build Coastguard Worker :columns: 12 733*61c4878aSAndroid Build Coastguard Worker :class-header: font-monospace 734*61c4878aSAndroid Build Coastguard Worker 735*61c4878aSAndroid Build Coastguard Worker PREPEND $PATH $PW_ROOT/.env/bin 736*61c4878aSAndroid Build Coastguard Worker ^^^ 737*61c4878aSAndroid Build Coastguard Worker .. grid:: 2 738*61c4878aSAndroid Build Coastguard Worker :margin: 0 739*61c4878aSAndroid Build Coastguard Worker :padding: 0 740*61c4878aSAndroid Build Coastguard Worker 741*61c4878aSAndroid Build Coastguard Worker .. grid-item:: **Windows** 742*61c4878aSAndroid Build Coastguard Worker 743*61c4878aSAndroid Build Coastguard Worker .. grid-item:: **Linux & Mac (sh-compatible shells)** 744*61c4878aSAndroid Build Coastguard Worker 745*61c4878aSAndroid Build Coastguard Worker .. grid:: 2 746*61c4878aSAndroid Build Coastguard Worker :margin: 0 747*61c4878aSAndroid Build Coastguard Worker :padding: 0 748*61c4878aSAndroid Build Coastguard Worker 749*61c4878aSAndroid Build Coastguard Worker .. grid-item:: 750*61c4878aSAndroid Build Coastguard Worker 751*61c4878aSAndroid Build Coastguard Worker .. code-block:: dosbatch 752*61c4878aSAndroid Build Coastguard Worker 753*61c4878aSAndroid Build Coastguard Worker set PATH=%PW_ROOT%/.env/bin;%PATH% 754*61c4878aSAndroid Build Coastguard Worker 755*61c4878aSAndroid Build Coastguard Worker .. grid-item:: 756*61c4878aSAndroid Build Coastguard Worker 757*61c4878aSAndroid Build Coastguard Worker .. code-block:: shell 758*61c4878aSAndroid Build Coastguard Worker 759*61c4878aSAndroid Build Coastguard Worker PATH="$(\ 760*61c4878aSAndroid Build Coastguard Worker echo "$PATH" | \ 761*61c4878aSAndroid Build Coastguard Worker sed "s|:$PW_ROOT/.env/bin:|:|g;" | \ 762*61c4878aSAndroid Build Coastguard Worker sed "s|^$PW_ROOT/.env/bin:||g;" | \ 763*61c4878aSAndroid Build Coastguard Worker sed "s|:$PW_ROOT/.env/bin$||g;")" 764*61c4878aSAndroid Build Coastguard Worker PATH="$PW_ROOT/.env/bin;$PATH" 765*61c4878aSAndroid Build Coastguard Worker export PATH 766*61c4878aSAndroid Build Coastguard Worker 767*61c4878aSAndroid Build Coastguard Worker.. grid:: 1 768*61c4878aSAndroid Build Coastguard Worker :padding: 0 769*61c4878aSAndroid Build Coastguard Worker 770*61c4878aSAndroid Build Coastguard Worker .. grid-item-card:: 771*61c4878aSAndroid Build Coastguard Worker :columns: 12 772*61c4878aSAndroid Build Coastguard Worker :class-header: font-monospace 773*61c4878aSAndroid Build Coastguard Worker 774*61c4878aSAndroid Build Coastguard Worker ECHO "Setup Complete!" 775*61c4878aSAndroid Build Coastguard Worker ^^^ 776*61c4878aSAndroid Build Coastguard Worker 777*61c4878aSAndroid Build Coastguard Worker .. grid:: 2 778*61c4878aSAndroid Build Coastguard Worker :margin: 0 779*61c4878aSAndroid Build Coastguard Worker :padding: 0 780*61c4878aSAndroid Build Coastguard Worker 781*61c4878aSAndroid Build Coastguard Worker .. grid-item:: **Windows** 782*61c4878aSAndroid Build Coastguard Worker 783*61c4878aSAndroid Build Coastguard Worker .. grid-item:: **Linux & Mac (sh-compatible shells)** 784*61c4878aSAndroid Build Coastguard Worker 785*61c4878aSAndroid Build Coastguard Worker 786*61c4878aSAndroid Build Coastguard Worker .. grid:: 2 787*61c4878aSAndroid Build Coastguard Worker :margin: 0 788*61c4878aSAndroid Build Coastguard Worker :padding: 0 789*61c4878aSAndroid Build Coastguard Worker 790*61c4878aSAndroid Build Coastguard Worker .. grid-item:: 791*61c4878aSAndroid Build Coastguard Worker 792*61c4878aSAndroid Build Coastguard Worker .. code-block:: dosbatch 793*61c4878aSAndroid Build Coastguard Worker 794*61c4878aSAndroid Build Coastguard Worker echo Setup Complete! 795*61c4878aSAndroid Build Coastguard Worker 796*61c4878aSAndroid Build Coastguard Worker .. grid-item:: 797*61c4878aSAndroid Build Coastguard Worker 798*61c4878aSAndroid Build Coastguard Worker .. code-block:: shell 799*61c4878aSAndroid Build Coastguard Worker 800*61c4878aSAndroid Build Coastguard Worker echo "Setup Complete!" 801*61c4878aSAndroid Build Coastguard Worker 802*61c4878aSAndroid Build Coastguard Worker 803*61c4878aSAndroid Build Coastguard Worker.. _Requirements Files documentation: https://pip.pypa.io/en/stable/user_guide/#requirements-files 804*61c4878aSAndroid Build Coastguard Worker.. _Constraints Files documentation: https://pip.pypa.io/en/stable/user_guide/#constraints-files 805