1.. _seed-0101: 2 3================== 40101: pigweed.json 5================== 6.. seed:: 7 :number: 0101 8 :name: pigweed.json 9 :status: Accepted 10 :proposal_date: 2023-02-06 11 :cl: 128010 12 :authors: Rob Mohr 13 :facilitator: Ted Pudlik 14 15------- 16Summary 17------- 18Combine several of the configuration options downstream projects use to 19configure parts of Pigweed in one place, and use this place for further 20configuration options. 21 22---------- 23Motivation 24---------- 25Pigweed-based projects configure Pigweed and themselves in a variety of ways. 26The environment setup is controlled by a JSON file that's referenced in 27``bootstrap.sh`` files and in internal infrastructure repos that looks 28something like this: 29 30.. code-block:: 31 32 { 33 "root_variable": "<PROJNAME>_ROOT", 34 "cipd_package_files": ["tools/default.json"], 35 "virtualenv": { 36 "gn_args": ["dir_pw_third_party_stm32cube=\"\""], 37 "gn_root": ".", 38 "gn_targets": [":python.install"] 39 }, 40 "optional_submodules": ["vendor/shhh-secret"], 41 "gni_file": "build_overrides/pigweed_environment.gni" 42 } 43 44The plugins to the ``pw`` command-line utility are configured in ``PW_PLUGINS``, 45which looks like this: 46 47.. code-block:: 48 49 # <name> <Python module> <function> 50 console pw_console.__main__ main 51 format pw_presubmit.format_code _pigweed_upstream_main 52 53In addition, changes have been proposed to configure some of the behavior of 54``pw format`` and the formatting steps of ``pw presubmit`` from config files, 55but there's no standard place to put these configuration options. 56 57--------------- 58Guide reference 59--------------- 60This proposal affects two sets of people: developers looking to use Pigweed, 61and developers looking to add configurable features to Pigweed. 62 63Developers looking to use Pigweed will have one config file that contains all 64the options they need to set. Documentation for individual Pigweed modules will 65show only the configuration options relevant for that module, and multiple of 66these examples can simply be concatenated to form a valid config file. 67 68Developers looking to add configurable features to Pigweed no longer need to 69define a new file format, figure out where to find it in the tree (or how to 70have Pigweed-projects specify a location), or parse this format. 71 72--------------------- 73Problem investigation 74--------------------- 75There are multiple issues with the current system that need to be addressed. 76 77* ``PW_PLUGINS`` works, but is a narrow custom format with exactly one purpose. 78* The environment config file is somewhat extensible, but is still specific to 79 environment setup. 80* There's no accepted place for other modules to retrieve configuration options. 81 82These should be combined into a single file. There are several formats that 83could be selected, and many more arguments for and against each. Only a subset 84of these arguments are reproduced here. 85 86* JSON does not support comments 87* JSON5 is not supported in the Python standard library 88* XML is too verbose 89* YAML is acceptable, but implicit type conversion could be a problem, and it's 90 not supported in the Python standard library 91* TOML is acceptable, and `was selected for a similar purpose by Python 92 <https://snarky.ca/what-the-heck-is-pyproject-toml/>`_, but it's 93 not supported in the Python standard library before Python v3.11 94* Protobuf Text Format is acceptable and widely used within Google, but is not 95 supported in the Python standard library 96 97The location of the file is also an issue. Environment config files can be found 98in a variety of locations depending on the project—all of the following paths 99are used by at least one internal Pigweed-based project. 100 101* ``build/environment.json`` 102* ``build/pigweed/env_setup.json`` 103* ``environment.json`` 104* ``env_setup.json`` 105* ``pw_env_setup.json`` 106* ``scripts/environment.json`` 107* ``tools/environment.json`` 108* ``tools/env_setup.json`` 109 110``PW_PLUGINS`` files can in theory be in any directory and ``pw`` will search up 111for them from the current directory, but in practice they only exist at the root 112of checkouts. Having this file in a fixed location with a fixed name makes it 113significantly easier to find as a user, and the fixed name (if not path) makes 114it easy to find programmatically too. 115 116--------------- 117Detailed design 118--------------- 119The ``pw_env_setup`` Python module will provide an API to retrieve a parsed 120``pigweed.json`` file from the root of the checkout. ``pw_env_setup`` is the 121correct location because it can't depend on anything else, but other modules can 122depend on it. Code in other languages does not yet depend on configuration 123files. 124 125A ``pigweed.json`` file might look like the following. Individual option names 126and structures are not final but will evolve as those options are 127implemented—this is merely an example of what an actual file could look like. 128The ``pw`` namespace is reserved for Pigweed, but other projects can use other 129namespaces for their own needs. Within the ``pw`` namespace all options are 130first grouped by their module name, which simplifies searching for the code and 131documentation related to the option in question. 132 133.. code-block:: 134 135 { 136 "pw": { 137 "pw_cli": { 138 "plugins": { 139 "console": { 140 "module": "pw_console.__main__", 141 "function": "main" 142 }, 143 "format": { 144 "module": "pw_presubmit.format_code", 145 "function": "_pigweed_upstream_main" 146 } 147 } 148 }, 149 "pw_env_setup": { 150 "root_variable": "<PROJNAME>_ROOT", 151 "rosetta": "allow", 152 "gni_file": "build_overrides/pigweed_environment.gni", 153 "cipd": { 154 "package_files": [ 155 "tools/default.json" 156 ] 157 }, 158 "virtualenv": { 159 "gn_args": [ 160 "dir_pw_third_party_stm32cube=\"\"" 161 ], 162 "gn_targets": [ 163 "python.install" 164 ], 165 "gn_root": "." 166 }, 167 "submodules": { 168 "optional": [ 169 "vendor/shhh-secret" 170 ] 171 } 172 }, 173 "pw_presubmit": { 174 "format": { 175 "python": { 176 "formatter": "black", 177 "black_path": "pyink" 178 } 179 } 180 } 181 } 182 } 183 184Some teams will resist a new file at the root of their checkout, but this seed 185won't be adding any files, it'll be combining at least one top-level file, maybe 186two, into a new top-level file, so there won't be any additional files in the 187checkout root. 188 189------------ 190Alternatives 191------------ 192``pw format`` and the formatting steps of ``pw presubmit`` could read from yet 193another config file, further fracturing Pigweed's configuration. 194 195A different file format could be chosen over JSON. Since JSON is parsed into 196only Python lists, dicts, and primitives, switching to another format that can 197be parsed into the same internal structure should be trivial. 198 199-------------- 200Open questions 201-------------- 202None? 203