Lines Matching +full:build +full:- +full:packages
1 .. _docs-python-build:
4 Pigweed's GN Python Build
8 - :bdg-ref-primary-line:`module-pw_build-python` for detailed template usage.
9 - :bdg-ref-primary-line:`module-pw_build` for other GN templates available
11 - :bdg-ref-primary-line:`docs-build-system` for a high level guide and
12 background information on Pigweed's build system as a whole.
14 Pigweed uses a custom GN-based build system to manage its Python code. The
15 Pigweed Python build supports packaging, installation and distribution of
16 interdependent local Python packages. It also provides for fast, incremental
18 with :ref:`module-pw_watch`) or in continuous integration.
20 Pigweed's Python code is exclusively managed by GN, but the GN-based build may
21 be used alongside CMake, Bazel, or any other build system. Pigweed's environment
23 build system. As needed, non-GN projects can declare just their Python packages
28 In addition to compiler commands a Pigweed GN build will execute Python scripts
31 :ref:`module-pw_build-pw_python_action` GN template which will ultimately run
32 ``python``. Running Python on it's own by default will make any Python packages
34 lead to flaky builds when different packages are installed on each developer
37 set of Python packages separate from the host system.
39 When a Pigweed GN build starts a single venv is created for use by all
40 :ref:`pw_python_actions <module-pw_build-pw_python_action>` throughout the build
41 graph. Once created, all required third-party Python packages needed for the
43 the venv. Of course if a new third-party package dependency is added it will be
45 with the :ref:`module-pw_build-pw_python_venv` template if desired, but only one
56 out[GN Build Dir<br/>fa:fa-folder out]
58 out -->|ninja -C out| createvenvs
61 createvenvs --> pyactions1
62 createvenvs --> pyactions2
66 venv1(fa:fa-folder out/python-venv  )
69 venv1 --> a1
70 venv1 --> a2
75 venv2(fa:fa-folder out/another-venv  )
78 venv2 --> a3
79 venv2 --> a4
85 … <https://cs.opensource.google/pigweed/pigweed/+/main:pw_env_setup/BUILD.gn?q=pigweed_build_venv>`_
86 if a project does not specify it's own build venv. See
87 :bdg-ref-primary-line:`docs-python-build-python-gn-venv` on how to define
90 Having a static venv containing only third-party dependencies opens the flood
91 gates for python scripts to run. If the venv only contains third-party
92 dependencies you may be wondering how you can import your own in-tree Python
93 packages. Python code run in the build may still import any in-tree Python
94 packages created with :ref:`module-pw_build-pw_python_package`
97 :ref:`module-pw_build-pw_python_action`
99 <https://docs.python.org/3/using/cmdline.html#envvar-PYTHONPATH>`_ so that given
100 package can be imported. This has the benefit of the build failing if a
106 - Using venvs to execute Python in GN provides reproducible builds with fixed
107 third-party dependencies.
108 - Using ``PYTHONPATH`` coupled with ``python_deps`` to import in-tree Python
109 packages enforces dependency correctness.
115 .. _docs-python-build-python-gn-venv:
117 Build Time Python Virtualenv
118 ----------------------------
119 Pigweed's GN Python build infrastructure relies on `Python virtual environments
123 :ref:`module-pw_build-pw_python_action` targets are executed.
128 The default build venv is specified via a GN arg and is best set in the root
129 ``.gn`` or ``BUILD.gn`` file. For example:
131 .. code-block::
136 Additional :ref:`module-pw_build-pw_python_venv` targets can be created as
137 needed. The :ref:`module-pw_build-pw_python_action` template can take an
142 .. _docs-python-build-python-gn-requirements-files:
144 Third-party Python Requirements and Constraints
145 -----------------------------------------------
147 the bootstrapped environment and in the GN build venv. There are two main ways
153 :ref:`module-pw_build-pw_python_package` template. This is the best option
154 if your in-tree Python package requires an external Python package.
160 packages from pypi.org, the local file system and git repos. See `pip's
162 <https://pip.pypa.io/en/stable/user_guide/#requirements-files>`_ for more
165 The GN arg can be set in your project's root ``.gn`` or ``BUILD.gn`` file.
167 .. code-block::
174 See the :ref:`docs-python-build-python-gn-structure` section below for a full
181 <https://pip.pypa.io/en/stable/user_guide/#constraints-files>`_. Constraints
182 control which versions of packages get installed by ``pip`` if that package is
187 .. code-block::
194 In-tree ``pw_python_package`` Requirements
195 ------------------------------------------
202 To ensure the requirements of in-tree :ref:`module-pw_build-pw_python_package`
203 targets are installed :ref:`module-pw_build-pw_python_venv` introduces the
204 ``source_packages`` argument. This is a list of in-tree ``pw_python_package``
208 all in-tree packages and any in-tree transitive dependencies is then written to
214 two packages each depend on a few other ``pw_python_package`` targets. The
216 adds ``-c`` lines for constraint files.
220 <https://pip.pypa.io/en/stable/reference/requirements-file-format/#requirements-file-format>`_
222 .. literalinclude:: pw_build/py/gn_tests/BUILD.gn
223 :start-after: [downstream-project-venv]
224 :end-before: [downstream-project-venv]
226 .. code-block::
230 # Auto-generated requirements.txt from the following packages:
244 -c ../../../../../../../pw_env_setup/py/pw_env_setup/virtualenv_setup/constraint.list
247 build>=0.8.0
254 pip-tools>=6.12.3
255 prompt-toolkit>=3.0.26
266 types-pygments
267 types-pyserial>=3.5,<4.0
268 types-pyyaml
269 types-setuptools
270 types-six
278 ``pip-compile`` command from `the pip-tools package
279 <https://pypi.org/project/pip-tools>`_ to fully expand and pin each package with
287 by the :ref:`module-pw_build-pw_python_zip_with_setup` template when
288 producing a self contained zip of in-tree and third party Python packages.
291 :ref:`module-pw_build-pw_python_venv` target:
294 .. code-block::
299 # This file is autogenerated by pip-compile with Python 3.11
302 # pip-compile --allow-unsafe --generate-hashes
303 …# --output-file=python/gen/pw_build/py/gn_tests/downstream_tools_build_venv/compiled_requirem…
304 # --resolver=backtracking
308 --hash=sha256:7d5d0167b2b1ba821647616af46a749d1c653740dd0d2415100fe26e27afdf41 \
309 --hash=sha256:a841dacd6b99318a741b166adb07e19ee71a274450e68237b4650ca1055ab128
311 …# -c python/gen/pw_build/py/gn_tests/downstream_tools_build_venv/../../../../../../../pw_env_set…
314 --hash=sha256:0e0e3709d64fbffd3037e4ff403580550f14471fd3eaae9fa11cc9a5c7901153 \
315 --hash=sha256:a3cf9f02c53dd259144a7e8f3ccd75d67c9a8c716ef183e0c1f291bc5d7bb3cf
317 …# -c python/gen/pw_build/py/gn_tests/downstream_tools_build_venv/../../../../../../../pw_env_set…
322 ``pip_generate_hashes`` arg to the :ref:`module-pw_build-pw_python_venv`
325 Caching Python Packages for Offline Installation
326 ------------------------------------------------
328 .. _docs-python-build-downloading-packages:
330 Downloading Packages
332 The :ref:`module-pw_build-pw_python_venv` target adds an optional sub target
333 that will download all Python packages from remote servers into a local
337 example again let's build a local cache. To run the download target append
342 To build that one gn target with ninja, pass the output name from gn as a target
345 .. code-block:: bash
348 ninja -C out \
349 $(gn ls out --as=output \
352 This creates a ``wheels`` folder with all downloaded packages and a
355 .. code-block::
356 :caption: :octicon:`file-directory;1em` Vendor wheels output directory
357 :name: vendor-wheel-output
362 ├── appdirs-1.4.4-py2.py3-none-any.whl
363 ├── astroid-2.14.2-py3-none-any.whl
364 ├── backcall-0.2.0-py2.py3-none-any.whl
365 ├── black-23.1.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
367 …├── websockets-10.4-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manyl…
368 ├── wheel-0.40.0-py3-none-any.whl
369 ├── wrapt-1.14.1.tar.gz
370 └── yapf-0.31.0-py2.py3-none-any.whl
373 ``.tar.gz`` files. The ``.whl`` may contain Python packages with precompiled C
375 ``cp311-cp311-manylinux_2_17_x86_64.whl``. These binary packages are selected by
380 If you need to cache Python packages for multiple platforms the
386 - cp311, manylinux_2_17_x86_64
387 - cp311, manylinux2014_x86_64
388 - cp311, macosx_11_0_arm64
389 - cp311, macosx_10_9_x86_64
390 - cp311, win_amd64
391 - cp311, win32
399 ``.vendor_wheels`` can attempt to download binary packages for multiple
402 .. code-block::
412 :start-after: [wheel-platform-args]
413 :end-before: [wheel-platform-args]
416 The set of Python packages that will be downloaded is determined by the
418 current host OS and Python version. `pip-tools
419 <https://pypi.org/project/pip-tools>`_ does not expand requirements for
421 …<https://setuptools.pypa.io/en/latest/userguide/dependency_management.html#platform-specific-depen…
424 .. code-block::
429 If pip-tools is run on Linux then the above packages will not appear in
433 .. _docs-python-build-installing-offline:
445 .. code-block::
447 # Adds --no-index forcing pip to not reach out to the internet (pypi.org) to
448 # download packages. Using this option requires setting
456 "//environment/cipd/packages/python_packages/universal",
457 "//environment/cipd/packages/python_packages/linux/cp311",
460 # Optional: Adds '--no-cache-dir' forcing pip to ignore any previously cached
461 # Python packages. On most systems this is located in ~/.cache/pip/
468 .. code-block::
470 :name: pip-conf-file
473 # Disable searching pypi.org for packages
474 no-index = True
475 # Find packages in these directories:
476 find-links =
480 This tells pip to not search pypi.org for packages and only look in
488 .. code-block:: bash
499 .. _docs-python-build-python-gn-structure:
503 Here is a full example of what is required to build Python packages using
504 Pigweed's GN build system. A brief file hierarchy is shown here with file
505 content following. See also :ref:`docs-python-build-structure` below for details
506 on the structure of Python packages.
508 .. code-block::
509 :caption: :octicon:`file-directory;1em` Top level GN file hierarchy
510 :name: gn-python-file-tree
517 ├── BUILD.gn
520 │ ├── BUILD.gn
540 - :octicon:`file-directory;1em` project_root/
542 - :octicon:`file;1em` .gn
544 .. code-block::
563 # Default gn build virtualenv target.
570 performed during the build.
573 :start-after: [default-pip-gn-args]
574 :end-before: [default-pip-gn-args]
576 - :octicon:`file;1em` BUILDCONFIG.gn
578 .. code-block::
586 - :octicon:`file-directory;1em` build_overrides / :octicon:`file;1em` pigweed.gni
588 .. code-block::
598 - :octicon:`file;1em` BUILD.gn
600 .. code-block::
609 # Lists all the targets build by default with e.g. `ninja -C out`.
626 # In-tree Python packages
631 # Pigweed Python packages to include
642 # Set this gn arg in a declare_args block in this file 'BUILD.gn' or in '.gn' to
648 path = "$root_build_dir/python-venv"
653 # This works by checking the setup.cfg files for all packages listed here and
654 # installing the packages listed in the [options].install_requires field.
658 # This template collects all python packages and their dependencies into a
662 packages = _all_python_packages
664 name = "project-tools"
671 # Install the project-tools super Python package into the bootstrapped
674 packages = [ ":generate_project_python_distribution" ]
677 .. _docs-python-build-structure:
681 Pigweed Python code is structured into standard Python packages. This makes it
682 simple to package and distribute Pigweed Python packages with common Python
685 Like all Pigweed source code, Python packages are organized into Pigweed
687 :ref:`Pigweed Module Stucture <docs-module-structure>`).
689 .. code-block::
690 :caption: :octicon:`file-directory;1em` Example layout of a Pigweed Python package.
691 :name: python-file-tree
695 │ ├── BUILD.gn
709 The ``BUILD.gn`` declares this package in GN. For upstream Pigweed, a presubmit
710 check in ensures that all Python files are listed in a ``BUILD.gn``.
712 Pigweed prefers to define Python packages using ``setup.cfg`` files. In the
716 .. code-block::
718 :name: pyproject-toml-stub
720 [build-system]
722 build-backend = 'setuptools.build_meta'
724 Each ``pyproject.toml`` file is required to specify which build system should be
730 - ``setup.cfg`` examples at `Configuring setup() using setup.cfg files`_
731 - ``pyproject.toml`` background at `Build System Support - How to use it?`_
733 .. _module-pw_build-python-target:
736 -------------------------
737 The key abstraction in the Python build is the ``pw_python_package``.
740 in :ref:`module-pw_build-python`.
744 - a ``setup.cfg`` and ``pyproject.toml`` file,
745 - source files,
746 - test files,
747 - dependencies on other ``pw_python_package`` targets.
750 subtarget represents different functionality in the Python build.
752 - ``<name>`` - Represents the Python files in the build, but does not take any
754 - ``<name>.tests`` - Runs all tests for this package.
756 - ``<name>.tests.<test_file>`` - Runs the specified test.
758 - ``<name>.lint`` - Runs static analysis tools on the Python code. This is a
761 - ``<name>.lint.mypy`` - Runs Mypy on all Python files, if enabled.
762 - ``<name>.lint.pylint`` - Runs Pylint on all Python files, if enabled.
763 - ``<name>.lint.ruff`` - Runs ruff on all Python files, if enabled.
765 - ``<name>.install`` - Installs the package in a Python virtual environment.
766 - ``<name>.wheel`` - Builds a Python wheel for this package.
775 Python package. The build will run it when the test, the package, or one of its
782 and ``ruff_toml`` files to use on a per-package basis. The configuration files
787 Packages may opt out of static analysis as necessary by setting
795 :start-after: [python-static-analysis-tools]
796 :end-before: [python-static-analysis-tools]
803 :start-after: [default-mypy-args]
804 :end-before: [default-mypy-args]
809 distributing Python packages. The Pigweed Python build supports creating wheels
810 for individual packages and groups of packages. Building the ``.wheel``
811 subtarget creates a ``.whl`` file for the package using the PyPA's `build
812 <https://pypa-build.readthedocs.io/en/stable/>`_ tool.
815 :ref:`module-pw_build-pw_python_distribution` records the location of the
820 :ref:`module-pw_build-python-dist`.
824 The Pigweed GN build supports protocol buffers with the ``pw_proto_library``
825 target (see :ref:`module-pw_protobuf_compiler`). Python protobuf modules are
826 generated as standalone Python packages by default. Protocol buffers may also be
827 nested within existing Python packages. In this case, the Python package in the
833 The ``pw_python_package`` target in the ``BUILD.gn`` duplicates much of the
839 Pigweed packages containing protobufs are generated in full or in part. These
840 packages may use generated setup files, since they are always packaged or
841 installed from the build output directory.
848 ----------
852 Build systems automate these auxiliary tasks of software development, making it
853 possible to build larger, more complex systems quickly and robustly.
855 Python is an interpreted language, but it shares most build automation concerns
860 ------------------
861 The Python programming langauge does not have an official build automation
862 system. However, there are numerous Python-focused build automation tools with
870 but are not intended for general build automation. Tools like `PyBuilder
872 provide more general build automation for Python.
874 The `Bazel <http://bazel.build/>`_ build system has first class support for
878 ----------
880 multi-language, modular project. It serves both as a library or middleware and
883 This section describes Python build automation challenges encountered by
890 <https://pypi.org/>`_ packages.
892 The basic Python packaging tools lack dependency tracking for local packages.
894 ``pip`` is not aware of local packages until they are installed. Packages must
906 by contributing to the long-term resilience of a codebase. Despite their
909 bug-prone codebases.
924 :bdg-ref-primary-line:`docs-automated-analysis` for info on other static
928 are `Pylint <https://www.pylint.org/>`_ and `Mypy <http://mypy-lang.org/>`_.
937 These tools do not have built-in support for incremental runs or dependency
948 `Protocol buffers <https://developers.google.com/protocol-buffers>`_ are an
956 protobufs with existing packages awkward.
959 ------------
961 flexible development experience for its customers. Pigweed's high-level goals
963 Python build.
965 - Integrate seamlessly with the other Pigweed build tools.
966 - Easy to use independently, even if primarily using a different build system.
967 - Support standard packaging and distribution with setuptools, wheel, and pip.
968 - Correctly manage interdependent local Python packages.
969 - Out-of-the-box support for writing and running tests.
970 - Preconfigured, trivial-to-run static analysis integration for Pylint and Mypy.
971 - Fast, dependency-aware incremental rebuilds and test execution, suitable for
972 use with :ref:`module-pw_watch`.
973 - Seamless protocol buffer support.
976 ---------------
978 is more limited in a multi-language project like Pigweed. The cost of bringing
979 up and maintaining an additional build automation system for a single language
982 Pigweed uses GN as its primary build system for all languages. While GN does not
985 GN has strong multi-toolchain and multi-language capabilities. In GN, it is
988 declaration. When using GN for multiple languages, Ninja schedules build steps
989 for all languages together, resulting in faster total build times.
991 Not all Pigweed users build with GN. Of Pigweed's three supported build systems,
996 Given these considerations, GN is an ideal choice for Pigweed's Python build.
999 …ild System Support - How to use it?: https://setuptools.readthedocs.io/en/latest/build_meta.html?h…