1.. _module-pw_presubmit: 2 3============ 4pw_presubmit 5============ 6.. pigweed-module:: 7 :name: pw_presubmit 8 9The presubmit module provides Python tools for running presubmit checks and 10checking and fixing code format. It also includes the presubmit check script for 11the Pigweed repository, ``pigweed_presubmit.py``. 12 13Presubmit checks are essential tools, but they take work to set up, and 14projects don’t always get around to it. The ``pw_presubmit`` module provides 15tools for setting up high quality presubmit checks for any project. We use this 16framework to run Pigweed’s presubmit on our workstations and in our automated 17building tools. 18 19The ``pw_presubmit`` module also includes ``pw format``, a tool that provides a 20unified interface for automatically formatting code in a variety of languages. 21With ``pw format``, you can format Bazel, C, C++, Python, GN, and Go code 22according to configurations defined by your project. ``pw format`` leverages 23existing tools like ``clang-format``, and it’s simple to add support for new 24languages. (Note: Bazel formatting requires ``buildifier`` to be present on your 25system. If it's not Bazel formatting passes without checking.) 26 27.. image:: docs/pw_presubmit_demo.gif 28 :alt: ``pw format`` demo 29 :align: left 30 31The ``pw_presubmit`` package includes presubmit checks that can be used with any 32project. These checks include: 33 34.. todo-check: disable 35 36* Check code format of several languages including C, C++, and Python 37* Initialize a Python environment 38* Run all Python tests 39* Run pylint 40* Run mypy 41* Ensure source files are included in the GN and Bazel builds 42* Build and run all tests with GN 43* Build and run all tests with Bazel 44* Ensure all header files contain ``#pragma once`` (or, that they have matching 45 ``#ifndef``/``#define`` lines) 46* Ensure lists are kept in alphabetical order 47* Forbid non-inclusive language 48* Check format of TODO lines 49* Apply various rules to ``.gitmodules`` or ``OWNERS`` files 50* Ensure all source files are in the build 51 52.. todo-check: enable 53 54------------- 55Compatibility 56------------- 57Python 3 58 59------------------------------------------- 60Creating a presubmit check for your project 61------------------------------------------- 62Creating a presubmit check for a project using ``pw_presubmit`` is simple, but 63requires some customization. Projects must define their own presubmit check 64Python script that uses the ``pw_presubmit`` package. 65 66A project's presubmit script can be registered as a 67:ref:`pw_cli <module-pw_cli>` plugin, so that it can be run as ``pw 68presubmit``. 69 70Setting up the command-line interface 71===================================== 72The ``pw_presubmit.cli`` module sets up the command-line interface for a 73presubmit script. This defines a standard set of arguments for invoking 74presubmit checks. Its use is optional, but recommended. 75 76Common ``pw presubmit`` command line arguments 77---------------------------------------------- 78.. argparse:: 79 :module: pw_presubmit.cli 80 :func: _get_default_parser 81 :prog: pw presubmit 82 :nodefaultconst: 83 :nodescription: 84 :noepilog: 85 86 87``pw_presubmit.cli`` Python API 88------------------------------- 89.. automodule:: pw_presubmit.cli 90 :members: add_arguments, run 91 92 93Presubmit output directory 94-------------------------- 95The ``pw_presubmit`` command line interface includes an ``--output-directory`` 96option that specifies the working directory to use for presubmits. The default 97path is ``out/presubmit``. A subdirectory is created for each presubmit step. 98This directory persists between presubmit runs and can be cleaned by deleting it 99or running ``pw presubmit --clean``. 100 101.. _module-pw_presubmit-presubmit-checks: 102 103Presubmit checks 104================ 105A presubmit check is defined as a function or other callable. The function must 106accept one argument: a ``PresubmitContext``, which provides the paths on which 107to run. Presubmit checks communicate failure by raising an exception. 108 109Presubmit checks may use the ``filter_paths`` decorator to automatically filter 110the paths list for file types they care about. 111 112Either of these functions could be used as presubmit checks: 113 114.. code-block:: python 115 116 @pw_presubmit.filter_paths(endswith='.py') 117 def file_contains_ni(ctx: PresubmitContext): 118 for path in ctx.paths: 119 with open(path) as file: 120 contents = file.read() 121 if 'ni' not in contents and 'nee' not in contents: 122 raise PresumitFailure('Files must say "ni"!', path=path) 123 124 def run_the_build(_): 125 subprocess.run(['make', 'release'], check=True) 126 127Presubmit checks functions are grouped into "programs" -- a named series of 128checks. Projects may find it helpful to have programs for different purposes, 129such as a quick program for local use and a full program for automated use. The 130:ref:`example script <example-script>` uses ``pw_presubmit.Programs`` to define 131``quick`` and ``full`` programs. 132 133By default, presubmit steps are only run on files changed since ``@{upstream}``. 134If all such files are filtered out by ``filter_paths``, then that step will be 135skipped. This can be overridden with the ``--base`` and ``--full`` arguments to 136``pw presubmit``. In automated testing ``--full`` is recommended, except for 137lint/format checks where ``--base HEAD~1`` is recommended. 138 139.. autoclass:: pw_presubmit.presubmit_context.PresubmitContext 140 :members: 141 :noindex: 142 143Additional members can be added by subclassing ``PresubmitContext`` and 144``Presubmit``. Then override ``Presubmit._create_presubmit_context()`` to 145return the subclass of ``PresubmitContext``. Finally, add 146``presubmit_class=PresubmitSubClass`` when calling ``cli.run()``. 147 148.. autoclass:: pw_presubmit.presubmit_context.LuciContext 149 :members: 150 :noindex: 151 152.. autoclass:: pw_presubmit.presubmit_context.LuciPipeline 153 :members: 154 :noindex: 155 156.. autoclass:: pw_presubmit.presubmit_context.LuciTrigger 157 :members: 158 :noindex: 159 160Substeps 161-------- 162Presubmit steps can define substeps that can run independently in other tooling. 163These steps should subclass ``SubStepCheck`` and must define a ``substeps()`` 164method that yields ``SubStep`` objects. ``SubStep`` objects have the following 165members: 166 167* ``name``: Name of the substep 168* ``_func``: Substep code 169* ``args``: Positional arguments for ``_func`` 170* ``kwargs``: Keyword arguments for ``_func`` 171 172``SubStep`` objects must have unique names. For a detailed example of a 173``SubStepCheck`` subclass see ``GnGenNinja`` in ``build.py``. 174 175Existing Presubmit Checks 176------------------------- 177A small number of presubmit checks are made available through ``pw_presubmit`` 178modules. 179 180Code Formatting 181^^^^^^^^^^^^^^^ 182Formatting checks for a variety of languages are available from 183``pw_presubmit.format_code``. These include C/C++, Java, Go, Python, GN, and 184others. All of these checks can be included by adding 185``pw_presubmit.format_code.presubmit_checks()`` to a presubmit program. These 186all use language-specific formatters like clang-format or black. 187 188Example changes demonstrating how to add formatters: 189 190* `CSS <https://pigweed-review.googlesource.com/c/pigweed/pigweed/+/178810>`_ 191* `JSON <https://pigweed-review.googlesource.com/c/pigweed/pigweed/+/171991>`_ 192* `reStructuredText <https://pigweed-review.googlesource.com/c/pigweed/pigweed/+/168541>`_ 193* `TypeScript <https://pigweed-review.googlesource.com/c/pigweed/pigweed/+/164825>`_ 194 195These will suggest fixes using ``pw format --fix``. 196 197Options for code formatting can be specified in the ``pigweed.json`` file 198(see also :ref:`SEED-0101 <seed-0101>`). These apply to both ``pw presubmit`` 199steps that check code formatting and ``pw format`` commands that either check 200or fix code formatting. 201 202* ``python_formatter``: Choice of Python formatter. Options are ``black`` 203 (default, used by Pigweed itself) and ``yapf``. 204* ``black_path``: If ``python_formatter`` is ``black``, use this as the 205 executable instead of ``black``. 206* ``black_config_file``: Set the config file for the black formatter. 207* ``exclude``: List of path regular expressions to ignore. Will be evaluated 208 against paths relative to the checkout root using ``re.search``. 209 210Example section from a ``pigweed.json`` file: 211 212.. code-block:: json 213 214 { 215 "pw": { 216 "pw_presubmit": { 217 "format": { 218 "python_formatter": "black", 219 "black_config_file": "$pw_env{PW_PROJECT_ROOT}/config/.black.toml" 220 "black_path": "black", 221 "exclude": [ 222 "\\bthird_party/foo/src" 223 ] 224 } 225 } 226 } 227 } 228 229Sorted Blocks 230^^^^^^^^^^^^^ 231Blocks of code can be required to be kept in sorted order using comments like 232the following: 233 234.. code-block:: python 235 236 # keep-sorted: start 237 bar 238 baz 239 foo 240 # keep-sorted: end 241 242This can be included by adding ``pw_presubmit.keep_sorted.presubmit_check`` to a 243presubmit program. Adding ``ignore-case`` to the start line will use 244case-insensitive sorting. 245 246By default, duplicates will be removed. Lines that are identical except in case 247are preserved, even with ``ignore-case``. To allow duplicates, add 248``allow-dupes`` to the start line. 249 250Prefixes can be ignored by adding ``ignore-prefix=`` followed by a 251comma-separated list of prefixes. The list below will be kept in this order. 252Neither commas nor whitespace are supported in prefixes. 253 254.. code-block:: python 255 256 # keep-sorted: start ignore-prefix='," 257 'bar', 258 "baz", 259 'foo', 260 # keep-sorted: end 261 262Inline comments are assumed to be associated with the following line. For 263example, the following is already sorted. This can be disabled with 264``sticky-comments=no``. 265 266.. todo-check: disable 267 268.. code-block:: python 269 270 # keep-sorted: start 271 # TODO: b/1234 - Fix this. 272 bar, 273 # TODO: b/5678 - Also fix this. 274 foo, 275 # keep-sorted: end 276 277.. todo-check: enable 278 279By default, the prefix of the keep-sorted line is assumed to be the comment 280marker used by any inline comments. This can be overridden by adding lines like 281``sticky-comments=%,#`` to the start line. 282 283Lines indented more than the preceding line are assumed to be continuations. 284Thus, the following block is already sorted. keep-sorted blocks can not be 285nested, so there's no ability to add a keep-sorted block for the sub-items. 286 287.. code-block:: 288 289 # keep-sorted: start 290 * abc 291 * xyz 292 * uvw 293 * def 294 # keep-sorted: end 295 296The presubmit check will suggest fixes using ``pw keep-sorted --fix``. 297 298Future versions may support additional multiline list items. 299 300.gitmodules 301^^^^^^^^^^^ 302Various rules can be applied to .gitmodules files. This check can be included 303by adding ``pw_presubmit.gitmodules.create()`` to a presubmit program. This 304function takes an optional argument of type ``pw_presubmit.gitmodules.Config``. 305``Config`` objects have several properties. 306 307* ``allow_submodules: bool = True`` — If false, don't allow any submodules. 308* ``allow_non_googlesource_hosts: bool = False`` — If false, all submodule URLs 309 must be on a Google-managed Gerrit server. 310* ``allowed_googlesource_hosts: Sequence[str] = ()`` — If set, any 311 Google-managed Gerrit URLs for submodules most be in this list. Entries 312 should be like ``pigweed`` for ``pigweed-review.googlesource.com``. 313* ``require_relative_urls: bool = False`` — If true, all submodules must be 314 relative to the superproject remote. 315* ``allow_sso: bool = True`` — If false, ``sso://`` and ``rpc://`` submodule 316 URLs are prohibited. 317* ``allow_git_corp_google_com: bool = True`` — If false, ``git.corp.google.com`` 318 submodule URLs are prohibited. 319* ``require_branch: bool = False`` — If true, all submodules must reference a 320 branch. 321* ``validator: Callable[[PresubmitContext, Path, str, dict[str, str]], None] = None`` 322 — A function that can be used for arbitrary submodule validation. It's called 323 with the ``PresubmitContext``, the path to the ``.gitmodules`` file, the name 324 of the current submodule, and the properties of the current submodule. 325 326#pragma once 327^^^^^^^^^^^^ 328There's a ``pragma_once`` check that confirms the first non-comment line of 329C/C++ headers is ``#pragma once``. This is enabled by adding 330``pw_presubmit.cpp_checks.pragma_once`` to a presubmit program. 331 332#ifndef/#define 333^^^^^^^^^^^^^^^ 334There's an ``ifndef_guard`` check that confirms the first two non-comment lines 335of C/C++ headers are ``#ifndef HEADER_H`` and ``#define HEADER_H``. This is 336enabled by adding ``pw_presubmit.cpp_checks.include_guard_check()`` to a 337presubmit program. ``include_guard_check()`` has options for specifying what the 338header guard should be based on the path. 339 340This check is not used in Pigweed itself but is available to projects using 341Pigweed. 342 343.. todo-check: disable 344 345TODO(b/###) Formatting 346^^^^^^^^^^^^^^^^^^^^^^ 347There's a check that confirms ``TODO`` lines match a given format. Upstream 348Pigweed expects these to look like ``TODO: https://pwbug.dev/### - 349Explanation``, but projects may define their own patterns instead. 350 351For information on supported TODO expressions, see Pigweed's 352:ref:`docs-pw-todo-style`. 353 354.. todo-check: enable 355 356Python Checks 357^^^^^^^^^^^^^ 358There are two checks in the ``pw_presubmit.python_checks`` module, ``gn_pylint`` 359and ``gn_python_check``. They assume there's a top-level ``python`` GN target. 360``gn_pylint`` runs Pylint and Mypy checks and ``gn_python_check`` runs Pylint, 361Mypy, and all Python tests. 362 363Bazel Checks 364^^^^^^^^^^^^ 365There is one Bazel-related check: the ``includes_presubmit_check`` verifies 366that ``cc_library`` Bazel targets don't use the ``includes`` attribute. See 367:bug:`378564135` for a discussion of why this attribute should be avoided. 368 369Inclusive Language 370^^^^^^^^^^^^^^^^^^ 371.. inclusive-language: disable 372 373The inclusive language check looks for words that are typical of non-inclusive 374code, like using "master" and "slave" in place of "primary" and "secondary" or 375"sanity check" in place of "consistency check". 376 377.. inclusive-language: enable 378 379These checks can be disabled for individual lines with 380"inclusive-language: ignore" on the line in question or the line above it, or 381for entire blocks by using "inclusive-language: disable" before the block and 382"inclusive-language: enable" after the block. 383 384.. In case things get moved around in the previous paragraphs the enable line 385.. is repeated here: inclusive-language: enable. 386 387OWNERS 388^^^^^^ 389There's a check that requires folders matching specific patterns contain 390``OWNERS`` files. It can be included by adding 391``module_owners.presubmit_check()`` to a presubmit program. This function takes 392a callable as an argument that indicates, for a given file, where a controlling 393``OWNERS`` file should be, or returns None if no ``OWNERS`` file is necessary. 394Formatting of ``OWNERS`` files is handled similary to formatting of other 395source files and is discussed in `Code Formatting`. 396 397JSON 398^^^^ 399The JSON check requires all ``*.json`` files to be valid JSON files. It can be 400included by adding ``json_check.presubmit_check()`` to a presubmit program. 401 402Source in Build 403^^^^^^^^^^^^^^^ 404Pigweed provides checks that source files are configured as part of the build 405for GN, Bazel, CMake, and Soong. These can be included by adding 406``source_in_build.gn(filter)`` and similar functions to a presubmit check. The 407CMake check additionally requires a callable that invokes CMake with appropriate 408options. 409 410pw_presubmit 411------------ 412.. automodule:: pw_presubmit 413 :members: filter_paths, call, PresubmitFailure, Programs 414 415.. _example-script: 416 417 418Git hook 419-------- 420You can run a presubmit program or step as a `git hook 421<https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks>`_ using 422``pw_presubmit.install_hook``. This can be used to run certain presubmit 423checks before a change is pushed to a remote. 424 425We strongly recommend that you only run fast (< 15 seconds) and trivial checks 426as push hooks, and perform slower or more complex ones in CI. This is because, 427 428* Running slow checks in the push hook will force you to wait longer for 429 ``git push`` to complete, and 430* If your change fails one of the checks at this stage, it will not yet be 431 uploaded to the remote, so you'll have a harder time debugging any failures 432 (sharing the change with your colleagues, linking to it from an issue 433 tracker, etc). 434 435Example 436======= 437A simple example presubmit check script follows. This can be copied-and-pasted 438to serve as a starting point for a project's presubmit check script. 439 440See ``pigweed_presubmit.py`` for a more complex presubmit check script example. 441 442.. code-block:: python 443 444 """Example presubmit check script.""" 445 446 import argparse 447 import logging 448 import os 449 from pathlib import Path 450 import re 451 import sys 452 453 try: 454 import pw_cli.log 455 except ImportError: 456 print("ERROR: Activate the environment before running presubmits!", file=sys.stderr) 457 sys.exit(2) 458 459 import pw_presubmit 460 from pw_presubmit import ( 461 build, 462 cli, 463 cpp_checks, 464 format_code, 465 inclusive_language, 466 python_checks, 467 ) 468 from pw_presubmit.presubmit import filter_paths 469 from pw_presubmit.presubmit_context import PresubmitContext 470 from pw_presubmit.install_hook import install_git_hook 471 472 # Set up variables for key project paths. 473 PROJECT_ROOT = Path(os.environ["MY_PROJECT_ROOT"]) 474 PIGWEED_ROOT = PROJECT_ROOT / "pigweed" 475 476 # Rerun the build if files with these extensions change. 477 _BUILD_EXTENSIONS = frozenset( 478 [".rst", ".gn", ".gni", *format_code.C_FORMAT.extensions] 479 ) 480 481 482 # 483 # Presubmit checks 484 # 485 def release_build(ctx: PresubmitContext): 486 build.gn_gen(ctx, build_type="release") 487 build.ninja(ctx) 488 build.gn_check(ctx) # Run after building to check generated files. 489 490 491 def host_tests(ctx: PresubmitContext): 492 build.gn_gen(ctx, run_host_tests="true") 493 build.ninja(ctx) 494 build.gn_check(ctx) 495 496 497 # Avoid running some checks on certain paths. 498 PATH_EXCLUSIONS = ( 499 re.compile(r"^external/"), 500 re.compile(r"^vendor/"), 501 ) 502 503 504 # Use the upstream pragma_once check, but apply a different set of path 505 # filters with @filter_paths. 506 @filter_paths(endswith=".h", exclude=PATH_EXCLUSIONS) 507 def pragma_once(ctx: PresubmitContext): 508 cpp_checks.pragma_once(ctx) 509 510 511 # 512 # Presubmit check programs 513 # 514 OTHER = ( 515 # Checks not ran by default but that should be available. These might 516 # include tests that are expensive to run or that don't yet pass. 517 build.gn_gen_check, 518 ) 519 520 QUICK = ( 521 # List some presubmit checks to run 522 pragma_once, 523 host_tests, 524 # Use the upstream formatting checks, with custom path filters applied. 525 format_code.presubmit_checks(exclude=PATH_EXCLUSIONS), 526 # Include the upstream inclusive language check. 527 inclusive_language.presubmit_check, 528 # Include just the lint-related Python checks. 529 python_checks.gn_python_lint.with_filter(exclude=PATH_EXCLUSIONS), 530 ) 531 532 FULL = ( 533 QUICK, # Add all checks from the 'quick' program 534 release_build, 535 # Use the upstream Python checks, with custom path filters applied. 536 # Checks listed multiple times are only run once. 537 python_checks.gn_python_check.with_filter(exclude=PATH_EXCLUSIONS), 538 ) 539 540 PROGRAMS = pw_presubmit.Programs(other=OTHER, quick=QUICK, full=FULL) 541 542 543 # 544 # Allowlist of remote refs for presubmit. If the remote ref being pushed to 545 # matches any of these values (with regex matching), then the presubmits 546 # checks will be run before pushing. 547 # 548 PRE_PUSH_REMOTE_REF_ALLOWLIST = ("refs/for/main",) 549 550 551 def run(install: bool, remote_ref: str | None, **presubmit_args) -> int: 552 """Process the --install argument then invoke pw_presubmit.""" 553 554 # Install the presubmit Git pre-push hook, if requested. 555 if install: 556 # '$remote_ref' will be replaced by the actual value of the remote ref 557 # at runtime. 558 install_git_hook( 559 "pre-push", 560 [ 561 "python", 562 "-m", 563 "tools.presubmit_check", 564 "--base", 565 "HEAD~", 566 "--remote-ref", 567 "$remote_ref", 568 ], 569 ) 570 return 0 571 572 # Run the checks if either no remote_ref was passed, or if the remote ref 573 # matches anything in the allowlist. 574 if remote_ref is None or any( 575 re.search(pattern, remote_ref) 576 for pattern in PRE_PUSH_REMOTE_REF_ALLOWLIST 577 ): 578 return cli.run(root=PROJECT_ROOT, **presubmit_args) 579 return 0 580 581 582 def main() -> int: 583 """Run the presubmit checks for this repository.""" 584 parser = argparse.ArgumentParser(description=__doc__) 585 cli.add_arguments(parser, PROGRAMS, "quick") 586 587 # Define an option for installing a Git pre-push hook for this script. 588 parser.add_argument( 589 "--install", 590 action="store_true", 591 help="Install the presubmit as a Git pre-push hook and exit.", 592 ) 593 594 # Define an optional flag to pass the remote ref into this script, if it 595 # is run as a pre-push hook. The destination variable in the parsed args 596 # will be `remote_ref`, as dashes are replaced with underscores to make 597 # valid variable names. 598 parser.add_argument( 599 "--remote-ref", 600 default=None, 601 nargs="?", # Make optional. 602 help="Remote ref of the push command, for use by the pre-push hook.", 603 ) 604 605 return run(**vars(parser.parse_args())) 606 607 608 if __name__ == "__main__": 609 pw_cli.log.install(logging.INFO) 610 sys.exit(main()) 611 612--------------------- 613Code formatting tools 614--------------------- 615The ``pw_presubmit.format_code`` module formats supported source files using 616external code format tools. The file ``format_code.py`` can be invoked directly 617from the command line or from ``pw`` as ``pw format``. 618 619Example 620======= 621A simple example of adding support for a custom format. This code wraps the 622built in formatter to add a new format. It could also be used to replace 623a formatter or remove/disable a PigWeed supplied one. 624 625.. code-block:: python 626 627 #!/usr/bin/env python 628 """Formats files in repository. """ 629 630 import logging 631 import sys 632 633 import pw_cli.log 634 from pw_presubmit import format_code 635 from your_project import presubmit_checks 636 from your_project import your_check 637 638 YOUR_CODE_FORMAT = CodeFormat('YourFormat', 639 filter=FileFilter(suffix=('.your', )), 640 check=your_check.check, 641 fix=your_check.fix) 642 643 CODE_FORMATS = (*format_code.CODE_FORMATS, YOUR_CODE_FORMAT) 644 645 def _run(exclude, **kwargs) -> int: 646 """Check and fix formatting for source files in the repo.""" 647 return format_code.format_paths_in_repo(exclude=exclude, 648 code_formats=CODE_FORMATS, 649 **kwargs) 650 651 652 def main(): 653 return _run(**vars(format_code.arguments(git_paths=True).parse_args())) 654 655 656 if __name__ == '__main__': 657 pw_cli.log.install(logging.INFO) 658 sys.exit(main()) 659 660.. pw_presubmit-nav-end 661 662.. toctree:: 663 :maxdepth: 1 664 :hidden: 665 666 format 667
