1.. _docs-ci-cq-intro: 2 3=========== 4CI/CQ Intro 5=========== 6 7.. _essentials: 8 9---------- 10Essentials 11---------- 12 13Submitting Changes 14================== 15The Gerrit ``Submit`` button is hidden, replaced by the ``Submit to CQ`` 16button. The ``Submit`` button is still accessible behind the ``...`` button 17in the top right corner of the Gerrit UI, but in some cases requires elevated 18permissions. 19 20Triggering presubmits 21===================== 22Presubmits are not automatically run when a patch set is uploaded. Click 23``CQ Dry Run`` to trigger them. (You can also use 24``git push origin +HEAD:refs/for/main%l=Commit-Queue``). 25 26Presubmit validity duration 27=========================== 28If you don't have recent passing results from a ``CQ Dry Run`` (within 24 29hours) then ``Submit to CQ`` will run presubmits. After any newly run 30presubmits pass this will submit the change. 31 32User interface 33============== 34If a presubmit fails you'll get a Gerrit comment with a link to the failing 35build. The status of tryjobs (pending, running, failed, passed, etc.) is 36shown directly in the Gerrit UI (see :ref:`tryjobs`). 37 38Auto-Submit 39=========== 40If you want your change to be automatically submitted when all requirements 41are met (``Code-Review +2``, ``OWNERS``-approval, all comments resolved, 42etc.) set the ``Auto-Submit`` label to +1. If submission fails it will be 43retried a couple times with backoff and then the auto submit job will give up. 44 45 46.. _further-details: 47 48--------------- 49Further Details 50--------------- 51 52Applying changes in testing 53=========================== 54Changes are always rebased on the most recent commit when tested. If they 55fail to rebase the build fails. 56``CQ Dry Run`` is the same as voting ``Commit-Queue +1`` label and 57``Submit to CQ`` is the same as voting ``Commit-Queue +2``. If you vote both 58``Code-Review +2`` and ``Commit-Queue +2`` on somebody's change you are 59submitting it for them. 60 61Post-Submit builders 62==================== 63Jobs are run post-submission too and can be seen at 64https://ci.chromium.org/p/pigweed (for public projects) and 65https://ci.chromium.org/p/pigweed-internal (for internal projects). Builders can 66also be viewed from Pigweed's :ref:`builder visualization <docs-builder-viz>`. 67 68Non-``main`` branches 69===================== 70CQ is enabled for all branches. If you upload to an individual repository 71branch X and the manifest or superproject also has a branch X, that branch of 72the manifest will be used. 73 74Rollers 75======= 76Just because a change has been submitted doesn't mean it's live in the 77project. Submodules and Android Repo Tool projects often need to be 78:ref:`rolled <docs-rollers>` before they're in the most recent checkout of the 79project. 80 81Presubmit result 82================ 83The ``Presubmit-Verified`` label is set at the completion of CQ runs. It does 84not block submission, but can be used by Copybara to see if CQ has passed. 85If it looks incorrect, do another CQ run and it will be updated. 86 87.. _docs-builder-viz: 88 89Builder Visualization 90===================== 91 92Pigweed's builder visualization simplifies the process of browsing, viewing, and 93triggering builders. The source-of-truth for it is Google-internal, but there's 94a public version without the Google-internal bits. 95 96 * `Builder Viz link for external contributors <https://pigweed.googlesource.com/infra/config/+/main/generated/pigweed/for_review_only/viz/index.md>`_ 97 * `Builder Viz link for Googlers <https://pigweed-internal.googlesource.com/infra/config/+/main/generated/pigweed/for_review_only/viz/index.md>`_ 98 99.. _tryjobs: 100 101------- 102Tryjobs 103------- 104The colors of tryjobs in the Gerrit UI indicate the status of the tryjob: gray 105is pending/running, green is passed, red is failed or cancelled, and purple is 106an infra failure. 107 108Some tryjobs are not yet stable and are run as "experimental". These can be 109viewed with ``Show Experimental Tryjobs``. Experimental tryjobs run with CQ but 110do not block it. 111 112Individual tryjobs can be run additional times using the ``Choose Tryjobs`` 113dialog. This can also be used to run tryjobs that would not normally run on the 114change. Tryjobs ran this way can be used to satisfy CQ requirements, but don't 115block CQ. 116 117.. _prod-vs-dev: 118 119Prod vs Dev 120=========== 121Most builders have "prod" and "dev" versions. The "prod" versions block changes 122in CQ and may cause emails to be sent out if they fail in CI. The "dev" builders 123test new VM images before they go to "prod", so if a "dev" builder is failing 124when a "prod" builder is not failing, then the "dev" builder is failing because 125of an upcoming VM change, and teams should take time to get the "dev" builder to 126pass. For most projects, "dev" builders show up on the far right of console 127views in the LUCI UI. 128 129.. _tryjobs-cli: 130 131Tryjobs CLI 132=========== 133 134``bb`` command 135-------------- 136The ``bb`` command is available in a bootstrapped Pigweed environment and the 137environments of many downstream projects. It is also available from 138`Chromium's depot tools <https://chromium.googlesource.com/chromium/tools/depot_tools.git>_`. 139 140 141Querying tryjobs 142---------------- 143In addition to viewing tryjobs in the Gerrit UI, you can use the ``bb`` command 144to query the tryjobs that ran on a change. The command to use is 145``bb ls -cl $URL``, but ``$URL`` has two non-obvious requirements: 146 147* It needs to be a ".googlesource.com" URL and not a Google-internal version of 148 that URL. 149* It needs to include the patchset number. 150 151.. code-block:: bash 152 153 $ bb ls -cl https://pigweed-review.googlesource.com/c/pigweed/sample_project/+/53684/1 | egrep -v '^(Tag|By):' 154 http://ci.chromium.org/b/8841234941714219488 SUCCESS 'pigweed/try/sample-project-xref-generator' 155 Created on 2021-07-19 at 16:45:32, waited 14.8s, started at 16:45:47, ran for 2m43s, ended at 16:48:30 156 CL: https://pigweed-review.googlesource.com/c/pigweed/sample_project/+/53684/1 157 158 http://ci.chromium.org/b/8841234941714219504 INFRA_FAILURE 'pigweed/try/sample-project-renode-test' 159 Summary: Infra Failure: Step('run pigweed/pw_unit_test/renode/test.sh') (retcode: None) 160 Created on 2021-07-19 at 16:45:32, waited 51.6s, started at 16:46:24, ran for 2m40s, ended at 16:49:04 161 CL: https://pigweed-review.googlesource.com/c/pigweed/sample_project/+/53684/1 162 163To exclude non-experimental tryjobs, add ``-t cq_experimental:false`` to the 164command. 165 166Manually launching tryjobs 167-------------------------- 168In most cases, individual tryjobs can be launched using 169``Choose Additional Tryjobs``. If any relevant tryjobs are not listed here 170please file a bug. 171 172The ``bb`` command can also be used to launch tryjobs, which can useful for 173tracking down race conditions by launching many copies of a tryjob. Please be 174careful using this, especially during working hours in California. 175 176.. code-block:: shell 177 178 URL="https://pigweed-review.googlesource.com/c/pigweed/sample_project/+/53684/1" 179 TRYJOB="pigweed/sample_project.try/sample-project-xref-generator" 180 for i in $(seq 1 25); do 181 bb add -cl "$URL" "$TRYJOB" 182 done 183 184.. _why-didnt-lintformat-catch: 185 186Why didn't lintformat catch this formatting change? 187=================================================== 188Rolls of tools like clang can update the preferred format of clang-format. There 189are two possibilities for addressing this. First, the tool roll could be blocked 190until formatting passes. This could require coordinating several changes across 191many repositories. This is further complicated if the new formatting preferred 192by clang-format is not accepted by the previous version. Second, lintformat can 193be configured to only run on changed files. This means downstream project 194lintformat tryjobs would not be run on Pigweed changes, nor on rolls of Pigweed 195into these projects. 196 197The second choice was selected. This means when tools roll lintformat jobs may 198start failing in CI, but they only fail in CQ on changes that touch files 199currently failing in CI. Teams should watch their build alert email list and 200proactively fix lintformat failures when they come. 201 202.. _dependent-changes: 203 204----------------- 205Dependent changes 206----------------- 207 208.. _creating: 209 210Creating 211======== 212To pull in other changes when testing a change add a ``patches.json`` file to 213the root of the repository. An example is below. 214 215.. code-block:: json 216 217 [ 218 { 219 "gerrit_name": "pigweed", 220 "number": 123456 221 }, 222 { 223 "gerrit_name": "pigweed", 224 "number": 654321 225 } 226 ] 227 228Patches can be uni- or bidirectional and are transitive. The tryjob will parse 229this file, and then parse any ``patches.json`` files found in the referenced 230changes. If requirements are truly one-way, don't list them as two-way. Only the 231Gerrit instance name (the part before "-review") is permitted. The repository 232name is not included. 233 234.. admonition:: Note 235 :class: warning 236 237 ``patches.json`` cannot be used for changes to *the same repo* on the same 238 Gerrit host (`b/230610752 <https://issuetracker.google.com/230610752>`_). 239 Just stack these changes instead. 240 241.. _submitting: 242 243Submitting 244========== 245Pigweed's infrastructure does not support submitting multiple changes together. 246The best option is to find a way to have changes not depend on each other and 247submit them separately, or to have a one-way requirement instead of codependent 248changes, and submit the changes in dependency order, waiting for any necessary 249rolls before submitting the next change. 250 251Pigweed-related Gerrit hosts are configured to reject submission of all changes 252containing ``patches.json`` files. If the dependency is one-way, then submit the 253change without dependencies, wait for it to roll (if necessary), remove 254``patches.json`` from the dependent change, and vote ``Commit-Queue +2``. 255 256If the changes are truly codependent—both (or all) changes need each other—then 257follow the instructions below. 258 259First, get both changes passing CQ with ``patches.json`` files. 260 261Second, if one of the codependent changes is a submodule and another is the 262parent project, update the submodule change to no longer include the 263``patches.json`` file. Then directly submit the change that lives in the child 264submodule, bypassing CQ. This will break the roller, but not the source tree, so 265others on your team are unaffected. 266 267.. admonition:: Note 268 269 For the main Pigweed repository, only core Pigweed team members can force 270 submit, and they must first 271 `request a temporary ACL <http://go/pwi-cookbook#bypass-cq>`_ to do so. This 272 process requires an associated bug, so have one on hand before reaching out 273 with a force submission request. 274 275 276Finally, once the change has merged into the child project, update the submodule 277pointer in the parent project: 278 279.. admonition:: Note 280 :class: warning 281 282 Some projects have limitations on submission outside of CQ. Reach out to a 283 core Pigweed team member to bypass CQ for Pigweed itself. 284 285#. Update your submodule pin to the submitted commit hash (in most cases 286 ``git submodule update --remote path/to/submodule`` should be sufficient, 287 but see the 288 `git submodule documentation <https://git-scm.com/book/en/v2/Git-Tools-Submodules>`_ 289 for full details) 290#. Add that change to the parent project change (``git add path/to/submodule``) 291#. Remove the ``patches.json`` file from the change (``git rm patches.json``) 292#. Commit and push to Gerrit 293#. Click ``Submit to CQ`` 294 295After this change is submitted the roller will start working again. 296 297If all changes are to submodules, remove the ``patches.json`` files from both 298changes and directly submit, bypassing CQ. Then create a manual roll change that 299updates the submodules in question 300(``git submodule update --remote submodule1 submodule2`` 301should be sufficient), upload it, and ``Submit to CQ``. 302 303.. _details: 304 305Details 306======= 307Sometimes codependent changes must be made in multiple repositories within an 308Android Repo Tool workspace or across multiple submodules. This can be done with 309the ``patches.json`` files. Given a situation where pigweed change would break 310the sample_project, the ``patches.json`` files must each refer to the other 311change. 312 313Pigweed ``patches.json`` 314 ``[{"gerrit_name": "pigweed", "number": B}]`` 315 316Sample Project ``patches.json`` 317 ``[{"gerrit_name": "pigweed", "number": A}]`` 318 319When running tryjobs for change A, builders will attempt to patch in change B as 320well. For pure Pigweed tryjobs this fails but the build continues. For the 321tryjobs that are there to ensure Pigweed doesn't break the Sample Project, both 322change A and change B will be applied to the checkout. 323 324There is some validation of the format of the ``patches.json`` file, but there's 325no error checking on the resolution of the required changes. The assumption is 326that changes that actually require other changes to pass CQ will fail if those 327changes aren't patched into the workspace. 328 329Requirements are transitive. If A requires B and B requires C then tryjobs for A 330will attempt to patch in A, B, and C. Requirements can also be one-way. If a 331change has been submitted it's assumed to already be in the checkout and is not 332patched in, nor are any transitive requirements processed. Likewise, abandoned 333changes are ignored. 334 335.. _banned-codewords: 336 337Banned codewords 338================ 339Sometimes the name of an internal Gerrit instance is a codeword we don't allow 340on the Pigweed Gerrit instance. For example, you may wish to do the following. 341 342Pigweed change A ``patches.json`` 343 ``[{"gerrit_name": "secret-project", "number": B}]`` 344 345Secret-Project change B ``patches.json`` 346 ``[{"gerrit_name": "pigweed", "number": A}]`` 347 348This will be rejected by the Pigweed Gerrit instance because using 349"secret-project" is banned on that Gerrit instance and you won't be able to 350push. Instead, do the following, using the 351`requires-helper <https://pigweed-internal.googlesource.com/requires-helper>`_ 352repository on the Pigweed-Internal Gerrit instance. 353 354Pigweed change A ``patches.json`` 355 ``[{"gerrit_name": "pigweed-internal", "number": C}]`` 356 357Secret-Project change B ``patches.json`` 358 ``[{"gerrit_name": "pigweed", "number": A}]`` 359 360Pigweed-Internal change C ``patches.json`` 361 ``[{"gerrit_name": "secret-project", "number": B}]`` 362 363The ``pw requires`` command simplifies creation of the Pigweed-Internal change. 364In this case the command would be ``pw requires secret-project:B``. Run this 365inside the Pigweed repository after committing change A and it will create 366change C and add ``[{"gerrit_name": "pigweed-internal", "number": C}]`` to 367change A. Multiple changes can be handled by passing multiple arguments to 368``pw requires``. 369 370Public builders won't have access to the Pigweed-Internal Gerrit instance so 371they won't even be able to see the ``secret-project`` reference. Internal 372builders for other internal projects will see the ``secret-project`` reference 373but won't be able to resolve it. Builders having access to ``secret-project`` 374will see all three changes and attempt to patch all three in. Pigweed-Internal 375change C is not included in any workspaces so it will never be patched in, but 376it transitively applies requirements to public changes. 377