README.md
1# Luci Infrastructure
2
3This directory contains the configuration and build recipes run by our luci infrastructure for CI
4and presubmit testing.
5
6Note: Luci applies config and recipes changes asynchronously. Do not submit changes to this
7directory in the same commit as changes to other crosvm source.
8
9## Recipes
10
11### Recipe Documentation
12
13A few links to relevant documentation needed to write recipes:
14
15- [Recipe Engine](https://chromium.googlesource.com/infra/luci/recipes-py.git/+/HEAD/README.recipes.md)
16- [Depot Tools Recipes](https://chromium.googlesource.com/chromium/tools/depot_tools.git/+/HEAD/recipes/README.recipes.md))
17- [ChromiumOS Recipes](https://chromium.googlesource.com/chromiumos/infra/recipes.git/+/HEAD/README.recipes.md)
18
19Luci also provides a
20[User Guide](https://chromium.googlesource.com/infra/luci/recipes-py/+/master/doc/user_guide.md) and
21[Walkthrough](https://chromium.googlesource.com/infra/luci/recipes-py/+/refs/heads/main/doc/walkthrough.md)
22for getting started with recipes.
23
24For writing tests, documentation can be found in the
25[Recipe test API](https://chromium.googlesource.com/infra/luci/recipes-py/+/HEAD/recipe_engine/recipe_test_api.py)
26and
27[Post Process API](https://chromium.googlesource.com/infra/luci/recipes-py/+/HEAD/recipe_engine/post_process.py)
28
29### Running recipe tests
30
31Recipes must have 100% code coverage to have tests pass. Tests can be run with:
32
33```
34cd infra && ./recipes.py test run
35```
36
37Most tests execute a few example invocations, record the commands that would be executed and compare
38them to the json files in `*.expected`. This allows developers to catch unwanted side-effects of
39their changes.
40
41To regenerate the expectation files, run:
42
43```
44cd infra && ./recipes.py test train
45```
46
47Then verify the `git diff` to make sure all changes to outcomes are intentional.
48
49### Testing recipes locally
50
51We try to build our recipes to work well locally, so for example build_linux.py can be invoked in
52the recipe engine via:
53
54```
55cd infra && ./recipes.py run build_linux
56```
57
58When run locally, recipes that check out crosvm, will run against the current HEAD of the main
59branch.
60
61The recipe will run in the local `infra/.recipe_deps/recipe_engine/workdir` directory and is
62preserved between runs in the same way data is preserved on bots, so incremental builds or the use
63of cached files can be tested.
64
65### Testing recipes on a bot (Googlers only)
66
67Note: See internal [crosvm/infra](http://go/crosvm/infra) documentation on access control.
68
69Some things cannot be tested locally and need to be run on one of our build bots. This can be done
70with the [led](http://go/luci-how-to-led) tool.
71
72Commonly used led commands are:
73
74- `led get-builder $NAME` will download and output the job template for that builder.
75- `led get-build $BBID` will download the job definition of a previous build.
76- `led edit-recipe-bundle` will update the job to use your local version recipes
77- `led edit-cr-cl` will update the job to run on a gerrit change
78- `led launch` launches a new job using the input job definition.
79
80Important: Changes to recipes are applied separately from changes to crosvm code.
81
82#### Testing recipe changes on post-submit builders
83
84To test a local recipe change, you can launch a post-submit build using `led`. First `git commit`
85your recipe changes locally, then combine the led commands to:
86
87```
88led get-builder luci.crosvm.ci:linux_x86_64
89 | led-edit-recipe-bundle
90 | led launch
91```
92
93This will run the `linux_x86_64` builder on the current `main` revision of crosvm using the local
94version of recipes.
95
96Important: Changes to crosvm source outside of recipes will not be part of the build.
97
98#### Testing recipe and source changes on pre-submit builders
99
100If we want to test a combination of both recipe and source changes, we can test those on a
101pre-submit builder, which patch in a gerrit change to test.
102
103We can specify that gerrit change via `led edit-cr-cl`.
104
105So to test, first `git commit` and `./tools/cl upload` your local changes. Then build a job
106definition to run:
107
108```
109led get-builder luci.crosvm.try:linux_x86_64
110 | led-edit-recipe-bundle
111 | led-edit-cr-cl $GERRIT_URL
112 | led launch
113```
114
115This will launch a presubmit builder using the local version of recipes, and runs it on the gerrit
116change at $GERRIT_URL.
117
118#### Testing a new recipe
119
120This is a little tricker, but you can change the recipe name that is launched by `led`. So you can:
121
122```
123led get-builder luci.crosvm.ci:linux_x86_64 > job.json
124```
125
126Then edit the `job.json` file to use the newly added recipe, and launch the job using the local
127version of recipes.
128
129```
130cat job.json | led-edit-recipe-bundle | led launch
131```
132
README.recipes.md
1<!--- AUTOGENERATED BY `./recipes.py test train` -->
2# Repo documentation for [crosvm](https://chromium.googlesource.com/crosvm/crosvm.git)
3## Table of Contents
4
5**[Recipe Modules](#Recipe-Modules)**
6 * [crosvm](#recipe_modules-crosvm)
7
8**[Recipes](#Recipes)**
9 * [build_chromeos_hatch](#recipes-build_chromeos_hatch)
10 * [build_docs](#recipes-build_docs)
11 * [build_linux](#recipes-build_linux)
12 * [build_windows](#recipes-build_windows)
13 * [crosvm:examples/container_build_context](#recipes-crosvm_examples_container_build_context)
14 * [crosvm:examples/cros_container_build_context](#recipes-crosvm_examples_cros_container_build_context)
15 * [crosvm:examples/host_build_context](#recipes-crosvm_examples_host_build_context)
16 * [crosvm:examples/source_context](#recipes-crosvm_examples_source_context)
17 * [health_check](#recipes-health_check)
18 * [presubmit](#recipes-presubmit)
19 * [push_to_github](#recipes-push_to_github)
20 * [update_chromeos_merges](#recipes-update_chromeos_merges)
21## Recipe Modules
22
23### *recipe_modules* / [crosvm](/infra/recipe_modules/crosvm)
24
25[DEPS](/infra/recipe_modules/crosvm/__init__.py#7): [depot\_tools/bot\_update][depot_tools/recipe_modules/bot_update], [depot\_tools/gclient][depot_tools/recipe_modules/gclient], [depot\_tools/git][depot_tools/recipe_modules/git], [recipe\_engine/buildbucket][recipe_engine/recipe_modules/buildbucket], [recipe\_engine/cipd][recipe_engine/recipe_modules/cipd], [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/file][recipe_engine/recipe_modules/file], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/platform][recipe_engine/recipe_modules/platform], [recipe\_engine/raw\_io][recipe_engine/recipe_modules/raw_io], [recipe\_engine/step][recipe_engine/recipe_modules/step]
26
27
28#### **class [CrosvmApi](/infra/recipe_modules/crosvm/api.py#11)([RecipeApi][recipe_engine/wkt/RecipeApi]):**
29
30Crosvm specific functionality shared between recipes.
31
32  **@property**<br>— **def [builder\_cache](/infra/recipe_modules/crosvm/api.py#43)(self):**
33
34Dedicated cache directory for each builder.
35
36Luci will try to run each builder on the same bot as previously to keep this cache present.
37
38  **@property**<br>— **def [cargo\_home](/infra/recipe_modules/crosvm/api.py#24)(self):**
39
40CARGO_HOME is cached between runs.
41
42  **@property**<br>— **def [cargo\_target\_dir](/infra/recipe_modules/crosvm/api.py#29)(self):**
43
44CARGO_TARGET_DIR is cleaned up between runs
45
46— **def [container\_build\_context](/infra/recipe_modules/crosvm/api.py#62)(self):**
47
48Prepares source and system to build crosvm via dev container.
49
50Usage:
51 with api.crosvm.container_build_context():
52 api.crosvm.step_in_container("build crosvm", ["cargo build"])
53
54— **def [cros\_container\_build\_context](/infra/recipe_modules/crosvm/api.py#79)(self):**
55
56Prepares source and system to build crosvm via cros container.
57
58Usage:
59 with api.crosvm.cros_container_build_context():
60 api.crosvm.step_in_container("build crosvm", ["cargo build"], cros=True)
61
62  **@property**<br>— **def [dev\_container\_cache](/infra/recipe_modules/crosvm/api.py#39)(self):**
63
64— **def [get\_git\_sha](/infra/recipe_modules/crosvm/api.py#186)(self):**
65
66— **def [host\_build\_context](/infra/recipe_modules/crosvm/api.py#113)(self):**
67
68Prepares source and system to build crosvm directly on the host.
69
70This will install the required rust version via rustup. However no further dependencies
71are installed.
72
73Usage:
74 with api.crosvm.host_build_context():
75 api.step("build crosvm", ["cargo build"])
76
77  **@property**<br>— **def [local\_bin](/infra/recipe_modules/crosvm/api.py#34)(self):**
78
79Directory used to install local tools required by the build.
80
81— **def [prepare\_git](/infra/recipe_modules/crosvm/api.py#161)(self):**
82
83  **@property**<br>— **def [rustup\_home](/infra/recipe_modules/crosvm/api.py#19)(self):**
84
85RUSTUP_HOME is cached between runs.
86
87— **def [source\_context](/infra/recipe_modules/crosvm/api.py#52)(self):**
88
89Updates the source to the revision to be tested and drops into the source directory.
90
91Use when no build commands are needed.
92
93  **@property**<br>— **def [source\_dir](/infra/recipe_modules/crosvm/api.py#14)(self):**
94
95Where the crosvm source will be checked out.
96
97— **def [step\_in\_container](/infra/recipe_modules/crosvm/api.py#144)(self, step_name, command, cros=False, \*\*kwargs):**
98
99Runs a luci step inside the crosvm dev container.
100
101— **def [upload\_coverage](/infra/recipe_modules/crosvm/api.py#194)(self, filename):**
102## Recipes
103
104### *recipes* / [build\_chromeos\_hatch](/infra/recipes/build_chromeos_hatch.py)
105
106[DEPS](/infra/recipes/build_chromeos_hatch.py#7): [crosvm](#recipe_modules-crosvm), [depot\_tools/depot\_tools][depot_tools/recipe_modules/depot_tools], [recipe\_engine/buildbucket][recipe_engine/recipe_modules/buildbucket], [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/step][recipe_engine/recipe_modules/step]
107
108
109— **def [RunSteps](/infra/recipes/build_chromeos_hatch.py#17)(api):**
110### *recipes* / [build\_docs](/infra/recipes/build_docs.py)
111
112[DEPS](/infra/recipes/build_docs.py#7): [crosvm](#recipe_modules-crosvm), [depot\_tools/gsutil][depot_tools/recipe_modules/gsutil], [recipe\_engine/buildbucket][recipe_engine/recipe_modules/buildbucket], [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/file][recipe_engine/recipe_modules/file], [recipe\_engine/step][recipe_engine/recipe_modules/step]
113
114
115— **def [RunSteps](/infra/recipes/build_docs.py#20)(api):**
116
117Builds crosvm mdbook and api docs, then uploads them to GCS.
118
119This recipe requires ambient luci authentication. To test locally run:
120 $ luci-auth context ./infra/recipes.py run build_docs
121### *recipes* / [build\_linux](/infra/recipes/build_linux.py)
122
123[DEPS](/infra/recipes/build_linux.py#11): [crosvm](#recipe_modules-crosvm), [recipe\_engine/buildbucket][recipe_engine/recipe_modules/buildbucket], [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/json][recipe_engine/recipe_modules/json], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/raw\_io][recipe_engine/recipe_modules/raw_io], [recipe\_engine/step][recipe_engine/recipe_modules/step]
124
125
126— **def [RunSteps](/infra/recipes/build_linux.py#74)(api, properties):**
127
128— **def [collect\_binary\_sizes](/infra/recipes/build_linux.py#24)(api, properties):**
129### *recipes* / [build\_windows](/infra/recipes/build_windows.py)
130
131[DEPS](/infra/recipes/build_windows.py#7): [crosvm](#recipe_modules-crosvm), [recipe\_engine/buildbucket][recipe_engine/recipe_modules/buildbucket], [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/step][recipe_engine/recipe_modules/step]
132
133
134— **def [RunSteps](/infra/recipes/build_windows.py#16)(api):**
135### *recipes* / [crosvm:examples/container\_build\_context](/infra/recipe_modules/crosvm/examples/container_build_context.py)
136
137[DEPS](/infra/recipe_modules/crosvm/examples/container_build_context.py#7): [crosvm](#recipe_modules-crosvm)
138
139
140— **def [RunSteps](/infra/recipe_modules/crosvm/examples/container_build_context.py#12)(api):**
141### *recipes* / [crosvm:examples/cros\_container\_build\_context](/infra/recipe_modules/crosvm/examples/cros_container_build_context.py)
142
143[DEPS](/infra/recipe_modules/crosvm/examples/cros_container_build_context.py#7): [crosvm](#recipe_modules-crosvm)
144
145
146— **def [RunSteps](/infra/recipe_modules/crosvm/examples/cros_container_build_context.py#12)(api):**
147### *recipes* / [crosvm:examples/host\_build\_context](/infra/recipe_modules/crosvm/examples/host_build_context.py)
148
149[DEPS](/infra/recipe_modules/crosvm/examples/host_build_context.py#7): [crosvm](#recipe_modules-crosvm), [recipe\_engine/platform][recipe_engine/recipe_modules/platform], [recipe\_engine/step][recipe_engine/recipe_modules/step]
150
151
152— **def [RunSteps](/infra/recipe_modules/crosvm/examples/host_build_context.py#14)(api):**
153### *recipes* / [crosvm:examples/source\_context](/infra/recipe_modules/crosvm/examples/source_context.py)
154
155[DEPS](/infra/recipe_modules/crosvm/examples/source_context.py#7): [crosvm](#recipe_modules-crosvm), [recipe\_engine/buildbucket][recipe_engine/recipe_modules/buildbucket]
156
157
158— **def [RunSteps](/infra/recipe_modules/crosvm/examples/source_context.py#13)(api):**
159### *recipes* / [health\_check](/infra/recipes/health_check.py)
160
161[DEPS](/infra/recipes/health_check.py#7): [crosvm](#recipe_modules-crosvm), [recipe\_engine/buildbucket][recipe_engine/recipe_modules/buildbucket], [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/raw\_io][recipe_engine/recipe_modules/raw_io], [recipe\_engine/step][recipe_engine/recipe_modules/step]
162
163
164— **def [RunSteps](/infra/recipes/health_check.py#17)(api):**
165### *recipes* / [presubmit](/infra/recipes/presubmit.py)
166
167[DEPS](/infra/recipes/presubmit.py#13): [crosvm](#recipe_modules-crosvm), [recipe\_engine/buildbucket][recipe_engine/recipe_modules/buildbucket], [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/json][recipe_engine/recipe_modules/json], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/raw\_io][recipe_engine/recipe_modules/raw_io], [recipe\_engine/step][recipe_engine/recipe_modules/step]
168
169
170— **def [RunSteps](/infra/recipes/presubmit.py#26)(api, properties):**
171### *recipes* / [push\_to\_github](/infra/recipes/push_to_github.py)
172
173[DEPS](/infra/recipes/push_to_github.py#7): [crosvm](#recipe_modules-crosvm), [recipe\_engine/buildbucket][recipe_engine/recipe_modules/buildbucket], [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/file][recipe_engine/recipe_modules/file], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/raw\_io][recipe_engine/recipe_modules/raw_io], [recipe\_engine/step][recipe_engine/recipe_modules/step]
174
175
176— **def [RunSteps](/infra/recipes/push_to_github.py#18)(api):**
177### *recipes* / [update\_chromeos\_merges](/infra/recipes/update_chromeos_merges.py)
178
179[DEPS](/infra/recipes/update_chromeos_merges.py#7): [crosvm](#recipe_modules-crosvm), [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/step][recipe_engine/recipe_modules/step]
180
181
182— **def [RunSteps](/infra/recipes/update_chromeos_merges.py#14)(api):**
183
184[depot_tools/recipe_modules/bot_update]: https://chromium.googlesource.com/chromium/tools/depot_tools.git/+/2515d3513f7b15bf0cf40994599989b5dd7128ac/recipes/README.recipes.md#recipe_modules-bot_update
185[depot_tools/recipe_modules/depot_tools]: https://chromium.googlesource.com/chromium/tools/depot_tools.git/+/2515d3513f7b15bf0cf40994599989b5dd7128ac/recipes/README.recipes.md#recipe_modules-depot_tools
186[depot_tools/recipe_modules/gclient]: https://chromium.googlesource.com/chromium/tools/depot_tools.git/+/2515d3513f7b15bf0cf40994599989b5dd7128ac/recipes/README.recipes.md#recipe_modules-gclient
187[depot_tools/recipe_modules/git]: https://chromium.googlesource.com/chromium/tools/depot_tools.git/+/2515d3513f7b15bf0cf40994599989b5dd7128ac/recipes/README.recipes.md#recipe_modules-git
188[depot_tools/recipe_modules/gsutil]: https://chromium.googlesource.com/chromium/tools/depot_tools.git/+/2515d3513f7b15bf0cf40994599989b5dd7128ac/recipes/README.recipes.md#recipe_modules-gsutil
189[recipe_engine/recipe_modules/buildbucket]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/3624a48ac89993276cb80e675a88fcd3b39a0f39/README.recipes.md#recipe_modules-buildbucket
190[recipe_engine/recipe_modules/cipd]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/3624a48ac89993276cb80e675a88fcd3b39a0f39/README.recipes.md#recipe_modules-cipd
191[recipe_engine/recipe_modules/context]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/3624a48ac89993276cb80e675a88fcd3b39a0f39/README.recipes.md#recipe_modules-context
192[recipe_engine/recipe_modules/file]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/3624a48ac89993276cb80e675a88fcd3b39a0f39/README.recipes.md#recipe_modules-file
193[recipe_engine/recipe_modules/json]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/3624a48ac89993276cb80e675a88fcd3b39a0f39/README.recipes.md#recipe_modules-json
194[recipe_engine/recipe_modules/path]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/3624a48ac89993276cb80e675a88fcd3b39a0f39/README.recipes.md#recipe_modules-path
195[recipe_engine/recipe_modules/platform]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/3624a48ac89993276cb80e675a88fcd3b39a0f39/README.recipes.md#recipe_modules-platform
196[recipe_engine/recipe_modules/properties]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/3624a48ac89993276cb80e675a88fcd3b39a0f39/README.recipes.md#recipe_modules-properties
197[recipe_engine/recipe_modules/raw_io]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/3624a48ac89993276cb80e675a88fcd3b39a0f39/README.recipes.md#recipe_modules-raw_io
198[recipe_engine/recipe_modules/step]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/3624a48ac89993276cb80e675a88fcd3b39a0f39/README.recipes.md#recipe_modules-step
199[recipe_engine/wkt/RecipeApi]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/3624a48ac89993276cb80e675a88fcd3b39a0f39/recipe_engine/recipe_api.py#433
200