1# Grpc.Tools MSBuild integration overview 2 3This is an overview for maintainers of Grpc.Tools. 4 5The Grpc.Tools NuGet package provides custom build targets to make it easier to specify `.proto` files 6in a project and for those files to be compiled and their generated files to be included in the project. 7 8# Files in the NuGet package 9 10## .props and .target files 11 12MSBuild properties and targets included from the Grpc.Tools NuGet package are in: 13 14* `build\Grpc.Tools.props`, which imports 15 * `build\_grpc\_Grpc.Tools.props` 16 * `build\_protobuf\Google.Protobuf.Tools.props` 17* `build\Grpc.Tools.targets`, which imports 18 * `build\_grpc\_Grpc.Tools.targets` 19 * `build\_protobuf\Google.Protobuf.Tools.targets` 20 21Details of how NuGet packages can add custom build targets and properties to a project is documented 22here: [MSBuild .props and .targets in a package](https://learn.microsoft.com/en-us/nuget/concepts/msbuild-props-and-targets) 23 24Basically the `.props` and `.targets` files are automatically included in the projects - the `.props` at the top 25of the project and the `.targets` are added to the bottom of the project. 26 27## Visual Studio property pages 28 29For Visual Studio integration - these files provide the properties pages: 30 31* `build\_protobuf\Protobuf.CSharp.xml` (included from `Google.Protobuf.Tools.targets`) 32* `build\_grpc\Grpc.CSharp.xml` (included from `_Grpc.Tools.targets`) 33 34## Custom tasks DLLs 35 36DLLs containing the custom tasks are in: 37 38* `build\_protobuf\netstandard1.3` 39* `build\_protobuf\net45` 40 41## Protobuf compiler and C# gRPC plugin binaries 42 43Native binary executables for the protobuf compiler (_protoc_) and C# gRPC plugin (_grpc_csharp_plugin_) are 44included in the NuGet package. Included are binaries for various OSes (Windows, Linux, macOS) and 45CPU architectures (x86, x64, arm64). 46 47The build determines which executables to use for the particular machine that the it is being run on. 48These can be overridden by specifying MSBuild properties or environment variables to give the paths to custom executables: 49 50* `Protobuf_ProtocFullPath` property or `PROTOBUF_PROTOC` environment variable \ 51Full path of protoc executable 52* `gRPC_PluginFullPath` property or `GRPC_PROTOC_PLUGIN` environment variable \ 53Full path of gRPC C# plugin 54 55# Grpc.Tools custom build targets 56 57## Hooking the custom targets into the project build 58 59The custom targets hook into various places in a normal MSBuild build by specifying 60before/after targets at the relevant places. See 61[msbuild-targets](https://learn.microsoft.com/en-us/visualstudio/msbuild/msbuild-targets) 62for predefined targets. 63 64* Before `PrepareForBuild` 65 * we add `Protobuf_SanityCheck` that checks this is a supported project type, e.g. a C# project. 66* Before `BeforeCompile` 67 * we add all the targets that compile the `.proto` files and generate the expected `.cs` files. These files are added to those that get compiled by the C# compiler. 68 * The target `_Protobuf_Compile_BeforeCsCompile` is the _glue_ inserting the targets into the build. 69 It may look like it isn't doing anything but by specifying `BeforeTargets` and `DependsOnTargets` it inserts `Protobuf_Compile` into the build - but only doing so if this is a C# project. 70* After `CoreClean` 71 * we add `Protobuf_Clean` that cleans the files generated by the protobuf compiler. 72 * The target `_Protobuf_Clean_AfterCsClean` is the _glue_ inserting `Protobuf_Clean` into the build - but only doing so if this is a C# project. 73 74## Custom tasks 75 76There are a few custom tasks needed by the targets. These are implemented in C# in the Grpc.Tools project 77and packaged in the file `Protobuf.MSBuild.dll` in the NuGet package. See [task writing](https://learn.microsoft.com/en-us/visualstudio/msbuild/task-writing) for information about implementing custom tasks. 78 79* ProtoToolsPlatform 80 * Works out the operating system and CPU architecture 81* ProtoCompilerOutputs 82 * Tries to work out the names and paths of the files that would be generated by the protobuf compiler and returns these as a list of items 83 * Also returns list of items that is the same as the `Protobuf` items passed in with the output directory metadata updated 84* ProtoReadDependencies 85 * Read generated files from previously written dependencies file and return as items. 86* ProtoCompile 87 * Runs the protobuf compiler for a proto file. The executable to run is specified by the property `Protobuf_ProtocFullPath` 88 * To do this it: 89 * first writes out a response file containing the parameters for protobuf compiler 90 * runs the executable, which generates the `.cs` files and a `.protodep` dependencies file 91 * reads the dependencies file to find the files that were generated and these are returned as a list of _items_ that can then be used in the MSBuild targets 92 93## Build steps 94 95The names of these items and properties are correct at the time of writing this document. 96 97High level builds steps: 98 99* Prepare list of `.proto` files to compile 100 * Makes sure all needed metadata is set for the `<Protobuf>` item, defaulting some values 101 * Removes `<Protobuf>` items that no longer exist or are marked as don't compile 102* Handling incremental builds 103 * Work out files that need to be created or have changed 104* Compile the `.proto` files 105* Add generated files to the list of files for the C# compiler 106 107### Prepare the list of .proto files to compile 108 109At various stages of the build copies of the original `<Protobuf>` items are created 110and/or updated to set metadata and to prune out unwanted items. 111 112Firstly, the build makes sure `ProtoRoot` metadata is set for all `Protobuf` items. 113A new list of items - `Protobuf_Rooted` - is created from the `Protobuf` items with `ProtoRoot` metadata set: 114 115* If `ProtoRoot` already set in the `<Protobuf>` item in the project file then it is left as-is. 116* If the `.proto` file is under the project's directory then set `ProtoRoot="."`. 117* If the `.proto` file is outside of the project's directory then set `ProtoRoot="<relative path to project directory>"`. 118 119Now prune out from `Protobuf_Rooted` the items that the user doesn't want to compile - those don't have 120`ProtoCompile` metadata as `true`. The pruned list is now called `Protobuf_Compile`. 121 122Set the `Source` metadata on `Protobuf_Compile` items to be the name of the `.proto` file. 123The `Source` metadata is used later as a key to map generated files to `.proto` files. 124 125### Handling incremental builds 126 127#### Gathering files to check for incremental builds 128 129The target `Protobuf_PrepareCompile` tries to work out which files the protobuf compiler will 130generate without actually calling the protobuf compiler. This is a best-effort guess. 131The custom task `ProtoCompilerOutputs` is called to do this. The results are stored in the 132item list `Protobuf_ExpectedOutputs`. 133 134The target `Protobuf_PrepareCompile` also reads previously written `.protodep` files to get 135any actual files previously generated. The custom task `ProtoReadDependencies` is called to 136do this. The results are stored in the item list `Protobuf_Dependencies`. 137This is in case the list of actual files is different from the previous best-effort guess 138from `ProtoCompilerOutputs`. 139 140The expected outputs and previous outputs are needed so that the timestamps of those files 141can be checked later when handling an incremental build. 142 143#### Understanding incremental builds 144 145To avoid unnecessarily recompiling the `.proto` files during an incremental build the 146target `_Protobuf_GatherStaleBatched` tries to work out if any files have changed. 147 148It checks for out of date files using MSBuilds incremental build feature that compares the 149timestamps on a target's _Input_ files to its _Output_ files. 150See [How to: Build incrementally](https://learn.microsoft.com/en-us/visualstudio/msbuild/how-to-build-incrementally) 151 152The _Inputs_ that are checked are: 153* Timestamps of the `.proto` files 154* Timestamps of previously generated files (list of these files read from `.protodep` files) 155* Timestamps of MSBuild project files 156 157These are checked against the _Outputs_: 158* Timestamps of the expected generated files 159 160[MSBuild target batching](https://learn.microsoft.com/en-us/visualstudio/msbuild/item-metadata-in-target-batching) 161is used to check each `.proto` file against its expected outputs. The batching is done by specifying the `Source` 162metadata in the _Input_. Items where `Source` metadata matches in both input and output are in each batch. 163 164The target `_Protobuf_GatherStaleBatched` sets the metadata `_Exec=true` on `_Protobuf_OutOfDateProto` 165items that are out of date. 166 167Later in the target `_Protobuf_GatherStaleFiles`, the items in `_Protobuf_OutOfDateProto` that don't have 168metadata `_Exec==true` are removed from the list of items, leaving only those that need compiling. 169 170### Compile the .proto files 171 172The target `_Protobuf_CoreCompile` is run for each `.proto` file that needs compiling. 173These are in the item list `_Protobuf_OutOfDateProto`. The custom task `ProtoCompile` is called to run the 174protobuf compiler. The files that were generated are returned in the item list `_Protobuf_GeneratedFiles`. 175 176If there are expected files that were not actually generated then the behaviour depends on whether the 177generated files should have been within the project (e.g. in the intermediate directories) or were 178specified to be outside of the project. 179* If within the project - empty files are created to prevent incremental builds doing unnecessary recompiles 180* If outside the project - by default empty files are not created and a warning is output (this behaviour is configurable) 181 182**TODO:** why are files inside and outside the project treated differently? 183 184### Add generated .cs files to the list of files for the C# compiler 185 186The target `_Protobuf_AugmentLanguageCompile` adds to the `Compile` item list 187(the list of files that CSC compiles) the expected generated files. 188 189**Note** - this is the _expected_ files not the _actual_ generated files and this is done 190before the protobuf compiler is called. 191 192**TODO:** why are the _expected_ files not the _actual_ generated files added? 193 194## Handling design time builds 195 196Design-time builds are special builds that Visual Studio uses to gather information about the project. 197They are not user-initiated but may be triggered whenever files are added, removed or saved. 198See [Design-Time Builds](https://github.com/dotnet/project-system/blob/main/docs/design-time-builds.md). 199 200The Grpc.Tools build targets used to try and optimise design time builds by disabling calling the 201protobuf compiler during a design time build. However this optimisation can lead to errors in 202Visual Studio because the generated files may not exist or be out of date and any code that relies 203on them will then have errors. 204 205Now design time builds behave exactly the same as a normal build. 206The old behaviour can be enabled by setting the setting `DisableProtobufDesignTimeBuild` property 207to `true` in the project file **_if_** it is a design time build, e.g. by adding 208 209```xml 210<PropertyGroup Condition="'$(DesignTimeBuild)' == 'true' "> 211 <DisableProtobufDesignTimeBuild>true</DisableProtobufDesignTimeBuild> 212</PropertyGroup> 213``` 214 215## Automatically including .proto files 216 217For SDK projects it is possible to automatically include `.proto` files found in the project 218directory or sub-directories, without having to specify them with a `<Protobuf>` item. 219To do this the property `EnableDefaultProtobufItems` has be set to `true` in the project file. 220 221By default it is not set and `<Protobuf>` items must be included in the project for 222the `.proto` files to be compiled. 223