# Grpc.Tools MSBuild integration overview This is an overview for maintainers of Grpc.Tools. The Grpc.Tools NuGet package provides custom build targets to make it easier to specify `.proto` files in a project and for those files to be compiled and their generated files to be included in the project. # Files in the NuGet package ## .props and .target files MSBuild properties and targets included from the Grpc.Tools NuGet package are in: * `build\Grpc.Tools.props`, which imports * `build\_grpc\_Grpc.Tools.props` * `build\_protobuf\Google.Protobuf.Tools.props` * `build\Grpc.Tools.targets`, which imports * `build\_grpc\_Grpc.Tools.targets` * `build\_protobuf\Google.Protobuf.Tools.targets` Details of how NuGet packages can add custom build targets and properties to a project is documented here: [MSBuild .props and .targets in a package](https://learn.microsoft.com/en-us/nuget/concepts/msbuild-props-and-targets) Basically the `.props` and `.targets` files are automatically included in the projects - the `.props` at the top of the project and the `.targets` are added to the bottom of the project. ## Visual Studio property pages For Visual Studio integration - these files provide the properties pages: * `build\_protobuf\Protobuf.CSharp.xml` (included from `Google.Protobuf.Tools.targets`) * `build\_grpc\Grpc.CSharp.xml` (included from `_Grpc.Tools.targets`) ## Custom tasks DLLs DLLs containing the custom tasks are in: * `build\_protobuf\netstandard1.3` * `build\_protobuf\net45` ## Protobuf compiler and C# gRPC plugin binaries Native binary executables for the protobuf compiler (_protoc_) and C# gRPC plugin (_grpc_csharp_plugin_) are included in the NuGet package. Included are binaries for various OSes (Windows, Linux, macOS) and CPU architectures (x86, x64, arm64). The build determines which executables to use for the particular machine that the it is being run on. These can be overridden by specifying MSBuild properties or environment variables to give the paths to custom executables: * `Protobuf_ProtocFullPath` property or `PROTOBUF_PROTOC` environment variable \ Full path of protoc executable * `gRPC_PluginFullPath` property or `GRPC_PROTOC_PLUGIN` environment variable \ Full path of gRPC C# plugin # Grpc.Tools custom build targets ## Hooking the custom targets into the project build The custom targets hook into various places in a normal MSBuild build by specifying before/after targets at the relevant places. See [msbuild-targets](https://learn.microsoft.com/en-us/visualstudio/msbuild/msbuild-targets) for predefined targets. * Before `PrepareForBuild` * we add `Protobuf_SanityCheck` that checks this is a supported project type, e.g. a C# project. * Before `BeforeCompile` * we add all the targets that compile the `.proto` files and generate the expected `.cs` files. These files are added to those that get compiled by the C# compiler. * The target `_Protobuf_Compile_BeforeCsCompile` is the _glue_ inserting the targets into the build. It may look like it isn't doing anything but by specifying `BeforeTargets` and `DependsOnTargets` it inserts `Protobuf_Compile` into the build - but only doing so if this is a C# project. * After `CoreClean` * we add `Protobuf_Clean` that cleans the files generated by the protobuf compiler. * The target `_Protobuf_Clean_AfterCsClean` is the _glue_ inserting `Protobuf_Clean` into the build - but only doing so if this is a C# project. ## Custom tasks There are a few custom tasks needed by the targets. These are implemented in C# in the Grpc.Tools project and packaged in the file `Protobuf.MSBuild.dll` in the NuGet package. See [task writing](https://learn.microsoft.com/en-us/visualstudio/msbuild/task-writing) for information about implementing custom tasks. * ProtoToolsPlatform * Works out the operating system and CPU architecture * ProtoCompilerOutputs * Tries to work out the names and paths of the files that would be generated by the protobuf compiler and returns these as a list of items * Also returns list of items that is the same as the `Protobuf` items passed in with the output directory metadata updated * ProtoReadDependencies * Read generated files from previously written dependencies file and return as items. * ProtoCompile * Runs the protobuf compiler for a proto file. The executable to run is specified by the property `Protobuf_ProtocFullPath` * To do this it: * first writes out a response file containing the parameters for protobuf compiler * runs the executable, which generates the `.cs` files and a `.protodep` dependencies file * reads the dependencies file to find the files that were generated and these are returned as a list of _items_ that can then be used in the MSBuild targets ## Build steps The names of these items and properties are correct at the time of writing this document. High level builds steps: * Prepare list of `.proto` files to compile * Makes sure all needed metadata is set for the `` item, defaulting some values * Removes `` items that no longer exist or are marked as don't compile * Handling incremental builds * Work out files that need to be created or have changed * Compile the `.proto` files * Add generated files to the list of files for the C# compiler ### Prepare the list of .proto files to compile At various stages of the build copies of the original `` items are created and/or updated to set metadata and to prune out unwanted items. Firstly, the build makes sure `ProtoRoot` metadata is set for all `Protobuf` items. A new list of items - `Protobuf_Rooted` - is created from the `Protobuf` items with `ProtoRoot` metadata set: * If `ProtoRoot` already set in the `` item in the project file then it is left as-is. * If the `.proto` file is under the project's directory then set `ProtoRoot="."`. * If the `.proto` file is outside of the project's directory then set `ProtoRoot=""`. Now prune out from `Protobuf_Rooted` the items that the user doesn't want to compile - those don't have `ProtoCompile` metadata as `true`. The pruned list is now called `Protobuf_Compile`. Set the `Source` metadata on `Protobuf_Compile` items to be the name of the `.proto` file. The `Source` metadata is used later as a key to map generated files to `.proto` files. ### Handling incremental builds #### Gathering files to check for incremental builds The target `Protobuf_PrepareCompile` tries to work out which files the protobuf compiler will generate without actually calling the protobuf compiler. This is a best-effort guess. The custom task `ProtoCompilerOutputs` is called to do this. The results are stored in the item list `Protobuf_ExpectedOutputs`. The target `Protobuf_PrepareCompile` also reads previously written `.protodep` files to get any actual files previously generated. The custom task `ProtoReadDependencies` is called to do this. The results are stored in the item list `Protobuf_Dependencies`. This is in case the list of actual files is different from the previous best-effort guess from `ProtoCompilerOutputs`. The expected outputs and previous outputs are needed so that the timestamps of those files can be checked later when handling an incremental build. #### Understanding incremental builds To avoid unnecessarily recompiling the `.proto` files during an incremental build the target `_Protobuf_GatherStaleBatched` tries to work out if any files have changed. It checks for out of date files using MSBuilds incremental build feature that compares the timestamps on a target's _Input_ files to its _Output_ files. See [How to: Build incrementally](https://learn.microsoft.com/en-us/visualstudio/msbuild/how-to-build-incrementally) The _Inputs_ that are checked are: * Timestamps of the `.proto` files * Timestamps of previously generated files (list of these files read from `.protodep` files) * Timestamps of MSBuild project files These are checked against the _Outputs_: * Timestamps of the expected generated files [MSBuild target batching](https://learn.microsoft.com/en-us/visualstudio/msbuild/item-metadata-in-target-batching) is used to check each `.proto` file against its expected outputs. The batching is done by specifying the `Source` metadata in the _Input_. Items where `Source` metadata matches in both input and output are in each batch. The target `_Protobuf_GatherStaleBatched` sets the metadata `_Exec=true` on `_Protobuf_OutOfDateProto` items that are out of date. Later in the target `_Protobuf_GatherStaleFiles`, the items in `_Protobuf_OutOfDateProto` that don't have metadata `_Exec==true` are removed from the list of items, leaving only those that need compiling. ### Compile the .proto files The target `_Protobuf_CoreCompile` is run for each `.proto` file that needs compiling. These are in the item list `_Protobuf_OutOfDateProto`. The custom task `ProtoCompile` is called to run the protobuf compiler. The files that were generated are returned in the item list `_Protobuf_GeneratedFiles`. If there are expected files that were not actually generated then the behaviour depends on whether the generated files should have been within the project (e.g. in the intermediate directories) or were specified to be outside of the project. * If within the project - empty files are created to prevent incremental builds doing unnecessary recompiles * If outside the project - by default empty files are not created and a warning is output (this behaviour is configurable) **TODO:** why are files inside and outside the project treated differently? ### Add generated .cs files to the list of files for the C# compiler The target `_Protobuf_AugmentLanguageCompile` adds to the `Compile` item list (the list of files that CSC compiles) the expected generated files. **Note** - this is the _expected_ files not the _actual_ generated files and this is done before the protobuf compiler is called. **TODO:** why are the _expected_ files not the _actual_ generated files added? ## Handling design time builds Design-time builds are special builds that Visual Studio uses to gather information about the project. They are not user-initiated but may be triggered whenever files are added, removed or saved. See [Design-Time Builds](https://github.com/dotnet/project-system/blob/main/docs/design-time-builds.md). The Grpc.Tools build targets used to try and optimise design time builds by disabling calling the protobuf compiler during a design time build. However this optimisation can lead to errors in Visual Studio because the generated files may not exist or be out of date and any code that relies on them will then have errors. Now design time builds behave exactly the same as a normal build. The old behaviour can be enabled by setting the setting `DisableProtobufDesignTimeBuild` property to `true` in the project file **_if_** it is a design time build, e.g. by adding ```xml true ``` ## Automatically including .proto files For SDK projects it is possible to automatically include `.proto` files found in the project directory or sub-directories, without having to specify them with a `` item. To do this the property `EnableDefaultProtobufItems` has be set to `true` in the project file. By default it is not set and `` items must be included in the project for the `.proto` files to be compiled.