1# Regenerating project files 2 3Prerequisites 4- `python` 5- `pip install mako` (the template processor) 6- `pip install pyyaml` (to read the yaml files) 7- `go` (required by boringssl dependency) 8 9``` 10# Regenerate the projects files (and other generated files) using templates 11tools/buildgen/generate_projects.sh 12``` 13 14# Quick justification 15 16We've approached the problem of the build system from a lot of different 17angles. The main issue was that there isn't a single build system that 18was going to single handedly cover all of our usage cases. 19 20So instead we decided to work the following way: 21 22* A `build.yaml` file at the root is the source of truth for listing all the 23targets and files needed to build grpc and its tests, as well as a basic system 24for dependency description. 25 26* Most of the build systems supported by gRPC (e.g. Makefile, cmake, XCode) have a template defined in this directory. The templates use the information from the `build.yaml` file to generate the project files specific to a given build system. 27 28This way we can maintain as many project system as we see fit, without having 29to manually maintain them when we add or remove new code to the repository. 30Only the structure of the project file is relevant to the template. The actual 31list of source code and targets isn't. 32 33# Structure of `build.yaml` 34 35The `build.yaml` file has the following structure: 36 37``` 38settings: # global settings, such as version number 39 ... 40filegroups: # groups of files that are automatically expanded 41 ... 42libs: # list of libraries to build 43 ... 44targets: # list of targets to build 45 ... 46``` 47 48The `filegroups` are helpful to re-use a subset of files in multiple targets. 49One `filegroups` entry has the following structure: 50 51``` 52- name: "arbitrary string", # the name of the filegroup 53 public_headers: # list of public headers defined in that filegroup 54 - ... 55 headers: # list of headers defined in that filegroup 56 - ... 57 src: # list of source files defined in that filegroup 58 - ... 59``` 60 61The `libs` collection contains the list of all the libraries we describe. Some may be 62helper libraries for the tests. Some may be installable libraries. Some may be 63helper libraries for installable binaries. 64 65The `targets` array contains the list of all the binary targets we describe. Some may 66be installable binaries. 67 68One `libs` or `targets` entry has the following structure (see below for 69details): 70 71``` 72name: "arbitrary string", # the name of the library 73build: "build type", # in which situation we want that library to be 74 # built and potentially installed (see below). 75language: "...", # the language tag; "c" or "c++" 76public_headers: # list of public headers to install 77headers: # list of headers used by that target 78src: # list of files to compile 79baselib: boolean, # this is a low level library that has system 80 # dependencies 81filegroups: # list of filegroups to merge to that project 82 # note that this will be expanded automatically 83deps: # list of libraries this target depends on 84dll: "..." # see below. 85``` 86 87## The `"build"` tag 88 89Currently, the "`build`" tag have these meanings: 90 91* `"all"`: library to build on `"make all"`, and install on the system. 92* `"plugin"`: library to build on `"make all"`, and install, but the corresponding CMake option defaults to off. The option needs to be declared manually at present to allow depending on third-party dependencies. 93* `"protoc"`: a protoc plugin to build on `"make all"` and install on the system. 94* `"plugin_test"`: A test that should only be built if the associated plugin is enabled. The plugin is mentioned in the `"plugin_option"` tag. 95* `"private"`: a library to only build for tests. 96* `"test"`: a test binary to run on `"make test"`. 97* `"tool"`: a binary to be built upon `"make tools"`. 98 99All of the targets should always be present in the generated project file, if 100possible and applicable. But the build tag is what should group the targets 101together in a single build command. 102 103## The `"baselib"` boolean 104 105This means this is a library that will provide most of the features for gRPC. 106In particular, if we're locally building OpenSSL, protobuf or zlib, then we 107should merge OpenSSL, protobuf or zlib inside that library. That effect depends 108on the `"language"` tag. OpenSSL and zlib are for `"c"` libraries, while 109protobuf is for `"c++"` ones. 110 111# The template system 112 113We're currently using the [mako templates](http://www.makotemplates.org/) 114renderer. That choice enables us to simply render text files without dragging 115with us a lot of other features. Feel free to explore the current templates 116in that directory. 117 118## The renderer engine 119 120As mentioned, the renderer is using [mako templates](http://www.makotemplates.org/), 121but some glue is needed to process all of that. See the [buildgen folder](../tools/buildgen) 122for more details. We're mainly loading the build.json file, and massaging it, 123in order to get the list of properties we need, into a Python dictionary, that 124is then passed to the template while rending it. 125 126## The plugins 127 128The file build.json itself isn't passed straight to the template files. It is 129first processed and modified by a few plugins. For example, the version 130expander is [a plugin](../tools/buildgen/plugins/expand_version.py). 131 132The structure of a plugin is simple. The plugin must defined the function 133`mako_plugin` that takes a Python dictionary. That dictionary represents the 134current state of the build.json contents. The plugin can alter it to whatever 135feature it needs to add. 136