1.. _module-pw_snapshot-design_discussion: 2 3================= 4Design Discussion 5================= 6There were a handful of key requirements going into the design of pw_snapshot: 7 8* **Pre-established file format** - Building and maintaining tooling to support 9 parsing binary snapshot data is a high maintenance burden that detracts from 10 the appeal of a pre-existing widely known/supported format. 11* **Incremental writing** - Needing to build an entire snapshot before 12 committing it as a finished file is a big limitation on embedded devices where 13 RAM is often very constrained. It is important that a snapshot can be built in 14 smaller in-memory segments that can be committed incrementally to a larger 15 sink (e.g. UART, off-chip flash). 16* **Extensible** - Pigweed doesn't know everything users might want to capture 17 in a snapshot. It's important that users have ways to include their own 18 information into snapshots with minimal friction. 19* **Relatively compact** - It's important that snapshots can contain useful 20 information even when they are limited to a few hundred bytes in size. 21 22Why Proto? 23========== 24Protobufs are widely used and supported across many languages and platforms. 25This greatly reduces the encode/decode tooling maintenance introduced by using 26custom or unstructured formats. While using a format like JSON provides 27similarly wide tooling support, encoding the same information as a proto 28significantly reduces the final file size. 29 30While protobuffer messages aren't truly streamable (i.e. can be written without 31any intermediate buffers) due to how message nesting works, a large message can 32be incrementally written as long as there's enough buffer space for encoding the 33largest single sub-message in the proto. 34 35Why overlay multiple protos? 36============================ 37Proto 2 supported a feature called "extensions" that explicitly allowed this 38behavior. While proto 3 removed this feature, it doesn't disallow the old 39behavior of serializing two 'overlayed' protos to the same data stream. Proto 3 40recommends using an "Any" proto instead of extensions, as it is more explicit 41and eliminates the issue of collisions in proto messages. Unfortunately, proto 42'Any' messages introduce unacceptable overhead. For a single integer that would 43encode to a few bytes using extensions, an Any submessage quickly expands to 44tens of bytes. 45 46pw_snapshot's proto format takes advantage of "extensions" from proto 2 without 47explicitly relying on the feature. To reduce the risk of colissions and maximize 48encoding efficiency, certain ranges are reserved to allow Pigweed to grow while 49ensuring downstream customers have equivalent flexibility when using the 50Snapshot proto format. 51 52Why no file header? 53=================== 54Right now it's assumed that anything that is storing or transferring a 55serialized snapshot implicitly tracks its size (and a checksum, if desired). 56While a container format might be introduced independently, pw_snapshot focuses 57on treating an encoded snapshot as raw serialized proto data. 58