1.. _module-pw_stream: 2 3.. cpp:namespace-push:: pw::stream 4 5========= 6pw_stream 7========= 8``pw_stream`` provides a foundational interface for streaming data from one part 9of a system to another. In the simplest use cases, this is basically a memcpy 10behind a reusable interface that can be passed around the system. On the other 11hand, the flexibility of this interface means a ``pw_stream`` could terminate is 12something more complex, like a UART stream or flash memory. 13 14-------- 15Overview 16-------- 17At the most basic level, ``pw_stream``'s interfaces provide very simple handles 18to enabling streaming data from one location in a system to an endpoint. 19 20Example: 21 22.. code-block:: cpp 23 24 Status DumpSensorData(pw::stream::Writer& writer) { 25 static char temp[64]; 26 ImuSample imu_sample; 27 imu.GetSample(&info); 28 size_t bytes_written = imu_sample.AsCsv(temp, sizeof(temp)); 29 return writer.Write(temp, bytes_written); 30 } 31 32In this example, ``DumpSensorData()`` only cares that it has access to a 33:cpp:class:`Writer` that it can use to stream data to using ``Writer::Write()``. 34The :cpp:class:`Writer` itself can be backed by anything that can act as a data 35"sink." 36 37--------------------- 38pw::stream Interfaces 39--------------------- 40There are three basic capabilities of a stream: 41 42* Reading -- Bytes can be read from the stream. 43* Writing -- Bytes can be written to the stream. 44* Seeking -- The position in the stream can be changed. 45 46``pw_stream`` provides a family of stream classes with different capabilities. 47The most basic class, :cpp:class:`Stream` guarantees no functionality, while the 48most capable class, :cpp:class:`SeekableReaderWriter` supports reading, writing, 49and seeking. 50 51Usage overview 52============== 53.. list-table:: 54 :header-rows: 1 55 56 * - pw::stream Interfaces 57 - Accept in APIs? 58 - Extend to create new stream? 59 * - :cpp:class:`pw::stream::Stream` 60 - ❌ 61 - ❌ 62 * - | :cpp:class:`pw::stream::Reader` 63 | :cpp:class:`pw::stream::Writer` 64 | :cpp:class:`pw::stream::ReaderWriter` 65 - ✅ 66 - ❌ 67 * - | :cpp:class:`pw::stream::SeekableReader` 68 | :cpp:class:`pw::stream::SeekableWriter` 69 | :cpp:class:`pw::stream::SeekableReaderWriter` 70 - ✅ 71 - ✅ 72 * - | :cpp:class:`pw::stream::RelativeSeekableReader` 73 | :cpp:class:`pw::stream::RelativeSeekableWriter` 74 | :cpp:class:`pw::stream::RelativeSeekableReaderWriter` 75 - ✅ (rarely) 76 - ✅ 77 * - | :cpp:class:`pw::stream::NonSeekableReader` 78 | :cpp:class:`pw::stream::NonSeekableWriter` 79 | :cpp:class:`pw::stream::NonSeekableReaderWriter` 80 - ❌ 81 - ✅ 82 83 84Interface documentation 85======================= 86Summary documentation for the ``pw_stream`` interfaces is below. See the API 87comments in `pw_stream/public/pw_stream/stream.h 88<https://cs.pigweed.dev/pigweed/+/main:pw_stream/public/pw_stream/stream.h>`_ 89for full details. 90 91.. doxygenclass:: pw::stream::Stream 92 :members: 93 :private-members: 94 95Reader interfaces 96----------------- 97.. doxygenclass:: pw::stream::Reader 98 :members: 99 100.. doxygenclass:: pw::stream::SeekableReader 101 :members: 102 103.. doxygenclass:: pw::stream::RelativeSeekableReader 104 :members: 105 106.. doxygenclass:: pw::stream::NonSeekableReader 107 :members: 108 109Writer interfaces 110----------------- 111.. doxygenclass:: pw::stream::Writer 112 :members: 113 114.. doxygenclass:: pw::stream::SeekableWriter 115 :members: 116 117.. doxygenclass:: pw::stream::RelativeSeekableWriter 118 :members: 119 120.. doxygenclass:: pw::stream::NonSeekableWriter 121 :members: 122 123 124ReaderWriter interfaces 125----------------------- 126.. doxygenclass:: pw::stream::ReaderWriter 127 :members: 128 129.. doxygenclass:: pw::stream::SeekableReaderWriter 130 :members: 131 132.. doxygenclass:: pw::stream::RelativeSeekableReaderWriter 133 :members: 134 135.. doxygenclass:: pw::stream::NonSeekableReaderWriter 136 :members: 137 138--------------- 139Implementations 140--------------- 141``pw_stream`` includes a few stream implementations for general use. 142 143.. cpp:class:: MemoryWriter : public SeekableWriter 144 145 The ``MemoryWriter`` class implements the :cpp:class:`Writer` interface by 146 backing the data destination with an **externally-provided** memory buffer. 147 ``MemoryWriterBuffer`` extends ``MemoryWriter`` to internally provide a memory 148 buffer. 149 150 The ``MemoryWriter`` can be accessed like a standard C++ container. The 151 contents grow as data is written. 152 153.. cpp:class:: MemoryReader : public SeekableReader 154 155 The ``MemoryReader`` class implements the :cpp:class:`Reader` interface by 156 backing the data source with an **externally-provided** memory buffer. 157 158.. cpp:class:: NullStream : public SeekableReaderWriter 159 160 ``NullStream`` is a no-op stream implementation, similar to ``/dev/null``. 161 Writes are always dropped. Reads always return ``OUT_OF_RANGE``. Seeks have no 162 effect. 163 164.. cpp:class:: CountingNullStream : public SeekableReaderWriter 165 166 ``CountingNullStream`` is a no-op stream implementation, like 167 :cpp:class:`NullStream`, that counts the number of bytes written. 168 169 .. cpp:function:: size_t bytes_written() const 170 171 Returns the number of bytes provided to previous ``Write()`` calls. 172 173.. cpp:class:: StdFileWriter : public SeekableWriter 174 175 ``StdFileWriter`` wraps an ``std::ofstream`` with the :cpp:class:`Writer` 176 interface. 177 178.. cpp:class:: StdFileReader : public SeekableReader 179 180 ``StdFileReader`` wraps an ``std::ifstream`` with the :cpp:class:`Reader` 181 interface. 182 183.. cpp:class:: SocketStream : public NonSeekableReaderWriter 184 185 ``SocketStream`` wraps posix-style TCP sockets with the :cpp:class:`Reader` 186 and :cpp:class:`Writer` interfaces. It can be used to connect to a TCP server, 187 or to communicate with a client via the ``ServerSocket`` class. 188 189.. cpp:class:: ServerSocket 190 191 ``ServerSocket`` wraps a posix server socket, and produces a 192 :cpp:class:`SocketStream` for each accepted client connection. 193 194------------------ 195Why use pw_stream? 196------------------ 197 198Standard API 199============ 200``pw_stream`` provides a standard way for classes to express that they have the 201ability to write data. Writing to one sink versus another sink is a matter of 202just passing a reference to the appropriate :cpp:class:`Writer`. 203 204As an example, imagine dumping sensor data. If written against a random HAL 205or one-off class, there's porting work required to write to a different sink 206(imagine writing over UART vs dumping to flash memory). Building a "dumping" 207implementation against the :cpp:class:`Writer` interface prevents a dependency 208on a bespoke API that would require porting work. 209 210Similarly, after building a :cpp:class:`Writer` implementation for a Sink that 211data could be dumped to, that same :cpp:class:`Writer` can be reused for other 212contexts that already write data to the :cpp:class:`pw::stream::Writer` 213interface. 214 215Before: 216 217.. code-block:: cpp 218 219 // Not reusable, depends on `Uart`. 220 void DumpSensorData(Uart& uart) { 221 static char temp[64]; 222 ImuSample imu_sample; 223 imu.GetSample(&info); 224 size_t bytes_written = imu_sample.AsCsv(temp, sizeof(temp)); 225 uart.Transmit(temp, bytes_written, /*timeout_ms=*/ 200); 226 } 227 228After: 229 230.. code-block:: cpp 231 232 // Reusable; no more Uart dependency! 233 Status DumpSensorData(Writer& writer) { 234 static char temp[64]; 235 ImuSample imu_sample; 236 imu.GetSample(&info); 237 size_t bytes_written = imu_sample.AsCsv(temp, sizeof(temp)); 238 return writer.Write(temp, bytes_written); 239 } 240 241Reduce intermediate buffers 242=========================== 243Often functions that write larger blobs of data request a buffer is passed as 244the destination that data should be written to. This *requires* a buffer to be 245allocated, even if the data only exists in that buffer for a very short period 246of time before it's written somewhere else. 247 248In situations where data read from somewhere will immediately be written 249somewhere else, a :cpp:class:`Writer` interface can cut out the middleman 250buffer. 251 252Before: 253 254.. code-block:: cpp 255 256 // Requires an intermediate buffer to write the data as CSV. 257 void DumpSensorData(Uart& uart) { 258 char temp[64]; 259 ImuSample imu_sample; 260 imu.GetSample(&info); 261 size_t bytes_written = imu_sample.AsCsv(temp, sizeof(temp)); 262 uart.Transmit(temp, bytes_written, /*timeout_ms=*/ 200); 263 } 264 265After: 266 267.. code-block:: cpp 268 269 // Both DumpSensorData() and RawSample::AsCsv() use a Writer, eliminating the 270 // need for an intermediate buffer. 271 Status DumpSensorData(Writer& writer) { 272 RawSample imu_sample; 273 imu.GetSample(&info); 274 return imu_sample.AsCsv(writer); 275 } 276 277Prevent buffer overflow 278======================= 279When copying data from one buffer to another, there must be checks to ensure the 280copy does not overflow the destination buffer. As this sort of logic is 281duplicated throughout a codebase, there's more opportunities for bound-checking 282bugs to sneak in. ``Writers`` manage this logic internally rather than pushing 283the bounds checking to the code that is moving or writing the data. 284 285Similarly, since only the :cpp:class:`Writer` has access to any underlying 286buffers, it's harder for functions that share a :cpp:class:`Writer` to 287accidentally clobber data written by others using the same buffer. 288 289Before: 290 291.. code-block:: cpp 292 293 Status BuildPacket(Id dest, span<const std::byte> payload, 294 span<std::byte> dest) { 295 Header header; 296 if (dest.size_bytes() + payload.size_bytes() < sizeof(Header)) { 297 return Status::ResourceExhausted(); 298 } 299 header.dest = dest; 300 header.src = DeviceId(); 301 header.payload_size = payload.size_bytes(); 302 303 memcpy(dest.data(), &header, sizeof(header)); 304 // Forgetting this line would clobber buffer contents. Also, using 305 // a temporary span instead could leave `dest` to be misused elsewhere in 306 // the function. 307 dest = dest.subspan(sizeof(header)); 308 memcpy(dest.data(), payload.data(), payload.size_bytes()); 309 } 310 311After: 312 313.. code-block:: cpp 314 315 Status BuildPacket(Id dest, span<const std::byte> payload, Writer& writer) { 316 Header header; 317 header.dest = dest; 318 header.src = DeviceId(); 319 header.payload_size = payload.size_bytes(); 320 321 writer.Write(header); 322 return writer.Write(payload); 323 } 324 325------------ 326Design notes 327------------ 328 329Sync & Flush 330============ 331The :cpp:class:`pw::stream::Stream` API does not include ``Sync()`` or 332``Flush()`` functions. There no mechanism in the :cpp:class:`Stream` API to 333synchronize a :cpp:class:`Reader`'s potentially buffered input with its 334underlying data source. This must be handled by the implementation if required. 335Similarly, the :cpp:class:`Writer` implementation is responsible for flushing 336any buffered data to the sink. 337 338``Flush()`` and ``Sync()`` were excluded from :cpp:class:`Stream` for a few 339reasons: 340 341* The semantics of when to call ``Flush()``/``Sync()`` on the stream are 342 unclear. The presence of these methods complicates using a :cpp:class:`Reader` 343 or :cpp:class:`Writer`. 344* Adding one or two additional virtual calls increases the size of all 345 :cpp:class:`Stream` vtables. 346 347.. _module-pw_stream-class-hierarchy: 348 349Class hierarchy 350=============== 351All ``pw_stream`` classes inherit from a single, common base with all possible 352functionality: :cpp:class:`pw::stream::Stream`. This structure has 353some similarities with Python's `io module 354<https://docs.python.org/3/library/io.html>`_ and C#'s `Stream class 355<https://docs.microsoft.com/en-us/dotnet/api/system.io.stream>`_. 356 357An alternative approach is to have the reading, writing, and seeking portions of 358the interface provided by different entities. This is how Go's `io 359package <https://pkg.go.dev/io>`_ and C++'s `input/output library 360<https://en.cppreference.com/w/cpp/io>`_ are structured. 361 362We chose to use a single base class for a few reasons: 363 364* The inheritance hierarchy is simple and linear. Despite the linear 365 hierarchy, combining capabilities is natural with classes like 366 :cpp:class:`ReaderWriter`. 367 368 In C++, separate interfaces for each capability requires either a complex 369 virtual inheritance hierarchy or entirely separate hierarchies for each 370 capability. Separate hierarchies can become cumbersome when trying to 371 combine multiple capabilities. A :cpp:class:`SeekableReaderWriter` would 372 have to implement three different interfaces, which means three different 373 vtables and three vtable pointers in each instance. 374* Stream capabilities are clearly expressed in the type system, while 375 naturally supporting optional functionality. A :cpp:class:`Reader` may 376 or may not support :cpp:func:`Stream::Seek`. Applications that can handle 377 seek failures gracefully way use seek on any :cpp:class:`Reader`. If seeking 378 is strictly necessary, an API can accept a :cpp:class:`SeekableReader` 379 instead. 380 381 Expressing optional functionality in the type system is cumbersome when 382 there are distinct interfaces for each capability. ``Reader``, ``Writer``, 383 and ``Seeker`` interfaces would not be sufficient. To match the flexibility 384 of the current structure, there would have to be separate optional versions 385 of each interface, and classes for various combinations. :cpp:class:`Stream` 386 would be an "OptionalReaderOptionalWriterOptionalSeeker" in this model. 387* Code reuse is maximized. For example, a single 388 :cpp:func:`Stream::ConservativeLimit` implementation supports many stream 389 implementations. 390 391Virtual interfaces 392================== 393``pw_stream`` uses virtual functions. Virtual functions enable runtime 394polymorphism. The same code can be used with any stream implementation. 395 396Virtual functions have inherently has more overhead than a regular function 397call. However, this is true of any polymorphic API. Using a C-style ``struct`` 398of function pointers makes different trade-offs but still has more overhead than 399a regular function call. 400 401For many use cases, the overhead of virtual calls insignificant. However, in 402some extremely performance-sensitive contexts, the flexibility of the virtual 403interface may not justify the performance cost. 404 405Asynchronous APIs 406================= 407At present, ``pw_stream`` is synchronous. All :cpp:class:`Stream` API calls are 408expected to block until the operation is complete. This might be undesirable 409for slow operations, like writing to NOR flash. 410 411Pigweed has not yet established a pattern for asynchronous C++ APIs. The 412:cpp:class:`Stream` class may be extended in the future to add asynchronous 413capabilities, or a separate ``AsyncStream`` could be created. 414 415.. cpp:namespace-pop:: 416 417------------ 418Dependencies 419------------ 420* :ref:`module-pw_assert` 421* :ref:`module-pw_preprocessor` 422* :ref:`module-pw_status` 423* :ref:`module-pw_span` 424 425------ 426Zephyr 427------ 428To enable ``pw_stream`` for Zephyr add ``CONFIG_PIGWEED_STREAM=y`` to the 429project's configuration. 430 431---- 432Rust 433---- 434Pigweed centric analogs to Rust ``std``'s ``Read``, ``Write``, ``Seek`` traits 435as well as a basic ``Cursor`` implementation are provided by the 436`pw_stream crate </rustdoc/pw_stream/>`_. 437 438 439.. toctree:: 440 :hidden: 441 :maxdepth: 1 442 443 Backends <backends> 444 445------ 446Python 447------ 448There are legacy Python utilities used for reading and writing a serial device 449for RPC purposes. 450 451.. toctree:: 452 :hidden: 453 454 Python <py/docs> 455