1.. _seed-0107: 2 3============================ 40107: Pigweed Communications 5============================ 6.. seed:: 7 :number: 107 8 :name: Communications 9 :status: Accepted 10 :proposal_date: 2023-07-19 11 :cl: 157090 12 :authors: Wyatt Hepler 13 :facilitator: Carlos Chinchilla 14 15------- 16Summary 17------- 18Pigweed does not currently offer an end-to-end solution for network 19communications. This SEED proposes that Pigweed adopt a new sockets API as its 20primary networking abstraction. The sockets API will be backed by a new, 21lightweight embedded-focused network protocol stack, inspired by the Internet 22protocol suite (TCP/IP). It will also support full TCP/IP via an open source 23embedded TCP/IP stack or OS sockets. The new communications APIs will support 24asynchronous use and zero-copy transmission. 25 26This work is comprised of the following subareas: 27 28- `Sockets API`_ 29- `Network protocol stack`_ 30- `Async`_ API pattern 31- `Buffer management`_ system 32 33The Pigweed team will revisit :ref:`pw_rpc <module-pw_rpc>` after deploying the 34sockets API and network protocol stack. 35 36---------- 37Background 38---------- 39Pigweed's primary communications system is :ref:`pw_rpc <module-pw_rpc>`. pw_rpc 40makes it possible to call functions and exchange data with a remote system. 41Requests and responses are encoded as protobufs. pw_rpc was initially deployed 42to a system with its own network protocol stack, so the Pigweed team did not 43invest in building a network stack of its own. 44 45The TCP/IP model, as described in `RFC 1122 46<https://datatracker.ietf.org/doc/html/rfc1122>`_, organizes communications 47systems and protocols into four layers: Application, Transport, Internet (or 48Network), and Link (or Network access). Pigweed's current communications 49offerings fit into the TCP/IP model as follows: 50 51+-----------------------+-----------------------------+ 52| TCP/IP Model | Pigweed Modules | 53+=======================+=============================+ 54| Application | | :ref:`module-pw_transfer` | 55| | | :ref:`module-pw_rpc` | 56+-----------------------+-----------------------------+ 57| Transport | | :ref:`module-pw_router` | 58+-----------------------+ | :ref:`module-pw_hdlc` | 59| Internet / Network | | 60+-----------------------+ | 61| Link / Network access | | 62+-----------------------+-----------------------------+ 63 64Notably, Pigweed provides little functionality below the application layer. The 65pw_router and pw_hdlc modules only implement a subset of features needed at 66their layer in the communications stack. 67 68Challenges deploying pw_rpc 69=========================== 70pw_rpc is application-layer communications module. It relies on a network layer 71to send packets between endpoints and doesn't provide any networking features 72itself. When initially developing pw_rpc, the Pigweed team focused its limited 73resources solely on this application-layer feature, which made it possible to 74deploy pw_rpc quickly to systems with existing networks. 75 76pw_rpc has been deployed to many projects with great results. However, since 77Pigweed does not provide a network stack, deploying pw_rpc to systems without 78existing stacks can be challenging. These systems have to develop their own 79solutions to transmit and route pw_rpc packets. 80 81As an example, one project based its network communications on Pigweed's 82:ref:`module-pw_hdlc` module. It used HDLC in a way more similar to IP, 83providing network-level addressing and features like quality-of-service. Source 84and destination addresses and ports were packed into the HDLC address field to 85facilitate routing and multiplexing. The :ref:`module-pw_router` module was 86developed to support static routing tables for HDLC frames through nodes in the 87system, and the :ref:`pw_transfer RPC service <module-pw_transfer>` was 88developed to provide reliable delivery of data. 89 90Learning from custom network stacks 91----------------------------------- 92Teams want to use Pigweed to build cool devices. Their goal isn't to build a 93network protocol stack, but they need one to use features like pw_rpc and 94pw_transfer. Given this, teams have little incentive to make the enormous time 95investment to develop a robust, reusable network stack. The practical approach 96is to assemble the minimum viable network stack from what's available. 97 98The Pigweed team has seen a few teams create custom network stacks for pw_rpc. 99While these projects were successful, their network stacks were not their 100primary focus. As a result, they had some shortcomings, including the following: 101 102- **Byte stuffing memory overhead** -- HDLC is a low-level protocol. It uses 103 `byte stuffing 104 <https://en.wikipedia.org/wiki/High-Level_Data_Link_Control#Asynchronous_framing>`_ 105 to ensure frame integrity across unreliable links. Byte stuffing makes sense 106 on the wire, but not in memory. Storing byte stuffed frames requires double 107 the memory to account for worst-case byte stuffing. Some projects use HDLC 108 frames as network layer packets, so they are buffered in memory for routing, 109 which requires more memory than necessary. 110- **HDLC protocol overhead** -- HDLC's frame recovery and integrity features are 111 not needed across all links. For example, these features are unnecessary for 112 Bluetooth. However, when projects use HDLC for both the network and link 113 layers, it has to be used across all links. 114- **pw_transfer at the application layer** -- :ref:`pw_transfer 115 <module-pw_transfer>` supports reliable data transfers with :ref:`pw_rpc 116 <module-pw_rpc>`. It required significant investment to develop, but since it 117 is layered on top of pw_rpc, it has additional overhead and limited 118 reusability. 119- **Custom routing** -- Some network nodes have multiple routes between them. 120 Projects have had to write custom, non-portable logic to handle routing. 121- **pw_rpc channel IDs in routing** -- Some projects used pw_rpc channel IDs as 122 a network addresses. Channel IDs were assigned for the whole network ahead of 123 time. This has several downsides: 124 125 - Requires nodes to have knowledge of the global channel ID assignments 126 and routes between them, which can be difficult to keep in sync. 127 - Implies that all traffic is pw_rpc packets. 128 - Requires decoding pw_rpc packets at lower levels of the network stack. 129 - Complicates runtime assignment of channel IDs. 130 131- **Flow control** -- Projects' communications stacks have not supported flow 132 control. The network layer simply has to drop packets it cannot process. 133 There is no mechanism to tell the producer to slow down or wait for the 134 receiver to be ready. 135- **Accounting for the MTU** -- HDLC and pw_rpc have variable overheads, so it 136 is difficult to know how much memory to allocate for RPC payloads. If packets 137 are not sized properly with respect to the maximum transmission unit (MTU), 138 packets may be silently dropped. 139 140Problem summary 141=============== 142These are the key issues of Pigweed's communications offerings based on the 143team's experiences deploying pw_rpc. 144 145**No cohesive full stack solution** 146 147Pigweed only provides a handful of communications modules. They were not 148designed to work together, and there is not enough to assemble a functioning 149network stack. Some projects have to create bespoke network protocols with 150limited reusability. 151 152**Layering violations** 153 154pw_transfer runs on top of pw_rpc instead of the transport layer, which adds 155overhead and prevents its use independent of pw_rpc. Using pw_rpc channels for 156routing ties the network to pw_rpc. Projects often use pw_hdlc for multiple 157network layers, which brings the encoding's overhead higher up the stack and 158across links that do not need it. 159 160**Inefficiency** 161 162Reliable data transfer requires pw_transfer, which runs on top of pw_rpc. This 163adds additional overhead and requires more CPU-intensive decoding operations. 164Using pw_rpc channel IDs in lower layers of the network requires expensive 165varint decodes, even when the packets are bound for other nodes. 166 167**Missing features** 168 169Each project has to develop its own version of common features, including: 170 171- **Addressing** -- There are no standard addressing schemes available to 172 Pigweed users. 173- **Routing** -- Projects must implement their own logic for routing packets, 174 which can be complex. 175- **Flow control** -- There is no way for the receiver to signal that it is ready 176 for more data or that it cannot receive any more, either at the protocol or 177 API level anywhere in the stack. Flow control is a crucial feature for 178 realistic networks with limited resources. 179- **Connections** -- Connections ensure the recipient is listening to 180 transmissions, and detect when the other end is no longer communicating. 181 pw_transfer maintains a connection, but it sits atop pw_rpc, so cannot be used 182 elsewhere. 183- **Quality of service (QoS)** -- Projects have developed basic QoS features in 184 HDLC, but there is no support in upstream Pigweed. Every project has to 185 develop its own custom implementation. 186 187----- 188Goals 189----- 190This SEED proposes a new communications system for Pigweed with the following 191goals: 192 193- **Practical end-to-end solution** -- Pigweed provides a full suite of APIs 194 and protocols that support simple and complex networking use cases. 195- **Robust, stable, and reliable** -- Pigweed communications "just work", even 196 under high load. The networking stack is thoroughly tested in both single and 197 multithreaded environments, with functional, load, fuzz, and performance 198 testing. Projects can easily test their own deployments with Pigweed tooling. 199- **Cohesive, yet modular** -- The network stack is holistically designed, but 200 modular. It is organized into layers that can be exchanged and configured 201 independently. Layering simplifies the stack, decouples protocol 202 implementations, and maximizes flexibility within a cohesive system. 203- **Efficient & performant** -- Pigweed’s network stack minimizes code size and 204 CPU usage. It provides for high throughput, low latency data transmission. 205 Memory allocation is configurable and adaptable to a project’s needs. 206- **Usable & easy to learn** -- Pigweed’s communications systems are backed by 207 thorough and up-to-date documentation. Getting started is easy using 208 Pigweed's tutorials and examples. 209 210-------- 211Proposal 212-------- 213Pigweed will unify its communications systems under a common sockets API. This 214entails the following: 215 216- **Sockets API** -- Pigweed will introduce a `sockets 217 API`_ to serve as its common networking interface. 218- **Lightweight protocol stack** -- Pigweed will provide a custom, 219 :ref:`lightweight network protocol stack <seed-0107-network-stack>` inspired 220 by IPv6, with UDP, TCP, and SCTP-like transport protocols. 221- **TCP/IP integration** -- Pigweed will offer sockets implementations for OS 222 sockets and an existing `embedded TCP/IP stack`_. 223- **Async** -- Pigweed will establish a new pattern for `async`_ programming and 224 use it in its networking APIs. 225- **Zero copy** -- Pigweed will develop a new `buffer management`_ system to 226 enable zero-copy networking. 227 228These features fit fit into the TCP/IP model as follows: 229 230+-------------------------------------+-------------------------------------+ 231| TCP/IP Model | Future Pigweed Comms Stack | 232+=====================================+=====================================+ 233| Application | | *Various modules including* | 234| | | *pw_rpc and pw_transfer.* | 235| | | 236| | | 237| | | 238+-------------------------------------+-------------------------------------+ 239| .. rst-class:: pw-text-center-align | .. rst-class:: pw-text-center-align | 240| | | 241| **OS Sockets** | **Pigweed Sockets** | 242+-------------------------------------+-------------------------------------+ 243| Transport | | UDP-like unreliable protocol | 244| | | TCP-like reliable protocol | 245| | | SCTP-like reliable protocol | 246+-------------------------------------+-------------------------------------+ 247| Network / Internet | | IPv6-like protocol | 248+-------------------------------------+-------------------------------------+ 249| Network access / Link | | HDLC | 250| | | others | 251+-------------------------------------+-------------------------------------+ 252 253Sockets API 254=========== 255The new sockets API will become the primary networking abstraction in Pigweed. 256The API will support the following: 257 258- Creating sockets for bidirectional communications with other nodes in the 259 network. 260- Opening and closing connections for connection-oriented socket types. 261- Sending and receiving data, optionally :ref:`asynchronously 262 <seed-0107-async>`. 263- Reporting errors. 264 265The sockets API will support runtime polymorphism. In C++, it will be a virtual 266interface. 267 268**Rationale** 269 270A network socket represents a bidirectional communications channel with another 271node, which could be local or across the Internet. Network sockets form the API 272between an application and the network. 273 274Sockets are a proven, well-understood concept. Socket APIs such as Berkeley / 275POSIX sockets are familiar to anyone with Internet programming experience. 276 277Sockets APIs hide the details of the network protocol stack. A socket provides 278well-defined semantics for a communications channel, but applications do not 279need to know how data is sent and received. The same API can be used to exchange 280data with another process on the same machine or with a device across the world. 281 282.. admonition:: Sockets SEEDs 283 284 The Pigweed sockets API is described in SEED-0116. The sockets API is based 285 on ``pw_channel``, which is proposed in SEED-0114. 286 287Socket types 288------------ 289Pigweed's sockets API will support the following sockets types. 290 291.. list-table:: 292 :header-rows: 1 293 294 * - Berkeley socket type 295 - Internet protocol 296 - Payload type 297 - Connection-oriented 298 - Guaranteed, ordered delivery 299 - Description 300 * - ``SOCK_DGRAM`` 301 - UDP 302 - Datagram 303 - ❌ 304 - ❌ 305 - Unreliable datagram 306 * - ``SOCK_STREAM`` 307 - TCP 308 - Byte stream 309 - ✅ 310 - ✅ 311 - Reliable byte stream 312 * - ``SOCK_SEQPACKET`` 313 - SCTP 314 - Datagram 315 - ✅ 316 - ✅ 317 - Reliable datagram 318 319Raw sockets (``SOCK_RAW``) may be supported in the future if required. 320``SOCK_CONN_DGRAM`` (unreliable connection-oriented datagram) sockets are 321uncommon and will not be supported. 322 323The socket's semantics will be expressed in the sockets API, e.g. with a 324different interface or class for each type. Instances of the connection-oriented 325socket types will be generated from a "listener" object. 326 327Pigweed's sockets API will draw inspiration from modern type safe APIs like 328Rust's `std::net sockets <https://doc.rust-lang.org/std/net/index.html>`_, 329rather than traditional APIs like POSIX sockets or Winsock. Pigweed sockets will 330map trivially to these APIs and implementations will be provided upstream. 331 332Using the sockets API 333--------------------- 334The Pigweed sockets API will provide the interface between applications and the 335network. Any application can open a socket to communicate across the network. 336A future revision of ``pw_rpc`` will use the sockets API in place of its current 337``Channel`` API. 338 339The sockets API will support both synchronous and :ref:`asynchronous 340<seed-0107-async>` use. The synchronous API may be built using the async API. 341It will also support :ref:`zero-copy <seed-0107-buffers>` data transmission. 342 343Addressing 344---------- 345The Pigweed sockets API will be aware of addresses. Addresses are used to refer 346to nodes in a network, including the socket's own node. With TCP/IP, the socket 347address includes an IP address and a port number. 348 349The POSIX sockets API supports different domains through address family 350constants such as ``AF_INET``, ``AF_INET6``, and ``AF_UNIX``. Addresses in these 351families are specified or accessed in various socket operations. Because the 352address format is not specified by the API, working with addresses is not type 353safe. 354 355Pigweed sockets will approach addressing differently, but details are yet to be 356determined. Possible approaches include: 357 358- Use IPv6 addresses exclusively. Systems with other addressing schemes map 359 these into IPv6 for use with Pigweed APIs. 360- Provide a polymorphic address class so sockets can work with addresses 361 generically. 362- Avoid addresses in the base sockets API. Instead, use implementation specific 363 derived classes to access addresses. 364 365Network protocol stack 366====================== 367The sockets API will be backed by a network protocol stack. Pigweed will provide 368sockets implementations for following network protocol stacks: 369 370* Third party embedded TCP/IP stack, most likely `lwIP 371 <https://savannah.nongnu.org/projects/lwip/>`_. 372* Operating system TCP/IP stack via POSIX sockets or `Winsock 373 <https://learn.microsoft.com/en-us/windows/win32/winsock/windows-sockets-start-page-2>`_. 374* Custom :ref:`lightweight network protocol stack <seed-0107-network-stack>`. 375 376Embedded TCP/IP stack 377--------------------- 378Pigweed will provide a sockets implementation for an embedded TCP/IP stack such 379as `lwIP <https://savannah.nongnu.org/projects/lwip/>`_. 380 381The sockets integration will be structured to avoid unnecessary dependencies on 382network stack features. For example, if a system is using IPv6 exclusively, the 383integration won't require IPv4 support, and the TCP/IP stack can be configured 384without it. 385 386**Rationale** 387 388The Internet protocol suite, or TCP/IP, is informed by decades of research and 389practical experience. It is much more than IP, TCP, and UDP; it's an alphabet 390soup of protocols that address a myriad of use cases and challenges. 391Implementing a functional TCP/IP stack is no small task. At time of writing, 392lwIP has about as many lines of C as Pigweed has C++ (excluding tests). 393 394The Pigweed team does not plan to implement a full TCP/IP stack. This is a major 395undertaking, and there are already established open source embedded TCP/IP 396stacks. Projects needing the full power of TCP/IP can use an embedded stack like 397`lwIP <https://savannah.nongnu.org/projects/lwip/>`_. 398 399Choosing between embedded TCP/IP and :ref:`Pigweed's stack <seed-0107-network-stack>` 400^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 401lwIP's `website <https://savannah.nongnu.org/projects/lwip/>`_ states that it 402requires tens of KB of RAM and about 40 KB of ROM. Using lwIP means using the 403same TCP/IP protocols that run the Internet. These protocols are feature rich, 404but have more overhead than is necessary for local communications within a small 405embedded system. 406 407Projects that can afford the resource requirements and protocol overhead of 408TCP/IP should use it. These projects can set up a local IPv4 or IPv6 network 409and use it for communications behind the Pigweed sockets API. Projects that 410cannot afford full TCP/IP can opt for Pigweed's :ref:`custom protocol stack 411<seed-0107-network-stack>`. Pigweed's custom stack will not have the depth of 412features and tooling of TCP/IP does, but will be sufficient for many systems. 413 414TCP/IP socket types 415^^^^^^^^^^^^^^^^^^^ 416With an embedded TCP/IP stack, the Pigweed sockets API will be implemented as 417follows: 418 419- Unreliable datagram (``SOCK_DGRAM``) -- UDP 420- Reliable byte stream (``SOCK_STREAM``) -- TCP 421- Reliable datagram (``SOCK_SEQPACKET``) -- Lightweight framing over TCP. This 422 will be semantically similar to `SCTP 423 <https://datatracker.ietf.org/doc/html/rfc9260>`_, but integrations will not 424 use SCTP since it is not widely supported. 425 426.. _seed-0107-network-stack: 427 428Pigweed's custom network protocol stack 429--------------------------------------- 430Pigweed will develop a custom, lightweight network protocol stack. 431 432This new protocol stack will be designed for small devices with relatively 433simple networks. It will scale to several interconnected cores that interface 434with a few external devices (e.g. over USB or Bluetooth). Depending on project 435requirements, it may or may not support dynamic network host configuration (e.g. 436DHCP or SLAAC). 437 438Pigweed's network protocol stack will be a strict subset of TCP/IP. This will 439include minimal, reduced overhead versions of UDP, TCP, and IPv6. Portions of 440other protocols such as ICMPv6 may be implemented as required. 441 442**Rationale** 443 444TCP/IP is too large and complex for some embedded systems. Systems for which 445TCP/IP is unnecessary can use Pigweed's lightweight embedded network protocol 446stack. 447 448Transport layer 449^^^^^^^^^^^^^^^ 450Pigweed will provide transport layer protocols that implement the semantics of 451``SOCK_DGRAM``, ``SOCK_STREAM``, and ``SOCK_SEQPACKET``-like sockets. 452 453- ``SOCK_DRAM``-like sockets will be backed by a UDP-like protocol. This will 454 add source and destination ports to the IP-style packets for multiplexing on 455 top of the network layer. 456- ``SOCK_STREAM``-like sockets will be backed by a TCP-like protocol that uses 457 network layer packets to implement a reliable byte stream. It will be based on 458 TCP, but will not implement all of its features. The :ref:`module-pw_transfer` 459 module may serve as a starting point for the new protocol implementation. 460- ``SOCK_SEQPACKET``-like sockets will be implemented with a simple 461 message-oriented protocol on top of the TCP-like protocol. Applications like 462 pw_rpc will use ``SOCK_SEQPACKET`` sockets. 463 464Network layer 465^^^^^^^^^^^^^ 466Pigweed will create a new network-layer protocol closely based on IPv6. Details 467are still to be determined, but the protocol is intended to be a strict subset 468of IPv6 and related protocols (e.g. ICMP, NDP) as needed. If a need arises, it 469is met by implementing the associated IP suite protocol. Packets will use 470compressed version of an IPv6 header (e.g. omit fields, use smaller addresses). 471 472This protocol will provide: 473 474- Unreliable packet delivery between source and destination. 475- Routing based on the source and destination addresses. 476- Quality of service (e.g. via the traffic class field). 477 478Packets may be routed at this layer independently of the link layer. Wire format 479details stay on the wire. 480 481Network access / link layer 482^^^^^^^^^^^^^^^^^^^^^^^^^^^ 483Pigweed's network stack will interact with the link layer through a generic 484interface. This will allow Pigweed to send network packets with any protocol 485over any physical interface. 486 487Pigweed already provides minimal support for one link layer protocol, HDLC. 488Other protocols (e.g. COBS, PPP) may be implemented. Some hardware interfaces 489(e.g. Bluetooth, USB) may not require an additional link-layer protocol. 490 491Language support 492---------------- 493Pigweed today is primarily C++, but it supports Rust, C, Python, TypeScript, and 494Java to varying extents. 495 496Pigweed’s communications stack will be developed in either C++ or Rust to start, 497but it will be ported to all supported languages in time. The stack may have C 498APIs to facilitate interoperability between C++ and Rust. 499 500.. admonition:: Network protocol stack SEED 501 502 Pigweed's network protocol stack will be explored in an upcoming SEED. 503 504.. _seed-0107-async: 505 506Async 507===== 508Pigweed will develop a model for asynchronous programming and use it in its 509networking APIs, including sockets. Sockets will also support synchronous 510operations, but these may be implemented in terms of the asynchronous API. 511 512The Pigweed async model has not been designed yet. The :ref:`pw_async 513<module-pw_async>` module has a task dispatcher, but the pattern for async APIs 514has not been established. Further exploration is needed, but C++20 coroutines 515may be used for Pigweed async APIs where supported. 516 517**Rationale** 518 519Synchronous APIs require the thread to block while an operation completes. The 520thread and its resources cannot be used by the system until the task completes. 521Async APIs allow a single thread to handle multiple simultaneous tasks. The 522thread advances tasks until they need to wait for an external operation to 523complete, then switches to another task to avoid blocking. 524 525Threads are expensive in embedded systems. Each thread requires significant 526memory for its stack and kernel structures for bookkeeping. They occupy this 527memory all the time, even when they are not running. Furthermore, context 528switches between threads can take significant CPU time. 529 530Asynchronous programming avoids these downsides. Many asynchronous threads run 531on a single thread. Fewer threads are needed, and the resources of one thread 532are shared by multiple tasks. Since asynchronous systems run within one thread, 533no thread context switches occur. 534 535Networking involves many asynchronous tasks. For example, waiting for data to be 536sent through a network interface, for a connection request, or for data to 537arrive on one or more interfaces are all operations that benefit from 538asynchronous APIs. Network protocols themselves are heavily asynchronous. 539 540.. admonition:: Async SEED 541 542 Pigweed's async pattern is proposed in :ref:`SEED-0112 <seed-0112>`. 543 544.. _seed-0107-buffers: 545 546Buffer management 547================= 548Pigweed's networking APIs will support zero-copy data transmission. Applications 549will be able to request a buffer from a socket. When one is available, they fill 550it with data for transmission. 551 552Pigweed will develop a general purpose module for allocating and managing 553buffers. This will be used to implement zero-copy features for Pigweed's 554networking stack. 555 556As an example, zero-copy buffer allocation could work as follows: 557 558- The user requests a buffer from a socket. 559- The network protocol layer under the socket requests a buffer from the next 560 lower layer. 561- The bottom protocol layer allocates a buffer. 562- Each layer reserves part of the buffer for its headers or footers. 563- The remaining buffer is provided to the user to populate with their payload. 564- When the user is done, the buffer is released. Each layer of the network stack 565 processes the buffer as necessary. 566- Finally, at the lowest layer, the final buffer is sent over the hardware 567 interface. 568 569Zero-copy APIs will be :ref:`asynchronous <seed-0107-async>`. 570 571**Rationale** 572 573Networking involves transmitting large amounts of data. Copying network traffic 574can result in substantial CPU usage, particularly in nodes that route traffic to 575other nodes. 576 577A buffer management system that minimizes copying saves precious CPU cycles and 578power on constrained systems. 579 580.. admonition:: Buffer management SEED 581 582 Pigweed's buffer management system is proposed in :ref:`SEED-0109 583 <seed-0109>`. 584 585Vectored I/O 586------------ 587Vectored or scatter/gather I/O allows users to serialize data from multiple 588buffers into a single output stream, or vice versa. For Pigweed's networking 589APIs, this could be used to, for example, store a packet header in one buffer 590and packet contents in one or more other buffers. These isolated chunks are 591serialized in order to the network interface. 592 593Vectored I/O minimizes copying, but is complex. Additionally, simple DMA engines 594may only operate on a single DMA buffer. Thus, vectored I/O could require 595either: 596 597- a copy into the DMA engine's buffer, which costs CPU time and memory, or 598- multiple, small DMAs, which involves extra interrupts and CPU time. 599 600Vectored I/O may be supported in Pigweed's communications stack, depending on 601project requirements. 602 603---------- 604Next steps 605---------- 606Pigweed's communications revamp will proceed loosely as follows: 607 608* Write SEEDs to explore existing solutions, distill requirements, and propose 609 new Pigweed features for these areas: 610 611 - Sockets API (SEED-0116) 612 - Async pattern (:ref:`SEED-0112 <seed-0112>`). 613 - Buffer management (:ref:`SEED-0109 <seed-0109>`) 614 - Network protocol stack 615 616* Implement the Sockets API. 617 618 - Document, integrate, and deploy the async programming pattern for Pigweed. 619 - Develop and test Pigweed's buffer management system. 620 - Use these features in the sockets API. If necessary, the synchronous, 621 copying API could be implemented first. 622 623* Deploy the sockets API for TCP/IP. 624 625 - Implement and unit test sockets for TCP/IP with POSIX and Winsock sockets. 626 - Implement and unit test sockets for an embedded TCP/IP stack. 627 628* Develop a test suite for Pigweed network communications. 629 630 - Create integration tests for networks with multiple nodes that cover basic 631 operation, high load, and packet loss. 632 - Write performance tests against the sockets API to measure network stack 633 performance. 634 635* Develop Pigweed's lightweight network protocol stack. 636 637 - Test the lightweight network protocol stack on hardware and in a simulated 638 environment. 639 - Write fuzz tests for the protocol stack. 640 - Write performance tests for the protocol stack. 641 642* Revisit other communications systems, including pw_rpc and pw_transfer. 643