Lines Matching +full:protocol +full:- +full:tests
1 .. _module-pw_transfer:
6 ``pw_transfer`` is a reliable data transfer protocol which runs on top of
13 -----
15 -----
21 ---------------
23 required. The transfer thread is used to process all transfer-related events
29 All user-defined transfer callbacks (i.e. the virtual interface of a
35 - The first is a *chunk buffer*. This is used to stage transfer packets received
39 - The second is an *encode buffer*. This is used by the transfer thread to
46 when creating a system thread. Refer to :ref:`module-pw_thread-thread-creation`
51 .. code-block:: cpp
71 // pw_transfer requires some additional per-packet overhead, so the actual
85 // pw::transfer::TransferThread is the generic, non-templated version of the
91 .. _pw_transfer-transfer-server:
94 ---------------
108 must have a globally-unique integer ID used to identify the resource.
117 .. code-block:: cpp
122 // A simple transfer handler which reads data from an in-memory buffer.
142 .. code-block:: cpp
161 .. code-block:: cpp
194 ---------------
212 return a non-OK status if it is called with bad arguments. Otherwise, it will
215 For using the offset parameter, please see :ref:`pw_transfer-nonzero-transfers`.
224 return a non-OK status if it is called with bad arguments. Otherwise, it will
227 For using the offset parameter, please see :ref:`pw_transfer-nonzero-transfers`.
244 .. code-block:: cpp
291 -------------------------
294 fixed-size may optionally indicate this to the transfer client and server,
309 .. code-block:: c++
313 handle->SetTransferSize(kMyResourceSize);
321 .. code-block:: c++
334 ----------------------------
342 .. _module-pw_transfer-config:
345 ----------------------------
346 The following configurations can be adjusted via compile-time configuration of
348 :ref:`module documentation <module-structure-compile-time-configuration>` for
354 chunk when no response is received. Can later be configured per-transfer when
370 This number should be high, particularly if long-running transfers are
378 per-transfer.
383 on the server before retrying. This can later be configured per-transfer.
389 per-transfer.
426 Turns on logging of individual non-data or non-parameter chunks. Default is
432 disable logging. These chunks are moderated (rate-limited) by the same
438 further queued events. In systems which can perform long-running operations
442 .. _pw_transfer-nonzero-transfers:
444 Non-zero Starting Offset Transfers
445 ----------------------------------
452 A resource can be read or written from a non-zero starting offset simply by
457 The data or stream passed to `read()` or `write()` will be used as-is. I.e.
465 operation, and the server responds with the offset to confirm the non-zero
467 check that the server has accepted the non-zero offset during the handshake, so
471 Due to the need to seek streams by the handler to support the non-zero offset,
475 See the :ref:`transfer handler <pw_transfer-transfer-server>` documentation for
476 further information about configuring resources for non-zero transfers and the
487 .. code-block:: python
517 .. code-block:: typescript
546 .. code-block:: java
576 --------
577 Protocol chapter
578 --------
588 :ref:`here <module-pw_transfer-proto-definition>`.
592 Transfers are run for a specific *resource* --- a stream of data which can be
593 read from or written to. Resources have a system-specific integral identifier
594 defined by the implementers of the server-side transfer node.
605 ``pw_transfer`` attempts to be a reliable data transfer protocol.
614 A non-seekable stream could prematurely terminate a transfer following a
617 At present, ``pw_transfer`` requires in-order data transmission. If packets are
618 received out-of-order, the receiver will request that the transmitter re-send
623 Transfers begin with a three-way handshake, whose purpose is to identify the
624 resource being transferred, assign a session ID, and synchronize the protocol
630 chunk additionally encodes the protocol version which the client is configured
636 additional user-specified setup. The server accepts the client's session ID,
638 session, and configured protocol version for the transfer.
640 .. _module-pw_transfer-windowing:
650 `(RFC 5681 §3.1) <https://datatracker.ietf.org/doc/html/rfc5681#section-3.1>`_,
673 received after a timeout, it re-sends its ``COMPLETION`` chunk. The session ends
677 .. _module-pw_transfer-proto-definition:
687 Protocol buffer definition
691 :lines: 14-
696 Protocol errors
697 ---------------
701 .. cpp:namespace-push:: pw::stream
703 +-------------------------+-------------------------+-------------------------+
709 +-------------------------+-------------------------+-------------------------+
714 +-------------------------+---------------------------------------------------+
716 +-------------------------+-------------------------+-------------------------+
721 +-------------------------+-------------------------+-------------------------+
723 +-------------------------+-------------------------+-------------------------+
725 +-------------------------+-------------------------+-------------------------+
726 | ``INTERNAL`` | An assumption of the protocol was violated. |
729 +-------------------------+-------------------------+-------------------------+
732 +-------------------------+-------------------------+-------------------------+
738 +-------------------------+-------------------------+-------------------------+
741 +-------------------------+-------------------------+-------------------------+
742 | ``UNIMPLEMENTED`` | Out-of-order chunk was | (not sent) |
745 +-------------------------+-------------------------+-------------------------+
747 .. cpp:namespace-pop::
760 -----------
764 start([Client initiates<br>transfer]) -->data_request
765 data_request[Receive transfer<br>parameters]-->send_chunk
767 send_chunk[Send chunk]-->sent_all
769 sent_all{Sent final<br>chunk?} -->|yes|wait
770 sent_all-->|no|sent_requested
772 sent_requested{Sent all<br>pending?}-->|yes|data_request
773 sent_requested-->|no|send_chunk
775 wait[Wait for receiver]-->is_done
777 is_done{Received<br>final chunk?}-->|yes|done
778 is_done-->|no|data_request
783 -------------
787 start([Client initiates<br>transfer]) -->request_bytes
788 request_bytes[Set transfer<br>parameters]-->wait
790 wait[Wait for chunk]-->received_chunk
792 received_chunk{Received<br>chunk by<br>deadline?}-->|no|request_bytes
793 received_chunk-->|yes|check_chunk
795 check_chunk{Correct<br>offset?} -->|yes|process_chunk
796 check_chunk --> |no|request_bytes
798 process_chunk[Process chunk]-->final_chunk
800 final_chunk{Final<br>chunk?}-->|yes|signal_completion
801 final_chunk{Final<br>chunk?}-->|no|received_requested
803 received_requested{Received all<br>pending?}-->|yes|request_bytes
804 received_requested-->|no|wait
806 signal_completion[Signal completion]-->done
810 Legacy protocol
813 reliability improvements of its modern protocol. As a result of this, transfer
814 implementations support a "legacy" protocol mode, in which transfers run without
820 - There is no distinction between a transfer resource and session --- a single
824 - The legacy protocol has no opening handshake phase. The client initiates with
827 - The legacy protocol has no terminating handshake phase. When either end
832 Transfer clients request the latest transfer protocol version by default, but
833 may be configured to request the legacy protocol. Transfer server and client
834 implementations detect if their transfer peer is running the legacy protocol and
835 automatically switch to it if required, even if they requested a newer protocol
836 version. It is **strongly** unadvised to use the legacy protocol in new code.
838 .. _module-pw_transfer-integration-tests:
840 -----------------
841 Integration tests
842 -----------------
843 The ``pw_transfer`` module has a set of integration tests that verify the
847 To run the tests on your machine, run
849 .. code-block:: bash
855 .. note:: There is a large test that tests transfers that are megabytes in size.
858 VERY slow, but exist for manual validation of real-world use cases.
860 The integration tests permit injection of client/server/proxy binaries to use
861 when running the tests. This allows manual testing of older versions of
864 .. code-block:: bash
868 $ bazel run pw_transfer/integration_test:cross_language_medium_test -- \
869 --cpp-client-binary ../old_pw_transfer_version/cpp_client
871 Backwards compatibility tests
873 ``pw_transfer`` includes a `suite of backwards-compatibility tests
875 that are intended to continuously validate a degree of backwards-compatibility
877 binaries hosted in CIPD and running tests between the older client/server
882 .. code-block::bash
887 $ cp bazel-bin/pw_transfer/integration_test/server \
889 $ cp bazel-bin/pw_transfer/integration_test/cpp_client \
893 updating a CIPD package <http://go/pigweed-cipd#installing-packages-into-cipd>`_.
897 …s://ci.chromium.org/ui/p/pigweed/builders/luci.pigweed.pigweed.ci/pigweed-linux-bzl-integration>`_.
899 By default, these tests are not run in CQ (on presubmit) because they are too
900 slow. However, you can request that the tests be run in presubmit on your
903 .. code-block::
905 Cq-Include-Trybots: luci.pigweed.try:pigweed-linux-bzl-integration
907 .. _module-pw_transfer-parallel-tests:
909 Running the tests many times
911 Because the tests bind to network ports, you cannot run more than one instance
916 -----
917 On Linux, you can add the ``"block-network"`` tag to the tests (`example
918 <https://pigweed-review.googlesource.com/c/pigweed/pigweed/+/181297>`_). This
919 enables network isolation for the tests, allowing you to run them in parallel
922 .. code-block::
924 bazel test --runs_per_test=10 //pw_transfer/integration_tests/...
927 -----
931 tag the tests ``"exclusive"``. This allows you to use ``--runs_per_test``, but
935 -------------------
936 Ideally, we would apply either the ``"block-network"`` or ``"exclusive"`` tag
937 to the tests depending on the OS. But this is not supported, `gh#2971
940 We don't want to tag the tests ``"exclusive"`` by default because that will
941 prevent *different* tests from running in parallel, significantly slowing them