1<?% config.freshness.reviewed = '2021-04-12' %?> 2<?% config.freshness.owner = 'sprang' %?> 3 4# Paced Sending 5 6The paced sender, often referred to as just the "pacer", is a part of the WebRTC 7RTP stack used primarily to smooth the flow of packets sent onto the network. 8 9## Background 10 11Consider a video stream at 5Mbps and 30fps. This would in an ideal world result 12in each frame being ~21kB large and packetized into 18 RTP packets. While the 13average bitrate over say a one second sliding window would be a correct 5Mbps, 14on a shorter time scale it can be seen as a burst of 167Mbps every 33ms, each 15followed by a 32ms silent period. Further, it is quite common that video 16encoders overshoot the target frame size in case of sudden movement especially 17dealing with screensharing. Frames being 10x or even 100x larger than the ideal 18size is an all too real scenario. These packet bursts can cause several issues, 19such as congesting networks and causing buffer bloat or even packet loss. Most 20sessions have more than one media stream, e.g. a video and an audio track. If 21you put a frame on the wire in one go, and those packets take 100ms to reach the 22other side - that means you have now blocked any audio packets from reaching the 23remote end in time as well. 24 25The paced sender solves this by having a buffer in which media is queued, and 26then using a _leaky bucket_ algorithm to pace them onto the network. The buffer 27contains separate fifo streams for all media tracks so that e.g. audio can be 28prioritized over video - and equal prio streams can be sent in a round-robin 29fashion to avoid any one stream blocking others. 30 31Since the pacer is in control of the bitrate sent on the wire, it is also used 32to generate padding in cases where a minimum send rate is required - and to 33generate packet trains if bitrate probing is used. 34 35## Life of a Packet 36 37The typical path for media packets when using the paced sender looks something 38like this: 39 401. `RTPSenderVideo` or `RTPSenderAudio` packetizes media into RTP packets. 412. The packets are sent to the [RTPSender] class for transmission. 423. The pacer is called via [RtpPacketSender] interface to enqueue the packet 43 batch. 444. The packets are put into a queue within the pacer awaiting opportune moments 45 to send them. 465. At a calculated time, the pacer calls the `PacingController::PacketSender()` 47 callback method, normally implemented by the [PacketRouter] class. 486. The router forwards the packet to the correct RTP module based on the 49 packet's SSRC, and in which the `RTPSenderEgress` class makes final time 50 stamping, potentially records it for retransmissions etc. 517. The packet is sent to the low-level `Transport` interface, after which it is 52 now out of scope. 53 54Asynchronously to this, the estimated available send bandwidth is determined - 55and the target send rate is set on the `RtpPacketPacer` via the `void 56SetPacingRates(DataRate pacing_rate, DataRate padding_rate)` method. 57 58## Packet Prioritization 59 60The pacer prioritized packets based on two criteria: 61 62* Packet type, with most to least prioritized: 63 1. Audio 64 2. Retransmissions 65 3. Video and FEC 66 4. Padding 67* Enqueue order 68 69The enqueue order is enforced on a per stream (SSRC) basis. Given equal 70priority, the [RoundRobinPacketQueue] alternates between media streams to ensure 71no stream needlessly blocks others. 72 73## Implementations 74 75The main class to use is called [TaskQueuePacedSender]. It uses a task queue to 76manage thread safety and schedule delayed tasks, but delegates most of the actual 77work to the `PacingController` class. 78This way, it's possible to develop a custom pacer with different scheduling 79mechanism - but ratain the same pacing logic. 80 81## The Packet Router 82 83An adjacent component called [PacketRouter] is used to route packets coming out 84of the pacer and into the correct RTP module. It has the following functions: 85 86* The `SendPacket` method looks up an RTP module with an SSRC corresponding to 87 the packet for further routing to the network. 88* If send-side bandwidth estimation is used, it populates the transport-wide 89 sequence number extension. 90* Generate padding. Modules supporting payload-based padding are prioritized, 91 with the last module to have sent media always being the first choice. 92* Returns any generated FEC after having sent media. 93* Forwards REMB and/or TransportFeedback messages to suitable RTP modules. 94 95At present the FEC is generated on a per SSRC basis, so is always returned from 96an RTP module after sending media. Hopefully one day we will support covering 97multiple streams with a single FlexFEC stream - and the packet router is the 98likely place for that FEC generator to live. It may even be used for FEC padding 99as an alternative to RTX. 100 101## The API 102 103The section outlines the classes and methods relevant to a few different use 104cases of the pacer. 105 106### Packet sending 107 108For sending packets, use 109`RtpPacketSender::EnqueuePackets(std::vector<std::unique_ptr<RtpPacketToSend>> 110packets)` The pacer takes a `PacingController::PacketSender` as constructor 111argument, this callback is used when it's time to actually send packets. 112 113### Send rates 114 115To control the send rate, use `void SetPacingRates(DataRate pacing_rate, 116DataRate padding_rate)` If the packet queue becomes empty and the send rate 117drops below `padding_rate`, the pacer will request padding packets from the 118`PacketRouter`. 119 120In order to completely suspend/resume sending data (e.g. due to network 121availability), use the `Pause()` and `Resume()` methods. 122 123The specified pacing rate may be overriden in some cases, e.g. due to extreme 124encoder overshoot. Use `void SetQueueTimeLimit(TimeDelta limit)` to specify the 125longest time you want packets to spend waiting in the pacer queue (pausing 126excluded). The actual send rate may then be increased past the pacing_rate to 127try to make the _average_ queue time less than that requested limit. The 128rationale for this is that if the send queue is say longer than three seconds, 129it's better to risk packet loss and then try to recover using a key-frame rather 130than cause severe delays. 131 132### Bandwidth estimation 133 134If the bandwidth estimator supports bandwidth probing, it may request a cluster 135of packets to be sent at a specified rate in order to gauge if this causes 136increased delay/loss on the network. Use the `void CreateProbeCluster(...)` 137method - packets sent via this `PacketRouter` will be marked with the 138corresponding cluster_id in the attached `PacedPacketInfo` struct. 139 140If congestion window pushback is used, the state can be updated using 141`SetCongestionWindow()` and `UpdateOutstandingData()`. 142 143A few more methods control how we pace: * `SetAccountForAudioPackets()` 144determines if audio packets count into bandwidth consumed. * 145`SetIncludeOverhead()` determines if the entire RTP packet size counts into 146bandwidth used (otherwise just media payload). * `SetTransportOverhead()` sets 147an additional data size consumed per packet, representing e.g. UDP/IP headers. 148 149### Stats 150 151Several methods are used to gather statistics in pacer state: 152 153* `OldestPacketWaitTime()` time since the oldest packet in the queue was 154 added. 155* `QueueSizeData()` total bytes currently in the queue. 156* `FirstSentPacketTime()` absolute time the first packet was sent. 157* `ExpectedQueueTime()` total bytes in the queue divided by the send rate. 158 159[RTPSender]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/modules/rtp_rtcp/source/rtp_sender.h;drc=77ee8542dd35d5143b5788ddf47fb7cdb96eb08e 160[RtpPacketSender]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/modules/rtp_rtcp/include/rtp_packet_sender.h;drc=ea55b0872f14faab23a4e5dbcb6956369c8ed5dc 161[RtpPacketPacer]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/modules/pacing/rtp_packet_pacer.h;drc=e7bc3a347760023dd4840cf6ebdd1e6c8592f4d7 162[PacketRouter]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/modules/pacing/packet_router.h;drc=3d2210876e31d0bb5c7de88b27fd02ceb1f4e03e 163[TaskQueuePacedSender]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/modules/pacing/task_queue_paced_sender.h;drc=5051693ada61bc7b78855c6fb3fa87a0394fa813 164[RoundRobinPacketQueue]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/modules/pacing/round_robin_packet_queue.h;drc=b571ff48f8fe07678da5a854cd6c3f5dde02855f 165