1# How ADB uses USB Zero-Length Packets (ZLP) 2 3## TLDR; 4There is no USB mechanism that lets a sender announce the size of a `Transfer`. This is not 5a problem when the host side receives packet since is leverage the aprotocol to know what size 6of transfer to expect. However, traffic towards the device must include zero length packets to 7mark the boundaries since the device does not leverage the aprotocol. 8 9## Introduction 10 11There is an asymmetry in how ADB communicates over USB. While all USB backends on the host side (Linux, 12Windows, Mac, and libusb) send ZLPs, the device side never does. Why is that? This document explains 13what ZLPs are, how ADB uses them, and why things are designed this way. 14 15## USB Transfer 101 16 17In the context of ADB, USB can be thought of as two unidirectional pipes per device. 18One pipe takes care of upload while the other takes care of the download. On the pipe 19transit payloads. The maximum size of a payload is set by the pipe buffers located 20on the device. These buffers are called the `Endpoint`. 21 22``` 23 ┌──────────┐ ┌──────────┐ 24 │ USB Host │ │USB Device│ 25 ├──────────┤ ├──────────┤ 26 │ │ │ │ 27 │ Pipe ◄──┼─────────────────────────┼ Endpoint │ 28 │ │ USB │ │ 29 │ Pipe ───┼─────────────────────────► Endpoint │ 30 └──────────┘ └──────────┘ 31``` 32 33In USB parlance, sending a buffer of data on a pipe and receiving it on the other end is called a `Transfer`. 34On the sender side, the USB Controller is presented with a [buffer,size] pair called IRP. On the receiver 35side, a similar IRP is provided to the USB controller. 36 37``` 38 ┌────────┐ ┌──────────┐ 39 │ Sender │ │ Receiver │ 40 └────┬───┘ └─────┬────┘ 41 │ │ 42 ┌──▼───┐ ┌──▼───┐ 43 │ IRP │ │ IRP │ 44 └──┬───┘ └──▲───┘ 45 ┌──────▼───────┐ ┌──────┴───────┐ 46 │USB Controller│ │USB Controller│ 47 │ │ │ │ 48 │ ─┼───►─DP──►─DP ─►──DP──►─┼─► │ 49 └──────────────┘ └──────────────┘ 50``` 51 52Because of the endpoint buffer size (`wMaxPacketSize`), an IRP is broken down in 53several data payloads called `Data Packets` (DP). 54 55Note: On the device, ADB uses `functionfs` which is not based on IRP. However, the logic is the same since received DP 56must be re-assembled on the device to rebuild the original IRP. To simplify this document we use the name "IRP" 57everywhere in this doc to mean "[buffer,size] pair provided to the USB Controller". 58 59## When does a USB Transfer ends? 60 61If an IRP is broken down in DPs by the sender, how does the receiver reassemble 62the DPs into an IRP on the other side? 63 64The key concept to get out of this whole document is that there is no mechanism 65in USB for the sender to announce the size of a `Transfer`. Instead, the receiving 66end uses the following rules. A `Transfer` is considered done when either of the following condition 67is met. 68 69- An error occurred (device disconnected, ...). 70- The IRP is full. 71- The size of the packet is less than `wMaxPacketSize` (this is a Short-Packet). This is 72a different behavior from the usual UNIX `read(3)` which is allowed to return less than required 73without meaning that the stream is over. 74- Too much data is received. The IRP overflows (this is also an error). 75 76See USB 2 specifications (5.8.3 Bulk Transfer Packet Size Constraints) for additional information. 77``` 78An endpoint must always transmit data payloads with a data field less than or equal to the endpoint’s 79reported wMaxPacketSize value. When a bulk IRP involves more data than can fit in one maximum-sized 80data payload, all data payloads are required to be maximum size except for the last data payload, which will 81contain the remaining data. A bulk transfer is complete when the endpoint does one of the following: 82 83• Has transferred exactly the amount of data expected 84• Transfers a packet with a payload size less than wMaxPacketSize or transfers a zero-length packet 85``` 86 87### Example 1: The IRP is full 88 89For a USB3 bulk pipe, the `wMaxPacketSize` is 1024. The sender "S" wishes 90to send 2048 bytes. It creates a IRP, fills it with the 2048 bytes, and gives the IRP 91to the USB controller. On the 92received side "R", the USB controller is provided with a IRP of side 2048. 93 94``` 95Traffic: 96S -> 1024 -> R 97S -> 1024 -> R IRP full, Transfer OK! 98``` 99 100At this point R's IRP is full. R USB controller declares the `Transfer` over 101and calls whatever callback the client provided the IRP. 102 103### Example 2: Short-Packet 104 105Same USB3 bulk as Example 1. The `wMaxPacketSize` is 1024. The sender wishes 106to send 2148 bytes. It creates a IRP of size 2148 bytes and fills it with data. 107On the received side, the USB controller is provided with a IRP of size 4096. 108 109``` 110Traffic: 111S -> 1024 -> R 112S -> 1024 -> R 113S -> 100 -> R Short-Packet, Transfer OK! 114``` 115 116The receiver end detects a short packet. Even though it was provided with a 4906 117byte IRP, it declares the `Transfer` completed (and records the actual size 118of the `Transfer` in the IRP). 119 120### Example 3: Overflow 121 122Same USB3 bulk as Example 1. The `wMaxPacketSize` is 1024. The sender wishes 123to send 4096 bytes. It creates a IRP, fills it with the 4096 bytes. On the 124receiver side, the USB controller is provided with an IRP of size 2148. 125 126``` 127Traffic: 128S -> 1024 -> R 129S -> 1024 -> R 130S -> 1024 -> R ERROR, Transfer failed! 131``` 132 133On the third packet, the receiver runs out of space in the IRP (it only had 100 134bytes available). Without a way to fully store this packet, 135it discards everything and returns an error stating that the `Transfer` was not successful. 136 137## Preventing overflow and the need for Zero-Length Packets 138 139There are two techniques to avoid overflows. 140 141### Using a protocol 142One technique is to create a protocol on top of `Transfers`. 143ADB does that with its "aprotocol" ([protocol.md](protocol.md)). 144 145In aprotocol, the sender creates a `Transfer` containing a header which is 146always 24 bytes. Then it sends a separate `Transfer` containing the payload. 147The size of the payload is in the header. This way the receiver always knows 148what size of IRP to provide to the USB controller: it first requests a 24 byte IRP 149read, extracts the size of the payload, then issues a second IRP read request 150with the extracted size of the payload. 151 152### Using a multiple of `wMaxPacketSize` 153 154The other technique to avoid overflows is for the receiver to always use a IRP with 155a size which is a multiple of the `wMaxPacketSize`. This way a `Transfer` always ends properly. 156* A max size packet will exactly finish to fill the IRP, ending the `Transfer`. 157* A short packet will end the `Transfer`. 158 159This technique comes with an edge case. Take the example of a USB3 pipe where 160`wMaxPacketSize` is 1024. The sender wishes to send 3072 byte. It creates a IRP 161of that size, fills in the data and gives it to the USB controller which breaks 162it into Packets. The receiver decides to read with a IRP of size 4096. 163 164``` 165Traffic: 166S -> 1024 -> R 167S -> 1024 -> R 168S -> 1024 -> R 169. 170. 171. 172Stalled! 173``` 174 175After the USB controller on the sender side has sent all the data in the IRP, it won't send anything else. 176But none of the ending conditions of a `Transfer` have been reached on the receiving end. No overflow, no short-packet, and the IRP is not 177full (there is still 1024 bytes unused). As is, the USB controller on the receiving end will never declare the `Transfer` 178either successful or failed. This is a stall (at least until another Packet is sent, if ever). 179 180This condition is entered when the size of a IRP to send is a multiple of `wMaxPacketSize` 181but less than the size of the IRP provided by the receiving end. To fix this condition, 182the sender MUST issue a Zero-Length Packet. Technically, this is a short packet (it is less 183than `wMaxPacketSize`). Upon receiving the ZLP, the receiver declares the `Transfer` 184finished. 185 186``` 187Traffic: 188S -> 1024 -> R 189S -> 1024 -> R 190S -> 1024 -> R 191S -> 0 -> R Short-Packet, Transfer is over! 192``` 193 194## Implementation choices 195 196By now, it should be clear that whether a sender needs to send a ZLP depends on the way 197the receiver end works. 198 199### ADB Device to Host pipe communication design 200 201The receiver on the host leverages ADB aprotocol ([protocol.md](protocol.md)). It 202first creates a IRP of size 24 bytes to receive the header. Then it creates a IRP 203`Transfer` 204of the size of the payload. Because the IRPs are always exactly the size of the `Transfer` 205the device sends, there is no need for LZP. The USB Controller on the host side will always be able 206to declare a `Transfer` complete when the IRP is full and there will never be any overflow. 207 208The drawback of this technique is that it can consume a lot of RAM because multiple 209IRPs can be in flight at a time. With the maximum size of 210a apacket payload being MAX_PAYLOAD (1MiB), things can quickly add up. 211 212 213### ADB Host to Device pipe communication design 214 215On the device side, the receiver does not leverage the ADB aprotocol ([protocol.md](protocol.md)). 216I suspect this was done to reduce memory consumption (the first Android device had a total RAM size of 192MiB). 217 218The UsbFS connection always requests the same 219`Transfer` size. To prevent overflows, the size is picked to be a multiple of the `wMaxPacketSize` (1x would be 220valid but the overhead would kill performances). Currently, the value is kUsbReadSize (16384). USB endpoints 221have a well known `wMaxPacketSize` so 16384 works for all of them (this list is for bulk transfers only which 222ADB exclusively uses). 223 224* Full Speed: 8, 16, 32, or 64 bytes. 225* High Speed: 512 bytes. 226* Super Speed: 1024 bytes. 227 228When the apacket payload size is a multiple 229of the `wMaxPacketSize`, the sender on the host side MUST send a ZLP to avoid stalling 230on the receiver end. 231 232 233## What happens if the host sender has a bug and ZLPs are not sent? 234 235If there is a bug on the host and ZLPs are not sent, several things can happen. 236You can observe normal behavior, stalled communication, or even device disconnection. 237 238Because there are many buffers before the USB controller layer 239is hit, the issue won't be deterministic. However my experience showed that attempting to 240push 10GiB rarely 241fails to bring up instabilities. 242 243``` 244$ dd if=/dev/urandom of=r.bin bs=1G count=10 iflag=fullblock` 245$ adb push -Z r.bin /datal/local/tmp 246``` 247 248### 1. Nothing breaks 249 250You could be unlucky and not trigger the fault. 251 252### 2. Stall 253 254A payload of a size that's a multiple of `wMaxPacketSize` but of size less than kUsbReadSize (16384) is sent. 255This is a stall as previously described. 256 257### 3. Disconnection (due to merged packets) 258 259In real-life usage, there is rarely a single thing happening on ADB. Users often also run logcat, Studio 260monitors metrics, or perhaps the user has a shell opened. What happens if a connection goes stalls 261but then something else sends an apacket? 262 263The first `Transfer` of the apacket will be an apacket header which is 24 bytes. This will be considered 264a short-packet. The previous stalled `Transfer` will be completed with the header appended. This will 265confuse UsbFS since the payload will be 24 bytes more than it should be. In this condition, the connection 266is closed. The log message is 267 268``` 269received too many bytes while waiting for payload 270``` 271 272or 273 274``` 275received packet of unexpected length while reading header 276``` 277 278A summary inspection of logs may make it look like a payload `Transfer` 279was merged with the next header `Transfer`.