1<?% config.freshness.reviewed = '2021-04-13' %?> 2<?% config.freshness.owner = 'jakobi' %?> 3 4# NetEq 5 6NetEq is the audio jitter buffer and packet loss concealer. The jitter buffer is 7an adaptive jitter buffer, meaning that the buffering delay is continuously 8optimized based on the network conditions. Its main goal is to ensure a smooth 9playout of incoming audio packets from the network with a low amount of audio 10artifacts (alterations to the original content of the packets) while at the same 11time keep the delay as low as possible. 12 13## API 14 15At a high level, the NetEq API has two main functions: 16[`InsertPacket`](https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/api/neteq/neteq.h;l=198;drc=4461f059d180fe8c2886d422ebd1cb55b5c83e72) 17and 18[`GetAudio`](https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/api/neteq/neteq.h;l=219;drc=4461f059d180fe8c2886d422ebd1cb55b5c83e72). 19 20### InsertPacket 21 22[`InsertPacket`](https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/api/neteq/neteq.h;l=198;drc=4461f059d180fe8c2886d422ebd1cb55b5c83e72) 23delivers an RTP packet from the network to NetEq where the following happens: 24 251. The packet is discarded if it is too late for playout (for example if it was 26 reordered). Otherwize it is put into the packet buffer where it is stored 27 until it is time for playout. If the buffer is full, discard all the 28 existing packets (this should be rare). 292. The interarrival time between packets is analyzed and statistics is updated 30 which is used to derive a new target playout delay. The interarrival time is 31 measured in the number of GetAudio ‘ticks’ and thus clock drift between the 32 sender and receiver can be accounted for. 33 34### GetAudio 35 36[`GetAudio`](https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/api/neteq/neteq.h;l=219;drc=4461f059d180fe8c2886d422ebd1cb55b5c83e72) 37pulls 10 ms of audio from NetEq for playout. A much simplified decision logic is 38as follows: 39 401. If there is 10 ms audio in the sync buffer then return that. 412. If the next packet is available (based on RTP timestamp) in the packet 42 buffer then decode it and append the result to the sync buffer. 43 1. Compare the current delay estimate (filtered buffer level) with the 44 target delay and time stretch (accelerate or decelerate) the contents of 45 the sync buffer if the buffer level is too high or too low. 46 2. Return 10 ms of audio from the sync buffer. 473. If the last decoded packet was a discontinuous transmission (DTX) packet 48 then generate comfort noise. 494. If there is no available packet for decoding due to the next packet having 50 not arrived or been lost then generate packet loss concealment by 51 extrapolating the remaining audio in the sync buffer or by asking the 52 decoder to produce it. 53 54In summary, the output is the result one of the following operations: 55 56* Normal: audio decoded from a packet. 57* Acceleration: accelerated playout of a decoded packet. 58* Preemptive expand: decelerated playout of a decoded packet. 59* Expand: packet loss concealment generated by NetEq or the decoder. 60* Merge: audio stitched together from packet loss concealment to decoded data 61 in case of a loss. 62* Comfort noise (CNG): comfort noise generated by NetEq or the decoder between 63 talk spurts due to discontinuous transmission of packets (DTX). 64 65## Statistics 66 67There are a number of functions that can be used to query the internal state of 68NetEq, statistics about the type of audio output and latency metrics such as how 69long time packets have waited in the buffer. 70 71* [`NetworkStatistics`](https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/api/neteq/neteq.h;l=273;drc=4461f059d180fe8c2886d422ebd1cb55b5c83e72): 72 instantaneous values or stats averaged over the duration since last call to 73 this function. 74* [`GetLifetimeStatistics`](https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/api/neteq/neteq.h;l=280;drc=4461f059d180fe8c2886d422ebd1cb55b5c83e72): 75 cumulative stats that persist over the lifetime of the class. 76* [`GetOperationsAndState`](https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/api/neteq/neteq.h;l=284;drc=4461f059d180fe8c2886d422ebd1cb55b5c83e72): 77 information about the internal state of NetEq (is only inteded to be used 78 for testing and debugging). 79 80## Tests and tools 81 82* [`neteq_rtpplay`](https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/modules/audio_coding/neteq/tools/neteq_rtpplay.cc;drc=cee751abff598fc19506f77de08bea7c61b9dcca): 83 Simulate NetEq behavior based on either an RTP dump, a PCAP file or an RTC 84 event log. A replacement audio file can also be used instead of the original 85 payload. Outputs aggregated statistics and optionally an audio file to 86 listen to. 87* [`neteq_speed_test`](https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/modules/audio_coding/neteq/test/neteq_speed_test.cc;drc=2ab97f6f8e27b47c0d9beeb8b6ca5387bda9f55c): 88 Measure performance of NetEq, used on perf bots. 89* Unit tests including bit exactness tests where RTP file is used as an input 90 to NetEq, the output is concatenated and a checksum is calculated and 91 compared against a reference. 92 93## Other responsibilities 94 95* Dual-tone multi-frequency signaling (DTMF): receive telephone events and 96 produce dual tone waveforms. 97* Forward error correction (RED or codec inband FEC): split inserted packets 98 and prioritize the payloads. 99* NACK (negative acknowledgement): keep track of lost packets and generate a 100 list of packets to NACK. 101* Audio/video sync: NetEq can be instructed to increase the latency in order 102 to keep audio and video in sync. 103