1<?% config.freshness.reviewed = '2021-02-21' %?> 2 3# DefaultVideoQualityAnalyzer 4 5## Audience 6 7This document is for users of 8[`webrtc::webrtc_pc_e2e::DefaultVideoQualityAnalyzer`][1]. 9 10## Overview 11 12`DefaultVideoQualityAnalyzer` implements 13[`webrtc::VideoQualityAnalyzerInterface`][2] and is a main 14implementation of video quality analyzer for WebRTC. To operate correctly it 15requires to receive video frame on each step: 16 171. On frame captured - analyzer will generate a unique ID for the frame, that 18 caller should attach to the it. 192. Immediately before frame enter the encoder. 203. Immediately after the frame was encoded. 214. After the frame was received and immediately before it entered the decoder. 225. Immediately after the frame was decoded. 236. When the frame was rendered. 24 25 26 27The analyzer updates its internal metrics per frame when it was rendered and 28reports all of them after it was stopped through 29[WebRTC perf results reporting system][10]. 30 31To properly inject `DefaultVideoQualityAnalyzer` into pipeline the following helpers can be used: 32 33### VideoQualityAnalyzerInjectionHelper 34 35[`webrtc::webrtc_pc_e2e::VideoQualityAnalyzerInjectionHelper`][3] provides 36factory methods for components, that will be used to inject 37`VideoQualityAnalyzerInterface` into the `PeerConnection` pipeline: 38 39* Wrappers for [`webrtc::VideoEncoderFactory`][4] and 40 [`webrtc::VideoDecodeFactory`][5] which will properly pass 41 [`webrtc::VideoFrame`][6] and [`webrtc::EncodedImage`][7] into analyzer 42 before and after real video encode and decoder. 43* [`webrtc::test::TestVideoCapturer::FramePreprocessor`][8] which is used to 44 pass generated frames into analyzer on capturing and then set the returned 45 frame ID. It also configures dumping of captured frames if requried. 46* [`rtc::VideoSinkInterface<VideoFrame>`][9] which is used to pass frames to 47 the analyzer before they will be rendered to compute per frame metrics. It 48 also configures dumping of rendered video if requried. 49 50Besides factories `VideoQualityAnalyzerInjectionHelper` has method to 51orchestrate `VideoQualityAnalyzerInterface` workflow: 52 53* `Start` - to start video analyzer, so it will be able to receive and analyze 54 video frames. 55* `RegisterParticipantInCall` - to add new participants after analyzer was 56 started. 57* `Stop` - to stop analyzer, compute all metrics for frames that were recevied 58 before and report them. 59 60Also `VideoQualityAnalyzerInjectionHelper` implements 61[`webrtc::webrtc_pc_e2e::StatsObserverInterface`][11] to propagate WebRTC stats 62to `VideoQualityAnalyzerInterface`. 63 64### EncodedImageDataInjector and EncodedImageDataExtractor 65 66[`webrtc::webrtc_pc_e2e::EncodedImageDataInjector`][14] and 67[`webrtc::webrtc_pc_e2e::EncodedImageDataInjector`][15] are used to inject and 68extract data into `webrtc::EncodedImage` to propagate frame ID and other 69required information through the network. 70 71By default [`webrtc::webrtc_pc_e2e::SingleProcessEncodedImageDataInjector`][16] 72is used. It assumes `webrtc::EncodedImage` payload as black box which is 73remaining unchanged from encoder to decoder and stores the information required 74for its work in the last 3 bytes of the payload, replacing the original data 75during injection and restoring it back during extraction. Also 76`SingleProcessEncodedImageDataInjector` requires that sender and receiver were 77inside single process. 78 79 80 81## Exported metrics 82 83Exported metrics are reported to WebRTC perf results reporting system. 84 85### General 86 87* *`cpu_usage`* - CPU usage excluding video analyzer 88 89### Video 90 91* *`psnr`* - peak signal-to-noise ratio: 92 [wikipedia](https://en.wikipedia.org/wiki/Peak_signal-to-noise_ratio) 93* *`ssim`* - structural similarity: 94 [wikipedia](https://en.wikipedia.org/wiki/Structural_similarity). 95* *`min_psnr`* - minimum value of psnr across all frames of video stream. 96* *`encode_time`* - time to encode a single frame. 97* *`decode_time`* - time to decode a single frame. 98* *`transport_time`* - time from frame encoded to frame received for decoding. 99* *`receive_to_render_time`* - time from frame received for decoding to frame 100 rendered. 101* *`total_delay_incl_transport`* - time from frame was captured on device to 102 time when frame was displayed on device. 103* *`encode_frame_rate`* - frame rate after encoder. 104* *`harmonic_framerate`* - video duration divided on squared sum of interframe 105 delays. Reflects render frame rate penalized by freezes. 106* *`time_between_rendered_frames`* - time between frames out to renderer. 107* *`dropped_frames`* - amount of frames that were sent, but weren't rendered 108 and are known not to be “on the way” from sender to receiver. 109 110Freeze is a pause when no new frames from decoder arrived for 150ms + avg time 111between frames or 3 * avg time between frames. 112 113* *`time_between_freezes`* - mean time from previous freeze end to new freeze 114 start. 115* *`freeze_time_ms`* - total freeze time in ms. 116* *`max_skipped`* - frames skipped between two nearest rendered. 117* *`pixels_per_frame`* - amount of pixels on frame (width * height). 118* *`target_encode_bitrate`* - target encode bitrate provided by BWE to 119 encoder. 120* *`actual_encode_bitrate -`* - actual encode bitrate produced by encoder. 121* *`available_send_bandwidth -`* - available send bandwidth estimated by BWE. 122* *`transmission_bitrate`* - bitrate of media in the emulated network, not 123 counting retransmissions FEC, and RTCP messages 124* *`retransmission_bitrate`* - bitrate of retransmission streams only. 125 126### Framework stability 127 128* *`frames_in_flight`* - amount of frames that were captured but wasn't seen 129 on receiver. 130 131## Debug metrics 132 133Debug metrics are not reported to WebRTC perf results reporting system, but are 134available through `DefaultVideoQualityAnalyzer` API. 135 136### [FrameCounters][12] 137 138Frame counters consist of next counters: 139 140* *`captured`* - count of frames, that were passed into WebRTC pipeline by 141 video stream source 142* *`pre_encoded`* - count of frames that reached video encoder. 143* *`encoded`* - count of encoded images that were produced by encoder for all 144 requested spatial layers and simulcast streams. 145* *`received`* - count of encoded images received in decoder for all requested 146 spatial layers and simulcast streams. 147* *`decoded`* - count of frames that were produced by decoder. 148* *`rendered`* - count of frames that went out from WebRTC pipeline to video 149 sink. 150* *`dropped`* - count of frames that were dropped in any point between 151 capturing and rendering. 152 153`DefaultVideoQualityAnalyzer` exports these frame counters: 154 155* *`GlobalCounters`* - frame counters for frames met on each stage of analysis 156 for all media streams. 157* *`PerStreamCounters`* - frame counters for frames met on each stage of 158 analysis separated per individual video track (single media section in the 159 SDP offer). 160 161### [AnalyzerStats][13] 162 163Contains metrics about internal state of video analyzer during its work 164 165* *`comparisons_queue_size`* - size of analyzer internal queue used to perform 166 captured and rendered frames comparisons measured when new element is added 167 to the queue. 168* *`comparisons_done`* - number of performed comparisons of 2 video frames 169 from captured and rendered streams. 170* *`cpu_overloaded_comparisons_done`* - number of cpu overloaded comparisons. 171 Comparison is cpu overloaded if it is queued when there are too many not 172 processed comparisons in the queue. Overloaded comparison doesn't include 173 metrics like SSIM and PSNR that require heavy computations. 174* *`memory_overloaded_comparisons_done`* - number of memory overloaded 175 comparisons. Comparison is memory overloaded if it is queued when its 176 captured frame was already removed due to high memory usage for that video 177 stream. 178* *`frames_in_flight_left_count`* - count of frames in flight in analyzer 179 measured when new comparison is added and after analyzer was stopped. 180 181[1]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/test/pc/e2e/analyzer/video/default_video_quality_analyzer.h;l=188;drc=08f46909a8735cf181b99ef2f7e1791c5a7531d2 182[2]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/api/test/video_quality_analyzer_interface.h;l=56;drc=d7808f1c464a07c8f1e2f97ec7ee92fda998d590 183[3]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/test/pc/e2e/analyzer/video/video_quality_analyzer_injection_helper.h;l=39;drc=08f46909a8735cf181b99ef2f7e1791c5a7531d2 184[4]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/api/video_codecs/video_encoder_factory.h;l=27;drc=08f46909a8735cf181b99ef2f7e1791c5a7531d2 185[5]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/api/video_codecs/video_decoder_factory.h;l=27;drc=08f46909a8735cf181b99ef2f7e1791c5a7531d2 186[6]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/api/video/video_frame.h;l=30;drc=08f46909a8735cf181b99ef2f7e1791c5a7531d2 187[7]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/api/video/encoded_image.h;l=71;drc=08f46909a8735cf181b99ef2f7e1791c5a7531d2 188[8]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/test/test_video_capturer.h;l=28;drc=08f46909a8735cf181b99ef2f7e1791c5a7531d2 189[9]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/api/video/video_sink_interface.h;l=19;drc=08f46909a8735cf181b99ef2f7e1791c5a7531d2 190[10]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/test/testsupport/perf_test.h;drc=0710b401b1e5b500b8e84946fb657656ba1b58b7 191[11]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/api/test/stats_observer_interface.h;l=21;drc=9b526180c9e9722d3fc7f8689da6ec094fc7fc0a 192[12]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/test/pc/e2e/analyzer/video/default_video_quality_analyzer.h;l=57;drc=08f46909a8735cf181b99ef2f7e1791c5a7531d2 193[13]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/test/pc/e2e/analyzer/video/default_video_quality_analyzer.h;l=113;drc=08f46909a8735cf181b99ef2f7e1791c5a7531d2 194[14]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/test/pc/e2e/analyzer/video/encoded_image_data_injector.h;l=23;drc=c57089a97a3df454f4356d882cc8df173e8b3ead 195[15]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/test/pc/e2e/analyzer/video/encoded_image_data_injector.h;l=46;drc=c57089a97a3df454f4356d882cc8df173e8b3ead 196[16]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/test/pc/e2e/analyzer/video/single_process_encoded_image_data_injector.h;l=40;drc=c57089a97a3df454f4356d882cc8df173e8b3ead 197
