1<?% config.freshness.reviewed = '2021-04-13' %?> 2<?% config.freshness.owner = 'eshr' %?> 3 4# Video Adaptation 5 6Video adaptation is a mechanism which reduces the bandwidth or CPU consumption 7by reducing encoded video quality. 8 9## Overview 10 11Adaptation occurs when a _Resource_ signals that it is currently underused or 12overused. When overused, the video quality is decreased and when underused, the 13video quality is increased. There are currently two dimensions in which the 14quality can be adapted: frame-rate and resolution. The dimension that is adapted 15is based on the degradation preference for the video track. 16 17## Resources 18 19_Resources_ monitor metrics from the system or the video stream. For example, a 20resource could monitor system temperature or the bandwidth usage of the video 21stream. A resource implements the [Resource][resource.h] interface. When a 22resource detects that it is overused, it calls `SetUsageState(kOveruse)`. When 23the resource is no longer overused, it can signal this using 24`SetUsageState(kUnderuse)`. 25 26There are two resources that are used by default on all video tracks: Quality 27scaler resource and encode overuse resource. 28 29### QP Scaler Resource 30 31The quality scaler resource monitors the quantization parameter (QP) of the 32encoded video frames for video send stream and ensures that the quality of the 33stream is acceptable for the current resolution. After each frame is encoded the 34[QualityScaler][quality_scaler.h] is given the QP of the encoded frame. Overuse 35or underuse is signalled when the average QP is outside of the 36[QP thresholds][VideoEncoder::QpThresholds]. If the average QP is above the 37_high_ threshold, the QP scaler signals _overuse_, and when below the _low_ 38threshold the QP scaler signals _underuse_. 39 40The thresholds are set by the video encoder in the `scaling_settings` property 41of the [EncoderInfo][EncoderInfo]. 42 43*Note:* that the QP scaler is only enabled when the degradation preference is 44`MAINTAIN_FRAMERATE` or `BALANCED`. 45 46### Encode Usage Resource 47 48The [encoder usage resource][encode_usage_resource.h] monitors how long it takes 49to encode a video frame. This works as a good proxy measurement for CPU usage as 50contention increases when CPU usage is high, increasing the encode times of the 51video frames. 52 53The time is tracked from when frame encoding starts to when it is completed. If 54the average encoder usage exceeds the thresholds set, *overuse* is triggered. 55 56### Injecting other Resources 57 58A custom resource can be injected into the call using the 59[Call::AddAdaptationResource][Call::AddAdaptationResource] method. 60 61## Adaptation 62 63When a a *resource* signals the it is over or underused, this signal reaches the 64`ResourceAdaptationProcessor` who requests an `Adaptation` proposal from the 65[VideoStreamAdapter][VideoStreamAdapter]. This proposal is based on the 66degradation preference of the video stream. `ResourceAdaptationProcessor` will 67determine if the `Adaptation` should be applied based on the current adaptation 68status and the `Adaptation` proposal. 69 70### Degradation Preference 71 72There are 3 degradation preferences, described in the 73[RtpParameters][RtpParameters] header. These are 74 75* `MAINTIAIN_FRAMERATE`: Adapt video resolution 76* `MAINTIAIN_RESOLUTION`: Adapt video frame-rate. 77* `BALANCED`: Adapt video frame-rate or resolution. 78 79The degradation preference is set for a video track using the 80`degradation_preference` property in the [RtpParameters][RtpParameters]. 81 82## VideoSinkWants and video stream adaptation 83 84Once an adaptation is applied it notifies the video stream. The video stream 85converts this adaptation to a [VideoSinkWants][VideoSinkWants]. These sink wants 86indicate to the video stream that some restrictions should be applied to the 87stream before it is sent to encoding. It has a few properties, but for 88adaptation the properties that might be set are: 89 90* `target_pixel_count`: The desired number of pixels for each video frame. The 91 actual pixel count should be close to this but does not have to be exact so 92 that aspect ratio can be maintained. 93* `max_pixel_count`: The maximum number of pixels in each video frame. This 94 value can not be exceeded if set. 95* `max_framerate_fps`: The maximum frame-rate for the video source. The source 96 is expected to drop frames that cause this threshold to be exceeded. 97 98The `VideoSinkWants` can be applied by any video source, or one may use the 99[AdaptedVideoTraceSource][adapted_video_track_source.h] which is a base class 100for sources that need video adaptation. 101 102[RtpParameters]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/api/rtp_parameters.h?q=%22RTC_EXPORT%20RtpParameters%22 103[resource.h]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/api/adaptation/resource.h 104[Call::AddAdaptationResource]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/call/call.h?q=Call::AddAdaptationResource 105[quality_scaler.h]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/modules/video_coding/utility/quality_scaler.h 106[VideoEncoder::QpThresholds]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/api/video_codecs/video_encoder.h?q=VideoEncoder::QpThresholds 107[EncoderInfo]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/api/video_codecs/video_encoder.h?q=VideoEncoder::EncoderInfo 108[encode_usage_resource.h]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/video/adaptation/encode_usage_resource.h 109[VideoStreamAdapter]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/call/adaptation/video_stream_adapter.h 110[adaptation_constraint.h]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/call/adaptation/adaptation_constraint.h 111[bitrate_constraint.h]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/video/adaptation/bitrate_constraint.h 112[AddOrUpdateSink]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/api/video/video_source_interface.h?q=AddOrUpdateSink 113[VideoSinkWants]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/api/video/video_source_interface.h?q=%22RTC_EXPORT%20VideoSinkWants%22 114[adapted_video_track_source.h]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/media/base/adapted_video_track_source.h 115