1*1b4853f5SAndroid Build Coastguard Worker# Virtio-media 2*1b4853f5SAndroid Build Coastguard Worker 3*1b4853f5SAndroid Build Coastguard WorkerThis is a virtio protocol definition, companion Linux guest kernel driver, and 4*1b4853f5SAndroid Build Coastguard Workerset of host-side devices for virtualizing media devices using virtio, following 5*1b4853f5SAndroid Build Coastguard Workerthe same model (and structures) as V4L2. It can be used to virtualize cameras, 6*1b4853f5SAndroid Build Coastguard Workercodec devices, or any other device supported by V4L2. 7*1b4853f5SAndroid Build Coastguard Worker 8*1b4853f5SAndroid Build Coastguard WorkerWant to try it? See the [TRY_IT_OUT document](/TRY_IT_OUT.md). 9*1b4853f5SAndroid Build Coastguard Worker 10*1b4853f5SAndroid Build Coastguard WorkerV4L2 is a UAPI that allows a less privileged entity (user-space) to use video 11*1b4853f5SAndroid Build Coastguard Workerhardware exposed by a more privileged entity (the kernel). Virtio-media is an 12*1b4853f5SAndroid Build Coastguard Workerencapsulation of this API into virtio, turning it into a virtualization API for 13*1b4853f5SAndroid Build Coastguard Workerall classes of video devices supported by V4L2, where the host plays the role of 14*1b4853f5SAndroid Build Coastguard Workerthe kernel and the guest the role of user-space. 15*1b4853f5SAndroid Build Coastguard Worker 16*1b4853f5SAndroid Build Coastguard WorkerThe host is therefore responsible for presenting a virtual device that behaves 17*1b4853f5SAndroid Build Coastguard Workerlike an actual V4L2 device, which the guest can control. 18*1b4853f5SAndroid Build Coastguard Worker 19*1b4853f5SAndroid Build Coastguard WorkerThis repository includes a simple guest Linux kernel module supporting this 20*1b4853f5SAndroid Build Coastguard Workerprotocol. On the host side, devices can be implemented in several ways: 21*1b4853f5SAndroid Build Coastguard Worker 22*1b4853f5SAndroid Build Coastguard Worker1. By forwarding a V4L2 device from the host into a guest. This works if the 23*1b4853f5SAndroid Build Coastguard Worker device is already supported by V4L2 on the host. 24*1b4853f5SAndroid Build Coastguard Worker2. By emulating a V4L2 device on the host, from the actual interface that the 25*1b4853f5SAndroid Build Coastguard Worker device provides. 26*1b4853f5SAndroid Build Coastguard Worker 27*1b4853f5SAndroid Build Coastguard WorkerNote that virtio-media does not require the use of a V4L2 device driver or of 28*1b4853f5SAndroid Build Coastguard WorkerLinux on the host or guest side - V4L2 is only used as a host-guest protocol, 29*1b4853f5SAndroid Build Coastguard Workerand both sides are free to convert it from/to any model that they wish to use. 30*1b4853f5SAndroid Build Coastguard Worker 31*1b4853f5SAndroid Build Coastguard WorkerThe complete definition of V4L2 structures and ioctls can be found under the 32*1b4853f5SAndroid Build Coastguard Worker[V4L2 UAPI documentation](https://www.kernel.org/doc/html/latest/userspace-api/media/index.html), 33*1b4853f5SAndroid Build Coastguard Workerwhich should be referred to alongside this document. 34*1b4853f5SAndroid Build Coastguard Worker 35*1b4853f5SAndroid Build Coastguard Worker## Driver status 36*1b4853f5SAndroid Build Coastguard Worker 37*1b4853f5SAndroid Build Coastguard WorkerThe driver (in the `driver/` directory) should be working and supporting most 38*1b4853f5SAndroid Build Coastguard WorkerV4L2 features, with the exception of the following: 39*1b4853f5SAndroid Build Coastguard Worker 40*1b4853f5SAndroid Build Coastguard Worker- [Read/Write API](https://www.kernel.org/doc/html/v4.8/media/uapi/v4l/rw.html), 41*1b4853f5SAndroid Build Coastguard Worker which is obsolete and inefficient. 42*1b4853f5SAndroid Build Coastguard Worker- Overlay interface and associated ioctls, i.e. 43*1b4853f5SAndroid Build Coastguard Worker - `VIDIOC_OVERLAY` 44*1b4853f5SAndroid Build Coastguard Worker - `VIDIOC_G/S_FBUF` 45*1b4853f5SAndroid Build Coastguard Worker- `DMABUF` buffers (this will be supported at least for virtio objects, other 46*1b4853f5SAndroid Build Coastguard Worker kinds of DMABUFs may or may not be usable) 47*1b4853f5SAndroid Build Coastguard Worker- `VIDIOC_EXPBUF` (to be implemented) 48*1b4853f5SAndroid Build Coastguard Worker- `VIDIOC_G/S_EDID` (to be implemented if it makes sense in a virtual context) 49*1b4853f5SAndroid Build Coastguard Worker- Media API and requests. This will probably be supported in the future behind a 50*1b4853f5SAndroid Build Coastguard Worker feature flag. 51*1b4853f5SAndroid Build Coastguard Worker 52*1b4853f5SAndroid Build Coastguard Worker## Devices status 53*1b4853f5SAndroid Build Coastguard Worker 54*1b4853f5SAndroid Build Coastguard WorkerThe `devices/` directory contains a Rust crate implementing helper functions to 55*1b4853f5SAndroid Build Coastguard Workerparse the protocol, interfaces to easily implement devices, and a couple of 56*1b4853f5SAndroid Build Coastguard Workerdevice implementations. It is written to be easily pluggable on any VMM. See the 57*1b4853f5SAndroid Build Coastguard Workerrustdoc in the `devices/` directory for more information. 58*1b4853f5SAndroid Build Coastguard Worker 59*1b4853f5SAndroid Build Coastguard WorkerImplemented devices are: 60*1b4853f5SAndroid Build Coastguard Worker 61*1b4853f5SAndroid Build Coastguard Worker- A simple video capture device generating a pattern (`simple_device.rs`), 62*1b4853f5SAndroid Build Coastguard Worker purely software-based and thus not requiring any kind of hardware. This is 63*1b4853f5SAndroid Build Coastguard Worker here for reference and testing purposes. 64*1b4853f5SAndroid Build Coastguard Worker- A proxy device for host V4L2 devices, i.e. a device allowing to expose a host 65*1b4853f5SAndroid Build Coastguard Worker V4L2 device to the guest almost as-is (`v4l2_device_proxy.rs`). 66*1b4853f5SAndroid Build Coastguard Worker 67*1b4853f5SAndroid Build Coastguard Worker* A FFmpeg-based video decoder device as a separate crate in 68*1b4853f5SAndroid Build Coastguard Worker `extras/ffmpeg-decoder`. 69*1b4853f5SAndroid Build Coastguard Worker 70*1b4853f5SAndroid Build Coastguard Worker## Virtio device ID 71*1b4853f5SAndroid Build Coastguard Worker 72*1b4853f5SAndroid Build Coastguard WorkerVirtio-media uses device ID `49`. 73*1b4853f5SAndroid Build Coastguard Worker 74*1b4853f5SAndroid Build Coastguard Worker## Virtqueues 75*1b4853f5SAndroid Build Coastguard Worker 76*1b4853f5SAndroid Build Coastguard WorkerThere are two queues in use: 77*1b4853f5SAndroid Build Coastguard Worker 78*1b4853f5SAndroid Build Coastguard Worker0 : commandq - queue for driver commands and device responses to these commands. 79*1b4853f5SAndroid Build Coastguard WorkerThe device MUST return the descriptor chains it receives as soon as possible, 80*1b4853f5SAndroid Build Coastguard Workerand must never hold to them for indefinite periods of time. 81*1b4853f5SAndroid Build Coastguard Worker 82*1b4853f5SAndroid Build Coastguard Worker1 : eventq - queue for events sent by the device to the driver. The driver MUST 83*1b4853f5SAndroid Build Coastguard Workerre-queue the descriptor chains returned by the device as soon as possible, and 84*1b4853f5SAndroid Build Coastguard Workermust never hold on them for indefinite periods of time. 85*1b4853f5SAndroid Build Coastguard Worker 86*1b4853f5SAndroid Build Coastguard Worker## Configuration area 87*1b4853f5SAndroid Build Coastguard Worker 88*1b4853f5SAndroid Build Coastguard WorkerThe configuration area contains the following information: 89*1b4853f5SAndroid Build Coastguard Worker 90*1b4853f5SAndroid Build Coastguard Worker```c 91*1b4853f5SAndroid Build Coastguard Workerstruct virtio_v4l2_config { 92*1b4853f5SAndroid Build Coastguard Worker /// The device_caps field of struct video_device. 93*1b4853f5SAndroid Build Coastguard Worker u32 device_caps; 94*1b4853f5SAndroid Build Coastguard Worker /// The vfl_devnode_type of the device. 95*1b4853f5SAndroid Build Coastguard Worker u32 device_type; 96*1b4853f5SAndroid Build Coastguard Worker /// The `card` field of v4l2_capability. 97*1b4853f5SAndroid Build Coastguard Worker u8 card[32]; 98*1b4853f5SAndroid Build Coastguard Worker} 99*1b4853f5SAndroid Build Coastguard Worker``` 100*1b4853f5SAndroid Build Coastguard Worker 101*1b4853f5SAndroid Build Coastguard Worker## Shared memory regions 102*1b4853f5SAndroid Build Coastguard Worker 103*1b4853f5SAndroid Build Coastguard WorkerShared memory region `0` is used to map `MMAP` buffers into the guest using the 104*1b4853f5SAndroid Build Coastguard Worker`VIRTIO_MEDIA_CMD_MMAP` command. If the host does not provide it, then `MMAP` 105*1b4853f5SAndroid Build Coastguard Workerbuffers cannot be mapped into the guest. 106*1b4853f5SAndroid Build Coastguard Worker 107*1b4853f5SAndroid Build Coastguard Worker## Protocol 108*1b4853f5SAndroid Build Coastguard Worker 109*1b4853f5SAndroid Build Coastguard WorkerAll structures managing the virtio protocol are defined and documented in 110*1b4853f5SAndroid Build Coastguard Worker`protocol.h`. Please refer to this file whenever a `virtio_media_cmd_*` or 111*1b4853f5SAndroid Build Coastguard Worker`virtio_media_resp_*` structure is mentioned. 112*1b4853f5SAndroid Build Coastguard Worker 113*1b4853f5SAndroid Build Coastguard WorkerCommands are queued on the `commandq` by the driver for the device to process. 114*1b4853f5SAndroid Build Coastguard WorkerThey all start by an instance of `struct virtio_media_cmd_header` and include 115*1b4853f5SAndroid Build Coastguard Workerdevice-writable descriptors for the device to write the result of the command in 116*1b4853f5SAndroid Build Coastguard Workera `struct virtio_media_resp_header`. 117*1b4853f5SAndroid Build Coastguard Worker 118*1b4853f5SAndroid Build Coastguard WorkerThe errors returned by each command are standard Linux kernel error codes. For 119*1b4853f5SAndroid Build Coastguard Workerinstance, a command that contains invalid options will return `EINVAL`. 120*1b4853f5SAndroid Build Coastguard Worker 121*1b4853f5SAndroid Build Coastguard WorkerEvents are sent on the `eventq` by the device for the driver to handle. They all 122*1b4853f5SAndroid Build Coastguard Workerstart by an instance of `struct virtio_media_event_header`. 123*1b4853f5SAndroid Build Coastguard Worker 124*1b4853f5SAndroid Build Coastguard Worker## Session management 125*1b4853f5SAndroid Build Coastguard Worker 126*1b4853f5SAndroid Build Coastguard WorkerIn order to use the device, the driver needs to open a session. This act is 127*1b4853f5SAndroid Build Coastguard Workerequivalent to opening the `/dev/videoX` device file of the V4L2 device. 128*1b4853f5SAndroid Build Coastguard WorkerDepending on the type of device, it may be possible to open several sessions 129*1b4853f5SAndroid Build Coastguard Workerconcurrently. 130*1b4853f5SAndroid Build Coastguard Worker 131*1b4853f5SAndroid Build Coastguard WorkerA session is opened by queueing a `struct virtio_media_cmd_open` along with a 132*1b4853f5SAndroid Build Coastguard Workerdescriptor to receive a `struct virtio_media_resp_open` to the commandq. An open 133*1b4853f5SAndroid Build Coastguard Workersession can be closed with `struct virtio_media_cmd_close`. 134*1b4853f5SAndroid Build Coastguard Worker 135*1b4853f5SAndroid Build Coastguard WorkerWhile the session is opened, its ID can be used to perform actions on it, most 136*1b4853f5SAndroid Build Coastguard Workercommonly V4L2 ioctls. 137*1b4853f5SAndroid Build Coastguard Worker 138*1b4853f5SAndroid Build Coastguard Worker## Ioctls 139*1b4853f5SAndroid Build Coastguard Worker 140*1b4853f5SAndroid Build Coastguard WorkerIoctls are the main way to interact with V4L2 devices, and therefore 141*1b4853f5SAndroid Build Coastguard Workervirtio-media features a command to perform an ioctl on an open session. 142*1b4853f5SAndroid Build Coastguard Worker 143*1b4853f5SAndroid Build Coastguard WorkerIn order to perform an ioctl, the driver queues a 144*1b4853f5SAndroid Build Coastguard Worker`struct virtio_media_cmd_ioctl` along with a descriptor to receive a 145*1b4853f5SAndroid Build Coastguard Worker`struct virtio_media_resp_ioctl` on the commandq. The code of the ioctl can be 146*1b4853f5SAndroid Build Coastguard Workerextracted from the 147*1b4853f5SAndroid Build Coastguard Worker[videodev2.h](https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/videodev.html) 148*1b4853f5SAndroid Build Coastguard Workerheader file, which defines the ioctls' codes, type of payload, and direction. 149*1b4853f5SAndroid Build Coastguard WorkerFor instance, the `VIDIOC_G_FMT` ioctl is defined as follows: 150*1b4853f5SAndroid Build Coastguard Worker 151*1b4853f5SAndroid Build Coastguard Worker```c 152*1b4853f5SAndroid Build Coastguard Worker#define VIDIOC_G_FMT _IOWR('V', 4, struct v4l2_format) 153*1b4853f5SAndroid Build Coastguard Worker``` 154*1b4853f5SAndroid Build Coastguard Worker 155*1b4853f5SAndroid Build Coastguard WorkerThis tells us that its ioctl code is `4`, that its payload is a 156*1b4853f5SAndroid Build Coastguard Worker`struct v4l2_format`, and that its direction is `WR`, i.e. the payload is 157*1b4853f5SAndroid Build Coastguard Workerwritten by both the driver and the device. 158*1b4853f5SAndroid Build Coastguard Worker 159*1b4853f5SAndroid Build Coastguard WorkerThe payload layout is always a 64-bit representation of the corresponding V4L2 160*1b4853f5SAndroid Build Coastguard Workerstructure, irrespective of the host and guest architecture. 161*1b4853f5SAndroid Build Coastguard Worker 162*1b4853f5SAndroid Build Coastguard Worker### Ioctls payload 163*1b4853f5SAndroid Build Coastguard Worker 164*1b4853f5SAndroid Build Coastguard WorkerThe payload of an ioctl in the descriptor chain follows the command structure, 165*1b4853f5SAndroid Build Coastguard Workerthe reponse structure, or both depending on the direction: 166*1b4853f5SAndroid Build Coastguard Worker 167*1b4853f5SAndroid Build Coastguard Worker- An `_IOR` ioctl is read-only for the driver, meaning the payload follows the 168*1b4853f5SAndroid Build Coastguard Worker response in the device-writable section of the descriptor chain. 169*1b4853f5SAndroid Build Coastguard Worker- An `_IOW` ioctl is read-only for the device, meaning the payload follows the 170*1b4853f5SAndroid Build Coastguard Worker command in the driver-writable section of the descriptor chain. 171*1b4853f5SAndroid Build Coastguard Worker- An `_IORW` ioctl is writable by both the device and driver, meaning the 172*1b4853f5SAndroid Build Coastguard Worker payload must follow both the command in the driver-writable section of the 173*1b4853f5SAndroid Build Coastguard Worker descriptor chain, and the response in the device-writable section. 174*1b4853f5SAndroid Build Coastguard Worker 175*1b4853f5SAndroid Build Coastguard WorkerFor instance, the `VIDIOC_G_STD` ioctl is defined as follows: 176*1b4853f5SAndroid Build Coastguard Worker 177*1b4853f5SAndroid Build Coastguard Worker```c 178*1b4853f5SAndroid Build Coastguard Worker#define VIDIOC_G_STD _IOR('V', 23, v4l2_std_id) 179*1b4853f5SAndroid Build Coastguard Worker``` 180*1b4853f5SAndroid Build Coastguard Worker 181*1b4853f5SAndroid Build Coastguard WorkerThus, its layout on the commandq will be: 182*1b4853f5SAndroid Build Coastguard Worker 183*1b4853f5SAndroid Build Coastguard Worker```text 184*1b4853f5SAndroid Build Coastguard Worker+-------------------------------------+ 185*1b4853f5SAndroid Build Coastguard Worker| struct virtio_media_cmd_ioctl | 186*1b4853f5SAndroid Build Coastguard Worker+=====================================+ 187*1b4853f5SAndroid Build Coastguard Worker| struct virtio_media_resp_ioctl | 188*1b4853f5SAndroid Build Coastguard Worker+-------------------------------------+ 189*1b4853f5SAndroid Build Coastguard Worker| v4l2_std_id | 190*1b4853f5SAndroid Build Coastguard Worker+-------------------------------------+ 191*1b4853f5SAndroid Build Coastguard Worker``` 192*1b4853f5SAndroid Build Coastguard Worker 193*1b4853f5SAndroid Build Coastguard Worker(in these diagrams, the `====` line signals the delimitation between 194*1b4853f5SAndroid Build Coastguard Workerdevice-readable and device-writable descriptors). 195*1b4853f5SAndroid Build Coastguard Worker 196*1b4853f5SAndroid Build Coastguard Worker`VIDIOC_SUBSCRIBE_EVENT` is defined as follows: 197*1b4853f5SAndroid Build Coastguard Worker 198*1b4853f5SAndroid Build Coastguard Worker```c 199*1b4853f5SAndroid Build Coastguard Worker#define VIDIOC_SUBSCRIBE_EVENT _IOW('V', 90, struct v4l2_event_subscription) 200*1b4853f5SAndroid Build Coastguard Worker``` 201*1b4853f5SAndroid Build Coastguard Worker 202*1b4853f5SAndroid Build Coastguard WorkerMeaning its layout on the commandq will be: 203*1b4853f5SAndroid Build Coastguard Worker 204*1b4853f5SAndroid Build Coastguard Worker```text 205*1b4853f5SAndroid Build Coastguard Worker+-------------------------------------+ 206*1b4853f5SAndroid Build Coastguard Worker| struct virtio_media_cmd_ioctl | 207*1b4853f5SAndroid Build Coastguard Worker+-------------------------------------+ 208*1b4853f5SAndroid Build Coastguard Worker| struct v4l2_event_subscription | 209*1b4853f5SAndroid Build Coastguard Worker+=====================================+ 210*1b4853f5SAndroid Build Coastguard Worker| struct virtio_media_resp_ioctl | 211*1b4853f5SAndroid Build Coastguard Worker+-------------------------------------+ 212*1b4853f5SAndroid Build Coastguard Worker``` 213*1b4853f5SAndroid Build Coastguard Worker 214*1b4853f5SAndroid Build Coastguard WorkerFinally, `VIDIOC_G_FMT` is a `WR` ioctl: 215*1b4853f5SAndroid Build Coastguard Worker 216*1b4853f5SAndroid Build Coastguard Worker```c 217*1b4853f5SAndroid Build Coastguard Worker#define VIDIOC_G_FMT _IOWR('V', 4, struct v4l2_format) 218*1b4853f5SAndroid Build Coastguard Worker``` 219*1b4853f5SAndroid Build Coastguard Worker 220*1b4853f5SAndroid Build Coastguard WorkerIts layout on the commandq will thus be: 221*1b4853f5SAndroid Build Coastguard Worker 222*1b4853f5SAndroid Build Coastguard Worker```text 223*1b4853f5SAndroid Build Coastguard Worker+-------------------------------------+ 224*1b4853f5SAndroid Build Coastguard Worker| struct virtio_media_cmd_ioctl | 225*1b4853f5SAndroid Build Coastguard Worker+-------------------------------------+ 226*1b4853f5SAndroid Build Coastguard Worker| struct v4l2_format | 227*1b4853f5SAndroid Build Coastguard Worker+=====================================+ 228*1b4853f5SAndroid Build Coastguard Worker| struct virtio_media_resp_ioctl | 229*1b4853f5SAndroid Build Coastguard Worker+-------------------------------------+ 230*1b4853f5SAndroid Build Coastguard Worker| struct v4l2_format | 231*1b4853f5SAndroid Build Coastguard Worker+-------------------------------------+ 232*1b4853f5SAndroid Build Coastguard Worker``` 233*1b4853f5SAndroid Build Coastguard Worker 234*1b4853f5SAndroid Build Coastguard WorkerA common optimization for `WR` ioctls is to provide the payload using 235*1b4853f5SAndroid Build Coastguard Workerdescriptors that both point to the same buffer. This mimics the behavior of V4L2 236*1b4853f5SAndroid Build Coastguard Workerioctls where the data is only passed once and used as both input and output by 237*1b4853f5SAndroid Build Coastguard Workerthe kernel. 238*1b4853f5SAndroid Build Coastguard Worker 239*1b4853f5SAndroid Build Coastguard WorkerIn case of success, the device MUST always write the payload in the 240*1b4853f5SAndroid Build Coastguard Workerdevice-writable part of the descriptor chain. 241*1b4853f5SAndroid Build Coastguard Worker 242*1b4853f5SAndroid Build Coastguard WorkerIn case of failure, the device is free to write the payload in the 243*1b4853f5SAndroid Build Coastguard Workerdevice-writable part of the descriptor chain or not. Some errors may still 244*1b4853f5SAndroid Build Coastguard Workerresult in the payload being updated, and in this case the device is expected to 245*1b4853f5SAndroid Build Coastguard Workerwrite the updated payload (for instance, `G_EXT_CTRLS` may return `EINVAL` but 246*1b4853f5SAndroid Build Coastguard Workerupdate the `size` member of the requested controls if the provided size was not 247*1b4853f5SAndroid Build Coastguard Workerenough). If the device has not written the payload after an error, the driver 248*1b4853f5SAndroid Build Coastguard WorkerMUST assume that the payload has not been modified. 249*1b4853f5SAndroid Build Coastguard Worker 250*1b4853f5SAndroid Build Coastguard Worker### Handling of pointers in ioctl payload 251*1b4853f5SAndroid Build Coastguard Worker 252*1b4853f5SAndroid Build Coastguard WorkerA few structures used as ioctl payloads contain pointers the link to further 253*1b4853f5SAndroid Build Coastguard Workerdata needed for the ioctl. There are notably: 254*1b4853f5SAndroid Build Coastguard Worker 255*1b4853f5SAndroid Build Coastguard Worker- The `planes` pointer of `struct v4l2_buffer`, which size is determined by the 256*1b4853f5SAndroid Build Coastguard Worker `length` member, 257*1b4853f5SAndroid Build Coastguard Worker- The `controls` pointer of `struct v4l2_ext_controls`, which size is determined 258*1b4853f5SAndroid Build Coastguard Worker by the `count` member. 259*1b4853f5SAndroid Build Coastguard Worker 260*1b4853f5SAndroid Build Coastguard WorkerIf the size of the pointed area is determined to be non-zero, then the main 261*1b4853f5SAndroid Build Coastguard Workerpayload is immediately followed by the pointed data in their order of appearance 262*1b4853f5SAndroid Build Coastguard Workerin the structure, and the pointer value itself is ignored by the device, which 263*1b4853f5SAndroid Build Coastguard Workermust also return the value initially passed by the driver. For instance, for a 264*1b4853f5SAndroid Build Coastguard Worker`struct v4l2_ext_controls` which `count` is `16`: 265*1b4853f5SAndroid Build Coastguard Worker 266*1b4853f5SAndroid Build Coastguard Worker```text 267*1b4853f5SAndroid Build Coastguard Worker+--------------------------------------+ 268*1b4853f5SAndroid Build Coastguard Worker| struct v4l2_ext_controls | 269*1b4853f5SAndroid Build Coastguard Worker+--------------------------------------+ 270*1b4853f5SAndroid Build Coastguard Worker| struct v4l2_ext_control for plane 0 | 271*1b4853f5SAndroid Build Coastguard Worker| struct v4l2_ext_control for plane 1 | 272*1b4853f5SAndroid Build Coastguard Worker| ... | 273*1b4853f5SAndroid Build Coastguard Worker| struct v4l2_ext_control for plane 15 | 274*1b4853f5SAndroid Build Coastguard Worker+--------------------------------------+ 275*1b4853f5SAndroid Build Coastguard Worker``` 276*1b4853f5SAndroid Build Coastguard Worker 277*1b4853f5SAndroid Build Coastguard WorkerSimilarly, a multiplanar `struct v4l2_buffer` with its `length` member set to 3 278*1b4853f5SAndroid Build Coastguard Workerwill be laid out as follows: 279*1b4853f5SAndroid Build Coastguard Worker 280*1b4853f5SAndroid Build Coastguard Worker```text 281*1b4853f5SAndroid Build Coastguard Worker+-------------------------------------+ 282*1b4853f5SAndroid Build Coastguard Worker| struct v4l2_buffer | 283*1b4853f5SAndroid Build Coastguard Worker+-------------------------------------+ 284*1b4853f5SAndroid Build Coastguard Worker| struct v4l2_plane for plane 0 | 285*1b4853f5SAndroid Build Coastguard Worker| struct v4l2_plane for plane 1 | 286*1b4853f5SAndroid Build Coastguard Worker| struct v4l2_plane for plane 2 | 287*1b4853f5SAndroid Build Coastguard Worker+-------------------------------------+ 288*1b4853f5SAndroid Build Coastguard Worker``` 289*1b4853f5SAndroid Build Coastguard Worker 290*1b4853f5SAndroid Build Coastguard Worker### Handling of pointers to userspace memory 291*1b4853f5SAndroid Build Coastguard Worker 292*1b4853f5SAndroid Build Coastguard WorkerA few pointers are special in that they point to userspace memory. They are: 293*1b4853f5SAndroid Build Coastguard Worker 294*1b4853f5SAndroid Build Coastguard Worker- The `m.userptr` member of `struct v4l2_buffer` and `struct v4l2_plane` 295*1b4853f5SAndroid Build Coastguard Worker (technically an `unsigned long`, but designated a userspace address), 296*1b4853f5SAndroid Build Coastguard Worker- The `ptr` member of `struct v4l2_ext_ctrl`. 297*1b4853f5SAndroid Build Coastguard Worker 298*1b4853f5SAndroid Build Coastguard WorkerThese pointers can cover large areas of scattered memory, which has the 299*1b4853f5SAndroid Build Coastguard Workerpotential to require more descriptors than the virtio queue can provide. For 300*1b4853f5SAndroid Build Coastguard Workerthese particular pointers only, a list of `struct virtio_media_sg_entry` that 301*1b4853f5SAndroid Build Coastguard Workercovers the needed amount of memory for the pointer is used instead of using 302*1b4853f5SAndroid Build Coastguard Workerdescriptors to map the pointed memory directly. 303*1b4853f5SAndroid Build Coastguard Worker 304*1b4853f5SAndroid Build Coastguard WorkerFor each such pointer to read, the device reads as many SG entries as needed to 305*1b4853f5SAndroid Build Coastguard Workercover the length of the pointed buffer, as described by its parent structure 306*1b4853f5SAndroid Build Coastguard Worker(`length` member of `struct v4l2_buffer` or `struct v4l2_plane` for buffer 307*1b4853f5SAndroid Build Coastguard Workermemory, and `size` member of `struct v4l2_ext_control` for control data). 308*1b4853f5SAndroid Build Coastguard Worker 309*1b4853f5SAndroid Build Coastguard WorkerSince the device never needs to modify the list of SG entries, it is only 310*1b4853f5SAndroid Build Coastguard Workerprovided by the driver in the device-readable section of the descriptor chain, 311*1b4853f5SAndroid Build Coastguard Workerand not repeated in the device-writable section, even for `WR` ioctls. 312*1b4853f5SAndroid Build Coastguard Worker 313*1b4853f5SAndroid Build Coastguard WorkerTo illustrate the data layout, here is what the descriptor chain of a 314*1b4853f5SAndroid Build Coastguard Worker`VIDIOC_QBUF` ioctl queueing a 3-planar `USERPTR` buffer would look like: 315*1b4853f5SAndroid Build Coastguard Worker 316*1b4853f5SAndroid Build Coastguard Worker```text 317*1b4853f5SAndroid Build Coastguard Worker+---------------------------------------------------+ 318*1b4853f5SAndroid Build Coastguard Worker| struct virtio_media_cmd_ioctl | 319*1b4853f5SAndroid Build Coastguard Worker+---------------------------------------------------+ 320*1b4853f5SAndroid Build Coastguard Worker| struct v4l2_buffer | 321*1b4853f5SAndroid Build Coastguard Worker+---------------------------------------------------+ 322*1b4853f5SAndroid Build Coastguard Worker| struct v4l2_plane for plane 0 | 323*1b4853f5SAndroid Build Coastguard Worker| struct v4l2_plane for plane 1 | 324*1b4853f5SAndroid Build Coastguard Worker| struct v4l2_plane for plane 2 | 325*1b4853f5SAndroid Build Coastguard Worker+---------------------------------------------------+ 326*1b4853f5SAndroid Build Coastguard Worker| array of struct virtio_media_sg_entry for plane 0 | 327*1b4853f5SAndroid Build Coastguard Worker+---------------------------------------------------+ 328*1b4853f5SAndroid Build Coastguard Worker| array of struct virtio_media_sg_entry for plane 1 | 329*1b4853f5SAndroid Build Coastguard Worker+---------------------------------------------------+ 330*1b4853f5SAndroid Build Coastguard Worker| array of struct virtio_media_sg_entry for plane 2 | 331*1b4853f5SAndroid Build Coastguard Worker+===================================================+ 332*1b4853f5SAndroid Build Coastguard Worker| struct virtio_media_resp_ioctl | 333*1b4853f5SAndroid Build Coastguard Worker+---------------------------------------------------+ 334*1b4853f5SAndroid Build Coastguard Worker| struct v4l2_buffer | 335*1b4853f5SAndroid Build Coastguard Worker+---------------------------------------------------+ 336*1b4853f5SAndroid Build Coastguard Worker| struct v4l2_plane for plane 0 | 337*1b4853f5SAndroid Build Coastguard Worker| struct v4l2_plane for plane 1 | 338*1b4853f5SAndroid Build Coastguard Worker| struct v4l2_plane for plane 2 | 339*1b4853f5SAndroid Build Coastguard Worker+---------------------------------------------------+ 340*1b4853f5SAndroid Build Coastguard Worker``` 341*1b4853f5SAndroid Build Coastguard Worker 342*1b4853f5SAndroid Build Coastguard WorkerSince the ioctl is `RW`, the payload is repeated in both the device-readable and 343*1b4853f5SAndroid Build Coastguard Workerdevice-writable sections of the descriptor chain, but the device-writable 344*1b4853f5SAndroid Build Coastguard Workersection does not include the SG lists to guest memory. 345*1b4853f5SAndroid Build Coastguard Worker 346*1b4853f5SAndroid Build Coastguard Worker### Unsupported ioctls 347*1b4853f5SAndroid Build Coastguard Worker 348*1b4853f5SAndroid Build Coastguard WorkerA few ioctls are replaced by other, more suitable mechanisms. If being requested 349*1b4853f5SAndroid Build Coastguard Workerthese ioctls, the device must return the same response as it would for an 350*1b4853f5SAndroid Build Coastguard Workerunknown ioctl, i.e. `ENOTTY`. 351*1b4853f5SAndroid Build Coastguard Worker 352*1b4853f5SAndroid Build Coastguard Worker- `VIDIOC_QUERYCAP` is replaced by reading the configuration area. 353*1b4853f5SAndroid Build Coastguard Worker- `VIDIOC_DQBUF` is replaced by a dedicated event. 354*1b4853f5SAndroid Build Coastguard Worker- `VIDIOC_DQEVENT` is replaced by a dedicated event. 355*1b4853f5SAndroid Build Coastguard Worker- `VIDIOC_G_JPEGCOMP` and `VIDIOC_S_JPEGCOMP` are deprecated and replaced by the 356*1b4853f5SAndroid Build Coastguard Worker controls of the JPEG class. 357*1b4853f5SAndroid Build Coastguard Worker- `VIDIOC_LOG_STATUS` is a guest-only operation and shall not be implemented by 358*1b4853f5SAndroid Build Coastguard Worker the host. 359*1b4853f5SAndroid Build Coastguard Worker 360*1b4853f5SAndroid Build Coastguard Worker## Events 361*1b4853f5SAndroid Build Coastguard Worker 362*1b4853f5SAndroid Build Coastguard WorkerEvents are a way for the device to inform the driver about asynchronous events 363*1b4853f5SAndroid Build Coastguard Workerthat it should know about. In virtio-media, they are used as a replacement for 364*1b4853f5SAndroid Build Coastguard Workerthe `VIDIOC_DQBUF` and `VIDIOC_DQEVENT` ioctls and the polling mechanism, which 365*1b4853f5SAndroid Build Coastguard Workerwould be impractical to implement on top of virtio. 366*1b4853f5SAndroid Build Coastguard Worker 367*1b4853f5SAndroid Build Coastguard Worker### Dequeued buffer events 368*1b4853f5SAndroid Build Coastguard Worker 369*1b4853f5SAndroid Build Coastguard WorkerA `struct virtio_media_event_dqbuf` event is queued on the eventq by the device 370*1b4853f5SAndroid Build Coastguard Workerevery time a buffer previously queued using the `VIDIOC_QBUF` ioctl is done 371*1b4853f5SAndroid Build Coastguard Workerbeing processed and can be used by the driver again. This is like an implicit 372*1b4853f5SAndroid Build Coastguard Worker`VIDIOC_DQBUF` ioctl. 373*1b4853f5SAndroid Build Coastguard Worker 374*1b4853f5SAndroid Build Coastguard WorkerPointer values in the `struct v4l2_buffer` and `struct v4l2_plane` are 375*1b4853f5SAndroid Build Coastguard Workermeaningless and must be ignored by the driver. It is recommended that the device 376*1b4853f5SAndroid Build Coastguard Workersets them to `NULL` in order to avoid leaking potential host addresses. 377*1b4853f5SAndroid Build Coastguard Worker 378*1b4853f5SAndroid Build Coastguard WorkerNote that in the case of a `USERPTR` buffer, the `struct v4l2_buffer` used as 379*1b4853f5SAndroid Build Coastguard Workerevent payload is not followed by the buffer memory: since that memory is the 380*1b4853f5SAndroid Build Coastguard Workersame that the driver submitted with the `VIDIOC_QBUF`, it would be redundant to 381*1b4853f5SAndroid Build Coastguard Workerhave it here. 382*1b4853f5SAndroid Build Coastguard Worker 383*1b4853f5SAndroid Build Coastguard Worker### Dequeued V4L2 event event 384*1b4853f5SAndroid Build Coastguard Worker 385*1b4853f5SAndroid Build Coastguard WorkerA `struct virtio_media_event_event` event is queued on the eventq by the device 386*1b4853f5SAndroid Build Coastguard Workerevery time an event the driver previously subscribed to using the 387*1b4853f5SAndroid Build Coastguard Worker`VIDIOC_SUBSCRIBE_EVENT` ioctl has been signaled. This is like an implicit 388*1b4853f5SAndroid Build Coastguard Worker`VIDIOC_DQEVENT` ioctl. 389*1b4853f5SAndroid Build Coastguard Worker 390*1b4853f5SAndroid Build Coastguard Worker## Memory types 391*1b4853f5SAndroid Build Coastguard Worker 392*1b4853f5SAndroid Build Coastguard WorkerThe semantics of the three V4L2 memory types (`MMAP`, `USERPTR` and `DMABUF`) 393*1b4853f5SAndroid Build Coastguard Workercan easily be mapped to a guest/host context. 394*1b4853f5SAndroid Build Coastguard Worker 395*1b4853f5SAndroid Build Coastguard Worker### MMAP 396*1b4853f5SAndroid Build Coastguard Worker 397*1b4853f5SAndroid Build Coastguard WorkerIn virtio-media, `MMAP` buffers are provisioned by the host, just like they are 398*1b4853f5SAndroid Build Coastguard Workerby the kernel in regular V4L2. Similarly to how userspace can map a `MMAP` 399*1b4853f5SAndroid Build Coastguard Workerbuffer into its address space using `mmap` and `munmap`, the virtio-media driver 400*1b4853f5SAndroid Build Coastguard Workercan map host buffers into the guest space by queueing the 401*1b4853f5SAndroid Build Coastguard Worker`struct virtio_media_cmd_mmap` and `struct virtio_media_cmd_munmap` commands to 402*1b4853f5SAndroid Build Coastguard Workerthe commandq. 403*1b4853f5SAndroid Build Coastguard Worker 404*1b4853f5SAndroid Build Coastguard Worker### USERPTR 405*1b4853f5SAndroid Build Coastguard Worker 406*1b4853f5SAndroid Build Coastguard WorkerIn virtio-media, `USERPTR` buffers and provisioned by the guest, just like they 407*1b4853f5SAndroid Build Coastguard Workerare by userspace in regular V4L2. Instances of `struct v4l2_buffer` and 408*1b4853f5SAndroid Build Coastguard Worker`struct v4l2_plane` of this type are followed by a series of descriptors mapping 409*1b4853f5SAndroid Build Coastguard Workerthe buffer backing memory in guest space. 410*1b4853f5SAndroid Build Coastguard Worker 411*1b4853f5SAndroid Build Coastguard WorkerFor the host convenience, the backing memory must start with a new descriptor - 412*1b4853f5SAndroid Build Coastguard Workerthis allows the host to easily map the buffer memory to render into it instead 413*1b4853f5SAndroid Build Coastguard Workerof having to do a copy. 414*1b4853f5SAndroid Build Coastguard Worker 415*1b4853f5SAndroid Build Coastguard WorkerThe host must not alter the pointer values provided by the guest, i.e. the 416*1b4853f5SAndroid Build Coastguard Worker`m.userptr` member of `struct v4l2_buffer` and `struct v4l2_plane` must be 417*1b4853f5SAndroid Build Coastguard Workerreturned to the guest with the same value as it was provided. 418*1b4853f5SAndroid Build Coastguard Worker 419*1b4853f5SAndroid Build Coastguard Worker### DMABUF 420*1b4853f5SAndroid Build Coastguard Worker 421*1b4853f5SAndroid Build Coastguard WorkerIn virtio-media, `DMABUF` buffers are provisioned by a virtio object, just like 422*1b4853f5SAndroid Build Coastguard Workerthey are by a DMABUF in regular V4L2. Virtio objects are 16-bytes UUIDs and do 423*1b4853f5SAndroid Build Coastguard Workernot fit in the placeholders for file descriptors, so they follow their embedding 424*1b4853f5SAndroid Build Coastguard Workerdata structure as needed and the device must leave the V4L2 structure 425*1b4853f5SAndroid Build Coastguard Workerplaceholder unchanged. For instance, a 3-planar `struct v4l2_buffer` with the 426*1b4853f5SAndroid Build Coastguard Worker`V4L2_MEMORY_DMABUF` memory type will have the following layout: 427*1b4853f5SAndroid Build Coastguard Worker 428*1b4853f5SAndroid Build Coastguard Worker```text 429*1b4853f5SAndroid Build Coastguard Worker+-------------------------------------+ 430*1b4853f5SAndroid Build Coastguard Worker| struct v4l2_buffer | 431*1b4853f5SAndroid Build Coastguard Worker+-------------------------------------+ 432*1b4853f5SAndroid Build Coastguard Worker| struct v4l2_plane for plane 0 | 433*1b4853f5SAndroid Build Coastguard Worker| struct v4l2_plane for plane 1 | 434*1b4853f5SAndroid Build Coastguard Worker| struct v4l2_plane for plane 2 | 435*1b4853f5SAndroid Build Coastguard Worker+-------------------------------------+ 436*1b4853f5SAndroid Build Coastguard Worker| 16 byte UUID for plane 0 | 437*1b4853f5SAndroid Build Coastguard Worker+-------------------------------------+ 438*1b4853f5SAndroid Build Coastguard Worker| 16 byte UUID for plane 1 | 439*1b4853f5SAndroid Build Coastguard Worker+-------------------------------------+ 440*1b4853f5SAndroid Build Coastguard Worker| 16 byte UUID for plane 2 | 441*1b4853f5SAndroid Build Coastguard Worker+-------------------------------------+ 442*1b4853f5SAndroid Build Coastguard Worker``` 443*1b4853f5SAndroid Build Coastguard Worker 444*1b4853f5SAndroid Build Coastguard WorkerContrary to `USERPTR` buffers, virtio objects UUIDs need to be added in both the 445*1b4853f5SAndroid Build Coastguard Workerdevice-readable and device-writable section of the descriptor chain. 446*1b4853f5SAndroid Build Coastguard Worker 447*1b4853f5SAndroid Build Coastguard WorkerHost-allocated buffers with the `V4L2_MEMORY_MMAP` memory type can also be 448*1b4853f5SAndroid Build Coastguard Workerexported as virtio objects for use with another virtio device using the 449*1b4853f5SAndroid Build Coastguard Worker`VIDIOC_EXPBUF` ioctl. The `fd` placefolder of `v4l2_exportbuffer` means that 450*1b4853f5SAndroid Build Coastguard Workerspace for the UUID needs to be reserved right after that structure, so the ioctl 451*1b4853f5SAndroid Build Coastguard Workerlayout will looks as follows: 452*1b4853f5SAndroid Build Coastguard Worker 453*1b4853f5SAndroid Build Coastguard Worker```text 454*1b4853f5SAndroid Build Coastguard Worker+-------------------------------------+ 455*1b4853f5SAndroid Build Coastguard Worker| struct virtio_media_cmd_ioctl | 456*1b4853f5SAndroid Build Coastguard Worker+-------------------------------------+ 457*1b4853f5SAndroid Build Coastguard Worker| struct v4l2_exportbuffer | 458*1b4853f5SAndroid Build Coastguard Worker+-------------------------------------+ 459*1b4853f5SAndroid Build Coastguard Worker| 16 bytes UUID for exported buffer | 460*1b4853f5SAndroid Build Coastguard Worker+=====================================+ 461*1b4853f5SAndroid Build Coastguard Worker| struct virtio_media_resp_ioctl | 462*1b4853f5SAndroid Build Coastguard Worker+-------------------------------------+ 463*1b4853f5SAndroid Build Coastguard Worker| struct v4l2_exportbuffer | 464*1b4853f5SAndroid Build Coastguard Worker+-------------------------------------+ 465*1b4853f5SAndroid Build Coastguard Worker| 16 bytes UUID for exported buffer | 466*1b4853f5SAndroid Build Coastguard Worker+-------------------------------------+ 467*1b4853f5SAndroid Build Coastguard Worker``` 468