xref: /aosp_15_r20/external/virtio-media/README.md (revision 1b4853f54772485c5dd4001ae33a7a958bcc97a1)
1*1b4853f5SAndroid Build Coastguard Worker# Virtio-media
2*1b4853f5SAndroid Build Coastguard Worker
3*1b4853f5SAndroid Build Coastguard WorkerThis is a virtio protocol definition, companion Linux guest kernel driver, and
4*1b4853f5SAndroid Build Coastguard Workerset of host-side devices for virtualizing media devices using virtio, following
5*1b4853f5SAndroid Build Coastguard Workerthe same model (and structures) as V4L2. It can be used to virtualize cameras,
6*1b4853f5SAndroid Build Coastguard Workercodec devices, or any other device supported by V4L2.
7*1b4853f5SAndroid Build Coastguard Worker
8*1b4853f5SAndroid Build Coastguard WorkerWant to try it? See the [TRY_IT_OUT document](/TRY_IT_OUT.md).
9*1b4853f5SAndroid Build Coastguard Worker
10*1b4853f5SAndroid Build Coastguard WorkerV4L2 is a UAPI that allows a less privileged entity (user-space) to use video
11*1b4853f5SAndroid Build Coastguard Workerhardware exposed by a more privileged entity (the kernel). Virtio-media is an
12*1b4853f5SAndroid Build Coastguard Workerencapsulation of this API into virtio, turning it into a virtualization API for
13*1b4853f5SAndroid Build Coastguard Workerall classes of video devices supported by V4L2, where the host plays the role of
14*1b4853f5SAndroid Build Coastguard Workerthe kernel and the guest the role of user-space.
15*1b4853f5SAndroid Build Coastguard Worker
16*1b4853f5SAndroid Build Coastguard WorkerThe host is therefore responsible for presenting a virtual device that behaves
17*1b4853f5SAndroid Build Coastguard Workerlike an actual V4L2 device, which the guest can control.
18*1b4853f5SAndroid Build Coastguard Worker
19*1b4853f5SAndroid Build Coastguard WorkerThis repository includes a simple guest Linux kernel module supporting this
20*1b4853f5SAndroid Build Coastguard Workerprotocol. On the host side, devices can be implemented in several ways:
21*1b4853f5SAndroid Build Coastguard Worker
22*1b4853f5SAndroid Build Coastguard Worker1. By forwarding a V4L2 device from the host into a guest. This works if the
23*1b4853f5SAndroid Build Coastguard Worker   device is already supported by V4L2 on the host.
24*1b4853f5SAndroid Build Coastguard Worker2. By emulating a V4L2 device on the host, from the actual interface that the
25*1b4853f5SAndroid Build Coastguard Worker   device provides.
26*1b4853f5SAndroid Build Coastguard Worker
27*1b4853f5SAndroid Build Coastguard WorkerNote that virtio-media does not require the use of a V4L2 device driver or of
28*1b4853f5SAndroid Build Coastguard WorkerLinux on the host or guest side - V4L2 is only used as a host-guest protocol,
29*1b4853f5SAndroid Build Coastguard Workerand both sides are free to convert it from/to any model that they wish to use.
30*1b4853f5SAndroid Build Coastguard Worker
31*1b4853f5SAndroid Build Coastguard WorkerThe complete definition of V4L2 structures and ioctls can be found under the
32*1b4853f5SAndroid Build Coastguard Worker[V4L2 UAPI documentation](https://www.kernel.org/doc/html/latest/userspace-api/media/index.html),
33*1b4853f5SAndroid Build Coastguard Workerwhich should be referred to alongside this document.
34*1b4853f5SAndroid Build Coastguard Worker
35*1b4853f5SAndroid Build Coastguard Worker## Driver status
36*1b4853f5SAndroid Build Coastguard Worker
37*1b4853f5SAndroid Build Coastguard WorkerThe driver (in the `driver/` directory) should be working and supporting most
38*1b4853f5SAndroid Build Coastguard WorkerV4L2 features, with the exception of the following:
39*1b4853f5SAndroid Build Coastguard Worker
40*1b4853f5SAndroid Build Coastguard Worker- [Read/Write API](https://www.kernel.org/doc/html/v4.8/media/uapi/v4l/rw.html),
41*1b4853f5SAndroid Build Coastguard Worker  which is obsolete and inefficient.
42*1b4853f5SAndroid Build Coastguard Worker- Overlay interface and associated ioctls, i.e.
43*1b4853f5SAndroid Build Coastguard Worker  - `VIDIOC_OVERLAY`
44*1b4853f5SAndroid Build Coastguard Worker  - `VIDIOC_G/S_FBUF`
45*1b4853f5SAndroid Build Coastguard Worker- `DMABUF` buffers (this will be supported at least for virtio objects, other
46*1b4853f5SAndroid Build Coastguard Worker  kinds of DMABUFs may or may not be usable)
47*1b4853f5SAndroid Build Coastguard Worker- `VIDIOC_EXPBUF` (to be implemented)
48*1b4853f5SAndroid Build Coastguard Worker- `VIDIOC_G/S_EDID` (to be implemented if it makes sense in a virtual context)
49*1b4853f5SAndroid Build Coastguard Worker- Media API and requests. This will probably be supported in the future behind a
50*1b4853f5SAndroid Build Coastguard Worker  feature flag.
51*1b4853f5SAndroid Build Coastguard Worker
52*1b4853f5SAndroid Build Coastguard Worker## Devices status
53*1b4853f5SAndroid Build Coastguard Worker
54*1b4853f5SAndroid Build Coastguard WorkerThe `devices/` directory contains a Rust crate implementing helper functions to
55*1b4853f5SAndroid Build Coastguard Workerparse the protocol, interfaces to easily implement devices, and a couple of
56*1b4853f5SAndroid Build Coastguard Workerdevice implementations. It is written to be easily pluggable on any VMM. See the
57*1b4853f5SAndroid Build Coastguard Workerrustdoc in the `devices/` directory for more information.
58*1b4853f5SAndroid Build Coastguard Worker
59*1b4853f5SAndroid Build Coastguard WorkerImplemented devices are:
60*1b4853f5SAndroid Build Coastguard Worker
61*1b4853f5SAndroid Build Coastguard Worker- A simple video capture device generating a pattern (`simple_device.rs`),
62*1b4853f5SAndroid Build Coastguard Worker  purely software-based and thus not requiring any kind of hardware. This is
63*1b4853f5SAndroid Build Coastguard Worker  here for reference and testing purposes.
64*1b4853f5SAndroid Build Coastguard Worker- A proxy device for host V4L2 devices, i.e. a device allowing to expose a host
65*1b4853f5SAndroid Build Coastguard Worker  V4L2 device to the guest almost as-is (`v4l2_device_proxy.rs`).
66*1b4853f5SAndroid Build Coastguard Worker
67*1b4853f5SAndroid Build Coastguard Worker* A FFmpeg-based video decoder device as a separate crate in
68*1b4853f5SAndroid Build Coastguard Worker  `extras/ffmpeg-decoder`.
69*1b4853f5SAndroid Build Coastguard Worker
70*1b4853f5SAndroid Build Coastguard Worker## Virtio device ID
71*1b4853f5SAndroid Build Coastguard Worker
72*1b4853f5SAndroid Build Coastguard WorkerVirtio-media uses device ID `49`.
73*1b4853f5SAndroid Build Coastguard Worker
74*1b4853f5SAndroid Build Coastguard Worker## Virtqueues
75*1b4853f5SAndroid Build Coastguard Worker
76*1b4853f5SAndroid Build Coastguard WorkerThere are two queues in use:
77*1b4853f5SAndroid Build Coastguard Worker
78*1b4853f5SAndroid Build Coastguard Worker0 : commandq - queue for driver commands and device responses to these commands.
79*1b4853f5SAndroid Build Coastguard WorkerThe device MUST return the descriptor chains it receives as soon as possible,
80*1b4853f5SAndroid Build Coastguard Workerand must never hold to them for indefinite periods of time.
81*1b4853f5SAndroid Build Coastguard Worker
82*1b4853f5SAndroid Build Coastguard Worker1 : eventq - queue for events sent by the device to the driver. The driver MUST
83*1b4853f5SAndroid Build Coastguard Workerre-queue the descriptor chains returned by the device as soon as possible, and
84*1b4853f5SAndroid Build Coastguard Workermust never hold on them for indefinite periods of time.
85*1b4853f5SAndroid Build Coastguard Worker
86*1b4853f5SAndroid Build Coastguard Worker## Configuration area
87*1b4853f5SAndroid Build Coastguard Worker
88*1b4853f5SAndroid Build Coastguard WorkerThe configuration area contains the following information:
89*1b4853f5SAndroid Build Coastguard Worker
90*1b4853f5SAndroid Build Coastguard Worker```c
91*1b4853f5SAndroid Build Coastguard Workerstruct virtio_v4l2_config {
92*1b4853f5SAndroid Build Coastguard Worker    /// The device_caps field of struct video_device.
93*1b4853f5SAndroid Build Coastguard Worker    u32 device_caps;
94*1b4853f5SAndroid Build Coastguard Worker    /// The vfl_devnode_type of the device.
95*1b4853f5SAndroid Build Coastguard Worker    u32 device_type;
96*1b4853f5SAndroid Build Coastguard Worker    /// The `card` field of v4l2_capability.
97*1b4853f5SAndroid Build Coastguard Worker    u8 card[32];
98*1b4853f5SAndroid Build Coastguard Worker}
99*1b4853f5SAndroid Build Coastguard Worker```
100*1b4853f5SAndroid Build Coastguard Worker
101*1b4853f5SAndroid Build Coastguard Worker## Shared memory regions
102*1b4853f5SAndroid Build Coastguard Worker
103*1b4853f5SAndroid Build Coastguard WorkerShared memory region `0` is used to map `MMAP` buffers into the guest using the
104*1b4853f5SAndroid Build Coastguard Worker`VIRTIO_MEDIA_CMD_MMAP` command. If the host does not provide it, then `MMAP`
105*1b4853f5SAndroid Build Coastguard Workerbuffers cannot be mapped into the guest.
106*1b4853f5SAndroid Build Coastguard Worker
107*1b4853f5SAndroid Build Coastguard Worker## Protocol
108*1b4853f5SAndroid Build Coastguard Worker
109*1b4853f5SAndroid Build Coastguard WorkerAll structures managing the virtio protocol are defined and documented in
110*1b4853f5SAndroid Build Coastguard Worker`protocol.h`. Please refer to this file whenever a `virtio_media_cmd_*` or
111*1b4853f5SAndroid Build Coastguard Worker`virtio_media_resp_*` structure is mentioned.
112*1b4853f5SAndroid Build Coastguard Worker
113*1b4853f5SAndroid Build Coastguard WorkerCommands are queued on the `commandq` by the driver for the device to process.
114*1b4853f5SAndroid Build Coastguard WorkerThey all start by an instance of `struct virtio_media_cmd_header` and include
115*1b4853f5SAndroid Build Coastguard Workerdevice-writable descriptors for the device to write the result of the command in
116*1b4853f5SAndroid Build Coastguard Workera `struct virtio_media_resp_header`.
117*1b4853f5SAndroid Build Coastguard Worker
118*1b4853f5SAndroid Build Coastguard WorkerThe errors returned by each command are standard Linux kernel error codes. For
119*1b4853f5SAndroid Build Coastguard Workerinstance, a command that contains invalid options will return `EINVAL`.
120*1b4853f5SAndroid Build Coastguard Worker
121*1b4853f5SAndroid Build Coastguard WorkerEvents are sent on the `eventq` by the device for the driver to handle. They all
122*1b4853f5SAndroid Build Coastguard Workerstart by an instance of `struct virtio_media_event_header`.
123*1b4853f5SAndroid Build Coastguard Worker
124*1b4853f5SAndroid Build Coastguard Worker## Session management
125*1b4853f5SAndroid Build Coastguard Worker
126*1b4853f5SAndroid Build Coastguard WorkerIn order to use the device, the driver needs to open a session. This act is
127*1b4853f5SAndroid Build Coastguard Workerequivalent to opening the `/dev/videoX` device file of the V4L2 device.
128*1b4853f5SAndroid Build Coastguard WorkerDepending on the type of device, it may be possible to open several sessions
129*1b4853f5SAndroid Build Coastguard Workerconcurrently.
130*1b4853f5SAndroid Build Coastguard Worker
131*1b4853f5SAndroid Build Coastguard WorkerA session is opened by queueing a `struct virtio_media_cmd_open` along with a
132*1b4853f5SAndroid Build Coastguard Workerdescriptor to receive a `struct virtio_media_resp_open` to the commandq. An open
133*1b4853f5SAndroid Build Coastguard Workersession can be closed with `struct virtio_media_cmd_close`.
134*1b4853f5SAndroid Build Coastguard Worker
135*1b4853f5SAndroid Build Coastguard WorkerWhile the session is opened, its ID can be used to perform actions on it, most
136*1b4853f5SAndroid Build Coastguard Workercommonly V4L2 ioctls.
137*1b4853f5SAndroid Build Coastguard Worker
138*1b4853f5SAndroid Build Coastguard Worker## Ioctls
139*1b4853f5SAndroid Build Coastguard Worker
140*1b4853f5SAndroid Build Coastguard WorkerIoctls are the main way to interact with V4L2 devices, and therefore
141*1b4853f5SAndroid Build Coastguard Workervirtio-media features a command to perform an ioctl on an open session.
142*1b4853f5SAndroid Build Coastguard Worker
143*1b4853f5SAndroid Build Coastguard WorkerIn order to perform an ioctl, the driver queues a
144*1b4853f5SAndroid Build Coastguard Worker`struct virtio_media_cmd_ioctl` along with a descriptor to receive a
145*1b4853f5SAndroid Build Coastguard Worker`struct virtio_media_resp_ioctl` on the commandq. The code of the ioctl can be
146*1b4853f5SAndroid Build Coastguard Workerextracted from the
147*1b4853f5SAndroid Build Coastguard Worker[videodev2.h](https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/videodev.html)
148*1b4853f5SAndroid Build Coastguard Workerheader file, which defines the ioctls' codes, type of payload, and direction.
149*1b4853f5SAndroid Build Coastguard WorkerFor instance, the `VIDIOC_G_FMT` ioctl is defined as follows:
150*1b4853f5SAndroid Build Coastguard Worker
151*1b4853f5SAndroid Build Coastguard Worker```c
152*1b4853f5SAndroid Build Coastguard Worker#define VIDIOC_G_FMT _IOWR('V',  4, struct v4l2_format)
153*1b4853f5SAndroid Build Coastguard Worker```
154*1b4853f5SAndroid Build Coastguard Worker
155*1b4853f5SAndroid Build Coastguard WorkerThis tells us that its ioctl code is `4`, that its payload is a
156*1b4853f5SAndroid Build Coastguard Worker`struct v4l2_format`, and that its direction is `WR`, i.e. the payload is
157*1b4853f5SAndroid Build Coastguard Workerwritten by both the driver and the device.
158*1b4853f5SAndroid Build Coastguard Worker
159*1b4853f5SAndroid Build Coastguard WorkerThe payload layout is always a 64-bit representation of the corresponding V4L2
160*1b4853f5SAndroid Build Coastguard Workerstructure, irrespective of the host and guest architecture.
161*1b4853f5SAndroid Build Coastguard Worker
162*1b4853f5SAndroid Build Coastguard Worker### Ioctls payload
163*1b4853f5SAndroid Build Coastguard Worker
164*1b4853f5SAndroid Build Coastguard WorkerThe payload of an ioctl in the descriptor chain follows the command structure,
165*1b4853f5SAndroid Build Coastguard Workerthe reponse structure, or both depending on the direction:
166*1b4853f5SAndroid Build Coastguard Worker
167*1b4853f5SAndroid Build Coastguard Worker- An `_IOR` ioctl is read-only for the driver, meaning the payload follows the
168*1b4853f5SAndroid Build Coastguard Worker  response in the device-writable section of the descriptor chain.
169*1b4853f5SAndroid Build Coastguard Worker- An `_IOW` ioctl is read-only for the device, meaning the payload follows the
170*1b4853f5SAndroid Build Coastguard Worker  command in the driver-writable section of the descriptor chain.
171*1b4853f5SAndroid Build Coastguard Worker- An `_IORW` ioctl is writable by both the device and driver, meaning the
172*1b4853f5SAndroid Build Coastguard Worker  payload must follow both the command in the driver-writable section of the
173*1b4853f5SAndroid Build Coastguard Worker  descriptor chain, and the response in the device-writable section.
174*1b4853f5SAndroid Build Coastguard Worker
175*1b4853f5SAndroid Build Coastguard WorkerFor instance, the `VIDIOC_G_STD` ioctl is defined as follows:
176*1b4853f5SAndroid Build Coastguard Worker
177*1b4853f5SAndroid Build Coastguard Worker```c
178*1b4853f5SAndroid Build Coastguard Worker#define VIDIOC_G_STD _IOR('V', 23, v4l2_std_id)
179*1b4853f5SAndroid Build Coastguard Worker```
180*1b4853f5SAndroid Build Coastguard Worker
181*1b4853f5SAndroid Build Coastguard WorkerThus, its layout on the commandq will be:
182*1b4853f5SAndroid Build Coastguard Worker
183*1b4853f5SAndroid Build Coastguard Worker```text
184*1b4853f5SAndroid Build Coastguard Worker+-------------------------------------+
185*1b4853f5SAndroid Build Coastguard Worker| struct virtio_media_cmd_ioctl       |
186*1b4853f5SAndroid Build Coastguard Worker+=====================================+
187*1b4853f5SAndroid Build Coastguard Worker| struct virtio_media_resp_ioctl      |
188*1b4853f5SAndroid Build Coastguard Worker+-------------------------------------+
189*1b4853f5SAndroid Build Coastguard Worker| v4l2_std_id                         |
190*1b4853f5SAndroid Build Coastguard Worker+-------------------------------------+
191*1b4853f5SAndroid Build Coastguard Worker```
192*1b4853f5SAndroid Build Coastguard Worker
193*1b4853f5SAndroid Build Coastguard Worker(in these diagrams, the `====` line signals the delimitation between
194*1b4853f5SAndroid Build Coastguard Workerdevice-readable and device-writable descriptors).
195*1b4853f5SAndroid Build Coastguard Worker
196*1b4853f5SAndroid Build Coastguard Worker`VIDIOC_SUBSCRIBE_EVENT` is defined as follows:
197*1b4853f5SAndroid Build Coastguard Worker
198*1b4853f5SAndroid Build Coastguard Worker```c
199*1b4853f5SAndroid Build Coastguard Worker#define VIDIOC_SUBSCRIBE_EVENT _IOW('V', 90, struct v4l2_event_subscription)
200*1b4853f5SAndroid Build Coastguard Worker```
201*1b4853f5SAndroid Build Coastguard Worker
202*1b4853f5SAndroid Build Coastguard WorkerMeaning its layout on the commandq will be:
203*1b4853f5SAndroid Build Coastguard Worker
204*1b4853f5SAndroid Build Coastguard Worker```text
205*1b4853f5SAndroid Build Coastguard Worker+-------------------------------------+
206*1b4853f5SAndroid Build Coastguard Worker| struct virtio_media_cmd_ioctl       |
207*1b4853f5SAndroid Build Coastguard Worker+-------------------------------------+
208*1b4853f5SAndroid Build Coastguard Worker| struct v4l2_event_subscription      |
209*1b4853f5SAndroid Build Coastguard Worker+=====================================+
210*1b4853f5SAndroid Build Coastguard Worker| struct virtio_media_resp_ioctl      |
211*1b4853f5SAndroid Build Coastguard Worker+-------------------------------------+
212*1b4853f5SAndroid Build Coastguard Worker```
213*1b4853f5SAndroid Build Coastguard Worker
214*1b4853f5SAndroid Build Coastguard WorkerFinally, `VIDIOC_G_FMT` is a `WR` ioctl:
215*1b4853f5SAndroid Build Coastguard Worker
216*1b4853f5SAndroid Build Coastguard Worker```c
217*1b4853f5SAndroid Build Coastguard Worker#define VIDIOC_G_FMT _IOWR('V',  4, struct v4l2_format)
218*1b4853f5SAndroid Build Coastguard Worker```
219*1b4853f5SAndroid Build Coastguard Worker
220*1b4853f5SAndroid Build Coastguard WorkerIts layout on the commandq will thus be:
221*1b4853f5SAndroid Build Coastguard Worker
222*1b4853f5SAndroid Build Coastguard Worker```text
223*1b4853f5SAndroid Build Coastguard Worker+-------------------------------------+
224*1b4853f5SAndroid Build Coastguard Worker| struct virtio_media_cmd_ioctl       |
225*1b4853f5SAndroid Build Coastguard Worker+-------------------------------------+
226*1b4853f5SAndroid Build Coastguard Worker| struct v4l2_format                  |
227*1b4853f5SAndroid Build Coastguard Worker+=====================================+
228*1b4853f5SAndroid Build Coastguard Worker| struct virtio_media_resp_ioctl      |
229*1b4853f5SAndroid Build Coastguard Worker+-------------------------------------+
230*1b4853f5SAndroid Build Coastguard Worker| struct v4l2_format                  |
231*1b4853f5SAndroid Build Coastguard Worker+-------------------------------------+
232*1b4853f5SAndroid Build Coastguard Worker```
233*1b4853f5SAndroid Build Coastguard Worker
234*1b4853f5SAndroid Build Coastguard WorkerA common optimization for `WR` ioctls is to provide the payload using
235*1b4853f5SAndroid Build Coastguard Workerdescriptors that both point to the same buffer. This mimics the behavior of V4L2
236*1b4853f5SAndroid Build Coastguard Workerioctls where the data is only passed once and used as both input and output by
237*1b4853f5SAndroid Build Coastguard Workerthe kernel.
238*1b4853f5SAndroid Build Coastguard Worker
239*1b4853f5SAndroid Build Coastguard WorkerIn case of success, the device MUST always write the payload in the
240*1b4853f5SAndroid Build Coastguard Workerdevice-writable part of the descriptor chain.
241*1b4853f5SAndroid Build Coastguard Worker
242*1b4853f5SAndroid Build Coastguard WorkerIn case of failure, the device is free to write the payload in the
243*1b4853f5SAndroid Build Coastguard Workerdevice-writable part of the descriptor chain or not. Some errors may still
244*1b4853f5SAndroid Build Coastguard Workerresult in the payload being updated, and in this case the device is expected to
245*1b4853f5SAndroid Build Coastguard Workerwrite the updated payload (for instance, `G_EXT_CTRLS` may return `EINVAL` but
246*1b4853f5SAndroid Build Coastguard Workerupdate the `size` member of the requested controls if the provided size was not
247*1b4853f5SAndroid Build Coastguard Workerenough). If the device has not written the payload after an error, the driver
248*1b4853f5SAndroid Build Coastguard WorkerMUST assume that the payload has not been modified.
249*1b4853f5SAndroid Build Coastguard Worker
250*1b4853f5SAndroid Build Coastguard Worker### Handling of pointers in ioctl payload
251*1b4853f5SAndroid Build Coastguard Worker
252*1b4853f5SAndroid Build Coastguard WorkerA few structures used as ioctl payloads contain pointers the link to further
253*1b4853f5SAndroid Build Coastguard Workerdata needed for the ioctl. There are notably:
254*1b4853f5SAndroid Build Coastguard Worker
255*1b4853f5SAndroid Build Coastguard Worker- The `planes` pointer of `struct v4l2_buffer`, which size is determined by the
256*1b4853f5SAndroid Build Coastguard Worker  `length` member,
257*1b4853f5SAndroid Build Coastguard Worker- The `controls` pointer of `struct v4l2_ext_controls`, which size is determined
258*1b4853f5SAndroid Build Coastguard Worker  by the `count` member.
259*1b4853f5SAndroid Build Coastguard Worker
260*1b4853f5SAndroid Build Coastguard WorkerIf the size of the pointed area is determined to be non-zero, then the main
261*1b4853f5SAndroid Build Coastguard Workerpayload is immediately followed by the pointed data in their order of appearance
262*1b4853f5SAndroid Build Coastguard Workerin the structure, and the pointer value itself is ignored by the device, which
263*1b4853f5SAndroid Build Coastguard Workermust also return the value initially passed by the driver. For instance, for a
264*1b4853f5SAndroid Build Coastguard Worker`struct v4l2_ext_controls` which `count` is `16`:
265*1b4853f5SAndroid Build Coastguard Worker
266*1b4853f5SAndroid Build Coastguard Worker```text
267*1b4853f5SAndroid Build Coastguard Worker+--------------------------------------+
268*1b4853f5SAndroid Build Coastguard Worker| struct v4l2_ext_controls             |
269*1b4853f5SAndroid Build Coastguard Worker+--------------------------------------+
270*1b4853f5SAndroid Build Coastguard Worker| struct v4l2_ext_control for plane 0  |
271*1b4853f5SAndroid Build Coastguard Worker| struct v4l2_ext_control for plane 1  |
272*1b4853f5SAndroid Build Coastguard Worker| ...                                  |
273*1b4853f5SAndroid Build Coastguard Worker| struct v4l2_ext_control for plane 15 |
274*1b4853f5SAndroid Build Coastguard Worker+--------------------------------------+
275*1b4853f5SAndroid Build Coastguard Worker```
276*1b4853f5SAndroid Build Coastguard Worker
277*1b4853f5SAndroid Build Coastguard WorkerSimilarly, a multiplanar `struct v4l2_buffer` with its `length` member set to 3
278*1b4853f5SAndroid Build Coastguard Workerwill be laid out as follows:
279*1b4853f5SAndroid Build Coastguard Worker
280*1b4853f5SAndroid Build Coastguard Worker```text
281*1b4853f5SAndroid Build Coastguard Worker+-------------------------------------+
282*1b4853f5SAndroid Build Coastguard Worker| struct v4l2_buffer                  |
283*1b4853f5SAndroid Build Coastguard Worker+-------------------------------------+
284*1b4853f5SAndroid Build Coastguard Worker| struct v4l2_plane for plane 0       |
285*1b4853f5SAndroid Build Coastguard Worker| struct v4l2_plane for plane 1       |
286*1b4853f5SAndroid Build Coastguard Worker| struct v4l2_plane for plane 2       |
287*1b4853f5SAndroid Build Coastguard Worker+-------------------------------------+
288*1b4853f5SAndroid Build Coastguard Worker```
289*1b4853f5SAndroid Build Coastguard Worker
290*1b4853f5SAndroid Build Coastguard Worker### Handling of pointers to userspace memory
291*1b4853f5SAndroid Build Coastguard Worker
292*1b4853f5SAndroid Build Coastguard WorkerA few pointers are special in that they point to userspace memory. They are:
293*1b4853f5SAndroid Build Coastguard Worker
294*1b4853f5SAndroid Build Coastguard Worker- The `m.userptr` member of `struct v4l2_buffer` and `struct v4l2_plane`
295*1b4853f5SAndroid Build Coastguard Worker  (technically an `unsigned long`, but designated a userspace address),
296*1b4853f5SAndroid Build Coastguard Worker- The `ptr` member of `struct v4l2_ext_ctrl`.
297*1b4853f5SAndroid Build Coastguard Worker
298*1b4853f5SAndroid Build Coastguard WorkerThese pointers can cover large areas of scattered memory, which has the
299*1b4853f5SAndroid Build Coastguard Workerpotential to require more descriptors than the virtio queue can provide. For
300*1b4853f5SAndroid Build Coastguard Workerthese particular pointers only, a list of `struct virtio_media_sg_entry` that
301*1b4853f5SAndroid Build Coastguard Workercovers the needed amount of memory for the pointer is used instead of using
302*1b4853f5SAndroid Build Coastguard Workerdescriptors to map the pointed memory directly.
303*1b4853f5SAndroid Build Coastguard Worker
304*1b4853f5SAndroid Build Coastguard WorkerFor each such pointer to read, the device reads as many SG entries as needed to
305*1b4853f5SAndroid Build Coastguard Workercover the length of the pointed buffer, as described by its parent structure
306*1b4853f5SAndroid Build Coastguard Worker(`length` member of `struct v4l2_buffer` or `struct v4l2_plane` for buffer
307*1b4853f5SAndroid Build Coastguard Workermemory, and `size` member of `struct v4l2_ext_control` for control data).
308*1b4853f5SAndroid Build Coastguard Worker
309*1b4853f5SAndroid Build Coastguard WorkerSince the device never needs to modify the list of SG entries, it is only
310*1b4853f5SAndroid Build Coastguard Workerprovided by the driver in the device-readable section of the descriptor chain,
311*1b4853f5SAndroid Build Coastguard Workerand not repeated in the device-writable section, even for `WR` ioctls.
312*1b4853f5SAndroid Build Coastguard Worker
313*1b4853f5SAndroid Build Coastguard WorkerTo illustrate the data layout, here is what the descriptor chain of a
314*1b4853f5SAndroid Build Coastguard Worker`VIDIOC_QBUF` ioctl queueing a 3-planar `USERPTR` buffer would look like:
315*1b4853f5SAndroid Build Coastguard Worker
316*1b4853f5SAndroid Build Coastguard Worker```text
317*1b4853f5SAndroid Build Coastguard Worker+---------------------------------------------------+
318*1b4853f5SAndroid Build Coastguard Worker| struct virtio_media_cmd_ioctl                     |
319*1b4853f5SAndroid Build Coastguard Worker+---------------------------------------------------+
320*1b4853f5SAndroid Build Coastguard Worker| struct v4l2_buffer                                |
321*1b4853f5SAndroid Build Coastguard Worker+---------------------------------------------------+
322*1b4853f5SAndroid Build Coastguard Worker| struct v4l2_plane for plane 0                     |
323*1b4853f5SAndroid Build Coastguard Worker| struct v4l2_plane for plane 1                     |
324*1b4853f5SAndroid Build Coastguard Worker| struct v4l2_plane for plane 2                     |
325*1b4853f5SAndroid Build Coastguard Worker+---------------------------------------------------+
326*1b4853f5SAndroid Build Coastguard Worker| array of struct virtio_media_sg_entry for plane 0 |
327*1b4853f5SAndroid Build Coastguard Worker+---------------------------------------------------+
328*1b4853f5SAndroid Build Coastguard Worker| array of struct virtio_media_sg_entry for plane 1 |
329*1b4853f5SAndroid Build Coastguard Worker+---------------------------------------------------+
330*1b4853f5SAndroid Build Coastguard Worker| array of struct virtio_media_sg_entry for plane 2 |
331*1b4853f5SAndroid Build Coastguard Worker+===================================================+
332*1b4853f5SAndroid Build Coastguard Worker| struct virtio_media_resp_ioctl                    |
333*1b4853f5SAndroid Build Coastguard Worker+---------------------------------------------------+
334*1b4853f5SAndroid Build Coastguard Worker| struct v4l2_buffer                                |
335*1b4853f5SAndroid Build Coastguard Worker+---------------------------------------------------+
336*1b4853f5SAndroid Build Coastguard Worker| struct v4l2_plane for plane 0                     |
337*1b4853f5SAndroid Build Coastguard Worker| struct v4l2_plane for plane 1                     |
338*1b4853f5SAndroid Build Coastguard Worker| struct v4l2_plane for plane 2                     |
339*1b4853f5SAndroid Build Coastguard Worker+---------------------------------------------------+
340*1b4853f5SAndroid Build Coastguard Worker```
341*1b4853f5SAndroid Build Coastguard Worker
342*1b4853f5SAndroid Build Coastguard WorkerSince the ioctl is `RW`, the payload is repeated in both the device-readable and
343*1b4853f5SAndroid Build Coastguard Workerdevice-writable sections of the descriptor chain, but the device-writable
344*1b4853f5SAndroid Build Coastguard Workersection does not include the SG lists to guest memory.
345*1b4853f5SAndroid Build Coastguard Worker
346*1b4853f5SAndroid Build Coastguard Worker### Unsupported ioctls
347*1b4853f5SAndroid Build Coastguard Worker
348*1b4853f5SAndroid Build Coastguard WorkerA few ioctls are replaced by other, more suitable mechanisms. If being requested
349*1b4853f5SAndroid Build Coastguard Workerthese ioctls, the device must return the same response as it would for an
350*1b4853f5SAndroid Build Coastguard Workerunknown ioctl, i.e. `ENOTTY`.
351*1b4853f5SAndroid Build Coastguard Worker
352*1b4853f5SAndroid Build Coastguard Worker- `VIDIOC_QUERYCAP` is replaced by reading the configuration area.
353*1b4853f5SAndroid Build Coastguard Worker- `VIDIOC_DQBUF` is replaced by a dedicated event.
354*1b4853f5SAndroid Build Coastguard Worker- `VIDIOC_DQEVENT` is replaced by a dedicated event.
355*1b4853f5SAndroid Build Coastguard Worker- `VIDIOC_G_JPEGCOMP` and `VIDIOC_S_JPEGCOMP` are deprecated and replaced by the
356*1b4853f5SAndroid Build Coastguard Worker  controls of the JPEG class.
357*1b4853f5SAndroid Build Coastguard Worker- `VIDIOC_LOG_STATUS` is a guest-only operation and shall not be implemented by
358*1b4853f5SAndroid Build Coastguard Worker  the host.
359*1b4853f5SAndroid Build Coastguard Worker
360*1b4853f5SAndroid Build Coastguard Worker## Events
361*1b4853f5SAndroid Build Coastguard Worker
362*1b4853f5SAndroid Build Coastguard WorkerEvents are a way for the device to inform the driver about asynchronous events
363*1b4853f5SAndroid Build Coastguard Workerthat it should know about. In virtio-media, they are used as a replacement for
364*1b4853f5SAndroid Build Coastguard Workerthe `VIDIOC_DQBUF` and `VIDIOC_DQEVENT` ioctls and the polling mechanism, which
365*1b4853f5SAndroid Build Coastguard Workerwould be impractical to implement on top of virtio.
366*1b4853f5SAndroid Build Coastguard Worker
367*1b4853f5SAndroid Build Coastguard Worker### Dequeued buffer events
368*1b4853f5SAndroid Build Coastguard Worker
369*1b4853f5SAndroid Build Coastguard WorkerA `struct virtio_media_event_dqbuf` event is queued on the eventq by the device
370*1b4853f5SAndroid Build Coastguard Workerevery time a buffer previously queued using the `VIDIOC_QBUF` ioctl is done
371*1b4853f5SAndroid Build Coastguard Workerbeing processed and can be used by the driver again. This is like an implicit
372*1b4853f5SAndroid Build Coastguard Worker`VIDIOC_DQBUF` ioctl.
373*1b4853f5SAndroid Build Coastguard Worker
374*1b4853f5SAndroid Build Coastguard WorkerPointer values in the `struct v4l2_buffer` and `struct v4l2_plane` are
375*1b4853f5SAndroid Build Coastguard Workermeaningless and must be ignored by the driver. It is recommended that the device
376*1b4853f5SAndroid Build Coastguard Workersets them to `NULL` in order to avoid leaking potential host addresses.
377*1b4853f5SAndroid Build Coastguard Worker
378*1b4853f5SAndroid Build Coastguard WorkerNote that in the case of a `USERPTR` buffer, the `struct v4l2_buffer` used as
379*1b4853f5SAndroid Build Coastguard Workerevent payload is not followed by the buffer memory: since that memory is the
380*1b4853f5SAndroid Build Coastguard Workersame that the driver submitted with the `VIDIOC_QBUF`, it would be redundant to
381*1b4853f5SAndroid Build Coastguard Workerhave it here.
382*1b4853f5SAndroid Build Coastguard Worker
383*1b4853f5SAndroid Build Coastguard Worker### Dequeued V4L2 event event
384*1b4853f5SAndroid Build Coastguard Worker
385*1b4853f5SAndroid Build Coastguard WorkerA `struct virtio_media_event_event` event is queued on the eventq by the device
386*1b4853f5SAndroid Build Coastguard Workerevery time an event the driver previously subscribed to using the
387*1b4853f5SAndroid Build Coastguard Worker`VIDIOC_SUBSCRIBE_EVENT` ioctl has been signaled. This is like an implicit
388*1b4853f5SAndroid Build Coastguard Worker`VIDIOC_DQEVENT` ioctl.
389*1b4853f5SAndroid Build Coastguard Worker
390*1b4853f5SAndroid Build Coastguard Worker## Memory types
391*1b4853f5SAndroid Build Coastguard Worker
392*1b4853f5SAndroid Build Coastguard WorkerThe semantics of the three V4L2 memory types (`MMAP`, `USERPTR` and `DMABUF`)
393*1b4853f5SAndroid Build Coastguard Workercan easily be mapped to a guest/host context.
394*1b4853f5SAndroid Build Coastguard Worker
395*1b4853f5SAndroid Build Coastguard Worker### MMAP
396*1b4853f5SAndroid Build Coastguard Worker
397*1b4853f5SAndroid Build Coastguard WorkerIn virtio-media, `MMAP` buffers are provisioned by the host, just like they are
398*1b4853f5SAndroid Build Coastguard Workerby the kernel in regular V4L2. Similarly to how userspace can map a `MMAP`
399*1b4853f5SAndroid Build Coastguard Workerbuffer into its address space using `mmap` and `munmap`, the virtio-media driver
400*1b4853f5SAndroid Build Coastguard Workercan map host buffers into the guest space by queueing the
401*1b4853f5SAndroid Build Coastguard Worker`struct virtio_media_cmd_mmap` and `struct virtio_media_cmd_munmap` commands to
402*1b4853f5SAndroid Build Coastguard Workerthe commandq.
403*1b4853f5SAndroid Build Coastguard Worker
404*1b4853f5SAndroid Build Coastguard Worker### USERPTR
405*1b4853f5SAndroid Build Coastguard Worker
406*1b4853f5SAndroid Build Coastguard WorkerIn virtio-media, `USERPTR` buffers and provisioned by the guest, just like they
407*1b4853f5SAndroid Build Coastguard Workerare by userspace in regular V4L2. Instances of `struct v4l2_buffer` and
408*1b4853f5SAndroid Build Coastguard Worker`struct v4l2_plane` of this type are followed by a series of descriptors mapping
409*1b4853f5SAndroid Build Coastguard Workerthe buffer backing memory in guest space.
410*1b4853f5SAndroid Build Coastguard Worker
411*1b4853f5SAndroid Build Coastguard WorkerFor the host convenience, the backing memory must start with a new descriptor -
412*1b4853f5SAndroid Build Coastguard Workerthis allows the host to easily map the buffer memory to render into it instead
413*1b4853f5SAndroid Build Coastguard Workerof having to do a copy.
414*1b4853f5SAndroid Build Coastguard Worker
415*1b4853f5SAndroid Build Coastguard WorkerThe host must not alter the pointer values provided by the guest, i.e. the
416*1b4853f5SAndroid Build Coastguard Worker`m.userptr` member of `struct v4l2_buffer` and `struct v4l2_plane` must be
417*1b4853f5SAndroid Build Coastguard Workerreturned to the guest with the same value as it was provided.
418*1b4853f5SAndroid Build Coastguard Worker
419*1b4853f5SAndroid Build Coastguard Worker### DMABUF
420*1b4853f5SAndroid Build Coastguard Worker
421*1b4853f5SAndroid Build Coastguard WorkerIn virtio-media, `DMABUF` buffers are provisioned by a virtio object, just like
422*1b4853f5SAndroid Build Coastguard Workerthey are by a DMABUF in regular V4L2. Virtio objects are 16-bytes UUIDs and do
423*1b4853f5SAndroid Build Coastguard Workernot fit in the placeholders for file descriptors, so they follow their embedding
424*1b4853f5SAndroid Build Coastguard Workerdata structure as needed and the device must leave the V4L2 structure
425*1b4853f5SAndroid Build Coastguard Workerplaceholder unchanged. For instance, a 3-planar `struct v4l2_buffer` with the
426*1b4853f5SAndroid Build Coastguard Worker`V4L2_MEMORY_DMABUF` memory type will have the following layout:
427*1b4853f5SAndroid Build Coastguard Worker
428*1b4853f5SAndroid Build Coastguard Worker```text
429*1b4853f5SAndroid Build Coastguard Worker+-------------------------------------+
430*1b4853f5SAndroid Build Coastguard Worker| struct v4l2_buffer                  |
431*1b4853f5SAndroid Build Coastguard Worker+-------------------------------------+
432*1b4853f5SAndroid Build Coastguard Worker| struct v4l2_plane for plane 0       |
433*1b4853f5SAndroid Build Coastguard Worker| struct v4l2_plane for plane 1       |
434*1b4853f5SAndroid Build Coastguard Worker| struct v4l2_plane for plane 2       |
435*1b4853f5SAndroid Build Coastguard Worker+-------------------------------------+
436*1b4853f5SAndroid Build Coastguard Worker| 16 byte UUID for plane 0            |
437*1b4853f5SAndroid Build Coastguard Worker+-------------------------------------+
438*1b4853f5SAndroid Build Coastguard Worker| 16 byte UUID for plane 1            |
439*1b4853f5SAndroid Build Coastguard Worker+-------------------------------------+
440*1b4853f5SAndroid Build Coastguard Worker| 16 byte UUID for plane 2            |
441*1b4853f5SAndroid Build Coastguard Worker+-------------------------------------+
442*1b4853f5SAndroid Build Coastguard Worker```
443*1b4853f5SAndroid Build Coastguard Worker
444*1b4853f5SAndroid Build Coastguard WorkerContrary to `USERPTR` buffers, virtio objects UUIDs need to be added in both the
445*1b4853f5SAndroid Build Coastguard Workerdevice-readable and device-writable section of the descriptor chain.
446*1b4853f5SAndroid Build Coastguard Worker
447*1b4853f5SAndroid Build Coastguard WorkerHost-allocated buffers with the `V4L2_MEMORY_MMAP` memory type can also be
448*1b4853f5SAndroid Build Coastguard Workerexported as virtio objects for use with another virtio device using the
449*1b4853f5SAndroid Build Coastguard Worker`VIDIOC_EXPBUF` ioctl. The `fd` placefolder of `v4l2_exportbuffer` means that
450*1b4853f5SAndroid Build Coastguard Workerspace for the UUID needs to be reserved right after that structure, so the ioctl
451*1b4853f5SAndroid Build Coastguard Workerlayout will looks as follows:
452*1b4853f5SAndroid Build Coastguard Worker
453*1b4853f5SAndroid Build Coastguard Worker```text
454*1b4853f5SAndroid Build Coastguard Worker+-------------------------------------+
455*1b4853f5SAndroid Build Coastguard Worker| struct virtio_media_cmd_ioctl       |
456*1b4853f5SAndroid Build Coastguard Worker+-------------------------------------+
457*1b4853f5SAndroid Build Coastguard Worker| struct v4l2_exportbuffer            |
458*1b4853f5SAndroid Build Coastguard Worker+-------------------------------------+
459*1b4853f5SAndroid Build Coastguard Worker| 16 bytes UUID for exported buffer   |
460*1b4853f5SAndroid Build Coastguard Worker+=====================================+
461*1b4853f5SAndroid Build Coastguard Worker| struct virtio_media_resp_ioctl      |
462*1b4853f5SAndroid Build Coastguard Worker+-------------------------------------+
463*1b4853f5SAndroid Build Coastguard Worker| struct v4l2_exportbuffer            |
464*1b4853f5SAndroid Build Coastguard Worker+-------------------------------------+
465*1b4853f5SAndroid Build Coastguard Worker| 16 bytes UUID for exported buffer   |
466*1b4853f5SAndroid Build Coastguard Worker+-------------------------------------+
467*1b4853f5SAndroid Build Coastguard Worker```
468