1# vhost-user protocol extensions: sleep/wake/snapshot/restore 2 3WORK IN PROGRESS 4 5Documentation for the vhost-user protocol extensions added to crosvm as part of the snapshot-restore 6project. Written in the style of https://qemu-project.gitlab.io/qemu/interop/vhost-user.html so that 7we can send it upstream as a proposal. 8 9These extensions might be redundant with the VHOST_USER_PROTOCOL_F_DEVICE_STATE features recently 10added to the spec. 11 12## Suspended device state 13 14(proposed additions are **bold**) 15 16While all vrings are stopped, the device is suspended. In addition to not processing any vring 17(because they are stopped), the device must: 18 19- not write to any guest memory regions, 20- not send any notifications to the guest, 21- not send any messages to the front-end, 22- **NEW: not interact with host resources. For example, a block device should not read or modify the 23 disk image file** 24- still process and reply to messages from the front-end. 25 26**NEW: The frontend can assume those requirements are obeyed both (1) before the first queue is 27started and (2) as soon as it receives a response for the message that stopped the last queue.** 28 29## Snapshot-Restore 30 31TODO: write an overview for the feature 32 33### Frontend 34 35Snapshot sequence: 36 371. Frontend connects to vhost-user devices. 381. ... proceed as usual ... 391. For each vhost-user device 40 - Frontend stops all the queues using VHOST_USER_GET_VRING_BASE and saves the vring bases 41 somewhere. 42 - Backend enters the "suspended device state" when the last queue is stopped. 431. For each vhost-user device 44 - Frontend sends VHOST_USER_SET_DEVICE_STATE_FD and VHOST_USER_CHECK_DEVICE_STATE requests with 45 transfer direction "save" to save the device state somewhere. 461. For each vhost-user device 47 - Frontend sends VHOST_USER_SET_MEM_TABLE request. 48 - Frontend starts all the queues as if from scratch, using the saved vring base in the 49 VHOST_USER_SET_VRING_BASE request. 50 - Backend exits the "suspended device state" (as early as) when the first queue is started. 511. ... proceed as usual ... 52 53Restore sequence: 54 551. Frontend connects to vhost-user devices. 561. For each vhost-user device 57 - Frontend stops all the queues using VHOST_USER_GET_VRING_BASE and saves the vring bases 58 somewhere. 59 - Backend enters the "suspended device state" when the last queue is stopped. 601. For each vhost-user device 61 - Frontend sends VHOST_USER_SET_DEVICE_STATE_FD and VHOST_USER_CHECK_DEVICE_STATE requests with 62 transfer direction "load" restore the device state. 631. For each vhost-user device 64 - Frontend sends VHOST_USER_SET_MEM_TABLE request. 65 - Frontend starts all the queues as if from scratch, using the saved vring base in the 66 VHOST_USER_SET_VRING_BASE request. 67 - Backend exits the "suspended device state" (as early as) when the first queue is started. 681. ... proceed as usual ... 69 70### Backend 71 72TODO: anything interesting to write here? 73