README.rst
1// SPDX-License-Identifier: GPL-2.0
2
3========
4ublk-nbd
5========
6
7Motivation
8==========
9
10As one attempt of using io_uring to implement network storage based on ublk
11framework, make one basic nbd client with ublk/io_uring, which could be for
12replacing linux kernel nbd driver.
13
14Howto
15=====
16
17ublk add -t nbd [-q $NR_QUEUES ] [ -d $QUEUE_DEPTH ] [--host $HOST_IP_OR_NAME | --unix $UNIX_SOCKET_PATH] [--send_zc]
18
19Like ``nbd-client`` [#nbd_client]_, ublk-nbd supports both tcp and unix socket.
20
21``--host $HOST_IP_OR_NAME`` points to nbd server's IP address or domain name if
22tcp socket is used.
23
24``--unix $UNIX_SOCKET_PATH`` points to unix socket path if unix socket is used.
25
26The option of ``--send_zc`` enables ``io_uring send zero copy``
27[#io_uring_send_zc]_, which is only used for handling ublk write IO.
28
29Design
30======
31
32Handshake
33---------
34
35Borrow code from ``nbd`` [#nbd]_ project.
36
37Transmission
38------------
39
40Traditionally the transmission phase is implemented as kernel driver of
41``nbd driver`` [#nbd_driver]_. Now we have ublk framework, so it is
42possible to move it out of linux kernel.
43
44NBD protocol [#nbd_protocol]_ is simple, for each block IO request,
45nbd client sends 24byte request header, nbd server responds with one
4616 byte nbd reply. For READ request, the returned IO data follows the
47reply, and now ublk-nbd implements nbd simple reply only, and doesn't
48support structured reply which isn't implemented by ``nbd driver``
49[#nbd_driver]_ too. For WRITE request, IO data needs to follow the
5024byte request header.
51
52For every IO request delivered from ublk driver, ublk-nbd target code
53handles this IO in one dedicated coroutine bound to IO tag, the
54IO handling includes:
55
56- sending nbd request
57- sending WRITE data
58- reading nbd reply
59- reading nbd READ data
60
61One extra dedicated coroutine is responsible for reading reply and
62data in case of READ request via io_uring & recv(nonblocking) hybrid
63approach. recv(nonblocking) is always tried first:
64
65- if the whole reply or data in case of READ is done by recv, wakeup
66 IO handling coroutine for completing this IO
67
68- if partial reply or data is read, keep to read via recv(nonblocking)
69 until the whole reply or data is read or the max tries are reached.
70
71- otherwise, io_uring is used for handling the remained reply/data
72
73If io_uring is used finally for reading reply or data, when the CQE is
74received, wakeup IO handling coroutine for completing the IO.
75
76Each IO's handling coroutine is responsible for sending nbd request and
77WRITE data in case of WRITE request via io_uring SQE, then wait for
78reply or data in case of READ request, which is notified from the recv
79coroutine.
80
81Even though everything is actually done asynchronously in single pthread
82for each nbd queue, programming with coroutine still looks like every
83step done step by step, so it becomes easier to write efficient async
84IO code with coroutine. Like other ublk targets, c++20 coroutine is used,
85which is stackless and efficient.
86
87Given stream socket is used by nbd, sending request header and data to
88socket has to be serialized, and io_uring's SQE chain is taken with
89help of IOSQE_IO_LINK. There are two chains, one is current chain, another
90chain is next chain. Before each socket send IO in current chain is sent
91to socket, new IO request is staggered into next chain. After the whole
92current chain is done, the next chain is started to be submitted. And
93the chain stuff is handled in ublk target callback of ->handle_io_background().
94
95Test
96====
97
98make test T=nbd
99
100
101TODO
102====
103
104TLS support
105-----------
106
107Timeout handling
108----------------
109
110More NBD features
111-----------------
112
113- structured replies
114
115References
116==========
117
118.. [#nbd] https://github.com/NetworkBlockDevice/nbd
119.. [#nbd_client] https://github.com/NetworkBlockDevice/nbd/blob/master/nbd-client.c
120.. [#nbd_driver] https://github.com/torvalds/linux/blob/master/drivers/block/nbd.c
121.. [#nbd_protocol] https://github.com/NetworkBlockDevice/nbd/blob/master/doc/proto.md
122.. [#io_uring_send_zc] https://lwn.net/Articles/879724/
123