Lines Matching +full:in +full:- +full:application
1 .. SPDX-License-Identifier: GPL-2.0
20 XDP programs to redirect frames to a memory buffer in a user-space
21 application.
29 descriptor ring points to a data buffer in a memory area called a
36 The UMEM consists of a number of equally sized chunks. A descriptor in
43 FILL ring is used by the application to send down addr for the kernel
44 to fill in with RX packet data. References to these frames will then
45 appear in the RX ring once each packet has been received. The
48 space, for either TX or RX. Thus, the frame addrs appearing in the
50 TX ring. In summary, the RX and FILL rings are used for the RX path
59 corresponding two rings, sets the XDP_SHARED_UMEM flag in the bind
62 then receive frame addr references in its own RX ring that point to
64 single-consumer / single-producer (for performance reasons), the new
71 is a BPF map called XSKMAP (or BPF_MAP_TYPE_XSKMAP in full). The
72 user-space application can place an XSK at an arbitrary place in this
73 map. The XDP program can then redirect a packet to a specific index in
74 this map and at this point XDP validates that the XSK in that map was
78 program loaded (and one XSK in the XSKMAP) to be able to get any
81 AF_XDP can operate in two different modes: XDP_SKB and XDP_DRV. If the
93 In order to use an AF_XDP socket, a number of associated objects need
94 to be setup. These objects and their options are explained in the
99 http://vger.kernel.org/lpc_net2018_talks/lpc18_paper_af_xdp_perf-v2.pdf. Do
106 ----
109 equal-sized frames. An UMEM is associated to a netdev and a specific
117 the next socket B can do this by setting the XDP_SHARED_UMEM flag in
121 The UMEM has two single-producer/single-consumer rings that are used
123 user-space application.
126 -----
129 TX. All rings are single-producer/single-consumer, so the user-space
130 application need explicit synchronization of multiple
145 calls and mmapped to user-space using the appropriate offset to mmap()
155 user-space to kernel-space. The UMEM addrs are passed in the ring. As
161 The user application produces UMEM addrs to this ring. Note that, if
162 running the application with aligned chunk mode, the kernel will mask
165 to the same chunk. If the user application is run in the unaligned
173 kernel-space to user-space. Just like the FILL ring, UMEM indices are
176 Frames passed from the kernel to user-space are frames that has been
177 sent (TX ring) and can be used by user-space again.
179 The user application consumes UMEM addrs from this ring.
185 The RX ring is the receiving side of a socket. Each entry in the ring
192 The user application consumes struct xdp_desc descriptors from this
202 be relaxed in the future.
204 The user application produces struct xdp_desc descriptors to this
212 in tools/lib/bpf/xsk.h for facilitating the use of AF_XDP. It
214 setup of AF_XDP socket easier and ones that can be used in the data
216 to use this API, please take a look at the sample application in
227 is used in conjunction with bpf_redirect_map() to pass the ingress
230 The user application inserts the socket into the map, via the bpf()
238 application (samples/bpf/) in for an example.
247 ------------------------------------
249 When you bind to a socket, the kernel will first try to use zero-copy
250 copy. If zero-copy is not supported, it will fall back on using copy
256 socket into zero-copy mode or fail.
259 -------------------------
263 netdevs/devices. In this mode, each socket has their own RX and TX
273 it in the normal way. Create a second socket and create an RX and a TX
275 ones from the first socket will be used. In the bind call, set he
276 XDP_SHARED_UMEM option and provide the initial socket's fd in the
281 program. Put all the sockets in the XSK_MAP and just indicate which
282 index in the array you would like to send each packet to. A simple
283 round-robin example of distributing packets is shown below:
285 .. code-block:: c
303 rr = (rr + 1) & (MAX_SOCKS - 1);
311 concurrently. There are no synchronization primitives in the
312 libbpf code that protects multiple users at this point in time.
318 built in one in libbpf that will route the traffic for you.
321 bound to different queue ids and/or netdevs. In this case you have to
325 and bind it in the normal way. Create a second socket and create an RX
327 COMPLETION ring for this socket. Then in the bind call, set he
328 XDP_SHARED_UMEM option and provide the initial socket's fd in the
332 There is no need to supply an XDP program like the one in the previous
335 the packets to the right queue. In the previous example, there is only
339 In libbpf, you need to use the xsk_socket__create_shared() API as it
351 -----------------------------
354 present in the FILL ring and the TX ring, the rings for which user
355 space is a producer. When this option is set in the bind call, the
360 If the flag is set on the FILL ring, the application needs to call
364 the NIC. In this case, interrupts are turned off as the NIC cannot
365 receive any packets (as there are no buffers to put them in), and the
370 If the flag is set for the TX ring, it means that the application
372 TX ring. This can be accomplished either by a poll() call, as in the
375 An example of how to use this flag can be found in
379 .. code-block:: c
387 better performance especially if you run the application and the
389 application and the kernel driver, as it reduces the number of
393 ------------------------------------------------------
399 application, but if you only want to do one of them, you can save
403 first one does not have a UMEM and should in that case not have any
405 be used. Note, that the rings are single-producer single-consumer, so
409 In libbpf, you can create Rx-only and Tx-only sockets by supplying
413 If you create a Tx-only socket, we recommend that you do not put any
415 going to receive something when you in fact will not, and this can
419 -----------------------
422 contain all the buffers that packet can reside in. The call takes a
427 will be able to hold a maximum of 128K / 2K = 64 packets in your UMEM
430 There is also an option to set the headroom of each single buffer in
433 application to use. The final option is the flags field, but it will
434 be dealt with in separate sections for each UMEM flag.
437 --------------------------
441 is created by a privileged process and passed to a non-privileged one.
446 -------------------------
451 .. code-block:: c
460 ----------------------
463 XDP_OPTIONS_ZEROCOPY which tells you if zero-copy is on or not.
465 Multi-Buffer Support
468 With multi-buffer support, programs using AF_XDP sockets can receive
469 and transmit packets consisting of multiple buffers both in copy and
470 zero-copy mode. For example, a packet can consist of two
479 * A descriptor in one of the AF_XDP rings always refers to a single
480 frame. In the case the packet consists of a single frame, the
483 To enable multi-buffer support for an AF_XDP socket, use the new bind
484 flag XDP_USE_SG. If this is not provided, all multi-buffer packets
486 needs to be in multi-buffer mode. This can be accomplished by using
490 XDP_PKT_CONTD is introduced in the options field of the Rx and Tx
493 of the packet. Why the reverse logic of end-of-packet (eop) flag found
494 in many NICs? Just to preserve compatibility with non-multi-buffer
511 * For copy mode, the maximum supported number of frames in a packet is
514 invalid. To produce an application that will work on any system
518 * For zero-copy mode, the limit is up to what the NIC HW
521 CONFIG_MAX_SKB_FRAGS + 1) for zero-copy mode, as it would have
522 resulted in copy actions under the hood to fit into what limit the
523 NIC supports. Kind of defeats the purpose of zero-copy mode. How to
524 probe for this limit is explained in the "probe for multi-buffer
527 On the Rx path in copy-mode, the xsk core copies the XDP data into
529 detailed before. Zero-copy mode works the same, though the data is not
530 copied. When the application gets a descriptor with the XDP_PKT_CONTD
532 and it continues with the next buffer in the following
535 that only a complete packet (all frames in the packet) is sent to the
536 application. If there is not enough space in the AF_XDP Rx ring, all
539 If application reads a batch of descriptors, using for example the libxdp
541 packet. It might end in the middle of a packet and the rest of the
546 An example program each for Rx and Tx multi-buffer support can be found
547 later in this document.
550 -----
552 In order to use AF_XDP sockets two parts are needed. The
553 user-space application and the XDP program. For a complete setup and
554 usage example, please refer to the sample application. The user-space
557 The XDP code sample included in tools/lib/bpf/xsk.c is the following:
559 .. code-block:: c
563 int index = ctx->rx_queue_index;
576 .. code-block:: c
598 __u32 entries = *ring->producer - *ring->consumer;
601 return -1;
603 // read-barrier!
605 *item = ring->desc[*ring->consumer & (RING_SIZE - 1)];
606 (*ring->consumer)++;
612 u32 free_entries = RING_SIZE - (*ring->producer - *ring->consumer);
615 return -1;
617 ring->desc[*ring->producer & (RING_SIZE - 1)] = *item;
619 // write-barrier!
621 (*ring->producer)++;
628 Usage Multi-Buffer Rx
629 ---------------------
631 Here is a simple Rx path pseudo-code example (using libxdp interfaces
634 .. code-block:: c
642 int rcvd = xsk_ring_cons__peek(&xsk->rx, opt_batch_size, &idx_rx);
644 xsk_ring_prod__reserve(&xsk->umem->fq, rcvd, &idx_fq);
647 struct xdp_desc *desc = xsk_ring_cons__rx_desc(&xsk->rx, idx_rx++);
648 char *frag = xsk_umem__get_data(xsk->umem->buffer, desc->addr);
649 bool eop = !(desc->options & XDP_PKT_CONTD);
661 *xsk_ring_prod__fill_addr(&xsk->umem->fq, idx_fq++) = desc->addr;
664 xsk_ring_prod__submit(&xsk->umem->fq, rcvd);
665 xsk_ring_cons__release(&xsk->rx, rcvd);
668 Usage Multi-Buffer Tx
669 ---------------------
671 Here is an example Tx path pseudo-code (using libxdp interfaces for
672 simplicity) ignoring that the umem is finite in size, and that we
674 points to a valid location in the umem.
676 .. code-block:: c
683 xsk_ring_prod__reserve(&xsk->tx, batch_size, &idx);
692 tx_desc = xsk_ring_prod__tx_desc(&xsk->tx, idx + i++);
693 tx_desc->addr = addr;
696 tx_desc->len = xsk_frame_size;
697 tx_desc->options = XDP_PKT_CONTD;
699 tx_desc->len = len;
700 tx_desc->options = 0;
703 len -= tx_desc->len;
715 xsk_ring_prod__submit(&xsk->tx, i);
718 Probing for Multi-Buffer Support
719 --------------------------------
721 To discover if a driver supports multi-buffer AF_XDP in SKB or DRV
722 mode, use the XDP_FEATURES feature of netlink in linux/netdev.h to
724 querying for XDP multi-buffer support. If XDP supports multi-buffer in
725 a driver, then AF_XDP will also support that in SKB and DRV mode.
727 To discover if a driver supports multi-buffer AF_XDP in zero-copy
729 flag. If it is set, it means that at least zero-copy is supported and
731 NETDEV_A_DEV_XDP_ZC_MAX_SEGS in linux/netdev.h. An unsigned integer
733 supported by this device in zero-copy mode. These are the possible
736 1: Multi-buffer for zero-copy is not supported by this device, as max
737 one fragment supported means that multi-buffer is not possible.
739 >=2: Multi-buffer is supported in zero-copy mode for this device. The
745 Multi-Buffer Support for Zero-Copy Drivers
746 ------------------------------------------
748 Zero-copy drivers usually use the batched APIs for Rx and Tx
751 to facilitate extending a zero-copy driver with multi-buffer support.
753 Sample application
756 There is a xdpsock benchmarking/test application included that
758 you would like your UDP traffic from port 4242 to end up in queue 16,
761 ethtool -N p3p2 rx-flow-hash udp4 fn
762 ethtool -N p3p2 flow-type udp4 src-port 4242 dst-port 4242 \
765 Running the rxdrop benchmark in XDP_DRV mode can then be done
768 samples/bpf/xdpsock -i p3p2 -q 16 -r -N
770 For XDP_SKB mode, use the switch "-S" instead of "-N" and all options
771 can be displayed with "-h", as usual.
773 This sample application uses libbpf to make the setup and usage of
776 code in tools/lib/bpf/xsk.[ch].
785 queue ids 0 to 7 will be allocated, one per core. In the AF_XDP
788 towards that queue you are going to get on you socket. So in the
799 sudo ethtool -L <interface> combined 1
803 that you can bind your XDP socket to. Here is one example in which
806 sudo ethtool -N <interface> rx-flow-hash udp4 fn
807 sudo ethtool -N <interface> flow-type udp4 src-port 4242 dst-port \
814 in copy mode?
817 XSKMAP can only be used to switch traffic coming in on queue id X
820 traffic goming in from queue id Y can be directed to sockets bound
821 to the same queue id Y. In zero-copy mode, you should use the
822 switch, or other distribution mechanism, in your NIC to direct
827 A: Care has to be taken not to feed the same buffer in the UMEM into
839 - Björn Töpel (AF_XDP core)
840 - Magnus Karlsson (AF_XDP core)
841 - Alexander Duyck
842 - Alexei Starovoitov
843 - Daniel Borkmann
844 - Jesper Dangaard Brouer
845 - John Fastabend
846 - Jonathan Corbet (LWN coverage)
847 - Michael S. Tsirkin
848 - Qi Z Zhang
849 - Willem de Bruijn