VPP P3C - TAP, AF_XDP, vhost-user, AF_PACKET

VPP MASTERY · PHASE 3C · WEEKS 11–13

🐧 TAP · AF_XDP · vhost-user · AF_PACKET

Linux kernel integration · eBPF bypass · VM connectivity · Full interface comparison matrix

src/vnet/devices/tap/ src/plugins/af_xdp/ src/vnet/devices/virtio/ linux-cp plugin

ALL VPP INTERFACE TYPES - SELECTION MATRIX

Property	DPDK	memif	TAP v2	AF_XDP	vhost-user	AF_PACKET
Max throughput	Line rate	8–12 Mpps	500K–2Mpps	2–8 Mpps	5–10 Mpps	<500Kpps
Kernel bypass	Full	Full	No	Partial (eBPF)	Full	No
Linux stack access	No	No	Yes	Yes	VM side only	Yes
Zero-copy	Yes	Optional	No	Yes (UMEM)	Partial	No
VM/QEMU support	No	No	No	No	Yes	No
Setup complexity	Medium	Low	Low	Medium	Medium	Trivial
Primary use	Physical NIC	Container-to-container	Management plane, Linux integration	High-perf + kernel visibility	QEMU/KVM VM	Dev/test
VPP source	`plugins/dpdk/`	`plugins/memif/`	`vnet/devices/tap/`	`plugins/af_xdp/`	`vnet/devices/virtio/`	`vnet/devices/af_packet/`

💡 Decision rule for your Docker + Mellanox environment: Physical traffic → DPDK. Container-to-container fast path → memif. Management/control plane access to Linux → TAP v2 or linux-cp. Testing without hugepages → AF_PACKET. VMs → vhost-user. You want NIC speed + Linux visibility → AF_XDP.

TAP v2 AND linux-cp PLUGIN

🐧

TAP v2 - VPP ↔ Linux Kernel Bridge

INTERNALS

TAP v2 creates a Linux virtual interface visible to the kernel alongside a VPP interface. It uses virtio vrings shared between VPP and the kernel's TUN/TAP driver - the same in-kernel virtio used by VMs, re-used for host networking. This gives the Linux kernel full visibility into traffic passing through VPP.

Primary use cases:

Management-plane traffic - SSH into a node via VPP-terminated interface
Running OSPFd/BGPd (e.g., FRRouting) on Linux while VPP handles the data plane
Control-plane protocols that need the kernel socket API
Sending packets from a VPP node to a regular Linux process

# Create TAP - VPP side gets tapN, Linux side gets vpp0 (or custom name)
create tap id 0 host-if-name vpp0 host-ip4-addr 10.10.0.2/30

# TAP with custom MAC, MTU, and namespace
create tap id 1 \
  host-if-name vpp-ctrl \
  host-ip4-addr 192.168.1.10/24 \
  host-mac-addr aa:bb:cc:dd:ee:ff \
  host-mtu-size 1500 \
  host-ns myns

# Bring up VPP side and assign L3
set interface state tap0 up
set interface ip address tap0 10.10.0.1/30

# Verify both sides
show interface          # VPP side: tap0 should be up
show tap tap0           # virtio queue details

# Linux side (inside the container):
# ip link show vpp0    - interface should be visible
# ping 10.10.0.1       - reaches VPP tap0 interface

🔗

linux-cp Plugin - Mirroring DPDK Interfaces to Linux

LINUX-CP

The linux-cp (Linux Control Plane) plugin (src/plugins/linux-cp/) solves a harder problem: you want DPDK to handle the fast path, but you also want Linux to see the same interfaces for control-plane routing protocols (FRR, Bird). linux-cp mirrors each VPP DPDK interface to a Linux netdev - punting control-plane traffic (ARP, OSPF hellos, BGP) to Linux while VPP handles the forwarding plane.

# Enable linux-cp for a DPDK interface
lcp create GigabitEthernet0/8/0 host-if lcp-eth0

# Linux now sees lcp-eth0 as a real interface
# Linux side: ip addr add 10.0.0.1/24 dev lcp-eth0
# FRR OSPF runs on lcp-eth0
# VPP data plane handles all forwarded traffic at line rate

show lcp              # list all linux-cp mirrors
lcp default netns myns  # create mirrors in a specific network namespace

💡 linux-cp vs TAP v2: TAP creates a NEW interface that only exists in VPP + Linux. linux-cp creates a Linux mirror of an existing DPDK interface. For a router deployment where you want FRR running alongside VPP on the same physical ports, linux-cp is the right tool. For a container needing management access, TAP is simpler.

AF_XDP - HIGH-PERFORMANCE WITH KERNEL VISIBILITY

⚡

AF_XDP Architecture - XSK + UMEM

ARCHITECTURE

AF_XDP (eXpress Data Path socket) lets a userspace process receive and transmit packets from a NIC queue without full DPDK kernel bypass. An eBPF XDP program in the kernel redirects selected packets from the NIC into a UMEM (userspace memory region), from which the AF_XDP socket reads them. The rest of the NIC's traffic continues through the normal kernel path.

This gives you: kernel-controlled NIC (no VFIO binding, kernel still owns the interface), with near-DPDK performance for the traffic you redirect to userspace.

/* AF_XDP components */

NIC → XDP eBPF hook → XDP_REDIRECT → UMEM (shared memory)
                                         ↑
                                    AF_XDP socket (VPP)
                                    reads RX ring
                                    writes TX ring

/* UMEM: a single large memory region, subdivided into frames */
UMEM frame size = 4096 (one per packet)
Fill ring:    VPP refills with free frame addresses
Completion ring: kernel notifies which TX frames are done
RX ring:      kernel deposits received frame addresses
TX ring:      VPP places frames to transmit

VPP AF_XDP plugin setup:

# Create AF_XDP interface on eth0 (NIC still owned by kernel mlx5_core)
create interface af-xdp host-if eth0 name afxdp0

# Or in startup.conf for persistent config

# startup.conf stanza
# (AF_XDP is configured via CLI/API, not startup.conf)

# Bring up and configure
set interface state afxdp0 up
set interface ip address afxdp0 10.0.0.1/24

# Verify
show interface afxdp0
show af-xdp interface

AF_XDP Mode	Description	Performance	Requirement
`native` (XDP_DRV)	XDP runs in NIC driver, before SKB allocation	Best - near DPDK	Driver must support native XDP (mlx5 does)
`generic` (XDP_SKB)	XDP runs after SKB allocation in generic kernel code	~2× slower than native	Any driver - universal fallback
`zero-copy`	NIC DMA directly into UMEM - no copy between kernel and userspace	Highest	Driver must support zero-copy XDP (mlx5 on kernel 5.3+)

💡 AF_XDP on Mellanox ConnectX-5: mlx5 supports native XDP and zero-copy XDP on Linux 5.3+. Your AMD + Docker environment should support this - check kernel version with uname -r. With zero-copy mode, AF_XDP throughput approaches DPDK for single-queue workloads while the NIC remains visible to ip link and ethtool.

VHOST-USER - VIRTUAL MACHINE CONNECTIVITY

🖥️

vhost-user Architecture - VPP ↔ QEMU VM

ARCHITECTURE

vhost-user is the standard mechanism for connecting a QEMU/KVM virtual machine to VPP at high performance. The VM sees a virtio-net device (standard paravirtualized NIC). The vhost-user protocol moves the virtio vring management from the kernel (vhost-net) into VPP userspace, enabling zero-copy forwarding between VPP and the VM.

# ── VPP side: create vhost-user server ──
create vhost-user socket /run/vpp/vm0.sock server

# The socket is created by VPP (server mode)
# QEMU connects to it as client

set interface state VirtualEthernet0/0/0 up
set interface ip address VirtualEthernet0/0/0 192.168.100.1/24

# ── QEMU side: connect VM to VPP ──
qemu-system-x86_64 \
  -m 2G -smp 2 \
  -chardev socket,id=char0,path=/run/vpp/vm0.sock \
  -netdev vhost-user,id=net0,chardev=char0,vhostforce \
  -device virtio-net-pci,netdev=net0,mac=52:54:00:01:02:03 \
  ...

# Inside the VM: the interface appears as eth0 or ens3
# Configure with: ip addr add 192.168.100.2/24 dev eth0
# Ping VPP: ping 192.168.100.1

# For multi-queue (improves VM throughput significantly)
create vhost-user socket /run/vpp/vm0.sock server \
  rx-queue-size 1024 tx-queue-size 1024

# QEMU multi-queue requires:
# -device virtio-net-pci,netdev=net0,mq=on,vectors=10
# -netdev vhost-user,id=net0,chardev=char0,queues=4

show vhost-user    # VPP: show all vhost-user interfaces and queue state

Performance optimisation for vhost-user:

Use huge pages in the VM - map VM memory with 2MB pages for fewer TLB misses in VPP's shared memory access
CPU pinning - pin QEMU vCPUs to cores that are NUMA-local to the VPP worker thread handling the vhost interface
Multi-queue - configure multiple vhost queues (equal to vCPU count) for parallel TX/RX
Packed virtqueue - newer QEMU/kernel supports packed vring format, reducing cache traffic vs split-ring. Enable with packed=on in QEMU device args

AF_PACKET - DEVELOPMENT AND TESTING

🔬

AF_PACKET - When You Don't Have Hugepages

DEV/TEST

AF_PACKET connects VPP to a Linux network interface via the kernel's AF_PACKET socket family (raw socket that receives all frames). It requires no hugepages, no VFIO, no special setup - just a Linux interface name. This makes it invaluable for development in environments where you can't provision hugepages (shared build servers, CI, laptops).

Performance is low - each packet crosses the kernel socket boundary. Use only for:

Functional testing of plugins before deploying to DPDK hardware
CI/CD pipelines where test performance doesn't matter
VPP development on a laptop without DPDK-capable NIC
Quick experiments with VPP's L2/L3 features

# Create AF_PACKET interface on Linux interface eth0
create host-interface name eth0

# Configure and bring up
set interface state host-eth0 up
set interface ip address host-eth0 10.0.0.1/24

show interface host-eth0
show af-packet interfaces

# Use with veth pairs for container testing without real NICs
# (run on Linux host, not inside container):
ip link add vpp0 type veth peer name vpp1
ip link set vpp0 up
ip link set vpp1 up
# VPP: create host-interface name vpp0
# External process uses vpp1

⚠️ Do not performance-test with AF_PACKET. AF_PACKET throughput (~100–500Kpps) is not representative of VPP's real capabilities. All performance benchmarking must use DPDK, memif, or AF_XDP. Use AF_PACKET only to verify functional correctness - that packets are processed correctly, not how fast they're processed.

P3C COMPLETION CHECKLIST

Can select the right interface type for a given scenario using the comparison matrix
Understand TAP v2 internals: virtio vrings shared between VPP and kernel tun/tap driver
Can create a TAP interface with custom host-if-name, IP, and namespace
Know the difference between TAP v2 (new interface) and linux-cp (mirror of DPDK interface)
Know when to use linux-cp: routing daemon (FRR) alongside DPDK-accelerated VPP
Understand AF_XDP three components: eBPF XDP hook, UMEM, XSK rings
Know the three AF_XDP modes: native, generic, zero-copy - and which mlx5 supports
Know AF_XDP's key advantage: NIC stays in kernel while giving near-DPDK speed
Understand vhost-user architecture: virtio vring moved from kernel to VPP userspace
Can configure VPP as vhost-user server and write the matching QEMU command line
Know vhost-user performance tuning: huge pages in VM, CPU pinning, multi-queue, packed vring
Understand AF_PACKET's role: dev/test only, no hugepages, not for performance measurement
Completed Project 4 (interface comparison lab from P3A)

✅ Phase 3 complete. You now know every VPP interface type in depth. Move to Phase 4 - Plugin Development, where you build production-quality plugins using everything learned so far.

← memif 🗺️ Roadmap Next: Plugin Dev →