VPP P5 - Control Plane and GoVPP

VPP MASTERY · PHASE 5 · WEEKS 19–22+

🎛️ Control Plane & GoVPP

GoVPP binary API client · Stats API · vpp_papi · Observability · NUMA tuning · Production patterns

github.com/FDio/govpp Stats API vpp_papi Projects 8 & 9

GOVPP - GO CLIENT FOR VPP BINARY API

🐹

GoVPP Architecture and Setup

GOVPP

GoVPP (github.com/FDio/govpp) is the official Go library for VPP's binary API. It connects to VPP via a Unix socket or shared memory, sends request messages, and receives reply/notification messages. GoVPP auto-generates Go structs from VPP's .api.json files - so every VPP API is accessible with full type safety.

// ── go.mod setup ──
// go get go.fd.io/govpp@latest

package main

import (
    "context"
    "fmt"
    "log"

    "go.fd.io/govpp"
    "go.fd.io/govpp/api"
    "go.fd.io/govpp/binapi/interface_types"
    "go.fd.io/govpp/binapi/interfaces"
    "go.fd.io/govpp/binapi/ip"
    "go.fd.io/govpp/binapi/ip_types"
    "go.fd.io/govpp/core"
)

func main() {
    // Connect to VPP binary API socket
    conn, err := govpp.Connect("/run/vpp/api.sock")
    if err != nil {
        log.Fatalf("connect: %v", err)
    }
    defer conn.Disconnect()

    // Open a channel - each goroutine should have its own channel
    ch, err := conn.NewAPIChannel()
    if err != nil {
        log.Fatalf("channel: %v", err)
    }
    defer ch.Close()

    // ── Example 1: Show VPP version ──
    req := &vpe.ShowVersion{}
    reply := &vpe.ShowVersionReply{}
    if err := ch.SendRequest(req).ReceiveReply(reply); err != nil {
        log.Fatalf("ShowVersion: %v", err)
    }
    fmt.Printf("VPP version: %s\n", reply.Version)
}

🔌

Interface Operations

API PATTERNS

// ── List all interfaces ──
reqCtx := ch.SendMultiRequest(&interfaces.SwInterfaceDump{
    SwIfIndex: interface_types.InterfaceIndex(^uint32(0)), // ~0 = all
})
for {
    details := &interfaces.SwInterfaceDetails{}
    stop, err := reqCtx.ReceiveReply(details)
    if stop { break }
    if err != nil { log.Fatalf("recv: %v", err) }
    fmt.Printf("  [%d] %s  admin:%v link:%v\n",
        details.SwIfIndex,
        details.InterfaceName,
        details.AdminUpDown, details.LinkUpDown)
}

// ── Set interface state up ──
_, err = ch.SendRequest(&interfaces.SwInterfaceSetFlags{
    SwIfIndex: interface_types.InterfaceIndex(swIfIndex),
    Flags:     interface_types.IF_STATUS_API_FLAG_ADMIN_UP,
}).ReceiveReply(&interfaces.SwInterfaceSetFlagsReply{})

// ── Add IPv4 address ──
_, err = ch.SendRequest(&interfaces.SwInterfaceAddDelAddress{
    SwIfIndex: interface_types.InterfaceIndex(swIfIndex),
    IsAdd:     true,
    Prefix: ip_types.AddressWithPrefix{
        Address: ip_types.Address{
            Af: ip_types.ADDRESS_IP4,
            Un: ip_types.AddressUnionIP4(ip_types.IP4Address{10, 0, 0, 1}),
        },
        Len: 24,
    },
}).ReceiveReply(&interfaces.SwInterfaceAddDelAddressReply{})

// ── Add a static route ──
_, err = ch.SendRequest(&ip.IPRouteAddDel{
    IsAdd: true,
    Route: ip.IPRoute{
        TableID: 0,
        Prefix: ip_types.Prefix{
            Address: ip_types.Address{
                Af: ip_types.ADDRESS_IP4,
                Un: ip_types.AddressUnionIP4(ip_types.IP4Address{10, 1, 0, 0}),
            },
            Len: 24,
        },
        Paths: []ip.FibPath{{
            SwIfIndex: interface_types.InterfaceIndex(swIfIndex),
            Proto:     ip.FIB_API_PATH_NH_PROTO_IP4,
            Nh: ip.FibPathNh{
                Address: ip_types.AddressUnionIP4(
                    ip_types.IP4Address{10, 0, 0, 2}),
            },
            Weight:     1,
            Preference: 0,
        }},
    },
}).ReceiveReply(&ip.IPRouteAddDelReply{})

GOVPP - NOTIFICATIONS AND CHANNELS

📨

Event Subscriptions and Multi-Channel Patterns

ADVANCED

<pre">// ── Subscribe to interface state change events ── notifChan := make(chan api.Message, 100) sub, err := conn.WatchEvent(context.Background(), (*interfaces.SwInterfaceEvent)(nil)) if err != nil { log.Fatalf("subscribe: %v", err) } // Enable notifications (VPP won't send events without this) ch.SendRequest(&interfaces.WantInterfaceEvents{ EnableDisable: 1, PID: uint32(os.Getpid()), }).ReceiveReply(&interfaces.WantInterfaceEventsReply{}) // Process events in a goroutine go func() { for { msg, ok := <-sub.Events() if !ok { return } ev := msg.(*interfaces.SwInterfaceEvent) fmt.Printf("Interface %d: admin=%v link=%v\n", ev.SwIfIndex, ev.AdminUpDown, ev.LinkUpDown) } }() // ── Multi-channel pattern: one channel per worker goroutine ── type VPPWorker struct { ch api.Channel } func NewWorker(conn api.Connection) (*VPPWorker, error) { ch, err := conn.NewAPIChannel() if err != nil { return nil, err } return &VPPWorker{ch: ch}, nil } // Each goroutine has its OWN channel - no sharing, no locking for i := 0; i < numWorkers; i++ { w, _ := NewWorker(conn) go w.processBatch(routes[i]) } // ── Bulk route programming - batch via channel ── func (w *VPPWorker) programRoutes(routes []Route) error { for _, r := range routes { req := buildIPRouteAddDel(r) reply := &ip.IPRouteAddDelReply{} if err := w.ch.SendRequest(req).ReceiveReply(reply); err != nil { return err } if reply.Retval != 0 { return fmt.Errorf("route add retval %d", reply.Retval) } } return nil }</pre>

STATS API - HIGH-FREQUENCY TELEMETRY

📊

VPP Stats Segment - Zero-Copy Telemetry

STATS

The Stats API is VPP's high-performance telemetry interface. It exposes per-node, per-interface, per-worker, and per-error counters via a shared memory segment - no IPC, no socket round-trip. A monitoring agent can read millions of counters per second without impacting the VPP dataplane.

// ── GoVPP Stats client ──
import "go.fd.io/govpp/adapter/statsclient"

func monitorVPP() {
    // Connect to stats segment (separate from binary API socket)
    client := statsclient.NewStatsClient("/run/vpp/stats.sock")
    if err := client.Connect(); err != nil {
        log.Fatalf("stats connect: %v", err)
    }
    defer client.Disconnect()

    // ── Poll interface counters ──
    ifCounters, err := client.GetInterfaceCounters()
    for _, ifc := range ifCounters {
        fmt.Printf("%-30s  rx: %8d pkts %12d bytes  tx: %8d pkts %12d bytes\n",
            ifc.InterfaceName,
            ifc.RxPackets, ifc.RxBytes,
            ifc.TxPackets, ifc.TxBytes)
    }

    // ── Poll per-node stats (show run equivalent) ──
    nodeCounters, err := client.GetNodeCounters()
    for _, nc := range nodeCounters {
        if nc.Calls == 0 { continue }
        fmt.Printf("%-40s  calls:%8d vectors:%8d vecs/call:%.1f\n",
            nc.NodeName, nc.Calls, nc.Vectors,
            float64(nc.Vectors)/float64(nc.Calls))
    }

    // ── Poll error counters (show error equivalent) ──
    errCounters, err := client.GetErrorCounters()
    for _, ec := range errCounters {
        if ec.Value == 0 { continue }
        fmt.Printf("%-50s  %d\n", ec.CounterName, ec.Value)
    }

    // ── Continuous monitoring loop ──
    ticker := time.NewTicker(1 * time.Second)
    for range ticker.C {
        // Stats segment uses version counter for consistency
        // GetInterfaceCounters handles the epoch check internally
        ifc, _ := client.GetInterfaceCounters()
        exportMetrics(ifc)  // Prometheus, InfluxDB, etc.
    }
}

💡 Stats segment vs binary API for telemetry: The Stats API reads from shared memory - it costs ~1 microsecond per read. The binary API requires a socket round-trip - ~50–100 microseconds. For polling counters at 1Hz or faster, always use the Stats API. Use the binary API only for configuration operations (add route, set interface state).

VPP_PAPI - PYTHON BINDINGS

🐍

vpp_papi - Scripting and Automation

PYTHON

vpp_papi (src/vpp-api/python/vpp_papi/) provides Python bindings for VPP's binary API. It is the same library used by VPP's Python test framework. Use it for automation scripts, management integrations, and quick prototyping.

from vpp_papi import VPPApiClient
import socket

# Connect to VPP
vpp = VPPApiClient(apifiles=["/usr/share/vpp/api/core/"],
                   server_address="/run/vpp/api.sock")
vpp.connect("my-python-agent")

# ── Show version ──
rv = vpp.api.show_version()
print(f"VPP: {rv.version}")

# ── List interfaces ──
for intf in vpp.api.sw_interface_dump():
    print(f"  [{intf.sw_if_index}] {intf.interface_name.rstrip(chr(0))} "
          f"link={'up' if intf.link_up_down else 'down'}")

# ── Create a TAP interface ──
rv = vpp.api.tap_create_v3(
    id=0,
    host_if_name_set=True,
    host_if_name=b"vpp0\x00",
    host_ip4_prefix_set=True,
    host_ip4_prefix={
        "address": {"af": "ADDRESS_IP4",
                    "un": {"ip4": socket.inet_aton("10.10.0.2")}},
        "len": 30
    }
)
print(f"TAP created: sw_if_index={rv.sw_if_index}")

# ── Add an IP route ──
vpp.api.ip_route_add_del(
    is_add=True,
    route={
        "prefix": {"address": {"af": "ADDRESS_IP4",
                               "un": {"ip4": socket.inet_aton("10.1.0.0")}},
                   "len": 24},
        "paths": [{"sw_if_index": rv.sw_if_index,
                   "proto": "FIB_API_PATH_NH_PROTO_IP4",
                   "nh": {"address": {"ip4": socket.inet_aton("10.10.0.1")}},
                   "weight": 1, "preference": 0}]
    }
)

# ── Subscribe to interface events ──
@vpp.register_event_callback
def on_interface_event(msg_name, msg):
    if msg_name == "sw_interface_event":
        print(f"Interface {msg.sw_if_index} link {'up' if msg.link_up_down else 'down'}")

vpp.api.want_interface_events(enable_disable=1, pid=0)

vpp.disconnect()

PERFORMANCE TUNING AND PRODUCTION PATTERNS

⚡

NUMA Awareness and CPU Topology

PERFORMANCE

VPP performance is highly sensitive to NUMA placement. Accessing memory across NUMA nodes adds ~100ns latency and reduces throughput by 30–50%. The goal is to keep NIC, hugepages, CPU cores, and worker threads all on the same NUMA node.

# Step 1: Find which NUMA node your Mellanox NIC is on
cat /sys/bus/pci/devices/0000:03:00.0/numa_node
# e.g. output: 0  → NUMA 0

# Step 2: Find NUMA-local CPU cores
lscpu | grep -A5 "NUMA node0"
# e.g. NUMA node0 CPU(s): 0-11,24-35

# Step 3: Configure startup.conf to use NUMA-local cores
cpu {
  main-core 0          # core 0 on NUMA 0
  corelist-workers 2-5 # cores 2-5 on NUMA 0
}
dpdk {
  socket-mem 4096,0    # 4GB on NUMA 0, 0 on NUMA 1
}
buffers {
  buffers-per-numa 262144   # 256K buffers on NUMA 0
}

# Step 4: Verify with VPP
# vppctl: show interface rx-placement
# Verify each queue is on the worker thread whose core is NUMA-local to the NIC

Tuning Area	Recommendation	How to Verify
NUMA placement	NIC, hugepages, and workers all on same NUMA node	numastat, show interface rx-placement
Workers = queues	num-rx-queues == num worker threads for full saturation	show run - vectors/call should be 64–256
Huge pages size	1GB pages preferred over 2MB at high load (fewer TLB misses)	perf stat -e dTLB-load-misses
CPU isolation	isolcpus=2-5 in kernel cmdline; no other processes on worker cores	cat /sys/devices/system/cpu/isolated
IRQ affinity	Move all NIC IRQs to non-worker cores (set_irq_affinity.sh)	cat /proc/irq/*/smp_affinity_list
Buffer sizing	buffers-per-numa ≥ 2× (num_workers × (rx_desc + tx_desc))	show buffers - free% should stay above 20%
Turbo / C-states	Disable CPU power management (cpufreq governor=performance)	cpupower frequency-info

PROJECT 8

GoVPP Control Plane Agent

Objective: Build a Go agent that manages a VPP instance - configures interfaces, programs routes, polls stats, and exposes a REST API for a management frontend.

Implement Connect(socketPath string) that establishes GoVPP connection and opens a pool of channels (one per goroutine). Handle reconnect with exponential backoff on disconnect.

Implement ConfigureInterface(name string, ip string, prefix int): list interfaces, find by name, set admin-up, add IP address. Return error if interface not found.

Implement ProgramRoutes(routes []Route): batch-program a list of static routes using a dedicated goroutine + channel. Measure time to program 1000 routes and report routes/second.

Implement a Stats poller: connect to Stats segment, poll interface counters every 1 second, compute RX/TX PPS (delta / interval), expose via Prometheus HTTP endpoint at :9090/metrics.

Implement event subscription: subscribe to SwInterfaceEvent, log all link state changes with timestamp. Test by toggling an interface up/down via vppctl and verifying the agent logs the event.

Add a REST API: GET /interfaces returns JSON list of all VPP interfaces with counters. POST /routes programs a new route. DELETE /routes/{prefix} removes it. Test with curl.

PROJECT 9

End-to-End Production Topology

Objective: Integrate all phases into a complete topology: VPP + DPDK physical ports + memif container connections + linux-cp for control plane + GoVPP management agent + observability.

Design the topology: 2 DPDK ports (physical NIC), 2 memif ports (connecting to application containers), 1 TAP for management. VPP acts as the central packet forwarder.

Deploy linux-cp mirroring both DPDK interfaces to Linux for FRRouting OSPF. Verify FRR forms OSPF adjacency and installs routes. VPP dataplane uses these routes for forwarding.

Enable the classify plugin (from Phase 4) on ip4-unicast arc. Program DROP rules for RFC-1918 sources via the GoVPP agent's REST API. Verify drops with show error.

Deploy the Prometheus + Grafana stack. Import the GoVPP agent's metrics. Build a dashboard showing: RX/TX PPS per interface, vectors/call for dpdk-input, buffer utilisation, error counter rates.

Run a 30-minute traffic test at 50% line rate. Verify: zero packet loss (show error), stable vectors/call for dpdk-input (32–256), free buffer% stays above 30%, FRR OSPF adjacency stays up throughout.

P5 COMPLETION CHECKLIST

Can connect to VPP from Go using GoVPP, open channels, and send request/reply messages
Know the multi-channel pattern: one channel per goroutine, no sharing
Can implement interface dump, set interface state, add IP address in Go
Can implement IP route add/delete with correct ip_types structures
Can subscribe to VPP events (want_interface_events) and handle them in a goroutine
Understand the Stats API architecture: shared memory, zero IPC cost
Can connect to Stats segment and poll interface, node, and error counters
Know when to use Stats API vs binary API (telemetry vs configuration)
Can write a vpp_papi Python script: connect, API call, event subscription
Know the 7 key NUMA/performance tuning areas and the CLI to verify each
Understand workers=queues constraint and how to size buffers-per-numa
Completed Project 8 (GoVPP agent with Prometheus) and Project 9 (full production topology)

🎉 Phase 5 complete. You can now build production VPP deployments end-to-end: from DPDK physical interfaces through custom plugins to a fully automated GoVPP control plane with observability. Bonus: continue to the Host Stack module to explore VPP's TCP/Session layer, VCL, and application namespaces.

← Plugin Dev 🗺️ Roadmap 🌐 Host Stack (Bonus) →