NETWORKING MASTERY · PHASE 5 · MODULE 21 · WEEK 19
🛡️ IPsec and IKEv2
ESP/AH protocols · SA and SPD · IKEv2 exchange · Transport vs tunnel mode · NAT traversal · StrongSwan
Advanced Prerequisite: M19 Cryptography RFC 7296 · RFC 4303 VPN and NGFW Core 2 Labs

IPsec — INTERNET PROTOCOL SECURITY

🛡️

IPsec Architecture

OVERVIEW

IPsec is a suite of protocols for securing IP communications at the network layer — below TCP/UDP, meaning all applications benefit without modification. Unlike TLS (application layer), IPsec is transparent to applications. It is the standard for site-to-site VPNs, remote access VPNs, and is used between NGFW gateways.

IPsec has two distinct planes:

  • Data plane — ESP (Encapsulating Security Payload, RFC 4303) encrypts and authenticates actual traffic. AH (Authentication Header, RFC 4302) authenticates only — rarely used today.
  • Control plane — IKEv2 (Internet Key Exchange v2, RFC 7296) negotiates SAs (Security Associations) — the parameters (algorithms, keys, lifetimes) that ESP uses. Runs over UDP 500 (and UDP 4500 for NAT traversal).

ESP — ENCAPSULATING SECURITY PAYLOAD

📦

ESP Packet Format and AEAD

ESP
/* ESP packet format (tunnel mode — wraps entire IP packet) */
[Outer IP: src=GW1, dst=GW2, proto=50 (ESP)]
[ESP Header: SPI(4B) + Sequence Number(4B)]
[IV/Nonce: 8-16 bytes (algorithm-dependent)]
[Encrypted Payload:
  [Inner IP: src=10.1.0.5, dst=10.2.0.5]
  [TCP/UDP/ICMP payload]
  [Padding: 0-255 bytes]
  [Pad Length: 1B]
  [Next Header: 4=IPv4, 41=IPv6, 6=TCP, 17=UDP]
]
[Auth Tag / ICV: 12-16 bytes]

/* SPI — Security Parameters Index */
32-bit value that identifies which SA to use for decryption
Receiver uses (SPI, dst_IP, protocol=ESP) to look up the SA
SA contains: decryption key, algorithm, anti-replay window

/* Sequence Number */
32-bit (or 64-bit ESN) counter, starts at 1, never wraps
Anti-replay: receiver maintains a sliding window of acceptable SN
Duplicate or out-of-window SN → packet dropped

/* Modern ESP cipher suites (RFC 8221) */
AES-256-GCM-16:     AES-256 GCM with 16-byte ICV (recommended)
AES-128-GCM-16:     AES-128 GCM with 16-byte ICV
ChaCha20-Poly1305:  For devices without AES-NI

/* Combined mode AEAD (GCM): */
Authenticated:  ESP header + IV (as AAD)
Encrypted:      payload + padding + next_header
ICV:            16-byte authentication tag

/* AH vs ESP */
AH (proto 51):  authenticates IP header + payload (no encryption)
                Problem: AH covers IP header fields that NAT changes → NAT breaks AH
ESP (proto 50): encrypts payload + authenticates ESP header
                Does NOT authenticate outer IP header → NAT compatible
Recommendation: always use ESP with authentication (GCM or AES-CBC + HMAC)

SA, SPD, AND SAD — THE IPsec POLICY AND STATE DATABASES

🗄️

Security Policy Database and Security Association Database

SA DATABASES
/* SPD — Security Policy Database */
/* Defines what to do with each traffic flow */

Policy selector (matches on):
  Source IP/prefix, Destination IP/prefix
  Source port range, Destination port range
  Protocol (TCP, UDP, ICMP, any)

Policy action:
  BYPASS:  send without IPsec (for IKE traffic, local management)
  PROTECT: encrypt with IPsec (most traffic)
  DISCARD: drop (block)

Example SPD:
  10.1.0.0/24 → 10.2.0.0/24  any   PROTECT (tunnel to remote site)
  0.0.0.0/0   → 0.0.0.0/0    UDP:500 BYPASS  (allow IKE to negotiate)
  0.0.0.0/0   → 0.0.0.0/0    any   DISCARD (block unprotected traffic)

/* SAD — Security Association Database */
/* Each SA contains the actual keys and parameters for one direction */

SA = {
  SPI:             32-bit identifier (receiver chooses, sender uses)
  Protocol:        ESP (50) or AH (51)
  Mode:            tunnel or transport
  Encryption alg:  AES-256-GCM
  Encryption key:  256-bit key (derived by IKEv2)
  Auth alg:        (not needed for GCM — AEAD provides both)
  Lifetime:        3600 seconds or 4GB bytes (whichever first)
  Sequence number: current counter
  Anti-replay window: 64-bit bitmap
}

/* SAs are unidirectional — one pair per IPsec connection */
Inbound SA:  (SPI=0x12345678) → decrypt packets FROM remote
Outbound SA: (SPI=0xabcdef01) → encrypt packets TO remote

/* Linux kernel IPsec state */
ip xfrm state          # show SA database (SAD)
ip xfrm policy         # show policy database (SPD)
ip xfrm state add src 10.0.0.1 dst 10.0.0.2 proto esp spi 0x12345678 \
    mode tunnel enc "aes-gcm-esp" 0x... aead-icv-len 128

IKEv2 — THE KEY EXCHANGE AND NEGOTIATION PROTOCOL

🤝

IKEv2 Exchange Flow

IKEv2
/* IKEv2 establishes the IPsec SA parameters via two exchanges */
/* All IKE runs over UDP 500 (or 4500 for NAT-T) */

INITIATOR                              RESPONDER
    │                                      │
    │── IKE_SA_INIT ─────────────────────→ │
    │   SA: proposed IKE algorithms         │
    │     (enc: AES-256-CBC/GCM,            │
    │      prf: HMAC-SHA256/SHA384,         │
    │      integ: HMAC-SHA256,              │
    │      dh: ECP-256/384, DH-2048)       │
    │   KE: Diffie-Hellman public value     │
    │   Ni: Nonce (random bytes)            │
    │                                      │
    │←── IKE_SA_INIT ───────────────────── │
    │   SA: selected IKE algorithms         │
    │   KE: Responder DH public value       │
    │   Nr: Responder nonce                 │
    │                                      │
    │ [Both compute SKEYSEED = PRF(Ni|Nr, DH_shared)]
    │ [Derive 7 IKE SA keys via PRF+ ]      │
    │ [All subsequent messages ENCRYPTED]   │
    │                                      │
    │── IKE_AUTH ─────────────────────────→│
    │   IDi: Initiator identity (FQDN or IP)│
    │   CERT: optional certificate          │
    │   AUTH: signature or PSK HMAC        │
    │   SAi: proposed Child SA (ESP) params │
    │   TSi: Traffic selector (initiator)   │
    │   TSr: Traffic selector (responder)   │
    │                                      │
    │←── IKE_AUTH ────────────────────────  │
    │   IDr: Responder identity             │
    │   CERT: certificate                   │
    │   AUTH: signature/PSK                 │
    │   SAr: selected Child SA params       │
    │   TSi/TSr: confirmed traffic selectors│
    │                                      │
    │ [Child SA (ESP) established]          │
    │ [Keys derived from IKE SA keys]       │

/* Authentication methods */
RSA signatures:    certificate-based (common for site-to-site)
ECDSA:             modern certificate auth
Pre-Shared Key:    shared secret (PSK) — simpler but less scalable
EAP:               for remote access users (EAP-TLS, EAP-MSCHAPv2)

/* IKEv2 additional exchanges */
CREATE_CHILD_SA:   create additional SAs (multiple tunnels, rekey)
INFORMATIONAL:     SA deletion, liveness check (dead peer detection)

/* Key material derivation */
SKEYSEED = PRF(Ni | Nr, g^ir)           /* g^ir = DH shared secret */
{SK_d | SK_ai | SK_ar | SK_ei | SK_er | SK_pi | SK_pr} =
    PRF+(SKEYSEED, Ni | Nr | SPIi | SPIr)
/* SK_d: used to derive Child SA keys */
/* SK_e: encryption keys for IKE SA */
/* SK_a: integrity keys for IKE SA */
/* SK_p: authentication (PRF input) */

TRANSPORT VS TUNNEL MODE

🔀

Mode Comparison

MODES

Transport Mode

ESP header inserted between IP header and the transport (TCP/UDP) payload. The original IP header is preserved and visible. Only the transport layer and above are encrypted.

/* Transport mode packet */
[IP: src=A dst=B]
[ESP Header]
[Encrypted: TCP + data]
[ESP Auth]

/* Use: host-to-host encryption */
/* Both endpoints must have IPsec */
/* Lower overhead (no extra IP hdr) */
/* Used with GRE: GRE+IPsec transport */

Tunnel Mode

The entire original IP packet is encrypted and encapsulated inside a new IP packet with gateway addresses as src/dst. Inner IP (with real endpoint addresses) is hidden from network observers.

/* Tunnel mode packet */
[Outer IP: GW1→GW2]
[ESP Header]
[Encrypted:
  [Inner IP: A→B]
  [TCP + data]]
[ESP Auth]

/* Use: site-to-site VPN */
/* Gateways encrypt/decrypt */
/* Endpoints unaware of IPsec */
/* Standard for NGFW VPN */

NAT TRAVERSAL — IPsec THROUGH NAT

🔧

UDP Encapsulation for NAT Traversal

NAT-T
/* Problem: ESP is IP protocol 50 (not TCP/UDP) */
/* NAT translates TCP/UDP port numbers — has no concept of ESP SPI */
/* Multiple ESP sessions through same NAT → ambiguity (which client?) */

/* NAT-T (NAT Traversal) — RFC 3948 */
/* Encapsulate ESP inside UDP to allow NAT translation */

IKEv2 NAT detection:
  IKE_SA_INIT includes NAT_DETECTION_SOURCE_IP and NAT_DETECTION_DESTINATION_IP
  payloads (hashes of IP:port). If hash mismatch → NAT detected.

If NAT detected:
  IKE switches to UDP 4500 (instead of 500)
  ESP packets wrapped in UDP 4500 header
  NAT translates the UDP port → multiple clients possible

/* NAT-T packet structure */
[Outer IP: src=client_public_IP, dst=VPN_GW]
[UDP: sport=4500 dport=4500]
[Non-ESP Marker: 0x00000000 (4 bytes, distinguishes from IKE)]
[ESP header + encrypted payload]

/* Keepalive for NAT mappings */
/* NAT state tables expire idle UDP sessions (often 30-120s) */
/* IKEv2 NAT-T keepalive: single 0xFF byte every 20s on UDP 4500 */
dpd-timeout 30         # StrongSwan: dead peer detection
nat-keepalive 20       # keepalive interval

/* AH cannot work through NAT */
/* AH authenticates the outer IP header including src IP */
/* NAT changes src IP → AH authentication fails → AH is dead */
/* Always use ESP for NAT-compatible IPsec */

IPsec IN LINUX — XFRM AND STRONGSWAN

🐧

Linux IPsec Configuration

LINUX
/* Linux IPsec: kernel handles ESP (via xfrm), StrongSwan handles IKEv2 */

/* /etc/swanctl/swanctl.conf — StrongSwan IKEv2 config */
connections {
  site-to-site {
    local_addrs  = 203.0.113.1      # our gateway IP
    remote_addrs = 198.51.100.1     # peer gateway IP

    local {
      auth  = pubkey                # certificate authentication
      certs = gw1-cert.pem
      id    = "gw1.example.com"
    }
    remote {
      auth  = pubkey
      id    = "gw2.example.com"
    }

    children {
      tunnel {
        local_ts   = 10.1.0.0/24   # traffic selectors
        remote_ts  = 10.2.0.0/24
        esp_proposals = aes256gcm128-prfsha384-ecp384  # cipher suite
        mode      = tunnel
        dpd_action = restart       # restart tunnel on DPD failure
      }
    }
    ike_proposals = aes256gcm16-prfsha384-ecp384
    version = 2
    dpd_delay = 30s
  }
}

# Start IKEv2 negotiation
swanctl --load-all
swanctl --initiate --child tunnel

# Monitor
swanctl --list-sas        # show active IKE and Child SAs
ip xfrm state             # kernel ESP SAs (keys, algorithms, byte counts)
ip xfrm policy            # kernel SPD (traffic selectors, actions)
ip xfrm monitor           # real-time SA events

# Check tunnel traffic
tcpdump -i eth0 esp       # ESP packets
tcpdump -i eth0 udp port 500 or udp port 4500  # IKE packets

/* XFRM offload to hardware (Intel QAT, Mellanox IPsec offload) */
# For your Mellanox ConnectX cards:
# ip xfrm state add ... offload dev eth0 dir in
# Pushes ESP encryption/decryption to NIC — reduces CPU overhead significantly
LAB 1

Site-to-Site IPsec VPN with StrongSwan

Objective: Configure a complete IKEv2/IPsec tunnel between two Linux VMs using StrongSwan. Capture and analyse IKEv2 and ESP traffic.

1
Set up two VMs: GW1 (10.0.0.1, serves 192.168.1.0/24) and GW2 (10.0.0.2, serves 192.168.2.0/24). Install StrongSwan: sudo apt install strongswan strongswan-swanctl. Generate certificates with StrongSwan's PKI: pki --gen --type ecdsa --size 256 > ca.der; pki --self --ca --in ca.der --dn "C=IN, O=Test, CN=Test CA" > ca.crt.
2
Create gateway certificates signed by the test CA. Configure swanctl.conf on both gateways as shown in Tab 6. Start StrongSwan: sudo systemctl start strongswan. Initiate the tunnel: sudo swanctl --initiate --child tunnel. Verify: sudo swanctl --list-sas shows ESTABLISHED.
3
Capture IKEv2: sudo tcpdump -i eth0 -w /tmp/ike.pcap udp port 500 while initiating the tunnel. Open in Wireshark — decode as IKEv2. Identify: IKE_SA_INIT (two messages), IKE_AUTH (two messages). Note which packets are plaintext (IKE_SA_INIT) vs encrypted (IKE_AUTH).
4
Test the tunnel: ping from 192.168.1.5 (behind GW1) to 192.168.2.5 (behind GW2). Capture ESP packets: tcpdump -i eth0 esp. Verify: IP protocol is 50 (ESP). Check the kernel SA byte counters: ip xfrm state | grep bytes — should increase as traffic flows. Try pinging an address outside the traffic selector — verify it's NOT tunnelled.
LAB 2

Manual XFRM SA — IPsec Without IKE

Objective: Configure IPsec manually using ip xfrm commands (no IKEv2). This shows exactly what IKEv2 does automatically and deepens understanding of the SA/SPD model.

1
On both VMs, manually add SAs using ip xfrm state add. Choose a random 256-bit key: openssl rand -hex 32. Create the inbound and outbound SAs on each machine with matching SPIs and the same key. Use AES-256-GCM: aead "rfc4106(gcm(aes))" 0xKEY 128.
2
Add security policies: ip xfrm policy add src 192.168.1.0/24 dst 192.168.2.0/24 dir out tmpl src GW1 dst GW2 proto esp mode tunnel. Add the reverse policy. Verify: ip xfrm state shows 2 SAs; ip xfrm policy shows 2 policies.
3
Ping across the tunnel. Capture and verify ESP packets appear. Check ip xfrm state | grep -A5 "bytes" — counters should increment. Now intentionally set a wrong key on one side. Verify pings fail (authentication error) and find the error counter in ip xfrm state.

M21 MASTERY CHECKLIST

When complete: Move to M22 - SSL/TLS Inspection and PKI Operations — the final Phase 5 module, bringing together TLS and PKI knowledge into operational NGFW inspection workflows.

← M20 TLS 🗺️ Roadmap Next: M22 - SSL Inspection →