IPsec — INTERNET PROTOCOL SECURITY
IPsec Architecture
OVERVIEWIPsec is a suite of protocols for securing IP communications at the network layer — below TCP/UDP, meaning all applications benefit without modification. Unlike TLS (application layer), IPsec is transparent to applications. It is the standard for site-to-site VPNs, remote access VPNs, and is used between NGFW gateways.
IPsec has two distinct planes:
- Data plane — ESP (Encapsulating Security Payload, RFC 4303) encrypts and authenticates actual traffic. AH (Authentication Header, RFC 4302) authenticates only — rarely used today.
- Control plane — IKEv2 (Internet Key Exchange v2, RFC 7296) negotiates SAs (Security Associations) — the parameters (algorithms, keys, lifetimes) that ESP uses. Runs over UDP 500 (and UDP 4500 for NAT traversal).
ESP — ENCAPSULATING SECURITY PAYLOAD
ESP Packet Format and AEAD
ESP/* ESP packet format (tunnel mode — wraps entire IP packet) */ [Outer IP: src=GW1, dst=GW2, proto=50 (ESP)] [ESP Header: SPI(4B) + Sequence Number(4B)] [IV/Nonce: 8-16 bytes (algorithm-dependent)] [Encrypted Payload: [Inner IP: src=10.1.0.5, dst=10.2.0.5] [TCP/UDP/ICMP payload] [Padding: 0-255 bytes] [Pad Length: 1B] [Next Header: 4=IPv4, 41=IPv6, 6=TCP, 17=UDP] ] [Auth Tag / ICV: 12-16 bytes] /* SPI — Security Parameters Index */ 32-bit value that identifies which SA to use for decryption Receiver uses (SPI, dst_IP, protocol=ESP) to look up the SA SA contains: decryption key, algorithm, anti-replay window /* Sequence Number */ 32-bit (or 64-bit ESN) counter, starts at 1, never wraps Anti-replay: receiver maintains a sliding window of acceptable SN Duplicate or out-of-window SN → packet dropped /* Modern ESP cipher suites (RFC 8221) */ AES-256-GCM-16: AES-256 GCM with 16-byte ICV (recommended) AES-128-GCM-16: AES-128 GCM with 16-byte ICV ChaCha20-Poly1305: For devices without AES-NI /* Combined mode AEAD (GCM): */ Authenticated: ESP header + IV (as AAD) Encrypted: payload + padding + next_header ICV: 16-byte authentication tag /* AH vs ESP */ AH (proto 51): authenticates IP header + payload (no encryption) Problem: AH covers IP header fields that NAT changes → NAT breaks AH ESP (proto 50): encrypts payload + authenticates ESP header Does NOT authenticate outer IP header → NAT compatible Recommendation: always use ESP with authentication (GCM or AES-CBC + HMAC)
SA, SPD, AND SAD — THE IPsec POLICY AND STATE DATABASES
Security Policy Database and Security Association Database
SA DATABASES/* SPD — Security Policy Database */ /* Defines what to do with each traffic flow */ Policy selector (matches on): Source IP/prefix, Destination IP/prefix Source port range, Destination port range Protocol (TCP, UDP, ICMP, any) Policy action: BYPASS: send without IPsec (for IKE traffic, local management) PROTECT: encrypt with IPsec (most traffic) DISCARD: drop (block) Example SPD: 10.1.0.0/24 → 10.2.0.0/24 any PROTECT (tunnel to remote site) 0.0.0.0/0 → 0.0.0.0/0 UDP:500 BYPASS (allow IKE to negotiate) 0.0.0.0/0 → 0.0.0.0/0 any DISCARD (block unprotected traffic) /* SAD — Security Association Database */ /* Each SA contains the actual keys and parameters for one direction */ SA = { SPI: 32-bit identifier (receiver chooses, sender uses) Protocol: ESP (50) or AH (51) Mode: tunnel or transport Encryption alg: AES-256-GCM Encryption key: 256-bit key (derived by IKEv2) Auth alg: (not needed for GCM — AEAD provides both) Lifetime: 3600 seconds or 4GB bytes (whichever first) Sequence number: current counter Anti-replay window: 64-bit bitmap } /* SAs are unidirectional — one pair per IPsec connection */ Inbound SA: (SPI=0x12345678) → decrypt packets FROM remote Outbound SA: (SPI=0xabcdef01) → encrypt packets TO remote /* Linux kernel IPsec state */ ip xfrm state # show SA database (SAD) ip xfrm policy # show policy database (SPD) ip xfrm state add src 10.0.0.1 dst 10.0.0.2 proto esp spi 0x12345678 \ mode tunnel enc "aes-gcm-esp" 0x... aead-icv-len 128
IKEv2 — THE KEY EXCHANGE AND NEGOTIATION PROTOCOL
IKEv2 Exchange Flow
IKEv2/* IKEv2 establishes the IPsec SA parameters via two exchanges */ /* All IKE runs over UDP 500 (or 4500 for NAT-T) */ INITIATOR RESPONDER │ │ │── IKE_SA_INIT ─────────────────────→ │ │ SA: proposed IKE algorithms │ │ (enc: AES-256-CBC/GCM, │ │ prf: HMAC-SHA256/SHA384, │ │ integ: HMAC-SHA256, │ │ dh: ECP-256/384, DH-2048) │ │ KE: Diffie-Hellman public value │ │ Ni: Nonce (random bytes) │ │ │ │←── IKE_SA_INIT ───────────────────── │ │ SA: selected IKE algorithms │ │ KE: Responder DH public value │ │ Nr: Responder nonce │ │ │ │ [Both compute SKEYSEED = PRF(Ni|Nr, DH_shared)] │ [Derive 7 IKE SA keys via PRF+ ] │ │ [All subsequent messages ENCRYPTED] │ │ │ │── IKE_AUTH ─────────────────────────→│ │ IDi: Initiator identity (FQDN or IP)│ │ CERT: optional certificate │ │ AUTH: signature or PSK HMAC │ │ SAi: proposed Child SA (ESP) params │ │ TSi: Traffic selector (initiator) │ │ TSr: Traffic selector (responder) │ │ │ │←── IKE_AUTH ──────────────────────── │ │ IDr: Responder identity │ │ CERT: certificate │ │ AUTH: signature/PSK │ │ SAr: selected Child SA params │ │ TSi/TSr: confirmed traffic selectors│ │ │ │ [Child SA (ESP) established] │ │ [Keys derived from IKE SA keys] │ /* Authentication methods */ RSA signatures: certificate-based (common for site-to-site) ECDSA: modern certificate auth Pre-Shared Key: shared secret (PSK) — simpler but less scalable EAP: for remote access users (EAP-TLS, EAP-MSCHAPv2) /* IKEv2 additional exchanges */ CREATE_CHILD_SA: create additional SAs (multiple tunnels, rekey) INFORMATIONAL: SA deletion, liveness check (dead peer detection) /* Key material derivation */ SKEYSEED = PRF(Ni | Nr, g^ir) /* g^ir = DH shared secret */ {SK_d | SK_ai | SK_ar | SK_ei | SK_er | SK_pi | SK_pr} = PRF+(SKEYSEED, Ni | Nr | SPIi | SPIr) /* SK_d: used to derive Child SA keys */ /* SK_e: encryption keys for IKE SA */ /* SK_a: integrity keys for IKE SA */ /* SK_p: authentication (PRF input) */
TRANSPORT VS TUNNEL MODE
Mode Comparison
MODESTransport Mode
ESP header inserted between IP header and the transport (TCP/UDP) payload. The original IP header is preserved and visible. Only the transport layer and above are encrypted.
/* Transport mode packet */ [IP: src=A dst=B] [ESP Header] [Encrypted: TCP + data] [ESP Auth] /* Use: host-to-host encryption */ /* Both endpoints must have IPsec */ /* Lower overhead (no extra IP hdr) */ /* Used with GRE: GRE+IPsec transport */
Tunnel Mode
The entire original IP packet is encrypted and encapsulated inside a new IP packet with gateway addresses as src/dst. Inner IP (with real endpoint addresses) is hidden from network observers.
/* Tunnel mode packet */ [Outer IP: GW1→GW2] [ESP Header] [Encrypted: [Inner IP: A→B] [TCP + data]] [ESP Auth] /* Use: site-to-site VPN */ /* Gateways encrypt/decrypt */ /* Endpoints unaware of IPsec */ /* Standard for NGFW VPN */
NAT TRAVERSAL — IPsec THROUGH NAT
UDP Encapsulation for NAT Traversal
NAT-T/* Problem: ESP is IP protocol 50 (not TCP/UDP) */ /* NAT translates TCP/UDP port numbers — has no concept of ESP SPI */ /* Multiple ESP sessions through same NAT → ambiguity (which client?) */ /* NAT-T (NAT Traversal) — RFC 3948 */ /* Encapsulate ESP inside UDP to allow NAT translation */ IKEv2 NAT detection: IKE_SA_INIT includes NAT_DETECTION_SOURCE_IP and NAT_DETECTION_DESTINATION_IP payloads (hashes of IP:port). If hash mismatch → NAT detected. If NAT detected: IKE switches to UDP 4500 (instead of 500) ESP packets wrapped in UDP 4500 header NAT translates the UDP port → multiple clients possible /* NAT-T packet structure */ [Outer IP: src=client_public_IP, dst=VPN_GW] [UDP: sport=4500 dport=4500] [Non-ESP Marker: 0x00000000 (4 bytes, distinguishes from IKE)] [ESP header + encrypted payload] /* Keepalive for NAT mappings */ /* NAT state tables expire idle UDP sessions (often 30-120s) */ /* IKEv2 NAT-T keepalive: single 0xFF byte every 20s on UDP 4500 */ dpd-timeout 30 # StrongSwan: dead peer detection nat-keepalive 20 # keepalive interval /* AH cannot work through NAT */ /* AH authenticates the outer IP header including src IP */ /* NAT changes src IP → AH authentication fails → AH is dead */ /* Always use ESP for NAT-compatible IPsec */
IPsec IN LINUX — XFRM AND STRONGSWAN
Linux IPsec Configuration
LINUX/* Linux IPsec: kernel handles ESP (via xfrm), StrongSwan handles IKEv2 */ /* /etc/swanctl/swanctl.conf — StrongSwan IKEv2 config */ connections { site-to-site { local_addrs = 203.0.113.1 # our gateway IP remote_addrs = 198.51.100.1 # peer gateway IP local { auth = pubkey # certificate authentication certs = gw1-cert.pem id = "gw1.example.com" } remote { auth = pubkey id = "gw2.example.com" } children { tunnel { local_ts = 10.1.0.0/24 # traffic selectors remote_ts = 10.2.0.0/24 esp_proposals = aes256gcm128-prfsha384-ecp384 # cipher suite mode = tunnel dpd_action = restart # restart tunnel on DPD failure } } ike_proposals = aes256gcm16-prfsha384-ecp384 version = 2 dpd_delay = 30s } } # Start IKEv2 negotiation swanctl --load-all swanctl --initiate --child tunnel # Monitor swanctl --list-sas # show active IKE and Child SAs ip xfrm state # kernel ESP SAs (keys, algorithms, byte counts) ip xfrm policy # kernel SPD (traffic selectors, actions) ip xfrm monitor # real-time SA events # Check tunnel traffic tcpdump -i eth0 esp # ESP packets tcpdump -i eth0 udp port 500 or udp port 4500 # IKE packets /* XFRM offload to hardware (Intel QAT, Mellanox IPsec offload) */ # For your Mellanox ConnectX cards: # ip xfrm state add ... offload dev eth0 dir in # Pushes ESP encryption/decryption to NIC — reduces CPU overhead significantly
Site-to-Site IPsec VPN with StrongSwan
Objective: Configure a complete IKEv2/IPsec tunnel between two Linux VMs using StrongSwan. Capture and analyse IKEv2 and ESP traffic.
sudo apt install strongswan strongswan-swanctl. Generate certificates with StrongSwan's PKI: pki --gen --type ecdsa --size 256 > ca.der; pki --self --ca --in ca.der --dn "C=IN, O=Test, CN=Test CA" > ca.crt.sudo systemctl start strongswan. Initiate the tunnel: sudo swanctl --initiate --child tunnel. Verify: sudo swanctl --list-sas shows ESTABLISHED.sudo tcpdump -i eth0 -w /tmp/ike.pcap udp port 500 while initiating the tunnel. Open in Wireshark — decode as IKEv2. Identify: IKE_SA_INIT (two messages), IKE_AUTH (two messages). Note which packets are plaintext (IKE_SA_INIT) vs encrypted (IKE_AUTH).tcpdump -i eth0 esp. Verify: IP protocol is 50 (ESP). Check the kernel SA byte counters: ip xfrm state | grep bytes — should increase as traffic flows. Try pinging an address outside the traffic selector — verify it's NOT tunnelled.Manual XFRM SA — IPsec Without IKE
Objective: Configure IPsec manually using ip xfrm commands (no IKEv2). This shows exactly what IKEv2 does automatically and deepens understanding of the SA/SPD model.
ip xfrm state add. Choose a random 256-bit key: openssl rand -hex 32. Create the inbound and outbound SAs on each machine with matching SPIs and the same key. Use AES-256-GCM: aead "rfc4106(gcm(aes))" 0xKEY 128.ip xfrm policy add src 192.168.1.0/24 dst 192.168.2.0/24 dir out tmpl src GW1 dst GW2 proto esp mode tunnel. Add the reverse policy. Verify: ip xfrm state shows 2 SAs; ip xfrm policy shows 2 policies.ip xfrm state | grep -A5 "bytes" — counters should increment. Now intentionally set a wrong key on one side. Verify pings fail (authentication error) and find the error counter in ip xfrm state.M21 MASTERY CHECKLIST
- Know IPsec two planes: data plane (ESP encrypts traffic) and control plane (IKEv2 negotiates keys)
- Know ESP packet format: outer IP + ESP header (SPI + SN) + IV + encrypted payload + auth tag
- Know SPI: 32-bit identifier in ESP header tells receiver which SA (and key) to use for decryption
- Know anti-replay: sequence number + sliding window; duplicate or out-of-window SN → drop
- Know modern ESP cipher: AES-256-GCM-16 (AEAD — no separate auth needed)
- Know why AH is rarely used: covers outer IP header → incompatible with NAT
- Know SPD (Security Policy Database): selector (src/dst IP, port) → action (PROTECT/BYPASS/DISCARD)
- Know SAD (Security Association Database): one SA per direction; contains SPI, keys, algorithm, lifetime, anti-replay state
- Know SAs are unidirectional: one pair (inbound + outbound) per IPsec connection
- Know IKEv2 exchange: IKE_SA_INIT (DH key exchange, algorithm negotiation) → IKE_AUTH (authentication, Child SA creation)
- Know IKEv2 key derivation: SKEYSEED = PRF(Ni|Nr, DH_shared) → 7 IKE SA keys via PRF+
- Know IKEv2 authentication methods: RSA/ECDSA certificates, PSK, EAP
- Know transport mode: protects IP payload, preserves IP header — for host-to-host
- Know tunnel mode: encapsulates entire IP packet in new IP — for site-to-site VPN (gateway-to-gateway)
- Know why NAT breaks ESP: NAT changes IP headers, but ESP SPI is not a port number NAT can track
- Know NAT-T: encapsulate ESP inside UDP 4500; IKE detects NAT via NAT_DETECTION payloads
- Know Linux IPsec: kernel xfrm handles ESP; StrongSwan handles IKEv2; ip xfrm state/policy commands
- Know XFRM offload: push ESP encrypt/decrypt to NIC (Mellanox ConnectX) — saves CPU cycles
- Completed Lab 1: configured site-to-site IKEv2 with StrongSwan; captured and decoded IKEv2/ESP traffic
- Completed Lab 2: manually configured XFRM SAs; verified encryption, tested wrong-key failure
✅ When complete: Move to M22 - SSL/TLS Inspection and PKI Operations — the final Phase 5 module, bringing together TLS and PKI knowledge into operational NGFW inspection workflows.