IPsec β INTERNET PROTOCOL SECURITY
IPsec Architecture
OVERVIEWIPsec is a suite of protocols for securing IP communications at the network layer β below TCP/UDP, meaning all applications benefit without modification. Unlike TLS (application layer), IPsec is transparent to applications. It is the standard for site-to-site VPNs, remote access VPNs, and is used between NGFW gateways.
IPsec has two distinct planes:
- Data plane β ESP (Encapsulating Security Payload, RFC 4303) encrypts and authenticates actual traffic. AH (Authentication Header, RFC 4302) authenticates only β rarely used today.
- Control plane β IKEv2 (Internet Key Exchange v2, RFC 7296) negotiates SAs (Security Associations) β the parameters (algorithms, keys, lifetimes) that ESP uses. Runs over UDP 500 (and UDP 4500 for NAT traversal).
ESP β ENCAPSULATING SECURITY PAYLOAD
ESP Packet Format and AEAD
ESP/* ESP packet format (tunnel mode β wraps entire IP packet) */
[Outer IP: src=GW1, dst=GW2, proto=50 (ESP)]
[ESP Header: SPI(4B) + Sequence Number(4B)]
[IV/Nonce: 8-16 bytes (algorithm-dependent)]
[Encrypted Payload:
[Inner IP: src=10.1.0.5, dst=10.2.0.5]
[TCP/UDP/ICMP payload]
[Padding: 0-255 bytes]
[Pad Length: 1B]
[Next Header: 4=IPv4, 41=IPv6, 6=TCP, 17=UDP]
]
[Auth Tag / ICV: 12-16 bytes]
/* SPI β Security Parameters Index */
32-bit value that identifies which SA to use for decryption
Receiver uses (SPI, dst_IP, protocol=ESP) to look up the SA
SA contains: decryption key, algorithm, anti-replay window
/* Sequence Number */
32-bit (or 64-bit ESN) counter, starts at 1, never wraps
Anti-replay: receiver maintains a sliding window of acceptable SN
Duplicate or out-of-window SN β packet dropped
/* Modern ESP cipher suites (RFC 8221) */
AES-256-GCM-16: AES-256 GCM with 16-byte ICV (recommended)
AES-128-GCM-16: AES-128 GCM with 16-byte ICV
ChaCha20-Poly1305: For devices without AES-NI
/* Combined mode AEAD (GCM): */
Authenticated: ESP header + IV (as AAD)
Encrypted: payload + padding + next_header
ICV: 16-byte authentication tag
/* AH vs ESP */
AH (proto 51): authenticates IP header + payload (no encryption)
Problem: AH covers IP header fields that NAT changes β NAT breaks AH
ESP (proto 50): encrypts payload + authenticates ESP header
Does NOT authenticate outer IP header β NAT compatible
Recommendation: always use ESP with authentication (GCM or AES-CBC + HMAC)
SA, SPD, AND SAD β THE IPsec POLICY AND STATE DATABASES
Security Policy Database and Security Association Database
SA DATABASES/* SPD β Security Policy Database */
/* Defines what to do with each traffic flow */
Policy selector (matches on):
Source IP/prefix, Destination IP/prefix
Source port range, Destination port range
Protocol (TCP, UDP, ICMP, any)
Policy action:
BYPASS: send without IPsec (for IKE traffic, local management)
PROTECT: encrypt with IPsec (most traffic)
DISCARD: drop (block)
Example SPD:
10.1.0.0/24 β 10.2.0.0/24 any PROTECT (tunnel to remote site)
0.0.0.0/0 β 0.0.0.0/0 UDP:500 BYPASS (allow IKE to negotiate)
0.0.0.0/0 β 0.0.0.0/0 any DISCARD (block unprotected traffic)
/* SAD β Security Association Database */
/* Each SA contains the actual keys and parameters for one direction */
SA = {
SPI: 32-bit identifier (receiver chooses, sender uses)
Protocol: ESP (50) or AH (51)
Mode: tunnel or transport
Encryption alg: AES-256-GCM
Encryption key: 256-bit key (derived by IKEv2)
Auth alg: (not needed for GCM β AEAD provides both)
Lifetime: 3600 seconds or 4GB bytes (whichever first)
Sequence number: current counter
Anti-replay window: 64-bit bitmap
}
/* SAs are unidirectional β one pair per IPsec connection */
Inbound SA: (SPI=0x12345678) β decrypt packets FROM remote
Outbound SA: (SPI=0xabcdef01) β encrypt packets TO remote
/* Linux kernel IPsec state */
ip xfrm state # show SA database (SAD)
ip xfrm policy # show policy database (SPD)
ip xfrm state add src 10.0.0.1 dst 10.0.0.2 proto esp spi 0x12345678 \
mode tunnel enc "aes-gcm-esp" 0x... aead-icv-len 128
IKEv2 β THE KEY EXCHANGE AND NEGOTIATION PROTOCOL
IKEv2 Exchange Flow
IKEv2/* IKEv2 establishes the IPsec SA parameters via two exchanges */
/* All IKE runs over UDP 500 (or 4500 for NAT-T) */
INITIATOR RESPONDER
β β
βββ IKE_SA_INIT ββββββββββββββββββββββ β
β SA: proposed IKE algorithms β
β (enc: AES-256-CBC/GCM, β
β prf: HMAC-SHA256/SHA384, β
β integ: HMAC-SHA256, β
β dh: ECP-256/384, DH-2048) β
β KE: Diffie-Hellman public value β
β Ni: Nonce (random bytes) β
β β
ββββ IKE_SA_INIT βββββββββββββββββββββ β
β SA: selected IKE algorithms β
β KE: Responder DH public value β
β Nr: Responder nonce β
β β
β [Both compute SKEYSEED = PRF(Ni|Nr, DH_shared)]
β [Derive 7 IKE SA keys via PRF+ ] β
β [All subsequent messages ENCRYPTED] β
β β
βββ IKE_AUTH βββββββββββββββββββββββββββ
β IDi: Initiator identity (FQDN or IP)β
β CERT: optional certificate β
β AUTH: signature or PSK HMAC β
β SAi: proposed Child SA (ESP) params β
β TSi: Traffic selector (initiator) β
β TSr: Traffic selector (responder) β
β β
ββββ IKE_AUTH ββββββββββββββββββββββββ β
β IDr: Responder identity β
β CERT: certificate β
β AUTH: signature/PSK β
β SAr: selected Child SA params β
β TSi/TSr: confirmed traffic selectorsβ
β β
β [Child SA (ESP) established] β
β [Keys derived from IKE SA keys] β
/* Authentication methods */
RSA signatures: certificate-based (common for site-to-site)
ECDSA: modern certificate auth
Pre-Shared Key: shared secret (PSK) β simpler but less scalable
EAP: for remote access users (EAP-TLS, EAP-MSCHAPv2)
/* IKEv2 additional exchanges */
CREATE_CHILD_SA: create additional SAs (multiple tunnels, rekey)
INFORMATIONAL: SA deletion, liveness check (dead peer detection)
/* Key material derivation */
SKEYSEED = PRF(Ni | Nr, g^ir) /* g^ir = DH shared secret */
{SK_d | SK_ai | SK_ar | SK_ei | SK_er | SK_pi | SK_pr} =
PRF+(SKEYSEED, Ni | Nr | SPIi | SPIr)
/* SK_d: used to derive Child SA keys */
/* SK_e: encryption keys for IKE SA */
/* SK_a: integrity keys for IKE SA */
/* SK_p: authentication (PRF input) */
TRANSPORT VS TUNNEL MODE
Mode Comparison
MODESTransport Mode
ESP header inserted between IP header and the transport (TCP/UDP) payload. The original IP header is preserved and visible. Only the transport layer and above are encrypted.
/* Transport mode packet */ [IP: src=A dst=B] [ESP Header] [Encrypted: TCP + data] [ESP Auth]/* Use: host-to-host encryption / / Both endpoints must have IPsec / / Lower overhead (no extra IP hdr) / / Used with GRE: GRE+IPsec transport */
Tunnel Mode
The entire original IP packet is encrypted and encapsulated inside a new IP packet with gateway addresses as src/dst. Inner IP (with real endpoint addresses) is hidden from network observers.
/* Tunnel mode packet */
[Outer IP: GW1βGW2]
[ESP Header]
[Encrypted:
[Inner IP: AβB]
[TCP + data]]
[ESP Auth]
/* Use: site-to-site VPN */
/* Gateways encrypt/decrypt */
/* Endpoints unaware of IPsec */
/* Standard for NGFW VPN */
NAT TRAVERSAL β IPsec THROUGH NAT
UDP Encapsulation for NAT Traversal
NAT-T/* Problem: ESP is IP protocol 50 (not TCP/UDP) */
/* NAT translates TCP/UDP port numbers β has no concept of ESP SPI */
/* Multiple ESP sessions through same NAT β ambiguity (which client?) */
/* NAT-T (NAT Traversal) β RFC 3948 */
/* Encapsulate ESP inside UDP to allow NAT translation */
IKEv2 NAT detection:
IKE_SA_INIT includes NAT_DETECTION_SOURCE_IP and NAT_DETECTION_DESTINATION_IP
payloads (hashes of IP:port). If hash mismatch β NAT detected.
If NAT detected:
IKE switches to UDP 4500 (instead of 500)
ESP packets wrapped in UDP 4500 header
NAT translates the UDP port β multiple clients possible
/* NAT-T packet structure */
[Outer IP: src=client_public_IP, dst=VPN_GW]
[UDP: sport=4500 dport=4500]
[Non-ESP Marker: 0x00000000 (4 bytes, distinguishes from IKE)]
[ESP header + encrypted payload]
/* Keepalive for NAT mappings */
/* NAT state tables expire idle UDP sessions (often 30-120s) */
/* IKEv2 NAT-T keepalive: single 0xFF byte every 20s on UDP 4500 */
dpd-timeout 30 # StrongSwan: dead peer detection
nat-keepalive 20 # keepalive interval
/* AH cannot work through NAT */
/* AH authenticates the outer IP header including src IP */
/* NAT changes src IP β AH authentication fails β AH is dead */
/* Always use ESP for NAT-compatible IPsec */
IPsec IN LINUX β XFRM AND STRONGSWAN
Linux IPsec Configuration
LINUX/* Linux IPsec: kernel handles ESP (via xfrm), StrongSwan handles IKEv2 */
/* /etc/swanctl/swanctl.conf β StrongSwan IKEv2 config */
connections {
site-to-site {
local_addrs = 203.0.113.1 # our gateway IP
remote_addrs = 198.51.100.1 # peer gateway IP
local {
auth = pubkey # certificate authentication
certs = gw1-cert.pem
id = "gw1.example.com"
}
remote {
auth = pubkey
id = "gw2.example.com"
}
children {
tunnel {
local_ts = 10.1.0.0/24 # traffic selectors
remote_ts = 10.2.0.0/24
esp_proposals = aes256gcm128-prfsha384-ecp384 # cipher suite
mode = tunnel
dpd_action = restart # restart tunnel on DPD failure
}
}
ike_proposals = aes256gcm16-prfsha384-ecp384
version = 2
dpd_delay = 30s
}
}
# Start IKEv2 negotiation
swanctl --load-all
swanctl --initiate --child tunnel
# Monitor
swanctl --list-sas # show active IKE and Child SAs
ip xfrm state # kernel ESP SAs (keys, algorithms, byte counts)
ip xfrm policy # kernel SPD (traffic selectors, actions)
ip xfrm monitor # real-time SA events
# Check tunnel traffic
tcpdump -i eth0 esp # ESP packets
tcpdump -i eth0 udp port 500 or udp port 4500 # IKE packets
/* XFRM offload to hardware (Intel QAT, Mellanox IPsec offload) */
# For your Mellanox ConnectX cards:
# ip xfrm state add ... offload dev eth0 dir in
# Pushes ESP encryption/decryption to NIC β reduces CPU overhead significantly
Site-to-Site IPsec VPN with StrongSwan
Objective: Configure a complete IKEv2/IPsec tunnel between two Linux VMs using StrongSwan. Capture and analyse IKEv2 and ESP traffic.
sudo apt install strongswan strongswan-swanctl. Generate certificates with StrongSwan's PKI: pki --gen --type ecdsa --size 256 > ca.der; pki --self --ca --in ca.der --dn "C=IN, O=Test, CN=Test CA" > ca.crt.sudo systemctl start strongswan. Initiate the tunnel: sudo swanctl --initiate --child tunnel. Verify: sudo swanctl --list-sas shows ESTABLISHED.sudo tcpdump -i eth0 -w /tmp/ike.pcap udp port 500 while initiating the tunnel. Open in Wireshark β decode as IKEv2. Identify: IKE_SA_INIT (two messages), IKE_AUTH (two messages). Note which packets are plaintext (IKE_SA_INIT) vs encrypted (IKE_AUTH).tcpdump -i eth0 esp. Verify: IP protocol is 50 (ESP). Check the kernel SA byte counters: ip xfrm state | grep bytes β should increase as traffic flows. Try pinging an address outside the traffic selector β verify it's NOT tunnelled.Manual XFRM SA β IPsec Without IKE
Objective: Configure IPsec manually using ip xfrm commands (no IKEv2). This shows exactly what IKEv2 does automatically and deepens understanding of the SA/SPD model.
ip xfrm state add. Choose a random 256-bit key: openssl rand -hex 32. Create the inbound and outbound SAs on each machine with matching SPIs and the same key. Use AES-256-GCM: aead "rfc4106(gcm(aes))" 0xKEY 128.ip xfrm policy add src 192.168.1.0/24 dst 192.168.2.0/24 dir out tmpl src GW1 dst GW2 proto esp mode tunnel. Add the reverse policy. Verify: ip xfrm state shows 2 SAs; ip xfrm policy shows 2 policies.ip xfrm state | grep -A5 "bytes" β counters should increment. Now intentionally set a wrong key on one side. Verify pings fail (authentication error) and find the error counter in ip xfrm state.M21 MASTERY CHECKLIST
- Know IPsec two planes: data plane (ESP encrypts traffic) and control plane (IKEv2 negotiates keys)
- Know ESP packet format: outer IP + ESP header (SPI + SN) + IV + encrypted payload + auth tag
- Know SPI: 32-bit identifier in ESP header tells receiver which SA (and key) to use for decryption
- Know anti-replay: sequence number + sliding window; duplicate or out-of-window SN β drop
- Know modern ESP cipher: AES-256-GCM-16 (AEAD β no separate auth needed)
- Know why AH is rarely used: covers outer IP header β incompatible with NAT
- Know SPD (Security Policy Database): selector (src/dst IP, port) β action (PROTECT/BYPASS/DISCARD)
- Know SAD (Security Association Database): one SA per direction; contains SPI, keys, algorithm, lifetime, anti-replay state
- Know SAs are unidirectional: one pair (inbound + outbound) per IPsec connection
- Know IKEv2 exchange: IKE_SA_INIT (DH key exchange, algorithm negotiation) β IKE_AUTH (authentication, Child SA creation)
- Know IKEv2 key derivation: SKEYSEED = PRF(Ni|Nr, DH_shared) β 7 IKE SA keys via PRF+
- Know IKEv2 authentication methods: RSA/ECDSA certificates, PSK, EAP
- Know transport mode: protects IP payload, preserves IP header β for host-to-host
- Know tunnel mode: encapsulates entire IP packet in new IP β for site-to-site VPN (gateway-to-gateway)
- Know why NAT breaks ESP: NAT changes IP headers, but ESP SPI is not a port number NAT can track
- Know NAT-T: encapsulate ESP inside UDP 4500; IKE detects NAT via NAT_DETECTION payloads
- Know Linux IPsec: kernel xfrm handles ESP; StrongSwan handles IKEv2; ip xfrm state/policy commands
- Know XFRM offload: push ESP encrypt/decrypt to NIC (Mellanox ConnectX) β saves CPU cycles
- Completed Lab 1: configured site-to-site IKEv2 with StrongSwan; captured and decoded IKEv2/ESP traffic
- Completed Lab 2: manually configured XFRM SAs; verified encryption, tested wrong-key failure
β When complete: Move to M22 - SSL/TLS Inspection and PKI Operations β the final Phase 5 module, bringing together TLS and PKI knowledge into operational NGFW inspection workflows.