TLS — TRANSPORT LAYER SECURITY
TLS in the Protocol Stack
OVERVIEWTLS (Transport Layer Security) is the protocol that makes HTTPS, SMTPS, IMAPS, FTPS, and many other "S" protocols secure. It sits between TCP and the application layer, providing: server authentication (via certificates), optional client authentication (mTLS), forward-secret key exchange (ECDHE), and authenticated encryption of all application data.
TLS 1.3 (RFC 8446, 2018) is the current standard. It eliminated all deprecated algorithms, reduced handshake latency from 2 RTTs to 1 RTT (0 RTT for resumption), and simplified the protocol significantly. Understanding TLS 1.3 is essential for NGFW because HTTPS carries the majority of internet traffic — and inspecting it requires understanding or terminating the TLS session.
TLS 1.3 HANDSHAKE — 1 RTT TO ENCRYPTED DATA
TLS 1.3 Full Handshake
HANDSHAKE/* TLS 1.3 Handshake — message flow */
CLIENT SERVER
│ │
│─── ClientHello ────────────────────────────→ │
│ • client_random (32 bytes) │
│ • supported_versions: [TLS 1.3, TLS 1.2] │
│ • supported_groups: [X25519, P-256, P-384] │
│ • key_share: X25519 ephemeral pubkey │
│ • signature_algs: [Ed25519, ECDSA P-256, │
│ RSA-PSS-SHA256] │
│ • server_name (SNI): "example.com" │
│ • psk_ke_modes (if resuming) │
│ │
│←── ServerHello ────────────────────────────── │
│ • server_random (32 bytes) │
│ • selected cipher: TLS_AES_256_GCM_SHA384 │
│ • key_share: X25519 server ephemeral pubkey │
│ • selected TLS version: 1.3 │
│ │
│ [ECDH shared secret computed by both sides] │
│ [Handshake keys derived via HKDF] │
│ [All subsequent messages are ENCRYPTED] │
│ │
│←── {EncryptedExtensions} ─────────────────── │
│ • ALPN: "h2" (HTTP/2 negotiated) │
│ • max_fragment_length, server_cert_type │
│ │
│←── {Certificate} ──────────────────────────── │
│ • Server's X.509 certificate chain │
│ │
│←── {CertificateVerify} ────────────────────── │
│ • Signature over transcript hash │
│ (proves server has private key) │
│ │
│←── {Finished} ─────────────────────────────── │
│ • HMAC over entire handshake transcript │
│ (proves handshake integrity) │
│ │
│─── {Finished} ──────────────────────────────→ │
│ • Client's HMAC over transcript │
│ │
│←→ {Application Data (AEAD encrypted)} ←────→ │
RTT count: 1 full RTT before application data can flow
(ClientHello → ServerHello+Cert+Finished → ClientFinished+AppData)💡 Key insight: In TLS 1.3, the server can send encrypted extensions, its certificate, and its Finished message all in one flight — before receiving anything from the client beyond ClientHello. This is possible because ECDHE allows the server to derive encryption keys immediately after seeing the client's key share. The client verifies the server's Finished HMAC to confirm the handshake wasn't tampered with.
TLS 1.3 KEY SCHEDULE — HOW KEYS ARE DERIVED
HKDF-Based Key Schedule
KEY SCHEDULE/* TLS 1.3 Key Schedule (RFC 8446 §7.1) */ /* All derivations use HKDF with the negotiated hash (SHA-256 or SHA-384) */ 0 (Early Secret) ├─ Early Traffic Keys (for 0-RTT data, if resuming with PSK) │ ↓ HKDF-Extract(PSK or 0-bytes, Early Secret) Handshake Secret ├─ client_handshake_traffic_secret │ → client_handshake_key (AES key for client→server during handshake) │ → client_handshake_iv (nonce base) ├─ server_handshake_traffic_secret │ → server_handshake_key (AES key for server→client during handshake) │ → server_handshake_iv │ ↓ HKDF-Extract(ECDHE shared secret, Handshake Secret) Master Secret ├─ client_application_traffic_secret_0 │ → client_write_key (AES key for client→server application data) │ → client_write_iv ├─ server_application_traffic_secret_0 │ → server_write_key (AES key for server→client application data) │ → server_write_iv ├─ exporter_master_secret (for channel binding) └─ resumption_master_secret (for session tickets / PSK resumption) /* Nonce construction — prevents nonce reuse */ /* For each record: nonce = write_iv XOR sequence_number (64-bit, left-padded) */ /* Sequence number increments with each record → unique nonce per record */ /* Key update (post-handshake) */ /* Either side can send KeyUpdate message → derive new traffic keys */ new_secret = HKDF-Expand-Label(current_secret, "traffic upd", "", hash_len) /* Forward secrecy within a session: old keys deleted, new keys derived */
TLS RECORD PROTOCOL — WIRE FORMAT
TLS Record Structure
RECORD FORMAT/* TLS Record format (all TLS versions) */ +------------------+------------------+------------------+ | Content Type (1B)| Version (2B) | Length (2B) | +------------------+------------------+------------------+ | Payload (up to 16384 bytes) | +--------------------------------------------------------+ Content Types: 20 = change_cipher_spec (legacy, sent for TLS 1.2 compat) 21 = alert (error notification) 22 = handshake (ClientHello, ServerHello, Certificate, etc.) 23 = application_data (encrypted payload) Version field in TLS 1.3: Outer record: 0x0303 (TLS 1.2) — for middlebox compatibility Inner content_type (inside AEAD ciphertext): real type /* TLS 1.3 Application Data record layout */ +------+--------+--------+----------------------------------+----------+ | 0x17 | 0x0303 | length | Encrypted(application_data + | auth_tag | | 23 | TLS1.2 | 2B | inner_content_type) — AEAD | 16B | +------+--------+--------+----------------------------------+----------+ /* AEAD inputs for encrypting a record */ Plaintext: application_data bytes + inner_content_type (1 byte at end) AAD: TLS record header (5 bytes: type + version + length) Key: write_key (from key schedule) Nonce: write_iv XOR (seq_number as 12-byte big-endian) /* Maximum record size */ 16384 bytes (2^14) of plaintext per record + 256 bytes of padding (optional, hides true record size) + 16 bytes auth tag = up to 16657 bytes per record /* Alert record format (2 bytes inside TLS record) */ Level: 1=warning, 2=fatal Description: 0=close_notify, 10=unexpected_message, 20=bad_record_mac, 42=bad_certificate, 48=unknown_ca, 70=protocol_version, 80=internal_error, 100=no_renegotiation, 112=unrecognized_name (SNI) /* Wireshark TLS decryption */ SSLKEYLOGFILE=/tmp/keys.log curl https://example.com # In Wireshark: Edit → Preferences → TLS → Master-Secret log file → /tmp/keys.log # Wireshark will decrypt all TLS records and show plaintext handshake + data
TLS 1.2 vs TLS 1.3 — KEY DIFFERENCES
What Changed from TLS 1.2 to 1.3
COMPARISON| Feature | TLS 1.2 | TLS 1.3 |
|---|---|---|
| Handshake RTTs | 2 RTTs minimum | 1 RTT (0 RTT for resumption) |
| Forward secrecy | Optional (ECDHE or static RSA) | Mandatory (ECDHE always) |
| Cipher suites | Hundreds incl. RC4, 3DES, NULL, anon | 5 only, all AEAD |
| Key exchange | RSA, ECDHE, DHE, ECDH (static) | ECDHE, DHE (finite field) only |
| Certificate encryption | Plaintext in handshake (visible to network) | Encrypted (after server key derived) |
| Renegotiation | Allowed (caused vulnerabilities) | Removed entirely |
| Compression | Optional (CRIME attack) | Removed |
| MAC-then-Encrypt | Used in CBC mode (BEAST, POODLE) | Removed — AEAD only |
| Session IDs | Server stores session state | Stateless session tickets only |
| SNI encryption | No (SNI in cleartext ClientHello) | ECH (Encrypted ClientHello) — draft |
| Removed from TLS 1.3 | — | RSA key exchange, CBC, RC4, 3DES, MD5, SHA-1, renegotiation, compression, DSA |
TLS 1.3 Cipher Suites — Only 5
TLS_AES_128_GCM_SHA256 (most common, high performance) TLS_AES_256_GCM_SHA384 (higher security) TLS_CHACHA20_POLY1305_SHA256 (mobile/ARM performance) TLS_AES_128_CCM_SHA256 (constrained IoT) TLS_AES_128_CCM_8_SHA256 (constrained IoT, shorter tag) # Note: no key exchange or auth in TLS 1.3 cipher suites # Key exchange is always ECDHE (negotiated separately in supported_groups) # Authentication is always certificate-based (negotiated in signature_algs)
0-RTT AND SESSION RESUMPTION
PSK and 0-RTT Early Data
0-RTT/* TLS 1.3 session resumption via PSK (Pre-Shared Key) */ After a successful TLS 1.3 handshake, the server sends a NewSessionTicket: - Contains a PSK (pre-shared key) encrypted with a server-only ticket key - Includes a ticket_lifetime (e.g., 7 days) - Client stores this opaque blob On reconnect, client includes the PSK in ClientHello: - pre_shared_key extension: ticket blob - psk_key_exchange_modes: psk_dhe_ke (PSK + ephemeral DH — recommended) or psk_ke (PSK only — no forward secrecy!) - early_data extension: client wants to send 0-RTT data /* 0-RTT early data — zero round-trip cost */ Standard 1-RTT: ClientHello → ServerHello+Cert+Finished → {AppData} 0-RTT resumption: ClientHello + {EarlyData} → ServerHello → {AppData} ↑ Application data piggybacks on ClientHello! /* 0-RTT security limitations */ Replay attack risk: Attacker captures ClientHello+EarlyData, replays it to server. Server has no way to distinguish replay from original! Mitigations: 1. Only use 0-RTT for idempotent requests (GET, not POST) 2. Server-side replay detection (store nonces, use anti-replay window) 3. Accept risk for non-sensitive use (performance vs security tradeoff) 0-RTT does NOT provide forward secrecy for early data: If PSK ticket key is compromised → early data decryptable Post-handshake application data DOES have forward secrecy (ECDHE) /* NGFW considerations for 0-RTT */ # 0-RTT early data is encrypted with the early_traffic_key # Without the PSK or TLS session keys, NGFW cannot decrypt 0-RTT # SSL inspection proxy must handle 0-RTT specially # Many NGFW products simply reject 0-RTT by not returning early_data in EncryptedExtensions
mTLS — MUTUAL AUTHENTICATION
Client Certificate Authentication
mTLS/* Standard TLS: only server is authenticated */ /* mTLS (mutual TLS): both server AND client present certificates */ /* mTLS handshake additions */ After sending Certificate + CertificateVerify + Finished, server sends: CertificateRequest: list of acceptable CA DNs for client certificates Client responds with: Certificate: client's X.509 certificate (or empty if none available) CertificateVerify: signature over handshake transcript with client private key Finished: as normal /* Use cases for mTLS */ Service mesh (Istio, Linkerd): all microservices authenticate each other Zero-trust networks: every connection requires client cert (device identity) API security: client apps authenticate with cert instead of API keys IoT devices: device certificates for mutual auth to backend NGFW policy: require client cert for access to sensitive internal resources /* Configure nginx for mTLS */ ssl_client_certificate /etc/ssl/ca.pem; # CA that signed client certs ssl_verify_client on; # require client cert ssl_verify_depth 2; # allow one intermediate CA /* OpenSSL mTLS server in C */ SSL_CTX *ctx = SSL_CTX_new(TLS_server_method()); SSL_CTX_load_verify_locations(ctx, "ca.pem", NULL); SSL_CTX_use_certificate_file(ctx, "server.pem", SSL_FILETYPE_PEM); SSL_CTX_use_PrivateKey_file(ctx, "server.key", SSL_FILETYPE_PEM); SSL_CTX_set_verify(ctx, SSL_VERIFY_PEER | SSL_VERIFY_FAIL_IF_NO_PEER_CERT, NULL); /* After SSL_accept(): inspect client certificate */ X509 *client_cert = SSL_get_peer_certificate(ssl); X509_NAME *subj = X509_get_subject_name(client_cert); char cn[256]; X509_NAME_get_text_by_NID(subj, NID_commonName, cn, sizeof(cn)); printf("Client cert CN: %s\n", cn); X509_free(client_cert);
SSL INSPECTION — NGFW TLS INTERCEPTION
How SSL/TLS Inspection Works
SSL INSPECTION/* SSL inspection (TLS MITM proxy) — the NGFW's view */ Normal TLS: Client ←── TLS ──→ Server Client trusts server's cert from a real CA NGFW sees: encrypted bytes → cannot inspect content SSL inspection: Client ←── TLS ──→ NGFW ←── TLS ──→ Server NGFW-Server leg: NGFW establishes TLS to the real server Validates server's real certificate NGFW has the session keys → can decrypt/inspect server responses Client-NGFW leg: NGFW generates a certificate for the domain Signs it with the corporate CA (deployed to all managed devices) Client validates against corporate CA → succeeds NGFW has these session keys too → can decrypt/inspect client requests /* What SSL inspection reveals */ Full HTTP URL path (not just hostname) All HTTP request/response headers Request bodies (POST data, form submissions) Response bodies (file downloads → malware scanning) WebSocket data gRPC payloads /* What SSL inspection breaks */ Certificate pinning: apps that pin to specific certs (Twitter app, many banking apps) HPKP (deprecated): HTTP Public Key Pinning Client certificates (mTLS): NGFW must handle client cert forwarding QUIC/HTTP3: QUIC encrypts more aggressively, harder to intercept /* NGFW SSL inspection bypass list (do NOT inspect) */ Banking domains (privacy regulation) Healthcare portals (HIPAA) Legal/HR applications (attorney-client privilege) Apps known to use certificate pinning Internal PKI-protected services (use different trust chain) /* Implementing basic TLS termination in C with OpenSSL */ SSL_CTX *server_ctx = SSL_CTX_new(TLS_server_method()); /* Load your generated cert for the target domain */ SSL_CTX_use_certificate(server_ctx, generated_cert); SSL_CTX_use_PrivateKey(server_ctx, generated_key); SSL_CTX *client_ctx = SSL_CTX_new(TLS_client_method()); /* Connect to real server */ SSL *client_ssl = SSL_new(client_ctx); SSL_set_fd(client_ssl, server_socket_fd); SSL_connect(client_ssl); /* Verify real server cert */ X509 *real_cert = SSL_get_peer_certificate(client_ssl); /* Extract domain, generate matching cert for client, serve it */
⚠️ ECH (Encrypted ClientHello) — in development for TLS 1.3 — will encrypt the SNI extension and other ClientHello fields, preventing NGFW from seeing the destination hostname without decrypting the entire TLS session. This fundamentally challenges SNI-based filtering and makes SSL inspection the only way to identify destinations. Watch RFC drafts for ECH deployment timeline.
TLS 1.3 Handshake Dissection
Objective: Capture and fully decode a TLS 1.3 handshake using Wireshark with TLS key logging.
export SSLKEYLOGFILE=/tmp/tls_keys.log. Make a TLS 1.3 connection: curl --tlsv1.3 -v https://cloudflare.com 2>&1 | head -40. Capture simultaneously: sudo tcpdump -i eth0 -w /tmp/tls.pcap host cloudflare.com. Open pcap in Wireshark with key log configured.Build a TLS Client and Server with OpenSSL
Objective: Implement a TLS 1.3 echo server and client in C using OpenSSL. Extend with mTLS client authentication.
openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:P-256 -keyout server.key -out server.crt -days 365 -nodes -subj "/CN=localhost". Write a minimal TLS server: socket → SSL_CTX_new(TLS_server_method()) → SSL_CTX_use_certificate/PrivateKey → accept loop → SSL_accept → SSL_read/write.SSL_CTX_set_min_proto_version(ctx, TLS1_3_VERSION). Verify with: openssl s_client -connect localhost:8443 -tls1_3. Confirm the cipher suite selected. Check the certificate presented: openssl s_client -connect localhost:8443 2>/dev/null | openssl x509 -noout -text.openssl s_client -connect localhost:8443 -cert client.crt -key client.key. Verify the server prints the client cert's CN. Try without a cert — verify the server rejects with "alert handshake failure".SSL_get1_session(ssl). On the next connection, restore it: SSL_set_session(ssl, session). Capture both connections in Wireshark and verify the second one is shorter (no Certificate/CertificateVerify).M20 MASTERY CHECKLIST
- Know TLS provides: server auth (cert), optional client auth (mTLS), forward-secret key exchange, AEAD encryption
- Know TLS 1.3 handshake messages in order: ClientHello → ServerHello → {EncryptedExtensions, Certificate, CertificateVerify, Finished} → {Finished} → application data
- Know ClientHello key fields: client_random, supported_versions, key_share (X25519 pubkey), signature_algorithms, SNI, ALPN
- Know that TLS 1.3 takes 1 RTT (server sends cert + finished in same flight as ServerHello)
- Know TLS 1.3 key schedule: ECDHE shared secret + HKDF → handshake keys → master secret → application keys
- Know TLS record format: 5-byte header (content_type + version + length) + AEAD ciphertext + auth_tag
- Know TLS 1.3 nonce: write_iv XOR sequence_number (64-bit, prevents nonce reuse)
- Know the 5 TLS 1.3 cipher suites (all AEAD); know that key exchange and auth are negotiated separately
- Know what TLS 1.3 removed vs 1.2: static RSA, CBC, RC4, 3DES, compression, renegotiation, session IDs
- Know forward secrecy difference: TLS 1.3 always ECDHE (FS); TLS 1.2 could use static RSA (no FS)
- Know 0-RTT: PSK session ticket, early data sent with ClientHello, replay attack risk, only for idempotent requests
- Know 0-RTT does NOT have forward secrecy for early data
- Know mTLS: server sends CertificateRequest; client responds with Certificate + CertificateVerify; both sides authenticated
- Know mTLS use cases: service mesh, zero-trust, IoT device auth, API security
- Know SSL inspection architecture: NGFW as MITM; two separate TLS sessions; corporate CA signed cert for client
- Know what SSL inspection reveals: full URL, headers, request/response bodies, file downloads
- Know SSL inspection exclusions: banking, healthcare, apps with certificate pinning
- Know ECH (Encrypted ClientHello) as the upcoming challenge to SNI-based NGFW filtering
- Completed Lab 1: dissected TLS 1.3 handshake in Wireshark with key log; identified all extensions and messages
- Completed Lab 2: built TLS 1.3 server+client in C; implemented mTLS; verified session resumption
✅ When complete: Move to M21 - IPsec and IKEv2 — the VPN protocol stack used for site-to-site and remote access in enterprise networks, and the encryption layer for many NGFW deployments.