M22 - SSL Inspection and PKI Operations

NETWORKING MASTERY · PHASE 5 · MODULE 22 · WEEK 20 · PHASE 5 FINAL

🔍 SSL Inspection and PKI Operations

TLS MITM architecture · Certificate generation · CA management · Bypass policy · Certificate pinning · ECH

Advanced Prerequisite: M20 TLS NGFW Core Capability 2 Labs

SSL INSPECTION — THE NGFW'S WINDOW INTO ENCRYPTED TRAFFIC

🔍

Why SSL Inspection Is Essential for NGFW

OVERVIEW

Over 95% of internet traffic is now HTTPS. Without SSL inspection, an NGFW is essentially blind to the majority of traffic — it can see destination IPs and SNI hostnames, but cannot inspect URLs, request/response content, or file transfers. Malware increasingly uses HTTPS for C2 communication precisely because simple firewalls cannot inspect it.

SSL inspection (also called TLS inspection, SSL/TLS deep inspection, SSL bump) allows the NGFW to decrypt, inspect, and re-encrypt HTTPS traffic in real time. It is the single capability that separates a modern NGFW from a basic firewall.

Without SSL Inspection	With SSL Inspection
See: destination IP, SNI hostname	See: full URL, all headers, request/response body
URL filtering: by hostname only	URL filtering: exact path and query string
Malware scanning: impossible	Malware scanning: scan file downloads in real time
DLP: impossible for HTTPS data	DLP: inspect POST bodies, detect sensitive data exfiltration
App identification: by certificate only	App identification: by HTTP headers, content, API patterns

MITM PROXY ARCHITECTURE — HOW SSL INSPECTION WORKS

🏗️

Full SSL Inspection Flow

ARCHITECTURE

/* SSL Inspection — two separate TLS sessions */

STEP 1: Client initiates connection
  Client → NGFW: TCP SYN to server IP
  NGFW intercepts (transparent proxy mode — no client config needed)

STEP 2: NGFW connects to real server
  NGFW → Real Server: TCP connect, TLS ClientHello
  NGFW validates server certificate (chain, expiry, revocation)
  NGFW completes TLS handshake with real server
  NGFW now has session keys → can decrypt all server responses

STEP 3: NGFW generates certificate for client
  NGFW reads SNI from client's ClientHello (or from server cert)
  NGFW dynamically generates a certificate:
    Subject: CN=<target domain> (e.g., CN=accounts.google.com)
    SAN: original server's SAN list (preserved for correctness)
    Issuer: NGFW's internal CA certificate
    Validity: short (24–72 hours, not original cert lifetime)
    Key: new ephemeral key (never the real server's key)
  NGFW signs cert with its private CA key

STEP 4: NGFW completes TLS with client
  NGFW → Client: TLS ServerHello with generated certificate
  Client validates: checks chain → NGFW cert → NGFW CA → trust store
  Client trusts because NGFW CA was deployed to device trust store
  Client completes TLS handshake with NGFW

STEP 5: Inspection
  Every HTTP request from client → NGFW decrypts → inspects → forwards to server
  Every HTTP response from server → NGFW decrypts → inspects → forwards to client
  Inspection can: scan for malware, check URLs, apply DLP, log everything

/* Transparency — is the client aware? */
Technical: YES — the certificate presented is signed by NGFW CA, not real CA.
         The certificate serial/key differ from the real server's cert.
         A careful user or security-aware app CAN detect this.
User-visible: Usually NOT — URL bar still shows padlock, correct hostname.
Legal: Must inform users that inspection is occurring (employee policy, consent).

/* Performance considerations */
Each SSL inspection = 2× TLS sessions: NGFW-client + NGFW-server
CPU cost: 2× TLS handshakes per connection
Latency: +5-20ms per new connection (TLS handshake cost)
Session resumption: NGFW must manage its own session cache client-side
                    AND use session resumption server-side

CERTIFICATE GENERATION — CREATING CERTIFICATES ON THE FLY

📜

Dynamic Certificate Generation with OpenSSL

CERT GEN

/* Dynamic certificate generation — runs for every new HTTPS connection */

int generate_inspection_cert(
        const char *hostname,        /* from SNI or server cert CN */
        X509 *real_server_cert,      /* for SAN list preservation */
        X509 *ca_cert,               /* NGFW CA certificate */
        EVP_PKEY *ca_key,            /* NGFW CA private key */
        X509 **out_cert,
        EVP_PKEY **out_key)
{
    /* 1. Generate ephemeral key (EC P-256 for speed) */
    EVP_PKEY *pkey = EVP_EC_gen("P-256");

    /* 2. Create certificate */
    X509 *cert = X509_new();
    X509_set_version(cert, 2);      /* version 3 */
    
    /* Random serial number (each cert needs unique serial) */
    BIGNUM *serial = BN_new();
    BN_rand(serial, 64, 0, 0);
    BN_to_ASN1_INTEGER(serial, X509_get_serialNumber(cert));
    
    /* Short validity — 24 hours */
    X509_gmtime_adj(X509_getm_notBefore(cert), 0);
    X509_gmtime_adj(X509_getm_notAfter(cert), 86400);
    
    /* Subject: CN = hostname */
    X509_NAME *name = X509_NAME_new();
    X509_NAME_add_entry_by_txt(name, "CN", MBSTRING_ASC,
        (unsigned char *)hostname, -1, -1, 0);
    X509_set_subject_name(cert, name);
    
    /* Issuer: our CA */
    X509_set_issuer_name(cert, X509_get_subject_name(ca_cert));
    
    /* Subject Alt Names — copy from real server cert */
    copy_san_extension(cert, real_server_cert);
    
    /* Key usage */
    add_key_usage(cert, KU_DIGITAL_SIGNATURE | KU_KEY_ENCIPHERMENT);
    add_ext_key_usage(cert, NID_server_auth);
    
    /* Attach public key */
    X509_set_pubkey(cert, pkey);
    
    /* Sign with CA key */
    X509_sign(cert, ca_key, EVP_sha256());
    
    *out_cert = cert;
    *out_key  = pkey;
    return 0;
}

/* Performance optimisation: certificate caching */
/* Generating a new cert takes ~1ms (EC key gen + sign) */
/* Cache by hostname: hash(SNI) → (cert, key) */
/* Cache lifetime: 1 hour (shorter than real cert validity) */
/* Cache size: 10,000 entries covers most enterprise browsing patterns */

CA MANAGEMENT — BUILDING AND OPERATING AN INSPECTION CA

🏛️

NGFW CA Lifecycle and Deployment

CA MANAGEMENT

/* Creating the NGFW inspection CA */

# Generate 4096-bit RSA CA key (CA key compromise = all inspection certs compromised)
openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:4096 -out ngfw-ca.key

# Create self-signed CA certificate (10 year validity)
openssl req -new -x509 -key ngfw-ca.key -out ngfw-ca.crt -days 3650 \
  -subj "/C=IN/O=Jio Platforms/OU=Network Security/CN=Jio NGFW Inspection CA" \
  -extensions v3_ca

# Verify CA cert
openssl x509 -in ngfw-ca.crt -text -noout | grep -A5 "Basic Constraints"
# Must show: CA:TRUE

/* Security hardening of CA key */
# Store CA private key in HSM (Hardware Security Module) if possible
# If software: encrypt with strong passphrase, restrict file permissions
chmod 400 ngfw-ca.key
# Or use PKCS#11 engine to access HSM from OpenSSL:
# openssl engine pkcs11 -pre MODULE_PATH:/usr/lib/softhsm/libsofthsm2.so

/* CA distribution — push to all managed endpoints */

Windows (Group Policy):
  Computer Configuration → Policies → Windows Settings → Public Key Policies
  → Trusted Root Certification Authorities → Import ngfw-ca.crt

Linux (system-wide):
  sudo cp ngfw-ca.crt /usr/local/share/ca-certificates/ngfw-ca.crt
  sudo update-ca-certificates

macOS (MDM):
  Deploy via MDM profile → Certificate payload → Always Trust

Android/iOS (MDM):
  Deploy via MDM → Certificate profile → Install as trusted CA

/* CA rotation — periodic key replacement */
# CA key should be rotated every 2-3 years
# Process: generate new CA key → deploy new CA cert → phase out old
# Overlap period: both CAs active simultaneously during rollout
# Use sub-CA: root CA signs a sub-CA cert used for signing; rotate sub-CA annually

/* Sub-CA architecture (recommended for enterprise) */
Offline Root CA (air-gapped, 4096-bit RSA, 20-year cert)
  └── Online Sub-CA (HSM-backed, ECDSA P-384, 5-year cert)
        └── NGFW dynamically signs per-hostname certs

BYPASS POLICY — WHEN NOT TO INSPECT

🚦

SSL Inspection Bypass Categories and Rules

BYPASS

Category	Why Bypass	How to Identify	Risk of Inspecting
Banking and finance	Privacy regulations (GDPR, RBI guidelines); credentials in transit	URL category: financial; TLD: .bank	Legal liability; user trust erosion
Healthcare portals	Protected health information (PHI); HIPAA equivalent	URL category: health; hospital domains	Regulatory violation
HR and payroll	Employee personal data; company confidentiality	URL category: HR; known HR SaaS domains	Privacy regulations
Personal email	Employee personal communications; legal considerations	URL: gmail.com, outlook.com personal access	Privacy expectations
Certificate-pinned apps	App will fail — cannot accept NGFW CA cert	Known list: Twitter app, many banking apps, Signal	App breakage
Internal PKI services	Use private CA already trusted; NGFW CA not in chain	Internal domains, RFC 1918 destinations	Certificate chain errors
Update servers	Signed packages — inspection adds latency; code signing	URL: windowsupdate.com, apt.canonical.com	Update failures possible
Legal counsel / IR tools	Attorney-client privilege	Known legal SaaS, law firm domains	Privileged communication exposure

/* Bypass decision flowchart in NGFW */
For each new TLS connection:

1. Is destination in bypass-by-IP list? → BYPASS
2. Is destination in bypass-by-domain list? → BYPASS
3. Is URL category in bypass-category list? → BYPASS
4. Does server cert have a CT SCT? (all trusted certs should) → if missing: BLOCK
5. Is server cert EV (Extended Validation)? → optional bypass
6. Does server cert use domain-validated DV from known CA? → INSPECT
7. Is client in inspection-exempt group? → BYPASS
8. Default: INSPECT

/* Squeezing more visibility without full inspection */
/* Even without decryption, TLS metadata reveals a lot: */

JA3/JA3S fingerprint:
  Hash of TLS ClientHello parameters (cipher suites, extensions, curves)
  Identifies the TLS client library (Chrome, Firefox, curl, Python, malware)
  Malware often has distinctive JA3 hashes
  JA3 = MD5(SSLVersion, Ciphers, Extensions, EllipticCurves, EllipticCurveFormats)

Certificate analysis:
  Self-signed cert for common domains → likely malware C2
  Cert issued <1 hour ago → suspicious (malware uses short-lived Let's Encrypt certs)
  Cert from unknown CA → block or alert

QUIC/HTTP3 detection:
  Alt-Svc header in HTTP response suggests QUIC support
  NGFW must block UDP 443 to force HTTP/2 (for inspection)
  Or deploy QUIC-capable inspection proxy

CERTIFICATE PINNING — THE INSPECTION ADVERSARY

📌

How Certificate Pinning Works and Breaks Inspection

PINNING

/* Certificate pinning */
App embeds expected certificate or public key directly in its binary.
On each TLS connection: compares server cert to pinned value.
If mismatch → connection refused (even if chain validates correctly).

Types:
  Leaf pinning:   pin the exact leaf certificate
  Public key pin: pin the subject public key (survives cert renewal)
  CA pin:         pin a specific CA — accepts any cert from that CA only
  SPKI pin:       Subject Public Key Info hash (RFC 7469 HPKP — now deprecated)

Result with SSL inspection:
  NGFW presents forged cert signed by NGFW CA, with different key.
  Pinned app: "Expected key X, got key Y" → TLS abort.
  App logs: "Certificate validation failed" / "SSL error" / silent failure.

/* Detection: how to identify pinned apps */
# Symptom: app works without SSL inspection, fails with inspection
# Test: enable inspection → app fails; disable → app works
# Tool: mitmproxy bypass detection log
# Android: frida-based SSLUnpinning script (for testing/research)

/* NGFW handling strategies */

Option 1: Bypass list (most practical)
  Add known-pinning apps/domains to bypass list.
  Risk: bypass allows uninspected traffic.
  Examples: Twitter app, Facebook app, many banking apps, Signal, WhatsApp.

Option 2: MDM enforcement
  Deploy MDM policy that disables cert pinning override.
  Only works for MDM-managed devices — not personal devices.

Option 3: Application block
  If app uses pinning and you can't inspect it → block the app entirely.
  Heavy-handed but used for high-security environments.

Option 4: Zero-trust network access (ZTNA)
  Replace SSL inspection with ZTNA agent on endpoint.
  Agent inspects traffic before it leaves the device (no MITM needed).
  Increasingly the modern alternative to network-based SSL inspection.

/* Known apps using certificate pinning (partial list) */
Strong pinning: WhatsApp, Signal, most banking apps, Twitter native app
Partial pinning: Chrome (HSTS preload), Firefox (for mozilla.org)
Historical: Google Chrome (Chrome pins Symantec certs — removed 2018)

ECH AND FUTURE CHALLENGES TO SSL INSPECTION

🔮

Encrypted ClientHello — The Coming Inspection Challenge

ECH

/* Current TLS: SNI is cleartext */
ClientHello contains:
  server_name (SNI) extension: "accounts.google.com"  ← VISIBLE TO NGFW

This allows:
  - NGFW URL filtering by hostname
  - ISP traffic monitoring
  - Country-level censorship
  - Passive TLS fingerprinting

/* ECH — Encrypted ClientHello (TLS 1.3 extension, draft RFC) */
New model:
  Outer ClientHello: SNI = "cloudflare.com" (the CDN/front)  ← visible
  Inner ClientHello: SNI = "accounts.google.com"              ← ENCRYPTED

How:
  Server publishes an ECH public key in DNS (HTTPS record type 65)
  Client fetches ECH key via DoH (bypassing ISP DNS)
  Client encrypts inner ClientHello with ECH public key
  Only the target server (with ECH private key) can decrypt inner ClientHello

/* ECH impact on NGFW */
Without SSL inspection: NGFW sees only outer SNI (CDN domain, not real destination)
                        URL filtering by hostname becomes impossible for ECH sites
                        Malware can hide its C2 domain behind Cloudflare/CDN with ECH

With SSL inspection: NGFW terminates TLS before ECH → can still decrypt everything
                     But NGFW must now connect to CDN as outer client,
                     then somehow route to real backend — complex for transparent proxy

/* Current status (2025) */
ECH is in RFC draft stage (draft-ietf-tls-esni)
Cloudflare, Fastly deploying ECH for their customers
Firefox and Chrome have ECH support behind flags or partial rollout
Major browser adoption + CDN deployment = ECH becomes common by 2026-2027

/* NGFW strategic responses to ECH */
1. DNS-based filtering:
   Block DNS HTTPS records (type 65) → ECH key not available → fallback to plain SNI
   Risk: browsers may treat as network error and retry with DoH

2. TLS fingerprint-based detection:
   JA4+ fingerprints identify TLS libraries even without SNI
   Combine with IP reputation, QUIC fingerprinting

3. Endpoint-based inspection (ZTNA agent):
   Agent on endpoint can inspect before encryption
   Avoids the network MITM problem entirely

4. DNS-over-HTTPS interception:
   Intercept all DoH queries (force internal resolver)
   Prevents ECH key retrieval for controlled environments

/* Takeaway for NGFW engineers */
# The long-term trend is toward more encryption, not less
# ECH + QUIC + DoH create a world where network-based inspection weakens
# Future NGFW: endpoint agent + cloud policy + AI-based behavioural detection
# Network inspection remains viable for on-premise managed environments

LAB 1

Build a Working SSL Inspection Proxy

Objective: Build a working TLS MITM proxy using mitmproxy and your own CA. Observe inspection capabilities and test bypass scenarios.

Install mitmproxy: pip install mitmproxy. Run in transparent mode: mitmproxy --mode transparent --showhost. Configure iptables to redirect traffic: iptables -t nat -A OUTPUT -p tcp --dport 443 -j REDIRECT --to-port 8080. Browse to HTTPS sites and observe mitmproxy decrypting all traffic.

Install mitmproxy's CA certificate: it auto-generates at ~/.mitmproxy/mitmproxy-ca.pem. Import to your browser: Settings → Certificates → Import. Verify HTTPS sites now show the mitmproxy CA in the certificate chain instead of the real CA. Find a site and compare the real cert vs the generated cert.

Test certificate pinning: try accessing an app or site that uses pinning (Twitter's web app, some banking sites). Observe the error. Document which applications fail. Use curl as a test: curl --proxy http://127.0.0.1:8080 https://github.com (should work) vs curl --pinnedpubkey sha256//... https://github.com (should fail if key changes).

Write a mitmproxy addon (Python script) that: logs the full URL, request headers, and response status for every request, saves POST body content to a file (DLP simulation), and blocks any response body containing the string "password" (test with a custom HTTP server). This demonstrates what real NGFW SSL inspection engines do.

LAB 2

Enterprise CA Setup and Certificate Management

Objective: Build a two-tier CA hierarchy (root CA + sub-CA) using OpenSSL. Generate inspection certificates dynamically and verify the chain.

Create an offline root CA: generate a 4096-bit RSA key, create a self-signed certificate with CA:TRUE and pathlen:1 (allows one sub-CA). Mark this as the root — in production it would be kept offline.

openssl req -x509 -newkey rsa:4096 -keyout root-ca.key -out root-ca.crt -days 7300 -nodes -extensions v3_ca -subj "/CN=Test Root CA"

Create the NGFW sub-CA: generate an EC P-384 key, create a CSR, sign it with the root CA. The sub-CA gets: CA:TRUE, pathlen:0 (no further sub-CAs), shorter validity (5 years). This is what the NGFW uses for daily signing operations.

Write a Python script that generates an inspection certificate for a given hostname in under 1ms (target). The cert must: copy SANs from the real server cert (fetch real cert with socket+ssl), use the sub-CA for signing, cache results by hostname. Benchmark: time 1000 certificate generations.

Verify the chain: openssl verify -CAfile root-ca.crt -untrusted sub-ca.crt generated-cert.crt. Should show "OK". Also verify with a browser: import root-ca.crt as trusted root, then use nginx with the generated cert. Browser should show a padlock with the custom CA chain.

M22 MASTERY CHECKLIST

Know why SSL inspection is essential: 95%+ traffic is HTTPS; without it NGFW is blind to URLs, content, malware in downloads
Know SSL inspection reveals vs not: reveals full URL/headers/body; does NOT reveal source code or keys of real server
Know the MITM proxy architecture: two separate TLS sessions (NGFW↔client and NGFW↔server)
Know the 5-step inspection flow: intercept → connect to real server → validate real cert → generate forged cert → serve to client → inspect bidirectionally
Know why clients trust forged certs: corporate CA is pre-deployed to managed device trust stores
Know what generated certs must contain: hostname in CN/SAN, CA:FALSE, short validity, copied SAN list, signed by NGFW sub-CA
Know certificate caching: generate once per hostname, cache 1 hour — ~1ms/cert without cache, sub-µs with cache
Know two-tier CA hierarchy: offline root CA + online NGFW sub-CA (root CA signs sub-CA; sub-CA signs inspection certs)
Know CA distribution methods: Windows GPO, Linux update-ca-certificates, MDM for mobile
Know SSL inspection bypass categories: banking, healthcare, HR/payroll, personal email, cert-pinned apps, internal PKI, update servers
Know TLS metadata visible without decryption: SNI, JA3/JA3S fingerprint, certificate subject/issuer, cipher suites, ALPN
Know JA3: hash of TLS ClientHello parameters; identifies TLS client library; malware has distinctive JA3
Know certificate pinning: app embeds expected cert/key; NGFW forged cert → mismatch → connection refused
Know three pinning types: leaf cert pin, public key pin, CA pin
Know NGFW strategies for pinning: bypass list (practical), MDM disable, block app, ZTNA agent
Know ECH (Encrypted ClientHello): encrypts SNI in ClientHello; only CDN/server can decrypt; blinds NGFW to real destination
Know ECH NGFW responses: block DNS HTTPS records, TLS fingerprinting, ZTNA endpoint agent, DoH interception
Completed Lab 1: ran mitmproxy SSL inspection proxy; tested cert pinning failure; wrote DLP addon
Completed Lab 2: built two-tier CA hierarchy; generated inspection certs programmatically; benchmarked cert generation

🎉 Phase 5 Complete — Security Protocols

You have completed all 4 modules of Phase 5: Cryptography Foundations (M19), TLS Internals (M20), IPsec and IKEv2 (M21), and SSL Inspection and PKI Operations (M22). You now have a thorough understanding of the cryptographic protocols underpinning modern network security — the same protocols your NGFW must implement, inspect, and in some cases circumvent. Move to Phase 6 — NGFW Development, the capstone phase.

← M21 IPsec 🗺️ Roadmap Next: M23 - Conntrack and NAT →