M08 - HTTP/1.1, HTTP/2, HTTP/3 and QUIC

NETWORKING MASTERY · PHASE 2 · MODULE 08 · WEEK 6

🌍 HTTP/1.1, HTTP/2, HTTP/3 and QUIC

Request/response · Methods · Headers · Status codes · Pipelining · Multiplexing · QUIC frames · NGFW inspection

Intermediate Prerequisite: M05 TCP, M07 DNS RFC 9110 · RFC 9113 · RFC 9114 Dominant Web Protocol 3 Labs

FROM HTTP/0.9 TO HTTP/3 — 30 YEARS OF EVOLUTION

📜

Why HTTP Has Evolved So Dramatically

OVERVIEW

HTTP (HyperText Transfer Protocol) is the application protocol that powers the web. Every browser request, every API call, every web service interaction uses HTTP. It began in 1991 as a trivially simple protocol for fetching HTML documents, and has evolved into a sophisticated multiplexed binary protocol because the web it carries has changed beyond recognition.

The core problem each version solves: latency. The web went from single HTML pages (1991) to pages requiring 100+ assets — CSS, JavaScript, images, fonts, API calls — all of which need to be fetched to render a page. Each protocol version attempts to reduce the number of round-trips and the total time to first byte.

HTTP/0.9

1991

GET only. No headers. No status codes. Response is the file content.

HTTP/1.0

1996 RFC 1945

Headers added. POST, HEAD methods. Status codes. 1 request per TCP connection.

HTTP/1.1

1997 RFC 2068 / 2616 / 9110

Persistent connections. Chunked encoding. Host header (virtual hosting). Pipelining (limited use). Still dominant for simple APIs.

HTTP/2

2015 RFC 7540 / 9113

Binary framing. Multiplexing. Header compression (HPACK). Server push. One TCP connection per origin. Solves HOL blocking at HTTP layer but not TCP layer.

HTTP/3

2022 RFC 9114

Runs over QUIC (UDP). Eliminates TCP HOL blocking. 0-RTT connection resumption. Connection migration. Mandatory encryption.

🌐

HTTP in the Protocol Stack

POSITION

/* Protocol stack comparison */

HTTP/1.1 and HTTP/2:          HTTP/3:
┌─────────────────┐           ┌─────────────────┐
│   HTTP/1.1 or   │           │     HTTP/3       │
│     HTTP/2      │           ├─────────────────┤
├─────────────────┤           │      QUIC        │
│   TLS 1.2/1.3   │           │  (includes TLS) │
├─────────────────┤           ├─────────────────┤
│      TCP        │           │      UDP         │
├─────────────────┤           ├─────────────────┤
│      IP         │           │      IP          │
└─────────────────┘           └─────────────────┘

/* HTTP is always text-based in HTTP/1.1 (line-delimited) */
/* HTTP/2 is binary framing over the same TCP+TLS */
/* HTTP/3 moves the entire stack to UDP with QUIC handling */
/* reliability, ordering, and encryption that TCP+TLS provided */

/* Check which HTTP version a server uses */
curl -v --http1.1 https://google.com 2>&1 | grep '< HTTP'
curl -v --http2    https://google.com 2>&1 | grep '< HTTP'
curl -v --http3    https://google.com 2>&1 | grep '< HTTP'  # if curl has HTTP/3

HTTP/1.1 — THE CLASSIC TEXT-BASED PROTOCOL

📨

HTTP/1.1 Request Format

FORMAT

HTTP/1.1 is a plain-text protocol. Every message — request or response — is human-readable ASCII, line-delimited. A request consists of a request line, followed by headers, a blank line (CRLF), and optionally a body.

POST /api/v1/users HTTP/1.1            ← Request line: Method SP Path SP Version CRLF
Host: api.example.com                  ← Mandatory in HTTP/1.1 (virtual hosting)
Content-Type: application/json         ← Body format
Content-Length: 27                    ← Body length in bytes
Authorization: Bearer eyJhbGciOiJIUzI1...
Accept: application/json
User-Agent: MyApp/2.1 (Linux x86_64)
Connection: keep-alive
                                         ← Empty line (CRLF) separates headers from body
{"name":"Ajay","role":"admin"}         ← Request body (27 bytes)

HTTP/1.1 201 Created                   ← Status line: Version SP Status-Code SP Reason CRLF
Content-Type: application/json
Content-Length: 45
Location: /api/v1/users/42              ← URL of created resource
Set-Cookie: session=abc123; HttpOnly; Secure; SameSite=Strict
X-Request-ID: 7f3a9c2d
Date: Wed, 18 Mar 2026 10:00:00 GMT

{"id":42,"name":"Ajay","role":"admin"}

🔄

Persistent Connections and Pipelining

CONNECTIONS

HTTP/1.0 opened a new TCP connection for every request — 3-way handshake + TLS handshake for every single asset. A page with 30 assets = 30 TCP connections. HTTP/1.1 introduced persistent connections (keep-alive) as the default: after a response, the TCP connection stays open for subsequent requests.

HTTP/1.0

TCP SYN

Request 1

Response 1

FIN

TCP SYN

Request 2

Response 2

FIN

HTTP/1.1

TCP SYN

Request 1

Response 1

Request 2

Response 2

Req 3…

HOL block

TCP SYN

Large Request 1 (slow)

Req 2 BLOCKED

Req 3 BLOCKED

Resp 1

Head-of-Line (HOL) blocking is HTTP/1.1's critical weakness: responses must be returned in request order on a single connection. If request 1 is slow (large file, slow server), requests 2, 3, 4 all wait behind it even if they'd complete instantly. Browsers work around this by opening 6 parallel TCP connections per origin — but this wastes resources and still isn't a clean solution.

Chunked Transfer Encoding

/* Server sends response body in chunks — no Content-Length needed */
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: text/html

1a               ← chunk size in hex: 0x1a = 26 bytes
This is the first chunk.
13               ← 0x13 = 19 bytes
And the second chunk.
0                ← zero-length chunk = end of body
                 ← trailing CRLF

/* Used for: streaming responses, server-sent events */
/* server starts sending before it knows total size */

HTTP METHODS AND STATUS CODES — THE COMPLETE REFERENCE

📋

HTTP Request Methods

METHODS

Method	Semantics	Safe?	Idempotent?	Has Body?	Common Use
`GET`	Retrieve a resource	Yes	Yes	No (ignored)	Fetch web page, API data, images
`POST`	Create/submit data	No	No	Yes	Create resource, form submit, login
`PUT`	Replace a resource completely	No	Yes	Yes	Replace entire user record
`PATCH`	Partially update a resource	No	No	Yes	Change user's email only
`DELETE`	Remove a resource	No	Yes	Optional	Delete a user, file, record
`HEAD`	GET but response body omitted	Yes	Yes	No	Check if resource exists/changed (ETag/Last-Modified)
`OPTIONS`	Describe communication options	Yes	Yes	No	CORS preflight, discover allowed methods
`CONNECT`	Establish a tunnel through proxy	No	No	No	HTTPS through HTTP proxy (CONNECT example.com:443)
`TRACE`	Echo request for debugging	Yes	Yes	No	Diagnostic (mostly disabled — XST attack risk)

💡 Safe = no side effects (read-only). Idempotent = calling multiple times has same effect as calling once. These properties matter for retry logic (retrying a non-idempotent POST could create duplicates) and NGFW policy (blocking unsafe methods on read-only APIs).

🔢

HTTP Status Codes — Every Category

STATUS CODES

1xx — Informational

100 Continue — server received request headers, client should send body
101 Switching Protocols — upgrading to WebSocket or HTTP/2

2xx — Success

200 OK — standard success
201 Created — resource created (POST)
204 No Content — success, no body (DELETE)
206 Partial Content — range request fulfilled

3xx — Redirection

301 Moved Permanently — permanent redirect, update bookmarks
302 Found — temporary redirect
304 Not Modified — cached version is still valid (ETag/If-None-Match)
307 Temporary Redirect — keep method (don't change POST → GET)
308 Permanent Redirect — like 301 but preserves method

4xx — Client Errors

400 Bad Request — malformed request syntax
401 Unauthorized — authentication required
403 Forbidden — authenticated but not authorised
404 Not Found — resource doesn't exist
405 Method Not Allowed — wrong HTTP method
408 Request Timeout — client too slow
409 Conflict — state conflict (duplicate resource)
429 Too Many Requests — rate limited

5xx — Server Errors

500 Internal Server Error — generic server error
502 Bad Gateway — upstream proxy/backend error
503 Service Unavailable — server overloaded or maintenance
504 Gateway Timeout — upstream took too long

💡 NGFW status code monitoring: A spike in 401/403 responses from a single source may indicate a credential stuffing or brute-force attack. A flood of 500 responses may indicate a vulnerability scan or SQL injection attempt. A stream of 404s from a single source is likely a web crawler or directory enumeration scan. These patterns are detectable in your NGFW's HTTP inspection logs.

KEY HTTP HEADERS — REQUEST, RESPONSE, AND SECURITY HEADERS

📋

Critical Request Headers

REQUEST HEADERS

Header	Format / Values	Purpose	NGFW Relevance
`Host`	`api.example.com`	Target virtual host. Mandatory in HTTP/1.1. Enables multiple sites on one IP.	URL filtering — identifies domain even when IP is shared. SNI cross-check for TLS inspection.
`User-Agent`	`Mozilla/5.0 (Linux...)`	Client application identity. Browser, OS, version.	Detect bots, scanners, known malware user-agents (C2 beaconing uses custom UA strings).
`Authorization`	`Bearer token` / `Basic base64`	Authentication credential.	DLP: detect credential exfiltration in plaintext HTTP. Inspect for known compromised tokens.
`Cookie`	`session=abc; pref=dark`	Session state sent to server.	Session hijacking detection. Cookie flags (HttpOnly, Secure) enforcement.
`Referer`	`https://google.com/search?q=...`	Where the user came from. (Yes, historically misspelled.)	Data leakage — Referer may contain sensitive search queries or internal URLs.
`Content-Type`	`application/json` / `multipart/form-data`	Format of request body.	DPI dispatch: parse body as JSON/XML/form-data for content inspection.
`X-Forwarded-For`	`203.x.x.x, 10.0.0.1`	Original client IP when going through proxies.	Real client IP for logging and policy. Must validate chain — easily forged by clients.
`Origin`	`https://app.example.com`	Origin of cross-origin request (CORS).	Detect cross-origin attacks. Origin ≠ Host may indicate CSRF or XSS attempt.

🛡️

Security Response Headers — What Your NGFW Should Enforce

SECURITY HEADERS

Security headers are HTTP response headers that instruct the browser to enable security protections. A WAF or NGFW proxy can inject missing security headers into responses:

/* Security headers — every production server should send these */

Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
  # HSTS: browser must use HTTPS for this domain for 1 year
  # Prevents SSL stripping attacks
  # preload: submit to browser preload list

Content-Security-Policy: default-src 'self'; script-src 'self' cdn.example.com; object-src 'none'
  # CSP: whitelist of allowed content sources
  # Prevents XSS by blocking inline scripts and untrusted sources

X-Frame-Options: DENY
  # Prevent clickjacking — page cannot be embedded in iframe
  # Superseded by CSP frame-ancestors, but still widely needed

X-Content-Type-Options: nosniff
  # Browser must not MIME-sniff — prevents content-type confusion attacks

Referrer-Policy: strict-origin-when-cross-origin
  # Controls what goes in Referer header on cross-origin requests
  # Prevents leaking sensitive URLs to third parties

Permissions-Policy: camera=(), microphone=(), geolocation=(self)
  # Restrict browser APIs — prevent malicious pages accessing camera/mic

Set-Cookie: session=abc; HttpOnly; Secure; SameSite=Strict; Path=/
  # HttpOnly: JS cannot access cookie (prevents XSS cookie theft)
  # Secure: only sent over HTTPS
  # SameSite=Strict: not sent on cross-site requests (prevents CSRF)

HTTP/2 — BINARY FRAMING AND MULTIPLEXING

⚡

HTTP/2 Core Innovations

HTTP/2

HTTP/2 (RFC 9113) keeps the same HTTP semantics (methods, status codes, headers) but completely replaces the wire format. It moves from text-based line-by-line messages to a binary framing layer that enables multiplexing — multiple concurrent request/response exchanges over a single TCP connection.

Four key innovations:

Binary framing — messages broken into typed binary frames instead of text lines. More efficient to parse, less error-prone, enables features impossible in text
Multiplexing — multiple streams (each stream = one request/response pair) interleaved on the same TCP connection. No more 6-connection limit. No more HOL blocking at the HTTP layer
HPACK header compression — headers compressed using a shared static table + dynamic table. Eliminates the overhead of repeating User-Agent, Cookie, Authorization on every request
Server push — server can proactively send resources the client will need (push CSS/JS when it sends HTML) without waiting for the client to request them

📊

HTTP/2 Streams and Framing

STREAMS

An HTTP/2 stream is a bidirectional sequence of frames on a single TCP connection. Each stream has an integer ID (client-initiated streams use odd IDs: 1, 3, 5…; server-initiated use even IDs). Multiple streams run concurrently — frames from different streams are interleaved:

HTTP/1.1
(6 conns)

Conn1: HTML

Conn2: CSS

Conn3: JS

Conn4: img1

Conn5: img2

Wait...

HTTP/2
(1 conn)

S1:H

S3:H

S5:H

S1:DATA

S7:H

S3:DATA

S9:H

S5:DATA

S7:DATA

S9:DATA

/* HTTP/2 Frame format */
+-----------------------------------------------+
| Length (24 bits) | Type (8 bits) | Flags (8b) |
+-----------------------------------------------+
|R|            Stream ID (31 bits)               |
+-----------------------------------------------+
|                  Frame Payload                 |
+-----------------------------------------------+

/* Frame types */
Type 0x0 DATA:        request/response body data
Type 0x1 HEADERS:     request/response headers (HPACK compressed)
Type 0x2 PRIORITY:    stream dependency and weight for scheduling
Type 0x3 RST_STREAM:  cancel/error a specific stream
Type 0x4 SETTINGS:    negotiate connection parameters (max frame size, max streams)
Type 0x5 PUSH_PROMISE:server announces pushed resource
Type 0x6 PING:        keepalive and RTT measurement
Type 0x7 GOAWAY:      graceful connection shutdown — last processed stream ID
Type 0x8 WINDOW_UPDATE:flow control — increase receive window
Type 0x9 CONTINUATION:continue HEADERS frame if too large for one frame

/* HEADERS frame example — HPACK compressed */
/* Static table entry: :method GET = index 2, :path / = index 4 */
\x82           # :method = GET  (index 2 from static table)
\x84           # :path = /       (index 4)
\x86           # :scheme = https (index 6)
\x41 \x8a...   # :authority = www.google.com (literal with indexing)

⚠️ HTTP/2 still has TCP HOL blocking. While HTTP/2 eliminates application-layer head-of-line blocking (requests no longer wait behind each other at the HTTP layer), it still runs over TCP. If a single TCP packet is lost, TCP holds ALL streams until retransmission completes — even streams that have no dependency on the lost data. This is the core motivation for HTTP/3's move to QUIC/UDP.

🗜️

HPACK Header Compression

HPACK

HTTP headers are verbose and repetitive. A typical request sends the same User-Agent, Accept, Cookie, and Authorization headers on every request — hundreds of bytes of overhead per request. HPACK (RFC 7541) compresses these to a handful of bytes using two techniques:

Static table — 61 pre-defined name/value pairs that are indexed by number. :method GET = index 2 (1 byte). :status 200 = index 8 (1 byte). Eliminates overhead for common headers entirely.
Dynamic table — previously seen name/value pairs are added to a shared dynamic table. Subsequent occurrences are replaced by their table index. A long Cookie header sent once can be referenced by 1 byte on all future requests.

/* HPACK compression example */
Request 1 — first time User-Agent is sent:
  Header: user-agent: Mozilla/5.0 (Linux; Android 11)
  Wire bytes: ~45 bytes (literal encoding, added to dynamic table at index 62)

Request 2 — same User-Agent:
  Wire bytes: \xbe (1 byte = index 62 in dynamic table)
  Savings: 98%

/* CRIME and BREACH attacks — why HPACK must be careful */
# If attacker can inject chosen plaintext adjacent to secrets in compressed data,
# they can observe compression ratio to infer secret values byte by byte.
# TLS compression was disabled due to CRIME. HPACK deliberately does NOT
# compress across streams (separate compression contexts) to mitigate.

HTTP/3 AND QUIC — REIMAGINING THE WEB TRANSPORT STACK

🚀

Why QUIC Was Invented

MOTIVATION

HTTP/2 over TCP still had TCP's fundamental limitations. Fixing these would require changing TCP itself — which means changing every operating system kernel worldwide. That's essentially impossible at any reasonable timescale. Google's solution: build a new transport protocol in user space over UDP, implement it in the browser and server, and deploy it without any OS changes.

QUIC (RFC 9000) addresses four problems that TCP cannot solve without OS-level changes:

TCP HOL blocking — a lost TCP packet stalls all HTTP/2 streams. QUIC streams are independent: a lost QUIC packet only stalls the one stream it belongs to
Connection establishment latency — TCP requires 1 RTT for handshake + 1 RTT for TLS = 2 RTTs before first byte. QUIC combines transport and crypto handshake in 1 RTT. With 0-RTT, returning clients can send data immediately
Connection migration — TCP connections are tied to a 4-tuple (src IP, src port, dst IP, dst port). If you walk from WiFi to cellular, your IP changes, TCP connection breaks. QUIC uses Connection IDs — connections survive IP changes
Ossification — TCP middleboxes (firewalls, NATs, proxies) inspect and sometimes modify TCP headers. This prevented TCP from evolving. QUIC is encrypted — middleboxes see only UDP packets, cannot inspect or modify internals

📦

QUIC Architecture

QUIC INTERNALS

/* QUIC packet structure (simplified) */
┌─────────────────────────────────────────────────────┐
│  UDP Header (8 bytes)                               │
├─────────────────────────────────────────────────────┤
│  QUIC Long/Short Header                             │
│  - Header Form (1 bit): Long=1, Short=0             │
│  - Connection ID (0–20 bytes): identifies connection│
│    without IP/port (enables migration!)             │
│  - Packet Number (1–4 bytes): sequence for ACK      │
├─────────────────────────────────────────────────────┤
│  QUIC Frames (encrypted with TLS 1.3)              │
│  - STREAM frames: carry application data            │
│  - ACK frames: acknowledge received packets         │
│  - CRYPTO frames: TLS handshake messages            │
│  - PADDING, PING, CONNECTION_CLOSE, etc.            │
└─────────────────────────────────────────────────────┘

/* QUIC connection establishment — 1-RTT */
Client → Server: Initial (ClientHello inside CRYPTO frame)
Server → Client: Initial (ServerHello, certificates, Finished) + Handshake + 1-RTT data
Client → Server: Handshake (Finished) + 1-RTT data ← FIRST DATA HERE

vs TLS over TCP: SYN → SYN+ACK → ACK → ClientHello → ServerHello...Finished → data
                 That's 2 RTTs minimum before data flows.

/* 0-RTT resumption */
Client received a "session ticket" from prior connection:
Client → Server: 0-RTT data immediately (with replay-protected pre-shared key)
                 Server accepts 0-RTT data — zero round trips!
Caveat: 0-RTT data is replay-vulnerable — only safe for idempotent requests (GET)

📊

QUIC Frame Types

FRAMES

STREAM

Type 0x08–0x0f

Carries application data for a specific stream. Flags control FIN (end of stream), LEN, and OFF (offset). Multiple streams multiplexed, each independently ordered.

ACK

Type 0x02–0x03

Acknowledges received QUIC packets. ACK ranges allow selective acknowledgement natively — no need for SACK option like TCP. Includes ACK delay measurement.

CRYPTO

Type 0x06

Carries TLS handshake messages (ClientHello, ServerHello, Certificate, Finished). Used during connection establishment before 1-RTT keys are available.

NEW_CONNECTION_ID

Type 0x18

Server provides new Connection IDs client can use. Enables connection migration — client switches to new ID on network change, preventing linkability.

MAX_DATA / MAX_STREAM_DATA

Type 0x10–0x13

Flow control — increase the connection-level or stream-level receive window. Equivalent to TCP's window update but per-stream.

CONNECTION_CLOSE

Type 0x1c–0x1d

Graceful connection termination. Carries error code and reason phrase. Unlike TCP FIN — closes entire connection, not individual streams.

🌐

HTTP/3 — HTTP over QUIC

HTTP/3

HTTP/3 (RFC 9114) is HTTP/2's semantics (same methods, status codes, header compression) re-implemented over QUIC instead of TCP+TLS. Key differences from HTTP/2:

QPACK instead of HPACK — header compression adapted for QUIC's independent stream ordering. HPACK's dynamic table requires strict ordering (which TCP guaranteed); QPACK uses separate encoder/decoder streams to avoid this dependency
No TCP HOL blocking — QUIC streams are truly independent. A lost UDP packet only pauses the stream it belongs to
Stream IDs — HTTP/3 uses QUIC stream IDs directly. Bidirectional streams for request/response, unidirectional for control and QPACK encoder/decoder
Mandatory HTTPS — QUIC always uses TLS 1.3; there is no "plain HTTP/3 without TLS"

# Detect HTTP/3 support — look for Alt-Svc header in HTTP/1.1 or HTTP/2 response
curl -I https://cloudflare.com | grep -i alt-svc
# alt-svc: h3=":443"; ma=86400
# "I support HTTP/3 (h3) on port 443, this hint is valid for 86400 seconds"

# Test HTTP/3 with curl (requires --http3 support)
curl --http3 -v https://cloudflare.com 2>&1 | head -20

# Wireshark capture of QUIC traffic
# Filter: udp.port == 443 (QUIC uses UDP 443)
# QUIC packets appear as "QUIC" in protocol column
# Content is encrypted — only metadata visible without TLS keys

# Provide TLS keys to Wireshark for decryption
SSLKEYLOGFILE=/tmp/keys.log curl --http3 https://cloudflare.com
# In Wireshark: Edit → Preferences → Protocols → TLS → Master Secret log

HTTPS — HTTP OVER TLS, AND WHY THE URL BAR IS GREEN

🔒

HTTPS — What It Does and What It Doesn't

HTTPS

HTTPS is simply HTTP running inside a TLS tunnel. TLS (Transport Layer Security) adds three properties to the connection:

Confidentiality — all HTTP content (headers, body, URL path) is encrypted. An on-path attacker can see the destination IP and SNI hostname, but cannot read the request path, headers, cookies, or response body
Integrity — TLS MAC ensures the content hasn't been tampered with in transit. Attacker cannot modify HTTP responses (inject ads, malware) without detection
Authentication — the server's TLS certificate proves its identity. Your browser verifies the certificate was issued by a trusted CA and matches the hostname

⚠️ What HTTPS does NOT protect: The destination hostname is visible in the TLS SNI (Server Name Indication) extension — the server needs to know which certificate to present before decryption. The destination IP is always visible (IP routing requires it). HTTP/2 header sizes are visible even though content is encrypted. HTTPS proves the server is who it claims to be — it does NOT prove the site is legitimate or safe.

🤝

TLS Handshake Timeline — HTTP/1.1 vs HTTP/2 vs HTTP/3

HANDSHAKE

/* HTTP/1.1 over TLS 1.3 — minimum 2 RTTs before data */
RTT 0: → TCP SYN
        ← TCP SYN+ACK
RTT 1: → TCP ACK + TLS ClientHello (with key_share)
        ← TLS ServerHello + Certificate + CertVerify + Finished (encrypted)
RTT 2: → TLS Finished + HTTP GET   ← HTTP response arrives
                                      ← DATA arrives

/* HTTP/2 over TLS 1.3 — same, but ALPN negotiates h2 in handshake */
TLS ClientHello includes: ALPN extension ["h2", "http/1.1"]
TLS ServerHello includes: ALPN selected "h2"
After handshake: HTTP/2 binary framing on the same connection

/* HTTP/3 over QUIC — 1 RTT (0-RTT for returning clients) */
RTT 0: → QUIC Initial (ClientHello in CRYPTO frame)
        ← QUIC Initial+Handshake (ServerHello+Cert+Finished)
           + 1-RTT data (server can already send response!)
RTT 1: → QUIC Handshake Finished + HTTP/3 request arrives
        ← (server was already sending response from RTT 0)

/* ALPN — Application-Layer Protocol Negotiation */
# Allows HTTP version negotiation within TLS handshake
# No extra round-trip needed
# Values: "h3" = HTTP/3, "h2" = HTTP/2, "http/1.1" = HTTP/1.1

NGFW HTTP INSPECTION — URL FILTERING, DPI, AND SSL INSPECTION

🛡️

HTTP Inspection in an NGFW — What Gets Inspected

INSPECTION

For plain HTTP traffic (port 80), an NGFW can inspect everything. For HTTPS, the NGFW must either trust the SNI (limited information) or perform SSL inspection (full access but requires certificate deployment).

Element	Visible in HTTP	Visible in HTTPS (no SSL inspection)	Visible with SSL Inspection
Destination IP	Yes	Yes	Yes
Destination hostname	Host header	TLS SNI only	Host header
URL path and query string	Yes (`GET /path?q=...`)	No (encrypted)	Yes
HTTP method	Yes	No	Yes
Request headers (Cookie, Auth)	Yes	No	Yes
Response status code	Yes	No	Yes
Response body (file content)	Yes	No	Yes
File downloads (malware scan)	Yes	No	Yes
TLS certificate	N/A	Yes (server cert visible)	Yes
TLS version and cipher suite	N/A	Yes (during handshake)	Yes

🔬

SSL Inspection — How NGFW Decrypts HTTPS

SSL INSPECTION

SSL inspection (also called TLS inspection, SSL bump, or MITM proxy) allows an NGFW to decrypt, inspect, and re-encrypt HTTPS traffic. It is the most powerful — and most controversial — NGFW capability.

Client initiates TLS connection to server

Client sends TLS ClientHello to server IP. The NGFW intercepts this connection — it acts as a transparent proxy.

NGFW establishes TLS with the real server

NGFW opens its own separate TLS connection to the destination server. It receives and validates the server's real certificate. NGFW now has the session keys — it can decrypt all server traffic.

NGFW ↔ Real Server: legitimate TLS with server's real cert

NGFW generates a forged certificate for the client

NGFW dynamically generates a certificate for the domain (e.g., google.com), signed by the NGFW's own CA certificate. This "forged" cert has the correct hostname but is signed by the corporate CA — not by DigiCert or Let's Encrypt.

NGFW → Client: "Here is google.com's cert" (signed by corporate CA)

Client trusts the corporate CA — verifies successfully

The corporate CA cert was pushed to all managed devices via MDM/GPO. Client's browser trusts it. TLS handshake with NGFW completes — client thinks it has a direct connection to google.com.

NGFW decrypts, inspects, re-encrypts all traffic

All HTTPS traffic passes through the NGFW in cleartext. Full HTTP inspection is possible: URL filtering, DLP, malware scanning, application identification. After inspection, NGFW re-encrypts and forwards to the real server.

Client → [NGFW decrypts] → inspect → [re-encrypt] → Server

💡 Exclusion list: SSL inspection should exclude financial and healthcare sites (banks, healthcare portals) where privacy regulations require client-to-server TLS integrity. Certificate pinning (used by some mobile apps) will break — those apps must be excluded. Most NGFWs maintain an exclusion list of domains where SSL inspection is bypassed.

🔍

HTTP-Based Attack Detection

ATTACKS

Attack	HTTP Indicators	NGFW Detection
SQL Injection	URL query: `?id=1' OR '1'='1`; POST body with SQL keywords (SELECT, UNION, DROP)	Regex/signature match on URL params and POST body. Rate-limit 400 errors from same source.
Cross-Site Scripting (XSS)	URL/body contains: `<script>`, `javascript:`, `onload=`, encoded variants (`%3Cscript%3E`)	Decode URL encoding first, then signature match. Block reflected XSS patterns in GET params.
Path Traversal	URL contains: `../`, `..\`, encoded variants (`%2e%2e%2f`)	Normalise and decode URL, detect directory traversal sequences.
Command Injection	Body contains: `; ls`, `\| cat /etc/passwd`, `whoami`	Signature match on shell metacharacters in POST body.
Web Shell Upload	POST to `/upload` with Content-Type: multipart; file content contains PHP/JSP code	Scan uploaded file content for webshell signatures, not just extension.
Data Exfiltration	Large POST bodies to external sites. Unusual User-Agent. HTTP CONNECT to non-standard ports.	DLP policy on POST body size/content. Block CONNECT to non-80/443 ports.
HTTP DDoS	Flood of GET requests to same URL; slowloris (partial requests held open)	Rate-limit requests per source IP per second. Detect incomplete request bodies.

LAB 1

HTTP Protocol Analysis with curl and Wireshark

Objective: Capture and fully decode HTTP/1.1, HTTP/2, and observe the protocol upgrade mechanism. Understand every header in a real request/response exchange.

Start Wireshark capture. Make an HTTP/1.1 request: curl -v --http1.1 http://httpbin.org/get. In Wireshark, filter http. Expand the HTTP request packet: find method (GET), URI path (/get), HTTP version (1.1), all request headers. Expand the response: status code (200), all response headers, JSON body.

Examine the raw TCP stream: right-click the HTTP GET packet → Follow → TCP Stream. You'll see the full plaintext HTTP/1.1 exchange. Identify the empty CRLF line that separates headers from body. Count header bytes vs body bytes — calculate overhead ratio.

Test persistent connections: curl -v --http1.1 http://httpbin.org/get http://httpbin.org/headers (two URLs to same server). In Wireshark, verify both requests use the same TCP connection (same port numbers in stream). Look for Connection: keep-alive header.

HTTP/2 analysis: curl -v --http2 https://nghttp2.org. In Wireshark: because HTTP/2 is encrypted, use SSL key log: SSLKEYLOGFILE=/tmp/keys.log curl --http2 https://nghttp2.org. Configure Wireshark to use the key log (Edit → Preferences → TLS → Master Secret log). Now HTTP/2 frames are visible — find HEADERS, DATA frame types.

Test various HTTP methods with httpbin:
curl -X POST -d '{"key":"val"}' -H "Content-Type: application/json" https://httpbin.org/post
curl -X DELETE https://httpbin.org/delete
curl -X PUT -d 'data' https://httpbin.org/put.
In each response, examine the json/data/form fields showing what the server received.

Bonus — QUIC traffic: SSLKEYLOGFILE=/tmp/keys.log curl --http3 https://cloudflare.com. In Wireshark: filter udp.port == 443. Without key log you see QUIC packet metadata but encrypted content. With key log configured, QUIC frames become visible. Compare frame structure to what you studied in Tab 5.

LAB 2

Build an HTTP/1.1 Server and Client in C

Objective: Write a minimal HTTP/1.1 server from scratch using TCP sockets. Manually construct and parse HTTP request lines, headers, and bodies. This gives you the deepest possible understanding of the protocol.

Write a minimal HTTP/1.1 server: create a TCP socket, bind to port 8080, listen, accept connections. For each connection, read bytes until you find the double CRLF (\r\n\r\n) marking the end of headers. Parse the request line: method, path, version.

Parse headers line-by-line: split on : to get name/value pairs. Store in a simple key-value array. Implement: extract Content-Length header (to know how many body bytes to read after the header CRLF).

Implement routing: if path == "/", return 200 OK with an HTML body. If path == "/api/echo", return the request body as JSON. If path is anything else, return 404 Not Found. Construct response by writing the status line, headers (Content-Type, Content-Length, Connection: keep-alive), blank line, body.

Test your server: curl -v http://localhost:8080/ and curl -v -X POST -d "hello" http://localhost:8080/api/echo. In Wireshark, verify your responses have correct HTTP/1.1 format. Then test with a browser — it should render your HTML response.

Bonus — implement chunked encoding: For the /stream endpoint, set Transfer-Encoding: chunked and send the response body in 5 chunks of 10 bytes each with proper hex length prefixes. Use curl -v http://localhost:8080/stream to verify the chunked response is reassembled correctly.

LAB 3

HTTP Security Header Audit and Attack Simulation

Objective: Audit a web server's security headers, simulate HTTP-based attacks, and observe how headers prevent them. Understand what an NGFW sees in HTTP attack traffic.

Security header audit: Use curl to inspect security headers: curl -sI https://google.com | grep -iE 'strict|content-security|x-frame|x-content|referrer|permissions'. Do the same for 5 different sites. Which ones have all security headers? Which are missing? Note which are missing CSP — those are vulnerable to XSS.

SQL injection patterns: Send test SQL injection payloads to httpbin (a safe echo server): curl "https://httpbin.org/get?id=1%27%20OR%20%271%27%3D%271". The %27 is URL-encoded single quote. In the response, verify the server received the decoded SQL. Write a simple Python script that scans a list of URLs for SQL injection patterns in the response (errors, SQL keywords in output).

XSS detection: Send an XSS payload: curl "https://httpbin.org/get?q=%3Cscript%3Ealert(1)%3C%2Fscript%3E". Capture in Wireshark. Write a filter rule (in Python or BPF syntax) that would detect this: match URL-encoded or decoded <script> tags in HTTP GET query strings.

User-Agent analysis: Make requests with various user agents and observe server responses: curl -A "sqlmap/1.0" https://httpbin.org/headers (known scanner UA). curl -A "" https://httpbin.org/headers (empty UA). curl -A "Mozilla/5.0" https://httpbin.org/headers (normal browser). Build a list of malicious/scanner User-Agent strings and write a detection rule.

Rate limiting simulation: Write a Python script that sends 100 GET requests per second to your HTTP server from Lab 2. Observe: does your server handle it? Add rate limiting: track requests per source IP in a hash map, return 429 Too Many Requests if rate exceeds 10 req/sec. Test the rate limiter works.

M08 MASTERY CHECKLIST

Can explain the key problem each HTTP version solved: 1.0=persistent, 1.1=keep-alive+chunked, 2=multiplexing, 3=no TCP HOL blocking
Know HTTP/1.1 message format: request line + headers + CRLF + body; status line + headers + CRLF + body
Know the mandatory HTTP/1.1 header: Host (enables virtual hosting)
Know all 9 HTTP methods: GET, POST, PUT, PATCH, DELETE, HEAD, OPTIONS, CONNECT, TRACE
Know safe vs idempotent: GET/HEAD/OPTIONS are safe; GET/PUT/DELETE/HEAD/OPTIONS are idempotent; POST/PATCH are neither
Know all 5 status code ranges and 15+ specific codes (200, 201, 204, 301, 302, 304, 400, 401, 403, 404, 405, 429, 500, 502, 503)
Know Head-of-Line blocking in HTTP/1.1: responses must return in request order; one slow response blocks all subsequent
Know chunked transfer encoding: hex length prefix per chunk, zero-length chunk = end
Know 8 critical request headers: Host, User-Agent, Authorization, Cookie, Referer, Content-Type, X-Forwarded-For, Origin
Know 6 security response headers: HSTS, CSP, X-Frame-Options, X-Content-Type-Options, Referrer-Policy, Permissions-Policy
Know cookie security flags: HttpOnly (no JS access), Secure (HTTPS only), SameSite=Strict (no cross-site)
Know HTTP/2's four innovations: binary framing, multiplexing, HPACK compression, server push
Know HTTP/2 frame types: DATA, HEADERS, RST_STREAM, SETTINGS, PUSH_PROMISE, PING, GOAWAY, WINDOW_UPDATE
Know HTTP/2 still has TCP HOL blocking — a lost TCP packet stalls all streams
Know HPACK: static table (61 entries), dynamic table (per-connection learned headers), both compressed to 1-byte index
Know QUIC's 4 motivations: TCP HOL blocking, 2-RTT connection setup, no connection migration, TCP ossification
Know QUIC Connection ID: enables connection migration when client IP changes (WiFi → cellular)
Know QUIC 1-RTT vs 0-RTT: 1-RTT = standard first connection; 0-RTT = session resumption, replay-vulnerable
Know 6 QUIC frame types: STREAM, ACK, CRYPTO, NEW_CONNECTION_ID, MAX_DATA, CONNECTION_CLOSE
Know QPACK vs HPACK: QPACK adapted for out-of-order QUIC streams using separate encoder/decoder streams
Know HTTP/3 uses ALPN "h3", runs on UDP 443, always requires TLS 1.3
Know what HTTPS encrypts: URL path, headers, body. What it doesn't hide: destination IP, TLS SNI hostname
Know how SSL inspection works: NGFW → server (real TLS), NGFW → client (forged cert signed by corporate CA)
Know SSL inspection limitations: certificate pinning, privacy regulations, requires CA cert deployment
Know HTTP inspection visibility matrix: what's visible in plain HTTP vs HTTPS vs with SSL inspection
Know 7 HTTP attack types for NGFW detection: SQLi, XSS, path traversal, command injection, webshell, exfiltration, DDoS
Know NGFW HTTP monitoring: 401/403 spikes = brute force; 404 flood = directory enumeration; 500 flood = injection scan
Completed Lab 1: captured and decoded HTTP/1.1 and HTTP/2 traffic in Wireshark with TLS key log
Completed Lab 2: built HTTP/1.1 server from scratch in C with routing, header parsing, and chunked encoding
Completed Lab 3: audited security headers, simulated SQLi/XSS patterns, implemented rate limiting

✅ When complete: Move to M09 - SMTP, FTP, and DHCP. These are the protocols an NGFW must parse to protect email infrastructure, file transfers, and network bootstrapping — each introduces unique ALG (Application-Level Gateway) challenges for firewall traversal.

← M07 DNS 🗺️ Roadmap Next: M09 - SMTP, FTP, DHCP →