infrastructure

Connection Pooling

Connection pooling reuses open TCP/TLS connections across multiple HTTP requests to reduce handshake overhead and improve scraping throughput.

Establishing a TCP connection involves a three-way handshake; establishing a TLS connection adds certificate exchange and key negotiation — together adding 100–500 ms of latency before the first byte of data arrives. Connection pooling maintains a set of already-established connections that can be reused for subsequent requests to the same host, eliminating this overhead.

HTTP/1.1 introduced `Keep-Alive` to reuse connections serially. HTTP/2 goes further by multiplexing multiple concurrent requests over a single connection, making connection pooling even more impactful. Most modern HTTP client libraries (Python's `httpx`, `requests.Session`, Node's `axios`) maintain connection pools automatically.

For scrapers making many requests to the same domain, connection pooling can improve throughput significantly. However, reusing connections also means the server can track session state across requests, which may be used by anti-bot systems to build a longer behavioural profile.

Examples

# httpx: use a persistent client (connection pool enabled)
import httpx

with httpx.Client(http2=True) as client:
    for url in urls:
        response = client.get(url)
        process(response)
# Connections are reused across all requests in the with block