infrastructure

Request Throttling

Request throttling deliberately limits the rate at which a scraper sends requests to a target site, preventing detection and avoiding overloading the server.

Throttling is both a courtesy and a self-preservation strategy. Sending requests faster than a site can process them degrades the target's performance for real users and triggers rate-limit defences sooner. Throttling — capping requests to a fixed rate (e.g., 1 request per second per domain) — keeps scraping traffic below detection thresholds and mimics natural browsing patterns.

Implementations range from simple `time.sleep()` calls between requests to sophisticated token-bucket or leaky-bucket rate limiters that maintain a smooth request rate regardless of processing time variance. Token buckets accumulate credits at a constant rate (e.g., 1 token/second) and each request consumes one token — when tokens run out, the next request waits.

For distributed scraping across many workers, a centralised rate-limit store (Redis) coordinates throttling so that workers collectively respect the per-domain limit rather than each worker independently hitting the limit at full speed.

Examples

import asyncio

class TokenBucket:
    def __init__(self, rate: float):
        self.rate = rate  # tokens per second
        self.tokens = rate
        self.last = asyncio.get_event_loop().time()

    async def acquire(self):
        now = asyncio.get_event_loop().time()
        self.tokens = min(self.rate, self.tokens + (now - self.last) * self.rate)
        self.last = now
        if self.tokens < 1:
            await asyncio.sleep((1 - self.tokens) / self.rate)
        self.tokens -= 1