Rate Limiting, Retries, Backoff and Connection Pooling in Python
Short answer
How do you add rate limiting, retries with backoff and connection pooling to a Python scraper?
Use one shared httpx or requests session so connections are pooled, cap concurrency with a semaphore, and retry only on timeouts, 429 and 5xx responses. Wait with exponential backoff plus random jitter, honour Retry-After headers and stop after a few attempts. AlterLab applies the same retry and pacing logic on its side for every request.
Step by step
- 1
Share one client
Create a single httpx.AsyncClient with connection limits and reuse it, so TCP and TLS connections are pooled.
- 2
Limit concurrency
Wrap requests in an asyncio.Semaphore sized to what the target can handle.
- 3
Retry selectively
Retry timeouts, 429 and 5xx. Do not retry 4xx client errors such as 404.
- 4
Back off with jitter
Sleep base * 2^attempt plus random jitter, or the Retry-After value when the server sends one.
Example
import asyncio, random
import httpx
limits = httpx.Limits(max_connections=20, max_keepalive_connections=10)
sem = asyncio.Semaphore(5)
async def fetch(client: httpx.AsyncClient, url: str, attempts: int = 4) -> str:
for attempt in range(attempts):
async with sem:
try:
r = await client.get(url, timeout=20)
if r.status_code not in (429, 500, 502, 503, 504):
r.raise_for_status()
return r.text
wait = float(r.headers.get("Retry-After", 0))
except httpx.TimeoutException:
wait = 0
await asyncio.sleep(wait or 2 ** attempt + random.random())
raise RuntimeError(f"gave up on {url}")
async def main(urls: list[str]) -> None:
async with httpx.AsyncClient(limits=limits) as client:
pages = await asyncio.gather(*(fetch(client, u) for u in urls))
print(len(pages))Frequently Asked Questions
Why add jitter to backoff?
Without jitter, many workers that failed together retry together and hit the server at the same instant again. Random jitter spreads the retries out.
How many retries are reasonable?
Three or four attempts cover most transient failures. More attempts mostly add load and delay without improving success.
Read the full guide
Python web scraping guideTry it with AlterLab
Send a URL, get the page back as HTML, markdown or JSON. Rendering, proxy rotation and retries are handled for you.
Your first scrape.
Sixty seconds.
Up to 5,000 free requests — about 5,000 Light, 500 Protected or 250 Dynamic requests. No credit card.
Just a POST request.
No credit card required · Up to 5,000 free requests (about 5,000 Light, 500 Protected or 250 Dynamic requests) · Balance never expires