protocol

Rate Limit Headers

Rate limit headers are HTTP response headers that communicate how many requests remain in the current window and when the limit resets, allowing clients to self-throttle.

The IETF draft standard for rate limit headers defines three headers: `RateLimit-Limit` (the maximum requests allowed in a window), `RateLimit-Remaining` (requests left in the current window), and `RateLimit-Reset` (when the window resets, in Unix epoch seconds). The older `X-RateLimit-*` prefix is also widely used by APIs before the standard was proposed.

For scrapers calling APIs, reading rate limit headers allows the client to proactively throttle before hitting the limit rather than reactively backing off after receiving a 429. A polite client monitors `RateLimit-Remaining` and slows down as it approaches zero, then waits until `RateLimit-Reset` before resuming full speed.

Some anti-bot systems embed rate limit information in custom headers specific to their platform (Cloudflare: `cf-cache-status`, Fastly: `Fastly-RateLimit-*`). Understanding these platform-specific headers helps scrapers optimise their request pacing without triggering hard blocks.

Examples

import requests, time

response = requests.get("https://api.example.com/data")
remaining = int(response.headers.get("RateLimit-Remaining", 100))
reset_at = int(response.headers.get("RateLimit-Reset", 0))

if remaining < 5:
    wait = max(0, reset_at - int(time.time()))
    print(f"Approaching rate limit — sleeping {wait}s")
    time.sleep(wait)

Related Terms

Rate Limiting
A server-side control that caps the number of requests accepted from a single IP or session within a time window, returning HTTP 429 when exceeded.
HTTP Headers
Key-value metadata fields sent with HTTP requests and responses that communicate authentication, content type, caching, and other directives.
Exponential Backoff
Exponential backoff is a retry strategy where the wait time between successive retries increases exponentially, reducing load on rate-limited or temporarily unavailable endpoints.
Retry Logic
Automated re-sending of failed requests with backoff strategies, essential for handling transient errors, rate limits, and flaky anti-bot challenges.
HTTP Status Code
HTTP status codes are three-digit numbers in server responses that indicate the outcome of a request — success, redirection, client error, or server error.

Extract Rate Limit Headers data from any website

AlterLab returns clean, structured data from any public URL — no scraper infrastructure needed. Start free, no credit card required.

View API docs
Back to Glossary

Your first scrape.
Sixty seconds.

$1 free balance. No credit card. No SDK.Just a POST request.

terminal
curl -X POST https://api.alterlab.io/v1/scrape \
-H "X-API-Key: YOUR_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com", "formats": ["markdown"]}'

No credit card required · Up to 5,000 free scrapes · Balance never expires

    Rate Limit Headers — Web Scraping Glossary | AlterLab