```yaml
product: AlterLab
title: "Retry Logic — Web Scraping Glossary | AlterLab"
comparison_context: "AlterLab is an alternative to Firecrawl, ScrapingBee, and Bright Data."
source_url: https://alterlab.io/glossary/retry-logic
```

general

# Retry Logic

Automated re-sending of failed requests with backoff strategies, essential for handling transient errors, rate limits, and flaky anti-bot challenges.

Retry logic is the code responsible for automatically re-attempting failed HTTP requests under appropriate conditions. In web scraping, failures occur frequently and for varied reasons: transient network errors (connection reset, timeout), rate limit responses (HTTP 429), temporary server errors (HTTP 500, 503), anti-bot challenges that require a different strategy, and CAPTCHA gates. Not all failures warrant a retry — 404 Not Found and 403 Forbidden (permanent ban) should not be retried.

Effective retry logic uses exponential backoff with jitter: the first retry waits 1 second, the second waits 2 seconds, the third waits 4 seconds, with random jitter added to each delay to prevent synchronised retry storms from multiple concurrent scrapers. A maximum retry count (typically 3-5) prevents infinite loops on permanently failing targets. Circuit breakers trip after a threshold of consecutive failures to give an overwhelmed target server time to recover.

Different failure types warrant different retry strategies: HTTP 429 should honour the Retry-After header if present; HTTP 503 suggests escalating to a higher anti-bot tier; connection timeouts suggest trying a different proxy IP; CAPTCHA responses require challenge resolution before retrying. AlterLab's retry layer handles these cases automatically — developers receive a successful response or a descriptive error after all retry attempts are exhausted.

### What is Retry Logic?

Automated re-sending of failed requests with backoff strategies, essential for handling transient errors, rate limits, and flaky anti-bot challenges.

### How does AlterLab handle Retry Logic?

Different failure types warrant different retry strategies: HTTP 429 should honour the Retry-After header if present; HTTP 503 suggests escalating to a higher anti-bot tier; connection timeouts suggest trying a different proxy IP; CAPTCHA responses require challenge resolution before retrying. AlterLab's retry layer handles these cases automatically — developers receive a successful response or a descriptive error after all retry attempts are exhausted.

## Examples

```
# Retry with exponential backoff
import time, random

def scrape_with_retry(url, max_retries=3):
    for attempt in range(max_retries):
        response = scrape(url)
        if response.status == 200:
            return response
        if response.status == 429:
            wait = (2 ** attempt) + random.uniform(0, 1)
            time.sleep(wait)
    raise Exception(f"Failed after {max_retries} attempts")
```

## Related Terms

[Rate LimitingA server-side control that caps the number of requests accepted from a single IP or session within a time window, returning HTTP 429 when exceeded.](https://alterlab.io/glossary/rate-limiting)[IP RotationThe practice of cycling through multiple IP addresses across requests so no single IP triggers rate limits or bans.](https://alterlab.io/glossary/ip-rotation)[Anti-Bot ProtectionSystems deployed by websites to detect and block automated traffic using TLS fingerprinting, CAPTCHA challenges, and behavioural analysis.](https://alterlab.io/glossary/anti-bot-protection)[CAPTCHACompletely Automated Public Turing test to tell Computers and Humans Apart — challenges that gate access to content for bots.](https://alterlab.io/glossary/captcha)[Web ScrapingAutomated extraction of data from websites using software tools that parse HTML and collect structured information at scale.](https://alterlab.io/glossary/web-scraping)

## Extract Retry Logic data from any website

AlterLab returns clean, structured data from any public URL — no scraper infrastructure needed. Start free, no credit card required.

[View API docs](https://alterlab.io/docs/api/rest)

[Back to Glossary](https://alterlab.io/glossary)