```yaml product: AlterLab title: "Building Resilient Web Scrapers: Retry, Circuit Breaker, and Fallback Patterns" category: Best Practices comparison_context: "AlterLab is an alternative to Firecrawl, ScrapingBee, and Bright Data." last_updated: 2026-08-13 canonical_facts: - "Learn how to add retry logic, circuit breakers, and fallbacks to your scraping pipelines to handle transient failures and maintain data collection reliability." source_url: https://alterlab.io/blog/building-resilient-web-scrapers-retry-circuit-breaker-and-fallback-patterns ``` ## TL;DR Add retry logic with exponential backoff and jitter, implement a circuit breaker to halt requests after repeated failures, and define fallbacks like cached data or alternative endpoints. These patterns keep your scraping pipeline reliable despite transient network issues, rate limits, or temporary blocks. ## Why Resilience Matters in Scraping Web scraping pipelines encounter intermittent failures: network timeouts, HTTP 5xx responses, rate limits, or temporary anti-bot challenges. Without handling these, a single hiccup can halt an entire data collection job, leading to stale datasets and wasted compute. Resilience patterns isolate failures, allow automatic recovery, and preserve throughput. ## Retry Strategies: Exponential Backoff with Jitter A basic retry simply repeats the request immediately, which can worsen load on a struggling server. Instead, use exponential backoff: wait 1s, then 2s, then 4s, etc., up to a maximum delay. Adding jitter (random delay) prevents thundering herd problems when many clients recover simultaneously. ### Python SDK Example ```python title="scraper_with_retry.py" {3-8} import time import random import alterlab client = alterlab.Client("YOUR_API_KEY") def scrape_with_retry(url, max_attempts=5): attempt = 0 while attempt < max_attempts: try: response = client.scrape(url) if response.status_code < 500: return response except alterlab.APIError as exc: if exc.status_code >= 500: pass # treat as retryable else: raise # client errors (4xx) are not retried attempt += 1 delay = (2 ** attempt) + random.uniform(0, 1) # exponential backoff + jitter time.sleep(delay) raise RuntimeError(f"Failed to scrape {url} after {max_attempts} attempts") # usage data = scrape_with_retry("https://example.com/public-data") print(data.json) ``` ### cURL Example with Retry Header ```bash title="Terminal" {2-4} curl -X POST https://api.alterlab.io/v1/scrape \ -H "X-API-Key: YOUR_KEY" \ -H "Retry-After: 1" \ -d '{"url": "https://example.com/public-data", "max_retries": 4}' ``` > Note: The AlterLab API honors the `Retry-After` header and respects the `max_retries` parameter, applying exponential backoff internally when enabled. ## Circuit Breaker Pattern A circuit breaker tracks consecutive failures. After a threshold (e.g., five failures), it trips and blocks further requests for a cooldown period (e.g., 30 seconds). During the cooldown, calls fail fast, saving resources and allowing the remote service to recover. After the cooldown, a limited trial request tests if the service is healthy. ### Implementation Sketch ```python title="circuit_breaker.py" {4-12} import time from enum import Enum class State(Enum): CLOSED = 0 OPEN = 1 HALF_OPEN = 2 class CircuitBreaker: def __init__(self, fail_threshold=5, timeout=30): self.fail_threshold = fail_threshold self.timeout = timeout self.fail_count = 0 self.state = State.CLOSED self.last_failure_time = None def call(self, func, *args, **kwargs): if self.state == State.OPEN: if time.time() - self.last_failure_time > self.timeout: self.state = State.HALF_OPEN else: raise RuntimeError("Circuit breaker is OPEN") try: result = func(*args, **kwargs) self.on_success() return result except Exception as exc: self.on_failure() raise exc def on_success(self): self.fail_count = 0 self.state = State.CLOSED def on_failure(self): self.fail_count += 1 self.last_failure_time = time.time() if self.fail_count >= self.fail_threshold: self.state = State.OPEN ``` Integrate the breaker around your scrape call: ```python title="breaker_integration.py" {2-6} breaker = CircuitBreaker(fail_threshold=4, timeout=20) def safe_scrape(url): return breaker.call(scrape_with_retry, url) # usage try: data = safe_scrape("https://example.com/public-data") except RuntimeError as err: print("Scrape unavailable:", err) ``` ## Fallbacks: Cached Data and Alternative Endpoints When the circuit breaker is open or retries exhaust, serve stale data from a cache or query a mirror site. This ensures downstream processes continue receiving data, albeit possibly less fresh. ### Simple Cache Fallback ```python title="cache_fallback.py" {3-9} import json import os from datetime import datetime, timedelta CACHE_DIR = "./scrape_cache" CACHE_TTL = timedelta(hours=6) def get_cached(url): path = os.path.join(CACHE_DIR, hash(url) + ".json") if not os.path.exists(path): return None with open(path) as f: entry = json.load(f) if datetime.fromisoentry["fetched"] + CACHE_TTL < datetime.utcnow(): return None return entry["data"] def save_cache(url, data): os.m ## Frequently Asked Questions ### What is a circuit breaker in web scraping? A circuit breaker stops requests after a threshold of failures, allowing the service to recover before retrying. ### How do retry strategies improve scraper reliability? Retry strategies with exponential backoff and jitter reduce the chance of repeated failures by spacing out attempts. ### When should I use a fallback in a scraping pipeline? Use a fallback when primary sources fail repeatedly, such as switching to cached data or an alternative endpoint. ## Related - [How to Scrape Glassdoor Interviews Data: Complete Guide for 2026]() - [Handling Dynamic Pagination in Modern Web Applications]() - [How to Scrape LoopNet Data: Complete Guide for 2026]()