" in response.text, "content_length": len(response.text), "latency_ms": response.elapsed.total_seconds() * 1000, } if not result["has_price"] or not result["has_title"]: logger.warning("Content validation failed: %s", url) return result ``` Log these metrics to your monitoring system. Set alerts on threshold breaches. Catching a block within minutes saves hours of missed data. ## Cost Considerations Running your own scraping infrastructure has hidden costs: - Proxy subscriptions: $50-500/month depending on pool size and type - CAPTCHA solving: $2-5 per 1000 solves - Server costs for headless browser instances - Engineering time for maintenance and debugging - Data loss from undetected failures A managed API converts these variable costs into a predictable per-request cost. You pay for successful scrapes, not for infrastructure that might fail. Check [pricing](https://alterlab.io/pricing) to model costs against your expected volume. ## Takeaway Building resilient scraping infrastructure requires handling proxy rotation, CAPTCHA solving, TLS fingerprinting, browser fingerprinting, rate limiting, and session management. Each layer adds complexity and failure modes. The practical path: use a managed API that handles anti-bot detection, proxy rotation, and CAPTCHA solving automatically. Focus your engineering time on data processing and pipeline reliability, not on staying one step ahead of bot detection systems. Start with the [quickstart guide](https://alterlab.io/docs/quickstart/installation) to get an API key and run your first scrape. The [documentation](https://alterlab.io/docs) covers advanced parameters for JavaScript rendering, scheduling, and structured data extraction.</article></body></html>

## The Problem With Building Your Own Proxy Infrastructure You spin up a scraper. It works for 200 requests. Then the target site starts returning 403s. Then CAPTCHAs. Then your IP gets banned entirely. The standard response: buy a proxy list, wire up rotation logic, add CAPTCHA solving, implement retry logic with exponential backoff, handle fingerprinting, manage session cookies, and deal with geographic targeting. That is a full-time infrastructure project before you scrape a single page of useful data. Most teams do not need to build this. They need data. This guide covers the patterns that work for production scraping at scale, with code you can deploy today. ## How Bot Detection Works in 2026 Modern anti-bot systems layer multiple signals. Understanding what they check tells you what your scraper needs to handle. **IP reputation.** Datacenter IP ranges are flagged. AWS, GCP, and Azure subnets are well-known. Sites maintain blocklists updated in real time from threat intelligence feeds. **TLS fingerprinting.** The JA3/JA4 hash of your TLS handshake reveals your HTTP client. Python requests, Go net/http, and raw curl each produce distinct fingerprints. Headless browsers have their own signatures. **Browser fingerprinting.** Canvas rendering, WebGL vendor strings, font enumeration, audio context, and screen resolution combine into a fingerprint that distinguishes automated browsers from real users. **Behavioral analysis.** Mouse movement patterns, scroll velocity, click timing, and navigation sequences are tracked. Bots move in straight lines. Humans do not. **CAPTCHA challenges.** reCAPTCHA v3 runs silently and scores each visitor. hCaptcha and Cloudflare Turnstile present visual puzzles when your score drops below threshold. **Rate limiting.** Request frequency per IP, per session, and per account is tracked. Burst patterns trigger blocks faster than steady traffic. A scraper that handles only one of these signals will fail. You need a system that addresses all of them simultaneously. ## Proxy Rotation: The Right Way ### Why Single-IP Scraping Fails Every request from the same IP to the same domain creates a pattern. After a threshold, the target site flags the IP. The threshold varies: some sites block after 50 requests per minute, others after 500 per hour. E-commerce sites during product launches are the most aggressive. ### Residential vs Datacenter vs Mobile Proxies

Proxy Type	Cost	Success Rate	Speed	Best For
Datacenter	$	40-60%	Fast	Low-security targets
Residential	$$	80-95%	Medium	Most production scraping
Mobile	$$$	95-99%	Slow	High-security targets

Datacenter proxies are cheap and fast but get flagged quickly. Residential proxies route through real ISP connections and blend in with normal traffic. Mobile proxies use carrier networks and have the highest success rate but cost more and add latency. The practical approach: start with residential proxies, escalate to mobile only when the target requires it. ### Implementing Proxy Rotation If you manage your own proxy pool, rotation logic looks like this: ```python title="proxy_rotator.py" {8-15} import random import requests from collections import deque class ProxyRotator: def __init__(self, proxy_list: list[str]): self.proxies = deque(proxy_list) def rotate(self) -> dict: self.proxies.rotate(1) proxy = self.proxies[0] return {"http": proxy, "https": proxy} def remove_bad(self, proxy: str): if proxy in self.proxies: self.proxies.remove(proxy) rotator = ProxyRotator([ "http://user:pass@proxy1:8080", "http://user:pass@proxy2:8080", "http://user:pass@proxy3:8080", ]) for url in urls_to_scrape: proxy = rotator.rotate() try: response = requests.get(url, proxies=proxy, timeout=10) response.raise_for_status() except requests.exceptions.RequestException: rotator.remove_bad(proxy["http"]) ``` This works for simple cases. It does not handle TLS fingerprinting, browser fingerprinting, CAPTCHAs, or behavioral detection. For those, you need a higher-level solution.

## CAPTCHA Solving at Scale ### Types of CAPTCHAs You Will Encounter **reCAPTCHA v2.** The classic "I'm not a robot" checkbox. Requires solving image puzzles or selecting objects. Solvable via third-party services with 15-30 second latency. **reCAPTCHA v3.** Runs silently. Returns a score from 0.0 to 1.0. Sites set a threshold, usually 0.5. Below that, you get blocked or redirected. No puzzle to solve, just a score to maintain. **hCaptcha.** Similar to reCAPTCHA v2 but with different image sets. Popular among sites that want an alternative to Google's ecosystem. **Cloudflare Turnstile.** Newer, privacy-focused. Presents challenges based on browser behavior rather than image puzzles. Harder to bypass without a real browser environment. **Custom CAPTCHAs.** Some sites build their own. Math problems, text recognition, or logic puzzles. These require custom solving logic. ### The CAPTCHA Solving Pipeline A production CAPTCHA solver works in four stages:

Building this pipeline yourself means integrating with solving services like 2Captcha or Anti-Captcha, handling their APIs, managing solve latency, and retrying on failures. Each CAPTCHA type requires different extraction and submission logic. ### Using a Managed Anti-Bot Bypass The alternative is using a service that handles all of this automatically. The [anti-bot bypass API](https://alterlab.io/anti-bot-bypass-api) detects CAPTCHAs, solves them, and returns the page content without your code needing to know a CAPTCHA existed. ```python title="scraper.py" {3-5} import alterlab client = alterlab.Client("YOUR_API_KEY") response = client.scrape("https://example.com/protected-page") print(response.text) ``` The request goes through a headless browser environment with rotating residential IPs, realistic TLS fingerprints, and automatic CAPTCHA solving. You get back the rendered HTML or clean JSON. No proxy management, no CAPTCHA integration, no fingerprint handling. ## Production Scraping Patterns ### Exponential Backoff with Jitter Blind retries make blocking worse. Use exponential backoff with random jitter to avoid thundering herd problems. ```python title="retry_with_backoff.py" {6-12} import time import random import requests def scrape_with_backoff(url: str, max_retries: int = 5): for attempt in range(max_retries): try: response = requests.get(url, timeout=15) response.raise_for_status() return response.text except requests.exceptions.RequestException as e: if attempt == max_retries - 1: raise wait = min(2 ** attempt + random.uniform(0, 1), 60) time.sleep(wait) ``` The jitter prevents multiple scraper instances from retrying simultaneously. The cap at 60 seconds prevents unbounded waits. ### Session Management Maintaining sessions reduces detection risk. Reusing cookies and connection pools looks more like a real browser than opening fresh connections for every request. ```python title="session_scraper.py" {4-7} import requests session = requests.Session() session.headers.update({ "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36", "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "Accept-Language": "en-US,en;q=0.5", }) # First request establishes session response = session.get("https://example.com/login") csrf_token = extract_csrf(response.text) # Second request reuses cookies and connection response = session.post("https://example.com/login", data={ "csrf_token": csrf_token, "username": "user", "password": "pass", }) ``` ### Parallel Scraping with Rate Limits Scraping one page at a time is slow. Scraping 100 pages at once gets you blocked. The middle ground: controlled concurrency with per-domain rate limiting. ```python title="parallel_scraper.py" {10-14} import asyncio import aiohttp from asyncio import Semaphore async def scrape_with_limit(urls: list[str], max_concurrent: int = 5): semaphore = Semaphore(max_concurrent) results = [] async def scrape_one(session: aiohttp.ClientSession, url: str): async with semaphore: try: async with session.get(url, timeout=15) as resp: return await resp.text() except Exception: return None async with aiohttp.ClientSession() as session: tasks = [scrape_one(session, url) for url in urls] results = await asyncio.gather(*tasks) return results ``` The semaphore caps concurrent requests. Adjust `max_concurrent` based on the target site's tolerance. Start at 3, monitor response codes, increase gradually. ## When to Use Headless Browsers Some sites require JavaScript execution to render content. Static HTTP requests return empty pages or login walls. Headless browsers handle this but add complexity. **Use headless browsers when:** - Content renders client-side via React, Vue, or Angular - The site requires JavaScript to set authentication cookies - You need to interact with the page (click, scroll, fill forms) - The site uses WebSocket connections for data **Skip headless browsers when:** - The page returns complete HTML on initial load - You only need API response data (intercept network calls instead) - Speed is critical and the target serves server-rendered HTML Running headless browsers at scale requires managing browser instances, memory, and crash recovery. A managed [web scraping API](https://alterlab.io/web-scraping-api-python) handles browser lifecycle, memory limits, and crash recovery automatically. ```bash title="Terminal" curl -X POST https://api.alterlab.io/v1/scrape \ -H "X-API-Key: YOUR_KEY" \ -d '{ "url": "https://example.com/js-heavy-page", "render_js": true, "wait_for": ".content-loaded", "formats": ["json"] }' ``` The `render_js` parameter spins up a headless browser. `wait_for` pauses until a CSS selector appears. `formats` returns structured data instead of raw HTML. ## Monitoring and Alerting Scraping pipelines fail silently. A site changes its layout, adds a new CAPTCHA type, or starts blocking your proxy range. Your scraper returns empty results and nobody notices for weeks. Set up monitoring at three levels: **Response codes.** Track 200 vs 403 vs 429 ratios. A spike in 403s means your proxies are burning. A spike in 429s means you are hitting rate limits. **Content validation.** Check that responses contain expected data. If you scrape product pages, verify each response has a price, title, and description. Empty fields mean the page structure changed. **Latency tracking.** Sudden latency increases often mean the site added new anti-bot checks or your proxies are routing through congested nodes. ```python title="monitoring.py" {12-18} import requests import logging logger = logging.getLogger("scraper.monitor") def scrape_and_validate(url: str) -> dict: response = requests.get(url, timeout=15) result = { "url": url, "status_code": response.status_code, "has_price": "price" in response.text, "has_title": "" in response.text, "content_length": len(response.text), "latency_ms": response.elapsed.total_seconds() * 1000, } if not result["has_price"] or not result["has_title"]: logger.warning("Content validation failed: %s", url) return result ``` Log these metrics to your monitoring system. Set alerts on threshold breaches. Catching a block within minutes saves hours of missed data. ## Cost Considerations Running your own scraping infrastructure has hidden costs: - Proxy subscriptions: $50-500/month depending on pool size and type - CAPTCHA solving: $2-5 per 1000 solves - Server costs for headless browser instances - Engineering time for maintenance and debugging - Data loss from undetected failures A managed API converts these variable costs into a predictable per-request cost. You pay for successful scrapes, not for infrastructure that might fail. Check [pricing](https://alterlab.io/pricing) to model costs against your expected volume. ## Takeaway Building resilient scraping infrastructure requires handling proxy rotation, CAPTCHA solving, TLS fingerprinting, browser fingerprinting, rate limiting, and session management. Each layer adds complexity and failure modes. The practical path: use a managed API that handles anti-bot detection, proxy rotation, and CAPTCHA solving automatically. Focus your engineering time on data processing and pipeline reliability, not on staying one step ahead of bot detection systems. Start with the [quickstart guide](https://alterlab.io/docs/quickstart/installation) to get an API key and run your first scrape. The [documentation](https://alterlab.io/docs) covers advanced parameters for JavaScript rendering, scheduling, and structured data extraction.</article></body></html>

#CAPTCHA

How to Rotate Proxies and Solve CAPTCHAs at Scale in 2026