How to Scrape Without Getting Blocked: The Complete Guide

You write a scraper. It works for a day. Then you start getting 403s, challenge pages, or empty responses. Your IP is flagged, your browser environment is easy to fingerprint, and the data pipeline you built stops working.

Getting blocked is the most common reason scraping projects fail. Not the parsing logic. Not the data model. The blocks.

This guide covers the techniques that keep a scraper running reliably, from the basics to browser environment management. We will also be honest about when it makes more sense to use a scraping API instead of building all this yourself.

Why Sites Block Scrapers

Before fixing the problem, understand what you are actually working around.

Websites deploy automated compatibility checks for legitimate reasons: preventing abuse, protecting server resources, guarding proprietary data, and stopping credential stuffing attacks. A scraper that looks identical to abusive traffic gets caught in the same net.

The checks websites commonly run include:

1. IP-level analysis - rate, reputation, datacenter vs residential
2. HTTP signature analysis - headers, TLS signature, protocol behavior
3. Browser environment checks - JavaScript APIs, rendering behavior
4. Behavioral analysis - timing patterns, navigation flow
5. Challenge pages - proof-of-humanity gates

Each layer catches a different class of automated traffic. A reliable scraper needs to look legitimate on all of them.

Level 1: HTTP Headers

The lowest-hanging fruit. Most beginners send requests with default library headers that immediately read as a script.

The Problem

import requests

# This sends headers like:
# User-Agent: python-requests/2.31.0
# Accept: */*
# Accept-Encoding: gzip, deflate
response = requests.get("https://target-site.com")

No real browser sends python-requests/2.31.0 as its User-Agent. Many sites reject this instantly.

The Fix

import requests

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
    "Accept-Encoding": "gzip, deflate, br",
    "Cache-Control": "max-age=0",
    "Sec-Ch-Ua": "\"Chromium\";v=\"122\", \"Not(A:Brand\";v=\"24\", \"Google Chrome\";v=\"122\"",
    "Sec-Ch-Ua-Mobile": "?0",
    "Sec-Ch-Ua-Platform": "\"Windows\"",
    "Sec-Fetch-Dest": "document",
    "Sec-Fetch-Mode": "navigate",
    "Sec-Fetch-Site": "none",
    "Sec-Fetch-User": "?1",
    "Upgrade-Insecure-Requests": "1",
}

response = requests.get("https://target-site.com", headers=headers)

Key points:

• Match ALL headers a real browser sends, not just User-Agent
• Include the Sec-Ch-Ua headers - modern Chrome sends these and their absence is a signal
• Keep your User-Agent current - using a two-year-old Chrome version when a current one is standard is a mismatch
• Rotate User-Agent strings but keep the header set consistent with the claimed browser

Level 2: TLS and Protocol Signature

This is where most developers get stuck. Even with perfect headers, your HTTP client's TLS handshake looks different from a real browser.

The Problem

Every TLS client sends a ClientHello message with a specific combination of cipher suites, extensions, and supported groups. Python's requests library (which uses urllib3 and OpenSSL) has a completely different TLS signature than Chrome, and automated systems can key off that difference alone, before your headers are even read.

The Fix

from curl_cffi import requests as cffi_requests

# curl_cffi uses a patched libcurl that mirrors real browser TLS behavior
response = cffi_requests.get(
    "https://target-site.com",
    impersonate="chrome"
)

Alternatively, use tls-client or run an actual browser via Playwright/Puppeteer. These send genuine browser TLS handshakes.

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=False)
    context = browser.new_context()
    page = context.new_page()
    response = page.goto("https://target-site.com")
    html = page.content()
    browser.close()

Level 3: Proxy Rotation

The same IP making hundreds of requests per minute is one of the most obvious signals there is.

IP Types Matter

Datacenter proxies come from cloud providers. Many sites maintain lists of datacenter IP ranges. These get caught fastest but they are cheap.

Residential proxies come from real ISP connections. They look like normal home internet users. Much harder to flag but cost 10-20x more per GB.

Mobile proxies come from cellular networks. CGNAT means many real users share these IPs, which makes them extremely hard to block without also blocking real mobile users.

Smart Rotation Strategy

Do not rotate proxies on every request. That itself is a signal - real users keep the same IP for a session.

import random
import time

class SmartProxyRotator:
    def __init__(self, proxy_list):
        self.proxies = proxy_list
        self.current = random.choice(proxy_list)
        self.request_count = 0
        self.max_per_ip = random.randint(15, 30)

    def get_proxy(self):
        self.request_count += 1
        if self.request_count >= self.max_per_ip:
            self.current = random.choice(self.proxies)
            self.request_count = 0
            self.max_per_ip = random.randint(15, 30)
        return self.current

Level 4: Rate Limiting and Timing

Humans do not make requests at machine speed. A 50ms gap between page loads is not something a real user browsing a website produces.

Natural Timing Patterns

import random
import time

def human_delay():
    """Generate a delay that looks like human browsing."""
    # Base delay: 2-5 seconds (reading time)
    base = random.uniform(2.0, 5.0)
    # Occasional longer pause (checking something)
    if random.random() < 0.1:
        base += random.uniform(5.0, 15.0)
    # Small random jitter
    base += random.gauss(0, 0.3)
    return max(0.5, base)

for url in urls_to_scrape:
    response = scrape(url)
    time.sleep(human_delay())

Request Rate Guidelines

• Conservative: 1 request every 3-5 seconds per IP
• Moderate: 1 request every 1-2 seconds per IP
• Aggressive: Multiple requests per second (requires proxy rotation)

Most sites will tolerate moderate speeds from residential IPs. Datacenter IPs should stick to conservative rates.

Level 5: Browser Environment Management

When sites run JavaScript-based checks, they collect detailed browser information. A default headless browser configuration leaks signals that give away automation.

Common Detection Vectors

// Automated checks commonly read these
navigator.webdriver          // true in automation mode
navigator.languages          // empty or wrong in headless
navigator.plugins            // empty array in headless
navigator.hardwareConcurrency // 0 or 2 in headless (real: 4-16)
window.chrome                // missing in headless
Notification.permission      // "denied" in headless (real: "default")

The Fix: Configure a Realistic Browser Environment

# For Playwright
from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(
        headless=False,  # headed mode passes more checks
        args=[
            "--disable-blink-features=AutomationControlled",
            "--no-sandbox",
        ]
    )
    context = browser.new_context(
        viewport={"width": 1920, "height": 1080},
        locale="en-US",
        timezone_id="America/New_York",
    )

    # Inject environment patches before any page loads
    context.add_init_script("""
        Object.defineProperty(navigator, 'webdriver', {
            get: () => undefined
        });
        Object.defineProperty(navigator, 'plugins', {
            get: () => [1, 2, 3, 4, 5]
        });
    """)

    page = context.new_page()
    page.goto("https://target-site.com")

For Node.js, puppeteer-extra-plugin-stealth patches most of these automatically (despite the package name, this is browser environment configuration, not a network-layer technique).

Level 6: Session and Cookie Management

Many sites track sessions. A request without cookies to a page that should have cookies from a previous visit is a mismatch worth avoiding.

Maintain Sessions

import requests

session = requests.Session()

# Visit homepage first (get cookies)
session.get("https://target-site.com")

# Now visit product pages with cookies set
for product_url in product_urls:
    response = session.get(product_url)
    # Cookies are automatically maintained

Handle Cookie-Based Challenges

Some sites set trust cookies via JavaScript after a client-side check completes. You need a real browser to satisfy these:

# Get cookies from browser, use in requests
from playwright.sync_api import sync_playwright
import requests

with sync_playwright() as p:
    browser = p.chromium.launch(headless=False)
    page = browser.new_page()
    page.goto("https://target-site.com")
    page.wait_for_timeout(5000)  # Wait for the check to resolve

    # Extract cookies
    cookies = page.context.cookies()
    browser.close()

# Transfer cookies to requests session
session = requests.Session()
for cookie in cookies:
    session.cookies.set(cookie["name"], cookie["value"])

# Now use the session for fast HTTP requests
for url in urls:
    response = session.get(url, headers=headers)

Level 7: Behavioral Patterns

More advanced site protections analyze how you navigate. Going directly to deep URLs without visiting the homepage first is a pattern real users rarely produce.

Navigation Flow

import random

def natural_crawl(session, base_url, target_urls):
    """Crawl with natural navigation patterns."""
    # Always start at homepage
    session.get(base_url)
    time.sleep(human_delay())

    # Visit a category page first
    categories = ["/products", "/catalog", "/shop"]
    session.get(base_url + random.choice(categories))
    time.sleep(human_delay())

    # Shuffle target URLs (do not crawl in order)
    random.shuffle(target_urls)

    for url in target_urls:
        response = session.get(url)
        time.sleep(human_delay())

        # Occasionally visit a non-target page
        if random.random() < 0.2:
            session.get(base_url + "/about")
            time.sleep(human_delay())

When to Use a Scraping API Instead

Building the full compatibility stack yourself is educational but expensive in engineering time. Here is a realistic assessment of when each approach makes sense.

Build It Yourself When:

• You scrape fewer than 5K pages per month
• Your targets have minimal automated protection
• You have engineering time to maintain the infrastructure
• You need full control over the browser and network layer

Use a Scraping API When:

• You scrape more than 10K pages per month
• Your targets run aggressive automated compatibility checks
• You do not want to maintain proxy rotation and browser environment patching
• Your engineering time is better spent on data processing, not scraping infrastructure

The math usually tips toward an API when you factor in engineering hours. A senior developer spending 10 hours per month maintaining a custom scraping stack costs more than most API subscriptions.

Using AlterLab to Handle All Seven Levels

curl -X POST https://api.alterlab.io/api/v1/scrape \
  -H "X-API-Key: your_api_key" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://target-site.com/product/123"}'

from alterlab import AlterLabSync

with AlterLabSync(api_key="your_api_key") as client:
    # AlterLab handles headers, TLS, proxies, browser environment, sessions
    result = client.scrape("https://target-site.com/product/123")
    print(result["content"])

    # Need structured data? Add a schema
    result = client.scrape(
        "https://target-site.com/product/123",
        extraction_schema={
            "type": "object",
            "properties": {
                "name": {"type": "string"},
                "price": {"type": "number"},
            },
        },
    )
    print(result["extraction"])

import { AlterLab } from 'alterlab';

const client = new AlterLab({ apiKey: 'your_api_key' });

const result = await client.scrape({
  url: 'https://target-site.com/product/123',
  extraction_schema: {
    type: 'object',
    properties: { name: { type: 'string' }, price: { type: 'number' } },
  },
});
console.log(result.extraction);

Quick Reference: Compatibility Checklist

Before running your scraper at scale, verify each layer:

• Headers: Complete browser header set including Sec-Ch-Ua
• TLS: Using curl_cffi, tls-client, or a real browser
• Proxies: Residential for moderate targets, mobile for hard targets
• Rate limiting: 1-5 second delays with random jitter
• Browser environment: navigator.webdriver patched, plugins populated
• Sessions: Cookies maintained across requests
• Behavior: Natural navigation flow, not direct deep links

Conclusion

Getting blocked is not inevitable. Each layer has a known, legitimate way to stay compatible with it. The question is whether the engineering investment is worth it for your use case and volume.

For small-scale scraping, the techniques in this guide will keep you running. For production-scale data collection, the maintenance burden of the full stack becomes a real cost center. That is where scraping APIs earn their keep.

Try AlterLab free and skip the infrastructure maintenance. $1 starting credit, no monthly commitment.