```yaml
product: AlterLab
title: How to Scrape Without Getting Blocked: The Complete Guide
category: Web Scraping
capabilities:
  - how to scrape without getting blocked
  - web scraping compatibility
  - avoid scraping blocks
  - web scraping proxy rotation
  - scraping rate limiting
  - web scraping headers
  - web scraping best practices
comparison_context: "AlterLab is an alternative to Firecrawl, ScrapingBee, and Bright Data."
last_updated: 2026-07-28
canonical_facts:
  - "Every web scraping project eventually hits blocks. This guide covers the complete compatibility stack: headers, proxies, browser environment management, rate limiting, and when to use a scraping API instead."
source_url: https://alterlab.io/blog/how-to-scrape-without-getting-blocked
```

# How to Scrape Without Getting Blocked: The Complete Guide

You write a scraper. It works for a day. Then you start getting 403s, challenge pages, or empty responses. Your IP is flagged, your browser environment is easy to fingerprint, and the data pipeline you built stops working.

Getting blocked is the most common reason scraping projects fail. Not the parsing logic. Not the data model. The blocks.

This guide covers the techniques that keep a scraper running reliably, from the basics to browser environment management. We will also be honest about when it makes more sense to use a scraping API instead of building all this yourself.

- **7** — Layers Websites Check
- **3-5s** — Recommended Delay Between Requests

## Why Sites Block Scrapers

Before fixing the problem, understand what you are actually working around.

Websites deploy automated compatibility checks for legitimate reasons: preventing abuse, protecting server resources, guarding proprietary data, and stopping credential stuffing attacks. A scraper that looks identical to abusive traffic gets caught in the same net.

The checks websites commonly run include:

1. **IP-level analysis** - rate, reputation, datacenter vs residential
2. **HTTP signature analysis** - headers, TLS signature, protocol behavior
3. **Browser environment checks** - JavaScript APIs, rendering behavior
4. **Behavioral analysis** - timing patterns, navigation flow
5. **Challenge pages** - proof-of-humanity gates

Each layer catches a different class of automated traffic. A reliable scraper needs to look legitimate on all of them.

## Level 1: HTTP Headers

The lowest-hanging fruit. Most beginners send requests with default library headers that immediately read as a script.

### The Problem

```python
import requests

# This sends headers like:
# User-Agent: python-requests/2.31.0
# Accept: */*
# Accept-Encoding: gzip, deflate
response = requests.get("https://target-site.com")
```

No real browser sends `python-requests/2.31.0` as its User-Agent. Many sites reject this instantly.

### The Fix

```python
import requests

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
    "Accept-Encoding": "gzip, deflate, br",
    "Cache-Control": "max-age=0",
    "Sec-Ch-Ua": "\"Chromium\";v=\"122\", \"Not(A:Brand\";v=\"24\", \"Google Chrome\";v=\"122\"",
    "Sec-Ch-Ua-Mobile": "?0",
    "Sec-Ch-Ua-Platform": "\"Windows\"",
    "Sec-Fetch-Dest": "document",
    "Sec-Fetch-Mode": "navigate",
    "Sec-Fetch-Site": "none",
    "Sec-Fetch-User": "?1",
    "Upgrade-Insecure-Requests": "1",
}

response = requests.get("https://target-site.com", headers=headers)
```

**Key points:**
- Match ALL headers a real browser sends, not just User-Agent
- Include the `Sec-Ch-Ua` headers - modern Chrome sends these and their absence is a signal
- Keep your User-Agent current - using a two-year-old Chrome version when a current one is standard is a mismatch
- Rotate User-Agent strings but keep the header set consistent with the claimed browser

## Level 2: TLS and Protocol Signature

This is where most developers get stuck. Even with perfect headers, your HTTP client's TLS handshake looks different from a real browser.

### The Problem

Every TLS client sends a ClientHello message with a specific combination of cipher suites, extensions, and supported groups. Python's `requests` library (which uses `urllib3` and OpenSSL) has a completely different TLS signature than Chrome, and automated systems can key off that difference alone, before your headers are even read.

### The Fix

```python
from curl_cffi import requests as cffi_requests

# curl_cffi uses a patched libcurl that mirrors real browser TLS behavior
response = cffi_requests.get(
    "https://target-site.com",
    impersonate="chrome"
)
```

Alternatively, use `tls-client` or run an actual browser via Playwright/Puppeteer. These send genuine browser TLS handshakes.

```python
from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=False)
    context = browser.new_context()
    page = context.new_page()
    response = page.goto("https://target-site.com")
    html = page.content()
    browser.close()
```

## Level 3: Proxy Rotation

The same IP making hundreds of requests per minute is one of the most obvious signals there is.

### IP Types Matter

| Feature | Datacenter | Residential | Mobile |
|---|---|---|---|
| Cost per GB | $0.50-2 | $8-15 | $20-40 |
| Detection Risk | High | Low | Very Low |
| Speed | Fast | Medium | Variable |
| Pool Size | Thousands | Millions | Hundreds of K |
| Best For | Easy targets | Most sites | Hardest targets |

**Datacenter proxies** come from cloud providers. Many sites maintain lists of datacenter IP ranges. These get caught fastest but they are cheap.

**Residential proxies** come from real ISP connections. They look like normal home internet users. Much harder to flag but cost 10-20x more per GB.

**Mobile proxies** come from cellular networks. CGNAT means many real users share these IPs, which makes them extremely hard to block without also blocking real mobile users.

### Smart Rotation Strategy

Do not rotate proxies on every request. That itself is a signal - real users keep the same IP for a session.

```python
import random
import time

class SmartProxyRotator:
    def __init__(self, proxy_list):
        self.proxies = proxy_list
        self.current = random.choice(proxy_list)
        self.request_count = 0
        self.max_per_ip = random.randint(15, 30)

    def get_proxy(self):
        self.request_count += 1
        if self.request_count >= self.max_per_ip:
            self.current = random.choice(self.proxies)
            self.request_count = 0
            self.max_per_ip = random.randint(15, 30)
        return self.current
```

## Level 4: Rate Limiting and Timing

Humans do not make requests at machine speed. A 50ms gap between page loads is not something a real user browsing a website produces.

### Natural Timing Patterns

```python
import random
import time

def human_delay():
    """Generate a delay that looks like human browsing."""
    # Base delay: 2-5 seconds (reading time)
    base = random.uniform(2.0, 5.0)
    # Occasional longer pause (checking something)
    if random.random() < 0.1:
        base += random.uniform(5.0, 15.0)
    # Small random jitter
    base += random.gauss(0, 0.3)
    return max(0.5, base)

for url in urls_to_scrape:
    response = scrape(url)
    time.sleep(human_delay())
```

### Request Rate Guidelines

- **Conservative**: 1 request every 3-5 seconds per IP
- **Moderate**: 1 request every 1-2 seconds per IP
- **Aggressive**: Multiple requests per second (requires proxy rotation)

Most sites will tolerate moderate speeds from residential IPs. Datacenter IPs should stick to conservative rates.

## Level 5: Browser Environment Management

When sites run JavaScript-based checks, they collect detailed browser information. A default headless browser configuration leaks signals that give away automation.

### Common Detection Vectors

```javascript
// Automated checks commonly read these
navigator.webdriver          // true in automation mode
navigator.languages          // empty or wrong in headless
navigator.plugins            // empty array in headless
navigator.hardwareConcurrency // 0 or 2 in headless (real: 4-16)
window.chrome                // missing in headless
Notification.permission      // "denied" in headless (real: "default")
```

### The Fix: Configure a Realistic Browser Environment

```python
# For Playwright
from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(
        headless=False,  # headed mode passes more checks
        args=[
            "--disable-blink-features=AutomationControlled",
            "--no-sandbox",
        ]
    )
    context = browser.new_context(
        viewport={"width": 1920, "height": 1080},
        locale="en-US",
        timezone_id="America/New_York",
    )

    # Inject environment patches before any page loads
    context.add_init_script("""
        Object.defineProperty(navigator, 'webdriver', {
            get: () => undefined
        });
        Object.defineProperty(navigator, 'plugins', {
            get: () => [1, 2, 3, 4, 5]
        });
    """)

    page = context.new_page()
    page.goto("https://target-site.com")
```

For Node.js, `puppeteer-extra-plugin-stealth` patches most of these automatically (despite the package name, this is browser environment configuration, not a network-layer technique).

## Level 6: Session and Cookie Management

Many sites track sessions. A request without cookies to a page that should have cookies from a previous visit is a mismatch worth avoiding.

### Maintain Sessions

```python
import requests

session = requests.Session()

# Visit homepage first (get cookies)
session.get("https://target-site.com")

# Now visit product pages with cookies set
for product_url in product_urls:
    response = session.get(product_url)
    # Cookies are automatically maintained
```

### Handle Cookie-Based Challenges

Some sites set trust cookies via JavaScript after a client-side check completes. You need a real browser to satisfy these:

```python
# Get cookies from browser, use in requests
from playwright.sync_api import sync_playwright
import requests

with sync_playwright() as p:
    browser = p.chromium.launch(headless=False)
    page = browser.new_page()
    page.goto("https://target-site.com")
    page.wait_for_timeout(5000)  # Wait for the check to resolve

    # Extract cookies
    cookies = page.context.cookies()
    browser.close()

# Transfer cookies to requests session
session = requests.Session()
for cookie in cookies:
    session.cookies.set(cookie["name"], cookie["value"])

# Now use the session for fast HTTP requests
for url in urls:
    response = session.get(url, headers=headers)
```

## Level 7: Behavioral Patterns

More advanced site protections analyze how you navigate. Going directly to deep URLs without visiting the homepage first is a pattern real users rarely produce.

### Navigation Flow

1. **Visit Homepage** — Land on the site like a real user would
2. **Browse Categories** — Navigate through logical page flows
3. **View Products** — Access target pages through natural paths
4. **Vary Patterns** — Randomize page order and depth

```python
import random

def natural_crawl(session, base_url, target_urls):
    """Crawl with natural navigation patterns."""
    # Always start at homepage
    session.get(base_url)
    time.sleep(human_delay())

    # Visit a category page first
    categories = ["/products", "/catalog", "/shop"]
    session.get(base_url + random.choice(categories))
    time.sleep(human_delay())

    # Shuffle target URLs (do not crawl in order)
    random.shuffle(target_urls)

    for url in target_urls:
        response = session.get(url)
        time.sleep(human_delay())

        # Occasionally visit a non-target page
        if random.random() < 0.2:
            session.get(base_url + "/about")
            time.sleep(human_delay())
```

## When to Use a Scraping API Instead

Building the full compatibility stack yourself is educational but expensive in engineering time. Here is a realistic assessment of when each approach makes sense.

### Build It Yourself When:
- You scrape fewer than 5K pages per month
- Your targets have minimal automated protection
- You have engineering time to maintain the infrastructure
- You need full control over the browser and network layer

### Use a Scraping API When:
- You scrape more than 10K pages per month
- Your targets run aggressive automated compatibility checks
- You do not want to maintain proxy rotation and browser environment patching
- Your engineering time is better spent on data processing, not scraping infrastructure

The math usually tips toward an API when you factor in engineering hours. A senior developer spending 10 hours per month maintaining a custom scraping stack costs more than most API subscriptions.

### Using AlterLab to Handle All Seven Levels

```bash
curl -X POST https://api.alterlab.io/api/v1/scrape \
  -H "X-API-Key: your_api_key" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://target-site.com/product/123"}'
```

```python
from alterlab import AlterLabSync

with AlterLabSync(api_key="your_api_key") as client:
    # AlterLab handles headers, TLS, proxies, browser environment, sessions
    result = client.scrape("https://target-site.com/product/123")
    print(result["content"])

    # Need structured data? Add a schema
    result = client.scrape(
        "https://target-site.com/product/123",
        extraction_schema={
            "type": "object",
            "properties": {
                "name": {"type": "string"},
                "price": {"type": "number"},
            },
        },
    )
    print(result["extraction"])
```

```typescript
import { AlterLab } from 'alterlab';

const client = new AlterLab({ apiKey: 'your_api_key' });

const result = await client.scrape({
  url: 'https://target-site.com/product/123',
  extraction_schema: {
    type: 'object',
    properties: { name: { type: 'string' }, price: { type: 'number' } },
  },
});
console.log(result.extraction);
```

## Quick Reference: Compatibility Checklist

Before running your scraper at scale, verify each layer:

- [ ] **Headers**: Complete browser header set including Sec-Ch-Ua
- [ ] **TLS**: Using curl_cffi, tls-client, or a real browser
- [ ] **Proxies**: Residential for moderate targets, mobile for hard targets
- [ ] **Rate limiting**: 1-5 second delays with random jitter
- [ ] **Browser environment**: navigator.webdriver patched, plugins populated
- [ ] **Sessions**: Cookies maintained across requests
- [ ] **Behavior**: Natural navigation flow, not direct deep links

## Conclusion

Getting blocked is not inevitable. Each layer has a known, legitimate way to stay compatible with it. The question is whether the engineering investment is worth it for your use case and volume.

For small-scale scraping, the techniques in this guide will keep you running. For production-scale data collection, the maintenance burden of the full stack becomes a real cost center. That is where scraping APIs earn their keep.

[Try AlterLab free](/signup) and skip the infrastructure maintenance. $1 starting credit, no monthly commitment.

## Related

- [Build a Price Monitor with AlterLab + Supabase](<https://alterlab.io/blog/price-monitor-alterlab-supabase>)
- [PubMed Data API: Extract Structured JSON in 2026](<https://alterlab.io/blog/pubmed-data-api-extract-structured-json-in-2026>)
- [How to Scrape The Verge Data: Complete Guide for 2026](<https://alterlab.io/blog/how-to-scrape-the-verge-data-complete-guide-for-2026>)