```yaml
product: AlterLab
title: "Concurrent Crawling — Web Scraping Glossary | AlterLab"
comparison_context: "AlterLab is an alternative to Firecrawl, ScrapingBee, and Bright Data."
source_url: https://alterlab.io/glossary/concurrent-crawling
```

general

# Concurrent Crawling

Concurrent crawling runs multiple URL fetches in parallel, balancing throughput against politeness by limiting simultaneous connections to any single domain.

A crawler that fetches one URL at a time (sequential) is vastly underutilising network and CPU resources. Concurrent crawling issues multiple requests simultaneously, dramatically increasing throughput. The key constraint is per-domain politeness: while issuing 50 concurrent requests globally is fine when targeting 50 different domains, issuing 50 concurrent requests to a single small website is an aggressive load that will trigger rate limiting or anti-bot defences.

Well-designed crawlers implement per-domain concurrency limits: a global semaphore limits total concurrent fetches (say, 100), while per-domain semaphores limit simultaneous requests to any single host (say, 2–5). Scrapy implements this via `CONCURRENT_REQUESTS_PER_DOMAIN` and Crawl-delay settings.

For website compatibility, low per-domain concurrency is often essential: sites that detect more than N requests per second from a session will trigger challenges. Staying below this threshold — even at the cost of lower throughput — is preferable to being blocked.

### What is Concurrent Crawling?

Concurrent crawling runs multiple URL fetches in parallel, balancing throughput against politeness by limiting simultaneous connections to any single domain.

### How does AlterLab handle Concurrent Crawling?

For website compatibility, low per-domain concurrency is often essential: sites that detect more than N requests per second from a session will trigger challenges. Staying below this threshold — even at the cost of lower throughput — is preferable to being blocked.

## Related Terms

[Web CrawlerA program that automatically navigates websites by following hyperlinks, systematically indexing pages across an entire domain.](https://alterlab.io/glossary/web-crawler)[ConcurrencyConcurrency in scraping refers to the number of requests or browser sessions running in parallel, which directly controls throughput and resource consumption.](https://alterlab.io/glossary/concurrency)[Rate LimitingA server-side control that caps the number of requests accepted from a single IP or session within a time window, returning HTTP 429 when exceeded.](https://alterlab.io/glossary/rate-limiting)[Async ScrapingAsync scraping uses asynchronous I/O to issue multiple HTTP requests concurrently without blocking, dramatically increasing throughput compared to sequential scraping.](https://alterlab.io/glossary/async-scraping)[Proxy PoolA managed collection of proxy IP addresses used to distribute outbound requests across multiple origins and stay below per-IP rate limits.](https://alterlab.io/glossary/proxy-pool)

## Extract Concurrent Crawling data from any website

AlterLab returns clean, structured data from any public URL — no scraper infrastructure needed. Start free, no credit card required.

[View API docs](https://alterlab.io/docs/api/rest)

[Back to Glossary](https://alterlab.io/glossary)