A crawler that fetches one URL at a time (sequential) is vastly underutilising network and CPU resources. Concurrent crawling issues multiple requests simultaneously, dramatically increasing throughput. The key constraint is per-domain politeness: while issuing 50 concurrent requests globally is fine when targeting 50 different domains, issuing 50 concurrent requests to a single small website is an aggressive load that will trigger rate limiting or anti-bot defences.
Well-designed crawlers implement per-domain concurrency limits: a global semaphore limits total concurrent fetches (say, 100), while per-domain semaphores limit simultaneous requests to any single host (say, 2–5). Scrapy implements this via `CONCURRENT_REQUESTS_PER_DOMAIN` and Crawl-delay settings.
For website compatibility, low per-domain concurrency is often essential: sites that detect more than N requests per second from a session will trigger challenges. Staying below this threshold — even at the cost of lower throughput — is preferable to being blocked.