```yaml
product: AlterLab
title: "Crawl Depth — Web Scraping Glossary | AlterLab"
comparison_context: "AlterLab is an alternative to Firecrawl, ScrapingBee, and Bright Data."
source_url: https://alterlab.io/glossary/crawl-depth
```

general

# Crawl Depth

Crawl depth is the maximum number of link hops a crawler will follow from its seed URLs, limiting the scope of the crawl to pages within N clicks of the starting point.

In a web crawl, depth 0 is the seed URL itself, depth 1 is every page linked from the seed, depth 2 is pages linked from depth-1 pages, and so on. Setting a maximum crawl depth prevents the crawler from venturing into deeply nested pages that are unlikely to contain the target data and helps control crawl budget.

Depth limits are especially important for sites with infinite or near-infinite link graphs generated by parameter combinations (calendar pages, faceted search results). A crawler with no depth limit encountering such a site will generate an unbounded crawl frontier.

In Scrapy, depth is controlled by the `DEPTH_LIMIT` setting. Individual spiders can override depth per request by checking `response.meta.get('depth')` and declining to follow links past a threshold. Some crawlers use a breadth-first strategy (process all depth-N pages before depth-N+1) to ensure the highest-level pages are captured first even if the crawl is terminated early.

### What is Crawl Depth?

Crawl depth is the maximum number of link hops a crawler will follow from its seed URLs, limiting the scope of the crawl to pages within N clicks of the starting point.

### How does AlterLab handle Crawl Depth?

In Scrapy, depth is controlled by the `DEPTH_LIMIT` setting. Individual spiders can override depth per request by checking `response.meta.get('depth')` and declining to follow links past a threshold. Some crawlers use a breadth-first strategy (process all depth-N pages before depth-N+1) to ensure the highest-level pages are captured first even if the crawl is terminated early.

## Examples

```
# Scrapy settings: limit crawl depth
# settings.py
DEPTH_LIMIT = 3  # only follow links up to 3 hops from seed
```

## Related Terms

[Web CrawlerA program that automatically navigates websites by following hyperlinks, systematically indexing pages across an entire domain.](https://alterlab.io/glossary/web-crawler)[Link FollowingLink following (or crawling) is the process of discovering and visiting URLs by extracting hyperlinks from already-scraped pages, enabling systematic traversal of a website.](https://alterlab.io/glossary/link-following)[Crawl BudgetCrawl budget is the number of pages a search engine crawler (or a custom crawler) will fetch from a site within a given timeframe, influenced by server capacity and site size.](https://alterlab.io/glossary/crawl-budget)[robots.txtA plain-text file at a website's root specifying which URL paths crawlers are permitted or disallowed from accessing.](https://alterlab.io/glossary/robots-txt)[SitemapAn XML file listing all URLs on a website with metadata like last modification date, helping crawlers discover and prioritise pages.](https://alterlab.io/glossary/sitemap)

## Extract Crawl Depth data from any website

AlterLab returns clean, structured data from any public URL — no scraper infrastructure needed. Start free, no credit card required.

[View API docs](https://alterlab.io/docs/api/rest)

[Back to Glossary](https://alterlab.io/glossary)