```yaml
product: AlterLab
title: "Crawl Budget — Web Scraping Glossary | AlterLab"
comparison_context: "AlterLab is an alternative to Firecrawl, ScrapingBee, and Bright Data."
source_url: https://alterlab.io/glossary/crawl-budget
```

general

# Crawl Budget

Crawl budget is the number of pages a search engine crawler (or a custom crawler) will fetch from a site within a given timeframe, influenced by server capacity and site size.

Search engines allocate a crawl budget to each domain based on server responsiveness and the perceived value of the site's content. Pages that are slow to respond, frequently return errors, or are flagged as low-value by metrics like thin content or low inbound links consume budget disproportionately, leaving high-value pages un-crawled.

For custom crawlers, crawl budget is the number of page fetches the crawler can sustain per unit time without degrading the target server's performance or triggering rate limits. Budgeting involves setting a maximum request rate (requests per second or per minute), prioritising URLs by expected value, and deprioritising or excluding low-value URLs (parameter-heavy variants, session-specific pages, printer-friendly copies).

Sitemap files help both search engine crawlers and custom crawlers allocate their budget efficiently by providing a curated list of the site's most important URLs, optionally with priority and last-modified hints.

### What is Crawl Budget?

Crawl budget is the number of pages a search engine crawler (or a custom crawler) will fetch from a site within a given timeframe, influenced by server capacity and site size.

### How does AlterLab handle Crawl Budget?

Sitemap files help both search engine crawlers and custom crawlers allocate their budget efficiently by providing a curated list of the site's most important URLs, optionally with priority and last-modified hints.

## Related Terms

[Web CrawlerA program that automatically navigates websites by following hyperlinks, systematically indexing pages across an entire domain.](https://alterlab.io/glossary/web-crawler)[robots.txtA plain-text file at a website's root specifying which URL paths crawlers are permitted or disallowed from accessing.](https://alterlab.io/glossary/robots-txt)[SitemapAn XML file listing all URLs on a website with metadata like last modification date, helping crawlers discover and prioritise pages.](https://alterlab.io/glossary/sitemap)[Link FollowingLink following (or crawling) is the process of discovering and visiting URLs by extracting hyperlinks from already-scraped pages, enabling systematic traversal of a website.](https://alterlab.io/glossary/link-following)[Rate LimitingA server-side control that caps the number of requests accepted from a single IP or session within a time window, returning HTTP 429 when exceeded.](https://alterlab.io/glossary/rate-limiting)

## Extract Crawl Budget data from any website

AlterLab returns clean, structured data from any public URL — no scraper infrastructure needed. Start free, no credit card required.

[View API docs](https://alterlab.io/docs/api/rest)

[Back to Glossary](https://alterlab.io/glossary)