```yaml
product: AlterLab
title: "Web Crawler — Web Scraping Glossary | AlterLab"
comparison_context: "AlterLab is an alternative to Firecrawl, ScrapingBee, and Bright Data."
source_url: https://alterlab.io/glossary/web-crawler
```

general

# Web Crawler

A program that automatically navigates websites by following hyperlinks, systematically indexing pages across an entire domain.

A web crawler (also called a spider or bot) is a program that discovers and visits web pages by following hyperlinks from one page to the next. Unlike scraping a single URL, a crawler maps and traverses an entire site — or the whole web — by queuing every link it encounters and processing each URL in turn.

Crawlers need to handle URL deduplication to avoid revisiting the same page, respect robots.txt exclusion rules, manage politeness delays to avoid overwhelming servers, handle redirects, and deal with session cookies and authentication. Large-scale crawlers distribute work across multiple machines and maintain persistent queues for fault tolerance.

Search engines like Google and Bing run massive crawlers to index the web. Developers use site-specific crawlers to map content structure, build datasets across hundreds of pages, or discover new URLs before scraping their content. For shallow crawling tasks, AlterLab's `/crawl` endpoint accepts a root URL and returns all discovered links within a domain.

### What is Web Crawler?

A program that automatically navigates websites by following hyperlinks, systematically indexing pages across an entire domain.

### How does AlterLab handle Web Crawler?

Search engines like Google and Bing run massive crawlers to index the web. Developers use site-specific crawlers to map content structure, build datasets across hundreds of pages, or discover new URLs before scraping their content. For shallow crawling tasks, AlterLab's `/crawl` endpoint accepts a root URL and returns all discovered links within a domain.

## Examples

```
# Discover all links from a domain
{
  "url": "https://example.com",
  "depth": 2,
  "follow_links": true,
  "same_domain": true
}
```

## Related Terms

[Web ScrapingAutomated extraction of data from websites using software tools that parse HTML and collect structured information at scale.](https://alterlab.io/glossary/web-scraping)[robots.txtA plain-text file at a website's root specifying which URL paths crawlers are permitted or disallowed from accessing.](https://alterlab.io/glossary/robots-txt)[SitemapAn XML file listing all URLs on a website with metadata like last modification date, helping crawlers discover and prioritise pages.](https://alterlab.io/glossary/sitemap)[Rate LimitingA server-side control that caps the number of requests accepted from a single IP or session within a time window, returning HTTP 429 when exceeded.](https://alterlab.io/glossary/rate-limiting)[Session ManagementThe handling of cookies, authentication tokens, and browser state across multiple requests to maintain a continuous browsing session.](https://alterlab.io/glossary/session-management)

## Extract Web Crawler data from any website

AlterLab returns clean, structured data from any public URL — no scraper infrastructure needed. Start free, no credit card required.

[View API docs](https://alterlab.io/docs/api/rest)

[Back to Glossary](https://alterlab.io/glossary)