```yaml
product: AlterLab
title: "Link Following — Web Scraping Glossary | AlterLab"
comparison_context: "AlterLab is an alternative to Firecrawl, ScrapingBee, and Bright Data."
source_url: https://alterlab.io/glossary/link-following
```

general

# Link Following

Link following (or crawling) is the process of discovering and visiting URLs by extracting hyperlinks from already-scraped pages, enabling systematic traversal of a website.

A scraper that follows links autonomously becomes a crawler: it starts from one or more seed URLs, extracts all anchor hrefs from the fetched HTML, adds new URLs to a queue, and visits them in turn. The crawl frontier (the queue of URLs to visit) grows as new links are discovered. Visited URLs are tracked to prevent re-processing.

Effective link-following requires normalising URLs (resolving relative paths, stripping fragment identifiers, canonicalising query parameters), respecting robots.txt exclusions, enforcing domain or path scope filters to avoid leaving the target site, and managing crawl depth limits to prevent infinite loops caused by dynamically generated URLs.

Politeness policies — rate limiting, request spacing, and honouring Crawl-delay directives in robots.txt — are important both ethically and practically: aggressive crawlers are more likely to trigger anti-bot defences and put load on the target server.

### What is Link Following?

Link following (or crawling) is the process of discovering and visiting URLs by extracting hyperlinks from already-scraped pages, enabling systematic traversal of a website.

### How does AlterLab handle Link Following?

Politeness policies — rate limiting, request spacing, and honouring Crawl-delay directives in robots.txt — are important both ethically and practically: aggressive crawlers are more likely to trigger anti-bot defences and put load on the target server.

## Examples

```
# Scrapy spider following links
import scrapy

class SiteSpider(scrapy.Spider):
    name = "site"
    start_urls = ["https://example.com"]

    def parse(self, response):
        yield {"url": response.url, "title": response.css("h1::text").get()}
        for href in response.css("a::attr(href)"):
            yield response.follow(href, self.parse)
```

## Related Terms

[Web CrawlerA program that automatically navigates websites by following hyperlinks, systematically indexing pages across an entire domain.](https://alterlab.io/glossary/web-crawler)[robots.txtA plain-text file at a website's root specifying which URL paths crawlers are permitted or disallowed from accessing.](https://alterlab.io/glossary/robots-txt)[SitemapAn XML file listing all URLs on a website with metadata like last modification date, helping crawlers discover and prioritise pages.](https://alterlab.io/glossary/sitemap)[PaginationThe division of large data sets across multiple pages, requiring scrapers to follow next-page links or increment page parameters to retrieve complete data.](https://alterlab.io/glossary/pagination)[Rate LimitingA server-side control that caps the number of requests accepted from a single IP or session within a time window, returning HTTP 429 when exceeded.](https://alterlab.io/glossary/rate-limiting)

## Extract Link Following data from any website

AlterLab returns clean, structured data from any public URL — no scraper infrastructure needed. Start free, no credit card required.

[View API docs](https://alterlab.io/docs/api/rest)

[Back to Glossary](https://alterlab.io/glossary)