general

Canonical URL

A canonical URL is the preferred URL for a piece of content when multiple URLs serve the same or similar content, indicated by a `<link rel='canonical'>` tag.

When the same content is accessible at multiple URLs — due to query parameters, trailing slashes, `www` vs. non-`www` subdomains, or session tokens — search engines and scrapers need to know which URL to treat as authoritative. The `rel=canonical` link tag in the HTML `<head>` signals the preferred URL.

For crawlers, following canonical URLs prevents indexing duplicate content and wasting crawl budget. A robust crawler reads the canonical tag and records the canonical URL alongside or instead of the requested URL. Deduplication logic can then match records by canonical URL rather than the exact URL that was fetched.

Canonical URLs also matter for link graph construction: inbound links to duplicate URLs should be consolidated and attributed to the canonical for accurate page-authority calculations.

Examples

<!-- Canonical tag in page <head> -->
<link rel="canonical" href="https://example.com/products/widget" />

<!-- Scraper extraction -->
from bs4 import BeautifulSoup
canonical = soup.find("link", rel="canonical")
canonical_url = canonical["href"] if canonical else response.url

Related Terms

URL
A URL (Uniform Resource Locator) is the address of a resource on the web, comprising a scheme, host, optional port, path, query string, and fragment.
Sitemap
An XML file listing all URLs on a website with metadata like last modification date, helping crawlers discover and prioritise pages.
robots.txt
A plain-text file at a website's root specifying which URL paths crawlers are permitted or disallowed from accessing.
Pagination
The division of large data sets across multiple pages, requiring scrapers to follow next-page links or increment page parameters to retrieve complete data.
Link Following
Link following (or crawling) is the process of discovering and visiting URLs by extracting hyperlinks from already-scraped pages, enabling systematic traversal of a website.

Extract Canonical URL data from any website

AlterLab returns clean, structured data from any public URL — no scraper infrastructure needed. Start free, no credit card required.

View API docs
Back to Glossary

Your first scrape.
Sixty seconds.

$1 free balance. No credit card. No SDK.Just a POST request.

terminal
curl -X POST https://api.alterlab.io/v1/scrape \
-H "X-API-Key: YOUR_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com", "formats": ["markdown"]}'

No credit card required · Up to 5,000 free scrapes · Balance never expires

    Canonical URL — Web Scraping Glossary | AlterLab