```yaml
product: AlterLab
title: "XML Sitemap — Web Scraping Glossary | AlterLab"
comparison_context: "AlterLab is an alternative to Firecrawl, ScrapingBee, and Bright Data."
source_url: https://alterlab.io/glossary/sitemap-xml
```

general

# XML Sitemap

An XML sitemap is a structured file listing a website's important URLs with optional metadata, helping search engine crawlers and custom scrapers discover all public pages.

An XML sitemap follows the Sitemap Protocol specification and is typically served at `/sitemap.xml` or referenced from `robots.txt`. Each `<url>` entry specifies a `<loc>` (the URL), optionally `<lastmod>` (last modified date), `<changefreq>` (expected change frequency), and `<priority>` (relative importance 0.0–1.0). Large sites use sitemap index files that reference multiple child sitemaps.

For scrapers, the sitemap is the most efficient starting point for a comprehensive crawl: rather than discovering URLs through link following, the scraper downloads the sitemap and directly queues all listed URLs. This avoids link-graph traversal overhead and ensures coverage of pages that may not be reachable through internal links.

Sitemaps also reveal site structure intelligence: the `<lastmod>` field enables incremental scraping (only fetch pages changed since the last run), and the priority field suggests which pages the publisher considers most important.

### What is XML Sitemap?

An XML sitemap is a structured file listing a website's important URLs with optional metadata, helping search engine crawlers and custom scrapers discover all public pages.

### How does AlterLab handle XML Sitemap?

Sitemaps also reveal site structure intelligence: the `<lastmod>` field enables incremental scraping (only fetch pages changed since the last run), and the priority field suggests which pages the publisher considers most important.

## Examples

```
# Parse a sitemap.xml with Python
import requests
from xml.etree import ElementTree as ET

response = requests.get("https://example.com/sitemap.xml")
ns = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}
root = ET.fromstring(response.content)
urls = [url.text for url in root.findall(".//sm:loc", ns)]
print(f"Found {len(urls)} URLs in sitemap")
```

## Related Terms

[SitemapAn XML file listing all URLs on a website with metadata like last modification date, helping crawlers discover and prioritise pages.](https://alterlab.io/glossary/sitemap)[robots.txtA plain-text file at a website's root specifying which URL paths crawlers are permitted or disallowed from accessing.](https://alterlab.io/glossary/robots-txt)[Web CrawlerA program that automatically navigates websites by following hyperlinks, systematically indexing pages across an entire domain.](https://alterlab.io/glossary/web-crawler)[Crawl BudgetCrawl budget is the number of pages a search engine crawler (or a custom crawler) will fetch from a site within a given timeframe, influenced by server capacity and site size.](https://alterlab.io/glossary/crawl-budget)[Canonical URLA canonical URL is the preferred URL for a piece of content when multiple URLs serve the same or similar content, indicated by a `<link rel='canonical'>` tag.](https://alterlab.io/glossary/canonical-url)

## Extract XML Sitemap data from any website

AlterLab returns clean, structured data from any public URL — no scraper infrastructure needed. Start free, no credit card required.

[View API docs](https://alterlab.io/docs/api/rest)

[Back to Glossary](https://alterlab.io/glossary)