general

XML Sitemap

An XML sitemap is a structured file listing a website's important URLs with optional metadata, helping search engine crawlers and custom scrapers discover all public pages.

An XML sitemap follows the Sitemap Protocol specification and is typically served at `/sitemap.xml` or referenced from `robots.txt`. Each `<url>` entry specifies a `<loc>` (the URL), optionally `<lastmod>` (last modified date), `<changefreq>` (expected change frequency), and `<priority>` (relative importance 0.0–1.0). Large sites use sitemap index files that reference multiple child sitemaps.

For scrapers, the sitemap is the most efficient starting point for a comprehensive crawl: rather than discovering URLs through link following, the scraper downloads the sitemap and directly queues all listed URLs. This avoids link-graph traversal overhead and ensures coverage of pages that may not be reachable through internal links.

Sitemaps also reveal site structure intelligence: the `<lastmod>` field enables incremental scraping (only fetch pages changed since the last run), and the priority field suggests which pages the publisher considers most important.

What is XML Sitemap?

An XML sitemap is a structured file listing a website's important URLs with optional metadata, helping search engine crawlers and custom scrapers discover all public pages.

How does AlterLab handle XML Sitemap?

Sitemaps also reveal site structure intelligence: the `<lastmod>` field enables incremental scraping (only fetch pages changed since the last run), and the priority field suggests which pages the publisher considers most important.

Examples

# Parse a sitemap.xml with Python
import requests
from xml.etree import ElementTree as ET

response = requests.get("https://example.com/sitemap.xml")
ns = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}
root = ET.fromstring(response.content)
urls = [url.text for url in root.findall(".//sm:loc", ns)]
print(f"Found {len(urls)} URLs in sitemap")

Related Terms

Sitemap
An XML file listing all URLs on a website with metadata like last modification date, helping crawlers discover and prioritise pages.
robots.txt
A plain-text file at a website's root specifying which URL paths crawlers are permitted or disallowed from accessing.
Web Crawler
A program that automatically navigates websites by following hyperlinks, systematically indexing pages across an entire domain.
Crawl Budget
Crawl budget is the number of pages a search engine crawler (or a custom crawler) will fetch from a site within a given timeframe, influenced by server capacity and site size.
Canonical URL
A canonical URL is the preferred URL for a piece of content when multiple URLs serve the same or similar content, indicated by a `<link rel='canonical'>` tag.

Extract XML Sitemap data from any website

AlterLab returns clean, structured data from any public URL — no scraper infrastructure needed. Start free, no credit card required.

View API docs
Back to Glossary

Your first scrape.
Sixty seconds.

$1 free credit — up to 5,000 scrapes. No credit card.Just a POST request.

terminal
curl -X POST https://api.alterlab.io/v1/scrape \
-H "X-API-Key: YOUR_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com", "formats": ["markdown"]}'

No credit card required · $1 free credit, up to 5,000 scrapes · Balance never expires