API Reference

POST

Map Endpoint

Discover all URLs on a website. Map returns a structured list of pages with depth, source, and optional metadata — perfect for planning targeted crawls or batch scrapes.

Endpoint

POST /api/v1/map

Low-Cost Discovery

Map costs 1 credit per call regardless of how many URLs are returned. Use it to plan before spending credits on scraping.

How It Works

Submit a URL

POST a target URL to /api/v1/map. AlterLab crawls the site following links and/or parsing the sitemap.

Discover Pages

The crawler follows internal links up to max_depth levels and collects up to max_pages URLs. You can filter with include/exclude patterns or a search query.

Get a URL List

Receive a flat list of discovered URLs with depth, source type, and optional metadata. Pipe them into /api/v1/batch or individual scrape calls.

Request

POST

/api/v1/map

Discover all URLs on a target website with optional filtering, search, and metadata.

Parameters

Name	Type	Required	Description
url	`string`	Required	The target URL to map. The crawler starts from this page and follows internal links.
max_pages	`integer`	Optional	Maximum number of URLs to return.Default: `100`
max_depth	`integer`	Optional	Maximum link depth to crawl from the starting URL. 0 returns only the starting page.Default: `3`
include_patterns	`string[]`	Optional	Glob patterns to include. Only URLs matching at least one pattern are returned. Example: ["/products/", "/blog/"].
exclude_patterns	`string[]`	Optional	Glob patterns to exclude. URLs matching any pattern are filtered out. Example: ["/admin/", ".pdf"].
search	`string`	Optional	Search query to filter URLs by relevance. Returns a relevance_score for each URL.
sitemap	`boolean`	Optional	If true, parse the sitemap.xml instead of (or in addition to) crawling links.Default: `false`
sitemap_path	`string`	Optional	Custom path to the sitemap file. Only used when sitemap is true.Default: `/sitemap.xml`
include_metadata	`boolean`	Optional	Include page metadata (title, description, last_modified) for each URL. Increases response time.Default: `false`
include_subdomains	`boolean`	Optional	Include URLs from subdomains of the target domain.Default: `false`
respect_robots	`boolean`	Optional	Respect robots.txt directives when crawling.Default: `true`

Request Example

Bash

curl -X POST https://api.alterlab.io/api/v1/map \
  -H "X-API-Key: your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://store.com",
    "max_pages": 500,
    "max_depth": 3,
    "include_patterns": ["/products/*"],
    "include_metadata": true
  }'

Parameter Details

Pattern Matching

include_patterns and exclude_patterns use glob syntax. Use * for single path segments and ** for recursive matching. Exclude patterns take priority over include patterns.

Response

A successful request returns 200 OK with the discovered URL list:

JSON

{
  "map_id": "map_a1b2c3d4e5f6",
  "total_urls": 47,
  "urls": [
    {
      "url": "https://store.com/products/widget-pro",
      "depth": 1,
      "source": "link",
      "title": "Widget Pro — Store",
      "description": "The best widget for professionals"
    },
    {
      "url": "https://store.com/products/widget-lite",
      "depth": 2,
      "source": "link",
      "title": "Widget Lite — Store",
      "description": "Affordable widget for everyone"
    },
    {
      "url": "https://store.com/products/accessories",
      "depth": 1,
      "source": "sitemap",
      "title": "Accessories — Store",
      "description": null
    }
  ],
  "sitemap_found": true,
  "robots_txt": {
    "exists": true,
    "disallowed_paths": ["/admin/*", "/api/*"],
    "crawl_delay": 1.0,
    "sitemaps": ["https://store.com/sitemap.xml"]
  },
  "credits_used": 1
}

URL Object

Field	Type	Description
url	string	Absolute URL of the discovered page
depth	integer	Number of link hops from the starting URL (0 = starting page)
source	string	How the URL was discovered: `link` or `sitemap`
relevance_score	number \| null	Relevance to the `search` query (0-1). Only present when search is used.
metadata	object \| null	Page title, description, and last_modified. Only present when `include_metadata` is true.

Search Filtering

Use the search parameter to find specific pages within a site. Results are ranked by relevance and include a relevance_score.

Bash

curl -X POST https://api.alterlab.io/api/v1/map \
  -H "X-API-Key: your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://docs.example.com",
    "search": "authentication oauth",
    "max_pages": 20
  }'

Search matches against URL paths, page titles (when include_metadata is enabled), and anchor text from linking pages.

Sitemap Mode

Set sitemap: true to parse the site's sitemap.xml instead of following links. This is faster and returns URLs the site owner considers canonical.

JSON

{
  "url": "https://store.com",
  "sitemap": true,
  "sitemap_path": "/sitemap-products.xml",
  "max_pages": 1000
}

Link Discovery vs Sitemap

Link discovery finds pages reachable by clicking through the site. Sitemap mode finds pages the site owner lists in XML. For best coverage, compare both — some pages only appear in one source.

List Map History

GET

/api/v1/map

List the authenticated user's past map operations, newest first. Supports pagination and domain filtering.

Parameters

Name	Type	Required	Description
page	`integer`	Optional	Page number (1-based).Default: `1`
per_page	`integer`	Optional	Items per page (1-100).Default: `20`
domain	`string`	Optional	Filter results by domain (e.g., store.com).

JSON

// List response
{
  "maps": [
    {
      "map_id": "map_a1b2c3d4e5f6",
      "url": "https://store.com",
      "domain": "store.com",
      "total_urls": 247,
      "credits_used": 1,
      "sitemap_found": true,
      "created_at": "2026-04-20T14:30:00Z"
    }
  ],
  "total": 12,
  "page": 1,
  "per_page": 20
}

Get Map Result

GET

/api/v1/map/{map_id}

Retrieve the full result of a past map operation including all discovered URLs. Only the owner can access their maps.

Bash

curl https://api.alterlab.io/api/v1/map/map_a1b2c3d4e5f6 \
  -H "X-API-Key: YOUR_API_KEY"

Returns the same shape as the POST response, plus the config object showing the original map configuration used.

Compare Maps

GET

/api/v1/map/compare

Compare two map snapshots of the same domain to detect structural changes. Returns added, removed, changed, and unchanged URLs. URL lists are paginated per category — use limit/offset to retrieve large result sets.

Parameters

Name	Type	Required	Description
map_id_a	`string`	Required	Baseline map ID (older snapshot).
map_id_b	`string`	Required	Comparison map ID (newer snapshot).
limit	`integer`	Optional	Maximum URL entries to return per category (added/removed/changed/unchanged). Range: 1–1000.Default: `100`
offset	`integer`	Optional	Number of entries to skip per category. Use with limit to page through large diffs.Default: `0`

JSON

// Compare response
{
  "map_id_a": "map_a1b2c3d4e5f6",
  "map_id_b": "map_f6e5d4c3b2a1",
  "domain": "store.com",
  "created_at_a": "2026-04-15T10:00:00Z",
  "created_at_b": "2026-04-22T10:00:00Z",
  "summary": {
    "added": 12,
    "removed": 3,
    "unchanged": 232,
    "total_a": 235,
    "total_b": 244,
    "sitemap_changed": false,
    "url_delta": 9
  },
  // Paginated URL lists — each category is paged independently
  "added_urls": [
    { "url": "https://store.com/products/new-widget", "source_b": "link", "depth_b": 2 }
  ],
  "removed_urls": [
    { "url": "https://store.com/products/discontinued", "source_a": "sitemap", "depth_a": 1 }
  ],
  "changed_urls": [],
  "unchanged_urls": [],
  // Pagination metadata
  "limit": 100,
  "offset": 0,
  "total_added": 12,    // Total across all pages
  "total_removed": 3,
  "total_changed": 0,
  "total_unchanged": 232
}

Paginated URL Lists

URL lists are paginated per category with a default of 100 entries each. If total_added (or other totals) exceeds limit, increment offset by limit to fetch the next page. The summary block always reflects the full totals regardless of pagination.

Same Domain Required

Both maps must belong to the same domain. Comparing maps of different domains returns a 400 error.

Export Map

GET

/api/v1/map/{map_id}/export

Download a past map result as CSV, JSON, or Sitemap XML. Streams the response for direct download.

Parameters

Name	Type	Required	Description
format	`string`	Optional	Export format: "csv", "json", or "xml" (Sitemap XML).Default: `"json"`

Bash

# Download as CSV
curl -O "https://api.alterlab.io/api/v1/map/map_a1b2c3d4e5f6/export?format=csv" \
  -H "X-API-Key: YOUR_API_KEY"

# Download as Sitemap XML
curl -O "https://api.alterlab.io/api/v1/map/map_a1b2c3d4e5f6/export?format=xml" \
  -H "X-API-Key: YOUR_API_KEY"

Credits & Billing

1 credit per map call regardless of how many URLs are discovered.
Map is designed as a planning tool — use it to discover site structure before committing credits to scraping.
Combine with include_patterns to narrow results before feeding them into batch scrape.
No additional charges for include_metadata or search — still 1 credit.

Error Codes

Code	Status	Description
400	Bad Request	Invalid URL, conflicting parameters, or malformed patterns
401	Unauthorized	Missing or invalid API key
402	Payment Required	Insufficient credit balance (need at least 1 credit)
429	Too Many Requests	Rate limit exceeded
500	Internal Error	Server error during crawl — retry with exponential backoff

cURL Example

Bash

# Discover product pages on a store
curl -X POST https://api.alterlab.io/api/v1/map \
  -H "X-API-Key: sk_live_..." \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://store.com",
    "max_pages": 500,
    "include_patterns": ["/products/*"],
    "include_metadata": true
  }'

# Search for pricing pages
curl -X POST https://api.alterlab.io/api/v1/map \
  -H "X-API-Key: sk_live_..." \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://docs.example.com",
    "search": "pricing",
    "max_pages": 20
  }'

# Parse sitemap only
curl -X POST https://api.alterlab.io/api/v1/map \
  -H "X-API-Key: sk_live_..." \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://blog.example.com",
    "sitemap": true,
    "max_pages": 1000
  }'

Python Example

Python

import alterlab

client = alterlab.AlterLab(api_key="your_api_key")

# Discover site structure
site_map = client.map(
    "https://store.com",
    max_pages=500,
    include_metadata=True,
)
print(f"Found {site_map['total_urls']} URLs")

# Filter for product pages
product_urls = [
    u["url"] for u in site_map["urls"]
    if "/products/" in u["url"]
]
print(f"{len(product_urls)} product pages found")

# Feed into batch scrape
batch = client.batch_scrape(
    urls=[{"url": u, "formats": ["json"]} for u in product_urls[:100]],
)
print(f"Batch {batch['batch_id']} submitted with {batch['total_urls']} URLs")

# Search for specific content
results = client.map(
    "https://docs.example.com",
    search="authentication oauth",
    max_pages=20,
)
for page in results["urls"]:
    score = page.get("relevance_score", 0)
    print(f"  {score:.2f}  {page['url']}")

Node.js Example

TYPESCRIPT

import AlterLab from "@alterlab/sdk";

const client = new AlterLab({ apiKey: "your_api_key" });

// Discover site structure
const siteMap = await client.map("https://store.com", {
  maxPages: 500,
  includeMetadata: true,
});
console.log(`Found ${siteMap.totalUrls} URLs`);

// Filter for product pages
const productUrls = siteMap.urls
  .filter((u) => u.url.includes("/products/"))
  .map((u) => u.url);
console.log(`${productUrls.length} product pages found`);

// Feed into batch scrape
const batch = await client.batchScrape({
  urls: productUrls.slice(0, 100).map((url) => ({
    url,
    formats: ["json"],
  })),
});
console.log(`Batch ${batch.batchId} submitted`);

// Search for specific content
const results = await client.map("https://docs.example.com", {
  search: "authentication oauth",
  maxPages: 20,
});
for (const page of results.urls) {
  console.log(`  ${page.relevanceScore?.toFixed(2) ?? "N/A"}  ${page.url}`);
}

Last updated: March 2026

API Reference

POST

Map Endpoint

Discover all URLs on a website. Map returns a structured list of pages with depth, source, and optional metadata — perfect for planning targeted crawls or batch scrapes.

Endpoint

POST /api/v1/map

Low-Cost Discovery

Map costs 1 credit per call regardless of how many URLs are returned. Use it to plan before spending credits on scraping.

How It Works

Submit a URL

POST a target URL to /api/v1/map. AlterLab crawls the site following links and/or parsing the sitemap.

Discover Pages

The crawler follows internal links up to max_depth levels and collects up to max_pages URLs. You can filter with include/exclude patterns or a search query.

Get a URL List

Receive a flat list of discovered URLs with depth, source type, and optional metadata. Pipe them into /api/v1/batch or individual scrape calls.

Request

POST

/api/v1/map

Discover all URLs on a target website with optional filtering, search, and metadata.

Parameters

Name	Type	Required	Description
url	`string`	Required	The target URL to map. The crawler starts from this page and follows internal links.
max_pages	`integer`	Optional	Maximum number of URLs to return.Default: `100`
max_depth	`integer`	Optional	Maximum link depth to crawl from the starting URL. 0 returns only the starting page.Default: `3`
include_patterns	`string[]`	Optional	Glob patterns to include. Only URLs matching at least one pattern are returned. Example: ["/products/", "/blog/"].
exclude_patterns	`string[]`	Optional	Glob patterns to exclude. URLs matching any pattern are filtered out. Example: ["/admin/", ".pdf"].
search	`string`	Optional	Search query to filter URLs by relevance. Returns a relevance_score for each URL.
sitemap	`boolean`	Optional	If true, parse the sitemap.xml instead of (or in addition to) crawling links.Default: `false`
sitemap_path	`string`	Optional	Custom path to the sitemap file. Only used when sitemap is true.Default: `/sitemap.xml`
include_metadata	`boolean`	Optional	Include page metadata (title, description, last_modified) for each URL. Increases response time.Default: `false`
include_subdomains	`boolean`	Optional	Include URLs from subdomains of the target domain.Default: `false`
respect_robots	`boolean`	Optional	Respect robots.txt directives when crawling.Default: `true`

Request Example

Bash

curl -X POST https://api.alterlab.io/api/v1/map \
  -H "X-API-Key: your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://store.com",
    "max_pages": 500,
    "max_depth": 3,
    "include_patterns": ["/products/*"],
    "include_metadata": true
  }'

Parameter Details

Pattern Matching

include_patterns and exclude_patterns use glob syntax. Use * for single path segments and ** for recursive matching. Exclude patterns take priority over include patterns.

Response

A successful request returns 200 OK with the discovered URL list:

JSON

{
  "map_id": "map_a1b2c3d4e5f6",
  "total_urls": 47,
  "urls": [
    {
      "url": "https://store.com/products/widget-pro",
      "depth": 1,
      "source": "link",
      "title": "Widget Pro — Store",
      "description": "The best widget for professionals"
    },
    {
      "url": "https://store.com/products/widget-lite",
      "depth": 2,
      "source": "link",
      "title": "Widget Lite — Store",
      "description": "Affordable widget for everyone"
    },
    {
      "url": "https://store.com/products/accessories",
      "depth": 1,
      "source": "sitemap",
      "title": "Accessories — Store",
      "description": null
    }
  ],
  "sitemap_found": true,
  "robots_txt": {
    "exists": true,
    "disallowed_paths": ["/admin/*", "/api/*"],
    "crawl_delay": 1.0,
    "sitemaps": ["https://store.com/sitemap.xml"]
  },
  "credits_used": 1
}

URL Object

Field	Type	Description
url	string	Absolute URL of the discovered page
depth	integer	Number of link hops from the starting URL (0 = starting page)
source	string	How the URL was discovered: `link` or `sitemap`
relevance_score	number \| null	Relevance to the `search` query (0-1). Only present when search is used.
metadata	object \| null	Page title, description, and last_modified. Only present when `include_metadata` is true.

Search Filtering

Use the search parameter to find specific pages within a site. Results are ranked by relevance and include a relevance_score.

Bash

curl -X POST https://api.alterlab.io/api/v1/map \
  -H "X-API-Key: your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://docs.example.com",
    "search": "authentication oauth",
    "max_pages": 20
  }'

Search matches against URL paths, page titles (when include_metadata is enabled), and anchor text from linking pages.

Sitemap Mode

Set sitemap: true to parse the site's sitemap.xml instead of following links. This is faster and returns URLs the site owner considers canonical.

JSON

{
  "url": "https://store.com",
  "sitemap": true,
  "sitemap_path": "/sitemap-products.xml",
  "max_pages": 1000
}

Link Discovery vs Sitemap

Link discovery finds pages reachable by clicking through the site. Sitemap mode finds pages the site owner lists in XML. For best coverage, compare both — some pages only appear in one source.

List Map History

GET

/api/v1/map

List the authenticated user's past map operations, newest first. Supports pagination and domain filtering.

Parameters

Name	Type	Required	Description
page	`integer`	Optional	Page number (1-based).Default: `1`
per_page	`integer`	Optional	Items per page (1-100).Default: `20`
domain	`string`	Optional	Filter results by domain (e.g., store.com).

JSON

// List response
{
  "maps": [
    {
      "map_id": "map_a1b2c3d4e5f6",
      "url": "https://store.com",
      "domain": "store.com",
      "total_urls": 247,
      "credits_used": 1,
      "sitemap_found": true,
      "created_at": "2026-04-20T14:30:00Z"
    }
  ],
  "total": 12,
  "page": 1,
  "per_page": 20
}

Get Map Result

GET

/api/v1/map/{map_id}

Retrieve the full result of a past map operation including all discovered URLs. Only the owner can access their maps.

Bash

curl https://api.alterlab.io/api/v1/map/map_a1b2c3d4e5f6 \
  -H "X-API-Key: YOUR_API_KEY"

Returns the same shape as the POST response, plus the config object showing the original map configuration used.

Compare Maps

GET

/api/v1/map/compare

Parameters

Name	Type	Required	Description
map_id_a	`string`	Required	Baseline map ID (older snapshot).
map_id_b	`string`	Required	Comparison map ID (newer snapshot).
limit	`integer`	Optional	Maximum URL entries to return per category (added/removed/changed/unchanged). Range: 1–1000.Default: `100`
offset	`integer`	Optional	Number of entries to skip per category. Use with limit to page through large diffs.Default: `0`

JSON

// Compare response
{
  "map_id_a": "map_a1b2c3d4e5f6",
  "map_id_b": "map_f6e5d4c3b2a1",
  "domain": "store.com",
  "created_at_a": "2026-04-15T10:00:00Z",
  "created_at_b": "2026-04-22T10:00:00Z",
  "summary": {
    "added": 12,
    "removed": 3,
    "unchanged": 232,
    "total_a": 235,
    "total_b": 244,
    "sitemap_changed": false,
    "url_delta": 9
  },
  // Paginated URL lists — each category is paged independently
  "added_urls": [
    { "url": "https://store.com/products/new-widget", "source_b": "link", "depth_b": 2 }
  ],
  "removed_urls": [
    { "url": "https://store.com/products/discontinued", "source_a": "sitemap", "depth_a": 1 }
  ],
  "changed_urls": [],
  "unchanged_urls": [],
  // Pagination metadata
  "limit": 100,
  "offset": 0,
  "total_added": 12,    // Total across all pages
  "total_removed": 3,
  "total_changed": 0,
  "total_unchanged": 232
}

Paginated URL Lists

Same Domain Required

Both maps must belong to the same domain. Comparing maps of different domains returns a 400 error.

Export Map

GET

/api/v1/map/{map_id}/export

Download a past map result as CSV, JSON, or Sitemap XML. Streams the response for direct download.

Parameters

Name	Type	Required	Description
format	`string`	Optional	Export format: "csv", "json", or "xml" (Sitemap XML).Default: `"json"`

Bash

# Download as CSV
curl -O "https://api.alterlab.io/api/v1/map/map_a1b2c3d4e5f6/export?format=csv" \
  -H "X-API-Key: YOUR_API_KEY"

# Download as Sitemap XML
curl -O "https://api.alterlab.io/api/v1/map/map_a1b2c3d4e5f6/export?format=xml" \
  -H "X-API-Key: YOUR_API_KEY"

Credits & Billing

1 credit per map call regardless of how many URLs are discovered.
Map is designed as a planning tool — use it to discover site structure before committing credits to scraping.
Combine with include_patterns to narrow results before feeding them into batch scrape.
No additional charges for include_metadata or search — still 1 credit.

Error Codes

Code	Status	Description
400	Bad Request	Invalid URL, conflicting parameters, or malformed patterns
401	Unauthorized	Missing or invalid API key
402	Payment Required	Insufficient credit balance (need at least 1 credit)
429	Too Many Requests	Rate limit exceeded
500	Internal Error	Server error during crawl — retry with exponential backoff

cURL Example

Bash

# Discover product pages on a store
curl -X POST https://api.alterlab.io/api/v1/map \
  -H "X-API-Key: sk_live_..." \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://store.com",
    "max_pages": 500,
    "include_patterns": ["/products/*"],
    "include_metadata": true
  }'

# Search for pricing pages
curl -X POST https://api.alterlab.io/api/v1/map \
  -H "X-API-Key: sk_live_..." \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://docs.example.com",
    "search": "pricing",
    "max_pages": 20
  }'

# Parse sitemap only
curl -X POST https://api.alterlab.io/api/v1/map \
  -H "X-API-Key: sk_live_..." \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://blog.example.com",
    "sitemap": true,
    "max_pages": 1000
  }'

Python Example

Python

import alterlab

client = alterlab.AlterLab(api_key="your_api_key")

# Discover site structure
site_map = client.map(
    "https://store.com",
    max_pages=500,
    include_metadata=True,
)
print(f"Found {site_map['total_urls']} URLs")

# Filter for product pages
product_urls = [
    u["url"] for u in site_map["urls"]
    if "/products/" in u["url"]
]
print(f"{len(product_urls)} product pages found")

# Feed into batch scrape
batch = client.batch_scrape(
    urls=[{"url": u, "formats": ["json"]} for u in product_urls[:100]],
)
print(f"Batch {batch['batch_id']} submitted with {batch['total_urls']} URLs")

# Search for specific content
results = client.map(
    "https://docs.example.com",
    search="authentication oauth",
    max_pages=20,
)
for page in results["urls"]:
    score = page.get("relevance_score", 0)
    print(f"  {score:.2f}  {page['url']}")

Node.js Example

TYPESCRIPT

import AlterLab from "@alterlab/sdk";

const client = new AlterLab({ apiKey: "your_api_key" });

// Discover site structure
const siteMap = await client.map("https://store.com", {
  maxPages: 500,
  includeMetadata: true,
});
console.log(`Found ${siteMap.totalUrls} URLs`);

// Filter for product pages
const productUrls = siteMap.urls
  .filter((u) => u.url.includes("/products/"))
  .map((u) => u.url);
console.log(`${productUrls.length} product pages found`);

// Feed into batch scrape
const batch = await client.batchScrape({
  urls: productUrls.slice(0, 100).map((url) => ({
    url,
    formats: ["json"],
  })),
});
console.log(`Batch ${batch.batchId} submitted`);

// Search for specific content
const results = await client.map("https://docs.example.com", {
  search: "authentication oauth",
  maxPages: 20,
});
for (const page of results.urls) {
  console.log(`  ${page.relevanceScore?.toFixed(2) ?? "N/A"}  ${page.url}`);
}

Last updated: March 2026