Web Crawling API
Crawl entire websites with depth control, link discovery, and sitemap-aware traversal. AlterLab's crawling API handles anti-bot protection, JavaScript rendering, and proxy rotation automatically. Extract structured data from thousands of pages in a single job.
Simple API, Powerful Results
Get started in minutes with our intuitive API. One request gives you structured data, screenshots, PDFs, and more. No browser management, no infrastructure headaches.
Multi-Format Output
Markdown, JSON, HTML, text
Adaptive Rendering
JS, SPAs, shadow DOM
3 Lines to Integrate
Any language, any stack
Up to 5,000 free scrapes included. No credit card required.
How Web Crawling Works
Submit a seed URL — AlterLab handles link discovery, anti-bot bypass, and structured data extraction automatically.
Seed URL Submission
Submit your starting URL along with depth limits, page count caps, and include/exclude patterns. AlterLab begins crawling immediately — no setup required. Sitemap.xml files are parsed automatically to discover all indexable URLs before the crawl begins.
Link Discovery & Queueing
Each scraped page is parsed for outbound links. Links matching your include patterns and within the depth limit are queued for crawling. AlterLab deduplicates URLs and respects robots.txt by default — override when needed for competitive intelligence use cases.
Anti-Bot Bypass Per Page
Each URL in the crawl queue is scraped through AlterLab's 5-tier pipeline. Basic pages use lightweight TLS fingerprinting at $0.0002/page. JavaScript-heavy pages automatically escalate to Playwright browser rendering. Anti-bot systems are detected and bypassed per page — no configuration needed.
Structured Results via Webhook
Results are delivered to your webhook when the crawl completes. Each page includes HTML, Markdown, extracted metadata, discovered links, and cost per page. Failed pages are retried automatically — you only pay for successful scrapes.
Built for Production Crawls
Enterprise-grade crawling with automatic anti-bot handling and structured output.
Depth Control
Set crawl depth from 1 to unlimited. Crawl single pages or full site hierarchies.
Sitemap-Aware
Automatically parses sitemap.xml for efficient, complete site coverage.
Anti-Bot Bypass
5-tier escalation handles Cloudflare, DataDome, and Akamai on every crawled page.
Include/Exclude Patterns
Glob patterns to scope crawls to specific sections — /blog/*, /products/*, etc.
Crawling Use Cases
From content indexing to competitive intelligence — web crawling at any scale.
Content Indexing
Crawl entire sites to build searchable indexes for internal search engines or AI knowledge bases
Competitive Monitoring
Track competitor sites for pricing changes, product launches, and content updates
Data Pipeline Feeds
Build automated crawl pipelines that feed structured data into databases and analytics platforms
SEO Auditing
Discover broken links, missing metadata, and crawl errors across entire site structures
Cheaper Than Firecrawl & Bright Data
$0.0002 per page. No monthly subscription. Scale to millions of pages without a contract.
Why Teams Switch to AlterLab
Smart routing, no subscriptions, balance that never expires
| Feature | AlterLab You are here | ScraperAPI | Bright Data | Firecrawl |
|---|---|---|---|---|
Avg cost (real workload) Smart routing vs flat rate | ~$0.001 | $0.00049/credit | $0.0015 | $0.0063 |
Simple scrape Basic HTTP request | $0.0002 | $0.00049 | $0.0015 | $0.0063 |
JS rendering Full browser render | $0.004 | $0.0049 | $0.0015 | $0.0063 |
Free tier Free requests to start | Up to 5,000 scrapes | 5,000 credits | None | 500 scrapes |
Minimum Smallest purchase | $10 one-time | $49/month | $0 (PAYG) / $499/mo (subscription) | $19/month |
Balance expires? Does unused balance expire | Never | Monthly | Never (prepaid) / Monthly (subscription) | Monthly |
CAPTCHA solving Built-in CAPTCHA bypass | $0.02/solve | Extra cost | Extra cost | Not available |
Avg cost based on typical content/SEO workload (75% simple, 25% protected). Competitor prices from public pricing pages, March 2026.
Web Crawling API FAQ
Crawling & Scraping Resources
Batch Scraping API
Submit up to 10,000 URLs at once with webhook delivery and auto-retries.
Anti-Bot Bypass API
Bypass Cloudflare, DataDome, and Akamai on every crawled page.
JavaScript Rendering API
Render SPAs and dynamic content with headless Chromium.
View Pricing
From $0.0002/page. No subscription. Pay only for what you scrape.
Your first scrape.
Sixty seconds.
$1 free balance. No credit card. No SDK.
Just a POST request.
No credit card required · Up to 5,000 free scrapes · Balance never expire