Programmatic Crawl Exports and Universal Batching
This release introduces the highly anticipated programmatic crawl export endpoint and opens up our Batch Scraping API to all subscription tiers. We have also refined our browser identity virtualization and cookie handling to deliver significantly higher scraping success rates on heavily protected retail domains. Additionally, developers will find clearer API error handling, helpful validation hints for batch payloads, and UI improvements in the dashboard playground.
You can now programmatically export completed crawl results using our new export endpoint. This enables fully automated, seamless retrieval of historical crawl data directly within your custom integrations and workflows.
You can now access the Batch Scraping API across all subscription tiers, rather than it being restricted to premium accounts. This allows you to scale your workflows with batch requests, managed dynamically through standard account balances.
New Features
1Programmatic crawl exports
You can now programmatically export completed crawl results using our new export endpoint. This enables fully automated, seamless retrieval of historical crawl data directly within your custom integrations and workflows.
Improvements
6Universal batch scraping
You can now access the Batch Scraping API across all subscription tiers, rather than it being restricted to premium accounts. This allows you to scale your workflows with batch requests, managed dynamically through standard account balances.
Enhanced request authenticity
Your scrapers now present fully aligned browser signatures and natural user permission profiles when loading target websites. This significantly boosts success rates on heavily protected domains that analyze browser authenticity.
Stale cookie protection
When reusing browser sessions, the platform now automatically discards location-specific verification cookies upon proxy IP rotation. This prevents stale session tracking from triggering blocks and keeps your requests looking fresh.
Reliable proxy sticky-sessions
Your multi-step browser tasks are now guaranteed to retain the same sticky IP address across consecutive actions. Geographic targeting parameters are also fully passed through to ensure consistently localized content.
Accurate protocol alignment
Our browser profiles now perfectly replicate the precise HTTP/2 settings and protocol flags of modern desktop browsers. This ensures your scraping traffic remains completely indistinguishable from real human visitors to detection systems.
General bug fixes and improvements
Plus 40 internal improvements for better reliability and performance.
Bug Fixes
9Actionable batch payload validation
If you submit a single-URL format to the batch scraping endpoint, you will now receive a helpful validation error with the exact schema needed. This provides immediate guidance to speed up your integration and migration tasks.
Consistent crawl error handling
You will now receive a consistent HTTP 404 status code when querying missing or expired crawl export resources. This aligns export behavior with standard crawl endpoints and provides a predictable error handling contract for your client code.
Silent crawl export validation
Passing an invalid status filter to crawl export endpoints now properly returns an HTTP 422 validation error instead of an empty success response. This prevents typos in query parameters from causing silent failures during development.
JSON extraction from HTML
Fixed a scraping bug where JSON payloads wrapped inside HTML `<pre>` tags resulted in empty extractions. You can now reliably scrape raw API responses rendered within the browser, retrieving fully structured JSON output.
Expired crawl warnings
The developer dashboard now alerts you when crawl results have expired and hides the download options. This prevents confusing, silent failures on old crawls and clearly displays the lifecycle of your data.
Playground overflow layout
We fixed a layout bug on the playground page where excessively long scraping responses would stretch the page vertically past the settings panel. The results container now uses a scrollable, capped-height frame for much cleaner readability.
Definitive block page handling
Our system now correctly identifies definitive block and challenge pages served by target websites, rather than falsely categorizing them as successful page content. This ensures your scraping success metrics are completely accurate.
API specification version alignment
We updated the public OpenAPI schema and version files to 2.7.0. This guarantees that your automated client generators and developer documentation always stay perfectly synchronized with our latest endpoints.
Improved retail scraping success
Resolved an issue where requests to certain major retail and apparel domains were intermittently blocked due to suboptimal browser routing. Our configurations are optimized to ensure highly reliable content retrieval from these sites.
Plus 40 internal changes for stability and performance.