Dashboard Enhancements & Billing Fixes
This release brings major dashboard improvements including a new job history page with search and filters, enhanced billing alerts, and performance optimizations for crawl operations. Users will notice faster job reconciliation and more accurate spending tracking.
A comprehensive job history page is now available at /dashboard/history, allowing users to browse all past jobs from PostgreSQL with advanced search, filtering by URL/status/mode, pagination, and response viewing capabilities. The sidebar navigation has been updated to include this new History entry
Job status counting operations now use batch Redis MGET commands instead of sequential GET calls, reducing Redis round-trips from O(n) to O(1) per reconciliation. This optimization cuts total GET operations for a crawl from O(n²) to O(n), significantly improving performance for large-scale scraping
New Features
1New Job History Dashboard
A comprehensive job history page is now available at /dashboard/history, allowing users to browse all past jobs from PostgreSQL with advanced search, filtering by URL/status/mode, pagination, and response viewing capabilities. The sidebar navigation has been updated to include this new History entry
Improvements
5Dashboard UI polish
Internal refinements to the dashboard layout and component structure for improved consistency and maintainability.
Faster Crawl Job Reconciliation
Job status counting operations now use batch Redis MGET commands instead of sequential GET calls, reducing Redis round-trips from O(n) to O(1) per reconciliation. This optimization cuts total GET operations for a crawl from O(n²) to O(n), significantly improving performance for large-scale scraping
Optimized Rate Limiting
Rate limit checks now occur before expensive network operations during URL discovery, preventing over-limit API keys from triggering unnecessary sitemap or start-page requests. This change reduces wasted resources and ensures users receive immediate 429 responses when they exceed their rate limits.
Enhanced API Key Management
API key scope fallbacks now include billing permissions by default, ensuring consistent access control across all API key operations. This change improves security and prevents unexpected permission issues when working with billing-related API endpoints.
General bug fixes and improvements
Plus 5 internal improvements for better reliability and performance.
Bug Fixes
5Enhanced Billing Alerts
Low balance alerts and auto-recharge triggers are now properly wired into both inline/sync and async queued billing paths. The system now correctly handles balance checks and sends timely notifications when account balances run low, preventing unexpected service interruptions during active scraping
Accurate Spending Display
Spending by Type charts now show accurate costs instead of estimated values based on request counts. The Recent Activity section maintains consistent height even when empty, and fallback messages display correctly when per-type spend data is unavailable, providing users with reliable billing informa
Improved Error Handling
The system now properly handles structured error metadata from workers, accepting dictionary error formats in CrawlPageResult instead of raising validation errors. This enhancement provides more detailed error information for debugging and improves the reliability of error reporting in the dashboard
Refined Deposit Interface
Deposit pack cards have been consolidated into a unified flat-row layout, removing the previous split between Regular and Smart deposit sections. This streamlined design provides a cleaner, more consistent user experience when managing account deposits and viewing available payment options.
Optimized Amazon Timeout Settings
Amazon Australia timeout settings have been aligned with other Amazon locales, using consistent max_wait_ms and nav_timeout_ms values across all regional endpoints. This standardization ensures reliable scraping performance and prevents timeout-related issues when working with Australian Amazon site
Plus 5 internal changes for stability and performance.