AlterLab Engineering Update: Fixing API Deadlocks & Worker Leaks

TL;DR

This update resolves critical system instabilities including a deadlock in infrastructure capacity gates during deployment, a 504 timeout caused by poisoned request deduplication slots, and memory leaks in the worker queue processing layer. These fixes improve API reliability for high-concurrency sync scrapes and ensure stable infrastructure scaling.

Infrastructure: Solving Capacity Gate Deadlocks

We identified a race condition in our live-capacity gate—the mechanism that regulates how many active scraping sessions are deployed across our Netcup infrastructure.

The Deadlock Scenario

During "shrink + grow" deployments (where the system scales down old nodes and scales up new ones simultaneously), the memory delta calculation was failing to account for the transition state. This resulted in a deadlock where the system believed it was at maximum capacity despite having available headroom, preventing new nodes from initializing.

We have implemented a fix to ensure the net memory delta is calculated atomically. Additionally, we updated the validation logic for ACTUAL_* replica counts. Previously, these counts were handled as generic integers; they are now strictly validated through netcup_is_uint to prevent negative value injections or type mismatches from triggering gate failures.

API: Eliminating Phantom Deduplication Hangs

A subset of users performing repeat synchronous scrapes experienced intermittent 85-second hangs followed by 504 Gateway Timeouts.

The Root Cause: Poisoned Slots

AlterLab uses an in-flight deduplication mechanism to optimize resources. If multiple requests for the same URL arrive simultaneously, the API holds them in a "slot" and fulfills all of them with a single upstream response.

We discovered that when resolve_extraction_route raised a 400 error, the deduplication slot was not being released. This "poisoned" the slot, causing subsequent identical requests to wait for a response that would never arrive until the global timeout was reached.

The fix ensures that the deduplication slot is released in a finally block, regardless of whether the route resolution succeeds or fails. This ensures that invalid requests do not block valid subsequent traffic.

try:
    route = resolve_extraction_route(request)
    result = execute_scrape(route)
except Exception as e:
    handle_error(e)
finally:
    # FIX: Always release the dedup slot to prevent 504 hangs
    dedup_manager.release_slot(request.dedup_key) 

Worker Layer: Queue Reaping and Proxy Logic

The worker nodes responsible for executing the actual scrapes required several stability updates to handle long-running tasks and network edge cases.

The Processing List Reaper

In high-throughput environments, tasks can occasionally vanish from the processing list without sending an acknowledgement (ACK). This leads to "ghost" tasks that occupy worker slots but perform no work.

We implemented a periodic reaper for the structure-queue :processing list. This reaper scans for tasks that have been in the processing state beyond a reasonable threshold and re-queues them. To prevent double-processing, we introduced a claimed-at hash invariant, ensuring that only the current owner of a task can acknowledge it.

FQDN Gateway Misclassification

We found a bug where the _is_system_proxy() check was misclassifying internal gateway hostnames that ended with a trailing dot (the formal FQDN notation). This caused the worker to treat internal traffic as "Bring Your Own Proxy" (BYOP), leading to routing errors. The logic has been updated to strip trailing dots before classification.

Web: Sitemap and SEO Hygiene

To improve the discoverability of our API docs, we cleaned up our sitemap generation logic.

1. Dead URL Removal: We identified ~84 dead URLs under the /docs-md/* path that were returning 404s. These have been purged from the sitemap.
2. Thin Content Gate: We updated the sitemap generator to exclude "thin" blog categories. If a category is marked with a noindex gate to prevent SEO devaluation, it is now automatically excluded from the XML sitemap to maintain a high crawl-to-index ratio.

Practical Implementation: Handling Sync Scrapes

For developers integrating these fixes into their pipelines, we recommend using the Python SDK to handle request timeouts and retries gracefully.

from alterlab import Client
import requests

client = Client(api_key="AL_12345")

try:
    # Use the /api/v1/scrape endpoint for synchronous data retrieval
    response = client.scrape(
        url="https://example-ecommerce-site.com/product/123",
        formats=["json"]
    )
    print(response.json())
except requests.exceptions.Timeout:
    print("Request timed out; check AlterLab status page.")

Alternatively, for simple integrations, you can use cURL:

curl -X POST https://api.alterlab.io/v1/scrape \
  -H "X-API-Key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example-ecommerce-site.com", "formats": ["json"]}'

Summary of Changes

Takeaway

This update focuses on the "unseen" parts of the platform—the plumbing of the worker queues and the infrastructure gates. By eliminating the phantom deduplication hangs and the deployment deadlocks, we've increased the overall reliability of the pay-as-you-go API for users running massive, concurrent scraping jobs.