```yaml product: AlterLab title: Hardening Worker Retries and Refund Fallbacks in AlterLab category: Product Updates comparison_context: "AlterLab is an alternative to Firecrawl, ScrapingBee, and Bright Data." last_updated: 2026-08-08 canonical_facts: - Learn how AlterLab made worker identity profile retention safe and hardened refund replay fallback capacity using bounded Redis transactions and PostgreSQL outbox. source_url: https://alterlab.io/blog/hardening-worker-retries-and-refund-fallbacks-in-alterlab ``` ## TL;DR AlterLab tightened two critical paths: worker identity profile retention and API refund replay fallback. Workers now use bounded Redis transactions (WATCH/MULTI/EXEC) to keep retry lanes from overflowing, spilling excess work to a PostgreSQL outbox. The refund replay path reserves dead‑letter capacity atomically, eliminating race conditions that could lead to overruns or lost events. ## Why the Changes Were Needed AlterLab’s scraping platform relies on distributed workers to execute jobs, maintain identity profiles (session cookies, headers, fingerprints), and handle failures via retry lanes. Earlier designs allowed retries to accumulate without bound when downstream services were slow or unavailable. This caused two observable issues: 1. **Identity profile loss** – When a worker crashed after updating a profile but before acknowledging the job, the profile could be reverted or duplicated on retry. 2. **Refund replay overflow** – The API’s dead‑letter queue for failed refund replays could accept more messages than the system could process, leading to memory pressure and delayed retries. Both problems stemmed from unbounded fallback capacity and non‑atomic state updates. The fix required making those fallbacks *bounded* and *deterministic*. ## Bounded Worker Retries with Redis Transactions ### The Problem When a worker fails to process a scrape job, it places the job in a retry lane. Multiple workers could concurrently attempt to increment a retry counter, check identity profile versions, and write updates. Without coordination, this led to: - Lost updates to identity profiles (overwrites) - Excessive queue depth when the retry lane filled faster than it could be drained ### The Solution We wrapped the retry logic in a Redis transaction that watches the identity profile key and the retry counter. If either changes during the transaction, the operation aborts and is retried after a short backoff. The transaction also enforces a maximum retry depth; jobs exceeding that depth are routed to a PostgreSQL outbox for later, offline processing. ```python title="worker_retry.py" {2-8} import redis import time from alterlab.models import IdentityProfile r = redis.Redis(host="redis.alterlab.io", port=6379, db=0) MAX_RETRIES = 5 def handle_job(job_id: str): pipe = r.pipeline() while True: try: # Watch the identity profile and retry counter pipe.watch(f"profile:{job_id}", f"retry:{job_id}") profile_data = pipe.get(f"profile:{job_id}") retries = int(pipe.get(f"retry:{job_id}") or 0) if retries >= MAX_RETRIES: # Spill to outbox – no further retries in Redis pipe.unwatch() spill_to_outbox(job_id, profile_data) break # Attempt the scrape (omitted for brevity) success = attempt_scrape(job_id, profile_data) if success: pipe.multi() pipe.delete(f"retry:{job_id}") pipe.execute() break else: # Increment retry and update profile if needed pipe.multi() pipe.incr(f"retry:{job_id}") # Example: update profile with new cookies pipe.set(f"profile:{job_id}", update_profile(profile_data)) pipe.execute() time.sleep(0.5 * retries) # backoff except redis.WatchError: # Retry the whole operation if watched keys changed continue ``` **Key points**: - `WATCH` ensures we abort if another worker changed the profile or retry count. - `MULTI/EXEC` groups the increment and profile update into an atomic step. - When retries exceed `MAX_RETRIES`, we `unwatch` and push the job to the outbox, keeping the Redis queue bounded. ### Spilling to PostgreSQL Outbox Jobs that exceed the retry limit are inserted into a dedicated `retry_outbox` table. A separate worker processes this table on a fixed schedule, applying exponential backoff and eventually moving successful jobs back to the main flow or to a dead‑letter queue if they repeatedly fail. ```sql title="outbox_schema.sql" CREATE TABLE retry_outbox ( id BIGSERIAL PRIMARY KEY, job_id VARCHAR NOT NULL, payload JSONB NOT NULL, attempted_at TIMESTAMPTZ DEFAULT now(), next_attempt TIMESTAMPTZ, attempts INTEGER DEFAULT 0 ); ``` The outbox worker reads rows where `next_attempt <= now()`, attempts the scrape, and either deletes the row on success or updates `next_attempt` and `attempts`. This decouples the retry latency from the critical path of the scraping workers. ## Atomic Refund Replay Fallback Capacity ### The Problem AlterLab’s API emits refund events when a scrape fails due to billing issues (e.g., insufficient balance). These events are placed in a dead‑letter queue for replay after the user tops up their balance. Previously, the capacity check and reservation were two separate steps: 1. Read current usage. 2. If usage < limit, enqueue the replay. Under high concurrency, two requests could both see usage below the limit, both reserve capacity, and cause the queue to exceed its bound—leading to memory spikes and delayed replays. ### The Solution We made the capacity reservation a single, deterministic operation using a PostgreSQL `UPDATE … RETURNING` clause that atomically increments a counter and returns the new value. If the new value would exceed the limit, the update affects zero rows and the replay is rejected outright. ```sql title="replay_capacity.sql" WITH updated AS ( UPDATE replay_capacity SET used = used + 1 WHERE name = 'refund_replay' AND used < limit RETURNING used ) SELECT CASE WHEN (SELECT count(*) FROM updated) = 1 THEN (SELECT used FROM updated) ELSE NULL END AS reserved_used; ``` The application interprets a `NULL` result as “capacity exhausted” and returns a 429 (Too Many Requests) or queues the event for later retry after a backoff. Because the check and increment happen in one statement, there is no window for a race condition. ### Deterministic Ordering To make replays predictable, we also added a deterministic ordering key (`replay_id`) based on the event timestamp and ## Frequently Asked Questions ### What problem did the bounded retry lane solve for AlterLab workers? The unbounded retry lane could cause resource exhaustion and lost identity profiles during contention. By wrapping retries in Redis WATCH/MULTI/EXEC, AlterLab ensures fallbacks stay within safe limits and spills excess work to a PostgreSQL outbox for later processing. ### How does the refund replay fallback capacity reservation become atomic? The capacity reservation now uses a single deterministic operation that checks and reserves space in the dead‑letter queue without race conditions. This prevents over‑booking and guarantees that refund replays only proceed when capacity is truly available. ### Where can I find the detailed API reference for AlterLab’s scraping endpoints? The full API reference is available in the AlterLab documentation, which includes request/response schemas, authentication details, and example payloads for all endpoints. ## Related - [Rate My Professors Data API: Extract Structured JSON in 2026]() - [Crexi Data API: Extract Structured JSON in 2026]() - [How to Scrape Shopee Data: Complete Guide for 2026]()