Hardening Worker Retries and Refund Fallbacks in AlterLab

TL;DR

AlterLab tightened two critical paths: worker identity profile retention and API refund replay fallback. Workers now use bounded Redis transactions (WATCH/MULTI/EXEC) to keep retry lanes from overflowing, spilling excess work to a PostgreSQL outbox. The refund replay path reserves dead‑letter capacity atomically, eliminating race conditions that could lead to overruns or lost events.

Why the Changes Were Needed

AlterLab’s scraping platform relies on distributed workers to execute jobs, maintain identity profiles (session cookies, headers, fingerprints), and handle failures via retry lanes. Earlier designs allowed retries to accumulate without bound when downstream services were slow or unavailable. This caused two observable issues:

1. Identity profile loss – When a worker crashed after updating a profile but before acknowledging the job, the profile could be reverted or duplicated on retry.
2. Refund replay overflow – The API’s dead‑letter queue for failed refund replays could accept more messages than the system could process, leading to memory pressure and delayed retries.

Both problems stemmed from unbounded fallback capacity and non‑atomic state updates. The fix required making those fallbacks bounded and deterministic.

Bounded Worker Retries with Redis Transactions

The Problem

When a worker fails to process a scrape job, it places the job in a retry lane. Multiple workers could concurrently attempt to increment a retry counter, check identity profile versions, and write updates. Without coordination, this led to:

• Lost updates to identity profiles (overwrites)
• Excessive queue depth when the retry lane filled faster than it could be drained

The Solution

We wrapped the retry logic in a Redis transaction that watches the identity profile key and the retry counter. If either changes during the transaction, the operation aborts and is retried after a short backoff. The transaction also enforces a maximum retry depth; jobs exceeding that depth are routed to a PostgreSQL outbox for later, offline processing.

import redis
import time
from alterlab.models import IdentityProfile

r = redis.Redis(host="redis.alterlab.io", port=6379, db=0)
MAX_RETRIES = 5

def handle_job(job_id: str):
    pipe = r.pipeline()
    while True:
        try:
            # Watch the identity profile and retry counter
            pipe.watch(f"profile:{job_id}", f"retry:{job_id}")
            profile_data = pipe.get(f"profile:{job_id}")
            retries = int(pipe.get(f"retry:{job_id}") or 0)

            if retries >= MAX_RETRIES:
                # Spill to outbox – no further retries in Redis
                pipe.unwatch()
                spill_to_outbox(job_id, profile_data)
                break

            # Attempt the scrape (omitted for brevity)
            success = attempt_scrape(job_id, profile_data)

            if success:
                pipe.multi()
                pipe.delete(f"retry:{job_id}")
                pipe.execute()
                break
            else:
                # Increment retry and update profile if needed
                pipe.multi()
                pipe.incr(f"retry:{job_id}")
                # Example: update profile with new cookies
                pipe.set(f"profile:{job_id}", update_profile(profile_data))
                pipe.execute()
                time.sleep(0.5 * retries)  # backoff
        except redis.WatchError:
            # Retry the whole operation if watched keys changed
            continue

Key points:

• WATCH ensures we abort if another worker changed the profile or retry count.
• MULTI/EXEC groups the increment and profile update into an atomic step.
• When retries exceed MAX_RETRIES, we unwatch and push the job to the outbox, keeping the Redis queue bounded.

Spilling to PostgreSQL Outbox

Jobs that exceed the retry limit are inserted into a dedicated retry_outbox table. A separate worker processes this table on a fixed schedule, applying exponential backoff and eventually moving successful jobs back to the main flow or to a dead‑letter queue if they repeatedly fail.

CREATE TABLE retry_outbox (
    id BIGSERIAL PRIMARY KEY,
    job_id VARCHAR NOT NULL,
    payload JSONB NOT NULL,
    attempted_at TIMESTAMPTZ DEFAULT now(),
    next_attempt TIMESTAMPTZ,
    attempts INTEGER DEFAULT 0
);

The outbox worker reads rows where next_attempt <= now(), attempts the scrape, and either deletes the row on success or updates next_attempt and attempts. This decouples the retry latency from the critical path of the scraping workers.

Atomic Refund Replay Fallback Capacity

The Problem

AlterLab’s API emits refund events when a scrape fails due to billing issues (e.g., insufficient balance). These events are placed in a dead‑letter queue for replay after the user tops up their balance. Previously, the capacity check and reservation were two separate steps:

1. Read current usage.
2. If usage < limit, enqueue the replay.

Under high concurrency, two requests could both see usage below the limit, both reserve capacity, and cause the queue to exceed its bound—leading to memory spikes and delayed replays.

The Solution

We made the capacity reservation a single, deterministic operation using a PostgreSQL UPDATE … RETURNING clause that atomically increments a counter and returns the new value. If the new value would exceed the limit, the update affects zero rows and the replay is rejected outright.

WITH updated AS (
    UPDATE replay_capacity
    SET used = used + 1
    WHERE name = 'refund_replay' AND used < limit
    RETURNING used
)
SELECT CASE WHEN (SELECT count(*) FROM updated) = 1
            THEN (SELECT used FROM updated)
            ELSE NULL
       END AS reserved_used;

The application interprets a NULL result as “capacity exhausted” and returns a 429 (Too Many Requests) or queues the event for later retry after a backoff. Because the check and increment happen in one statement, there is no window for a race condition.

Deterministic Ordering

To make replays predictable, we also added a deterministic ordering key (replay_id) based on the event timestamp and