Fixing CI Format Failures, Spend-Guard Atomicity & More

TL;DR

AlterLab’s engineering team recently shipped five changes: fixed formatting errors that blocked the CI gate, made the spend‑guard decrement atomic to prevent stuck credits, restored discovery cooldown after Redis write fallbacks, and added a replay worker that processes failed refund retries. These updates improve reliability, cost accuracy, and recovery for the scraping platform.

Fixing API Formatting CI Failures

The continuous integration pipeline blocks merges to main when code does not meet the project’s formatting standard. In a recent run, the gate flagged four API source files and one internal analytics service file. The violations were line‑length and spacing issues that differed from the service‑specific Black configuration.

To resolve the issue, engineers ran Black with the exact version and line‑length setting used by each service:

# API services use Black 23.9.0 with 100‑char lines
black --line-length 100 services/api/
black --line-length 100 services/internal/analytics/

After reformatting, the CI gate passed, and the changes were merged. The fix ensures that future edits will not stall the merge process as long as the same Black invocation is used in pre‑commit hooks.

Making Spend‑Guard Decrement Atomic

In the worker service, the function _finalize_structure_shutdown_handoff_failure guards against double‑decrementing a user’s spend balance. The original implementation performed two separate awaits:

1. redis.set(spend_guard_key, "1", nx=True, ex=86400) – sets a guard key only if it does not already exist.
2. If the guard was set, _decrement_spend_counters(...) is called to deduct the amount from Postgres.

A CancelledError raised between these two steps could leave the guard key set while the decrement never ran, effectively locking the user’s balance permanently.

The fix combines the guard acquisition and decrement into a single Lua script executed atomically by Redis:

async def _finalize_structure_shutdown_handoff_failure(
    redis: Redis, pg: Connection, user_id: str, amount: Decimal
) -> None:
    lua = """
    if redis.call('SETNX', KEYS[1], '1') == 1 then
        redis.call('EXPIRE', KEYS[1], ARGV[2])
        return 1
    else
        return 0
    end
    """
    # Acquire guard; if successful, run decrement in the same step
    guard_acquired = await redis.eval(lua, 1, spend_guard_key, str(amount), "86400")
    if guard_acquired:
        await _decrement_spend_counters(pg, user_id, amount)

Now the guard and decrement are inseparable; a cancellation cannot leave the system in an inconsistent state.

Restoring Discovery Cooldown After Redis Write Fallback

AlterLab caches provider discovery results in Redis. When Redis becomes temporarily unavailable for writes, the API falls back to a local in‑memory cache. Previously, after the write‑degraded snapshot expired, a discovery failure would cause the local cooldown timestamp to be cleared, leading to immediate retries on every call.

The updated logic re‑evaluates the process‑local write‑degraded latch after any locked cache check:

async def _maybe_fetch_discovery(self) -> ProviderInfo:
    if self._local_discovery_next_fetch_at and \
       time.time() < self._local_discovery_next_fetch_at:
        return await self._get_cached_discovery()

    try:
        async with self._discovery_lock:
            # Double‑check after acquiring the lock
            if self._local_discovery_next_fetch_at and \
               time.time() < self._local_discovery_next_fetch_at:
                return await self._get_cached_discovery()

            info = await self._fetch_from_provider()
            await self._cache_discovery(info)
            self._local_discovery_next_fetch_at = time.time() + self._cooldown
            return info
    except ProviderError:
        # On failure, arm the cooldown so we don't hammer the provider
        self._local_discovery_next_fetch_at = time.time() + self._cooldown
        raise

This ensures that even if the Redis write fails, the caller backs off for the configured cooldown period, preventing thundering‑herd behavior.

Adding a Replay Worker for Billing Refund Retries

When a refund attempt fails, the function record_failed_refund writes an entry to a bounded Redis list billing:refund_retry. Originally, this list was only an audit trail; no process consumed it, so failed refunds were never retried.

The new replay worker continuously pops items from the list and attempts to re‑apply the refund:

import asyncio
from services.api.app.billing import record_failed_refund, attempt_refund
from redis.asyncio import Redis

async def replay_worker(redis: Redis, poll_interval: float = 5.0) -> None:
    while True:
        # BRPOP blocks until an item is available or timeout
        item = await redis.brpop("billing:refund_retry", timeout=poll_interval)
        if not item:
            continue
        _, payload = item
        refund_data = json.loads(payload)
        try:
            await attempt_refund(refund_data["refund_id"])
            # Successfully reapplied; optionally remove from a dead‑letter set
        except Exception as exc:
            # Log and leave the item for a later retry
            logger.error(
                "Refund replay failed for %s: %s",
                refund_data["refund_id"],
                exc,
                exc_info=True,
            )

The worker runs as a separate asyncio task alongside the API service. It transforms the billing:refund_retry list from a passive log into an active recovery mechanism, reducing manual intervention and improving revenue reliability.

Infographic: Workflow of the Refund Replay Worker

Internal References

For developers looking to integrate these changes, see the Python SDK for a batteries‑included client that handles retries and formatting automatically. Detailed endpoint specifications are available in the API reference. If you need to review pricing or plan adjustments, visit the pricing page.

Takeaway

The