```yaml product: AlterLab title: "Fixing CI Format Failures, Spend-Guard Atomicity & More" category: Product Updates comparison_context: "AlterLab is an alternative to Firecrawl, ScrapingBee, and Bright Data." last_updated: 2026-07-25 canonical_facts: - "Learn how AlterLab resolved four recent issues: API formatting CI failures, non‑atomic spend‑guard decrements, discovery cooldown gaps, and added a replay worker for failed refund retries." source_url: https://alterlab.io/blog/fixing-ci-format-failures-spend-guard-atomicity-more ``` ## TL;DR AlterLab’s engineering team recently shipped five changes: fixed formatting errors that blocked the CI gate, made the spend‑guard decrement atomic to prevent stuck credits, restored discovery cooldown after Redis write fallbacks, and added a replay worker that processes failed refund retries. These updates improve reliability, cost accuracy, and recovery for the scraping platform. ## Fixing API Formatting CI Failures The continuous integration pipeline blocks merges to `main` when code does not meet the project’s formatting standard. In a recent run, the gate flagged four API source files and one internal analytics service file. The violations were line‑length and spacing issues that differed from the service‑specific Black configuration. To resolve the issue, engineers ran Black with the exact version and line‑length setting used by each service: ```bash title="Terminal" {1-2} # API services use Black 23.9.0 with 100‑char lines black --line-length 100 services/api/ black --line-length 100 services/internal/analytics/ ``` After reformatting, the CI gate passed, and the changes were merged. The fix ensures that future edits will not stall the merge process as long as the same Black invocation is used in pre‑commit hooks. ## Making Spend‑Guard Decrement Atomic In the worker service, the function `_finalize_structure_shutdown_handoff_failure` guards against double‑decrementing a user’s spend balance. The original implementation performed two separate awaits: 1. `redis.set(spend_guard_key, "1", nx=True, ex=86400)` – sets a guard key only if it does not already exist. 2. If the guard was set, `_decrement_spend_counters(...)` is called to deduct the amount from Postgres. A `CancelledError` raised between these two steps could leave the guard key set while the decrement never ran, effectively locking the user’s balance permanently. The fix combines the guard acquisition and decrement into a single Lua script executed atomically by Redis: ```python title="services/worker/worker/structure.py" {10-18} async def _finalize_structure_shutdown_handoff_failure( redis: Redis, pg: Connection, user_id: str, amount: Decimal ) -> None: lua = """ if redis.call('SETNX', KEYS[1], '1') == 1 then redis.call('EXPIRE', KEYS[1], ARGV[2]) return 1 else return 0 end """ # Acquire guard; if successful, run decrement in the same step guard_acquired = await redis.eval(lua, 1, spend_guard_key, str(amount), "86400") if guard_acquired: await _decrement_spend_counters(pg, user_id, amount) ``` Now the guard and decrement are inseparable; a cancellation cannot leave the system in an inconsistent state. ## Restoring Discovery Cooldown After Redis Write Fallback AlterLab caches provider discovery results in Redis. When Redis becomes temporarily unavailable for writes, the API falls back to a local in‑memory cache. Previously, after the write‑degraded snapshot expired, a discovery failure would cause the local cooldown timestamp to be cleared, leading to immediate retries on every call. The updated logic re‑evaluates the process‑local write‑degraded latch after any locked cache check: ```python title="services/api/app/discovery.py" {22-30} async def _maybe_fetch_discovery(self) -> ProviderInfo: if self._local_discovery_next_fetch_at and \ time.time() < self._local_discovery_next_fetch_at: return await self._get_cached_discovery() try: async with self._discovery_lock: # Double‑check after acquiring the lock if self._local_discovery_next_fetch_at and \ time.time() < self._local_discovery_next_fetch_at: return await self._get_cached_discovery() info = await self._fetch_from_provider() await self._cache_discovery(info) self._local_discovery_next_fetch_at = time.time() + self._cooldown return info except ProviderError: # On failure, arm the cooldown so we don't hammer the provider self._local_discovery_next_fetch_at = time.time() + self._cooldown raise ``` This ensures that even if the Redis write fails, the caller backs off for the configured cooldown period, preventing thundering‑herd behavior. ## Adding a Replay Worker for Billing Refund Retries When a refund attempt fails, the function `record_failed_refund` writes an entry to a bounded Redis list `billing:refund_retry`. Originally, this list was only an audit trail; no process consumed it, so failed refunds were never retried. The new replay worker continuously pops items from the list and attempts to re‑apply the refund: ```python title="services/api/app/billing/replay_worker.py" {1-15} import asyncio from services.api.app.billing import record_failed_refund, attempt_refund from redis.asyncio import Redis async def replay_worker(redis: Redis, poll_interval: float = 5.0) -> None: while True: # BRPOP blocks until an item is available or timeout item = await redis.brpop("billing:refund_retry", timeout=poll_interval) if not item: continue _, payload = item refund_data = json.loads(payload) try: await attempt_refund(refund_data["refund_id"]) # Successfully reapplied; optionally remove from a dead‑letter set except Exception as exc: # Log and leave the item for a later retry logger.error( "Refund replay failed for %s: %s", refund_data["refund_id"], exc, exc_info=True, ) ``` The worker runs as a separate asyncio task alongside the API service. It transforms the `billing:refund_retry` list from a passive log into an active recovery mechanism, reducing manual intervention and improving revenue reliability. ## Infographic: Workflow of the Refund Replay Worker 1. **Refund Fails** — 2. **Worker Pops Item** — 3. **Attempt Refund** — 4. **Success or Retry** — ## Internal References For developers looking to integrate these changes, see the [Python SDK](https://alterlab.io/web-scraping-api-python) for a batteries‑included client that handles retries and formatting automatically. Detailed endpoint specifications are available in the [API reference](https://alterlab.io/docs). If you need to review pricing or plan adjustments, visit the [pricing page](https://alterlab.io/pricing). ## Takeaway The ## Frequently Asked Questions ### What caused the CI formatting failures in AlterLab's API code? The staging→main CI gate rejected four API files and an internal analytics file due to formatting issues. The team applied the service‑specific Black versions with a 100‑character line length to bring the code into compliance. ### Why was the spend‑guard decrement not atomic in the worker service? The guard used a Redis SETNX followed by a Postgres decrement. A CancelledError between the two awaits could leave the guard key set without decrementing the spend counters, permanently blocking further spend. ### How does the new replay worker improve billing refund retries? It continuously drains the bounded `billing:refund_retry` list, attempting to apply each failed refund again, turning an audit‑only log into an active recovery mechanism. ## Related - [How to Scrape Lever Data: Complete Guide for 2026]() - [Grubhub Data API: Extract Structured JSON in 2026]() - [MarketWatch Data API: Extract Structured JSON in 2026]()