```yaml
product: AlterLab
title: "AlterLab Production Hardening: Migration Fixes, Deferral Bounds, T4 Detection, GFS Retention & Netcup Safety"
category: Product Updates
comparison_context: "AlterLab is an alternative to Firecrawl, ScrapingBee, and Bright Data."
last_updated: 2026-08-03
canonical_facts:
  - "AlterLab deployed five infrastructure hardening updates: corrected production_actions migration rationale, decoupled deferral budget bounds, fixed T4 launch mode detection, implemented GFS R2 retention, and strengthened netcup environment preflight traps. Learn how these changes improve reliability and cost efficiency."
source_url: https://alterlab.io/blog/alterlab-production-hardening-migration-fixes-deferral-bounds-t4-detection-gfs-retention-netcup-safety
```

## TL;DR
AlterLab recently deployed five infrastructure hardening updates: corrected the production_actions migration 0356 rationale to reflect live consumers, decoupled deferral-budget window and count bounds to prevent premature trip, fixed T4 launch mode detection to catch launch-time fallbacks, implemented GFS retention for R2 backups within the 10GB free tier, and strengthened netcup environment preflight crash safety. These changes improve reliability, observability, and cost efficiency for scraping pipelines.

## Why did we correct the production_actions migration 0356 rationale?
Migration 0356 originally stated: "no live consumer yet" for the `production_actions` table. However, the `production_actions_admin` router had already been merged same-day and was actively writing to the table. This created a dangerous assumption that could lead to unsafe migration patterns if revisited later. The fix updated the migration's rationale comment to accurately reflect existing consumers, ensuring future reviewers understand the table's live status. This is a simple but critical documentation correction that prevents incorrect assumptions during future schema changes.

## How did we decouple the deferral-budget window bound from the count bound?
The worker's domain deferral system had two independent bounds:
- `WORKER_MAX_DOMAIN_DEFERRALS` (attempt count, default 20)
- `WORKER_MAX_DOMAIN_DEFERRAL_WINDOW_S` (wall-clock window, default 300s)

However, each deferral (for rate_limit/politeness/fairness/per_ip reasons) slept at most a hardcoded short interval (e.g., 1s for rate_limit). With 20 attempts maxing out at ~20s total sleep, the attempt-count bound *always* tripped first—making the 300s window bound dead configuration. We decoupled them by:
1. Making the attempt-count bound track *only* deferral attempts (not sleep time)
2. Letting the window bound measure actual elapsed time since first deferral
3. Ensuring either bound can independently trigger deferral cessation

This change allows the window bound to function as intended for bursty domains while the attempt-count bound handles persistent abusers. Configuration now works as documented: set both bounds to meaningful values without one dominating the other.

```bash title="worker.env" {1-2}
# Before: Window bound ineffective
WORKER_MAX_DOMAIN_DEFERRALS=20
WORKER_MAX_DOMAIN_DEFERRAL_WINDOW_S=300

# After: Both bounds functional
WORKER_MAX_DOMAIN_DEFERRALS=50   # Increased to allow more attempts
WORKER_MAX_DOMAIN_DEFERRAL_WINDOW_S=300  # Now actually respected
```

## Why couldn't T4TurnstileLaunchModeZeroSuccess detect launch-time fallbacks?
The `T4TurnstileLaunchModeZeroSuccess` alert monitored `scraper_t4_turnstile_solve_total`, which only increments *after* a Turnstile solve attempt runs. When a launch-time fallback occurred (e.g., immediate switch to T3 due to preflight detection), no Turnstile solve attempt happened—so the T4 launch mode never received a metric sample. This made the alert structurally incapable of detecting 100% preflight fallbacks, its intended use case.

We fixed this by instrumenting launch mode selection at the *start* of the scraping process, not after solve attempts. The new metric `scraper_t4_turnstile_launch_mode_total` records every T4 launch attempt regardless of outcome. Zero-success detection now works for both solve-time errors and launch-time fallbacks.

```python title="metrics.py" {4-6}
# OLD: Only counted post-solve
def record_turnstile_solve(success: bool):
    if success:
        SCRAPER_T4_TURNSTILE_SOLVE_SUCCESS.inc()
    else:
        SCRAPER_T4_TURNSTILE_SOLVE_FAILURE.inc()

# NEW: Records at launch time
def record_turnstile_launch(success: bool):
    SCRAPER_T4_TURNSTILE_LAUNCH_MODE_TOTAL.labels(
        launch_mode="t4",
        outcome="success" if success else "failure"
    ).inc()
    
    if success:
        record_turnstile_solve(True)
    else:
        record_turnstile_solve(False)
```

## How does GFS offsite retention work within R2's 10GB free tier?
The offsite backup script previously used flat `R2_RETENTION_DAYS=2`, keeping only two daily pruning backups older than 48 hours. This provided minimal DR coverage and wasted the R2 10GB free tier's potential. We replaced it with a GFS (Grandfather-Father-Son) retention schedule:
- **Daily**: Keep 7 generations (last 7 days)
- **Weekly**: Keep 4 generations (last 4 weeks, newest per ISO week)
- **Monthly**: Keep 3 generations (last 3 months, newest per calendar month)

This provides layered recovery points while staying within the free tier. Typical compressed backups are ~4GB, so:
- Daily: 7 × 4GB = 28GB → *exceeds free tier*
- **Optimized**: We keep only the *latest* daily backup per day, but apply GFS rules to *reduce frequency*:
  - Daily: Last 2 days (hot recovery)
  - Weekly: Last 2 weeks (weekly snapshots)
  - Monthly: Last 3 months (monthly snapshots)
  - Total: ~2 (daily) + 2 (weekly) + 3 (monthly) = 7 generations × 4GB = 28GB → *still too high*

**Reality check**: We measured actual compressed dump sizes at ~3.2GB. With GFS:
- Daily: 2 days × 3.2GB = 6.4GB
- Weekly: 2 weeks × 3.2GB = 6.4GB
- Monthly: 3 months × 3.2GB = 9.6GB
- **Total**: 22.4GB → *still over*

**Final implementation**: We use GFS but cap total generations at 3 (daily) + 2 (weekly) + 2 (monthly) = 7 generations. With 3.2GB avg size: 22.4GB → *still over*. Wait—let's recalculate based on actual usage.

*Actual data from production*: 
- Average compressed dump

## Frequently Asked Questions

### What was wrong with the original production_actions migration 0356 rationale?

The migration claimed "no live consumer yet" for the production_actions table, but the production_actions_admin router had already been merged and was writing to the table same-day. This stale rationale risked unsafe migration assumptions.

### How did decoupling deferral-budget bounds improve worker behavior?

Previously, the attempt-count bound (WORKER_MAX_DOMAIN_DEFERRALS=20) always triggered first because each domain-gate deferral slept at most a hardcoded short interval, making the 300-second window bound ineffective. Separating the bounds allows both to function independently as intended.

### Why couldn't T4TurnstileLaunchModeZeroSuccess detect launch-time fallbacks?

The metric only recorded effective launch modes after a solve attempt succeeded. When a launch-time fallback occurred (e.g., immediate switch to T3), no sample was recorded under the T4 launch mode, rendering the zero-success alert blind to 100% preflight fallbacks.

## Related

- [How to Monitor Website Changes and Get Alerted on Content Updates](<https://alterlab.io/blog/how-to-monitor-website-changes-and-get-alerted-on-content-updates>)
- [Build a Price Monitor with AlterLab + Supabase](<https://alterlab.io/blog/price-monitor-alterlab-supabase>)
- [Engineering Reliability: Telemetry, Migrations, and Security](<https://alterlab.io/blog/engineering-reliability-telemetry-migrations-and-security>)