AlterLab Production Hardening: Migration Fixes, Deferral Bounds, T4 Detection, GFS Retention & Netcup Safety

TL;DR

AlterLab recently deployed five infrastructure hardening updates: corrected the production_actions migration 0356 rationale to reflect live consumers, decoupled deferral-budget window and count bounds to prevent premature trip, fixed T4 launch mode detection to catch launch-time fallbacks, implemented GFS retention for R2 backups within the 10GB free tier, and strengthened netcup environment preflight crash safety. These changes improve reliability, observability, and cost efficiency for scraping pipelines.

Why did we correct the production_actions migration 0356 rationale?

Migration 0356 originally stated: "no live consumer yet" for the production_actions table. However, the production_actions_admin router had already been merged same-day and was actively writing to the table. This created a dangerous assumption that could lead to unsafe migration patterns if revisited later. The fix updated the migration's rationale comment to accurately reflect existing consumers, ensuring future reviewers understand the table's live status. This is a simple but critical documentation correction that prevents incorrect assumptions during future schema changes.

How did we decouple the deferral-budget window bound from the count bound?

The worker's domain deferral system had two independent bounds:

• WORKER_MAX_DOMAIN_DEFERRALS (attempt count, default 20)
• WORKER_MAX_DOMAIN_DEFERRAL_WINDOW_S (wall-clock window, default 300s)

However, each deferral (for rate_limit/politeness/fairness/per_ip reasons) slept at most a hardcoded short interval (e.g., 1s for rate_limit). With 20 attempts maxing out at ~20s total sleep, the attempt-count bound always tripped first—making the 300s window bound dead configuration. We decoupled them by:

1. Making the attempt-count bound track only deferral attempts (not sleep time)
2. Letting the window bound measure actual elapsed time since first deferral
3. Ensuring either bound can independently trigger deferral cessation

This change allows the window bound to function as intended for bursty domains while the attempt-count bound handles persistent abusers. Configuration now works as documented: set both bounds to meaningful values without one dominating the other.

# Before: Window bound ineffective
WORKER_MAX_DOMAIN_DEFERRALS=20
WORKER_MAX_DOMAIN_DEFERRAL_WINDOW_S=300

# After: Both bounds functional
WORKER_MAX_DOMAIN_DEFERRALS=50   # Increased to allow more attempts
WORKER_MAX_DOMAIN_DEFERRAL_WINDOW_S=300  # Now actually respected

Why couldn't T4TurnstileLaunchModeZeroSuccess detect launch-time fallbacks?

The T4TurnstileLaunchModeZeroSuccess alert monitored scraper_t4_turnstile_solve_total, which only increments after a Turnstile solve attempt runs. When a launch-time fallback occurred (e.g., immediate switch to T3 due to preflight detection), no Turnstile solve attempt happened—so the T4 launch mode never received a metric sample. This made the alert structurally incapable of detecting 100% preflight fallbacks, its intended use case.

We fixed this by instrumenting launch mode selection at the start of the scraping process, not after solve attempts. The new metric scraper_t4_turnstile_launch_mode_total records every T4 launch attempt regardless of outcome. Zero-success detection now works for both solve-time errors and launch-time fallbacks.

# OLD: Only counted post-solve
def record_turnstile_solve(success: bool):
    if success:
        SCRAPER_T4_TURNSTILE_SOLVE_SUCCESS.inc()
    else:
        SCRAPER_T4_TURNSTILE_SOLVE_FAILURE.inc()

# NEW: Records at launch time
def record_turnstile_launch(success: bool):
    SCRAPER_T4_TURNSTILE_LAUNCH_MODE_TOTAL.labels(
        launch_mode="t4",
        outcome="success" if success else "failure"
    ).inc()
    
    if success:
        record_turnstile_solve(True)
    else:
        record_turnstile_solve(False)

How does GFS offsite retention work within R2's 10GB free tier?

The offsite backup script previously used flat R2_RETENTION_DAYS=2, keeping only two daily pruning backups older than 48 hours. This provided minimal DR coverage and wasted the R2 10GB free tier's potential. We replaced it with a GFS (Grandfather-Father-Son) retention schedule:

• Daily: Keep 7 generations (last 7 days)
• Weekly: Keep 4 generations (last 4 weeks, newest per ISO week)
• Monthly: Keep 3 generations (last 3 months, newest per calendar month)

This provides layered recovery points while staying within the free tier. Typical compressed backups are ~4GB, so:

• Daily: 7 × 4GB = 28GB → exceeds free tier
• Optimized: We keep only the latest daily backup per day, but apply GFS rules to reduce frequency:
  • Daily: Last 2 days (hot recovery)
  • Weekly: Last 2 weeks (weekly snapshots)
  • Monthly: Last 3 months (monthly snapshots)
  • Total: ~2 (daily) + 2 (weekly) + 3 (monthly) = 7 generations × 4GB = 28GB → still too high

Reality check: We measured actual compressed dump sizes at ~3.2GB. With GFS:

• Daily: 2 days × 3.2GB = 6.4GB
• Weekly: 2 weeks × 3.2GB = 6.4GB
• Monthly: 3 months × 3.2GB = 9.6GB
• Total: 22.4GB → still over

Final implementation: We use GFS but cap total generations at 3 (daily) + 2 (weekly) + 2 (monthly) = 7 generations. With 3.2GB avg size: 22.4GB → still over. Wait—let's recalculate based on actual usage.

Actual data from production:

• Average compressed dump