Changelog

daily Stripe reconciliation snapshot and per-source freshness alert

typed Stripe ids on deposit ledger grants, idempotent backfill, deposit_refunded/deposit_disputed

bot_reasons persistence and fixed-screen / spoofed-Google fingerprints

durable analytics_identity_links, OAuth anonymous_id carry-through and enriched user_touches

GSC brand/non-brand split metrics and Bing brand snapshot

enable B2B lead detection and deliver a weekly ranked prospect list

gate Cloudflare deploy-token scope drift before traffic switch

tier intelligence never records T3.5 successes and skips to T4 on a single (false) T4 success

follow-up: 1 deferred item(s) (deferred from )

stage shared registry entries, acquisition taxonomy and emit-failure metric for

day-35 outcome check for /best-web-scraping-api and /vs

server signup funnel events, visible emit failures and CF-IPCountry login country

Google/Bing parser coverage with golden fixtures and a parser-drift monitor

evaluate action results are Python repr strings, not JSON

disambiguate the AlterLab entity markup to recover brand sitelinks

follow-up: 4 deferred item(s) (deferred from )

follow-up: 5 deferred item(s) (deferred from )

zero signups since the Oct 25 passwordless deploy

pre-capture consent hider blanks the page — CMP hide selectors injected as a stylesheet bypass the html/body guard (Hu-manity .hu-visible on <body>)

Chrome header wire order from unautomated Chrome, per protocol and request kind

HTTP client 0.17.0 and a chrome152 TLS baseline

Google/Bing search jobs are enqueued on the HTTP queue with no browser-tier routing

batch cache hits are charged and batch item credits_used is always 1000

page/num_results contract differs per engine; DDG pages 4-10 fetch and bill for zero results

single-search spend-limit reservation is never reconciled or reversed

response and billing report the requested engine, not the engine that served the result after failover ( fixed /v1/serp only)

single search stays in 'scraping' forever when a scrape child fails

sync 30 s timeout returns 504 without search_id; the late result is unreachable

search engine search fails 32% over 7 days; worker_timeout bursts (10-24, 10-26) replace parse_error

result cache key ignores engine; cross-engine cache hits are refunded, so a Google request can be served search engine data for free

ASCII keyword relevance filter silently drops valid organic results

browser-earned clearance is never reusable by T3 — persona unbound, exit not shared, CF solved ~22 s per job (0 store hits)

HTTP client 0.17.0 and an opt-in chrome152 wire profile that carries 0xCA34

pin the restore drill's trusted verifier to the v11 backup contract commit

third round of staging→main deploy check fixes

second round of staging→main deploy check fixes

green the staging→main deploy checks

GSC snapshot skips late-published days and never backfills gaps

branded enterprise client portal with per-client settings

repin production-action host-key helper to the keyscan banner fix

unblock deploy run 40181642888 (keyscan banner, openapi sync, two test flakes)

follow-up: 1 deferred item(s) (deferred from )

atomic email-code verify, fixed-window analytics relay limit, batch actions_on_error (deploy review)

scope T4 action results per attempt; JSONL-only daily_rollup deferral (deploy review)

test-api/test-herald failures on deploy

test-web/test-worker failures on deploy

browser-actions docs suites from are unwired — check-test-suite-wiring fails on staging

volume-aware auto-promotion — rare-class auto_promote changes can never reach shadow_min_n=100 and sit in shadow forever

reconcile T4 by successful-attempt egress and report billing.egress ( part 5)

built-in consent dismissal that acts on the live page before screenshots

wire direct-first gate into the unified T4 path ( part 3)

align dedicated browser route with the direct-first gate ( part 4)

BROWSER_DIRECT_MICROCENTS and egress-aware browser cost helper ( part 1)

evidence-gated direct-first browser gate with shadow mode, egress caps and rollout guards ( part 2)

B2B enterprise provisioning UI and open-invoice pay link

follow-up: 2 deferred item(s) (deferred from )

browser actions product page with use cases

browser actions in batch, crawl, scheduler and monitors

competitive actions pricing — included in browser tier, screenshot parity, pre-debit floor

per-action timeout/optional/on_error and partial-result warnings for actions

add select, hover, click_text, extract and wait_for state actions

align dashboard actions builder with API action types and add Run on URL

typed browser action models and action_results in Python and Node SDKs

expose browser actions, screenshot and PDF options in MCP scrape tools

map competitor API actions in the competitor API-compat endpoint

follow-up: 1 deferred item(s) (deferred from )

proxy cost rate is 0.80/GB in config but real DataImpulse cost is 1.00/GB

action_results never returned — dropped in scrape→structure handoff (0/155 jobs in 30d)

browser actions silently dropped on dedicated JS consumer (screenshot/pdf/mode=js jobs)

action executor ignores wait.selector, scroll default mismatch, no tests

enforce a cumulative time budget in the browser action loop

follow-up: 2 deferred item(s) (deferred from )

The onboarding experience now guides you directly into a pre-configured sample request within the playground. You can execute a reliable initial scrape immediately upon signup without manual configuration.

Requests hitting irrecoverable block pages now terminate immediately without stalling through retry cascades. This reduces response latency and frees up request throughput when target sites are blocked.

The request engine now remembers failure outcomes across geographic locations to bypass blocked paths automatically. This improves success rates for location-sensitive target sites while optimizing overall request performance.

Signup grants and balance indicators now clearly display a $1 free credit alongside per-tier pricing details. This ensures predictable usage tracking across different scraping parameters and target complexities.

Plus 14 internal improvements for better reliability and performance.

Fixed an issue where search engine API calls returned gateway timeout errors on pages with no organic search results. Search requests now complete successfully and return valid structured payloads regardless of result volume.

Resolved a security policy conflict that blocked redirects during the OAuth consent step. External client applications can now authenticate and complete authorization flows without interruption.

Requests routed through remote tool proxies now correctly forward the original end-user IP address. This ensures IP allowlists and geographic access policies are enforced accurately across proxy connections.

server-authoritative deposit_succeeded event keyed by Stripe event id; browser purchase deduplicated

identity stitching — analytics.identify and hashed analytics user_id after login

temp-mail MX browser compatibility and per-domain signup velocity limits for non-mainstream domains

passwordless email-code signup to replace the three-step password path

one canonical alternative page each for competitor, competitor and competitor, built on unique data

first-party analytics relay with server-side bot filtering, replacing the static screen-size exclusion

shared event registry with generated TS/Python constants and analytics reserved-param guard

direct-answer pages for Bing's zero-click LLM fan-out queries, pushed via IndexNow

shared signup email assessor on every signup path with daily-synced disposable domain lists

reproducible public success-rate benchmark page with methodology and raw data

server-side bot classifier and analytics_events.is_bot excluded from Pathfinder rollups

daily analytics reconciliation guard and signup-burst monitoring system

anti-bot system device-check differential capture: real-browser vs. QuickJS plaintext signals on the same sticky exit

metric-exclusion flags and one canonical definition each of signup, activated and paying

Sync: Update SDKs — code, message (from )

Sync: Update SDKs — fallback_on_block (from )

forward visitor ASN (and entitled bot signals) to origin for analytics classification

follow-up: 2 deferred item(s) (deferred from )

follow-up: 2 deferred item(s) (deferred from )

OFAC geo-block middleware is not mounted in production (OFAC_ENABLED off) and GeoLite2-City.mmdb is absent, so makes the caller IP visible but sanctioned MCP callers are still not blocked until both are provisioned (pre-existing, deploy prerequisite) (review finding — )

/vs-style comparison CTAs and a playground embed on the top 20 blog posts

rebuild /best-web-scraping-api to reach page 1 and rewrite /alternative-to/oxylabs title and meta

follow-up: 1 deferred item(s) (deferred from )

weekly 60-prompt, 5-engine AI share-of-voice panel with competitor and cited-domain capture

expand /vs comparison pages — missing competitors, pricing and FAQ tables, blog internal links

activation sprint — onboarding that is hard to skip and pre-fills a first scrape known to succeed

publish LangChain, LlamaIndex and CrewAI integrations under discoverable package names

15-min status reports carry the digest section; replay 2026-10-21 alert mix

route Alertmanager warnings to the status-report digest instead of Discord

page standing conditions once per state transition, not per cycle

expected-class registry with escalation guards (plan-limiter 429s, origin-side 5xx, DNS 503, customer-capped)

execute the interstitial's wasm-bindgen module in the anti-bot system QuickJS device check (FeRLcb)

Growth & Health owner page with weekly KPIs and the unused internal service analytics snapshots

wire error-trigger.sh and log-monitor.sh Discord sends through the alert router (hotspot wiring only)

analytics_daily rollup and v_user_ltv_by_source view as the single query surface

route data-plane/backup senders through the alert router (pg-*, wal, amcheck, backup scripts, restore drills)

referral prompt at first successful scrape, under the cluster grant cap

route host/container/edge senders through the alert router (memory, redis, uptime, ssl, cloud provider-health, heartbeat, etc.)

full UTM capture, first- and last-touch attribution, and one ai_assistant channel

retroactive temp-mail account sweep as an owner-approved production action

shared Discord alert router and ALERT_ROUTING.md policy

route GitHub workflow Discord posts through the alert policy and widen the sender-classification guard

route internal service Discord alerts through the page/digest/suppress policy

cloud provider 15-min report shows paid success per tier via an cloud provider-published cache status key

derive the anti-bot system device-check environment from the shared persona, plus a parity test against the real-browser capture

paid_success_slo per-tier and TierSuccessRateLow count dead-origin/404 outcomes as T3/T4 failures (T3 reads 21-38%, ~83% excluding origin-side)

base-backup.sh writes a new dated, ownerless P1 internal tooling every night and never closes it

refresh growth materialized views daily, base activation on usage data, fix capped overview count

storage trend alerts read nightly backup peaks and deploy pulls as growth (StorageCategoryDaysToBudgetLow, DiskGrowthWarning)

pg-error-monitor vacuum-stale prefers an old autovacuum over a newer manual VACUUM and never closes its internal tooling

anti-bot system rejects the QuickJS device-check payload 19/19 and each failed check burns the exit — shadow the check, then fix the persona by differential capture

follow-up: 3 deferred item(s) (deferred from )

follow-up: 5 deferred item(s) (deferred from )

0 LLM successes since 6e90e2a — primary nemotron-3-super:free removed (404) and remaining free pool 429s; consecutive_failures=109

follow-up: 1 deferred item(s) (deferred from )

failed T4 browser runs record no proxy bytes (basis=unknown), keeping ValidResultEconomicsCoverageDegraded firing after

deploy never delivers CF_WORKERS_ANALYTICS_TOKEN — new cf-workers-quota cron fails every 15 min after promotes

real lastmod dates in place of the build-date stamp, and a sitemap index split by page type

DDG Lite challenged (202 anomaly) on 100% of attempts after proxy egress; every search falls back

rotate CLOUDFLARE_API_TOKEN to the scope manifest

cache pool 'Too many connections' fails structure handoff of completed scrapes; jobs hang ~20 min until stuck-recovery (since 2026-10-20)

replace '5,000 free scrapes' with an honest '$1 free credit' offer and per-tier request counts

zero-organic SERP pages fail as parse_error 504 with no escalation or logging (~15% of /v1/search)

GEO citation monitor has never written a row in production

honest "up to 5,000 free requests" copy in llms/agent/API/herald/email surfaces + repo-wide guard

honest "up to 5,000 free requests" copy in guides, docs, app pages and root metadata

honest "up to 5,000 free requests" copy in programmatic comparison/SEO data and pages

lead the free offer with "Up to 5,000 free requests" in shared CTAs and constants

derive the window-global presence bitmap per persona family in the anti-bot system QuickJS device check (oX2m3T)

emulate Function.caller through hooked natives in the anti-bot system QuickJS device check (cyZ0R9, uwUbv3)

answer the font-metric probe from persona text metrics in the anti-bot system QuickJS device check (jlHREb)

emulate the CSS animation timeline in the anti-bot system QuickJS device check (zi4P5h and second animation-frame samples)

populate document.fonts from the interstitial's linked font-face stylesheet in the anti-bot system QuickJS device check (i1Yjh2)

P1: backup restore drill failed (2026-10-22)

daily_rollup delivery overwrites one S3 object per job; deliver one file per bulk submission

re-engagement bonus must check fraud, disposable and cluster eligibility before email and grant

cap welcome grants per signup cluster (email domain + IP /24 + device), not per IP alone

repin production-action executor past

cut routed pages in the 2026-10-21 replay toward ~10 without reclassifying genuine incidents

follow-up: 1 deferred item(s) (deferred from )

T4 bills the bot protection system Edge deny page as success — nav-status tracker keeps an aborted 3xx reload status; edge-deny body gate can't read a marked-up reference ID

country fences without region wording never classify as geo_restricted — CF per-request beacon token breaks body fingerprints; invariance only runs inside the bot protection system/CF identity rotation

OAuth authorize/consent redirect to the standalone bind origin http://[server]:3000 — browser OAuth flow broken in prod

The test fixture builds via and now sets the attributes provides, which 's deferral path reads.

| File | Change | Requirements |.

The page query in mode does not bound its work by the page size:.

Both workflow selectors (introduced in c203331a45, "make logical publication atomic") copied the writer's strict whole-namespace inventory validation instead of the reader semantics of th.

These checks predate the fail-closed recovery contract (/).

log-monitor has no stable logical identity per issue class.

The Chrome binary install was written as a floating (introduced in b509d0ae8c, moved in c4a422b001/c320b45fc7 for root/su reasons) while the identity was later frozen at a HTTP client-verified major (#2.

Severity is a one-way ratchet: the filename and Severity line record the peak, never the current observation, and an age/cycle timer converts mere persistence into Critical.

This is policy, not a code defect.

The Workers request quota was never instrumented.

Plus 10 internal improvements for better reliability and performance.

short-expiry terminal-verdict memory per (host, vendor_class, exit_class) with per-host singleflight

fleet-shared host×country outcome memory, per-host exploration singleflight, all-countries-denied negative cache, fleet prior

fail fast on terminal vendor verdicts: skip redirect wait, nodriver and advanced browser; one gated alternative leg; coded unbilled failure (shadow-first)

record exit country and a bounded body browser compatibility on every tier attempt; geo_restricted failure category

T3 egress-country exploration on geo_restricted (RC_EGRESS_COUNTRY, ≤2 extra T3 legs, cohort rollout)

count T4-deferral queue time against the T4 deadline; fail before browser/proxy acquisition when no useful attempt fits

per-(vendor_class, verdict, engine, exit_class) outcome/cost cells and Wilson-LB alternative-leg gate (record-only)

geo_restricted recovery subtype (HTTP 451, CF 1009, regional-restriction text, identity-invariance signature), shadow only

first-class vendor verdicts — parse/persist anti-bot system rt/t/view and map verdicts to recovery classes

replace hand-picked TLD_COUNTRY_MAP with a complete ISO 3166 ccTLD table

Milestone: Remote MCP Server with OAuth → staging

bingoplus.com PH-exit residual: T4 cleared page rejected as semantic soft_block

do not book IP/ASN reputation from geo_restricted outcomes; add exit country to the routing arm and outcome key

identify the device signal anti-bot system rejects (exit vs payload vs persona) and bisect the skruvat.se 53/61→0/8 regression

route CF headed-promote probe through UnifiedDetector — non-CF pages must never take the headed slot

keep the T3 stage outcome in tier_failures after a T4 deferral (tier4_pool_exhausted_requeued)

engine selection by active challenge vendor, not passive CDN marker; enable anti-bot system T4 engine A/B

ship log-monitor-internal tooling-keys.sh in the root cron runtime

upgrade certifi to 2026.10.19 and Pillow to 13.0.0 (Trivy CRITICAL/HIGH)

close milestone 115 merge gaps found on deploy

reconcile guards with intended staging changes (deploy )

omit CSP form-action on /oauth/consent so consent redirects reach the client

oauth_tokens classified bounded but every refresh inserts a row and nothing ever purges expired/rotated/revoked tokens or grants (review finding — )

mcp-remote forwards no end-user IP to the API so OFAC geo-block fails open and IP allowlist/attribution see the container IP (review finding — )

Singleflight lock keyed by host only while verdict keyed by host+vendor+exit_class; followers on a different cell wait for the holder with no possible hit (review finding — )

correct OAuth2 guide step count, revocation timing and client registration

Scheduled jobs now track consecutive failures and automatically pause when targets remain broken. You will receive instant notifications so you can investigate before wasting API credits.

You can now enable opt-in fallbacks for high-tier extraction requests. If a request encounters a challenge or timeout, the system automatically attempts secondary extraction modes.

Request pacing now dynamically adjusts based on real-time target domain responsiveness. This improves overall extraction throughput while minimizing blocks and strain on target servers.

Consecutive failures on specific URLs now trigger automatic back-off delays. This halts repetitive retries against unreachable endpoints and conserves your API credit quota.

Sticky session connections now enforce concurrency limits across requests. This guarantees consistent session reuse and improves extraction success rates during multi-step web flows.

Plus 58 internal improvements for better reliability and performance.

Resolved an issue where responses returning challenge pages were marked as successful extractions. Requests blocked by target site challenges are no longer charged full credits.

Fixed a ledger calculation discrepancy for batch extractions. Failed individual jobs inside a batch now correctly receive credit refunds against the initial batch debit.

Fixed cache ordering gaps during credit balance updates. Account balances and API entitlement limits now update immediately in real time following any transaction.

Fixed network egress timeouts occurring on search extraction calls. Search requests now experience consistent response times and higher overall completion rates.

consecutive-failure tracking with notify and auto-pause for all schedule types ( option 4)

opt-in fallback from forced T4 to T3/T3.5 on challenge or timeout ( option 5)

per-(user, URL) back-off after consecutive identical failures ( option 3)

bound concurrency per sticky exit/session (clearance reuse, cookie-cohort, force_sticky)

anti-bot system clearance-store replay (enforcement phase of ) after shadow metrics

surface failure hints and steer examples to render_js="auto" ( options 1+2)

obs-pgstat top [n] wrapper for pg_stat_statements attribution

stable check keys and fail-closed recovery closer for log-monitor internal tooling, plus stale-memo dry-run report

alert on Cloudflare Workers daily request quota before Error 1027

detailed-logs crawl grouping via partial index and keyset pagination

exhaustion-nurture scans read balances from credit_balance_cache instead of re-summing credit_ledger

add FailureBackoffErrorResponse to the published 429 contract test and docs (deploy blocker)

drop per_domain_concurrent as a customer 429 source; bound per-domain by account max_in_flight

anti-bot system device-check HTTP client sessions bypass the egress guard — route via proxied, SSRF-checked egress (deploy )

follow-up: 2 deferred item(s) (deferred from )

update DomainConcurrencyDeferralHigh promtool expectation after description change (deploy )

expire_abandoned_drafts while-True batch loop hangs the internal service test suite (and is unbounded in prod); add pytest-timeout (deploy )

anti-bot system device-check HTTP client session sends impersonate without browser compatibility kwargs — Firefox TLS incoherence on T3 (deploy )

api-changelog behaviour-change test must not assume the entry is in the top release block (deploy )

test-check-deploy-contracts.sh bundle helpers break on 's shared line in bootstrap-body.sh (deploy )

test lists mandate-banned words as literals and trips the wording scanner (deploy )

schedule auto-pause counts platform queue-full and stale-claim failures — capacity incident pauses customer schedules

follow-up: alterlab_api_worker_rss_bytes soak verification of api memory fix (deferred from )

production-action executor crashes on first real dispatch — maintenance_windows FK target not loaded (NoReferencedTableError)

T4 `page.route('**/*')` adds Pragma/Cache-Control no-cache to every request

publish recovery outcomes for internal service, AI engine, LLM and log-analysis error-trigger checks

logical-generation selectors fail closed on an in-progress generation instead of skipping it (shadow dry run raced the nightly upload)

publish recovery outcomes for billing/payment error-trigger checks

pg cache-hit alert attributes misses to recurring statements and ignores calls=1 one-offs

degrade gracefully and alert accurately when shared OpenRouter free-tier budget is exhausted

CLOUDFLARE_API_TOKEN lacks Cache Settings Edit and Single Redirect Edit — edge cache and redirect rules not deployed

worker image installs floating latest Chrome (155) while identity constants claim Chrome 150 — pin and gate the binary version

stop labelling warning-threshold counts Critical in error-trigger internal tooling and list unlinked memos first in aging handoff

re-measure Cloudflare T4 after and set telemetry-gated policy for cloudflare_js and cloudflare_blocked

cache recreate onto volatile-lru — OOM-on-write exposure, cloud provider reconnect gap, state-dependent rollback (review finding — )

origin-lock installer self-probe always fails from the host — lock is effective (external probe 403), deploy goes red

publish recovery outcomes for container, edge and database error-trigger checks

cache volatile-lru rollback runbook lost writer-fencing contract after — test_redis_backfill_job_ttl red on deploy

RESERVED_HARD-promoted hosts bypass cohort sticky identity and share one rotating reserved session, breaking IP-bound clearance reuse (review finding — )

enforce vendor-class pool routing from exit_reputation — replace build-time RESERVED_HARD domain classification ( / enforcement phase)

Adaptive Unblocking Control Loop → staging

require cross-egress consistency as positive evidence before delivering non-placeholder thin T4 pages

spec + epic tracker — support@ inbound filtering and in-house CRM leads

Adaptive Unblocking Control Loop → staging

merge post-auth, scrape-per-key and concurrency checks into one Lua call

win back the lapsed top-revenue accounts after reliability fixes

activate cloud provider storage backup lane (relay account, keys, host-key pin, monitor, restore drill)

monitor read access to all internal tooling, /var/log/monitoring, single-instance state and node-exporter textfiles on both hosts ( part 4)

earn anti-bot system clearance on cold T3 contact (in-house QuickJS device-check on the unified T3 path)

adaptive per-domain backoff on target strain signals; replace static preemptive domain caps

hint render_js="auto" when a forced-render job fails on challenge or timeout ( option 1)

obs-internal tooling close — the monitor account's single write (open→closed rename with lifecycle note) ( part 3)

follow-up: 1 deferred item(s) (deferred from )

Worker quota exhaustion must never take marketing pages offline

Incident-ID issue-link cron that sets canonical_issue on open Critical/High internal tooling

obs-redis read-only allowlist wrapper (SLOWLOG/INFO/LATENCY/CLIENT LIST) absorbing obs-redis-keyspace ( part 2)

make monitor_ro inherit pg_monitor/pg_read_all_stats so pg_stat_statements shows query text ( part 1)

bot protection system Bot Manager denies deep links without interaction telemetry — route to a T4 -warm session (vendor class, replaces identity rotation)

prevent cannibalizing blog posts with a durable intent registry enforced on every generator

consolidate existing cannibalizing blog posts — audit never runs in production

4h45m cloud provider host outage (2026-10-16 21:39–02:24Z) went undetected; no off-host availability probe

cloud provider scraper-lite stuck in autoheal restart loop for ~6h (restarts never recovered; only recreate did)

Phantom T4.5 returns bot protection system Edge deny page as a billed success — re-check drops T3 status/headers

Inline-path options snapshot hand-mirrors the worker payload and records has_* flags as false for headers/body/extraction/custom keys (review finding — )

pre-T3 origin probe counts a stalled proxy CONNECT (tunnel_timeout) as dead-origin witness, fast-failing live sites (review finding — )

paid-request failure rate regressed from ~5% (Apr) to 30-40% (Sep-Oct)

10x capacity headroom on existing hardware — 0.8 -> 8 jobs/sec with zero 5xx (supersedes , fitted to the retired 8 vCPU box)

bot protection system identity-rotation, stage-span and event-loop test failures on deploy

mandate-wording guard breaks after waf_block marker moved to (deploy )

map new cron jobs in the RSI periodic-job inventory (deploy )

renumber duplicate 0451 rollout-cohorts migration blocking deploy

worker pytest session dies at 95% with no summary on deploy

wire five unaccounted scripts/tests suites into CI jobs

Chrome upgrade safety gate cannot parse refactored

add external-availability-probe and storage-category-metrics to trusted-runtime closure

classify status-0 T3 failures by cause and deny T4 only on a confirmed reputation reject

blog generation publishes nothing when the free LLM is rate-limited because exhausts the static topic pool

DatabaseBudgetStatusUnavailable fires permanently because worker processes export alterlab_db_budget_status_ok=0

render_js browser-tier jobs run on WORKER_TYPE=http lite workers and OOM-kill them

internal tooling-bridge leaves memos open when an alert vanishes without a resolved webhook; reconcile against Alertmanager

OpenRouterKeyLimitLow and openrouter_balance_low fire permanently against the deliberate $0.20 free-only key cap

cache stall cascades into cluster-wide DB budget exhaustion because failed lease releases leak slots until expiry

T4 content-validation rejections and time-budget stops are recorded as unknown failure_category

job-failure alerts count origin 404 (not_found) and 5xx (target_server_error) as platform failures

publish recovery outcomes for load balancer, scanner, security_rate_limit and api_cf_routing checks so memos auto-close

treat per-account 429s and customer-input 400s as expected so traefik_4xx_flood and security_rate_limit can clear

log-monitor titles every ERROR Critical Infrastructure Failure and re-pages every 15 min for a standing condition

web error scan pages Critical on Next.js server-action scanner probes; filter and classify them

refund_retry_all_stores_failed log drops its extra payload, so the documented recovery can't be done from API logs

apex DNS check alerts DNS Mismatch on every Cloudflare edge IP; port the proxied-path check from check_api_dns

log-monitor.sh exits 1 when the LLM reply lacks ROOT_CAUSE, failing every ERROR cycle and the heartbeat

LLM auth/connection failure alert fires on free-tier 429s; classify as degraded and auto-close on recovery

worker package manager.lock regenerated with package manager 2.4.3 breaks pinned 2.2.1 installs (deploy )

wire 3 new scripts/tests suites and make the wiring guard runnable locally (deploy )

entitlement cache epoch ordering and invalidation gaps (review finding — )

remove DDG Lite direct-egress burst timeout regression

Cloudflare Worker route audit has never passed — token lacks Workers read scopes

event-loop isolation from hangs the worker suite and breaks asyncpg diagnostics tests

r2-evidence-lifecycle token recipe must use exact internal tooling YAML path (deploy )

auto-resolve cloud provider worker-only error-trigger memos

successes whose body is a classified challenge are billed in full and invisible to FalseSuccessRateHigh

provision internal service analytics/GSC/analytics credentials in prod secrets and fail loud instead of reporting "stale"

drop the per-IP rate-limit-global middleware from web-router so crawlers on public pages are never 429'd at origin

dev.to syndication returns 401 and 4,100 legacy content drafts are never dispatched or expired

remove bm-verify from vendor-marker extras now derived from bot protection system provider (deploy )

refund all-stores test vs budget-exempt fallback; economics contract test vs refactor (deploy )

update deploy checkout-index and cutover-gate contract tests for and (deploy )

map ovh-internal tooling-issue-link in rsi-coverage.json (deploy )

lost-data alerts (RefundRetryAllStoresFailed) must not auto-close their internal tooling on resolve

cache public marketing HTML so crawl volume never reaches origin (keep crawlers unthrottled)

bot protection system bm_mi and vendor-script markers on complete real pages are classified as challenges

register test-deploy-internal tooling-issue-link-token-contract.sh with the CI suite guard (deploy CI)

follow-up: /etc/server/sites-enabled/alterlab.conf (deferred from )

register internal tooling-issue-link.sh in the trusted-runtime closure and make the closure check part of lane validation

proxy cost provenance marks zero-byte metered attempts unknown, so ValidResultEconomicsCoverageDegraded fires permanently

heartbeat-watchdog and ssh-guard drift alerts never create internal tooling; deploy should re-install guard shims

Web: SEO structured data, redirects and CLS

log-monitor billed-5xx 'retained' check matches any customer's debit, keeping a standing Critical REVENUE IMPACT memo

follow-up: Worker-scope WebGL/deviceMemory coherence (deferred from )

log-monitor tier-stats Lua SCAN freezes cache 2-6s every 15 min — root cause of the 2026-10-19 06:00Z outage

DB budget cap strands claimed jobs while database is idle

node-exporter cannot read the textfile collector dir under 0700 /var/lib/alterlab, so all textfile metrics are invisible

ChallengeCookieCacheZeroHitRate can't distinguish nothing-stored-to-reuse from a broken cache

Bing submit/verify ignores the 100-URL daily quota and ThrottleHost, so discovery reports 0 verified indexed

OPERATOR RUNBOOK: restore production recovery posture (ordered — backup, database, credential, cloud provider)

obs-redis-keyspace reports "no running api/worker container" on cloud provider while obs-queues works

settlement checker flags failed batch jobs whose job-keyed refund covers the batch-keyed base debit

test_force_t4_preserves_executed_penetrator_engine hangs on an unreaped child process (deploy )

T3 reachable SLI buckets transport no-response failures as content, hiding the real T3 gap

cache 'Too many connections' ~55/h on stream trim and alert processing paths

log-monitor runs an LLM agent with shell/write tools on the prod host, fed attacker-influenced log text

refund compensation must persist when cache or the DB budget gate is unavailable (all-stores failure drops refunds)

a single Reddit 401 trips a 7-day permanent circuit and the alert mislabels it as a ban

Browser-rendered requests now cache static assets like scripts and stylesheets. This significantly reduces response latency and accelerates page rendering.

Single-page app shells, directory listings, and minimal landing pages are now recognized and delivered accurately. Content is returned immediately without being misclassified as blocked or triggering redundant processing.

Eliminated redundant retry loops when requests encounter pages with naturally minimal markup. Scrapes now complete much faster when pages contain limited body text or basic layouts.

Plus 1 internal improvement for better reliability and performance.

Resolved an issue where lightweight tracking scripts injected by target site security systems were falsely flagged as challenges. Scraped pages now return their complete response body without false block errors.

Fixed an operational issue where browser rendering processes could fail on worker nodes without active virtual displays. Display environments are now verified prior to execution to guarantee seamless rendering.

shared RFC 9111 cache for T4 browser subresources behind an A/B

require a live X server before headed launches

stop classifying Cloudflare's bot-detection beacon as a challenge

deliver and bill genuine thin origin pages

stop T4 proxy re-rolls on sparse-page soft blocks

stop the analytics retry interval when window is gone (deploy-blocking test flake)

linear noscript removal in rendered thin-page shape check

linear noscript text strip, id-anchored app-mount scan ( re-review pass 8)

linear empty app-root scan, bounded app-shell/noscript scans ( re-review pass 7)

anchor server_default phrase, exempt short placeholders from empty-shell gate ( re-review pass 6)

placeholder delivery off app shells and Han denials, restore recall ( re-review pass 5)

positive evidence for tiny and uncovered-script thin pages ( re-review pass 4)

asset cache off customer egress + tiny origin page delivery ( re-review pass 3)

staging re-review findings for (asset-cache arm scope, sparse-deny retries)

address staging review findings on

force-remove cloud provider slots restarted during drain

Plus 3 internal improvements for better reliability and performance.

Scraping requests for identical URLs using different output formats now correctly bypass cached data generated in other formats. This ensures you receive properly formatted scraped content every time you request a new output format for a previously scraped page.

key repeat-detection cache on requested formats

Scraper jobs now store a snapshot of submitted options like JavaScript rendering, timeouts, and cost caps. This allows you to easily audit and reproduce request behavior from your dashboard.

When target servers are unreachable, requests now fail fast instead of running extended retries. This reduces response times by up to 80 seconds on offline targets and avoids extra charges.

Domain health tracking now ignores user-level rate limits and skipped retry steps. This prevents false domain pauses and keeps your extraction pipelines running smoothly.

Plus 11 internal improvements for better reliability and performance.

Pages containing large inline hydration state are no longer incorrectly flagged as empty shells. This ensures complete content extraction on modern web applications.

Target site server errors and 5xx status codes are now accurately classified as origin errors rather than security blocks. This improves status reporting clarity and retry routing.

Connection timeouts during target site checks are now handled more accurately. Inconclusive proxy connections are distinguished from down targets for predictable request resolution.

Fixed an issue where automatic balance top-ups could stay stuck pending after payment gateway errors. Failed recharges now resolve cleanly so you can retry or update billing details.

High server load no longer triggers false unauthorized errors or IP security flags on valid API keys. When authentication capacity is briefly saturated, the API now returns 503 with error auth_unavailable and a Retry-After header instead of 401, and WebSocket connections close with code 1013 (Try Again Later) instead of 4001. Retry after the indicated delay; do not rotate the key. Newly created or regenerated API keys also authenticate immediately without lockouts.

Requests that set advanced.render_js to true with cost_controls.max_tier below "4" are now rejected with 422 at submission, before any charge, on /api/v1/scrape, /api/v1/batch items and /api/v1/crawl. Browser rendering runs on tier 4, so set max_tier to "4", omit it, or use render_js "auto".

deliver synthetic probe credentials to cloud provider monitoring config

T3 attempt-success SLI on a legal-reachable denominator — track the 90% T3 goal

vendor-class Chromium vs Firefox T4 engine A/B switch (anti-bot system class)

persist a sanitized request-options snapshot per job (tier cap, render_js, timeout, BYOP flag, cost caps)

repair stale tests failing on deploy

retain pre-debited use_proxy add-on in worker reconciliation

block listed abuser IPs before the WAF skip rule and stop trusting crawler UAs

revert cloud provider workers when the cloud provider deploy tail fails after the overlapped rollout

avoid bare else under drone-ssh script_stop

CF edge rate-limit exporter fails every poll — zone lacks access to GraphQL path

SLA rules and SlaMeasurementAbsent select entrypoint=websecure; prod traffic is on web

synthetic probe unverified since 10-08 — SYNTHETIC_PROBE_API_KEY not provisioned

pg-compose-drift can't read shared_preload_libraries as NOSUPERUSER alterlab

discovery verifier status write fails with AmbiguousParameterError on reused :status bind

deterministic sleep counts in T4 solver deadline tests

anti-bot system T4 Firefox-route clearance regressed with the Windows Firefox identity (allegro 5/5 to 0/6)

validate impersonation bundles against real-browser captures; choose vendor engine/profile by telemetry (firefox135 23 majors stale, chrome150 JA4 skew)

static cloudflare_blocked and bot protection system edge-deny are identity-specific — rotate identity before concluding IP reputation